RetroSearch Browse

Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from https://ibm.github.io/prompt-declaration-language/contrib below:

Contribute - Prompt Declaration Language

Contributing to PDL

You can report issues or open a pull request (PR) to suggest changes.

Reporting an issue

To report an issue, or to suggest an idea for a change that you haven't had time to write-up yet:

Review existing issues to see if a similar issue has been opened or discussed.
Open an issue. Be sure to include any helpful information, such as version used, error messages, or logs that you might have.

To report vulnerabilities privately, you can contact the authors by email (see pyproject.toml).

Suggesting a change

To suggest a change to this repository, submit a pull request with the complete set of changes that you want to suggest. Before creating a PR, make sure that your changes pass all of the tests and add tests for new features.

The test suite can be executed with the following command in the top-level folder:

Also, please make sure that your changes pass static checks such as code styles by executing the following command:

pre-commit run --all-files

Development environment PDL development

Follow the following instructions to set up a dev environment to get started with contributing to PDL.

Create a fork of https://github.com/IBM/prompt-declaration-language
Clone your fork

Set up a Python virtual environment and install dependencies

cd prompt-declaration-language
python -m venv .venv
source .venv/bin/activate
pip install -e .

Test that you can run an editable version of PDL
```
pdl examples/demo/1-hello.pdl

Hello
Hello
```

Documentation updates

When you make changes to PDL, ensure to document any new features in the docs section. You can serve the docs locally to preview changes.

Install the required dependencies for documentation.

Then serve the docs to load a preview.

You are all set!

Run examples

PDL executes nightly runs for Run Examples, which searches for all the .pdl programs in the repo and runs the interpreter against each file. The config file for Run Examples describes how to handle each file. There are four conditions:

skip: a list of PDL files that are skipped in Run Examples
with_inputs: PDL files that require user input. Each file name is mapped to two fields that describe how inputs are patched
stdin: separated by a newline, each line represents the user input (string)
scope: scope for the PDL program (a dictionary of key-value pairs)
expected_parse_error: a list of PDL files that expect parse errors
expected_runtime_error: a list of PDL files that expect runtime errors

If you wish to make a contribution to PDL and modify or add any PDL program to the repo, it is important that you provide the new expected results for those files so that the Run Examples nightly test does not break.

Local dev

Under check, you can provide a list of files that you want to run Pytest against. If you leave it empty (check: or check: []), then by default, Pytest will be executed against all files in the repo, except for those under skip. For local development, it is useful to only test against a subset of files so that Pytest executes faster.

If you expect the files to produce a different result, setting update_results: true will automatically create a new file under tests/examples/results capturing the new output for each of the file in check. It is useful to set this field to true before opening a PR.

Run this Pytest command for Run Examples, which is the same command for the nightly test.

pytest --capture=tee-sys -rfE -s tests/test_examples_run.py --disable-pytest-warnings

Opening a pull request

A slight variation in the Python version and OS environment can cause a different LLM response, thus Run Examples might fail because it uses exact string matching for PDL outputs.

When you open a pull request (PR) against the main branch, a series of status checks will be executed. Specificially, three Run Examples test will be initiated against the PDL files you have added and modified as part of the PR. If there's any variation, you should manually examine the results produced in the Github Actions environment, then copy and paste the results to a new file, and push another commit to your PR so the CI can pass. Be aware of whitespaces in between sentences.

Below is what the CI might look like if the results differ. If the results look reasonable, you should 1. copy and paste the result for that file 2. create a new file in tests/results in the format of tests/results/<path/to/file>.<i>.result where <i> should be incremented from the highest number of <i>.result in the folder

If the test fails, look for something similar like the following:

tests/test_examples_run.py 
============================================================================
File that produced wrong result: examples/demo/1-hello.pdl
Actual result (copy everything below this line):
✂️ ------------------------------------------------------------
Hello
Hello
-------------------------------------------------------------

============================================================================
File that produced wrong result: examples/demo/2-model-chaining.pdl
Actual result (copy everything below this line):
✂️ ------------------------------------------------------------
Hello
Hello
Did you just say Hello?
Yes, I did. It's a common greeting, similar to how humans might respond when they first interact with an artificial intelligence like me. How can I assist you today?
-------------------------------------------------------------
F

Note: Your PR should always set update_results: false before merging, though the CI pipeline will not attempt to write the results to your PR even if you forget to set this field to false.

Here's a preview of the current configuration file for Run Examples:

update_results: true
check:
  - examples/tutorial/defs.pdl
  - examples/tutorial/function_definition.pdl
  - examples/tutorial/function_empty_context.pdl
  - examples/tutorial/calling_llm_with_input_messages_var.pdl
  - examples/tutorial/local_computation.pdl
  - examples/tutorial/role.pdl
  - examples/tutorial/lastOf.pdl
  - examples/tutorial/muting_block_output.pdl
  - examples/tutorial/function_optional_params.pdl
  - examples/tutorial/function_call_in_jinja.pdl
skip:
  - examples/demos/react.pdl
  - examples/cldk/cldk-assistant.pdl
  - examples/gsm8k/gsm8.pdl
  - examples/gsm8k/gsm8k-plan.pdl
  - examples/gsm8k/gsm8k-plan-few-shots.pdl
  - examples/gsm8k/gsm8k-tot-few-shot.pdl
  - examples/gsm8k/gsm8k-tot-multiplan.pdl
  - examples/gsm8k/gsm8k-tot.pdl
  - examples/intrinsics/demo-hallucination.pdl
  - examples/tutorial/programs/demo-hallucination.pdl
  - examples/rag/pdf_index.pdl
  - examples/rag/pdf_query.pdl
  - examples/rag/rag_library1.pdl
  - examples/tutorial/structured_decoding.pdl
  - pdl-live-react/src-tauri/tests/cli/code-python.pdl
  - pdl-live-react/demos/error.pdl
  - pdl-live-react/demos/demo1.pdl
  - pdl-live-react/demos/demo2.pdl
  - pdl-live-react/src-tauri/tests/cli/read-stdin.pdl
  - examples/granite-io/granite_io_transformers.pdl
  - examples/granite-io/granite_io_thinking.pdl
  - examples/granite-io/granite_io_object.pdl
  - examples/granite-io/granite_io_openai.pdl
  - examples/granite-io/granite_io_hallucinations.pdl
  - examples/demos/granite_io_hallucinations.pdl
  - examples/optimizer/mbpp.pdl
  - examples/optimizer/fever.pdl
  - examples/optimizer/gsm8k.pdl
  - examples/optimizer/bea19.pdl
  - examples/optimizer/eval_levenshtein.pdl
  - examples/requirements/email.pdl
with_inputs:
  examples/tutorial/programs/chatbot.pdl:
    stdin: |
      What is APR?
      yes
    scope: null
  examples/chatbot/chatbot.pdl:
    stdin: |
      What is APR?
      yes
    scope: null
  examples/demo/7-chatbot-roles.pdl:
    stdin: |
      What is APR?
      quit
    scope: null
  examples/tutorial/input_stdin.pdl:
    stdin: |
      What is APR?
      yes
    scope: null
  examples/tutorial/input_stdin_multiline.pdl:
    stdin: |
      Hello
      Bye
    scope: null
  examples/input/input_test1.pdl:
    stdin: |
      Hello
    scope: null
  examples/input/input_test2.pdl:
    stdin: |
      Hello
    scope: null
  examples/tutorial/free_variables.pdl:
    stdin: null
    scope:
      something: ABC
  tests/data/optimizer_gsm8k.pdl:
    stdin: null
    scope: 
      model: watsonx_text/ibm/granite-3-8b-instruct
      prompt_pattern: cot
      num_demonstrations: 0
      demonstrations: []
      question: "The sky currently has 4 times as many cirrus clouds as cumulus clouds, and 12 times as many cumulus clouds as cumulonimbus clouds. If the sky currently has 3 cumulonimbus clouds, how many cirrus clouds are in the sky at this moment?"

expected_parse_error:
  - tests/data/line/hello.pdl
  - tests/data/line/hello1.pdl
  - tests/data/line/hello4.pdl
  - tests/data/line/hello7.pdl
  - tests/data/line/hello8.pdl
  - tests/data/line/hello10.pdl
  - tests/data/line/hello11.pdl
  - tests/data/line/hello31.pdl
expected_runtime_error:
  - examples/callback/repair_prompt.pdl
  - examples/demos/repair_prompt.pdl
  - examples/tutorial/type_list.pdl
  - examples/tutorial/type_checking.pdl
  - tests/data/line/hello3.pdl
  - tests/data/line/hello9.pdl
  - tests/data/line/hello12.pdl
  - tests/data/line/hello13.pdl
  - tests/data/line/hello14.pdl
  - tests/data/line/hello15.pdl
  - tests/data/line/hello16.pdl
  - tests/data/line/hello17.pdl
  - tests/data/line/hello18.pdl
  - tests/data/line/hello19.pdl
  - tests/data/line/hello20.pdl
  - tests/data/line/hello21.pdl
  - tests/data/line/hello22.pdl
  - tests/data/line/hello23.pdl
  - tests/data/line/hello24.pdl
  - tests/data/line/hello25.pdl
  - tests/data/line/hello26.pdl
  - tests/data/line/hello27.pdl
  - tests/data/line/hello28.pdl
  - tests/data/line/hello29.pdl
  - tests/data/line/hello30.pdl

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4