TypeScript support for testing is still in closed alpha - please contact us if you would like to use it!

Common pytest issues

Make sure your test file, the Test class AND/or the test function is prefixed with Test or test respectively. Otherwise, pytest will not pick up your tests. E.g. test_foo.py, TestFoo, test_foo

Make sure you are running these commands from your python virtual environment (or poetry shell if you use poetry).

No module named baml_lib.

Try running poetry run python -m pytest -m baml_test instead if you are using poetry. Double check you are in a poetry shell and that there is a baml_client and baml dependency in your project

Helpful Pytest commands

# From your project root
# Lists all tests with the baml_test marker
pytest -m baml_test --collect-only
# From your project root
# Runs all tests
# For every function, for every impl
pytest -m baml_test

To run tests for a subdirectory

# From your project root
# Note the underscore at the end of the folder name
pytest -m baml_test ./your-tests-folder/

To run tests that match a specific name E.g. if your test is called “test_thing_123”, the following command will run this test:

# From your project root
pytest -m baml_test -k thing

You can read more about the -k arg of pytest here (PyTest Docs)

-k will match any tests with that given name.

To run a specific test case in a test group

# From your project root
pytest -m baml_test -k 'test_group_name and test_case_name'

Testing multiple impls using fixtures (advanced)

We automatically export a pytest fixture for each of your defined functions that will automatically convert a test function into N test functions, where N is the number of impls you have defined for that function.

Instead of writing:

@baml_test
async def test_impl1_foo():
  assert b.Foo.get_impl("v1").run(...)

@baml_test
async def test_impl2_foo():
  assert b.Foo.get_impl("v2").run(...)

You can import the fixture in:

Test Function
# Import your baml-generated functions
from baml_client import baml as b
# Import any custom types defined in .baml files
from baml_client.baml_types import Sentiment, IClassifySentiment

# This automatically generates a test case for each impl
# of ClassifySentiment.
@b.ClassifySentiment.test
async def test_happy_user(ClassifySentimentImpl: IClassifySentiment):
    # Note that the parameter name "ClassifySentimentImpl"
    # must match the name of the function you're testing
    response = await ClassifySentimentImpl("I am ecstatic")
    assert response == Sentiment.POSITIVE

Grouping tests

You can also group tests in test classes. We won’t use the impl-specific fixture in this case for simplicity. The dashboard will show you grouped tests as well if you use this method, which can be handy for organizing your tests.

@baml_test
class TestClassifySentiment:
    async def test_happy_user(self):
        response = await b.ClassifySentiment("I am ecstatic")
        assert response == Sentiment.POSITIVE

    async def test_sad_user(self):
        response = await b.ClassifySentiment("I am sad")
        assert response == Sentiment.NEGATIVE

The class name must start with “Test” or it won’t get picked up by Pytest.

Parameterization

You can also parameterize your tests. This is useful if you want to test a function with a variety of inputs.

The parameters to the parametrize annotation indicate the name of the arguments sent into the test.

...
import pytest

@baml_test
class TestClassifySentiment:
    @pytest.mark.parametrize(
        "input, expected",
        [
            ("I am ecstatic", Sentiment.POSITIVE),
            ("I am sad", Sentiment.NEGATIVE),
        ],
    )
    # Note the name of the args matches what we defined in the parametrize annotation. The first arg is "self" since this is inside a class.
    async def test_sentiment(self, input, expected):
        response = await b.ClassifySentiment(input)
        assert response == expected

Or alternatively, you can group things by sentiment. You dont need to use a class.

import pytest
from baml_client.testing import baml_test
from baml_client import baml as b
from baml_client.baml_types import Sentiment

@baml_test
@pytest.mark.asyncio
@pytest.mark.parametrize(
    "input",
    [
        "I am ecstatic",
        "I am super happy!",
        "I am thrilled",
        "I am overjoyed",
    ],
)
async def test_happy_sentiments(input):
    response = await b.ClassifySentiment(input)
    assert response == Sentiment.POSITIVE

@baml_test
@pytest.mark.asyncio
@pytest.mark.parametrize(
    "input",
    [
        "I am sad",
        "I am angry",
        "I am upset",
        "I am frustrated",
    ],
)
async def test_sad_sentiments(input):
    response = await b.ClassifySentiment(input)
    assert response == Sentiment.NEGATIVE

You can read more about it here (PyTest Docs)