baml-cli

test

The test command runs BAML function tests defined in your BAML files. It provides comprehensive testing capabilities including filtering, parallel execution, and various output formats.

Usage

baml-cli test [OPTIONS]

Options

OptionDescriptionDefault
--from <PATH>Path to the baml_src directory./baml_src
--listOnly list selected tests without running themfalse
-i, --include <PATTERN>Include specific functions or tests (can be used multiple times)Run all tests
-x, --exclude <PATTERN>Exclude specific functions or tests (can be used multiple times)None
--parallel <NUM>Number of tests to run in parallel10
--pass-if-no-testsPass if no tests are selectedfalse
--require-human-evalFail if any tests need human evaluationtrue
--dotenvLoad environment variables from .env filetrue
--dotenv-path <PATH>Path to custom .env fileNone

Description

The test command performs the following actions:

  1. Discovers and parses all test cases defined in BAML files
  2. Applies include/exclude filters to select which tests to run
  3. Executes tests in parallel (configurable concurrency)
  4. Reports results with detailed output and assertions
  5. Supports various output formats and CI integration

Test Filtering

The --include and --exclude options support powerful pattern matching:

Pattern Syntax

  • * matches any characters within a name
  • FunctionName::TestName matches a specific test in a specific function
  • FunctionName:: matches all tests in a function
  • ::TestName matches a test name across all functions
  • Multiple patterns can be combined

Examples

$# Run all tests
>baml-cli test
>
># List all available tests
>baml-cli test --list
>
># Run tests for a specific function
>baml-cli test -i "ExtractResume::"
>
># Run a specific test
>baml-cli test -i "ExtractResume::TestBasicResume"
>
># Run all tests matching a pattern
>baml-cli test -i "Extract*::"
>
># Run tests with multiple include patterns
>baml-cli test -i "Extract*::" -i "Classify*::"
>
># Exclude specific tests
>baml-cli test -x "ExtractResume::TestComplexResume"
>
># Combine include and exclude (exclude takes precedence)
>baml-cli test -i "Extract*::" -x "*::TestSlow*"

Parallel Execution

Control the number of tests running simultaneously:

$# Run tests with default parallelism (10)
>baml-cli test
>
># Run tests sequentially
>baml-cli test --parallel 1
>
># Run with high parallelism
>baml-cli test --parallel 20

Environment Variables

The test command automatically loads environment variables:

$# Use default .env file loading
>baml-cli test
>
># Disable .env file loading
>baml-cli test --no-dotenv
>
># Use custom .env file
>baml-cli test --dotenv-path .env.test

Environment variables can also be set directly:

$# Set API keys for testing
>OPENAI_API_KEY=... ANTHROPIC_API_KEY=... baml-cli test

Exit Codes

The test command returns different exit codes based on results:

Exit CodeMeaning
0All tests passed
1Test failures occurred
2Tests require human evaluation
3Test execution was cancelled
4No tests were found to run

Examples

Basic Testing

$# Run all tests in the project
>baml-cli test
>
># Run tests from a specific directory
>baml-cli test --from /path/to/my/baml_src

Development Workflow

$# Run tests for a function you're developing
>baml-cli test -i "MyNewFunction::"
>
># Run specific test while debugging
>baml-cli test -i "MyFunction::TestEdgeCase"
>
># List tests to see what's available
>baml-cli test --list -i "Extract*::"

CI/CD Integration

$# Fail fast on first assertion failure
>baml-cli test --require-human-eval
>
># Allow tests that need human evaluation to pass
>baml-cli test --no-require-human-eval
>
># Run with controlled parallelism for CI
>baml-cli test --parallel 5

Test Discovery

$# See all available tests
>baml-cli test --list
>
># See tests for specific functions
>baml-cli test --list -i "ClassifyMessage::"
>
># See what tests would run with filters
>baml-cli test --list -i "Extract*::" -x "*::TestSlow*"

Test Definition

Tests are defined in BAML files using the test block syntax:

1function ExtractResume(resume: string) -> Resume {
2  client GPT4
3  prompt #"Extract resume information: {{resume}}"#
4}
5
6test TestBasicResume {
7  functions [ExtractResume]
8  args {
9    resume "John Doe\njohn@example.com\nSoftware Engineer"
10  }
11  @@assert({{ this.name == "John Doe" }})
12  @@assert({{ this.email == "john@example.com" }})
13}
  • baml dev - Development server with hot reload for interactive testing
  • baml serve - Production server for HTTP API testing
  • baml generate - Generate client code before running tests