test

The test command runs BAML function tests defined in your BAML files. It provides comprehensive testing capabilities including filtering, parallel execution, and various output formats.

Usage

baml-cli test [OPTIONS]

Options

Option	Description	Default
`--from <PATH>`	Path to the `baml_src` directory	`./baml_src`
`--list`	Only list selected tests without running them	`false`
`-i, --include <PATTERN>`	Include specific functions or tests (can be used multiple times)	Run all tests
`-x, --exclude <PATTERN>`	Exclude specific functions or tests (can be used multiple times)	None
`--parallel <NUM>`	Number of tests to run in parallel	`10`
`--pass-if-no-tests`	Pass if no tests are selected	`false`
`--require-human-eval`	Fail if any tests need human evaluation	`true`
`--dotenv`	Load environment variables from .env file	`true`
`--dotenv-path <PATH>`	Path to custom .env file	None

Description

The test command performs the following actions:

Discovers and parses all test cases defined in BAML files
Applies include/exclude filters to select which tests to run
Executes tests in parallel (configurable concurrency)
Reports results with detailed output and assertions
Supports various output formats and CI integration

Test Filtering

The --include and --exclude options support powerful pattern matching:

Pattern Syntax

* matches any characters within a name
FunctionName::TestName matches a specific test in a specific function
FunctionName:: matches all tests in a function
::TestName matches a test name across all functions
Multiple patterns can be combined

Examples

$ # Run all tests
$ baml-cli test
$ 
$ # List all available tests
$ baml-cli test --list
$ 
$ # Run tests for a specific function
$ baml-cli test -i "ExtractResume::"
$ 
$ # Run a specific test
$ baml-cli test -i "ExtractResume::TestBasicResume"
$ 
$ # Run all tests matching a pattern
$ baml-cli test -i "Extract*::"
$ 
$ # Run tests with multiple include patterns
$ baml-cli test -i "Extract*::" -i "Classify*::"
$ 
$ # Exclude specific tests
$ baml-cli test -x "ExtractResume::TestComplexResume"
$ 
$ # Combine include and exclude (exclude takes precedence)
$ baml-cli test -i "Extract*::" -x "*::TestSlow*"

Parallel Execution

Control the number of tests running simultaneously:

$ # Run tests with default parallelism (10)
$ baml-cli test
$ 
$ # Run tests sequentially
$ baml-cli test --parallel 1
$ 
$ # Run with high parallelism
$ baml-cli test --parallel 20

Environment Variables

The test command automatically loads environment variables:

$ # Use default .env file loading
$ baml-cli test
$ 
$ # Disable .env file loading
$ baml-cli test --no-dotenv
$ 
$ # Use custom .env file
$ baml-cli test --dotenv-path .env.test

Environment variables can also be set directly:

$ # Set API keys for testing
$ OPENAI_API_KEY=... ANTHROPIC_API_KEY=... baml-cli test

Exit Codes

The test command returns different exit codes based on results:

Exit Code	Meaning
`0`	All tests passed
`1`	Test failures occurred
`2`	Tests require human evaluation
`3`	Test execution was cancelled
`4`	No tests were found to run

Examples

Basic Testing

$ # Run all tests in the project
$ baml-cli test
$ 
$ # Run tests from a specific directory
$ baml-cli test --from /path/to/my/baml_src

Development Workflow

$ # Run tests for a function you're developing
> baml-cli test -i "MyNewFunction::"
> 
> # Run specific test while debugging
> baml-cli test -i "MyFunction::TestEdgeCase"
> 
> # List tests to see what's available
$ baml-cli test --list -i "Extract*::"

CI/CD Integration

$ # Fail fast on first assertion failure
$ baml-cli test --require-human-eval
$ 
$ # Allow tests that need human evaluation to pass
$ baml-cli test --no-require-human-eval
$ 
$ # Run with controlled parallelism for CI
$ baml-cli test --parallel 5

Test Discovery

$ # See all available tests
$ baml-cli test --list
$ 
$ # See tests for specific functions
$ baml-cli test --list -i "ClassifyMessage::"
$ 
$ # See what tests would run with filters
$ baml-cli test --list -i "Extract*::" -x "*::TestSlow*"

Test Definition

Tests are defined in BAML files using the test block syntax:

1 function ExtractResume(resume: string) -> Resume {
2   client GPT4
3   prompt #"Extract resume information: {{resume}}"#
4 }
5 
6 test TestBasicResume {
7   functions [ExtractResume]
8   args {
9     resume "John Doe\njohn@example.com\nSoftware Engineer"
10   }
11   @@assert({{ this.name == "John Doe" }})
12   @@assert({{ this.email == "john@example.com" }})
13 }

baml dev - Development server with hot reload for interactive testing
baml serve - Production server for HTTP API testing
baml generate - Generate client code before running tests

$	# Run all tests
$	baml-cli test
$
$	# List all available tests
$	baml-cli test --list
$
$	# Run tests for a specific function
$	baml-cli test -i "ExtractResume::"
$
$	# Run a specific test
$	baml-cli test -i "ExtractResume::TestBasicResume"
$
$	# Run all tests matching a pattern
$	baml-cli test -i "Extract*::"
$
$	# Run tests with multiple include patterns
$	baml-cli test -i "Extract::" -i "Classify::"
$
$	# Exclude specific tests
$	baml-cli test -x "ExtractResume::TestComplexResume"
$
$	# Combine include and exclude (exclude takes precedence)
$	baml-cli test -i "Extract::" -x "::TestSlow*"

$	# Run tests with default parallelism (10)
$	baml-cli test
$
$	# Run tests sequentially
$	baml-cli test --parallel 1
$
$	# Run with high parallelism
$	baml-cli test --parallel 20

$	# Use default .env file loading
$	baml-cli test
$
$	# Disable .env file loading
$	baml-cli test --no-dotenv
$
$	# Use custom .env file
$	baml-cli test --dotenv-path .env.test

$	# Set API keys for testing
$	OPENAI_API_KEY=... ANTHROPIC_API_KEY=... baml-cli test

$	# Run all tests in the project
$	baml-cli test
$
$	# Run tests from a specific directory
$	baml-cli test --from /path/to/my/baml_src

$	# Run tests for a function you're developing
>	baml-cli test -i "MyNewFunction::"
>
>	# Run specific test while debugging
>	baml-cli test -i "MyFunction::TestEdgeCase"
>
>	# List tests to see what's available
$	baml-cli test --list -i "Extract*::"

$	# Fail fast on first assertion failure
$	baml-cli test --require-human-eval
$
$	# Allow tests that need human evaluation to pass
$	baml-cli test --no-require-human-eval
$
$	# Run with controlled parallelism for CI
$	baml-cli test --parallel 5

$	# See all available tests
$	baml-cli test --list
$
$	# See tests for specific functions
$	baml-cli test --list -i "ClassifyMessage::"
$
$	# See what tests would run with filters
$	baml-cli test --list -i "Extract::" -x "::TestSlow*"

1	function ExtractResume(resume: string) -> Resume {
2	client GPT4
3	prompt #"Extract resume information: {{resume}}"#
4	}
5
6	test TestBasicResume {
7	functions [ExtractResume]
8	args {
9	resume "John Doe\njohn@example.com\nSoftware Engineer"
10	}
11	@@assert({{ this.name == "John Doe" }})
12	@@assert({{ this.email == "john@example.com" }})
13	}

Usage

Options

Description

Test Filtering

Pattern Syntax

Examples

Parallel Execution

Environment Variables

Exit Codes

Examples

Basic Testing

Development Workflow

CI/CD Integration

Test Discovery

Test Definition

Related Commands

Usage

Options

Description

Test Filtering

Pattern Syntax

Examples

Parallel Execution

Environment Variables

Exit Codes

Examples

Basic Testing

Development Workflow

CI/CD Integration

Test Discovery

Test Definition

Related Commands