LLM Client Strategies

Connection Pooling

Reuse HTTP connections across requests

BAML disables HTTP connection pooling by default. This preserves compatibility with applications that create a BAML runtime before using Python’s fork() start method.

Enable connection reuse for a leaf client with the http option:

1client<llm> MyClient {
2  provider openai
3  options {
4    model "gpt-4o"
5    api_key env.OPENAI_API_KEY
6    http {
7      enable_connection_pooling true
8    }
9  }
10}

Pooling avoids a new TCP and TLS handshake for every request. It is useful for asyncio, threaded, and multiprocessing applications that use the spawn start method.

Do not enable pooling when child processes inherit an existing BAML runtime or client through fork(). Create the BAML runtime inside each child process, or use the spawn start method instead.

Connection pooling is not available for fallback or round-robin clients. Set it on their leaf clients instead. It is also not available for AWS Bedrock clients, whose HTTP transport is managed by the AWS SDK.