LLM Client Providers

LLM Clients (client<llm>)

Clients are used to configure how LLMs are called, like so:

BAML
1function MakeHaiku(topic: string) -> string {
2  client "openai/gpt-4o"
3  prompt #"
4    Write a haiku about {{ topic }}.
5  "#
6}

<provider>/<model> shorthand for the Named Client version of MyClient:

BAML
1client<llm> MyClient {
2  provider "openai"
3  options {
4    model "gpt-4o"
5    // api_key defaults to env.OPENAI_API_KEY
6  }
7}
8
9function MakeHaiku(topic: string) -> string {
10  client MyClient
11  prompt #"
12    Write a haiku about {{ topic }}.
13  "#
14}

Consult the provider documentation for a list of supported providers and models, and the default options.

If you want to override options like api_key to use a different environment variable, or you want to point base_url to a different endpoint, you should use the latter form.

If you want to specify which client to use at runtime, in your Python/TS/Ruby code, you can use the client registry to do so.

This can come in handy if you’re trying to, say, send 10% of your requests to a different model.

Fields

provider
stringRequired

This configures which provider to use. The provider is responsible for handling the actual API calls to the LLM service. The provider is a required field.

The configuration modifies the URL request BAML runtime makes.

Provider NameDocsNotes
anthropicAnthropicSupports /v1/messages endpoint
aws-bedrockAWS BedrockSupports Converse and ConverseStream endpoint
google-aiGoogle AISupports Google AI’s generateContent and streamGenerateContent endpoints
vertex-aiVertex AISupports Vertex’s generateContent and streamGenerateContent endpoints
openaiOpenAISupports /chat/completions endpoint
azure-openaiAzure OpenAISupports Azure’s /chat/completions endpoint
openai-genericOpenAI (generic)Any other provider that supports OpenAI’s /chat/completions endpoint

A non-exhaustive list of providers you can use with openai-generic:

Inference ProviderDocs
Azure AI FoundaryAzure AI Foundary
GroqGroq
Hugging FaceHugging Face
Keywords AIKeywords AI
LitellmLitellm
LM StudioLM Studio
OllamaOllama
OpenRouterOpenRouter
TogetherAITogetherAI
Unify AIUnify AI
vLLMvLLM

We also have some special providers that allow composing clients together:

Provider NameDocsNotes
fallbackFallbackUsed to chain models conditional on failures
round-robinRound RobinUsed to load balance
options
dict[str, Any]Required

These vary per provider. Please see provider specific documentation for more information. Generally they are pass through options to the POST request made to the LLM.

retry_policy

The name of the retry policy. See Retry Policy.