Many LLMs are subject to fail due to transient errors. Setting up a fallback allows you to switch to a different LLM when prior LLMs fail (e.g. outage, high latency, rate limits, etc).

To accomplish this, instead of new syntax, you can simple define a client<llm> using a baml-fallback provider.

The baml-fallback provider takes a strategy option, which is a list of client<llm>s to try in order. If the first client fails, the second client is tried, and so on.

client<llm> MySafeClient {
    provider baml-fallback
    options {
      // First try GPT4 client, if it fails, try GPT35 client.
      strategy [
        GPT4,
        GPT35
        // If you had more clients, you could add them here.
        // Anthropic
      ]
  }
}

client<llm> GPT4 {
    provider baml-openai-chat
    options {
      // ...
    }
}

client<llm> GPT35 {
    provider baml-openai-chat
    options {
      // ...
    }
}

Fallbacks are triggered on any error.

Errors codes are:

NameError Code
BAD_REQUEST400
UNAUTHORIZED401
FORBIDDEN403
NOT_FOUND404
RATE_LIMITED429
INTERNAL_ERROR500
SERVICE_UNAVAILABLE503
UNKNOWN1