You can use the fallback provider to add more resilliancy to your application.

A fallback will attempt to use the first client, and if it fails, it will try the second client, and so on.

You can nest fallbacks inside of other fallbacks.
BAML
1client<llm> SuperDuperClient {
2 provider fallback
3 options {
4 strategy [
5 ClientA
6 ClientB
7 ClientC
8 ]
9 }
10}

Options

strategy
List[string]Required

The list of client names to try in order. Cannot be empty.

retry_policy

Like any other client, you can specify a retry policy for the fallback client. See retry_policy for more information.

The retry policy will test the fallback itself, after the entire strategy has failed.

BAML
1client<llm> SuperDuperClient {
2 provider fallback
3 retry_policy MyRetryPolicy
4 options {
5 strategy [
6 ClientA
7 ClientB
8 ClientC
9 ]
10 }
11}

Nesting multiple fallbacks

You can nest multiple fallbacks inside of each other. The fallbacks will just chain as you would expect.

BAML
1client<llm> SuperDuperClient {
2 provider fallback
3 options {
4 strategy [
5 ClientA
6 ClientB
7 ClientC
8 ]
9 }
10}
11
12client<llm> MegaClient {
13 provider fallback
14 options {
15 strategy [
16 SuperDuperClient
17 ClientD
18 ]
19 }
20}