round-robin

The round_robin provider allows you to distribute requests across multiple clients in a round-robin fashion. After each call, the next client in the list will be used.

BAML
1client<llm> MyClient {
2  provider round-robin
3  options {
4    strategy [
5      ClientA
6      ClientB
7      ClientC
8    ]
9  }
10}

Options

strategy
List[string]Required

The list of client names to try in order. Cannot be empty.

start
int

The index of the client to start with.

Default is random(0, len(strategy))

In the BAML Playground, Default is 0.

retry_policy

When using a retry_policy with a round-robin client, it will rotate the strategy list after each retry.

BAML
1client<llm> MyClient {
2  provider round-robin
3  retry_policy MyRetryPolicy
4  options {
5    strategy [
6      ClientA
7      ClientB
8      ClientC
9    ]
10  }
11}

Nesting multiple round-robin clients

You can nest multiple round-robin clients inside of each other. The round-robin as you would expect.

BAML
1client<llm> MyClient {
2  provider round-robin
3  options {
4    strategy [
5      ClientA
6      ClientB
7      ClientC
8    ]
9  }
10}
11
12client<llm> MegaClient {
13  provider round-robin
14  options {
15    strategy [
16      MyClient
17      ClientD
18      ClientE
19    ]
20  }
21}
22
23// Calling MegaClient will call:
24// MyClient(ClientA)
25// ClientD
26// ClientE
27// MyClient(ClientB)
28// etc.