aws-bedrock

AWS Bedrock provider for BAML

The aws-bedrock provider supports all text-output models available via the Converse API.

Example:

BAML
1client<llm> MyClient {
2  provider aws-bedrock
3  options {
4    api_key env.MY_OPENAI_KEY
5    model "gpt-3.5-turbo"
6    temperature 0.1
7  }
8}

Authorization

We use the AWS SDK under the hood, which will respect all authentication mechanisms supported by the SDK, including but not limited to:

  • AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as set in your environment variables
  • loading the specified AWS_PROFILE from ~/.aws/config
  • built-in authn for services running in EC2, ECS, Lambda, etc.

Non-forwarded options

default_role
string

The default role for any prompts that don’t specify a role. Default: system

We don’t have any checks for this field, you can pass any string you wish.

allowed_role_metadata
string[]

Which role metadata should we forward to the API? Default: []

For example you can set this to ["foo", "bar"] to forward the cache policy to the API.

If you do not set allowed_role_metadata, we will not forward any role metadata to the API even if it is set in the prompt.

Then in your prompt you can use something like:

1client<llm> Foo {
2  provider openai
3  options {
4    allowed_role_metadata: ["foo", "bar"]
5  }
6}
7
8client<llm> FooWithout {
9  provider openai
10  options {
11  }
12}
13template_string Foo() #"
14  {{ _.role('user', foo={"type": "ephemeral"}, bar="1", cat=True) }}
15  This will be have foo and bar, but not cat metadata. But only for Foo, not FooWithout.
16  {{ _.role('user') }}
17  This will have none of the role metadata for Foo or FooWithout.
18"#

You can use the playground to see the raw curl request to see what is being sent to the API.

Forwarded options

messages
DO NOT USE

BAML will auto construct this field for you from the prompt

model_id
string

The model to use.

ModelDescription
anthropic.claude-3-haiku-20240307-v1:0Fastest + Cheapest
anthropic.claude-3-sonnet-20240307-v1:0Smartest
meta.llama3-8b-instruct-v1:0
meta.llama3-70b-instruct-v1:0
mistral.mistral-7b-instruct-v0:2
mistral.mixtral-8x7b-instruct-v0:1

Run aws bedrock list-foundation-models | jq '.modelSummaries.[].modelId to get a list of available foundation models; you can also use any custom models you’ve deployed.

Note that to use any of these models you’ll need to request model access.

inference_configuration
object

Additional inference configuration to send with the request; see AWS Bedrock documentation.

Example:

BAML
1client<llm> MyClient {
2  provider aws-bedrock
3  options {
4    inference_configuration {
5      max_tokens 1000
6      temperature 1.0
7      top_p 0.8
8      stop_sequence ["_EOF"]
9    }
10  }
11}