vertex-ai

The vertex-ai provider is used to interact with the Google Vertex AI services.

Example using a Vertex API Key (Express Mode):

1
client<llm> MyClient {
2
  provider vertex-ai
3
  options {
4
    model gemini-2.5-pro
5
    location us-central1 // or "global"
6
    project_id my-project-id
7
    query_params {
8
      key env.VERTEX_API_KEY
9
    }
10
  }
11
}

Authentication

Using a Vertex API Key (Express Mode)

When using a Vertex API Key, set the key query parameter and specify your project_id and location:

1
client<llm> VertexApiKeyClient {
2
  provider vertex-ai
3
  options {
4
    model gemini-2.5-pro
5
    location us-central1 // you can also use "global"
6
    project_id my-project-id
7
    query_params {
8
      key env.VERTEX_API_KEY
9
    }
10
  }
11
}

When in doubt, check the ‘cURL’ tab in the playground to see the exact request being sent!

Notes:

• project_id cannot be inferred when using an API key; set it explicitly.
• Keep credentials unset when using an API key, so BAML does not prefer service account auth.

Using a Vertex API Key in the playground

You should see the VERTEX_API_KEY environment variable in the playground API Keys dialog. You can set it there and you’re all set!

Using Google Application Credentials

If no vertex api key is set, BAML will by default try to authenticate using application default credentials

client<llm> MyClient {
  provider vertex-ai
  options {
    model gemini-2.5-pro
    location us-central1
    project_id my-project-id
    // we will by default try to use this form of authentication.
    credentials env.MY_APPLICATION_CREDENTIALS_CONTENT
  }
}

This is what the MY_APPLICATION_CREDENTIALS_CONTENT environment variable looks like:

1
MY_APPLICATION_CREDENTIALS_CONTENT={
2
  "type": "service_account",
3
  "project_id": "my-project-id",
4
  "private_key_id": "string",
5
  "private_key": "-----BEGIN PRIVATE KEY-----string\n-----END PRIVATE KEY-----\n"
6
  ...other fields...
7
}

BAML accepts this blob as a string, a path to a file, or a JSON object.

More details on Google Application Credentials

Here is the order of authentication:

• If GOOGLE_APPLICATION_CREDENTIALS environment variable is set, it will use the specified service account
• If you have run gcloud auth application-default login, it will find the credentials generated by gcloud by the path convention. Note that you will still need to set either options.project_id or the GOOGLE_CLOUD_PROJECT environment variable.
• If running in GCP, it will query the metadata server to use the attached service account
• If gcloud is available on the PATH, it will use gcloud auth print-access-token

Requirements

You need to use an account with a ProjectID that has been authorized to use Vertex. When administering your Google Cloud account, be sure to enable Vertex, and set up ADC:

$
gcloud auth application-default login

If you’re using Google Cloud application default credentials, you can expect authentication to work out of the box.

Setting options.credentials will take precedence and force vertex-ai to load service account credentials from that file path.

Playground

To use a vertex-ai client in the playground, you need to run gcloud auth application-default login in the terminal and set the GOOGLE_CLOUD_PROJECT environment variable in the “API Keys” dialog. The playground will then use these credentials to auth all Vertex API calls.

Debugging

https://${LOCATION}-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/${LOCATION}/publishers/google/models/${MODEL_ID}:generateContent

https://${LOCATION}-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/${LOCATION}/publishers/google/models/${MODEL_ID}:streamGenerateContent

[modelname] was not found or your project does not have access to it. Please ensure you are using a valid model version.

1
client<llm> MyClient {
2
  provider vertex-ai
3
  options {
4
    model gemini-2.5-pro
5
    location global
6
    project_id my-project-id
7
  }
8
}

BAML-specific request options

These unique parameters (aka options) modify the API request sent to the provider.

You can use this to modify the headers and base_url for example.

https://{LOCATION}-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/{LOCATION}/publishers/google/models/

https://aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/global/publishers/google/models/

1
client<llm> Vertex {
2
  provider vertex-ai
3
  options {
4
    model gemini-2.5-pro
5
    location us-central1
6
    // credentials can be a block string containing service account credentials in JSON format
7
    credentials #"
8
      {
9
        "type": "service_account",
10
        "project_id": "my-project-id",
11
        "private_key_id": "string",
12
        "private_key": "-----BEGIN PRIVATE KEY-----string\n-----END PRIVATE KEY-----\n",
13
        "client_email": "john_doe@gmail.com",
14
        "client_id": "123456",
15
        "auth_uri": "https://accounts.google.com/o/oauth2/auth",
16
        "token_uri": "https://oauth2.googleapis.com/token",
17
        "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
18
        "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/...",
19
        "universe_domain": "googleapis.com"
20
      }
21
    "#
22
  }
23
}

1
client<llm> Vertex {
2
  provider vertex-ai
3
  options {
4
    model gemini-2.5-pro
5
    location us-central1
6
    credentials "path/to/credentials.json"
7
  }
8
}

1
client<llm> Vertex {
2
  provider vertex-ai
3
  options {
4
    model gemini-2.5-pro
5
    location us-central1
6
    // credentials can be a block string containing service account credentials in JSON format
7
    credentials {
8
      type "service_account",
9
      project_id "my-project-id",
10
      private_key_id "string",
11
      private_key "-----BEGIN PRIVATE KEY-----string\n-----END PRIVATE KEY-----\n",
12
      client_email "john_doe@gmail.com",
13
      client_id "123456",
14
      auth_uri "https://accounts.google.com/o/oauth2/auth",
15
      token_uri "https://oauth2.googleapis.com/token",
16
      auth_provider_x509_cert_url "https://www.googleapis.com/oauth2/v1/certs",
17
      client_x509_cert_url "https://www.googleapis.com/robot/v1/metadata/...",
18
      universe_domain "googleapis.com"
19
    }
20
  }
21
}

1
client<llm> Vertex {
2
  provider vertex-ai
3
  options {
4
    model gemini-2.5-pro
5
    location us-central1
6
    // credentials_content is a block string containing service account credentials in JSON format
7
    credentials_content #"
8
      {
9
        "type": "service_account",
10
        "project_id": "my-project-id",
11
        "private_key_id": "string",
12
        "private_key": "-----BEGIN PRIVATE KEY-----string\n-----END PRIVATE KEY-----\n",
13
        "client_email": "john_doe@gmail.com",
14
        "client_id": "123456",
15
        "auth_uri": "https://accounts.google.com/o/oauth2/auth",
16
        "token_uri": "https://oauth2.googleapis.com/token",
17
        "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
18
        "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/...",
19
        "universe_domain": "googleapis.com"
20
      }
21
    "#
22
  }
23
}

1
client<llm> MyClient {
2
  provider vertex-ai
3
  options {
4
    model gemini-2.5-pro
5
    project_id my-project-id
6
    location us-central1
7
    // Additional headers
8
    headers {
9
      "X-My-Header" "my-value"
10
    }
11
  }
12
}

1
client<llm> MyClient {
2
  provider vertex-ai
3
  options {
4
    model gemini-2.5-pro
5
    project_id my-project-id
6
    location us-central1
7
    query_params {
8
      key env.VERTEX_API_KEY
9
    }
10
  }
11
}

1
{
2
  "user": "human",
3
  "assistant": "ai",
4
}

1
client<llm> Foo {
2
  provider openai
3
  options {
4
    allowed_role_metadata: ["foo", "bar"]
5
  }
6
}
7
8
client<llm> FooWithout {
9
  provider openai
10
  options {
11
  }
12
}
13
template_string Foo() #"
14
  {{ _.role('user', foo={"type": "ephemeral"}, bar="1", cat=True) }}
15
  This will be have foo and bar, but not cat metadata. But only for Foo, not FooWithout.
16
  {{ _.role('user') }}
17
  This will have none of the role metadata for Foo or FooWithout.
18
"#

1
client<llm> MyClientWithoutStreaming {
2
  provider anthropic
3
  options {
4
    model claude-3-5-haiku-20241022
5
    api_key env.ANTHROPIC_API_KEY
6
    max_tokens 1000
7
    supports_streaming false
8
  }
9
}
10
11
function MyFunction() -> string {
12
  client MyClientWithoutStreaming
13
  prompt #"Write a short story"#
14
}

1
# This will be streamed from your python code perspective, 
2
# but under the hood it will call the non-streaming HTTP API
3
# and then return a streamable response with a single event
4
b.stream.MyFunction()
5
6
# This will work exactly the same as before
7
b.MyFunction()

1
client<llm> MyClient {
2
  provider "openai"
3
  options {
4
    model "gpt-5-mini"
5
    api_key env.OPENAI_API_KEY
6
    // Finish reason allow list will only allow the stop finish reason
7
    finish_reason_allow_list ["stop"]
8
  }
9
}

1
client<llm> MyClient {
2
  provider "openai"
3
  options {
4
    model "gpt-5-mini"
5
    api_key env.OPENAI_API_KEY
6
    // Finish reason deny list will allow all finish reasons except length
7
    finish_reason_deny_list ["length"]
8
  }
9
}

media_url_handler

Controls how media URLs are processed before sending to the provider. This allows you to override the default behavior for handling images, audio, PDFs, and videos.

1
client<llm> MyClient {
2
  provider openai
3
  options {
4
    media_url_handler {
5
      image "send_base64"                    // Options: send_base64 | send_url | send_url_add_mime_type | send_base64_unless_google_url
6
      audio "send_url"
7
      pdf "send_url_add_mime_type"
8
      video "send_url"
9
    }
10
  }
11
}

Options

Each media type can be configured with one of these modes:

• send_base64 - Always download URLs and convert to base64 data URIs
• send_url - Pass URLs through unchanged to the provider
• send_url_add_mime_type - Ensure MIME type is present (may require downloading to detect)
• send_base64_unless_google_url - Only process non-gs:// URLs (keep Google Cloud Storage URLs as-is)

Provider Defaults

If not specified, each provider uses these defaults:

When to Use

• Use send_base64 when your provider doesn’t support external URLs and you need to embed media content
• Use send_url when your provider handles URL fetching and you want to avoid the overhead of base64 conversion
• Use send_url_add_mime_type when your provider requires MIME type information (e.g., Vertex AI)
• Use send_base64_unless_google_url when working with Google Cloud Storage and want to preserve gs:// URLs

Provider request parameters

These are other parameters that are passed through to the provider, without modification by BAML. For example if the request has a temperature field, you can define it in the client here so every call has that set.

Consult the specific provider’s documentation for more information.

1
client<llm> MyClient {
2
  provider vertex-ai
3
  options {
4
    model gemini-2.5-pro
5
    project_id my-project-id
6
    location us-central1
7
8
    safetySettings {
9
      category HARM_CATEGORY_HATE_SPEECH
10
      threshold BLOCK_LOW_AND_ABOVE
11
      method SEVERITY
12
    }
13
  }
14
}

1
client<llm> MyClient {
2
  provider vertex-ai
3
  options {
4
    model gemini-2.5-pro
5
    project_id my-project-id
6
    location us-central1
7
8
    generationConfig {
9
      maxOutputTokens 100
10
      temperature 1
11
    }
12
  }
13
}

1
client<llm> GeminiThinking {
2
  provider vertex-ai
3
  options {
4
    model gemini-2.5-pro
5
    project_id my-project-id
6
    location us-central1
7
    generationConfig {
8
      thinkingConfig {
9
        thinkingBudget 1024
10
        includeThoughts true
11
      }
12
    }
13
  }
14
}

For all other options, see the official Vertex AI documentation.

Publishers Other Than Google

If you are using models from publishers other than Google, such as Llama from Meta, use your project endpoint as the base_url in BAML:

1
client<llm> VertexLlama {
2
  provider vertex-ai
3
  options {
4
    base_url "https://us-central1-aiplatform.googleapis.com/v1/projects/my-project-id/locations/us-central1/endpoints/"
5
    location us-central1
6
  }
7
}

Anthropic Claude Models on Vertex AI

For Anthropic Claude models, you can use the simplified configuration with location and project_id:

1
client<llm> VertexClaudeSonnet {
2
  provider vertex-ai
3
  options {
4
    model "claude-sonnet-4-5@20250929"
5
    location us-east5
6
    project_id my-project-id
7
    anthropic_version "vertex-2023-10-16"
8
    credentials env.GOOGLE_APPLICATION_CREDENTIALS
9
  }
10
}

Alternatively, you can use a custom base_url with environment variables:

1
client<llm> VertexClaudeWithBaseUrl {
2
  provider vertex-ai
3
  options {
4
    model "claude-sonnet-4-5@20250929"
5
    anthropic_version "vertex-2023-10-16"
6
    base_url env.VERTEX_CLAUDE_BASE_URL
7
  }
8
}

Where VERTEX_CLAUDE_BASE_URL would be set to something like:

$
VERTEX_CLAUDE_BASE_URL="https://us-east5-aiplatform.googleapis.com/v1/projects/your-project-id/locations/us-east5/publishers/anthropic/models"

Extended Thinking for Claude on Vertex AI

To enable extended thinking for Claude models on Vertex AI, use the thinking parameter:

1
client<llm> VertexClaudeThinking {
2
  provider vertex-ai
3
  options {
4
    model "claude-sonnet-4-5@20250929"
5
    location us-east5
6
    project_id my-project-id
7
    anthropic_version "vertex-2023-10-16"
8
    credentials env.GOOGLE_APPLICATION_CREDENTIALS
9
    max_tokens 4096
10
    thinking {
11
      type "enabled"
12
      budget_tokens 2048
13
    }
14
  }
15
}