vertex-ai

The base URL for the API.

Default: https://{LOCATION}-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/{LOCATION}/publishers/google/models/

Can be used in lieu of the project_id and location fields, to manually set the request URL.

Vertex requires a Google Cloud project ID for each request. See the Google Cloud Project ID Docs for more information.

Vertex requires a location for each request. Some locations may have different models avaiable.

Common locations include:

• us-central1
• us-west1
• us-east1
• us-south1

See the Vertex Location Docs for all locations and supported models.

Path to a JSON credentials file or a JSON object containing the credentials.

Default: env.GOOGLE_APPLICATION_CREDENTIALS

Overrides contents of the Google Cloud Application Credentials. Default: env.GOOGLE_APPLICATION_CREDENTIALS_CONTENT

Directly set Google Cloud Authentication Token in lieu of token generation via env.GOOGLE_APPLICATION_CREDENTIALS or env.GOOGLE_APPLICATION_CREDENTIALS_CONTENT fields.

The Google model to use for the request.

See the Google Model Docs for the latest models.

Additional headers to send with the request.

Example:

1
client<llm> MyClient {
2
  provider vertex-ai
3
  options {
4
    model gemini-1.5-pro
5
    project_id my-project-id
6
    location us-central1
7
    // Additional headers
8
    headers {
9
      "X-My-Header" "my-value"
10
    }
11
  }
12
}

The role to use if the role is not in the allowed_roles. Default: "user" usually, but some models like OpenAI’s gpt-4o will use "system"

Picked the first role in allowed_roles if not “user”, otherwise “user”.

Which roles should we forward to the API? Default: ["system", "user", "assistant"] usually, but some models like OpenAI’s o1-mini will use ["user", "assistant"]

When building prompts, any role not in this list will be set to the default_role.

Which role metadata should we forward to the API? Default: []

For example you can set this to ["foo", "bar"] to forward the cache policy to the API.

If you do not set allowed_role_metadata, we will not forward any role metadata to the API even if it is set in the prompt.

Then in your prompt you can use something like:

1
client<llm> Foo {
2
  provider openai
3
  options {
4
    allowed_role_metadata: ["foo", "bar"]
5
  }
6
}
7
8
client<llm> FooWithout {
9
  provider openai
10
  options {
11
  }
12
}
13
template_string Foo() #"
14
  {{ _.role('user', foo={"type": "ephemeral"}, bar="1", cat=True) }}
15
  This will be have foo and bar, but not cat metadata. But only for Foo, not FooWithout.
16
  {{ _.role('user') }}
17
  This will have none of the role metadata for Foo or FooWithout.
18
"#

You can use the playground to see the raw curl request to see what is being sent to the API.

Whether the internal LLM client should use the streaming API. Default: true

Then in your prompt you can use something like:

1
client<llm> MyClientWithoutStreaming {
2
  provider anthropic
3
  options {
4
    model claude-3-haiku-20240307
5
    api_key env.ANTHROPIC_API_KEY
6
    max_tokens 1000
7
    supports_streaming false
8
  }
9
}
10
11
function MyFunction() -> string {
12
  client MyClientWithoutStreaming
13
  prompt #"Write a short story"#
14
}

1
# This will be streamed from your python code perspective, 
2
# but under the hood it will call the non-streaming HTTP API
3
# and then return a streamable response with a single event
4
b.stream.MyFunction()
5
6
# This will work exactly the same as before
7
b.MyFunction()

Safety settings to apply to the request. You can stack different safety settings with a new safetySettings header for each one. See the Google Vertex API Request Docs for more information on what safety settings can be set.

1
client<llm> MyClient {
2
  provider vertex-ai
3
  options {
4
    model gemini-1.5-pro
5
    project_id my-project-id
6
    location us-central1
7
8
    safetySettings {
9
      category HARM_CATEGORY_HATE_SPEECH
10
      threshold BLOCK_LOW_AND_ABOVE
11
      method SEVERITY
12
    }
13
  }
14
}

Generation configurations to apply to the request. See the Google Vertex API Request Docs for more information on what properties can be set.

1
client<llm> MyClient {
2
  provider vertex-ai
3
  options {
4
    model gemini-1.5-pro
5
    project_id my-project-id
6
    location us-central1
7
    
8
    generationConfig {
9
      maxOutputTokens 100
10
      temperature 1
11
    }
12
  }
13
}