LLM Client Providers

llama-api

Llama API supports the OpenAI client, allowing you to use the openai-generic provider with an overridden base_url.

Note that to call Llama, you must use its OpenAI-compatible /compat/v1 endpoint. See Llama’s OpenAI compatibility documentation.

client<llm> LlamaAPI {
  provider openai-generic
  retry_policy Exponential
  options {
    base_url "https://llama-api.meta.com/compat/v1"
    model "Llama-3.3-8B-Instruct"
    api_key env.LLAMA_API_KEY
    // see openai-generic docs for more options
  }
}