ollama
Ollama supports the OpenAI client, allowing you to use the
openai-generic
provider with an
overridden base_url
.
Note that to call Ollama, you must use its OpenAI-compatible
/v1
endpoint. See Ollama’s OpenAI compatibility
documentation.
OLLAMA_ORIGINS='*' ollama serve
. Learn more in hereThe options are passed through directly to the API, barring a few. Here’s a shorthand of the options:
Non-forwarded options
The base URL for the API. Default: http://localhost:11434/v1
/v1
at the end of the URL. See Ollama’s OpenAI compatabilityThe default role for any prompts that don’t specify a role. Default: system
We don’t have any checks for this field, you can pass any string you wish.
Additional headers to send with the request.
Example:
Which role metadata should we forward to the API? Default: []
For example you can set this to ["foo", "bar"]
to forward the cache policy to the API.
If you do not set allowed_role_metadata
, we will not forward any role metadata to the API even if it is set in the prompt.
Then in your prompt you can use something like:
You can use the playground to see the raw curl request to see what is being sent to the API.
Whether the internal LLM client should use the streaming API. Default: true
Then in your prompt you can use something like:
Forwarded options
BAML will auto construct this field for you from the prompt
BAML will auto construct this field for you based on how you call the client in your code
The model to use.
For the most up-to-date list of models supported by Ollama, see their Model Library.
"mixtral:8x22b"