Configure LLMs and Embedding Models in AutoGraph

AutoGraph uses two kinds of models:

• A chat (LLM) model, used by the RAG Strategizer to generate the per-cluster ontology (the entity types for each domain).
• An embedding model, used during the corpus build to embed documents for similarity and clustering, and by the embed-field endpoint.

Each model can be backed by either any OpenAI-compatible API or a Triton Inference Server. That covers the OpenAI API itself, which is the recommended setup, as well as any other endpoint implementing the same contract — OpenRouter, Gemini, Anthropic, or a private corporate LLM.

“OpenAI-compatible” means the endpoint must implement the contract used by the OpenAI Chat Completions client (/v1/chat/completions, and /v1/embeddings for embedding models). An endpoint that exposes only a different API surface is not supported. Some newer OpenAI models require the Responses API (/v1/responses) instead; AutoGraph detects this and falls back automatically (see Chat payload compatibility).

Supported models

The following models are validated for use with the AutoGraph service. For the full list across all services, see Supported LLM and embedding models.

The recommended provider is openai with the OpenAI models below. That is the combination ArangoDB tests, so prefer it where you can; other endpoints can differ in behavior such as latency.

You can still point AutoGraph at any other OpenAI-compatible endpoint — OpenRouter, Google Gemini, Anthropic, Azure, or a corporate LLM — and run a model that is not on the list. Configure these with the custom provider and the chat_api_url / embedding_api_url of your endpoint, as described in Using OpenAI-compatible APIs. Models beyond the list below are outside ArangoDB’s testing, so validate them in your own environment. For the models served through Triton, see Using Triton Inference Server.

A model marked Yes under Default is the one applied automatically when the model name is not set with the openai provider. Whether the custom provider falls back to the same model varies by service — check the parameter reference for the service you are configuring.

Supported providers

For both the chat model and the embedding model, AutoGraph supports three provider values:

• openai: the OpenAI API itself.
• custom: any other OpenAI-compatible API, such as OpenRouter, Google Gemini, Anthropic, Azure, or a corporate LLM.
• triton: a Triton Inference Server.

The chat provider (chat_api_provider) and the embedding provider (embedding_api_provider) are configured independently: AutoGraph does not require them to match. In most deployments both are set to the same value (typically openai). Mixing providers (for example, openai for chat and triton for embeddings) is not rejected, but it is not a routinely tested combination, so validate it in your environment before relying on it.

URL defaults: When a provider is set to openai, chat_api_url and embedding_api_url default to https://api.openai.com/v1 if not specified. For triton, these URLs are required and must be provided explicitly.

Model defaults: The following default models are applied automatically when chat_model or embedding_model_name are not specified:

Using OpenAI-compatible APIs

AutoGraph reaches OpenAI-compatible APIs through two provider values:

• openai for the official OpenAI API. The URLs default to https://api.openai.com/v1, so you can omit them.
• custom for every other OpenAI-compatible endpoint, including OpenRouter, Google Gemini, Anthropic Claude, Azure, and corporate or self-hosted LLMs. Always set chat_api_url and embedding_api_url to your endpoint: AutoGraph otherwise falls back to the OpenAI URL, silently sending requests to OpenAI.

Pointing the openai provider at a non-OpenAI URL is not supported. Use custom for those endpoints.

Example using OpenAI

{
  "env": {
    "db_name": "your_database_name",
    "genai_project_name": "your_project_name",
    "chat_api_provider": "openai",
    "chat_api_url": "https://api.openai.com/v1",
    "chat_model": "gpt-5.4-nano",
    "chat_api_key": "your_openai_api_key",
    "embedding_api_provider": "openai",
    "embedding_api_url": "https://api.openai.com/v1",
    "embedding_model_name": "text-embedding-3-small",
    "embedding_api_key": "your_openai_api_key",
    "embedding_dimensions": "512"
  }
}

For a full description of all parameters, see Chat and Embedding Parameters.

Using different OpenAI-compatible services for chat and embedding

You can use different OpenAI-compatible services for chat and embedding. For example, you might use OpenRouter for chat and OpenAI for embeddings, depending on your needs for performance, cost, or model availability. Because the two providers are configured independently, set chat_api_provider to custom with the OpenRouter URL and leave embedding_api_provider on openai.

{
  "env": {
    "db_name": "your_database_name",
    "genai_project_name": "your_project_name",
    "chat_api_provider": "custom",
    "chat_api_url": "https://openrouter.ai/api/v1",
    "chat_model": "mistralai/mistral-nemo",
    "chat_api_key": "your_openrouter_api_key",
    "embedding_api_provider": "openai",
    "embedding_api_url": "https://api.openai.com/v1",
    "embedding_model_name": "text-embedding-3-small",
    "embedding_api_key": "your_openai_api_key",
    "embedding_dimensions": "512"
  }
}

For a full description of all parameters, see Chat and Embedding Parameters.

Using Triton Inference Server

The first step is to install the LLM Host service with the LLM and embedding models of your choice. The setup uses the Triton Inference Server and MLflow at the backend. For more details, see the Triton Inference Server and MLflow documentation.

Once the llmhost service is up and running, configure AutoGraph to use it for chat, embeddings, or both. The chat_api_url and embedding_api_url are required for Triton, and chat_model is required for the Triton chat provider.

{
  "env": {
    "db_name": "your_database_name",
    "genai_project_name": "your_project_name",
    "chat_api_provider": "triton",
    "chat_api_url": "your-arangodb-llm-host-url",
    "chat_model": "mistral-nemo-instruct",
    "embedding_api_provider": "triton",
    "embedding_api_url": "your-arangodb-llm-host-url",
    "embedding_model_name": "nomic-embed-text-v1",
    "embedding_dimensions": "768"
  }
}

For a full description of all parameters, see Chat and Embedding Parameters.

Mixing providers

Because the chat and embedding providers are configured independently, nothing prevents you from mixing them. The example below uses an OpenAI-compatible API for chat and a Triton Inference Server for embeddings. This combination is not routinely tested, so verify it works in your environment before relying on it.

{
  "env": {
    "db_name": "your_database_name",
    "genai_project_name": "your_project_name",
    "chat_api_provider": "openai",
    "chat_api_url": "https://api.openai.com/v1",
    "chat_model": "gpt-5.4-nano",
    "chat_api_key": "your_openai_api_key",
    "embedding_api_provider": "triton",
    "embedding_api_url": "your-arangodb-llm-host-url",
    "embedding_model_name": "nomic-embed-text-v1",
    "embedding_dimensions": "768"
  }
}

Chat and Embedding Parameters

This reference section covers the chat and embedding model parameters. Provider-specific defaults and requirements are noted where applicable. Parameter names are also accepted in uppercase (for example, CHAT_API_URL).

General parameters

• db_name (required): Name of the ArangoDB database where the corpus graph and knowledge graph are stored.
• genai_project_name (required): The project name, used as a prefix for all ArangoDB collections (for example, a project named docs creates docs_sources, docs_domains, docs_CorpusGraph, and so on). Set it to the same name you chose when creating the project via the web interface or the Project API. The Project API names this field project_name; the AutoGraph install request uses genai_project_name for the same value.

Chat API parameters

• chat_api_provider (required): The provider for the chat/LLM model. Set to openai for the OpenAI API, custom for any other OpenAI-compatible API, or triton for Triton Inference Server.
• chat_api_url: API endpoint URL for the chat model.
  • OpenAI: Defaults to https://api.openai.com/v1 if not provided.
  • Custom: Set it explicitly. AutoGraph falls back to https://api.openai.com/v1 when it is omitted, which sends requests to OpenAI instead of your endpoint.
  • Triton: Must be provided explicitly.
• chat_api_key (required for the openai and custom providers): API key for authenticating with the chat model. Alternatively, use chat_secret_profile_id.
• chat_model: Language model used for ontology generation and analysis.
  • OpenAI: Defaults to gpt-5.4-nano.
  • Custom: Also defaults to gpt-5.4-nano; set it explicitly to a model your endpoint exposes.
  • Triton: Required; there is no default.

Embedding API parameters

• embedding_api_provider (required): The provider for the embedding model. Set to openai for the OpenAI API, custom for any other OpenAI-compatible API, or triton for Triton Inference Server.
• embedding_api_url: API endpoint URL for the embedding model.
  • OpenAI: Defaults to https://api.openai.com/v1 if not provided.
  • Custom: Set it explicitly. AutoGraph falls back to https://api.openai.com/v1 when it is omitted, which sends requests to OpenAI instead of your endpoint.
  • Triton: Must be provided explicitly.
• embedding_api_key (required for the openai and custom providers): API key for authenticating with the embedding model. Alternatively, use embedding_secret_profile_id.
• embedding_model_name: Model used to generate text embeddings.
  • OpenAI: Defaults to text-embedding-3-small.
  • Custom: Also defaults to text-embedding-3-small; set it explicitly to a model your endpoint exposes.
  • Triton: Defaults to nomic-embed-text-v1.
• embedding_dimensions: Embedding dimension. Defaults to 512. It must match the embedding model’s output dimension; set it explicitly when using a model with a different dimension. For example, the default Triton embedding model nomic-embed-text-v1 produces 768-dimensional vectors, so set this to 768 for that model. Always match the embedding dimension to your model’s output when configuring Triton embeddings.
• embedding_api_keys: Optional comma-separated list of OpenAI-compatible API keys. When more than one key is provided, AutoGraph rotates across them in a round-robin pool to increase the effective rate limit. Falls back to embedding_api_key when not set.
• embedding_input_type: Optional input type (for example, passage or query) required by some providers, such as NVIDIA models.
• disabled_params: Optional JSON array of embedding parameters to strip from outgoing embedding requests, for example ["dimensions"]. This applies to embedding calls only, not chat.

Advanced parameters

• OPENAI_MAX_RETRIES: Maximum retries on the OpenAI-compatible HTTP client for chat. Defaults to 6.
• TRITON_TIMEOUT: Request timeout in seconds for the Triton chat provider. Defaults to 300.

Chat payload compatibility

These options apply when chat_api_provider is openai or custom, not for triton. OpenAI-style chat requests (used by the RAG Strategizer) are built from environment variables, with a one-shot retry when the API rejects an optional parameter, per-model caching of the working parameter shape within the process, and a Responses API fallback when a model rejects /v1/chat/completions.

Set chat_model to the model your provider exposes. With the openai provider, use one of the chat models in Supported models above, such as gpt-5.4-nano, gpt-5.4-mini, or gpt-5. With the custom provider, your endpoint may serve model families that are not in that table, including older ones such as the GPT-4 series; the options below exist so you can match their payload requirements. Different model families accept different optional fields on chat completions.

Optional parameters (also accepted in uppercase, for example CHAT_PARAMETER_POLICY):

Configuration Validation

When configuring AutoGraph, ensure you:

1. Use a supported provider for each model: chat_api_provider and embedding_api_provider must each be openai, custom, or triton. They are independent and do not need to match.
2. Provide all required parameters:
   • chat_api_provider and embedding_api_provider (both required).
   • chat_api_key and embedding_api_key (required for the openai and custom providers, or supply the corresponding secret profile ID).
   • chat_api_url and embedding_api_url (optional for openai and custom, which both fall back to the OpenAI URL, but always set them for custom; required for triton).
   • chat_model (required for the triton chat provider).
3. Follow provider-specific requirements:
   • The openai provider requires valid API keys.
   • The custom provider requires valid API keys, and you should always set the endpoint URLs and model names rather than rely on the OpenAI fallbacks.
   • The triton provider requires valid server URLs.

AutoGraph validates your configuration at startup and rejects an unsupported provider or a missing required parameter with an error message.

Next Steps

• Build a corpus: Generate document embeddings and similarity edges.
• Run the RAG Strategizer: Assign retrieval strategies and generate per-cluster ontologies with the chat model.
• Embed a field: Add embeddings to an existing ArangoDB collection.