Triton LLM Host

Overview

The Triton LLM Host service provides scalable deployment of Large Language Models (LLMs) using the NVIDIA Triton Inference Server. It efficiently serves machine learning models with support for HTTP and gRPC APIs, customizable routing, and seamless Kubernetes integration.

Workflow

The Triton LLM Host enables the services of the Agentic AI Suite to use privately hosted LLMs directly from the Arango Contextual Data Platform environment. The process involves the following steps:

1. Install the Triton LLM Host service.
2. Register your LLM model to MLflow by uploading the required files.
3. Configure the Importer service to use your LLM model.
4. Configure AutoRAG to use your LLM model.

Deployment

The Triton LLM Host service is deployed as a Kubernetes application using Helm charts in the Arango Contextual Data Platform ecosystem. It integrates with the:

• MLFlow model registry for model management.
• Storage sidecar for artifact storage.

Installation via ACP Service API

To install the Triton LLM Host service, send an API request to the ACP service using the following parameters:

Required parameters

{
  "models": "model_name",
}

You can also specify multiple models:

• Without versions: "model_name_1, model_name_2"
• With versions: "model_name_1@version1, model_name_2@version2"
• Mixed: "model_name_1, model_name_2@version4"

Optional parameters

{
  "log_level": "INFO",
  "profiles": "profile1,profile2",
  "resources_requests_memory": "",     // Minimum memory required for the container
  "resources_requests_cpu": "",        // Minimum CPU required for the container
  "resources_limits_memory": "",       // Maximum memory the container can use
  "resources_limits_cpu": "",          // Maximum CPU the container can use
  "resources_requests_ephemeral_storage": "",  // Minimum ephemeral storage required for the container
  "resources_limits_ephemeral_storage": ""     // Maximum ephemeral storage the container can use
}

Parameter descriptions

Model requirements

Python Backend

All models must use the Python backend to ensure compatibility with the Triton service. Each model requires the following three files:

1. model.py Implements the Python backend model. Triton uses this file to load and execute your model for inference.

   class TritonPythonModel:
       def initialize(self, args):
           # Load your model here
           pass
   
       def execute(self, requests):
           # Process inference requests
           pass
   
       def finalize(self):
           # Cleanup resources
           pass
   

2. config.pbtxt This is the Triton model configuration file that defines essential parameters such as the model name, backend, and input/output tensors.

   name: "your_model_name"
   backend: "python"
   max_batch_size: 1
   input: [...]
   output: [...]
   

3. MLmodel Required by MLflow 3.x at the root of the model bundle so that mlflow.register_model(...) can locate the logged artifacts. The Triton LLM Host service itself does not read this file; it exists solely to satisfy MLflow’s registration check. See an example below:

   artifact_path: model
   flavors:
     python_function:
       env: conda.yaml
       loader_module: mlflow.pyfunc.loader
       python_version: "3.11"
   mlflow_version: "3.10.1"
   

Model management with MLflow

How to register a model in MLflow

Registering a Python backend model in MLflow involves packaging your model.py, config.pbtxt, and MLmodel files and passing them as an artifact. The Triton service will look for a directory named after your model (e.g., my-private-llm-model) within the MLflow registry store and expects to find the MLmodel, config.pbtxt, and model.py files inside it.

The bundle directory must have the following layout:

<model_name>/
├── MLmodel          # required by MLflow 3.x
├── config.pbtxt
└── <version>/
    └── model.py

try:
    mlflow.set_tracking_uri(MLFLOW_SERVICE_URI)
    with mlflow.start_run() as run:
        run_id = run.info.run_id
        model_uri = f"runs:/{run_id}/model"
        mlflow.register_model(model_uri=model_uri, name=model_name)
        # Log the entire model directory as an artifact, preserving the Triton structure
        mlflow.log_artifact(local_path=str(local_model_dir))

Service endpoints

Once deployed, the service exposes two endpoints:

• Internal access (within Arango Contextual Data Platform): https://{SERVICE_ID}.{KUBERNETES_NAMESPACE}.svc:8000

  • KUBERNETES_NAMESPACE is available as an environment variable.
  • SERVICE_ID is returned by the ACP service API.

  Example: To check server health: GET https://{SERVICE_ID}.{KUBERNETES_NAMESPACE}.svc:8000/v2/health/ready

• External access (outside Arango Contextual Data Platform): https://{BASE_URL}:8529/llm/{SERVICE_POSTFIX}/

  • BASE_URL: Your Arango Contextual Data Platform base URL.
  • SERVICE_POSTFIX: Last 5 characters of the service ID.

  Example: To check server health: GET https://{BASE_URL}:8529/llm/{SERVICE_POSTFIX}/v2/health/ready

Triton Inference Server API

For complete documentation on available endpoints and their usage, refer to the Triton Inference Server HTTP API  documentation.