Set up the Importer

The Importer service is available in the Arango Contextual Data Platform through two interfaces:

• Web interface: A guided workflow for configuring and running the Importer step by step. See AutoGraph Studio.
• HTTP API: Full programmatic control over the Importer pipeline.

Document conversion and supported formats

The Importer does not parse documents itself. Every input that is not already plain text or Markdown is handed to the internal File Parser service, which converts it to Markdown and, where applicable, extracts the embedded images together with the text surrounding each one (if requested). The Importer then chunks that Markdown and builds the knowledge graph from it.

The File Parser is a data platform service installed once per environment. It has no web interface and you do not call it directly. AutoGraph uses the same service for its corpus build, so both paths accept the same inputs. The Markdown they get back differs: a corpus build only needs enough text to cluster a document, so it requests a truncated sample and never images, whereas an import converts the document in full.

Format support

Text extraction is what the knowledge graph is built from, and it is reliable for everything below. What varies between formats is image extraction, so check the last column before you rely on semantic units for a given document type.

Where images are extracted, each one is stored as a separate artifact and referenced at its position in the Markdown, together with the text surrounding it. That is what the Importer turns into semantic units.

Documents that legitimately contain no extractable text, such as a blank page, succeed with empty content and a warning rather than failing.

Tuning the File Parser for self-hosted deployments

The File Parser ships with defaults sized for a reference data platform deployment. Deployments on the Arango Managed Platform (AMP) run these defaults unchanged. For self-hosted clusters, the settings below are the most relevant ones to adjust limits and resource utilization.

Consider the combined resource footprint of the PDF workers. With the default settings, three PDF worker pods request 3 CPUs and 12 GiB of memory in total, and are limited to 12 CPUs and 18 GiB. The node pool must satisfy the requests for the pods to be scheduled, and the limits for all three to run at full capacity at the same time. Adjust the replica count and the per-pod limits together, based on the resources available in your cluster.

The worker resource limits are defined at deploy time of the service. Changing them in an active system is discouraged as it may have adverse effects on other components.

Applying values

The File Parser is installed once per environment as the arangodb-file-parser data platform service. Put your values in that service’s overrides block in the platform package (platform.yaml), the same file you install the data platform with, and apply the configuration using arangodb_operator_platform package install:

  arangodb-file-parser:
    package: arangodb-file-parser
    overrides:
      workerPdf:
        replicas: 5
        resources:
          limits:
            memory: 8Gi
      workerDefault:
        replicas: 15

The package is the right place for anything you want to keep: it is re-applied on every install and survives upgrades. Editing the running service directly with kubectl edit arangoplatformservice arangodb-file-parser takes effect immediately and is fine while you experiment, but the next package install replaces it. Changing a value restarts the pods, and workers finish the job they are on before stopping.

Checking what is applied

The pods log their effective non-default settings on startup, which is the quickest way to confirm a change landed:

kubectl logs -n <namespace> -l app.kubernetes.io/instance=arangodb-file-parser --tail=20

A setting name that does not exist is reported as a warning there rather than failing silently, so a typo is visible in the first log after a restart. To see the full applied set instead, read the service’s rendered configuration:

kubectl get cm arangodb-file-parser-config -n <namespace> -o yaml

If jobs queue for a long time, the fleet is too small: raise workerPdf.replicas / workerDefault.replicas, or give the pool more CPU. If jobs fail with resource or timeout errors, the per-job limits are too tight for your documents.

Prerequisites

• Arango Contextual Data Platform.
• LLM and embedding API access (OpenAI-compatible endpoints).
• Valid JWT for the API (Authorization: Bearer ...).
• A project in the target database. Projects keep datasets and configurations isolated from each other. For instructions, see the Projects section in the Arango Control Plane (ACP) documentation.

Installation

To install and start the Importer service, use the following endpoint:

This endpoint is part of the Arango Control Plane (ACP) service, which manages the lifecycle of all AI services in the platform. For detailed installation, monitoring, and lifecycle management instructions, see the Arango Control Plane (ACP) documentation.

Get started

Learn more

• Architecture: Knowledge graph collections, vector indexes, and the async-job lifecycle.
• LLM Configuration: Choose and configure your chat and embedding providers.
• Import Files: Single-file and multi-file imports with examples.
• AutoGraph Integration: How the Importer is driven by AutoGraph for multi-partition builds.
• Reference: HTTP endpoints, parameters, and error handling.