Importer Architecture

The Importer builds the Layer 3 Knowledge Graph in your ArangoDB database: the documents, chunks, entities, communities, and relationships that the AutoRAG (and downstream applications) query at runtime. This page describes the collections it creates, the vector indexes it adds, the asynchronous job lifecycle of an import, and the operational rules to keep in mind when deploying the service.

For Layer 1 and Layer 2 (modules and Corpus Graph), see AutoGraph Architecture.

Knowledge graph collections

After a successful import, the Importer creates a named graph {project_name}_kg containing the vertex and edge collections below. All collection names are prefixed with the project name configured at install time (see LLM Configuration).

graph LR
  Document -->|PART_OF| Chunk
  Chunk -->|MENTIONED_IN| Entity
  Entity -->|RELATED_TO| Entity2["Entity"]
  Entity -->|IN_COMMUNITY| Community
  Community -->|SUB_COMMUNITY_OF| Community2["Community"]

Every Layer 3 document carries a partition_id field so that data from different partitions coexists in the same collections. Filter by partition_id when querying a specific partition.

Documents

• Purpose: Original text documents that were processed.
• Key fields:
  • _key: Unique identifier, built from a fingerprint of the content plus the import_number, and prefixed with the partition_id when the target is a SmartGraph. Because the fingerprint covers the content, files with identical content collapse into a single Document within one import.
  • content: Full text content.
  • file_name: Original filename.
  • citable_url: URL used for inline citations at retrieval. Taken from the inline files[].citable_url field, or from the file’s File Manager custom metadata on the file_id and file_ids paths. Empty for file_content and file_url imports, which have no way to supply one. See Citation URLs.
  • partition_id: Partition the document belongs to.
  • file_ids: List of File Manager ids stamped at import time, when the source is a File Manager file. Empty for inline or file_url imports without one. It is a list rather than a single id because _key is a content hash: when several files hold the same content they share one Document, and the id of each of them is recorded here. Layer 3 removals resolve documents primarily by this field.
  • import_number: The import batch that produced the document. Repeated imports into the same partition add further batches, which reclustering consolidates.

Chunks

• Purpose: Text chunks extracted from documents for granular processing.
• Key fields:
  • _key: Unique identifier.
  • content: Chunk text.
  • tokens: Number of tokens.
  • chunk_order_index: Position within the source document.
  • embedding: Vector embedding (when enabled).
  • partition_id: Partition the chunk belongs to.

Entities

• Purpose: Entities extracted from text (people, organizations, concepts, etc.). Only populated in full_graphrag mode.
• Key fields:
  • _key: Unique identifier.
  • entity_name, entity_type, description.
  • embedding: Vector embedding.
  • clusters: Community clusters the entity belongs to.
  • partition_id: Partition the entity belongs to.

Communities

• Purpose: Clusters of related entities representing themes in the data. Only populated in full_graphrag mode.
• Key fields:
  • _key: Unique identifier.
  • title: Cluster ID.
  • report_string: Markdown report explaining the community.
  • report_json: Structured report with title, summary, rating, rating_explanation, and findings.
  • level: Hierarchical level (e.g., 1 for top-level).
  • occurrence: Normalized score (0-1) for relative frequency.
  • sub_communities: References to nested sub-communities.
  • embedding: Vector embedding (when enable_community_embeddings is true).
  • partition_id: Partition the community belongs to.

Relations

• Purpose: Edges connecting all nodes in the Knowledge Graph.
• Key fields:
  • _from, _to: Source and target node references.
  • type: Relationship type (see below).
  • For entity-to-entity edges: weight, description, source_id, order.
  • partition_id: Partition the relationship belongs to.

The Importer creates the following relationship types:

1. PART_OF: Links chunks to their parent documents.
2. MENTIONED_IN: Connects entities to chunks where they appear.
3. RELATED_TO: Relationships between different entities.
4. IN_COMMUNITY: Associates entities with their community groups.
5. SUB_COMMUNITY_OF: Relates a sub-community to its parent community.

The community edges (IN_COMMUNITY, SUB_COMMUNITY_OF) only exist in full_graphrag partitions. They are rebuilt if you recluster such a partition after incremental updates. The other relationship types are kept. A vector_rag partition has no Entities or Communities and therefore no community layer that could be rebuilt.

Semantic Units

• Purpose: Image references and web URLs extracted from documents. Only created when enable_semantic_units is true. See Semantic Units for details.

Vector indexes

The Importer automatically creates a vector index named vector_cosine on the embedding field of collections that received embeddings during an import:

• Entities: when the selected rag_mode imports entities (e.g., full_graphrag).
• Chunks: when chunk embeddings are enabled (enable_chunk_embeddings=true in full_graphrag, always on in vector_rag).
• Communities: when enable_community_embeddings=true (default).
• SemanticUnits: when enable_semantic_unit_embeddings=true.
• Relations: when enable_edge_embeddings=true and the mode imports relations.

These indexes power semantic similarity search and nearest-neighbor queries at retrieval time. Index parameters (vector_index_metric, vector_index_use_hnsw, vector_index_n_lists) are set per import; see the Parameter Reference.

For background on ArangoDB vector indexes, see Vector indexes.

Asynchronous import lifecycle

Both POST /v1/import and POST /v1/import-multiple start work in the background and return immediately. The diagram below shows how a client submits an import and monitors progress.

An import begins by converting its inputs: every document that is not already plain text or Markdown is submitted to the File Parser service, which returns Markdown and, when image extraction is enabled, the embedded images as separate artifacts referenced at their position in that Markdown. The Importer chunks the Markdown, and the image references are what semantic units are built from. This stage is not reported as a separate job status. A document that cannot be parsed fails the import with the File Parser’s verdict in the status message. See Document conversion and supported formats.

flowchart LR
  Client["Client / orchestrator"]
  API["Importer API"]
  Poll["<code>GET /v1/jobs/{job_id}</code>"]
  Status["Platform service status"]
  DB["ArangoDB knowledge graph"]

  Client -->|POST /v1/import| API
  Client -->|POST /v1/import-multiple| API
  API -->|Single-file: no job_id| Status
  API -->|Multifile: job_id| Poll
  Client --> Poll
  Client --> Status
  API --> DB

The same lock also applies to POST /v1/recluster: a replica can only run one import or recluster job at a time. It is a single global lock per replica, not keyed by partition, so any import blocks any recluster, no matter which partitions the two jobs touch.

How a rejection is reported depends on the endpoint you call, so your client needs to handle both cases:

This is independent of which operation holds the lock. A blocked import reports success: false even if a recluster job is running, and a blocked recluster reports 503 even if an import is running. In both cases, wait until the running job is done and try again. The exception is a single-file import, which has no job_id. In this case, watch the platform service status instead. For the recluster endpoint and how to poll its job, see Incremental Updates. For the different response shapes, see Error handling.

Terminal statuses include service_completed, service_failed, openai_graph_build_failed, import_graph_to_adb_failed, and create_index_failed. Status messages may include a [rag_mode=...] prefix or markers like [NO_ENTITIES_WRITTEN] or [KG_VERIFICATION_INCONCLUSIVE]; see Error handling for the full list.

Operational guidance

The recommendations below apply whether you call the Importer directly or via AutoGraph orchestration.

1. Choose rag_mode explicitly - full_graphrag (default when omitted) for entity and community graphs; vector_rag for chunk-only retrieval.
2. Use multi-file imports plus the jobs API for batches; use single-file import only when you already integrate with data platform service status.
3. Serialize imports per replica - wait for service_completed or a terminal job status before submitting again.
4. Match embedding_dim to the deployed embedding model. Mismatches cause vector-index creation to fail at runtime.
5. Prefer File Manager file_ids for large files already on the data platform instead of inline base64 payloads.
6. Plan for long-running jobs. Graph build and vector-index training can take tens of minutes; ensure tokens can renew (the data platform JWT lifetime applies).
7. SmartGraph constraints: use smart_graph_attribute="partition_id", a valid partition_id, and shard_count=1 when creating a new SmartGraph. See Import Files for details.