Import Parameter Reference

Overview

The Importer service supports a wide range of parameters to customize how your documents are processed and imported. Parameters are organized by functionality to help you find what you need.

File Source Parameters

Parameters differ between single file and multi-file import:

Single File Import

• file_content (required, or use file_url / file_id): Direct file content as base64-encoded string.
• file_url (required, or use file_content / file_id): URL to download the file from.
• file_id (required, or use file_content / file_url): RAG file ID from File Manager. When provided, the file is fetched by ID and file_content / file_url are ignored.
• file_name (required): Original filename with extension.

Single file import has no citable_url field. A file imported by file_id brings its citation URL along from File Manager; see Citation URLs.

Example with direct file content:

{
  "file_content": "base64_encoded_content_here",
  "file_name": "document.txt"
}

Example with URL download:

{
  "file_url": "https://example.com/documents/report.pdf",
  "file_name": "report.pdf"
}

Multi-File Import

You can provide files inline using the files array, or reference files already uploaded to File Manager using file_ids. When file_ids is provided, files are fetched by ID and the files array is ignored.

• file_ids (required, or use files): An array of RAG file IDs from File Manager.

Each file in the files array requires:

• name (required): Original filename with extension.
• content (required): File content as base64-encoded bytes.
• citable_url (optional): URL to be cited in inline citations. This URL is stored in the document metadata and used at retrieval. When querying your knowledge graph, whether citations are displayed is controlled by the show_citations parameter.

Example:

{
  "files": [
    {
      "name": "document1.txt",
      "content": "base64_encoded_content_1",
      "citable_url": "https://example.com/docs/document1"
    },
    {
      "name": "document2.md",
      "content": "base64_encoded_content_2",
      "citable_url": "https://example.com/docs/document2"
    }
  ]
}

There is no citable_url field next to file_ids. Files that come from File Manager bring their own; see Citation URLs.

Citation URLs

When AutoRAG answers a question, it cites the documents it used. A document that has a citable_url is cited as a clickable link; one without it is cited as a plain number.

If your files are in File Manager, there is nothing to set on the import request. Give the file a citable_url in its custom metadata when you upload it, and the Importer picks it up on its own:

Two things are worth knowing:

• Custom metadata belongs to one version of a file, so uploading a new version to File Manager drops the old citable_url. Set it again on the new upload.
• The Importer keeps only http and https links that contain no whitespace and no unmatched parentheses. It discards anything else, and the import still succeeds, so a bad link shows up as a plain citation number rather than an error. The reason is written to the Importer’s log. Values you set inline on files[].citable_url are not checked at all.

RAG Mode Configuration

The Importer supports two operational modes that determine how documents are processed and what knowledge graph elements are created:

• rag_mode: Specifies the retrieval mode. Valid values are:
  • "full_graphrag" (default): Full knowledge graph extraction with entities, relationships, and communities
  • "vector_rag": Simple vector-based retrieval using only chunk embeddings (skips entity extraction)

Example:

{
  "rag_mode": "full_graphrag"
}

Chunking Parameters

These parameters control how documents are split into smaller chunks for processing.

• chunk_token_size: Maximum tokens per chunk. The default value is 1024. It must not be smaller than chunk_overlap_token_size; equal values are accepted. A smaller value is rejected with a validation error.
• chunk_overlap_token_size: Number of overlapping tokens between consecutive chunks. The default value is 128.
• chunk_min_token_size: Minimum tokens per chunk. Chunks smaller than this are merged with adjacent chunks. The default value is 64.
• chunk_custom_separators: Custom separators for chunking (optional, e.g., ["\n\n", "\n", " "]).
• preserve_chunk_separator: Whether to preserve separator characters in chunks (default: false). Set it to true explicitly if you want separators kept. Omitting the field is the same as sending false.
• ignore_chunk_token_size: If true, chunks are split only by separators without enforcing token size limits. When enabled, chunk_token_size and chunk_overlap_token_size are ignored (default: false).

Example:

{
  "chunk_token_size": 1024,
  "chunk_overlap_token_size": 128,
  "chunk_min_token_size": 64,
  "chunk_custom_separators": ["\n\n", "\n", " "],
  "preserve_chunk_separator": true,
  "ignore_chunk_token_size": false
}

Example with separator-only chunking:

{
  "chunk_custom_separators": ["\n## ", "\n\n", "\n"],
  "ignore_chunk_token_size": true,
  "preserve_chunk_separator": false
}

Entity and Relationship Extraction

These parameters define what entities and relationships to extract from your documents.

• entity_types: Entity types to extract from the document. The default value is ["person", "organization", "geo", "event"].
• relationship_types: Relationship types to extract (e.g., ["RELATED_TO", "PART_OF", "USES"]). If not provided, relationships are extracted based on the LLM’s judgment.
• enable_strict_types: Enable strict filtering of entities and relationships from the specified types (default: false).
• entity_extract_max_gleaning: Maximum number of extraction iterations (default: 1).

Example:

{
  "entity_types": ["person", "organization", "location", "event", "technology"],
  "relationship_types": ["WORKS_FOR", "LOCATED_IN", "USES", "PARTICIPATES_IN"],
  "enable_strict_types": true,
  "entity_extract_max_gleaning": 3
}

Custom Prompts

You can customize the prompts used for entity extraction, community report generation, and other processing steps. This is an advanced feature for domain-specific customization.

• custom_prompts: A dictionary mapping prompt names to custom prompt text. When provided, custom prompts override the default prompts. Only the prompts you specify are overridden; all others use their defaults. If empty or not provided, default prompts are used.

Available prompt keys:

• "entity_extraction": Prompt for extracting entities and relationships from text chunks.
• "community_report": Prompt for generating community summary reports.
• "claim_extraction": Prompt for extracting claims against entities.
• "summarize_entity_descriptions": Prompt for combining multiple entity descriptions into one.
• "entity_continue_extraction": Follow-up prompt when entities may have been missed.
• "entity_if_loop_extraction": Prompt to determine if additional extraction iterations are needed.

Template variables:

Custom prompts support the following template variables that are automatically filled at runtime:

• {entity_types}: Comma-separated list of entity types.
• {relationship_types}: Comma-separated list of relationship types.
• {relationship_type_instruction}: Dynamic instruction about relationship types.
• {input_text}: The text chunk being processed.
• {tuple_delimiter}: Field separator (default: <|>).
• {record_delimiter}: Record separator (default: ##).
• {completion_delimiter}: End marker (default: <|COMPLETE|>).

Example:

{
  "custom_prompts": {
    "entity_extraction": "Extract technical entities including: software libraries, programming languages, APIs, and architectural components. For each entity provide: name, type, and a brief description.\n\nEntity_types: {entity_types}\nRelationship_types: {relationship_types}\nText: {input_text}\nOutput:",
    "community_report": "Generate a technical summary focusing on: 1) Key technologies and their relationships, 2) Architectural patterns, 3) Integration points. Provide 5-7 key insights.\n\nText:\n{input_text}\nOutput:"
  }
}

Embedding Parameters

These parameters control which graph elements receive embedding vectors for semantic search.

• enable_chunk_embeddings: Whether to enable embeddings for text chunks (default: false for "full_graphrag" mode, always true for "vector_rag" mode).
• enable_edge_embeddings: Whether to enable embeddings for relationship edges (default: false).
• enable_community_embeddings: Whether to enable embeddings for community reports (default: true).

Example:

{
  "enable_chunk_embeddings": true,
  "enable_edge_embeddings": false,
  "enable_community_embeddings": true
}

Vector Index Configuration

These parameters configure the vector indexes used for semantic search and similarity queries. Vector indexes are automatically created on embedding fields when embeddings are enabled.

• vector_index_metric: Distance metric for vector similarity search. The supported values are "cosine" (default), "l2", and "innerProduct".
• vector_index_n_lists: Number of lists for approximate search (optional). Must be a positive integer when set. If not set, it is automatically computed as 8 * sqrt(collection_size) and capped at the collection size. This parameter is ignored when using HNSW.
• vector_index_use_hnsw: Whether to use HNSW (Hierarchical Navigable Small World) index instead of the default inverted index (default: false).

Example:

{
  "vector_index_n_lists": 100,
  "vector_index_metric": "cosine",
  "vector_index_use_hnsw": false
}

Semantic Units and Image Processing

These parameters enable extraction of images and multimedia references. For detailed information, see the Semantic Units guide.

• enable_semantic_units: Collect image and web references, and ask the File Parser to extract the images embedded in the source documents. Default: false.
• process_images: Send each image to a vision-capable model and store the returned description on the semantic unit (requires enable_semantic_units=true). Default: false.
• enable_semantic_unit_embeddings: Generate vector embeddings for semantic units (requires enable_semantic_units=true). Default: false.

process_images and enable_semantic_unit_embeddings both require enable_semantic_units.

Example:

{
  "enable_semantic_units": true,
  "process_images": true,
  "enable_semantic_unit_embeddings": false
}

Graph Configuration

These parameters configure ArangoDB graph features for distributed deployments.

• smart_graph_attribute: SmartGraph attribute for graph sharding. When creating a new SmartGraph through the Importer, this must be set to "partition_id".
• shard_count: Number of shards for the collections. The current version supports only shard_count: 1 when the Importer is creating a new SmartGraph or a new sharded enterprise graph; other values are rejected with a validation error. Omit the field (or set 0) when the target graph already exists.
• is_disjoint: Whether the graphs must be disjoint. Must be false when creating a new sharded enterprise graph with a positive shard_count.
• satellite_collections: An array of collection names to create as Satellite Collections.

Example:

{
  "smart_graph_attribute": "partition_id",
  "shard_count": 1,
  "is_disjoint": false,
  "satellite_collections": ["entities"]
}

Storage and Organization

These parameters control data storage and organization.

• store_in_s3: Accepted but currently a no-op. The field is present on the request payload for proto compatibility but has no effect in the current service version.
• partition_id: Partition identifier for grouping related documents together.
• batch_size: Number of documents, entities, and relationships to insert in a single batch. The default value is 1000.

partition_id

The partition_id parameter enables semantic sharding and horizontal scaling by grouping related documents into logical partitions.

Key benefits:

• Semantic sharding: Documents from the same domain/cluster are stored together
• Horizontal scaling: Different partitions can be distributed across multiple machines
• Multi-tenancy: Isolate documents for different projects, customers, or use cases
• Efficient querying: AutoRAG can target specific partitions for faster searches

Usage patterns:

1. With AutoGraph: Use AutoGraph-generated rag_partition_id values (format: {cluster_index}_{a|b}) to group documents by semantic similarity and RAG strategy
2. Manual partitioning: Define your own partition scheme based on domain, project, or tenant
3. Auto-generated: If not specified, a random partition ID is generated

Example:

{
  "partition_id": "cluster_0_a",
  "batch_size": 1000
}

Examples

Standard Full GraphRAG Example

This example uses full GraphRAG mode with entity extraction, chunking, and embeddings:

{
  "file_content": "base64_encoded_content",
  "file_name": "document.txt",
  "rag_mode": "full_graphrag",
  "chunk_token_size": 1024,
  "chunk_overlap_token_size": 128,
  "entity_types": ["person", "organization", "location", "event"],
  "relationship_types": ["WORKS_FOR", "LOCATED_IN", "PARTICIPATES_IN"],
  "enable_chunk_embeddings": false,
  "enable_community_embeddings": true,
  "batch_size": 1000
}

Vector RAG Example

This example uses vector RAG mode for fast, simple semantic search without entity extraction:

{
  "file_content": "base64_encoded_content",
  "file_name": "document.txt",
  "rag_mode": "vector_rag",
  "chunk_token_size": 1024,
  "chunk_overlap_token_size": 128,
  "batch_size": 1000
}

Advanced Full GraphRAG Example

For advanced use cases, this configuration demonstrates custom prompts, separator-based chunking, semantic units for image processing, strict type filtering, and optimized vector index settings:

{
  "file_content": "base64_encoded_content",
  "file_name": "document.md",
  "rag_mode": "full_graphrag",
  "chunk_custom_separators": ["\n## ", "\n\n", "\n"],
  "ignore_chunk_token_size": true,
  "entity_types": ["person", "organization", "technology", "concept"],
  "relationship_types": ["USES", "RELATED_TO", "PART_OF"],
  "enable_strict_types": true,
  "entity_extract_max_gleaning": 2,
  "enable_chunk_embeddings": true,
  "enable_community_embeddings": true,
  "enable_semantic_units": true,
  "process_images": true,
  "vector_index_metric": "cosine",
  "vector_index_use_hnsw": true,
  "custom_prompts": {
    "entity_extraction": "Extract technical components, APIs, and architectural elements with their relationships.",
    "community_report": "Summarize key technical insights and system architecture patterns."
  },
  "batch_size": 500
}

API Reference

For detailed API documentation, see the GraphRAG Importer API Reference .