AutoGraph Corpus Build Reference

Create Corpus Build

Trigger a corpus build from File Manager categories or from files that you have imported directly.

Recommended path: Prefer categories, so that the service resolves the files of each category from the File Manager itself. Send only new categories on a first build and leave incremental at its default of false. To append documents to a category that has already been built, set incremental: true. Only one build can run at a time (409).

Selecting what to build

Provide at most one of the following selectors. A request that carries more than one is rejected with 400.

When you use the legacy path without a selector, every imported basename has to exist as a RAG input in the File Manager for the same database as well.

Request (preferred - by category)

{
  "embedding_strategy": "first_chunk",
  "categories": ["legal", "finance"],
  "strategy": {
    "top_k": 7,
    "cluster_threshold": 2
  },
  "incremental": false
}

Request (deprecated - by file IDs)

{
  "embedding_strategy": "first_chunk",
  "file_ids": ["file_id_1", "file_id_2", "file_id_3"],
  "strategy": {
    "top_k": 7,
    "cluster_threshold": 2
  }
}

Parameters

Citation URLs

There is no citable_url field on this request. For File Manager builds, set custom_metadata.citable_url on the RAG input when you upload it. AutoGraph reads it while fetching the file and stores it on the source document.

The URL has to be an http or https URL without whitespace or unbalanced parentheses. If the key is missing, empty, or invalid, the build still succeeds, citable_url is stored as an empty string, and the citation numbers stay unlinked. The API does not report the rejection, so check the build logs if links are missing.

Response

{
  "corpus_build_id": "cb_01ARZ3NDEKTSV4RRFFQ69G5FAV",
  "graph_name": "myproject_CorpusGraph"
}

Selector validation

The service looks the selector up in the File Manager while it handles the request, and rejects the build with 400 if there is nothing to build:

• A categories label that holds no files, on a full build with incremental: false.
• file_ids that the File Manager cannot resolve, because they are unknown or malformed, or because they belong to another database.

The response names what was missing. No build is created in these cases, so there is no corpus_build_id to poll. Check the status code before you read the ID out of the response body.

Two cases are deliberately not rejected:

• An incremental build (incremental: true) still accepts a category whose files are all gone, because reconciling such removals is part of what an incremental run is for. See Incremental Builds.
• If the File Manager cannot be reached at all, the service reaches no verdict and accepts the build as before. An outage is reported on the build, never as a bad request.

A file that is deleted after the check still fails the build the way it always did. The check makes the common mistake visible right away, it does not remove the deferred failure path.

Document parsing

For File Manager builds (categories, modules, or file_ids), the service extracts the text through the File Parsing Service instead of parsing in-process: preview scope, the first 4,800 characters, images excluded. Files uploaded through the deprecated POST /v1/import-multiple are still parsed in-process.

Plain text files (.txt and .md) are parsed directly. Every other input is submitted to the File Parser service, which converts a sample to Markdown before AutoGraph embeds and clusters it. PDF files (including scanned documents, read using OCR) and the Office formats .doc, .docx, .ppt, and .pptx are officially supported. See Supported file formats for details.

Parsing is durable. If the pod restarts in the middle of a build, it picks up the batches it already submitted instead of submitting them again.

Three outcomes are worth planning for:

• A file that yields no extractable text, such as a scanned image without OCR-readable content, is reported as a failed file. It is no longer embedded as an empty document.
• If some files parse and others fail, the build completes instead of failing, and GET /v1/corpus/builds/{id} reports error_code: FILE_PARSER_PARTIAL_FAILURE.
• If no file parses, the build fails with error_code: FILE_PARSER_NO_SUCCESS.

In every case the failing files are named individually in message, as filename (ID: file_id): error, so you can tell which document caused the failure rather than only which batch did:

Build partially completed: 2 file(s) failed in File Parser. Failed files:
report.pdf (ID: rag-input-abc): FILE_TOO_LARGE: source exceeds size limit;
scan.pdf (ID: rag-input-def): no extractable content

A completed build with partial failures lists the first five entries and then ; ... and N more. A build that failed because nothing parsed lists the first ten and then ... and N more file(s). On a large failure set, message alone therefore does not enumerate every affected file.

Scanned and image-heavy documents are much slower to parse than digital text, and the quality of the extraction depends on how legible the scan is. The mechanics belong to the File Parsing Service, which ships on its own release cadence, so consult its documentation for the current per-format behavior.

Chunk limit: by default the first 1200 tokens (~4800 characters) per document drive the embedding. Your operator can tune the embedding budget with CHUNK_SIZE and CHARACTERS_PER_TOKENS, but the 4,800-character extraction cap of the File Parsing Service is fixed and does not move with it.

HTTP Example

curl -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <token>" \
  -d '{
    "embedding_strategy": "first_chunk",
    "categories": ["legal", "finance"],
    "strategy": { "top_k": 10, "cluster_threshold": 2 }
  }' \
  https://<EXTERNAL_ENDPOINT>:8529/autograph/v1/corpus/builds

Monitoring Build Status

Check the progress of a corpus build.

Recommended path: Immediately after POST /v1/corpus/builds returns corpus_build_id, poll this endpoint until status is completed or failed. Typical interval: 5–30 seconds for short builds; 30–60 seconds for very large corpora to avoid load.

Parameters

Response

{
  "corpus_build_id": "cb_01ARZ3NDEKTSV4RRFFQ69G5FAV",
  "status": "running",
  "message": "Creating similarity edges...",
  "progress": 55,
  "error": "",
  "error_code": null,
  "started_at": 1705344000.0,
  "completed_at": 0.0,
  "graph_name": "myproject_CorpusGraph",
  "document_count": 0,
  "cluster_count": 0,
  "documents_added": 0,
  "documents_removed": 0,
  "documents_unchanged": 0,
  "files_written": 0,
  "documents_created": 0,
  "documents_deduplicated": 0,
  "dedup_groups": []
}

Every dedup_groups entry has the following fields:

Build error codes

Two different situations report STORAGE_FILE_TOO_LARGE:

• A file larger than the whole staging space is skipped on its own, and the wave carries on with the files behind it.

• Separately, if a wave can download nothing at all and no staging space can be reclaimed, every file still queued is abandoned and reported as Local storage budget exhausted; could not download remaining files. Those files are not too large and succeed on a retry once the staging space is free.

Check the per-file text before you split or shrink anything.

HTTP Example

curl -H "Authorization: Bearer <token>" \
  https://<EXTERNAL_ENDPOINT>:8529/autograph/v1/corpus/builds/cb_01ARZ3NDEKTSV4RRFFQ69G5FAV

Document identity and deduplication

A corpus document is identified by its category and its filename, and by nothing else. The document key is a hash of exactly these two values, and the writes use the update overwrite mode of ArangoDB. That has the following consequences:

• Two input files with the same filename in the same category produce a single document. The one written last provides the content, the other one is absorbed. This is what makes a re-import idempotent instead of duplicating rows, and it is reported in documents_deduplicated and dedup_groups.
• The deduplication is by name, never by content. No content is hashed, no embedding similarity is compared, and no similarity threshold is involved. Two documents whose extracted text is identical, a shared boilerplate cover page for example, stay two documents as long as their filenames differ.
• The category scopes the name. A cover.pdf in legal and a cover.pdf in finance are two separate documents.
• Filenames are disambiguated on the way in. If a build resolves several File Manager files that share a basename, they are renamed to report.pdf, report_1.pdf, and so on, before any of the above applies. The counter is shared by the entire build, so the renaming also occurs between files in different categories.
• A re-upload under an existing name never reaches this rule. The File Manager keys a RAG input by database, project, category, and name, so uploading the same name into the same category supersedes the earlier file with a new version instead of adding a second entry. Only the latest version of a file is built, and this loss occurs before AutoGraph sees anything.

When a collapse actually happens

Because names are unique within a category and are disambiguated across categories, few sequences reach a collapse. The known one is a filename that collides with the output of the disambiguation:

Three File Manager entries become two documents, with documents_deduplicated: 1. The renaming does not reserve the name it generates, so a file that is genuinely called a_1.md is not disambiguated against it. Watch out for this if you use systematic _1 and _2 suffixes in your filenames and reuse those basenames across categories.

Which files a document came from

Every document records its own mapping, so you do not have to reconstruct the key hash. The documents in the {project}_sources collection have the following fields:

Both are on every document, with a single entry in the ordinary case where nothing collapsed, so you never have to handle their absence. Documents that were written before these fields existed have neither. Import them again to populate them.

If documents_deduplicated is not what you expected:

1. Read dedup_groups. It names the exact files and their File Manager IDs.
2. If the collapse was unintended, upload the absorbed file under a filename that is distinct and not of the form name_1 or name_2 for a basename that is used elsewhere in the build, then add it with POST /v1/graph/insert. Uploading it under the same name in the same category does not help, because the File Manager supersedes the earlier version instead of creating a second entry.
3. To replace the content of a document deliberately, use POST /v1/graph/update instead of relying on a re-import under the same name.

If 5% or more of a batch collapses, the completion message of the build says so as well, so a high deduplication ratio is visible without reading the counts.

Incremental Builds

An incremental build appends documents to categories that already exist, without rebuilding the rest of the corpus. For guidance on structuring categories, see the Design Guide.

When to use an incremental build:

• Adding documents to a category that has already been built.
• Syncing deletions: on a File Manager category build, incremental: true also removes corpus documents that are no longer in the File Manager listing.
• Reducing the build time for large corpora.

When not to use one:

• For a first build, or when you add a category that is new to the corpus, leave incremental at its default of false.
• To change individual documents in a graph that is already built, use an incremental graph update instead.

Next Steps

• Run RAG Strategizer: Analyze clusters and get RAG strategy recommendations
• Orchestrate Pipeline: Automatically build knowledge graphs
• Project Operations: Inspect the project state and remove categories