Incremental Graph Updates in AutoGraph

Incremental Graph Updates (IGU) keep a knowledge graph up-to-date after it has been built. You can add new documents, remove obsolete ones, and replace documents whose content has changed, without running the corpus build, the RAG Strategizer, and a full orchestration pass again.

IGU updates all three layers. In Layers 1 and 2, it maintains the corpus graph with its sources, similarity edges, and cluster membership. In Layer 3, it adds or removes the knowledge graph data, such as documents, chunks, entities, communities, and relationships. Existing clusters and strategy profiles are kept, and a new document joins the cluster closest to it, so the whole category does not have to be clustered again.

Adding to Layer 3 is the Importer’s job. Removing from it is not. AutoGraph deletes knowledge graph data itself, with AQL queries, and only involves the Importer when new content has to be extracted, embedded, and clustered:

An update is a delete followed by an insert, which is why its Layer 3 work is split between the two services in exactly that way.

How you track an operation follows from the table. What AutoGraph does itself is reported by the call that did it. What the Importer does runs in the background, so you start it with an orchestration call and poll until it is finished.

For everyday document changes, this is much cheaper and faster than a rebuild, because only the documents that actually changed are processed.

Supported operations

Insert, delete, and update work on documents and start in Layers 1 and 2, as described above. Reclustering is different: it takes Layer 3 partitions directly, schedules the Importer work for them, and resets the divergence values if the job succeeds.

This page explains the concepts: when to use IGU, how it compares to a rebuild, and how partition divergence is measured. For the requests and responses of each endpoint, see the Graph Operations reference.

When to use IGU

Use IGU if all of the following is true:

• The initial corpus build has finished successfully, and usually the RAG Strategizer and orchestration have run as well.
• You want to add, remove, or replace documents in an existing category. Every endpoint takes a batch, so this is not limited to one document per call.
• The change is small compared to what the category already holds.
• The cluster topology is still valid. You are not redesigning categories, and you do not need to compute the similarity and clustering of a whole category again.

When not to use IGU

Rule of thumb: Use IGU for changes that are small in relation to the category, whether that is one document or a batch of them, an incremental corpus build for bulk additions, and a corpus build for a new category or a clean category rebuild.

Prerequisites

• Arango Contextual Data Platform 4.1.0, where IGU is only available through the HTTP API.
• An AutoGraph project with a corpus graph that has already been built. Typically, the RAG Strategizer and orchestration have run as well.
• The project was built through the File Manager, so that every document in the project has a file_id.
• The documents you want to change belong to existing categories.

Full rebuild vs. IGU

Insert vs. update

Workflow

flowchart TD
  A["Corpus graph already built"] --> B["Insert / Delete / Update\n(Layers 1-2)"]
  B --> C["Targeted orchestration\nfile_ids\n(Layer 3)"]
  C --> D{"needs_reclustering\ntrue?"}
  D -->|no| E["Done"]
  D -->|yes| F["POST /v1/graph/recluster\n(your decision)"]
  F --> G["Communities refreshed\ndivergence reset to 0"]

1. Change the documents. Call insert, delete, or update, depending on what changed.
2. Build Layer 3. After an insert or a successful update, run a targeted orchestration with the returned file_id, so that the knowledge graph contains the new content. AutoGraph works out which strategized clusters hold those ids, so you do not name a partition. Both insert and update take File Manager input only, so every changed document has a file_id you can target with. See Identifying documents for Layer 3. A delete needs no orchestration, because it has already removed its Layer 3 data.
3. Check the divergence. On a FullGraphRAG partition, a new divergence_score is calculated once Layer 3 reflects the change, and needs_reclustering is set if the score is above the partition’s threshold. Insert and update responses do not carry the score, so read it from GET /v1/orchestrate/{orchestration_id} after step 2, see Where to read the score. VectorRAG partitions have nothing to measure and are never flagged, so this step and the next one do not apply to them.
4. Recluster if you want to. If the flag is true and you think the refresh is worth the cost, call recluster with the affected rag_partition_id values, at most five per request. Reclusterings run one at a time and hold the service-wide write slot for the whole run, so plan them like a maintenance window. Confirm the outcome on the partition’s rags node: needs_reclustering is cleared and last_reclustered_at is set only if it really ran.

Identifying documents for Layer 3

A targeted orchestration identifies documents by file_id alone. doc_name does not work here, and it never has to: insert and update both require a File Manager file_id for every document, so the id you send is the id you pass on to the orchestration. IDs have the form rag-input-{base64url(db:path)} and refer to a File Manager path, which is why the prerequisites ask for a project built through the File Manager. Upload a document with POST /_platform/filemanager/_db/{database}/rag-input and pass the returned ID.

After an insert or an update, send the file_ids of the changed documents, together with the project, which is required. Non-empty file_ids is what keeps an already imported partition eligible, because it bypasses the filter that orchestration applies otherwise.

Sending categories alone does not bypass that filter. Orchestration then only builds the partitions that are not in the {project}_kg knowledge graph yet, which skips exactly the partitions that your insert or update touched. If nothing else is stale, the request is rejected with a 409 and a Nothing to orchestrate message, see Telling the three 409 responses apart.

Partition divergence and reclustering

After every insert, delete, and update, AutoGraph measures how far each affected FullGraphRAG partition in Layer 3 has drifted from the state it was in when it was last clustered with the Leiden algorithm. The result is a divergence_score. It is stored in the partition’s rags strategy profile. Where you can read it back depends on the operation, see Where to read the score.

Everything in this section applies to FullGraphRAG partitions only.

The divergence is only a signal. AutoGraph never starts a reclustering on its own. If the score gets above the partition’s threshold, AutoGraph sets needs_reclustering: true and leaves the decision to you.

How the score is calculated

The stored score is the higher one of two values:

divergence_score = max(gross_churn_score, multi_batch_score)
needs_reclustering = (divergence_score > divergence_threshold)

The divergence_threshold is stored per partition on the rags profile and is 0.25 (25%) in this release. It is not configurable through the API: no request field sets it, so every partition falls through to the default. The only way to change it is to edit the rags document in ArangoDB by hand. A score that is equal to the threshold does not set the flag, only a higher score does.

Gross churn score

gross_churn_score = cumulative_churn / baseline_entity_count

• baseline_entity_count is the number of distinct entity names at the time of the last successful clustering or reclustering. The first time the score is measured after a partition is built, the current count becomes the baseline and the score starts at 0.
• cumulative_churn is the total number of entities that have been added and deleted by all inserts, deletes, and updates since that baseline. The churn is gross, not net. An update that removes 100 entities and adds 100 new ones therefore adds about 200 to the total, not 0.

For example, with a baseline of 1000 entities, deleting 100 entities and then adding 100 gives a cumulative_churn of 200 and thus a gross_churn_score of 0.20.

Multi-batch score

If a partition has taken several incremental imports without being consolidated, the same entities can exist multiple times across the import batches. AutoGraph groups the entities by import_number, uses the largest batch as the baseline, and counts everything outside of it as changed:

multi_batch_score = (total_entities - largest_batch) / total_entities

A partition with a single batch, or with no entities at all, scores 0 here. Batch sizes of [500, 300, 200], for example, give (1000 - 500) / 1000 = 0.50.

Combined example

If the gross churn is 0.20 and the multi-batch value is 0.50, then the divergence_score is 0.50. With the threshold of 0.25, needs_reclustering is set to true.

Where the values are stored

The divergence values are stored in the partition’s rags strategy profile:

Divergence lifecycle

Where to read the score

A score is only meaningful once Layer 3 reflects the change, so insert and update do not report one at all. Their responses are produced while the work is still in Layers 1 and 2, before a targeted orchestration has created the entities the score would be measured against. Rather than return a value that is wrong by construction, AutoGraph leaves the field off.

That leaves three places to read it:

A delete can report its score directly because it removes the Layer 3 data itself, before it returns. There is no import to schedule afterwards, so nothing is still pending when the score is taken.

Next steps

• Graph Operations: The requests and responses of /v1/graph/insert, /delete, /update, and /recluster, as well as targeted orchestration with file_ids
• Design Guide: How categories, layers, and partitions fit together
• Corpus Build: Incremental builds for new categories and bulk additions
• Importer Incremental Updates: What the Importer does for Layer 3, and its reclustering endpoint
• Error Handling: HTTP codes and general troubleshooting