AutoGraph Incremental Graph Updates Reference

POST /v1/graph/insert, /delete, and /update add, remove, and replace individual documents in the existing categories of a built knowledge graph. POST /v1/graph/recluster reorganizes the communities of a partition that has changed a lot. For when to use them, see Incremental Graph Updates. For a step-by-step guide, see the Quick Start. For how the divergence is measured, see Divergence and Reorganization.

The endpoints are relative to the path of the project’s AutoGraph service, https://<EXTERNAL_ENDPOINT>:8529/autograph/{service_id_postfix}. service_id_postfix is the end of the service ID, such as vyk1t for arangodb-autograph-vyk1t. The serviceUrl in the project metadata of GET /_platform/acp/v1/project_by_name/{database}/{project} ends with it, for example .../autograph/vyk1t/.

Add documents

Adds new documents to an existing category. The API calls this operation insert. The call is synchronous.

The corpus and the target category have to exist already. If the project has exactly one category, you can omit category. Otherwise, it is required.

Request

Every document is identified by its File Manager file_id. Adding takes no inline content, so upload the document with POST /_platform/filemanager/_db/{database}/rag-input first and pass the returned ID.

{
  "files": [
    {
      "doc_name": "new-contract.pdf",
      "file_id": "<file-manager-file-id>"
    }
  ],
  "category": "legal"
}

Parameters

There is no citable_url request field. AutoGraph reads the citable URL from the custom_metadata of the File Manager file and validates it the same way a corpus build does, so it has to be an http or https URL with valid characters. Set it on the file when you upload it.

Response

{
  "results": [
    {
      "doc_name": "new-contract.pdf",
      "success": true,
      "cluster_key": "cluster_legal_0",
      "rag_partition_id": "legal_0_a"
    }
  ]
}

Adding extracts the text, creates an embedding, stores the document, assigns the closest existing cluster, and adds the membership and similarity edges. It only changes the corpus graph. To add the document to the knowledge graph, start a knowledge graph build with its file_id. AutoGraph resolves the partition itself, so you do not pass one, see Identifying documents for the knowledge graph.

The build only processes a document that is in a cluster with a strategy. Check cluster_key and rag_partition_id in the response before you start it:

The build lists skipped documents in unmatched_file_ids. If it skips all of them, it returns 409, see Telling the three 409 responses apart.

HTTP example

curl -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <token>" \
  -d '{
    "files": [{
      "doc_name": "new-contract.pdf",
      "file_id": "<file-manager-file-id>"
    }],
    "category": "legal"
  }' \
  https://<EXTERNAL_ENDPOINT>:8529/autograph/{service_id_postfix}/v1/graph/insert

Remove documents

Removes documents from an existing category. The API calls this operation delete. The call is synchronous and does the whole removal itself. It cleans up the knowledge graph first, then the corpus graph. No Importer worker is involved, and there is no background job to wait for. The response is the final result.

Once the deletion has committed, AutoGraph also deletes the files of the removed documents, with all their versions, from the File Manager. To use a document again, upload it again. If deleting a file from the File Manager fails, the removal still reports success, and the file stays in the File Manager.

Use File Manager IDs in file_ids if you can, as they are stable. Otherwise, provide doc_names, which are looked up in the requested category. You need to provide at least one of the two lists.

Request

{
  "file_ids": [
    "<file-manager-file-id-1>",
    "<file-manager-file-id-2>"
  ],
  "category": "legal"
}

By file name instead:

{
  "doc_names": ["old-contract.pdf"],
  "category": "legal"
}

Parameters

Response

{
  "results": [
    {
      "file_id": "<file-manager-file-id-1>",
      "status": "LAYER2_DELETE_STATUS_SUCCESS",
      "rag_partition_id": "legal_0_a",
      "cluster_key": "cluster_legal_0",
      "similarity_edges_removed": 8,
      "divergence_score": 0.31,
      "needs_reclustering": true
    }
  ],
  "affected_rag_partitions": ["legal_0_a"],
  "removed_rag_partitions": [],
  "layer3_results": [
    {
      "rag_partition_id": "legal_0_a",
      "status": "LAYER3_DELETE_STATUS_SUCCESS",
      "documents_removed": 1,
      "chunks_removed": 8,
      "entities_removed": 490,
      "communities_removed": 0,
      "semantic_units_removed": 0,
      "edges_removed": 0
    }
  ],
  "overall_status": "DELETE_OVERALL_STATUS_COMMITTED",
  "affected_cluster_ids": ["cluster_legal_0"],
  "removed_cluster_ids": [],
  "delete_id": "del_1711812345_a1b2c3d4"
}

The response reports the outcome of the complete deletion, knowledge graph included. There is no importerOrchestration entry to poll, because a standalone removal does not write one.

overall_status covers both stages and tells you whether you can retry.

In field and status names, LAYER3 refers to the knowledge graph and LAYER2 to the corpus graph. The status of a layer3_results entry is one of LAYER3_DELETE_STATUS_SUCCESS, LAYER3_DELETE_STATUS_FAILED, LAYER3_DELETE_STATUS_NOT_ATTEMPTED, or LAYER3_DELETE_STATUS_UNSPECIFIED.

When the deletion commits, AutoGraph calculates the divergence of every affected partition again and adds divergence_score and needs_reclustering to the result of each file. The fields are only present if overall_status is DELETE_OVERALL_STATUS_COMMITTED. A deletion that is rolled back or fails reports no divergence.

The removed documents can be in other partitions than the ones a later knowledge graph build processes, so keep needs_reclustering from this response. See Where to read the divergence.

HTTP example

curl -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <token>" \
  -d '{
    "file_ids": ["<file-manager-file-id>"],
    "category": "legal"
  }' \
  https://<EXTERNAL_ENDPOINT>:8529/autograph/{service_id_postfix}/v1/graph/delete

Replace documents

Replaces documents in an existing category with a new version. The API calls this operation update. The call is asynchronous.

Every document has to be in the requested category already. A single invalid document makes the whole batch fail before anything is changed.

Request

As when adding, every document is identified by its File Manager file_id. Replacing takes no inline content. AutoGraph always reads the new version from the File Manager. Upload the new version with POST /_platform/filemanager/_db/{database}/rag-input and pass the returned ID.

{
  "files": [
    {
      "doc_name": "existing-contract.txt",
      "file_id": "<file-manager-file-id>"
    }
  ],
  "category": "legal"
}

Parameters

POST /v1/graph/update still accepts the deprecated module field, which is only honored when category is empty. Send category. Insert and delete have no module field at all.

Immediate response

{
  "update_id": "update_1721840400_a1b2c3d4",
  "accepted": true,
  "message": "Update started",
  "results": []
}

accepted: true means that replacing has been validated and started, not that it is done. Poll the importerOrchestration entry in the metadata of your platform project, which GET /_platform/acp/v1/project_by_name/{database}/{project} returns, under projectMetadata.autographService.importerOrchestration. The progress is in its message field, as a JSON string. Replacing goes through the following phases. In their names, L3 stands for the knowledge graph and L12 for the corpus graph:

1. DELETE_L3: AutoGraph removes the old knowledge graph data with AQL queries of its own. This is synchronous, and no Importer is involved.
2. DELETE_L12: Removes the old document from the corpus graph.
3. INSERT_L12: Inserts the new version into the corpus graph and assigns a cluster to it.
4. DONE: Replacing has succeeded or failed.

Replacing uses the same logic as a standalone removal. It saves a snapshot of the knowledge graph data and removes that data before it changes the corpus graph.

The final message, parsed, looks like this:

{
  "phase": "DONE",
  "status": "completed",
  "summary": "Updated 1 file(s) in Layers 1-2; run orchestrate to re-import Layer 3",
  "files": [
    {
      "file": "existing-contract.txt",
      "result": "updated",
      "cluster": "cluster_legal_0",
      "previous_cluster": "cluster_legal_1",
      "partition": "legal_0_a",
      "file_id": "<file-manager-file-id>"
    }
  ]
}

Only file and result are always present. The service adds every other key conditionally, so read them as absent rather than empty when the value does not apply.

Replacing waits for the old knowledge graph data to be removed before it adds the new version, so that old and new knowledge graph data do not exist side by side. It is still not a database transaction.

After the documents have been replaced, start a knowledge graph build to import the new version into the knowledge graph.

HTTP example

curl -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <token>" \
  -d '{
    "files": [{
      "doc_name": "existing-contract.txt",
      "file_id": "<file-manager-file-id>"
    }],
    "category": "legal"
  }' \
  https://<EXTERNAL_ENDPOINT>:8529/autograph/{service_id_postfix}/v1/graph/update

Reorganize partitions

Rebuilds the communities of up to five partitions with the FullGraphRAG strategy from their current entities. The API calls this operation recluster. The call returns right away and reports whether each partition was taken up. This is not a queue position, see the response below. Reorganizing runs asynchronously, and only one partition is reorganized at a time.

Call this endpoint if needs_reclustering: true is reported for a partition and you decide to rebuild its communities, which is expensive. AutoGraph never reorganizes on its own. The flag is reported by GET /v1/orchestrate/{orchestration_id}, by a removal once it has committed, and by the rags profile of the partition. Adding and replacing do not report it. If a replaced document moves to another cluster, the partition it left is reported nowhere but in its rags profile. See Where to read the divergence.

Only partitions with the FullGraphRAG strategy have communities that can be rebuilt. A partition with the VectorRAG strategy has no Entities or Communities, so there is nothing to rebuild, and such a partition is never flagged.

The flag is set when the divergence_score is above the threshold, which is 0.25 and not configurable. For how the divergence is calculated, with examples, and how the values change over time, see Divergence and Reorganization.

Request

{
  "partition_ids": ["legal_0_a", "engineering_0_a"]
}

Response

{
  "results": [
    {
      "rag_partition_id": "legal_0_a",
      "accepted": true
    },
    {
      "rag_partition_id": "engineering_0_a",
      "accepted": true
    }
  ]
}

Read the result of each partition from its rags strategy profile, not from importerOrchestration. The importerOrchestration entry in your project metadata shows the status of the job that has the lock, not the result of each partition. A partition that gave up waiting for another reorganization does not appear there at all. A partition that gave up waiting for any other operation appears as failed. The rags node of the partition has the reliable result.

If needs_reclustering is still true and last_reclustered_at has not moved, the partition was not reorganized, whether it failed or never got the lock. Trigger it again. See Where the values are stored.

Notes:

• If a partition is already being reorganized, a second request is merged into it and does not start another Importer job.
• Reorganizing uses the same service-wide write lock as builds and as adding, removing, and replacing documents, so it cannot run at the same time as these operations.
• A reorganization that fails or is skipped does not clear needs_reclustering. Start it again once the lock is free.
• For a partition without entities, the job reports 0 communities over 0 entities and finishes successfully without changing anything. A completed job does not mean that anything was rebuilt. Check last_reclustered_at and the Communities of the partition.
• To learn what the Importer does while it reorganizes and how long it takes, see Importer Incremental Updates.

HTTP example

curl -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <token>" \
  -d '{
    "partition_ids": ["legal_0_a"]
  }' \
  https://<EXTERNAL_ENDPOINT>:8529/autograph/{service_id_postfix}/v1/graph/recluster

Build the knowledge graph for changed documents

Adding and replacing only change the corpus graph. To extract the entities and relationships of the new content and add them to the knowledge graph, call POST /v1/orchestrate with the file_ids of the added and replaced documents. Only those documents are processed, in the partitions that contain them. You do not name the partitions. AutoGraph finds them from the IDs.

Removing does not need this call. It removes the knowledge graph data itself.

Request

{
  "project": "my_project",
  "replicas": 1,
  "file_ids": ["<added-or-replaced-file-id>"]
}

Parameters

For the complete list of parameters, the response, the credential handling, and the three kinds of 409, see Trigger Orchestration.

Identifying documents for the knowledge graph

The build identifies documents by file_id only, not by doc_name. Adding and replacing both require a File Manager file_id for every document, so you pass the same IDs on to the build. IDs have the form rag-input-{base64url(db:path)} and refer to a File Manager path, which is why the project has to be built through the File Manager. Upload a document with POST /_platform/filemanager/_db/{database}/rag-input and pass the returned ID.

A non-empty file_ids is what keeps an already built partition eligible, because it bypasses the filter that otherwise only builds partitions that are not in the {project}_kg knowledge graph yet. Sending categories alone does not bypass that filter, so it skips exactly the partitions that your addition or replacement touched. If nothing else is stale, the request is rejected with a 409 and a Nothing to orchestrate message, see Telling the three 409 responses apart.

Response

The call returns 202 with an orchestration_id and runs in the background. IDs that matched no document are listed in unmatched_file_ids, which is only reported on this response.

{
  "orchestration_id": "orch_1711812345_a1b2c3d4",
  "success": true,
  "message": "Orchestration started",
  "unmatched_file_ids": []
}

Poll GET /v1/orchestrate/{orchestration_id} until status leaves running. Each entry of jobs has the divergence_score and needs_reclustering of its partition. jobs only lists the partitions that the build processed. The partitions of removed documents, and the partitions that replaced documents moved away from, are not in it. To see all of them, read the rags profiles, for example with GET /v1/rag-strategizer/strategy.

HTTP example

curl -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <token>" \
  -d '{
    "project": "my_project",
    "replicas": 1,
    "file_ids": ["<added-or-replaced-file-id>"]
  }' \
  https://<EXTERNAL_ENDPOINT>:8529/autograph/{service_id_postfix}/v1/orchestrate

For all steps in order, with example requests, see the Quick Start.