HTTP API for Graph Analytics Engines

Workflow

The following list outlines how you can use Graph Analytics Engines (GAEs). How to perform the steps is detailed in the subsequent sections.

1. Determine the approximate size of the data that you will load into the GAE and ensure the machine to run the engine on has sufficient memory. The data as well as the temporarily needed space for computations and results needs to fit in memory.
2. Start a graphanalytics service via the ACP service that manages various Platform components for graph intelligence and machine learning. It only takes a few seconds until the engine service can be used. The engine runs adjacent to the pods of the ArangoDB core.
3. Load graph data from the ArangoDB core into the engine. You can load named graphs or sets of node and edge collections. This loads the edge information and a configurable subset of the node attributes.
4. Run graph algorithms on the data. You only need to load the data once per engine and can then run various algorithms with different settings.
5. Check the progress of the job that runs the algorithm. The job is done when its progress is equal to its total, which needs to be the case before you can store the computation results.
6. Write the computation results to a dedicated collection in the ArangoDB core.
7. Stop the engine service once you are done.

Authentication

You can use any of the available authentication methods the Contextual Data Platform supports to start and stop graphanalytics services via the ACP service as well as to authenticate requests to the Engine API.

• HTTP Basic Authentication
• Access tokens
• JWT session tokens

Start and stop Graph Analytics Engines

GAEs are deployed and deleted via the Arango Control Plane (ACP) service in the Contextual Data Platform.

If you use cURL, you need to use the -k / --insecure option for requests if the data platform deployment uses a self-signed certificate (default).

Start a graphanalytics service

Start a GAE via the ACP service with an empty request body. This returns a serviceId that you need to construct the Engine API URL in the next section.

# Set your Contextual Data Platform endpoint
EXTERNAL_ENDPOINT="<your-endpoint>"  # Example: data-platform.example.org

# Get an authentication token (example with JWT session token)
ADB_TOKEN=$(curl -sSk -X POST \
  -d '{"username":"root","password":""}' \
  "https://$EXTERNAL_ENDPOINT:8529/_open/auth" | jq -r .jwt)

# Start the Graph Analytics service
SERVICE=$(curl -sSk -H "Authorization: bearer $ADB_TOKEN" \
  -X POST "https://$EXTERNAL_ENDPOINT:8529/_platform/acp/v1/graphanalytics")

# Extract the trailing segment of the serviceId (needed for Engine API URL construction)
SERVICE_ID=$(echo "$SERVICE" | jq ".serviceInfo.serviceId")

if [[ "$SERVICE_ID" == "null" ]]; then 
  echo "Error starting Graph Analytics Engine"
else
  SERVICE_ID_POSTFIX=$(echo "$SERVICE_ID" | jq -r 'split("-") | last')
  echo "Graph Analytics service started successfully"
  echo "serviceIdPostfix: $SERVICE_ID_POSTFIX"
  echo "Save the postfix for constructing the Engine API URL"
fi

echo "$SERVICE" | jq

Example response:

{
  "serviceInfo": {
    "serviceId": "arangodb-gral-tqcge",
    "description": "Install complete",
    "status": "DEPLOYED",
    "namespace": "arango",
    "managingEntity": "ACP"
  }
}

List the services

You can list all running services managed by the ACP service, including the graphanalytics services:

curl -sSk -H "Authorization: bearer <ADB_TOKEN>" \
  -X POST "https://data-platform.example.org:8529/_platform/acp/v1/list_services" | jq

Stop a graphanalytics service

Delete the desired engine via the ACP service using the serviceId:

curl -sSk -H "Authorization: bearer <ADB_TOKEN>" \
  -X DELETE "https://data-platform.example.org:8529/_platform/acp/v1/service/arangodb-gral-tqcge" | jq

Engine API

The Engine API lets you load data, run algorithms, and manage results.

The Engine API URL is constructed from multiple parts:

Where:

• <EXTERNAL_ENDPOINT>: Your Contextual Data Platform endpoint (e.g., data-platform.example.org)
• :serviceIdPostfix: From the ACP service response when you started the service (the serviceId segment after the last hyphen)

Example:

https://data-platform.example.org:8529/gral/tqcge/v1/pagerank

The port :8529 is added when constructing URLs.

You can also list all service IDs using kubectl:

kubectl -n arangodb get svc arangodb-gral -o jsonpath="{.spec.selector.release}"

For convenience, you can store the Engine API base URL in a variable:

# Your Platform endpoint (without port)
EXTERNAL_ENDPOINT="data-platform.example.org"

# Service ID from when you started the service
SERVICE_ID_POSTFIX="tqcge"

# Construct the Engine API base URL
ENGINE_URL="https://$EXTERNAL_ENDPOINT:8529/gral/$SERVICE_ID_POSTFIX"

This makes subsequent requests shorter and easier to manage. Alternatively, you can use the full URL directly in each request.

Verify the connection:

curl -sSk -H "Authorization: bearer $ADB_TOKEN" "$ENGINE_URL/v1/jobs"

Authentication

Authenticate Engine API requests using a bearer token in the HTTP header:

Authorization: bearer <TOKEN>

You can save the token in a variable to ease scripting. Note that this should be the token string only and not include quote marks. The following examples assume Bash as the shell and that the curl and jq commands are available.

Example with JWT session token:

# Platform endpoint (from previous section)
EXTERNAL_ENDPOINT="data-platform.example.org"

# Trailing segment of the serviceId from when you started the service (after last hyphen)
SERVICE_ID_POSTFIX="tqcge"

# Get authentication token
ADB_TOKEN=$(curl -sSk -X POST \
  -d '{"username":"<ADB_USER>","password":"<ADB_PASS>"}' \
  "https://$EXTERNAL_ENDPOINT:8529/_open/auth" | jq -r '.jwt')

# Example: Use token to verify connection
curl -sSk -H "Authorization: bearer $ADB_TOKEN" "https://$EXTERNAL_ENDPOINT:8529/gral/$SERVICE_ID_POSTFIX/v1/jobs"

All requests to the engine API start jobs, each representing an operation. You can check the progress of operations and check if errors occurred. You can submit jobs concurrently and they also run concurrently.

You can find the API reference documentation with detailed descriptions of the request and response data structures at https://apiref.arango.ai/#gral .

Request and response payloads are JSON-encoded in the engine API.

Load data

Import graph data from a database of the ArangoDB deployment. You can import named graphs as well as sets of node and edge collections (see Managed and unmanaged graphs).

Example:

curl -H "Authorization: bearer $ADB_TOKEN" -XPOST -d '{"database":"_system","graph_name":"connectedComponentsGraph"}' "https://data-platform.example.org:8529/gral/tqcge/v1/loaddata"

Parameters:

• database (string, required): The database to load the graph from.
• graph_name (string, optional): The name of a named graph to load. Required if vertex_collections and edge_collections are not specified.
• vertex_collections (array of strings, optional): Vertex collections to load. Required together with edge_collections if graph_name is not specified.
• edge_collections (array of strings, optional): Edge collections to load. Required together with vertex_collections if graph_name is not specified.
• vertex_attributes (array of strings, optional): The names of vertex attributes to load. Only the specified attributes are loaded into memory. If omitted or empty, only the graph topology is loaded without any vertex attributes.
• parallelism (integer, optional): The number of parallel threads to use for loading data (default: 4).
• batch_size (integer, optional): The number of documents per batch (default: 400000).

The response contains a job_id and graph_id. Use the job_id to track the loading progress via the Jobs API.

Example with vertex collections, edge collections, and vertex attributes:

curl -H "Authorization: bearer $ADB_TOKEN" -XPOST \
  -d '{
    "database": "_system",
    "vertex_collections": ["persons"],
    "edge_collections": ["knows"],
    "vertex_attributes": ["name", "age"],
    "parallelism": 8,
    "batch_size": 100000
  }' \
  "https://data-platform.example.org:8529/gral/tqcge/v1/loaddata"

Load data using AQL queries

Import graph data using custom AQL queries. This gives you full control over which data to load, including the ability to filter, transform, or traverse the graph during loading. Each AQL query must return documents containing vertices and/or edges arrays.

Queries are organized into phases, where each phase is a set of queries that run in parallel. You can have multiple phases (query groups), and they are executed sequentially, where each phase completes before the next begins. This lets you order dependencies, for example, loading vertices in the first phase and edges in the second.

Each query must return documents with the following format:

{"vertices": [{"_id": "collection/key", ...}], "edges": [{"_from": "coll/a", "_to": "coll/b", ...}]}

Vertices require the _id field. Edges require the _from and _to fields. A single query can return both vertices and edges at the same time, for example, when using AQL traversals.

Example:

curl -H "Authorization: bearer $ADB_TOKEN" -XPOST \
  -d '{
    "database": "_system",
    "vertex_attributes": [
      {"name": "name", "data_type": "String"},
      {"name": "age", "data_type": "U64"}
    ],
    "edge_attributes": [
      {"name": "weight", "data_type": "F64"}
    ],
    "phases": [
      {"queries": [{"query": "FOR v IN @@V RETURN {vertices: [{_id: v._id, name: v.name, age: v.age}]}", "bind_vars": {"@V": "myVertices"}}]},
      {"queries": [{"query": "FOR e IN @@E RETURN {edges: [{_from: e._from, _to: e._to, weight: e.weight}]}", "bind_vars": {"@E": "myEdges"}}]}
    ]
  }' \
  "https://data-platform.example.org:8529/gral/tqcge/v1/loaddataaql"

Parameters:

• database (string, required): The database to run the queries against.
• phases (array, required): A list of query groups, executed sequentially. Each group contains a queries array of AQL queries that run in parallel. Each query object has:
  • query (string): The AQL query string.
  • bind_vars (object): Bind parameters as key-value pairs.
• vertex_attributes (array, optional): A list of vertex attributes to load, each with a name (string) and data_type (one of Bool, String, U64, I64, F64, JSON).
• edge_attributes (array, optional): A list of edge attributes to load, with the same structure as vertex_attributes.
• batch_size (integer, optional): The number of documents per batch (default: 400000).

The response contains a job_id and graph_id. Use the job_id to track the loading progress via the Jobs API. The loading job reports a total of 2 progress steps: 1 after all vertices have been processed, and 2 when all edges are processed and the graph is ready.

Example using a graph traversal (single-phase load):

curl -H "Authorization: bearer $ADB_TOKEN" -XPOST \
  -d '{
    "database": "_system",
    "vertex_attributes": [
      {"name": "name", "data_type": "String"},
      {"name": "age", "data_type": "U64"}
    ],
    "edge_attributes": [
      {"name": "weight", "data_type": "F64"}
    ],
    "phases": [
      {"queries": [{"query": "FOR v, e IN 0..3 OUTBOUND \"V/0\" @@edgeCollection RETURN {vertices: [v], edges: e ? [e] : []}", "bind_vars": {"@edgeCollection": "myEdges"}}]}
    ]
  }' \
  "https://data-platform.example.org:8529/gral/tqcge/v1/loaddataaql"

Run algorithms

PageRank

Example:

GRAPH_ID="234"
curl -H "Authorization: bearer $ADB_TOKEN" -XPOST -d "{\"graph_id\":$GRAPH_ID,\"damping_factor\":0.85,\"maximum_supersteps\":500,\"seeding_attribute\":\"seed_attr\"}" "https://data-platform.example.org:8529/gral/tqcge/v1/pagerank"

PageRank is a well known algorithm to rank nodes in a graph: the more important a node, the higher rank it gets. It goes back to L. Page and S. Brin’s paper  and is used to rank pages in search engines (hence the name). The algorithm runs until the execution converges. To run for a fixed number of iterations, use the maximum_supersteps parameter.

The rank of a node is a positive real number. The algorithm starts with every node having the same rank (one divided by the number of nodes) and sends its rank to its out-neighbors. The computation proceeds in iterations. In each iteration, the new rank is computed according to the formula ( (1 - damping_factor) / total number of nodes) + (damping_factor * the sum of all incoming ranks). The value sent to each of the out-neighbors is the new rank divided by the number of those neighbors, thus every out-neighbor gets the same part of the new rank.

The algorithm stops when at least one of the two conditions is satisfied:

• The maximum number of iterations is reached. This is the same maximum_supersteps parameter as for the other algorithms.
• Every node changes its rank in the last iteration by less than a certain threshold. The threshold is hardcoded to 0.0000001.

It is possible to specify an initial distribution for the node documents in your graph. To define these seed ranks / centralities, you can specify a seeding_attribute in the properties for this algorithm. If the specified field is set on a document and the value is numeric, then it is used instead of the default initial rank of 1 / numNodes.

Parameters:

• graph_id
• damping_factor
• maximum_supersteps
• seeding_attribute (optional, for seeded PageRank)

The result is the rank of each node.

Weakly Connected Components (WCC)

Example:

GRAPH_ID="234"
curl -H "Authorization: bearer $ADB_TOKEN" -XPOST -d "{\"graph_id\":$GRAPH_ID}" "https://data-platform.example.org:8529/gral/tqcge/v1/wcc"

The weakly connected component algorithm partitions a graph into maximal groups of nodes, so that within a group, all nodes are reachable from each node by following the edges, ignoring their direction.

In other words, each weakly connected component is a maximal subgraph such that there is a path between each pair of nodes where one can also follow edges against their direction in a directed graph.

Parameters:

• graph_id

The result is a component ID for each node. All nodes from the same component obtain the same component ID, every two nodes from different components obtain different IDs.

Strongly Connected Components (SCC)

Example:

GRAPH_ID="234"
curl -H "Authorization: bearer $ADB_TOKEN" -XPOST -d "{\"graph_id\":$GRAPH_ID}" "https://data-platform.example.org:8529/gral/tqcge/v1/scc"

The strongly connected components algorithm partitions a graph into maximal groups of nodes, so that within a group, all nodes are reachable from each node by following the edges in their direction.

In other words, a strongly connected component is a maximal subgraph, where for every two nodes, there is a path from one of them to the other, forming a cycle. In contrast to a weakly connected component, one cannot follow edges against their direction.

Parameters:

• graph_id

The result is a component ID for each node. All nodes from the same component obtain the same component ID, every two nodes from different components obtain different IDs.

Vertex Centrality

Centrality measures help identify the most important nodes in a graph. They can be used in a wide range of applications: to identify influencers in social networks, or middlemen in terrorist networks.

There are various definitions for centrality, the simplest one being the node degree. These definitions were not designed with scalability in mind. It is probably impossible to discover an efficient algorithm which computes them in a distributed way. Fortunately there are scalable substitutions available, which should be equally usable for most use cases.

Betweenness Centrality

Example:

GRAPH_ID="234"
curl -H "Authorization: bearer $ADB_TOKEN" -XPOST -d "{\"graph_id\":$GRAPH_ID,\"k\":0,\"undirected\":false,\"normalized\":true}" "https://data-platform.example.org:8529/gral/tqcge/v1/betweennesscentrality"

A relatively expensive algorithm with complexity O(V*E) where V is the number of nodes and E is the number of edges in the graph.

Betweenness-centrality can be approximated by cheaper algorithms like Line Rank but this algorithm strives to compute accurate centrality measures.

Parameters:

• graph_id
• k (number of start nodes, 0 = all)
• undirected
• normalized
• parallelism

The result is a centrality measure for each node.

LineRank

Example:

GRAPH_ID="234"
curl -H "Authorization: bearer $ADB_TOKEN" -XPOST -d "{\"graph_id\":$GRAPH_ID,\"damping_factor\":0.0000001,\"maximum_supersteps\":500}" "https://data-platform.example.org:8529/gral/tqcge/v1/linerank"

Another common measure is the betweenness centrality : It measures the number of times a node is part of shortest paths between any pairs of nodes. For a node v betweenness is defined as:

Where the σ represents the number of shortest paths between x and y, and σ(v) represents the number of paths also passing through a node v. By intuition a node with higher betweenness centrality has more information passing through it.

LineRank approximates the random walk betweenness of every node in a graph. This is the probability that someone, starting on an arbitrary node, visits this node when they randomly choose edges to visit.

The algorithm essentially builds a line graph out of your graph (switches the nodes and edges), and then computes a score similar to PageRank. This can be considered a scalable equivalent to vertex betweenness, which can be executed distributedly in ArangoDB. The algorithm is from the paper Centralities in Large Networks: Algorithms and Observations (U Kang et.al. 2011).

Parameters:

• graph_id
• damping_factor
• maximum_supersteps

The result is the line rank of each node.

Community Detection

Graphs based on real world networks often have a community structure. This means it is possible to find groups of nodes such that each node group is internally more densely connected than outside the group. This has many applications when you want to analyze your networks, for example Social networks include community groups (the origin of the term, in fact) based on common location, interests, occupation, etc.

Label Propagation

Example:

GRAPH_ID="234"
curl -H "Authorization: bearer $ADB_TOKEN" -XPOST -d "{\"graph_id\":$GRAPH_ID,\"start_label_attribute\":\"start_attr\",\"synchronous\":false,\"random_tiebreak\":false,\"maximum_supersteps\":500}" "https://data-platform.example.org:8529/gral/tqcge/v1/labelpropagation"

Label Propagation  can be used to implement community detection on large graphs.

The algorithm assigns an initial community identifier to every node in the graph using a user-defined attribute. The idea is that each node should be in the community that most of its neighbors are in at the end of the computation.

In each iteration of the computation, a node sends its current community ID to all its neighbor nodes, inbound and outbound (ignoring edge directions). After that, each node adopts the community ID it received most frequently in the last step.

It can happen that a node receives multiple most frequent community IDs. In this case, one is chosen either randomly or using a deterministic choice depending on a setting for the algorithm. The rules for a deterministic tiebreak are as follows:

• If a node obtains only one community ID and the ID of the node from the previous step, its old ID, is less than the obtained ID, the old ID is kept.
• If a node obtains more than one ID, its new ID is the lowest ID among the most frequently obtained IDs. For example, if the initial IDs are numbers and the obtained IDs are 1, 2, 2, 3, 3, then 2 is the new ID.
• If, however, no ID arrives more than once, the new ID is the minimum of the lowest obtained IDs and the old ID. For example, if the old ID is 5 and the obtained IDs are 3, 4, 6, then the new ID is 3. If the old ID is 2, it is kept.

The algorithm runs until it converges or reaches the maximum iteration bound. It may not converge on large graphs if the synchronous variant is used.

• Synchronous: The new community ID of a node is based on the community IDs of its neighbors from the previous iteration. With (nearly) bipartite  subgraphs, this may lead to the community IDs changing back and forth in each iteration within the two halves of the subgraph.
• Asynchronous: A node determines the new community ID using the most up-to-date community IDs of its neighbors, whether those updates occurred in the current iteration or the previous one. The order in which nodes are updated in each iteration is chosen randomly. This leads to more stable community IDs.

Parameters:

• graph_id
• start_label_attribute
• synchronous
• random_tiebreak
• maximum_supersteps

The result is a community ID for each node.

Attribute Propagation

Example:

GRAPH_ID="234"
curl -H "Authorization: bearer $ADB_TOKEN" -XPOST -d "{\"graph_id\":$GRAPH_ID,\"start_label_attribute\":\"start_attr\",\"synchronous\":false,\"backwards\":false,\"maximum_supersteps\":500}" "https://data-platform.example.org:8529/gral/tqcge/v1/attributepropagation"

The attribute propagation algorithm can be used to implement community detection. It works similar to the label propagation algorithm, but every node additionally accumulates a memory of observed labels instead of forgetting all but one label.

The algorithm assigns an initial value to every node in the graph using a user-defined attribute. The attribute value can be a list of strings to initialize the set of labels with multiple labels.

In each iteration of the computation, the following steps are executed:

1. Each node propagates its set of labels along the edges to all direct neighbor nodes. Whether inbound or outbound edges are followed depends on an algorithm setting.
2. Each node adds the labels it receives to its own set of labels.

After a specified maximal number of iterations or if no label set changes any more, the algorithm stops.

Parameters:

• graph_id
• start_label_attribute: The attribute to initialize labels with. Use "@id" to use the document IDs of the nodes.
• synchronous: Whether synchronous or asynchronous label propagation is used.
• backwards: Whether labels are propagated in edge direction (false) or the opposite direction (true).
• maximum_supersteps: Maximum number of iterations.

The result is the set of accumulated labels of each node.

Store job results

Example:

JOB_ID="123"
curl -H "Authorization: bearer $ADB_TOKEN" -X POST -d "{\"database\":\"_system\",\"target_collection\":\"coll\",\"job_ids\":[$JOB_ID],\"attribute_names\":[\"attr\"]}" "https://data-platform.example.org:8529/gral/tqcge/v1/storeresults"

You need to specify to which ArangoDB database and target_collection to save the results to. They need to exist already.

Whereas an engine can load data from multiple collections, it writes the results to a single target_collection only and never updates the source documents. Use a dedicated collection for the results. It is considerably faster to create new documents that only hold the computed attributes than to update existing documents, and you can join the results with your source data at query time.

You also need to specify a list of job_ids with one or more jobs that have run graph algorithms.

Each algorithm outputs one value for each node, and you can define the target attribute to store the information in with attribute_names. It has to be a list with one attribute name for every job in the job_ids list.

You can optionally set the degree of parallelism and the batch_size for saving the data.

Parameters:

• database
• target_collection
• job_ids
• attribute_names
• parallelism
• batch_size

List all jobs

Example:

curl -H "Authorization: bearer $ADB_TOKEN" "https://data-platform.example.org:8529/gral/tqcge/v1/jobs"

List all active and finished jobs.

Get a job

Example:

JOB_ID="123"
curl -H "Authorization: bearer $ADB_TOKEN" "https://data-platform.example.org:8529/gral/tqcge/v1/jobs/$JOB_ID"

Get detailed information about a specific job.

Delete a job

Example:

JOB_ID="123"
curl -H "Authorization: bearer $ADB_TOKEN" -X DELETE "https://data-platform.example.org:8529/gral/tqcge/v1/jobs/$JOB_ID"

Delete a specific job.

List all graphs

Example:

curl -H "Authorization: bearer $ADB_TOKEN" "https://data-platform.example.org:8529/gral/tqcge/v1/graphs"

List all loaded sets of graph data that reside in the memory of the engine node.

Get a graph

Example:

GRAPH_ID="234"
curl -H "Authorization: bearer $ADB_TOKEN" "https://data-platform.example.org:8529/gral/tqcge/v1/graphs/$GRAPH_ID"

Get detailed information about a specific set of graph data.

Delete a graph

Example:

GRAPH_ID="234"
curl -H "Authorization: bearer $ADB_TOKEN" -X DELETE "https://data-platform.example.org:8529/gral/tqcge/v1/graphs/$GRAPH_ID"

Delete a specific set of graph data, removing it from the memory of the engine node.