ArangoDB v4.x is under development and not released yet.

This documentation is not final and potentially incomplete.

API Changes in ArangoDB 3.12

A summary of the changes to the HTTP API and other interfaces that are relevant for developers, like maintainers of drivers and integrations for ArangoDB

HTTP RESTful API

Behavior changes

VelocyStream protocol removed

ArangoDB’s own bi-directional asynchronous binary protocol VelocyStream is no longer supported.

The server immediately closes the connection if you attempt to use the VelocyStream protocol. If you specify any scheme starting with vst in the --server.endpoint startup option of a client tool, the HTTP protocol is used instead.

The following metrics related to VelocyStream have been removed:

arangodb_request_body_size_vst
arangodb_vst_connections_total

VelocyPack remains as ArangoDB’s binary storage format and you can continue to use it in transport over the HTTP protocol, as well as use JSON over the HTTP protocol.

HTTP headers

The following long-deprecated features have been removed from ArangoDB’s HTTP server:

Overriding the HTTP method by setting one of the HTTP headers:
- x-http-method
- x-http-method-override
- x-method-override
This functionality posed a potential security risk and was thus removed. Previously, it was only enabled when explicitly starting the server with the --http.allow-method-override startup option. The functionality has now been removed and setting the startup option does nothing.
Optionally hiding ArangoDB’s server response header. This functionality could optionally be enabled by starting the server with the startup option --http.hide-product-header. The functionality has now been removed and setting the startup option does nothing.

`--database.extended-names` enabled by default

The --database.extended-names startup option is now enabled by default. The names of databases, collections, Views, and indexes may contain Unicode characters using the default settings.

Collection API

When creating a collection using the POST /_api/collection endpoint, the server log now displays a deprecation message if illegal combinations and unknown attributes and values are detected in the request body.

Note that all invalid elements and value combinations will be rejected in future versions. The following options are already validated more strictly in v3.12 and incorrect use can lead to errors:

keyOptions: The increment and offset sub-attributes are only allowed if the type sub-attribute is "autoincrement". The lastValue sub-attribute is only allowed if the type sub-attribute is "traditional", "autoincrement", or "padded".
shardKeys: Each array element needs to be a string.

Index API

Stored values can contain the `_id` attribute

The usage of the _id system attribute was previously disallowed for persistent indexes inside of storedValues. This is now allowed in v3.12.

Note that it is still forbidden to use _id as a top-level attribute or sub-attribute in fields of persistent indexes. On the other hand, inverted indexes have been allowing to index and store the _id system attribute.

Optimizer rule changes

Due to the improved joins in AQL, there is a new join-index-nodes optimizer rule and a JoinNode that may appear in execution plans.

The remove-unnecessary-projections AQL optimizer rule has been renamed to optimize-projections and now includes an additional optimization.

Moreover, a remove-unnecessary-calculations-4 and batch-materialize-documents rule have been added.

A push-limit-into-index rule has been added in v3.12.2.

A replace-entries-with-object-iteration rule has been added in v3.12.3.

A use-index-for-collect and a use-vector-index rule have been added in v3.12.4.

A push-filter-into-enumerate-near rule has been added in v3.12.7.

A materialize-for-enumerate-near rule has been added in v3.12.10.

A replace-any-eq-with-in rule has been added in v3.12.10.

The affected endpoints are POST /_api/cursor, POST /_api/explain, and GET /_api/query/rules.

Gharial API

The PATCH /_api/gharial/{graph}/edge/{collection}/{edge} endpoint to update edges in named graphs now validates the referenced node when modifying either the _from or _to edge attribute. Previously, the validation only occurred if both were set in the request.

Validation of `smartGraphAttribute` in SmartGraphs

Introduced in: v3.10.13, v3.11.7

The attribute defined by the smartGraphAttribute graph property is not allowed to be changed in the documents of SmartGraph node collections. This is now strictly enforced. You must set the attribute when creating a document. Any attempt to modify or remove the attribute afterward by update or replace operations now throws an error. Previously, the smartGraphAttribute value was checked only when inserting documents into a SmartGraph node collection, but not for update or replace operations.

The missing checks on update and replace operations allowed to retroactively modify the value of the smartGraphAttribute for existing documents, which could have led to problems when the data of such a SmartGraph node collection was replicated to a new follower shard. On the new follower shard, the documents went through the full validation and led to documents with modified smartGraphAttribute values being rejected on the follower. This could have led to follower shards not getting in sync.

Now, the value of the smartGraphAttribute is fully validated with every insert, update, or replace operation, and every attempt to modify the value of the smartGraphAttribute retroactively fails with the 4003 error, ERROR_KEY_MUST_BE_PREFIXED_WITH_SMART_GRAPH_ATTRIBUTE. Additionally, if upon insertion the smartGraphAttribute is missing for a SmartGraph node, the error code is error 4001, ERROR_NO_SMART_GRAPH_ATTRIBUTE.

To retroactively repair the data in any of the affected collections, it is possible to update every (affected) document with the correct value of the smartGraphAttribute via an AQL query as follows:

FOR doc IN @@collection
  LET expected = SUBSTRING(doc._key, 0, FIND_FIRST(doc._key, ':'))
  LET actual = doc.@attr
  FILTER expected != actual
  UPDATE doc WITH {@attr: expected} IN @@collection
  COLLECT WITH COUNT INTO updated
  RETURN updated

This updates all documents with the correct (expected) value of the smartGraphAttribute if it deviates from the expected value. The query returns the number of updated documents as well.

The bind parameters necessary to run this query are:

@@collection: name of a SmartGraph node collection to be updated
@attr: attribute name of the smartGraphAttribute of the collection

Limit to the number of databases in a deployment

Introduced in: v3.10.10, v3.11.2

The new --database.max-databases startup option can cap the number of databases and creating databases using the POST /_api/database endpoint can thus now fail for this reason if your deployment is at or above the configured maximum. Example:

{
  "code": 400,
  "error": true,
  "errorMessage": "unable to create additional database because it would exceed the configured maximum number of databases (2)",
  "errorNum": 32
}

Adjustable Stream Transaction size

The Stream Transactions HTTP API may now allow larger transactions or be limited to smaller transactions because the maximum transaction size can now be configured with the --transaction.streaming-max-transaction-size startup option. The default value remains 128 MiB up to v3.12.3. From v3.12.4 onward, the default value is 512 MiB.

Analyzer API

The /_api/analyzer endpoints supports a new multi_delimiter Analyzer that accepts an array of strings in a delimiters attribute of the properties object.

Adjustable `writeConcern` for collections with `distributeShardsLike`

Collections that are sharded like another collection via the distributeShardsLike property use the replicationFactor, numberOfShards, and shardingStrategy properties of the prototype collection. In previous versions, the writeConcern property of the prototype collection was used as well. Now, you can independently set a writeConcern when creating a collection with distributeShardsLike. The property defaults to the writeConcern of the prototype collection if you don’t specify it explicitly. You can adjust the writeConcern later on in either case.

Log API

The /_admin/log/* endpoints no longer use the ldap log topic. Changing the log level of the ldap topic or any other unknown topic is not an error, however. Also see Incompatible changes in ArangoDB 3.12.

A new deprecation log topic has been added. It warns about deprecated features and the usage of options that will not be allowed or have no effect in a future version.

Error code `12` removed

The unused error ERROR_OUT_OF_MEMORY_MMAP with the number 12 has been removed.

`mmap` log topic removed

Introduced in: v3.12.1

The mmap log topic for logging information related to memory mapping has been unused since v3.12.0 and has now been removed. The /_admin/log/level endpoints no longer include this log topic in responses and attempts to set the log level for this topic are ignored.

Permission checks for Stream Transactions in cluster

Introduced in: v3.12.10-1

Operations that you execute as part of a Stream Transaction by setting the x-arango-trx-id header now explicitly check whether you have the required collection access level. This closes a gap in cluster deployments for collections that are not declared in the collections attribute when beginning the transaction. It affects the following operations:

All operations of the Document API
Getting the document count and truncating a collection via the Collection API

If you don’t have the required access level for the collection, these requests now fail with an HTTP 403 Forbidden error and the ERROR_FORBIDDEN (11) error number. Previously, Coordinators didn’t check the access level for undeclared collections:

Write operations failed with an HTTP 400 Bad Request error and the ERROR_TRANSACTION_UNREGISTERED_COLLECTION (1652) error number.
Reading a single document by key as well as getting the document count succeeded.

Single servers are unaffected. They check the access level when they add an undeclared collection to a running transaction and therefore already reject such requests with an HTTP 403 Forbidden error. Operations outside of Stream Transactions as well as AQL queries are unaffected, too.

Access token lifetime

Introduced in: v3.12.10-1

When requesting a personal access token via the POST /_api/token/{user} endpoint, the server may not honor the requested valid_until timestamp and issue the access token with a shorter validity. The maximum lifetime (in seconds) is controlled by the new --auth.maximal-access-token-expiry-time arangod startup option. The default is 604800 (1 week).

Refactored authorization system

Introduced in: v3.12.11

The authorization system has been refactored to support Role-Based Access Control (RBAC). The following behavior changes are side effects of this refactoring. They specifically apply to the classic authorization system, so when not using RBAC.

Note that enabling RBAC changes the behavior more significantly because different permissions are needed, and the API under RBAC is designed to not disclose whether a resource exists if a user has no permission to access it.

Error number for write operations in read-only mode

Nearly every endpoint that writes something refuses to perform the operation if the server is in read-only mode. For those that require write access to a collection, the HTTP status code remains 403 Forbidden, but the reported error number has intentionally been changed from ERROR_FORBIDDEN (11) to ERROR_ARANGO_READ_ONLY (1004).

This only affects requests where the user account you authenticate with actually has read/write access to the collection but the read-only mode prevents the write. If the access level is insufficient, the error number remains 11. The superuser is not restricted by the read-only mode.

Access token management in read-only mode

The following endpoints for managing access tokens now respect the read-only mode of the server:

POST /_api/token/{user}
DELETE /_api/token/{user}/{token-id}

Up to v3.12.10, they allowed creating and deleting access tokens even if the server was in read-only mode, provided that the user account you authenticate with has read/write access to the _system database. Now, such requests fail with an HTTP 403 Forbidden error and the ERROR_ARANGO_READ_ONLY (1004) error number. The superuser can still create and delete access tokens in read-only mode.

Permission checks for the AQL query results cache API

The following endpoints of the AQL query results cache API now require at least read access to the _system database, in addition to the read access to the specified database that was already required before:

PUT /_api/query-cache/properties
DELETE /_api/query-cache

Up to v3.12.10, they didn’t check the access level for the _system database. If you don’t have the required access level, these requests now fail with an HTTP 403 Forbidden error.

Error response for inaccessible databases in the Activities API

If the user account you authenticate with has no access to the database you target with the experimental GET /_arango/experimental/_admin/activities endpoint, the request now fails with an HTTP 404 Not Found error and the ERROR_ARANGO_DATABASE_NOT_FOUND (1228) error number. Up to v3.12.10, the request failed with an HTTP 401 Unauthorized error and the ERROR_FORBIDDEN (11) error number.

Endpoint return value changes

Storage engine API

The storage engine API at GET /_api/engine does not return the attribute dfdb anymore.
On single servers and DB-Servers, the GET /_api/engine endpoint now returns an endianness attribute. Currently, only Little Endian is supported as an architecture by ArangoDB. The value is therefore "little".

Storage engine statistics API

Introduced in: v3.12.8

The GET /_api/engine/stats endpoint previously returned hard-to-read strings under columnFamilies.*.dbstats:

{
  ...
  "columnFamilies" : {
    "definitions" : {
      "dbstats" : "\n** Compaction Stats [default] **\nLevel    Files   Size     Score Read(GB) ...",
      "memory" : 15673
    },
    "documents" : {
      "dbstats" : "\n** Compaction Stats [Documents] **\nLevel    Files   Size     Score Read(GB) ...",
      "memory" : 135430
    },
    ...
  }
}

It now returns all information in a structured way:

{
  ...
  "columnFamilies" : {
    "definitions" : {
      "compactionStats" : {
        "sum" : {
          "numFiles" : 2,
          "sizeBytes" : 230164,
          "readGB" : 0.000015,
          "writeGB" : 0.000015,
          "readMBps" : 0.797984,
          "writeMBps" : 0.80012,
          "compSec" : 0.01919,
          "compCount" : 4,
          "keyIn" : 176,
          "keyDrop" : 88
        },
        "levels" : [ ... ],
      },
    ...
    }
  }
}

Endpoints added

Effective and available startup options

The new GET /_admin/options and GET /_admin/options-description HTTP API endpoints allow you to return the effective configuration and the available startup options of the queried arangod instance.

Previously, it was only possible to fetch the current configuration on single servers and Coordinators using a JavaScript transaction, and to list the available startup options with --dump-options.

See the HTTP interface for administration for details.

Available key generators

You can now retrieve the available key generators for collections using the new GET /_api/key-generators endpoint.

See the HTTP API description

Shard usage metrics

With GET /_admin/usage-metrics you can retrieve detailed shard usage metrics on DB-Servers.

These metrics can be enabled by setting the --server.export-shard-usage-metrics startup option to enabled-per-shard to make DB-Servers collect per-shard usage metrics, or to enabled-per-shard-per-user to make DB-Servers collect usage metrics per shard and per user whenever a shard is accessed.

For more information, see the HTTP API description and Monitoring per collection/database/user.

Reset log levels

Introduced in: v3.12.1

A new DELETE /_admin/log/level endpoint has been added that lets you reset the log level settings to the values they had at server startup. This is useful for tools that temporarily change log levels but do not want to fetch and remember the previous log levels settings. Such tools can now simply call this new endpoint to restore the original log levels.

See the Log API for details.

Query plan cache API

Introduced in: v3.12.4

Two endpoints have been added to let you list the entries and clear the cache for AQL execution plans. Query plan caching works on a per-database basis.

GET /_api/query-plan-cache
DELETE /_api/query-plan-cache

See HTTP interface for the query plan cache for details.

API call recording

Introduced in: v3.12.5

A new /_admin/server/api-calls endpoint has been added to let you retrieve a list of the most recent requests with a timestamp and the endpoint. This feature is for debugging purposes.

See HTTP interface for server logs for details.

AQL query recording

Introduced in: v3.12.6

A new /_admin/server/aql-queries endpoint has been added to let you retrieve a list of the most recent AQL queries with a timestamp and information about the submitted queries. This feature is for debugging purposes.

See HTTP interface for server logs for details.

Access tokens

Introduced in: v3.12.5

New endpoints have been added to let you manage access tokens.

POST /_api/token/{user}
GET /_api/token/{user}
DELETE /_api/token/{user}/{token-id}

See the HTTP API documentation.

Also see Authentication with access tokens for related API changes.

Deployment ID

Introduced in: v3.12.6

Licenses are now bound to specific deployments. Each deployment has a unique identifier that you can retrieve via a new GET /_admin/deployment/id endpoint in the HTTP API.

Get public options configuration

Introduced in: v3.12.8

A new /_admin/options-public endpoint has been added for retrieving a small, curated subset of the configured server startup options that are safe to expose to any authenticated user.

Crash dump management

Introduced in: v3.12.8

New endpoints for viewing and managing crash dumps have been added to the HTTP API:

GET /_admin/crashes: List all crash dump directory identifiers (UUIDs).
GET /_admin/crashes/{id}: Get the contents of a specific crash dump as stored in <database-directory>/crashes/<uuid>/.
DELETE /_admin/crashes/{id}: Delete a specific crash dump.

See Crash dump management for details.

Activities API (experimental)

Introduced in: v3.12.8

A new activities API has been added as an observability feature. See the HTTP interface for server activities for details.

Activities of all servers (experimental)

Introduced in: v3.12.10

A new GET /_arango/experimental/_admin/activities/all endpoint has been added. You can call it on a Coordinator to get the activities of every server of a cluster deployment, grouped by server ID. The activities are returned in an activities_per_server object.

See Get the activities of all servers for details.

Endpoints augmented

View API

`optimizeTopK` property for `arangosearch` Views

Views of type arangosearch accept a new optimizeTopK View property for the ArangoSearch WAND optimization. It is an immutable array of strings, optional, and defaults to [].

See the optimizeTopK View property for details.

Changed consolidation defaults for `arangosearch` Views

Introduced in: v3.12.6

The default values for consolidating arangosearch Views have been changed. By consolidating less often and with more data, less file descriptors are used.

consolidationIntervalMsec increased from 1000 to 5000
consolidationPolicy (with type set to tier):
- segmentsMin increased from 1 to 50
- segmentsMax increased from 10 to 200
- segmentsBytesMax increased from 5368709120 (5 GiB) to 8589934592 (8 GiB)
- segmentsBytesFloor increased from 2097152 (2 MiB) to 25165824 (24 MiB)

Added and removed consolidation options for `arangosearch` Views

Introduced in: v3.12.7

The following options for consolidating arangosearch Views have been removed and are now ignored when specified in a request:

consolidationPolicy (with type set to tier):
- segmentsMin
- segmentsMax
- segmentsBytesFloor
- minScore

The following new options have been added:

consolidationPolicy (with type set to tier):
- maxSkewThreshold (number in range [0.0, 1.0], default: 0.4)
- minDeletionRatio (number in range [0.0, 1.0], default: 0.5)

Document API

The following endpoints accept a new versionAttribute query parameter that adds external versioning support:

PATCH /_api/document/{collection}/{key} (single document update)
PATCH /_api/document/{collection} (multiple document update)
PUT /_api/document/{collection} (single document replace)
PUT /_api/document/{collection}/{key} (multiple document replace)
POST /_api/document/{collection} (single document insert, when used to update/replace a document)
POST /_api/document/{collection}/{key} (multiple document insert, when used to update/replace a document)

If set, the attribute with the name specified by the option is looked up in the stored document and the attribute value is compared numerically to the value of the versioning attribute in the supplied document that is supposed to update/replace it. The document is only changed if the new number is higher. See the Document API for details.

Cursor API

`documentLookups` and `seeks` statistics

Two new statistics are included in the response when you execute an AQL query:

documentLookups: The number of real document lookups caused by late materialization as well as IndexNodes that had to load document attributes not covered by the index. This is how many documents had to be fetched from storage after an index scan that initially covered the attribute access for these documents.
seeks: The number of seek calls done by RocksDB iterators for merge joins (JoinNode in the execution plan).

{
  "result": [
    // ...
  ],
  // ...
  "extra": {
    "stats": {
      "documentLookups": 10,
      "seeks": 0,
      // ...
    }
  }
}

`searchParallelism` statistic

Introduced in: v3.12.9

The cursor API now returns an additional statistic under extra.stats:

searchParallelism (integer): The number of threads used by ArangoSearch for this query.

Query API

Introduced in: v3.12.2

The endpoints for the lists of currently running queries and slow queries (/_api/query/current and /_api/query/slow) now include the following attributes:

dataSources (array of strings), only present if tracking of data sources is enabled
modificationQuery (boolean)
warnings (integer)

Query plan cache attributes

Introduced in: v3.12.4

The following endpoints related to AQL queries support a new usePlanCache query option in the options object:

POST /_api/cursor
POST /_api/explain

An error is raised if usePlanCache is set to true but the query is not eligible for plan caching (a new error code ERROR_QUERY_NOT_ELIGIBLE_FOR_PLAN_CACHING with the number 1584). See The execution plan cache for AQL queries for details.

If a cached query plan is utilized, the above endpoints include a new planCacheKey attribute at the top-level of the response with the key of the cached plan (string).

See HTTP interfaces for AQL queries for details.

Index API

`optimizeTopK` for inverted indexes

Indexes of type inverted accept a new optimizeTopK property for the ArangoSearch WAND optimization. It is an array of strings, optional, and defaults to [].

See the inverted index optimizeTopK property for details.

Multi-dimensional indexes

The previously experimental zkd index type is now stable and has been renamed to mdi. Existing indexes keep the zkd type. The HTTP API still allows the old name to create new indexes that behave exactly like mdi indexes but this is discouraged. The zkd alias may get removed in a future version.

An additional mdi-prefixed index variant has been added. This is a new index type in the API with the same settings as the mdi index but with one additional prefixFields attribute. It is a required setting for the mdi-prefixed index type and accepts an array of strings similar to the fields attribute. You can use it to narrow down the search space using equality checks.

Both multi-dimensional index variants now support a sparse setting (boolean) and storedValues setting (array of strings) that were not supported by the zkd index type in previous versions.

See Working with multi-dimensional indexes for details.

Progress indication on the index generation

Introduced in: v3.10.13, v3.11.7

The GET /_api/index endpoint may now include a progress attribute for the elements in the indexes array. For every index that is currently being created, it indicates the progress of the index generation (in percent).

To return indexes that are not yet fully built but are in the building phase, add the withHidden=true query parameter to the call of the endpoint.

curl "http://localhost:8529/_api/index?collection=myCollection&withHidden=true"

Vector indexes

Introduced in: v3.12.4

A new vector index type has been added. See HTTP interface for vector indexes for details.

Introduced in: v3.12.9

Vector indexes now have two new attributes in success responses:

trainingState (string): Possible values:
- "unusable"
- "training"
- "ingesting"
- "ready"
errorMessage (string): Only present if there is a problem with the index/training.

Furthermore, a new error code ERROR_QUERY_VECTOR_INDEX_NOT_READY with the number 1555 has been added. It is thrown if a query tries to use a vector index that hasn’t been trained yet.

You can now create a vector index first and then populate the collection with vector data. However, it is still recommended to load the data first and then create the index to ensure that all documents participate in the training process as the training is only executed once. The training is triggered automatically if the vector index hasn’t been trained yet and the number of documents to index exceeds the threshold of nLists documents. If sparse is set to true, documents without the vector embedding field are not counted toward this threshold. Check the trainingState to see if the index is "ready" and errorMessage for the reason if it’s not.

Introduced in: v3.12.10

A new option to let you configure how many vectors per centroid to include in the random sample used for training the index has been added. You can set numberOfDocsPerCentroid in the params object to change the default of 100.

Up to v3.12.9, this is not configurable and a fixed value of 256 per centroid is used instead.

Introduced in: v3.12.10

The nLists attribute of the params object is optional now and it accepts an object with a scaling specification in addition to a number. The scaling specification lets ArangoDB compute the number of Voronoi cells respectively centroids from the document count at training time, per shard in cluster deployments. It has the following attributes:

strategy (string): The only available value is "autoSqrt", which computes max(minNLists, multiplier * sqrt(N)) where N is the number of documents.
multiplier (number): The factor for the autoSqrt strategy.
minNLists (number): The lower bound for the computed number of centroids.
tiers (array of objects, optional): Fixed numbers of centroids for large document counts, each with a threshold and a fixedValue attribute.

If you specify nLists as an object, you need to set strategy, multiplier, and minNLists. If you don’t specify nLists at all, a scaling specification with the autoSqrt strategy, a multiplier of 4, a minNLists of 2, and three tiers (1000000 → 16384, 10000000 → 65536, 300000000 → 131072) is used.

Success responses report nLists the same way it was specified, that is, either as a number or as an object.

Furthermore, the factory attribute of the params object accepts a {} placeholder in place of the number of centroids now, like "IVF{}_HNSW32,SQ8". It is substituted with the number of centroids that nLists resolves to. Success responses report the factory string as specified, including the placeholder.

Introduced in: v3.12.10

The GET /_api/index?collection=<collection-name>&withHidden=true endpoint now reports per-shard details for vector indexes in a shards attribute. The keys are the shard names, and every value is an object with the following attributes:

trainingState (string): The training state of this shard’s index.
error (string): The training error of this shard’s index, or an empty string.
resolvedNLists (number): The number of centroids this shard’s index has actually been trained with. If a fixed nLists value is configured, it matches this value. If a scaling specification is configured, it is the value computed from the document count at training time.

In single server deployments, the collection name is used as the shard key, mirroring the cluster format. The top-level trainingState is the least-progressed state across all shards.

Introduced in: v3.12.10

Creating a vector index with inBackground set to false blocks until the index training has finished. If the training fails permanently, for example, because there is not enough training data, the index is created nevertheless but cannot be used for queries.

Up to v3.12.9, the POST /_api/index request failed with an error in this case, like ERROR_QUERY_VECTOR_INDEX_NOT_READY (1555) or the underlying error such as ERROR_RESOURCE_LIMIT (32). From v3.12.10 onward, the request succeeds with an HTTP 201 Created response that reports the trainingState as "unusable" and the reason for the failed training in the errorMessage attribute:

{
  "id": "coll/68",
  "name": "vector_l2",
  "type": "vector",
  "isNewlyCreated": true,
  "trainingState": "unusable",
  "errorMessage": "not enough training data for vector index",
  "code": 201,
  "error": false
}

Errors that are not related to the training outcome, like a timeout while waiting for the index or a server shutdown, are still reported as errors.

See the inBackground option of vector indexes for details.

Changed consolidation defaults for inverted indexes

Introduced in: v3.12.6

The default values for consolidating inverted indexes have been changed. By consolidating less often and with more data, less file descriptors are used.

consolidationIntervalMsec increased from 1000 to 5000
consolidationPolicy (with type set to tier):
- segmentsMin increased from 1 to 50
- segmentsMax increased from 10 to 200
- segmentsBytesMax increased from 5368709120 (5 GiB) to 8589934592 (8 GiB)
- segmentsBytesFloor increased from 2097152 (2 MiB) to 25165824 (24 MiB)

Added and removed consolidation options for inverted indexes

Introduced in: v3.12.7

The following options for consolidating inverted indexes have been removed and are now ignored when specified in a request:

consolidationPolicy (with type set to tier):
- segmentsMin
- segmentsMax
- segmentsBytesFloor
- minScore

The following new options have been added:

consolidationPolicy (with type set to tier):
- maxSkewThreshold (number in range [0.0, 1.0], default: 0.4)
- minDeletionRatio (number in range [0.0, 1.0], default: 0.5)

Optimizer rule descriptions

Introduced in: v3.10.9, v3.11.2

The GET /_api/query/rules endpoint now includes a description attribute for every optimizer rule that briefly explains what it does.

Query parsing API

The POST /_api/query endpoint for parsing AQL queries now unconditionally returns the warnings attribute, even if no warnings were produced while parsing the query. In that case, warnings contains an empty array. In previous versions, no warnings attribute was returned when parsing a query produced no warnings.

Metrics API

The metrics endpoint includes the following new metrics about AQL queries, ongoing dumps, ArangoSearch parallelism and used file descriptors:

arangodb_aql_cursors_active
arangodb_dump_memory_usage
arangodb_dump_ongoing
arangodb_dump_threads_blocked_total
arangodb_search_execution_threads_demand
arangodb_search_file_descriptors

The following new metrics for improved memory observability have been added:

arangodb_agency_node_memory_usage
arangodb_aql_cursors_memory_usage
arangodb_index_estimates_memory_usage
arangodb_internal_cluster_info_memory_usage
arangodb_requests_memory_usage
arangodb_revision_tree_buffered_memory_usage
arangodb_scheduler_queue_memory_usage
arangodb_scheduler_stack_memory_usage
arangodb_search_consolidations_memory_usage
arangodb_search_mapped_memory
arangodb_search_readers_memory_usage
arangodb_search_writers_memory_usage
arangodb_transactions_internal_memory_usage
arangodb_transactions_rest_memory_usage

Introduced in: v3.11.2

The following metrics have been added about the LZ4 compression for values in the in-memory edge cache:

rocksdb_cache_edge_inserts_effective_entries_size_total
rocksdb_cache_edge_inserts_uncompressed_entries_size_total
rocksdb_cache_edge_compression_ratio

Introduced in: v3.10.11, v3.11.4

The following metrics have been added to improve the observability of in-memory cache subsystem:

rocksdb_cache_free_memory_tasks_total
rocksdb_cache_free_memory_tasks_duration_total
rocksdb_cache_migrate_tasks_total
rocksdb_cache_migrate_tasks_duration_total

Introduced in: v3.11.4

The following metrics have been added to improve the observability of in-memory edge cache:

rocksdb_cache_edge_compressed_inserts_total
rocksdb_cache_edge_empty_inserts_total
rocksdb_cache_edge_inserts_total

Introduced in: v3.11.5

The following metrics have been added to monitor and detect temporary or permanent connectivity issues as well as how many scheduler threads are in the detached state:

arangodb_network_connectivity_failures_coordinators
arangodb_network_connectivity_failures_dbservers_total
arangodb_scheduler_num_detached_threads

Introduced in: v3.10.13, v3.11.7

The following metrics have been introduced to track per-shard requests on DB-Servers:

arangodb_collection_leader_reads_total
arangodb_collection_leader_writes_total
arangodb_collection_requests_bytes_read_total
arangodb_collection_requests_bytes_written_total

Introduced in: v3.12.1

The following metrics have been added for observability:

arangodb_vocbase_shards_read_only_by_write_concern
arangodb_logger_messages_dropped_total
arangodb_scheduler_high_prio_dequeue_hist
arangodb_scheduler_medium_prio_dequeue_hist
arangodb_scheduler_low_prio_dequeue_hist
arangodb_scheduler_maintenance_prio_dequeue_hist

Introduced in: v3.12.4

The following metric about partially committed or aborted transactions on DB-Servers in a cluster has been added:

arangodb_vocbase_transactions_lost_subordinates_total

Introduced in: v3.12.5

A metric has been added for the time spent on API call recording:

arangodb_api_recording_call_time

Introduced in: v3.12.6

A metric has been added for the time spent on AQL query recording, along with multiple RocksDB metrics:

arangodb_aql_recording_call_time
rocksdb_block_cache_charge_per_entry
rocksdb_block_cache_entries
rocksdb_live_blob_file_garbage_size
rocksdb_live_blob_file_size
rocksdb_num_blob_files

Introduced in: v3.12.7

The following new metrics have been added to track the global number of databases, collections, and shards, as well as the CGroup version and effective CPU cores and physical memory:

arangodb_metadata_number_of_databases
arangodb_metadata_number_of_collections
arangodb_metadata_number_of_shards
arangodb_server_statistics_cpu_cgroup_version
arangodb_server_statistics_effective_cpu_cores
arangodb_server_statistics_effective_physical_memory

Introduced in: v3.12.8

The following new metrics have been introduced to provide visibility into shard distribution and replication health across your cluster:

arangodb_metadata_total_number_of_shards
arangodb_metadata_number_follower_shards
arangodb_metadata_number_out_of_sync_shards
arangodb_metadata_number_not_replicated_shards
arangodb_metadata_shard_followers_out_of_sync_number

Furthermore, the following metrics have been added as part of the experimental activities feature:

arangodb_activities_total
arangodb_activities_existing

Introduced in: v3.12.9

The following new metrics have been added for better visibility of the current state of the vector indexes:

arangodb_vector_index_ingestion_duration
arangodb_vector_index_training_duration
arangodb_vector_index_training_ongoing
arangodb_vector_index_unusable

Introduced in: v3.12.10

The following new metric has been added for tracking how often particular HTTP status codes are used in server responses:

arangodb_http_response_code_total

Introduced in: v3.12.11

The following new metrics have been added for monitoring how long requests to the Role-Based Access Control (RBAC) service take and what the health of the cluster servers is:

arangodb_rbac_request_duration
arangodb_server_health

Stream Transactions API

Introduced in: v3.12.1

A skipFastLockRound option has been added to the POST /_api/transaction/begin endpoint that lets you disable the fast lock round for Stream Transactions. The option defaults to false so that fast locking is tried.

See the HTTP API for details.

Log API

Introduced in: v3.12.2

The GET /_admin/log/level and PUT /_admin/log/level endpoints have been extended with a withAppenders query option to let you query and set log level settings for individual log outputs:

curl http://localhost:8529/_admin/log/level?withAppenders=true

If enabled, the response structure is as follows:

{
  "global": {
    "agency": "INFO",
    "agencycomm": "INFO",
    "agencystore": "WARNING",
    ...
  },
  "appenders": {
    "-": {
      "agency": "INFO",
      "agencycomm": "INFO",
      "agencystore": "WARNING",
      ...
    },
    "file:///path/to/file": {
      "agency": "INFO",
      "agencycomm": "INFO",
      "agencystore": "WARNING",
      ...
    },
    ...
  }
}

The keys under appenders correspond to the configured log outputs (--log.output startup option, - stands for the standard output). The global levels are automatically set to the most verbose log level for that topic across all appenders.

To change any of the log levels at runtime, you can send a request following the same structure:

curl -XPUT -d '{"global":{"queries":"DEBUG"},"appenders":{"-":{"requests":"ERROR"}}}' http://localhost:8529/_admin/log/level?withAppenders=true

Setting a global log level applies the value to all outputs for the specified topic. You can only change the log levels for individual log outputs (appenders) but not add new outputs at runtime.

Authentication with access tokens

Introduced in: v3.12.5

The newly added access tokens can be used for either creating JWT session tokens or directly authenticate with an access token instead of a password.

If you use an access token when calling the POST /_open/auth endpoint to create a session token, you only need to provide the access token as the password. You don’t need to specify the username, but if you do, it must match the user name encoded in the access token.

# Access token of user "root"
curl -d '{"password":"v1.7b2265...71227d"}' http://localhost:8529/_open/auth
curl -d '{"username":"root", "password":"v1.7b2265...71227d"}' http://localhost:8529/_open/auth

Similarly, if you use an access token for HTTP Basic authentication, you can leave out the user name. If you don’t, it needs to match the name in the token. Example:

# Access token of user "root" 
curl -u:v1.7b2265...71227d http://localhost:8529/_api/database
curl -uroot:v1.7b2265...71227d http://localhost:8529/_api/database

Note that it is recommended to use access tokens for creating JWT session tokens.

Activities API (experimental)

Threads of an activity

Introduced in: v3.12.10

The activity objects returned by the GET /_arango/experimental/_admin/activities endpoint have a new threads attribute. It lists the threads which currently execute the activity, each with the identifier of the light-weight process (LWPID) and the thread name. The list is empty for activities that no thread claims, like background maintenance activities.

ArangoSearch consolidation activities

Introduced in: v3.12.11

The GET /_arango/experimental/_admin/activities endpoint now also reports the background consolidation of arangosearch View and inverted index data. Such activities have the type ArangoSearchConsolidation and their data object lists the index segments that are being merged.

See ArangoSearch consolidation activities for details.

Endpoints deprecated

JavaScript Transactions API

JavaScript Transactions and thus the POST /_api/transaction endpoint is deprecated from v3.12.0 onward and will be removed in a future version. The endpoints for Stream Transactions (POST /_api/transaction/begin etc.) are unaffected.

Endpoints removed

JavaScript-based traversal using `/_api/traversal`

The long-deprecated JavaScript-based traversal functionality has been removed in v3.12.0, including the REST API endpoint /_api/traversal.

The functionality provided by this API was deprecated and unmaintained since v3.4.0. JavaScript-based traversals have been replaced with AQL traversals in v2.8.0. Additionally, the JavaScript-based traversal REST API could not handle larger amounts of data and was thus very limited.

Users of the /_api/traversal REST API should use AQL traversal queries instead.

Pregel API

The /_api/control_pregel/* endpoints have been removed in v3.12.0 as Pregel graph processing is no longer supported. The arangodb_pregel_* metrics and the pregel log topic have been removed as well from the respective endpoints.

Batch request API

Removed in: v3.12.3

The /_api/batch endpoints that let you send multiple operations in a single HTTP request was deprecated in v3.8.0 and has now been removed.

To send multiple documents at once to an ArangoDB instance, please use the HTTP interface for documents that can insert, update, replace, or remove arrays of documents.

Obsolete replication APIs

Removed in: v3.12.10

The following endpoints related to replication functionality that is no longer used have been removed:

GET /_api/replication/applier-config
PUT /_api/replication/applier-config
PUT /_api/replication/applier-start
PUT /_api/replication/applier-stop
GET /_api/replication/applier-state
GET /_api/replication/applier-state-all
PUT /_api/replication/make-follower
GET /_api/replication/logger-follow
GET /_api/replication/logger-first-tick
GET /_api/replication/logger-tick-ranges
GET /_api/replication/logger-last
GET /_api/replication/server-id
PUT /_api/replication/server-id
PUT /_api/replication/sync

JavaScript API

Collection creation

When creating a collection using the db._create(), db._createDocumentCollection(), or db._createEdgeCollection() method, the server log now displays a deprecation message if illegal combinations and unknown properties are detected in the properties object.

Note that all invalid elements and combinations will be rejected in future versions.

`@arangodb/graph/traversal` module

The long-deprecated JavaScript-based traversal functionality has been removed in v3.12.0, including the bundled @arangodb/graph/traversal JavaScript module.

The functionality provided by this traversal module was deprecated and unmaintained since v3.4.0. JavaScript-based traversals have been replaced with AQL traversals in v2.8.0. Additionally, the JavaScript-based traversals could not handle larger amounts of data and were thus very limited.

Users of the JavaScript-based traversal API should use AQL traversal queries instead.

`collection` object

The following methods now accept a versionAttribute option that adds external versioning support:

collection.update(object, data, options)
collection.replace(object, data, options)
collection.insert(data, options) when used to update/replace a document

`@arangodb/pregel` removed

The @arangodb/pregel module of the JavaScript API has been removed in v3.12.0 as Pregel is no longer supported.

`db._executeTransaction()` deprecated

JavaScript Transactions and thus the db._executeTransaction() method is deprecated from v3.12.0 onward and will be removed in a future version. The db._createTransaction() method for starting Stream Transactions is unaffected.

`@arangodb/request` certificate validation

Introduced in: v3.11.11, v3.12.2

The @arangodb/request module now supports two additional options for making HTTPS requests:

verifyCertificates (optional): if set to true, the server certificate of the remote server is verified using the default certificate store of the system. Default: false.
verifyDepth (optional): limit the maximum length of the certificate chain that counts as valid. Default: 10.

Stream Transactions API

Introduced in: v3.12.1

A skipFastLockRound option has been added to the db._createTransaction() method that lets you disable the fast lock round for Stream Transactions. The option defaults to false so that fast locking is tried.

See the JavaScript API for details.

Query plan cache module

Introduced in: v3.12.4

The new @arangodb/aql/plan-cache module lets you list the entries (.toArray()) and clear (.clear()) the AQL execution plan cache in the JavaScript API.

See The execution plan cache for AQL queries for details.

Stricter JavaScript security defaults

Introduced in: v3.12.9

Up to v3.12.8, the default access for server-side JavaScript code like Foxx, user-defined AQL functions (UDFs), and JavaScript Transactions was to allow everything. This included reading and writing arbitrary files, accessing environment variables, reading startup configuration values, and making outbound HTTP requests from within the server process.

From v3.12.9 onward, each of the following arangod startup options now defaults to disallow access to the respective resource unless configured otherwise, as if the given allowlist was set to '^$':

--javascript.files-allowlist
--javascript.environment-variables-allowlist
--javascript.startup-options-allowlist
--javascript.endpoints-allowlist

If you set denylist startup options, access is granted for everything except what matches the denylist of the respective resource, overwriting the default of disallowing everything:

--javascript.environment-variables-denylist
--javascript.startup-options-denylist
--javascript.endpoints-denylist

Up to v3.12.9, file access is exclusively controlled by --javascript.files-allowlist with no corresponding denylist. A --javascript.files-denylist option was added in v3.12.10.

JavaScript files denylist

Introduced in: v3.12.10

A --javascript.files-denylist startup option has been added to complement the existing --javascript.files-allowlist. It lets you forbid access to specific filesystem paths from client-side and server-side JavaScript, mirroring the allow/deny pairs already available for endpoints, environment variables, and startup options. File access is now controlled by both lists: a path must match the allowlist and must not match the denylist. The denylist is empty by default, so existing configurations are unaffected.

API Changes in ArangoDB 3.12

HTTP RESTful API

Behavior changes

VelocyStream protocol removed

HTTP headers

--database.extended-names enabled by default

Collection API

Index API

Stored values can contain the _id attribute

Optimizer rule changes

Gharial API

Validation of smartGraphAttribute in SmartGraphs

Limit to the number of databases in a deployment

Adjustable Stream Transaction size

Analyzer API

Adjustable writeConcern for collections with distributeShardsLike

Log API

Error code 12 removed

mmap log topic removed

Permission checks for Stream Transactions in cluster

Access token lifetime

Refactored authorization system

Error number for write operations in read-only mode

Access token management in read-only mode

Permission checks for the AQL query results cache API

Error response for inaccessible databases in the Activities API

Endpoint return value changes

Storage engine API

Storage engine statistics API

Endpoints added

Effective and available startup options

Available key generators

Shard usage metrics

Reset log levels

Query plan cache API

API call recording

AQL query recording

Access tokens

Deployment ID

Get public options configuration

Crash dump management

Activities API (experimental)

Activities of all servers (experimental)

Endpoints augmented

View API

optimizeTopK property for arangosearch Views

Changed consolidation defaults for arangosearch Views

Added and removed consolidation options for arangosearch Views

Document API

Cursor API

documentLookups and seeks statistics

searchParallelism statistic

Query API

Query plan cache attributes

Index API

optimizeTopK for inverted indexes

Multi-dimensional indexes

Progress indication on the index generation

Vector indexes

Changed consolidation defaults for inverted indexes

Added and removed consolidation options for inverted indexes

Optimizer rule descriptions

Query parsing API

Metrics API

Stream Transactions API

Log API

Authentication with access tokens

Activities API (experimental)

Threads of an activity

ArangoSearch consolidation activities

Endpoints deprecated

JavaScript Transactions API

Endpoints removed

JavaScript-based traversal using /_api/traversal

Pregel API

Batch request API

Obsolete replication APIs

JavaScript API

Collection creation

@arangodb/graph/traversal module

`--database.extended-names` enabled by default

Stored values can contain the `_id` attribute

Validation of `smartGraphAttribute` in SmartGraphs

Adjustable `writeConcern` for collections with `distributeShardsLike`

Error code `12` removed

`mmap` log topic removed

`optimizeTopK` property for `arangosearch` Views

Changed consolidation defaults for `arangosearch` Views

Added and removed consolidation options for `arangosearch` Views

`documentLookups` and `seeks` statistics

`searchParallelism` statistic

`optimizeTopK` for inverted indexes

JavaScript-based traversal using `/_api/traversal`

`@arangodb/graph/traversal` module

`collection` object

`@arangodb/pregel` removed

`db._executeTransaction()` deprecated

`@arangodb/request` certificate validation