Vector index HTTP API

The name of the database.

Example: _system

The collection name.

A list with exactly one attribute path to specify where the vector embedding is stored in each document.

If you want to index another vector embedding attribute, you need to create a separate vector index.

Up to ArangoDB v3.12.8, the vector data needs to be populated before creating the index. From v3.12.9 onward, you can create the vector index first and then populate the collection with vector data. However, it is still recommended to load the data first and then create the index to ensure that all documents participate in the training process as the training is only executed once. The training is triggered automatically if the vector index hasn’t been trained yet and the number of documents to index exceeds a threshold. The threshold is the nLists value if you set a fixed number of centroids, and the minNLists value if you use the scaling mode of nLists (from v3.12.10 onward). If sparse is set to true, documents without the vector embedding field are not counted toward this threshold. Check the trainingState to see if the index is "ready" and errorMessage for the reason if it’s not.

Set this option to true to keep the collection/shards available for write operations by not using an exclusive write lock for the duration of the index creation.

If the option is disabled, the call returns only after the index training has finished (but timeouts may occur).

• Up to v3.12.9, the call returns an error if the training fails, for example, because there is not enough training data. The index is created nevertheless but stays "unusable".
• From v3.12.10 onward, the call returns a success response with the trainingState set to "unusable" and the reason for the failed training in the errorMessage attribute.

A user-defined name for the index for easier identification. If not specified, a name is automatically generated.

The number of threads to use for indexing.

The parameters as used by the Faiss library.

• defaultNProbe integer (default: 1)

  How many neighboring centroids to consider for the search results by default. The larger the number, the slower the search but the better the search results. The default is 1. You should generally use a higher value here or per query via the nProbe option of the vector similarity functions.

• dimension integer, required

  The vector dimension. The attribute to index needs to have this many elements in the array that stores the vector embedding.

• factory string

  You can specify an index factory string that is forwarded to the underlying Faiss library, allowing you to combine different advanced options. Examples:

  • "IVF100_HNSW10,Flat"
  • "IVF100,SQ4"
  • "IVF10_HNSW5,Flat"
  • "IVF100_HNSW5,PQ256x16"

  The base index must be an inverted file (IVF) to work with ArangoDB. If you don’t specify an index factory, the value is equivalent to IVF<nLists>,Flat. For more information on how to create these custom indexes, see the Faiss Wiki .

  The number of centroids that the factory string specifies needs to match the nLists value, otherwise the training fails and the index stays "unusable". From v3.12.10 onward, you can use a {} placeholder in place of the number to avoid this problem, like "IVF{},SQ4". It is substituted with the number of centroids that nLists resolves to, per shard in cluster deployments. A factory string with a fixed number of centroids can be combined with the scaling mode of nLists, but only if the resolved value happens to match the number in the factory string.

  Example: "IVF{}_HNSW32,SQ8"

• metric string, required

  Possible values: "cosine", "innerProduct", "l2"

  The measure for calculating the vector similarity:

  • "cosine": Angular similarity. Vectors are automatically normalized before insertion and search.
  • "innerProduct" (introduced in v3.12.6): Similarity in terms of angle and magnitude. Vectors are not normalized, making it faster than cosine.
  • "l2": Euclidean distance.

• nLists

  The number of Voronoi cells to partition the vector space into, respectively the number of centroids in the index. What value to choose depends on the data distribution and chosen metric. According to The Faiss library paper , it should scale sublinearly with the document count. A bigger value produces more correct results but increases the training time and thus how long it takes to build the index. It cannot be bigger than the number of documents.

  Up to v3.12.9, you need to set this attribute to a number and it is required. From v3.12.10 onward, it is optional and you can either set a fixed number of centroids or let ArangoDB compute the number from the document count:

  • Fixed mode (number): Use exactly this number of centroids, for example 100. You can use c * sqrt(N) where c is a constant factor and N is the number of documents in the collection, respectively the number of documents in the shard for cluster deployments.

    Suggested values for c:

    • Between 15 and 20 for larger datasets (according to the Faiss paper)
    • Between 4 and 8 (used in practice by autofaiss)

  • Scaling mode (object, introduced in v3.12.10): Compute the number of centroids from the number of documents at training time. In cluster deployments, the computation is done per shard using the document count of the respective shard. This is especially useful if the data distribution across shards is unequal. The attributes of the object are the following:

    • strategy (string): How to compute the number of centroids if no tier applies. The only available value is "autoSqrt", which computes max(minNLists, multiplier * sqrt(N)) where N is the number of documents of the shard.
    • multiplier (number): The factor to use in the autoSqrt strategy. It must be 1 or greater.
    • minNLists (number): The lower bound for the number of centroids computed by the autoSqrt strategy. It must be 1 or greater. It is also the number of documents required to trigger the training.
    • tiers (array of objects, optional): Fixed numbers of centroids for large document counts. The tier with the highest threshold that is less than or equal to the number of documents wins and its fixedValue is used instead of computing a value with the strategy. Each tier has a threshold and a fixedValue attribute, both of which must be 1 or greater.

    If you specify nLists as an object, you need to set strategy, multiplier, and minNLists. Only tiers is optional.

  The recommendation for ArangoDB is to use the scaling mode. The default settings scale nLists sublinearly with the document count, and the tiers bound it for high document counts so that it doesn’t become impractically high.

  If you don’t specify nLists at all, the following scaling specification is used:

  {
    "nLists": {
      "strategy": "autoSqrt",
      "multiplier": 4,
      "minNLists": 2,
      "tiers": [
        { "threshold": 1000000,   "fixedValue": 16384 },
        { "threshold": 10000000,  "fixedValue": 65536 },
        { "threshold": 300000000, "fixedValue": 131072 }
      ]
    }
  }
  

  It resolves to the following numbers of centroids for N documents:

  • N < 1,000,000: max(2, 4 * sqrt(N))
  • 1,000,000 ≤ N < 10,000,000: 16384
  • 10,000,000 ≤ N < 300,000,000: 65536
  • N ≥ 300,000,000: 131072

  Note that the scaling mode cannot resolve a number of centroids for an empty collection respectively shard. The index stays "unusable" in this case.

• numberOfDocsPerCentroid integer (default: 100)

  Introduced in: v3.12.10

  How many vectors per centroid to include in the random sample used for training. It must be 1 or greater.

  Up to v3.12.9, this is not configurable and a fixed value of 256 per centroid is used instead.

  The training does not use the full dataset but a sample bounded to nLists × numberOfDocsPerCentroid vectors. A larger value can improve the training quality but increases the memory and time required for training. See Resource usage during index creation for details.

• trainingIterations integer (default: 25)

  The number of iterations in the training process. The default is 25. Smaller values lead to a faster index creation but may yield worse search results.

Whether to create a sparse index that excludes documents with the attribute for indexing missing or set to null. This attribute is defined by fields.

Introduced in: v3.12.7

Store additional attributes in the index.

The maximum number of attributes that you can use in storedValues is 32.

• Up to v3.12.9, these are not for covering projections with the index but for adding attributes that you filter on. This lets you make the lookup in the vector index more efficient because it avoids materializing documents twice, once for the filtering and once for the matches.
• From v3.12.10 onward, these are also used to cover projections. This lets you return the attributes directly from the index without materialization.

A list of attribute paths. The . character denotes sub-attributes.

The index type. Needs to be "vector".

Example: "vector"