arangorestore Examples

To restore data from a dump previously created with arangodump, ArangoDB provides the arangorestore tool.

Restore a dump

To restore a dump including its system collections into the _system database, run arangorestore with the directory that contains the dump and enable the --include-system-collections startup option:

arangorestore --include-system-collections true --input-directory "dump"

System collections hold data that other features depend on, for example, the definitions of named graphs in the _graphs collection. The dump needs to include them, which requires the same option when running arangodump.

Without further startup options, arangorestore connects to the ArangoDB server at the default endpoint http://127.0.0.1:8529 as the root user and prompts you for the password. It then restores the collection structure and the documents from the files found in the input directory dump. The input directory must have been created by running arangodump before.

You can also specify the input directory as a positional argument without --input-directory:

arangorestore --include-system-collections true dump

arangorestore prints out its progress while running, and ends with a line showing some aggregate statistics:

Processed 2 collection(s), read 2256 byte(s) from datafiles, sent 2 batch(es)

Connect and authenticate

To connect to a different server or as a different user, use the following startup options:

• --server.endpoint <string>: The endpoint to connect to. In a cluster, specify a Coordinator.
• --server.username <string>: The username.
• --server.password <string>: The password or access token. If you omit it, you are prompted for the password.
• --server.jwt-token <string>: A JWT to authenticate with instead of a username and password. Set it to - to be prompted for the token.
• --server.jwt-secret-keyfile <string>: A file with the JWT secret of the server, to authenticate with superuser rights instead of a username and password.
• --server.authentication <bool>: Whether to authenticate. Set it to false if authentication is disabled on the server.

For example, to restore a dump into a database of a server at a non-standard endpoint as the backup user:

arangorestore \
  --server.endpoint http://192.168.173.13:8531 \
  --server.username backup \
  --server.database mydb \
  --input-directory "dump"

You can specify more than one endpoint, for example, multiple Coordinators of a cluster. arangorestore then distributes the connections of its threads across the endpoints:

arangorestore \
  --server.endpoint http://192.168.173.13:8531 \
  --server.endpoint http://192.168.173.13:8532 \
  --server.username backup \
  --server.database mydb \
  --input-directory "dump"

From v3.12.13 onward, arangorestore renews JWTs automatically before they expire, including the token it obtains for a username and password, so that long-running restores don’t fail with an HTTP 401 Unauthorized error. For details about the authentication methods, how to avoid exposing credentials in the process list, and how the token renewal works, see Connect to an ArangoDB server.

Required permissions

The user you restore data as needs the following access levels:

• When restoring into an existing database, the user needs Administrate / rw access on this database.
• When creating a new database during the restore, the user needs Administrate / rw access on the _system database. The user is granted Administrate / rw access on the newly created database.

Choose the target database

arangorestore restores into the _system database by default. To restore into a different database, specify its name with the --server.database startup option.

If the target database doesn’t exist yet, you can let arangorestore create it by setting the --create-database startup option to true:

arangorestore --server.username backup --server.database newdb --create-database true --input-directory "dump"

The user that arangorestore authenticates as is granted access to the new database. If you authenticate with a JWT superuser token or the JWT secret, this is the user specified with --server.username (root by default). Up to v3.12.12, this is also the case for JWT session and user tokens instead of the user the token has been issued for.

To restore a dump that has been created with the --all-databases option of arangodump, set the --all-databases startup option to true and don’t specify --server.database. You can combine it with --create-database true to create the databases that don’t exist yet:

arangorestore --server.username backup --all-databases true --create-database true --input-directory "dump-multiple"

The --force-same-database startup option restricts arangorestore operations to a database with the same name as in the source dump’s dump.json file. You can use it to prevent restoring data into a wrong database by accident. For example, if a dump was taken from database A, and the restore is attempted into database B, then with the --force-same-database option set to true, arangorestore aborts instantly. The option is false by default.

Choose the collections and data to restore

By default, arangorestore re-creates all non-system collections found in the input directory and loads data into them. If the target database already contains collections which are also present in the input directory, the existing collections in the database are dropped and re-created with the properties and data found in the input directory.

The following startup options are available to adjust this behavior:

• --create-collection <bool>: Set to true to create collections in the target database if they don’t yet exist. If the target database already contains a collection with the same name, then it is dropped and recreated with the same properties as in the dump if the --overwrite option is enabled (default). If the --overwrite option is set to false, an existing collection is used as is, and its properties are not updated nor is its data discarded before restoring. If --create-collection is set to false, then arangorestore does not make any attempts to create the collection or modify its properties. Data is restored into the existing collections without wiping the collection beforehand. If set to false and arangorestore encounters a collection that is present in the input directory but not in the target database, it aborts with a “collection not found” error. The default value for --create-collection is true.
• --overwrite <bool>: Controls whether existing collections are dropped if --create-collection is enabled. The default value is true.
• --import-data <bool>: Set to true to load document data into the collections in the target database. Set to false to not load any document data. The default value is true.
• --include-system-collections <bool>: Whether to include system collections when re-creating collections or reloading data. The default value is false. Set it to true to restore data like the definitions of named graphs.

For example, to (re-)create all non-system collections and load document data into them, use:

arangorestore --create-collection true --import-data true --input-directory "dump"

This drops potentially existing collections in the target database that are also present in the input directory.

To include system collections too, use --include-system-collections true:

arangorestore --create-collection true --import-data true --include-system-collections true --input-directory "dump"

To (re-)create all non-system collections without loading document data, use:

arangorestore --create-collection true --import-data false --input-directory "dump"

This also drops existing collections in the target database that are also present in the input directory.

To just load document data into existing non-system collections, use:

arangorestore --create-collection false --import-data true --input-directory "dump"

To restrict reloading to just specific collections, use the --collection option. It can be specified multiple times if required:

arangorestore --collection myusers --collection myvalues --input-directory "dump"

Collections are processed in alphabetical order by arangorestore, with all document collections being processed before all edge collections. This remains valid also when multiple threads are in use.

Note however that when restoring an edge collection no internal checks are made in order to validate that the documents that the edges connect exist. As a consequence, when restoring individual collections which are part of a graph you are not required to restore in a specific order.

To restrict reloading to specific Views, there is the --view option. Should you specify the --collection parameter, Views are not restored unless you explicitly specify them via the --view option.

arangorestore --collection myusers --view myview --input-directory "dump"

In the case of an arangosearch View, you must make sure that the linked collections are either also restored or already present on the server.

Continue an interrupted restore

If a restore is interrupted, for example, because the connection to the server is lost or the arangorestore process is terminated, you can continue it instead of starting over by running arangorestore again with the same startup options and --continue true:

arangorestore --server.database mydb --input-directory "dump" --continue true

While restoring, arangorestore records its progress in a continue.json file in the input directory (in the directory of each database if you use --all-databases). Therefore, the input directory needs to be writable. For each collection, the file tracks whether the collection has been created and up to which position in the data files the server has confirmed the restored documents. The file also tracks whether the Views have been created.

With --continue true, arangorestore reads this file and does the following:

• It skips collections and Views that have been fully restored.
• It doesn’t drop and re-create collections that it has already created, even if --overwrite is enabled.
• It continues restoring the data of partially restored collections from the last confirmed position. Batches of documents that were sent before the interruption but not confirmed are sent again. This doesn’t create duplicates because restoring a document replaces an existing document with the same document key.

When the restore succeeds, arangorestore removes the continue.json file. If you run arangorestore without --continue true, it ignores an existing progress file and starts from scratch.

Encryption

See arangodump for details.

Compression

When restoring dumps that are compressed on disk, arangorestore automatically handles the decompression.

You can optionally let arangorestore compress the data for the network transfer with the --compress-transfer startup option. This can reduce the traffic and thus save time. Set the --compress-request-threshold startup option to define the minimum size for request bodies (in bytes) at which compression is applied.

arangorestore --input-directory "dump" --compress-transfer --compress-request-threshold 250

Reloading Data into a different Collection

arangorestore restores documents and edges with the exact same _key, _from, and _to values as found in the input directory.

With some creativity you can also use arangodump and arangorestore to transfer data from one collection into another (either on the same server or not). For example, to copy data from a collection myvalues in database mydb into a collection mycopyvalues in database mycopy, you can start with the following command:

arangodump --collection myvalues --server.database mydb --output-directory "dump"

This creates two files, myvalues.structure.json and myvalues.data.json, in the output directory. To load data from the datafile into an existing collection mycopyvalues in database mycopy, rename the files to mycopyvalues.structure.json and mycopyvalues.data.json.

After that, run the following command:

arangorestore --collection mycopyvalues --server.database mycopy --input-directory "dump"

Enabling revision trees for older dumps

Collections in ArangoDB 3.8 and later can use an internal format that is based on revision trees for replication. Using this format has advantages over the previous format, because changes to the collection on the leader can quickly be detected when trying to get follower shards in sync.

Dumps taken from older versions of ArangoDB, i.e. ArangoDB 3.7 or before, do not contain any information about revision trees. The arangorestore behavior for these collections is as follows:

• In ArangoDB versions before 3.8.7 and 3.9.2, the collections are restored without revision trees.
• In ArangoDB versions 3.8.7, 3.9.2 or later, the collections use revision trees by default, but you can opt out of this by invoking arangorestore with the --enable-revision-trees false option.

If the --enable-revision-trees startup option is true (which is the default value), then arangorestore adds the necessary attributes for using revision trees when restoring the collections. It’s only done for the attributes which are not contained in the dump. If the option is set to false, arangorestore does not add the attributes when restoring collections.

Regardless of the setting of this option, arangorestore does not add the attributes, when they are already present in the dump. You may modify the attributes manually in the dump if you want to change their values.

Restoring in a Cluster

To restore data into a Cluster, simply point arangorestore to one of the Coordinators in your Cluster.

If arangorestore is asked to restore a collection, it uses the same number of shards, replication factor, and shard keys as when the collection was dumped. The distribution of the shards to the servers are also the same as at the time of the dump, provided that the number of DB-Servers in the cluster dumped from is identical to the number of DB-Servers in the to-be-restored-to cluster.

To modify the number of shards or the replication factor for all or just some collections, arangorestore provides the options --number-of-shards and --replication-factor. These options can be specified multiple times as well, in order to override the settings for dedicated collections, e.g.

arangorestore --number-of-shards 2 --number-of-shards mycollection=3 --number-of-shards test=4

The above command restores all collections except “mycollection” and “test” with 2 shards. “mycollection” has 3 shards when restored, and “test” has 4. It is possible to omit the default value and only use collection-specific overrides. In this case, the number of shards for any collections not overridden is determined by looking into the numberOfShards values contained in the dump.

The --replication-factor options works in the same way, e.g.

arangorestore --replication-factor 2 --replication-factor mycollection=1

sets the replication factor to 2 for all collections but “mycollection”, which gets a replication factor of just 1.

If a collection was dumped from a single instance and is then restored into a cluster, the sharding is done by the _key attribute by default. You can manually edit the structural description for the shard keys in the dump files if required (*.structure.json).

If you restore a collection that was dumped from a cluster into a single ArangoDB instance, the number of shards, replication factor and shard keys are silently ignored.

Factors affecting speed of arangorestore in a Cluster

The following factors affect speed of arangorestore in a Cluster:

• Replication Factor: The higher the replication factor, the more time the restore takes. To speed up the restore you can restore using a replication factor of 1 and then increase it again after the restore. This reduces the number of network hops needed during the restore.
• Restore Parallelization: If the collections are not restored in parallel, the restore speed is highly affected. A parallel restore can be done by using the --threads option of arangorestore. Depending on your specific case, you might be able to achieve additional parallelization by restoring on multiple Coordinators at the same time.

Restoring collections with sharding prototypes

arangorestore yields an error when you try to restore a collection whose shard distribution follows a collection (distributeShardsLike property) which does not exist in the cluster and which was not dumped together with the other collection:

arangorestore --collection clonedCollection --server.database mydb --input-directory "dump"

WARNING [c6658] {restore} Error while creating document collection 'clonedCollection': got invalid response from server: HTTP 500: 'Collection not found: prototypeCollection in database _system' while executing restoring collection with this requestPayload: ...

ERROR [cb69f] {restore} got invalid response from server: HTTP 500: 'Collection not found: prototypeCollection in database _system' while executing restoring collection with this requestPayload: ...

INFO [a66e1] {restore} Processed 0 collection(s) from 1 database(s) in 0.04 s total time. Read 0 bytes from datafiles, sent 0 data batch(es) of 0 bytes total size.

You need to dump and restore collections that follow the sharding of a prototype collection together with the prototype collection:

arangorestore --collection clonedCollection --collection prototypeCollection --server.database mydb --input-directory "dump"

...
INFO [a66e1] {restore} Processed 2 collection(s) from 1 database(s) in 1.12 s total time. Read 0 bytes from datafiles, sent 0 data batch(es) of 0 bytes total size.