Backup and restore

Why the backup is a SQLite snapshot and not a file copy, the script that takes it, how to restore from it, and how to prove the restored installation is the one you backed up.

Everything an installation owns lives in one directory — the one that holds registry.db. So a backup is that directory, and a restore is putting it back. The two paragraphs that follow are the reason it is not cp -r.

Why not `cp`

The four databases are not opened the same way:

Database	`journal_mode`
`registry.db`, `nexus-audit.db`	`DELETE`, with `synchronous=FULL` — chosen so the directory can sit on a shared filesystem
`nexus-log.db`, `nexus-ui.db`	`WAL`

A cp over a WAL database with the server running takes the main file without its -wal companion: an older database that looks perfectly intact. Nothing fails at restore time. The last writes are simply not there, and nothing says so.

.backup is consistent whatever the journal mode, which is why the procedure below is written with it, and why sqlite3 is worth having on the machine.

The backup

#!/bin/sh
set -e
NEXUS_DIR=/var/lib/nexus
DEST=/backup/nexus/$(date +%Y%m%dT%H%M%S)
mkdir -p "$DEST"

# The databases: `.backup` takes a consistent snapshot, live server included.
for db in registry nexus-audit nexus-log nexus-ui; do
  sqlite3 "$NEXUS_DIR/$db.db" ".backup '$DEST/$db.db'"
done

# The queues: files, not a database. See the note below.
tar -C "$NEXUS_DIR" -cf "$DEST/queues.tar" queues

sqlite3 "$DEST/registry.db" "PRAGMA integrity_check;"
sqlite3 "$DEST/nexus-audit.db" "PRAGMA integrity_check;"

The note that matters: .backup is consistent per database, not across them, and queues/ is not a database at all. A backup taken with the server running can catch a message already pulled from the queue and not yet recorded in the log, or the reverse.

For a mutually consistent set — the only kind you restore a whole installation from, as opposed to reading one database out of — stop the server:

systemctl stop nexus && /backup/nexus-backup.sh && systemctl start nexus

The live backup keeps its job as the daily net. The one you restore from is taken with the server stopped.

In a container

Same procedure, with the destination mounted and with -u root:

docker stop nexus
docker run --rm -u root -v nexus-data:/data -v /backup/nexus:/backup \
  --entrypoint sh <the image you were given> -c '
    set -e
    DEST=/backup/$(date +%Y%m%dT%H%M%S); mkdir -p "$DEST"
    for db in registry nexus-audit nexus-log nexus-ui; do
      sqlite3 "/data/$db.db" ".backup \"$DEST/$db.db\""
    done
    tar -C /data -cf "$DEST/queues.tar" queues
    sqlite3 "$DEST/registry.db" "PRAGMA integrity_check;"
    sqlite3 "$DEST/nexus-audit.db" "PRAGMA integrity_check;"'
docker start nexus

-u root is not convenience. The process in the image runs as uid 10001, and a freshly created volume or host directory belongs to root — without the flag, .backup stops with cannot open before writing anything. If the destination already belongs to 10001, drop the flag; what does not work is the default form over a new destination.

The restore

systemctl stop nexus
mv /var/lib/nexus /var/lib/nexus.broken          # nothing is deleted until the new one works
install -d -o nexus -g nexus -m 0750 /var/lib/nexus
cp /backup/nexus/<timestamp>/*.db /var/lib/nexus/
tar -C /var/lib/nexus -xf /backup/nexus/<timestamp>/queues.tar
chown -R nexus:nexus /var/lib/nexus
systemctl start nexus

The -wal and -shm files are never copied from a backup. .backup produces complete databases, and a foreign -wal sitting next to a restored database is the most effective way to corrupt exactly what was just recovered.

Then verify, in this order:

curl -s localhost:9090/health                       # {"ok":true}
nexus audit --tenant acme --verify --db /var/lib/nexus/nexus-audit.db
#   -> chain integrity: OK (N entries checked)
nexus tenant list --db /var/lib/nexus/registry.db
nexus keys list --tenant acme --db /var/lib/nexus/registry.db
nexus state list --tenant acme --db /var/lib/nexus/registry.db

nexus audit --verify on an empty chain answers (no audit entries for tenant …) and exits successfully. That is not a check that passed; it is a chain that is not there. The entry count in the success message is the number to compare with what was there before.

What a restore does not bring back

A restore is a restart with older files, so everything a restart loses is lost here too. None of it is in any backup, because none of it is on disk.

Rate-limit buckets are in memory and start full. In the first second after a restart, a key limited to 10 requests per second can pass its burst on top of its allowance.
Cron triggers that fell inside the stop window are not caught up. The scheduler computes the next tick from the moment it starts.
The gRPC routing table is rebuilt at boot from the latest artifacts, so a flow deployed after the restore is routed only after the next restart.
Queue workers start lazily, on the first POST /enqueue for each (tenant, flow) pair.

And what survives, which is most of it: the retry heap is rebuilt from segment.log.retry with absolute deadlines; queue position comes from segment.log.ckpt, so an unacked message is redelivered — at-least-once, so a flow without a dedup_key: can run that delivery twice; UI sessions are in nexus-ui.db and stay valid; deduplication memory and open correlations are in registry.db and come back with it.

Those last two come back only if the state key does. Deduplication fingerprints and correlation keys are stored keyed by NEXUS_SECRET_NEXUS_STATE_KEY, which deliberately lives outside the databases — whoever holds the file with the fingerprints would otherwise hold the key that made them. Restore the directory under a different key and the rows are still there but nothing matches them: every delivery reads as fresh for one retention window. So the key travels with the backup, in whatever holds the installation’s other secrets, and it is the first thing to check before the restore rather than after. See the state key at install time.

Upgrades

An upgrade of the platform migrates the databases in place, on the first start of the new binary, and there is no down migration. So the backup taken with the server stopped — the mutually consistent one — is what makes an upgrade reversible: you go back by putting the old binary and those files back. The procedure is under What an upgrade of the platform does.