Running more than one engine

An installation scales by starting the same binary again. There is no orchestrator to install, no node registry to keep, no consensus protocol, no heartbeat and no leader-election service. Engines do not know about each other; they coordinate through the state directory they share, using POSIX advisory record locks. Add an engine and the work redistributes; stop one and the rest keep serving.

This page describes that shape, what each engine keeps to itself, and what the filesystem underneath has to support for it to hold.

The shape

Every engine is started the same way, pointed at the same state directory:

$ nexus serve --port 9090 --db /srv/nexus/registry.db     # engine 1
$ nexus serve --port 9091 --db /srv/nexus/registry.db     # engine 2
$ nexus serve --port 9092 --db /srv/nexus/registry.db     # engine 3

--db is the only path flag. Its parent directory is the state directory, and everything else is placed relative to it — see Server and configuration. Put a reverse proxy in front of the HTTP ports, terminate TLS there, and health-check each engine on GET /health, which needs no credential and answers {"ok":true}.

Each engine boots identically and says on its own console which roles it took:

INFO scheduler: acquired singleton lock — this node is the scheduler

or, on the engines that did not win it:

INFO scheduler: another node holds the scheduler lock — skipping

What is elected, and what is merely serialised

Three things in the state directory must not happen twice at once. Each is answered by a lock file, and the difference between them is whether the losing engine waits or walks away.

Concern	Lock file	Behaviour
One queue worker per `(tenant, flow)`	`queues/{tenant}/{flow}/worker.lock`	Non-blocking. The engine that takes it runs the worker; the others skip and do not run one.
One cron scheduler per installation	`queues/scheduler.lock`	Non-blocking, at boot. The engine that takes it runs the scheduled flows and the correlation deadline sweep; the others skip both.
Appends to one queue’s segment	`queues/{tenant}/{flow}/segment.log.lock`	Blocking. Every engine appends; the lock is held across seek, write, flush and fsync, so two engines submitting to the same queue interleave messages rather than corrupting the file.

The scheduler and the correlation sweep share one lock on purpose. Both answer the same question — “exactly one sweeper per installation” — and a second lock would be a second answer to it.

Everything else that crosses engines is settled by SQLite rather than by a lock file. The deduplication row and the correlation lease in flow_inbox are taken with a single atomic statement, so two engines racing on the same key produce one winner and one refusal, not two executions. The audit chain resolves the same race by retrying: two engines can read the same tail and try to write the same sequence number, one loses on the UNIQUE constraint, re-reads the tail with the new previous hash and appends behind the winner. The chain stays verifiable.

What the filesystem has to support

POSIX fcntl record locks. Not flock() — its semantics vary by kernel and by network filesystem implementation, which is why the platform never uses it. If fcntl locking does not work on the mount, nothing above holds.

The journal modes of the four databases. They are not the same, and the difference decides where the state directory can live:

File	Journal mode	Opened for sharing
`registry.db` (the `--db` file)	`DELETE`	Yes — `synchronous=FULL`, `busy_timeout=5000`
`nexus-audit.db`	`DELETE`	Yes — same, plus the chain retry above
`nexus-log.db`	`WAL`	`busy_timeout=5000` only
`nexus-ui.db`	`WAL`	—

The two that carry the shared state of a delivery — the registry and the audit chain — are opened in DELETE journal mode deliberately: WAL needs SQLite’s shared-memory index, the -shm file, which a network filesystem typically cannot share between processes on different hosts. The log database and the management-UI database are in WAL and were not changed, so they carry that requirement with them. There is one state directory and one flag, so the four cannot be placed on different filesystems.

The shape that works today is therefore engines sharing a filesystem that supports both fcntl record locking and SQLite’s shared-memory index. Several engines on one host over a local directory satisfy both. A mount that gives you fcntl but not shared memory carries the registry, the audit chain and the queues, and does not carry the other two.

The daily log file. logs/nexus-log-YYYY-MM-DD.jsonl is appended by every engine without a lock. On a local filesystem the appends do not tear; over a network mount they can interleave. The rows in nexus-log.db are the ordered record — see Logging and audit.

What each engine keeps to itself

Some state is per process by construction, and an operator sizing an installation has to count it per engine rather than per installation:

Rate limit buckets. The token bucket lives in memory, keyed by API key. With N engines, a key configured at r requests per second can spend up to N × r if the proxy spreads its requests. Size the ceiling for the engine, not for the installation.
The execution budgets of ADR-037 — the residency slots that decide how many queue workers one engine will start, and the in-flight ceiling. Each engine has its own.
The gRPC routing table. Built at boot from the deployed artifacts, on each engine independently. A flow published afterwards is routed by an engine only after that engine restarts.
The OAuth2 token cache and the per-endpoint circuit breaker. Each engine holds its own, so a breaker opened by one engine does not open on the others.
The in-memory retry heap of a queue worker, which belongs to the engine holding that queue’s worker lock and is persisted next to the segment.

The maintenance loop — log retention, queue compaction, the certificate sweep, the flow-state purge — runs on every engine rather than on the elected one. Its work is deletion of things past a window, so repeating it is harmless; a race is a file already gone, reported as a warning. Queue compaction is the one that could destroy data, and it asks for the worker lock before deleting anything, so it cannot delete the files under another engine’s worker.

Losing an engine

Requests in flight on that engine are lost to their callers the way any request is when the process behind a proxy goes away. A synchronous /run has no record; a message already written by /enqueue is durable and is drained by whichever engine holds that queue’s worker.

Queues keep draining, with one thing to know. An engine that found worker.lock held records that queue as served and does not ask for the lock again while it runs. So when the engine holding a queue’s worker stops, that queue is picked up by an engine that has not yet seen that (tenant, flow) pair — the next one to receive a submission for it — and otherwise at the next restart.

Scheduled flows stop, and so does the correlation deadline sweep. The election runs once, at boot. When the engine holding scheduler.lock exits, the lock is released, but the engines already running passed that point in their own boot and do not return to it. Restarting any engine re-runs the election and the schedules resume. Plan the restart; there is no automatic takeover.

A lock that cannot be read is treated as absent, on purpose. If the worker lock file cannot be opened — an I/O error on the mount, not a lock held elsewhere — the engine starts the worker anyway and logs a warning naming the tenant and the flow. Two workers on one queue deliver a message more than once, which is inside the at-least-once contract the queue already states; a queue nobody drains is outside it. See Queues and delivery.

Putting a new version live

The procedure is the one in Server and configuration — deploy the new flow version, then let it resolve — and it does not change with more engines, because an artifact is immutable and resolved per request. What does change is the engine restart: take one engine out of the proxy at a time, restart it, put it back. If the engine you restart was the scheduler, the election moves to whichever engine you restart next, so restart the scheduler engine last if you want the schedules to run throughout.

Planning an installation

Three properties of this shape decide how an installation is sized and operated. Each is described above; collected here because they are what an operator plans around.

Rate limits are counted per engine. A key configured at r requests per second can spend up to N × r across N engines, depending on how the proxy spreads its requests. Size the ceiling for the engine.
The cron scheduler is elected once, at boot. Restarting an engine re-runs the election, so restart the engine holding it last if the schedules are to keep running throughout.
The log and management-UI databases are in WAL, so they need a filesystem that supports SQLite’s shared-memory index. That is what decides whether all four databases can share one mount, or only the registry, the audit chain and the queues can.