Queues, retries and the DLQ

The asynchronous endpoint: what happens between the submission and the 202, how retries are configured, what ends up in the dead-letter queue, and how to get it out again.

The asynchronous path

$ curl -i -X POST http://localhost:9090/flows/acme/forward-order/enqueue \
        -H "Authorization: Bearer $NEXUS_KEY" \
        -H 'Content-Type: application/json' \
        -d '{"orderId":"A-1","total":42}'

HTTP/1.1 202 Accepted
Content-Type: application/json

{"queued":true,"offset":0,"tenant":"acme","flow":"forward-order"}

In order, before you get that response:

The access gate runs, before a byte of the body is read. The key is authenticated, the tenant checked, the flow allowlist and the rate limit applied. A caller refused here gets 401/403/404/429 without the body ever being pulled off the socket — so an unauthenticated caller announcing an oversized body is told 401, not 413. See Authentication.
The message-size ceiling for this access point is resolved and the body is read under it.
The signature is checked, if the server was started with --secret — after the body, because a signature is over the body. See Authentication.
The body is parsed. JSON by default; application/xml is parsed and then flattened to JSON — child elements become object keys and the root tag is dropped, so <order><id>A-1</id></order> is stored as {"id":"A-1"} and read in the flow as $.id.
A correlation ID is taken from X-Correlation-ID, then X-Request-ID, or generated.
The message is appended to the queue’s write-ahead log and fsynced. Only then does the 202 go out.

The flow has not run at the point you get the response. offset is the message’s byte offset in the log — it is what identifies the message internally, not a handle you can query.

Nothing about the flow’s own behaviour is reported here. A flow that would have answered 422 on /run still answers 202 on /enqueue, and its failure surfaces later, in the log and eventually in the dead-letter queue.

Where it is stored

One log per (tenant, flow) pair, under the state directory:

File	Holds
`queues/{tenant}/{flow}/segment.log`	The messages, appended in order.
`queues/{tenant}/{flow}/segment.log.ckpt`	Byte offset up to which everything has been processed.
`queues/{tenant}/{flow}/segment.log.retry`	The pending retry schedule.
`queues/{tenant}/{flow}/segment.log.seq`	The next message id. Durable, so ids keep counting across a drain and are never reissued.
`queues/{tenant}/{flow}/dlq.log`	Dead-lettered messages.
`queues/{tenant}/{flow}/dlq.log.acked`	Which dead-lettered messages you have already handled.

Each frame carries a checksum, and a frame that does not verify is not delivered.

Both path components are checked against the rule for a name that becomes a path before anything is created. A pair that breaks it — ../x, a name with a dot or an uppercase letter — is refused with an error naming the tenant, the flow and the character at fault, and no directory appears. This is the same rule the compiler applies at publication, so on a healthy installation it never fires; it exists for the paths that reach the queue without passing through publication: an operator’s nexus dlq replay --flow …, the Queue page of the UI, and an artifact published by a build that predates the rule.

The worker

A worker is a loop that dequeues, runs the flow, and acknowledges or reschedules. One worker per (tenant, flow). It is started on the first submission to that flow after the process starts, and then runs until the process exits, polling every 500 ms while the queue is idle. If several server processes share a state directory, a lock file elects one of them to run the worker for each queue.

That has a consequence worth knowing: after a restart, messages left unprocessed in a log sit there until something is submitted to that flow again. The submission that wakes the worker can be any message — the backlog is drained first, in order.

The worker resolves latest when it picks a message up, so a message submitted before a deploy is executed by the version that is live when its turn comes. See Server and configuration.

Publishing from a flow

/enqueue is not the only way in. A step with the queue_publish effect puts the current message on another flow’s queue in the same tenant:

## Step: build-task
```ntd
{ "orderId": "{{ $.orderId }}", "requestedBy": "{{ ctx.correlation_id }}" }
```

## Step: hand-off
effects: [queue_publish]
queue: order-fulfilment

Key	Required	Meaning
`queue`	yes	The flow whose queue receives the message. A step without it is refused at compile time, and so is one whose value breaks the rule for a name that becomes a path

Three things follow from how effects work, and none of them is guessable:

The message is the previous step’s output. Effects run before the step body, exactly as they do for http_egress, so you build the payload in one step and publish it in the next. There is no payload template on the publishing step.
The tenant is the running flow’s tenant, never a key you can set. Cross-tenant publishing is not expressible.
The message must have a JSON form. The queue stores JSON, so an XML document is refused naming the step, rather than being flattened into a shape the target flow cannot read back.

The correlation ID travels with the message, so the queued run continues the same trace.

The publish fails — and the step fails with it — if no worker can be started for the target queue, for instance because the residency budget is exhausted. That check happens before the write: a message in a queue nobody drains is a silent loss wearing the appearance of success.

Retry configuration

Four front-matter keys, all optional.

Key	Type	Default	Effect
`queue_max_attempts`	integer	`3`	Total delivery attempts. On the attempt that reaches this number, the message is dead-lettered instead of rescheduled.
`queue_initial_delay_ms`	integer	`1000`	Delay before the second attempt.
`queue_backoff_factor`	number, at least `1.0`	`2.0`	Multiplier applied per attempt. `1.0` gives a fixed delay.
`queue_max_delay_ms`	integer	`60000`	Ceiling on any single delay.

A queue_backoff_factor below 1.0 is rejected by nexus validate. A value that is not a number, for any of the four, is also rejected — a misspelled number does not silently fall back to the default.

The delay applied after a failed attempt n is:

delay = min(queue_initial_delay_ms × queue_backoff_factor ^ (n − 1), queue_max_delay_ms)

Worked example

---
flowmarkdown_version: "0.1"
flow: forward-order
tenant: acme
effects: [http_egress]
queue_max_attempts: 5
queue_initial_delay_ms: 500
queue_backoff_factor: 3.0
queue_max_delay_ms: 10000
---

## Step: forward
effects: [http_egress]
endpoint: https://orders.example.com/ingest
method: POST

With a downstream that is refusing every call:

Attempt	Outcome	Delay before the next attempt
1	fails	`500 × 3⁰` = 500 ms
2	fails	`500 × 3¹` = 1500 ms
3	fails	`500 × 3²` = 4500 ms
4	fails	`500 × 3³` = 13500 ms, capped to 10000 ms
5	fails	none — attempt 5 reaches `queue_max_attempts`, so the message is dead-lettered

Five attempts, four delays, 16.5 seconds of retrying. With the defaults it is three attempts, two delays (1000 ms and 2000 ms), and 3 seconds.

The schedule is written to disk on every reschedule, with absolute timestamps. A message whose next attempt is two minutes away survives a restart and is still two minutes away. A retry that came due while the process was down fires immediately.

What reaches the dead-letter queue

Two things:

Retries exhausted. The flow failed on the attempt numbered queue_max_attempts.
A corrupt artifact. The stored compiled flow failed its integrity check. Re-reading the same bytes cannot produce a different result, so this skips retrying entirely and dead-letters on the first attempt.

Three things that look similar and do not:

A condition fence that evaluated to false. The message is acknowledged and dropped. This is a filter, not a failure — it is recorded as queue_filtered and never retried. See Routing and validation.
No such flow in the registry. The message is acknowledged and dropped, with an error in the operational log. It could never succeed, so retrying it would loop forever.
A failure to read the registry itself. Retried at a fixed one-second interval. It does not reach the dead-letter queue.

A dead-lettered entry keeps the payload as submitted, the message’s id, when it was enqueued, when it was dead-lettered, how many attempts it consumed, and the text of the last error. It does not keep the correlation ID — to find the trace, look for the dlq_moved event in the structured log. See Logging and audit.

Managing the dead-letter queue

Listing

$ nexus dlq list
DLQ: 2 pending message(s)

  id=7 tenant=acme flow=forward-order
    enqueued_at: 2026-08-09T14:02:11Z
    failed_at:   2026-08-09T14:02:14Z
    attempts:    3
    last_error:  step 'forward': effect handler failed: HTTP 503 Service Unavailable

  id=2 tenant=acme flow=sync-invoices
    enqueued_at: 2026-08-09T09:41:00Z
    failed_at:   2026-08-09T09:41:07Z
    attempts:    3
    last_error:  step 'check': validation failed (Semantic): total must be positive

Both filters are optional and can be combined:

$ nexus dlq list --tenant acme
$ nexus dlq list --tenant acme --flow forward-order

An empty dead-letter queue prints DLQ is empty.

Replaying

$ nexus dlq replay --tenant acme --flow forward-order --id 7
replayed  id=7 tenant=acme flow=forward-order

The payload is appended to the main queue as a new message, at the back, and the dead-letter entry is marked handled so it no longer appears in list. The replayed message starts again at attempt 1 with the full retry budget, and it runs against whatever version is latest when the worker reaches it.

A replayed message gets a fresh correlation ID. The original run and the replay are two traces, not one.

Replay writes to the log; it does not run the flow. Something has to pick the message up. If a worker is already running for that flow it will be dequeued within the poll interval; if the process was restarted and nothing has been submitted to that flow since, the replay waits for the worker to be started by the next submission.

Fix the cause first. Replaying into a downstream that is still returning 503 spends the retry budget again and puts the message straight back.

Discarding

$ nexus dlq discard --tenant acme --flow forward-order --id 7
discarded id=7 tenant=acme flow=forward-order
  the body leaves the disk at this flow's next queue drain; until then it is only hidden from listings

Marks the entry handled without re-running anything. It is gone from list at once, and there is no command to bring it back.

The payload is deleted in two stages, and the gap matters if you are discarding a message because of what it contains. The command writes the id to dlq.log.acked; the flow’s worker rewrites dlq.log without the marked rows the next time it finds the queue empty — seconds on a queue with traffic. Two things to know about the gap:

A flow with no running worker never closes it. Workers start on the first POST /enqueue after a restart. If a flow has received nothing since, nobody applies its marks. nexus dlq list will show the entry gone while dlq.log has not shrunk.
A restart does not lose the mark. dlq.log.acked is durable; the rewrite happens at the first drain after the platform comes back.

To delete immediately, stop the platform and remove {queue_dir}/{tenant}/{flow}/dlq.log together with dlq.log.acked. That also takes the entries nobody marked.

Flags

Flag	`list`	`replay` / `discard`
`--tenant <t>`	optional filter	required
`--flow <f>`	optional filter	required
`--id <n>`	—	required, from `nexus dlq list`
`--db <path>`	optional	optional

An id is unique within one (tenant, flow) queue, not across queues — which is why replay and discard need all three. --tenant and --flow are checked against the rule for a name that becomes a path before the queue directory is opened: a value like ../x exits non-zero, names the offending pair, and creates nothing.

--db defaults to ~/.nexus/registry.db. The queue directory is derived from it, so if you started the server with a non-default --db, pass the same path here or you will be looking at an empty dead-letter queue in the wrong directory.

The Queue page of the management UI shows the same entries and has replay and discard buttons for them. See Logging and audit.

Delivery semantics

At-least-once. That is the whole guarantee.

The checkpoint only advances after a message has been processed successfully. Everything between the checkpoint and the end of the log is delivered again when the process starts, so a crash between “the flow ran” and “the checkpoint moved” produces a second delivery of a message whose effects already happened. A retry is the same thing by design: the flow restarts from step one with the original payload, and every step that succeeded last time succeeds again.

Order is FIFO on first delivery only. Within one queue, messages are delivered in the order they were submitted. A message that is rescheduled loses its place — it comes back when its delay expires, after messages that were submitted later. Do not build anything that depends on two messages arriving in a fixed order.

Each queue is independent. No ordering between flows, no ordering between tenants, no shared throughput.

What you must do to be safe under redelivery

Assume every message will be delivered more than once, because eventually one will be.

The platform’s guarantee stops at at-least-once; exactly-once is built at the destination, not declared on the platform. The pattern is the idempotent receiver: the message’s identifying key travels with it, and the destination recognises the second arrival and refuses it. On the platform’s side there are exactly two dials, and both belong to the flow: how often to retry — only as the flow’s own retry configuration says, nothing retries by default beyond it — and what counts as delivered.

Give the downstream something to deduplicate on. Send a key it can recognise as a repeat — an order number, an invoice id, the correlation ID — and let it reject or ignore the second arrival.
```
## Step: forward
effects: [http_egress]
endpoint: https://orders.example.com/ingest
method: POST
headers: {"Idempotency-Key": "{{ ctx.correlation_id }}"}
```
The correlation ID is stored in the queue frame, so it is the same on every attempt of the same message — which is what makes it usable for this. A replay is a different message and gets a different ID.
Define “delivered”, when a 2xx is not enough. By default a non-2xx from the backend fails the step — which is exactly what triggers the retry, the backoff, and at queue_max_attempts the dead-letter queue. When the backend answers 200 with its verdict in the body, write the condition as a validate step after the egress. On a 2xx the backend’s response is the current document, so the rule reads it as $.:
```
## Step: forward
effects: [http_egress]
endpoint: https://orders.example.com/ingest
method: POST
headers: {"Idempotency-Key": "{{ ctx.correlation_id }}"}

## Step: accepted-means-accepted
```validate
$.status == "ACCEPTED" | "backend did not accept the message"
```
```
A failed rule fails the delivery like any other error: retry with backoff, dead letter at the limit. The message is not delivered until the backend says so, in the form the flow declared.

Unless the flow has a ## Fault: section. A handler that succeeds becomes the delivery’s answer, so the delivery ends in success and the worker acks the message: no retry, no backoff, no dead letter. That is the whole point of a fault handler on the synchronous door, and it is the opposite of what you want here — the message you wanted parked for a human is gone instead, with queue_processed in the log and nothing in the dead-letter queue. The one thing that tells them apart is the outcome column on that row, which reads fault_handled rather than succeeded.

So on a queued flow, decide which of the two you are writing. A handler that turns a rejection into a notification and lets the message go is a deliberate policy; a handler you added for the synchronous door and forgot about is a silently disarmed dead-letter queue. If you need both — a handled answer and the retry — the handler has to fail: end it with a step that errors, and the delivery stays failed.
Derive keys from the payload, never from the clock or a random value. uuid() and now() produce a new value on every attempt, so a key built from them defeats the deduplication it was meant to enable. $.orderId does not.
Prefer effects that are safe to repeat. A PUT that sets a record to a known state is safe to run twice; a POST that appends a row is not.
Do not spread one logical operation across several steps and assume it either all happened or none of it did. There are no transactions across steps. A flow that fails at step 4 will re-run steps 1 to 3 on the next attempt, including their effects.

Housekeeping

A queue reclaims its own space. When its worker finds nothing left to deliver and no retry pending, it empties segment.log in place and rewrites dlq.log without the entries you discarded or replayed. Nothing is deleted while the worker holds the files open, so this is safe on a live installation.

A queue with no running worker — nothing submitted to that flow since the last restart — is handled differently: a background task deletes its files outright, and the next submission recreates them. That task refuses to touch a queue that still has messages, pending retries, or unhandled dead letters.

One consequence to expect on the queues page: total and processed drop to zero after a drain. They count what is in the segment file now, not what the queue has ever carried. The history is in the L1 journal. What survives a drain is the message id, which keeps counting from a durable counter and is never reissued. See Server and configuration.

Not supported

Topic fan-out to several flows from one submission, priority ordering within a queue, and cross-node replication of a queue.