Logging and audit

Two separate records. The structured log answers “what happened to message X” and is configured per flow. The audit trail is an append-only hash chain you can verify. Then the management UI, which reads both.

The structured log

Every event is written to two places, always both:

Sink	What it is for
nexus-log.db, table flow_log	Querying. Indexed by (tenant, flow, ts) and by correlation_id. This is what the UI reads.
logs/nexus-log-YYYY-MM-DD.jsonl	Shipping. One JSON object per line, one file per day, appended.

Both live under the state directory. See Server and configuration.

Events reach the writer through a bounded in-memory channel that holds 4096 of them. If the channel is full the event is dropped and a warning goes to the operational log. Logging never slows a flow down and never fails one, which is the right trade for a business journal and the wrong assumption to build billing on.

What a record contains

{
  "ts": "2026-08-09T14:02:14.318Z",
  "correlation_id": "3f8b1c42-9e70-4a51-b8d2-1c0e7a4f9b33",
  "tenant": "acme",
  "flow": "forward-order",
  "event": "completed",
  "channel": "http",
  "duration_ms": 87,
  "payload": { "status": "accepted", "orderId": "A-1" },
  "project": "orders",
  "scenario": "intake"
}

Field	Always present	Meaning
ts	yes	ISO 8601 UTC, milliseconds.
correlation_id	yes	See below.
tenant	yes
flow	yes
step	no	Step name. Omitted for the flow-level events, which is all of them today.
event	yes	One of the kinds below.
channel	yes	Which door the message came in by: http, grpc, queue or cron.
duration_ms	no	Omitted when zero. Wall time of the execution, not of a single step.
payload	no	Masked, and trimmed at payload_trimmed. Absent unless the flow asks for it.
error	no	The error text, on failure events. Names the step, the path and the type — never a value from the message. See below.
attempt	no	Delivery attempt number, on the queue events.
project	no	From log_project.
scenario	no	From log_scenario.

Omitted fields are absent from the JSON Lines record and NULL in the database column.

The unit is a delivery, not an HTTP request

One delivery leaves a received plus exactly one verdict — completed, fault or delivery_refused. Which door it came in by is the channel column, not the kind of event: a scheduled run emits the same received and verdict as a call to /run, with channel set to cron. So counting deliveries means counting received, on any channel, and a flow that runs on a schedule is as visible as one that is called.

Event kinds

A delivery, on any channel:

Kind	Emitted when
received	A message arrives, before the flow is loaded or executed.
completed	The execution finished and an answer is going out. Carries duration_ms and, if the flow asks for it, the output payload.
fault	The execution failed. Carries error and, if the flow asks for it, the payload the execution failed on.

Refusals — three of them, three different operator actions, and none of them is a fault:

Kind	Emitted when
access_denied	The caller has no right to this flow: no credential, a revoked one, a missing scope, a flow outside the key’s allowlist.
handoff_refused	The platform is at capacity. The message is durable, but no worker took it.
delivery_refused	The request cannot be identified, or the artifact cannot be loaded, or the body is over the size ceiling.

None of these counts toward the error rate on the Stats page, and none of them makes a flow look active. A port scanner would otherwise raise the error rate of flows it never touched.

Deduplication — only on flows that declare a dedup_key::

Kind	Emitted when
dedup_replayed	A second delivery of a key whose first execution had already finished.
dedup_held_back	A second delivery arrived while the first was still running.

Correlation — only on flows that open or close one:

Kind	Emitted when
correlation_opened	A step declared correlation_open: and the wait was recorded, with its deadline.
correlation_closed	The awaited answer arrived in time and the waiting state was rehydrated.
correlation_closed_late	The answer arrived after the deadline had already been reconciled.
correlation_timed_out	The deadline passed with no answer. Not a fault: a silent partner is not a defect of the flow.

Queue:

Kind	Emitted when
queue_enqueued	A message was written to the queue by /enqueue.
queue_processed	A queue worker ran a message successfully. Carries duration_ms, attempt and, if asked for, the output payload.
queue_filtered	A condition fence dropped the message. Acknowledged, not retried, no payload recorded.
queue_failed	A delivery attempt failed and another is scheduled. Carries error, attempt and, if asked for, the payload.
dlq_moved	A message was moved to the dead-letter queue. Carries error, attempt and, if asked for, the payload.

Connectors and certificates:

Kind	Emitted when
connector_received	A connector took a message off its source and is forwarding it.
connector_oversized	A connector skipped a message for exceeding its size limit.
cert_expiring	The maintenance sweep found a registered certificate inside its warning window.
cert_expired	The sweep found one already past its validity.

received is emitted before the artifact is loaded, so it is the one event that carries neither project nor scenario — the flow’s configuration is not known yet.

Every kind in this table is written by something. A kind with a filter in the UI and no producer is worse than a missing feature: you get zero rows and cannot tell “no traffic” from “this is never recorded”. Three kinds lived that way for months and were removed; a test now refuses to let the list grow a fourth.

Correlation IDs

One value tying every record of one message together.

Source	Precedence
X-Correlation-ID request header	first
X-Request-ID request header	second
A generated UUID v4	if neither is present

The value is taken verbatim — it is not validated, so if you send your own it can be any string your systems already use. Both /run and /enqueue behave identically.

For a queued message the ID is stored in the queue frame, so queue_enqueued, every queue_failed, and the final queue_processed or dlq_moved all share it. A retry does not get a new ID. A dead-letter replay is a new message and does get one. A cron tick gets a generated one, since there is no caller to take it from.

Reading it in a flow

It is available as ctx.correlation_id.

---
flowmarkdown_version: "0.1"
flow: forward-order
tenant: acme
effects: [http_egress]
---

## Step: forward
effects: [http_egress]
endpoint: https://orders.example.com/ingest
method: POST
headers: {"X-Correlation-ID": "{{ ctx.correlation_id }}"}

That passes the same ID downstream, so one trace spans both systems. It also works inside a template:

## Step: build-response
```ntd
{ "accepted": true, "trace": "{{ ctx.correlation_id }}" }
```

Log levels

One key per flow, log_level.

Value	Records the event	Records the payload
event_only	yes	no
payload_trimmed	yes	masked, then capped at log_payload_max_bytes
full	yes	masked, no cap

event_only is the default. There is no setting that stops a flow appearing in the log — every level records the event and its verdict; event_only just stops there, before the payload.

Key	Type	Default	Meaning
log_level	event_only | payload_trimmed | full	event_only	As above. Any other value is a parse error.
log_payload_max_bytes	integer	4096	Cap used by payload_trimmed. Ignored at every other level.

---
flowmarkdown_version: "0.1"
flow: forward-order
tenant: acme
effects: [http_egress]
log_level: payload_trimmed
log_payload_max_bytes: 8192
---

What gets recorded is the payload at the moment the verdict is written: the flow’s output on completed and queue_processed, and the value the execution failed on for fault, queue_failed and dlq_moved. Same masking, same trimming, same level. The inbound body of a successful call is never stored as such: received is emitted before the flow’s log level is known.

An error message names the shape, never the value

A runtime error tells you where it happened and what shape it found. It never prints the value it found:

step 'read': field access '.iban' on a String
step 'total': arithmetic on a non-number, got Object

That is deliberate, and it is why the failure events carry a payload at all. An error message goes to four places that are not masked — the process log, the dead-letter row, this error field, and the audit chain — so a message that printed the value would put message content in all four, at any log level, including the default. The value goes to payload instead, which is masked, is trimmed, and is governed by log_level.

So diagnosing a type error reads two fields, not one:

• error — the step, the path, the expected type and the type found. Always there.
• payload — the value, masked. Only at payload_trimmed or full.

At the default event_only the value is nowhere. If an incident needs it, raise log_level on that flow and reproduce; do not look for it in the message.

One exception, and it is the operator’s: the dead-letter row keeps the whole message body, in the clear, because a dead-letter queue that does not keep the message cannot replay it. Nothing expires it either — log_retention_days governs this log, not the queues — so retention there is a procedure you run, not a setting you turn on.

A flow whose output is XML records no payload, at any log level. That covers both a SOAP response and a plain XML response: a document is not kept in the log store. The event itself is still written, with its duration_ms and correlation id — you lose the body, not the trace.

If you need the contents of an XML answer in the log, log the parts and not the document: a step that reads the fields you care about into ctx.vars, or a flow that answers with an object and leaves the XML for an egress step. Note the trade-off before you do: masking works on object keys, so fields you extract are masked by name, while a document would not have been.

Payload masking

A fixed set of field names is always replaced with "***". This cannot be turned off, reduced, or overridden, at any log level.

user_name      username       user           password       newpassword
oldpassword    passwd         pwd            code           token
access_token   refresh_token  authorization  secret         apikey
x-api-key      value          credential     id             newcredential
oldcredential  identifier     key            authorizeduser phonenumber

Matching is case-insensitive on the field name, and it applies at every depth. Two consequences that surprise people:

• The whole value goes, not just a scalar. If a key matches, its value is replaced even when that value is an object. A payload of {"user":{"name":"Alice","city":"Oradea"}} is recorded as {"user":"***"} — the subtree is gone, not masked field by field.
• id, key, value and code are on the list. These are ordinary field names in ordinary payloads. If your messages use them, expect the log to show "***", and do not conclude the field was empty. Where you need a business identifier to stay readable, name it something else — orderId is not on the list and is recorded in full.

Adding fields

log_extra_mask_fields is additive. It cannot remove anything from the list above.

---
flowmarkdown_version: "0.1"
flow: onboard-customer
tenant: acme
log_level: payload_trimmed
log_extra_mask_fields: [taxNumber, accountNumber, dateOfBirth]
---

A comma-separated list, with or without brackets. Matching is case-insensitive, like the global list.

How truncation interacts with it

The order is fixed and it is the safe one: mask first, then measure.

At payload_trimmed, the payload is masked, the masked form is serialised, and only then is its length compared to log_payload_max_bytes. If it fits, it is stored as JSON. If it does not, the whole payload becomes a single string:

"payload": "[TRIMMED:4096B] {\"orderId\":\"A-1\",\"lines\":[{\"sku\":\"AB-1\","

The cut is at a character boundary, so the string is always valid UTF-8, but it is not valid JSON — it is a prefix. You cannot query into a trimmed payload; raise log_payload_max_bytes for the flows where you need to.

Because masking runs before the measurement, truncation can never expose a masked value, and a payload too large to store cannot leak one either.

At full there is no measurement and no cap. Masking still applies.

Business classification

Two free-text keys, copied onto every event that knows the flow’s configuration. They exist so you can filter across flows that belong to one system without matching on flow names.

Key	Type	Default	Meaning
log_project	string	none	The application or system this flow belongs to.
log_scenario	string	none	The use case within it.

---
flowmarkdown_version: "0.1"
flow: forward-order
tenant: acme
effects: [http_egress]
log_project: orders
log_scenario: intake
---

Nothing validates or interprets them. Pick a vocabulary and keep it consistent — they are only as useful as they are uniform.

The audit trail

A second, separate record in nexus-audit.db. Append-only: no update and no delete is ever issued against it. One chain per tenant, with its own sequence starting at 0.

What an entry contains

{
  "id": 42,
  "tenant_id": "acme",
  "sequence": 7,
  "event_type": "key.created",
  "correlation_id": "",
  "payload": {
    "action": "key.created",
    "key_id": 5,
    "key_label": "billing importer",
    "tenant": "acme",
    "scopes": ["run"],
    "expires_at": null,
    "allowed_flows": null,
    "subject": { "kind": "ui_user", "id": 7, "label": "ana" }
  },
  "prev_hash": "9c2e…",
  "hash": "4a9c…",
  "created_at": "2026-08-09T14:02:14Z"
}

Field	Meaning
id	Autoincrement row id. Assigned by the database, outside the hash.
tenant_id	Which chain this entry belongs to.
sequence	Position in that chain, starting at 0.
event_type	One of the kinds listed below, e.g. key.created.
correlation_id	The triggering request’s correlation id, empty string if there wasn’t one.
payload	Event-specific facts — see the tables below. Never the credential, the message content, or a password.
prev_hash	The hash of the entry immediately before this one in the same tenant’s chain. Empty string for the first entry.
hash	blake3 of this entry’s own content — see “The hash chain” below.
created_at	ISO 8601 UTC timestamp, set at insert time. Outside the hash, like id.

There is no subject column. Who acted is folded into payload, under the key "subject", with three fields:

subject field	Meaning
kind	One of ui_user, local_operator, api_key — see below.
id	The row id of the subject in its own table. Never cryptographic material — an i64 cannot hold a raw key or a blake3 hash.
label	A human-readable name: the account’s username, the key’s label, or a fixed placeholder for the CLI.

The three kind values, and what identifies the actor for each:

kind	id is	label is	When it appears
ui_user	ui_users.id, the real row of the account that was logged in	its username	The act came from the management UI.
api_key	api_keys.id, the key that was presented	the key’s own label	The act is access.denied for a caller who presented a valid, identifiable key but was refused on rights.
local_operator	always 0 — not a lookup, there is no table of CLI users	always the literal string "nexus CLI"	The act came from running the nexus binary. A declared placeholder: the platform never authenticates whoever is at the terminal, so two different administrators running the CLI on the same machine produce identical, indistinguishable entries.

An entry with no subject at all — a queue.message.* or grpc.call row — has no "subject" key in its payload, because nobody decided it; it happened as a consequence of traffic, not of an operator’s choice.

What is recorded

Two families, split by whether the entry can be attributed to someone.

Policy acts — somebody decided something — carry a subject: who acted, as a management-UI user, the local operator placeholder for a CLI command, or the API key at the gate. Most land on the chain of the tenant they concern:

Event type	Written when
tenant.registered	A tenant was registered. Payload: display name and the authentication methods granted. Normally the first row of that tenant’s chain
tenant.disabled	A tenant was taken out of service
tenant.enabled	A disabled tenant was put back. Note this restores nothing else — keys that were never revoked start authenticating again at that moment
key.created	An API key was issued. Payload: row id, label, scopes, expiry, flow allowlist — the rights granted, never the key or its hash
key.revoked	An API key was revoked. Payload: row id and label; the rights are not repeated, they are already in this chain
dlq.replayed	An operator put a dead-lettered message back on the queue. Payload: message id, attempts, timestamps, size — never the content
dlq.discarded	An operator gave up on a dead-lettered message. Same payload, different type: a message back on the wire and a message abandoned must be distinguishable without interpretation
state.lease_released	An operator released a delivery lease by hand, knowingly authorising a second concurrent execution. Payload: flow, key fingerprint, previous lease owner and expiry, attempt, correlation id
access.denied	A caller was refused at the gate and could be identified. Payload: route, flow, correlation id, operator-facing reason. On the credential’s own tenant chain, never the tenant in the request path
trace.enabled	A sysadmin turned on the payload trace for one (tenant, flow) pair. Payload: flow
trace.disabled	The counterpart — the trace was turned back off. Payload: flow

Still policy acts, still carrying a subject, but eleven of them have no tenant to land on: management-UI accounts, installation settings, and profile actions belong to the installation, not to any one tenant. The chain is per tenant, so these go on a single reserved tenant, nexus, that exists in the registry for exactly this purpose and cannot be called like a real one:

Event type	Written when
ui_user.created	A management-UI account was created. Payload: row id, username, role
ui_user.role_changed	An account’s role changed. Payload: row id, username, new role, and the previous role — the only place the old value survives
ui_user.disabled	An account was disabled. Payload: row id, username, role at the time
ui_user.password_changed	Someone changed their own password. Payload: row id, username — never the password or its hash
ui_session.revoked	One UI session was revoked. Payload: session id
ui_session.revoked_all	All other sessions of the account were revoked at once. Payload: how many
system_config.changed	An installation-wide setting was changed. Payload: the key, the previous value (null if it was never set), the new value. One row per key changed
maintenance.compaction_run	An operator ran queue compaction by hand from the Settings page. Payload: number of segments compacted
maintenance.retention_run	An operator ran the log-retention sweep by hand. Payload: rows deleted, files deleted, the retention-days threshold used

Processing events are the second family: no subject, because nobody decided them — they are a consequence of traffic, not of an operator’s choice.

Event type	Written when
queue.message.processed	A queued message ran successfully. Payload: flow, message id, attempt, duration
queue.message.failed	A delivery attempt failed and a retry is scheduled. Payload: flow, message id, attempt, error, delay before the retry
queue.message.dlq	A message was dead-lettered. Payload: flow, message id, attempt, error
grpc.call	A gRPC call finished, successfully or not. Payload: step, descriptor name and version, descriptor hash, service, method, outcome, number of messages received

That is the whole list, and note what is not in it. A refusal the platform cannot attribute — no credential, a malformed one, an unknown tenant asked for anonymously — writes no audit row at all, only a structured-log event: without a subject it is a counter, not an attribution, and it is also what stops an anonymous caller from starting a chain on a tenant id it invented. A 429 is the same: a load measure, not a verdict on rights.

Deployments are not here either, and not in the structured log — a deployed version is recorded by the registry itself, with its version, hash and timestamp, which is what the Registry page shows. Successful /run calls are in the structured log, not in the chain.

An audit write that fails is logged as a warning and does not interrupt the work that triggered it. Message processing is never held up by the audit database.

The hash chain

Each entry stores the hash of the one before it, and its own hash covers that link. The hash is blake3 over these fields, NUL-separated, in this order:

tenant_id
sequence          (8 bytes, big-endian)
event_type
correlation_id
payload           (JSON)
prev_hash         ("" for the first entry in a tenant's chain)

The row id and the timestamp are deliberately outside the hash — they are assigned by the database and say nothing about the entry’s content.

So changing any recorded field, reordering entries, or deleting one from the middle breaks the chain at that point and at every point after it. Appending a forged entry to the end requires knowing the current tail hash, which is in the same database — the chain proves tampering with history, not that a well-resourced attacker with write access never appended anything.

Querying and verifying

$ nexus audit --tenant acme --limit 5
seq     event_type                      created_at                  correlation_id
------------------------------------------------------------------------------------------
41      queue.message.processed         2026-08-09T14:05:02Z        7c1e9a20-...
       payload: {"attempt":1,"duration_ms":74,"flow":"forward-order","msg_id":12}
40      queue.message.dlq               2026-08-09T14:02:14Z        3f8b1c42-...
       payload: {"attempt":3,"error":"step 'forward': effect handler failed: HTTP 503 Service Unavailable","flow":"forward-order","msg_id":7}
39      queue.message.failed            2026-08-09T14:02:12Z        3f8b1c42-...
       payload: {"attempt":2,"error":"step 'forward': effect handler failed: HTTP 503 Service Unavailable","flow":"forward-order","msg_id":7,"retry_in_ms":2000}

Flag	Default	Meaning
--tenant <t>	required	Chains are per tenant; there is no cross-tenant view.
--limit <n>, -l	20	Most recent entries first.
--verify	off	Also walk the whole chain and check it.
--db <path>	~/.nexus/nexus-audit.db	The audit database.

--verify checks three things over every entry in the tenant’s chain: that sequence numbers are contiguous from 0, that each prev_hash equals the previous entry’s hash, and that each hash recomputes from the entry’s own fields.

$ nexus audit --tenant acme --verify
...
chain integrity: OK (41 entries checked)

It collects every problem rather than stopping at the first, and the command exits non-zero when there is one:

$ nexus audit --tenant acme --verify
...
chain integrity error: hash mismatch at sequence 17: stored=4a9c…, computed=81ef…
error: audit chain integrity check failed (1 error(s))

--limit does not affect verification; the whole chain is read either way. On a long chain this is not a cheap command.

The management UI

The same process serves a web UI at /ui. With the default port:

http://localhost:9090/ui

It reads the same databases the server writes. Its accounts and sessions are entirely separate from API keys — a UI login grants nothing on the flow endpoints, and an API key grants nothing in the UI. Creating the first account is described in Server and configuration.

Pages

Page	Shows
Dashboard	Volume and error indicators, recent deployments, health.
Registry	Every deployed flow, its versions, hashes and deployment times.
Logs	The structured log, filterable by tenant, flow, level and free text, paged.
Stats	Charts over the log: volume, errors, latency, busiest flows and tenants.
Queue	Per-queue counts of pending, retrying and dead-lettered messages. The replay and discard buttons are sysadmin only.
Profile	Your own display name, password, theme, language, and active sessions.
API Keys	Create, list and revoke keys, including the per-key flow allowlist.
Audit	The audit chain, with a verify button.
System Logs	The operational log the process writes about itself.
Users	Create users, change roles, disable accounts.
Settings	Log retention and queue compaction, and buttons to run either now.
Certificates	Expiry of the certificates the deployed flows use, reconciled against the registry.

Roles

Three, set per user, changed from the Users page.

Role	Navigation offers
support	Registry, Queue, Logs, Profile.
developer	The same, plus Dashboard and Stats.
sysadmin	Everything, including API Keys, Certificates, Audit, System Logs, Users and Settings.

Read that column precisely: it is what the sidebar shows, which is not the same as what the server enforces. Two different lines, and you need both.

The model below is a decision, not an accident of how the pages were written: one enforced boundary, plus one role that sees something the others do not.

What the server enforces is sysadmin, and only sysadmin. It guards the six pages in the lower half of the sidebar — API Keys, Certificates, Audit, System Logs, Users, Settings — and, on pages any role can open, three actions:

Action	Page	Why it is not a support action
Replay a dead-lettered message	Queue	It puts the message back on the wire
Discard a dead-lettered message	Queue	It destroys the message
Turn payload tracing on or off for a flow	Registry	It changes what gets recorded

Everything else needs a valid session and nothing more. A support user who types the Dashboard or Stats URL gets the page — those two handlers ask for a session, not a role. developer is a label on the person, useful for knowing who is who; it is not a privilege, and nothing checks it.

support is not a lesser role — it is the troubleshooting one, and it is the only role that reads the payload trace. A sysadmin who asks for step_traced or egress_traced by name gets zero rows: someone who can change every setting in the installation still cannot read a flow’s messages just by asking for them. That boundary is enforced where the rows are selected, not hidden in the page. See Payload tracing — and note the division of labour there: only sysadmin decides that a trace gets collected, only support reads it.

One consequence of the shape above is worth stating plainly, because a table of roles invites the opposite conclusion: every role reads the ordinary log, including recorded payloads. The Logs page shows them for any flow whose log_level records them, and a support account is exactly the kind of account that gets handed out widely. log_level is therefore the control that decides who can see message contents — not the role. Choose it with that in mind, and see Log levels.