Payload tracing

A second, independent trace for one purpose: a support operator investigating a specific delivery needs to see the payload at every step, not just what log_level happened to keep.

log_level is the flow author’s decision, made at deploy time, about what to retain long-term. It is the wrong lever for “what happened to this one message yesterday” — the flow would have had to be deployed with full logging before the incident, on the chance it would need it. Tracing is the other kind of switch: turned on by an operator, on a running flow, and it stays on — retroactively useful the next time something goes wrong, not just the next request.

Turning it on


Where	Registry page, per flow. A toggle next to the flow’s versions.
Who	`sysadmin` only. `support` reads a trace; only `sysadmin` decides one gets collected.
Stored	`flow_trace_config` in the registry database — not the flow’s frontmatter. Turning it on or off does not require a redeploy.
Scope	One `(tenant, flow)` pair. Turning it on for one flow does not affect any other.
Duration	Until explicitly turned off. Not “trace the next request” — every delivery is traced from the moment it is enabled.

Both actions are audited (trace_enabled / trace_disabled), attributed to the sysadmin who did it, on that tenant’s chain. Turning it on twice, or off when it is already off, writes nothing — same rule as revoking an already-revoked API key.

What it adds

Two event kinds, on top of the ones in Logging and audit:

Kind	Emitted when
`step_traced`	After a step finishes. Carries `ctx.input` as it stood at that point — masked, the same 25 global fields plus the flow’s own `log_extra_mask_fields`.
`egress_traced`	After one `http_egress` call, successful or not. Carries the response status, headers, and body — as one record, since a support investigation needs all three together, not three rows to reassemble.

Both are independent of log_level. A flow deployed with log_level: event_only still produces a full trace once tracing is turned on — the two settings answer different questions and neither implies the other.

`step_traced`

{
  "ts": "2026-08-20T09:14:02.118Z",
  "correlation_id": "3f8b1c42-9e70-4a51-b8d2-1c0e7a4f9b33",
  "tenant": "acme",
  "flow": "forward-order",
  "step": "build-order",
  "event": "step_traced",
  "payload": { "orderId": "A-1", "apiKey": "***" }
}

One row per step, in execution order. payload is ctx.input after that step ran, so a value that changes at step three and breaks at step five is visible as the change, not inferred from the before-and-after of the whole flow.

A plain http_egress step — no transform of its own — does not also get a step_traced row: egress_traced already reports the same value, and one step producing two identical rows would be noise. The exception is restore_body: if the step restores a snapshot into ctx.input, that value can differ from what egress_traced saw (the backend’s raw response), so both rows are kept.

`egress_traced`

{
  "ts": "2026-08-20T09:14:02.240Z",
  "correlation_id": "3f8b1c42-9e70-4a51-b8d2-1c0e7a4f9b33",
  "tenant": "acme",
  "flow": "forward-order",
  "step": "call-backend",
  "event": "egress_traced",
  "http_status": 502,
  "payload": {
    "headers": { "content-type": "application/json" },
    "body": { "error": "downstream unavailable" }
  }
}

Written whether the call succeeded or not — a failed backend call is exactly the case a trace exists for. headers has the same credential denylist applied as everywhere else on this platform (authorization, set-cookie, cookie, www-authenticate, proxy-authorization are stripped before this is ever recorded).

body is null in two different situations, and the trace does not collapse them:

the response genuinely had an empty body;
the response was abandoned before it was read, because it announced a size over max_message_bytes (see Limits and constraints). The call still shows up, with its status and headers — a rejected-for-size response is not silently missing from the trace.

Reading it

Trace rows are visible on the Logs page to support only. This is enforced where the rows are selected, not filtered out afterward in the UI — a developer or sysadmin who explicitly asks for step_traced or egress_traced by name gets zero rows, not a page that quietly omits them. The event filter dropdown and the step column only appear for support, for the same reason: offering a filter nobody with that role can ever get a result from would just be confusing.

The entry row

The very first traced row of a delivery has no step name yet — nothing has run. Instead of a blank step, it carries where the delivery came from:

`step` on the entry row	Delivery arrived via
`received`	`POST /run`
`enqueued`	`POST /enqueue`, what was written to the queue
`dequeued`	A queue worker, what it read back out

This is what lets two entry rows on the same correlation_id be told apart — what was written to the queue and what the worker later read can differ in principle, and both are worth keeping.

This label lives only on step_traced rows. The ordinary boundary events — received, completed, queue_enqueued, queue_processed, queue_filtered, queue_failed, dlq_moved — never carry anything in their own step field. event already says what they are; a step column duplicating it would suggest an intermediate stage that does not exist.

Fan-out and re-entry

A foreach, a split, a fork branch, and the ## Fault: sequence each start their own inner run of the executor — and each one writes its own entry row, the same way a delivery does. These are labelled too, distinctly from the top-level ones and from each other:

`step` on the entry row	Written by
`foreach[i]`	Iteration `i` of a `foreach`, in order.
`split[i]`	Branch `i` of a `split` — the index is stable even though branches run concurrently and can land in the log in any relative order.
`fork[i]`	Branch `i` of a `fork`.
`fault`	The `## Fault:` sequence. Not indexed — it never recurses into itself, so there is at most one per delivery. Its payload is not the original message but the synthetic `{ fault_step, fault_error, fault_correlation_id }` object it runs with.

So a trace of a flow with a ten-item foreach shows ten entry rows, foreach[0] through foreach[9], each followed by that iteration’s own step_traced rows — the index is what lets you follow one item’s path through the fan-out instead of guessing from the payload alone.

XML input

Masking works on JSON object keys. An XML value has none, so a step whose ctx.input is XML at the moment it is traced does not carry the document — it carries a marker instead:

{ "_unrepresentable": "XML value (root <Order>) — masking is JSON-only, not traced" }

This is deliberate, not a gap to be filled by converting XML to JSON first: NexusFabric already has an XML-to-JSON conversion (used on /enqueue ingestion), but its convention is built for field access, not for guaranteeing every attribute lands under a name the mask list recognizes. Running it here would risk a masked-looking payload that quietly leaked a value the conversion mapped somewhere unexpected — worse than admitting the document cannot be traced. See also the equivalent note for XML output in Logging and audit.

Volume

Trace payloads are masked but never truncated — unlike an ordinary completed or queue_processed payload at payload_trimmed, which is capped at log_payload_max_bytes. This is deliberate, not an oversight: a support trace exists precisely because that cap was not enough to answer “what happened to this message” in the first place. Applying the same cap to the tool built to get past it would just recreate the shortfall inside it.

The trade-off is storage, not correctness: a flow whose payloads are large writes large trace rows, for as long as tracing stays on for it. Turn it off once the investigation is done, the same way you would stop a tcpdump.

Cost

The enabled/disabled check runs on every /run, /enqueue, and dequeue, uncached, so the question of whether that matters was measured rather than assumed: it costs on the order of 10µs, regardless of how many flows have tracing on. Leaving it on for a flow you are actively debugging is not a capacity decision — the cost that matters is the one above, storage, not this one.