How to Instrument a Multi-Agent System for Complete Causal Lineage

This is the next how-to guide in the series, and the most technical so far. Earlier guides covered what each ZizkaDB function does. This one covers how to wire an agent system so those functions work: why() needs an unbroken causal chain, replay needs complete sessions, and checksums need a stable serialization. All of that is decided at instrumentation time, and mistakes made there cannot be fixed later.

It is written for engineers who own the agent runtime.

A note on the code: the snippets below are illustrative pseudocode in Python style, showing the design. Method names like emit are placeholders, so check the SDK documentation in the repository for the exact calls.

Before you start: what lineage requires from your data

A why() query walks a graph. For that walk to be correct, every event needs four things:

• A stable, unique event ID
• A session ID that groups related events
• A parent event ID pointing to the event that caused it
• A payload that captures the inputs and outputs of that step

The parent link is what separates a causal record from a flat log. The event distributions and parent to child sequences you see in the Agent Behavior view, such as llm_end -> tool_start, come from the same structure. If parent links are missing or wrong, lineage and drift both degrade silently.

Step 1: Define your event taxonomy

Start with a small, fixed set of event types and resist inventing new ones per feature. A workable baseline:

• user_message
• decision
• llm_start and llm_end
• tool_start and tool_end
• assistant_response
• error

Fixed types matter for drift detection, because baselines are built from event and transition frequencies. If every release introduces new event names, your baseline resets and drift scores become meaningless. Add fields to payloads instead of adding types.

Step 2: Generate IDs you can sort and dedupe

Use time-sortable unique IDs, such as UUIDv7 or ULID, generated at the point where the event occurs, not by the database on insert. Two reasons:

1. You need the ID before the write completes, because child events reference it.
2. Client-generated IDs make writes idempotent. If a network retry sends the same event twice, the second write is recognized as a duplicate instead of creating a phantom step in the timeline.

Step 3: Propagate context, do not pass it by hand

The most common lineage bug is a lost parent. It happens when context is passed manually and someone forgets in one code path. Store the current session and parent IDs in ambient context instead.

In Python, use contextvars. In Node, use AsyncLocalStorage.

python

# illustrative pseudocode
from contextvars import ContextVar

current_session: ContextVar[str] = ContextVar("session")
current_parent: ContextVar[str | None] = ContextVar("parent", default=None)def emit(event_type, payload):
    event_id = new_uuid7()
    event = {
        "event_id": event_id,
        "session_id": current_session.get(),
        "parent_event_id": current_parent.get(),
        "type": event_type,
        "ts": now_utc_iso(),
        "payload": payload,
    }
    write_async(with_checksum(event))
    return event_iddef traced_tool_call(tool, args):
    start_id = emit("tool_start", {"tool": tool.name, "args": args})
    token = current_parent.set(start_id)
    try:
        result = tool.run(args)
        emit("tool_end", {"tool": tool.name, "result": result})
        return result
    except Exception as e:
        emit("error", {"tool": tool.name, "error": str(e)})
        raise
    finally:
        current_parent.reset(token)

Wrap your LLM client and tool executor once, in one place, so every call path is traced by construction. The try and finally structure ensures an error event and a clean context reset even when a tool fails.

Step 4: Carry lineage across process and agent boundaries

In a multi-agent system, a parent agent often delegates to a sub-agent running in another task, process, or service. Ambient context does not cross that boundary, so you must pass it explicitly in the message envelope.

python

# illustrative pseudocode
def delegate(subagent, task):
    delegation_id = emit("decision", {"action": "delegate", "to": subagent.name})
    envelope = {
        "task": task,
        "lineage": {
            "session_id": current_session.get(),
            "parent_event_id": delegation_id,
        },
    }
    return subagent.send(envelope)

def subagent_entry(envelope):
    current_session.set(envelope["lineage"]["session_id"])
    current_parent.set(envelope["lineage"]["parent_event_id"])
    ...

Two design choices to make explicitly:

• Shared session or linked sessions. A shared session ID gives one continuous timeline for replay. Separate sessions with a link keep each agent’s baseline clean. Pick one and apply it consistently, because drift baselines are computed per agent.
• Over HTTP or queues. Put the lineage fields in headers or message attributes, and make the receiving side fail loudly, or at least emit an error event, when they are missing.

Step 5: Fix your serialization before you hash

Checksums only verify if the same event always serializes to the same bytes. JSON libraries differ on key order, whitespace, and float formatting, so define a canonical form:

• UTF-8 encoding
• Keys sorted lexicographically at every level
• No insignificant whitespace
• Timestamps in one format and timezone, UTC
• Numbers handled consistently, with decimals stored as strings where precision matters, as in financial amounts
• The checksum field itself excluded from the hashed content

python

# illustrative pseudocode
import hashlib, json

def canonical(event: dict) -> bytes:
    body = {k: v for k, v in event.items() if k != "checksum"}
    return json.dumps(body, sort_keys=True, separators=(",", ":"),
                      ensure_ascii=False).encode("utf-8")def with_checksum(event: dict) -> dict:
    event["checksum"] = hashlib.sha256(canonical(event)).hexdigest()
    return event

Write a test that serializes an event in two different runtimes or library versions and asserts identical hashes. Verification failures caused by serialization drift look exactly like tampering, and they will erode trust in the whole mechanism.

If you also need to detect deleted events, not only edited ones, consider including the previous event’s checksum or a per-session sequence number in the hashed content. This is a design option you can layer on top, so check what the platform provides natively before building it.

Step 6: Design the write path for failure

Decide what happens when ZizkaDB is unreachable, before it happens in production.

• Never block the agent on the write by default. Use a bounded in-memory queue with a background flusher, plus batching.
• Make delivery at least once. Combined with client-generated IDs from Step 2, duplicates are harmless.
• Persist on overflow. If the queue fills or the process is shutting down, spill to local disk and replay later, so you do not silently drop events.
• Choose fail-open or fail-closed per risk level. For a low-risk agent, losing a few events during an outage may be acceptable if you record that gap. For a high-risk action such as moving money, you may prefer to refuse the action when the audit write cannot be guaranteed.

Whatever you choose, emit a metric for dropped, retried, and spilled events. An audit trail with undocumented gaps is hard to defend.

Step 7: Control what goes into payloads

Payloads are evidence, and also a liability. Decide up front:

• Minimize. Log what is needed to explain the decision. Do not dump entire documents when a reference and a hash of the document would do.
• Redact or tokenize secrets and unnecessary personal data before the event is created, since altering an event afterwards breaks its checksum.
• Tag every event with a subject identifier where personal data is involved, so forget() can find and erase everything belonging to one person across events and vectors.
• Record the configuration identifiers that matter for at(): prompt version, model identifier, tool versions, and key parameters. Without these in the record, point-in-time reconstruction is guesswork.

Step 8: Test completeness automatically

Treat the audit trail as a feature with its own tests. Run these checks in CI against scripted sessions and continuously against production samples:

1. Orphan check. Every event except user_message roots has a parent that exists in the same session.
2. Pairing check. Every llm_start has a matching llm_end, and every tool_start has a tool_end or an error.
3. Ordering check. A child’s timestamp is not earlier than its parent’s, allowing for a small clock tolerance.
4. Coverage check. Replaying a scripted session reconstructs the expected sequence exactly.
5. Checksum check. Recomputing checksums from stored events matches on a full sample.

An orphan rate above zero is a bug in your instrumentation, and it means why() will return incomplete chains for some decisions.

Step 9: Gate releases on behavioral drift

Because baselines are built from event and transition frequencies, you can use drift as a release check. A practical pattern:

1. Run a fixed set of representative scenarios against the candidate build, ideally with recorded inputs and stubbed tool responses so the comparison is controlled.
2. Compare the resulting event and transition distribution to the baseline for the current production version.
3. Fail or flag the release when the drift score or specific transitions move beyond thresholds you set, for example a verification step dropping out of a flow where it must always occur.

Be careful with interpretation. LLM behavior is non-deterministic, so run enough scenarios to separate noise from signal, and treat the drift score as a trigger for review, not as an automatic verdict. After release, keep watching the live drift view for days, since production traffic differs from test scenarios.

Step 10: Schedule verification and retention jobs

Run these as routine jobs, not one-off scripts:

• A checksum verification job over recent and sampled historical events, alerting on any mismatch
• A completeness job reporting orphan, pairing, and gap metrics
• A retention job aligned with your per-tenant policy
• A periodic export of checksums to storage that database administrators cannot modify, if your threat model includes insider risk

Assign an owner to each alert, and test the alerts by deliberately corrupting a copy of an event.

What the business case looks like

These figures are modeled assumptions, not measured results. Replace them with your own.

Assume 20 agent incidents a month at 100 euro per engineering hour. When lineage is broken or incomplete, an investigation drifts back to manual log archaeology and takes about 7 hours, costing 14,000 euro a month. With complete, tested instrumentation, it takes about 1.5 hours, costing 3,000 euro a month, a saving of roughly 11,000 euro a month.

The instrumentation itself is a one-time cost. A reasonable estimate for a team with one agent framework is 3 to 6 engineer-days to wrap the LLM and tool layers, add context propagation, and write the completeness tests, so roughly 2,400 to 4,800 euro at 100 euro per hour with 8-hour days. Multi-service systems with many sub-agents will take longer, mostly because of Step 4.

The cost of getting this wrong is asymmetric. Missing lineage for last month cannot be recreated this month.

Common mistakes to avoid

Passing parent IDs by hand. One forgotten code path breaks lineage for every session that takes it. Use ambient context and wrap the clients once.

Letting the database assign IDs. Children need the parent ID before the write returns, and retries create duplicates.

Hashing non-canonical JSON. Verification will fail for reasons unrelated to tampering, and people will stop trusting the check.

Redacting after the fact. Editing a stored event invalidates its checksum. Redact before emitting.

Growing the event taxonomy freely. Every new event type fragments your baseline and weakens drift signals.

Skipping the failure design. Silent drops during an outage leave gaps exactly when an incident is most likely.

Where to go from here

Start with one agent. Wrap the LLM and tool layers, get parent links correct, and run the completeness checks until the orphan rate is zero. Then add sub-agent propagation, the drift gate, and the verification jobs. Each step makes the earlier guides’ functions more reliable, because lineage, replay, drift, at(), and forget() are all only as good as the events underneath them.

If you tell me your agent framework and whether you run single or multi-agent systems, I can adapt these steps and the pseudocode to your stack.

Want to test ZizkaDB on your agent? Try our open source version here: https://github.com/ZIZKA-AI-SL/ZizkaDB

Interested in a design partnership? Fill in the form on our site or reach me directly at founder@zizka.ai