Persistent Worlds Need Deterministic Governance
When AI agents stop responding to single prompts and start living in multi-day economies, the failure modes change. So must the evidence.
Series: Deterministic AI Engineering · Post 5
A trend is becoming visible: persistent game worlds — long-running, multi-agent, economically and socially complex environments — are starting to be treated as serious AI laboratories. Not because games are toys, but because the failure modes that matter at scale only show up over time. Static benchmarks test reasoning at a single point. Persistent worlds test time-extended behaviour: memory drift over weeks, capability scope-creep across sessions, multi-agent collusion, slow-burn manipulation, audit gaps after incidents the runtime never recorded.
This post is a structural argument: if your AI runs for more than one turn, the safety primitives the field defaults to — output filters and guardrails — are designed for a problem you no longer have. The problem you have is trajectory, not response. The evidence you need has to be produced at run time, not reconstructed after the fact.
Static tests tell you about static systems
A benchmark prompt evaluates how an AI responds to a single question. The answer is right or wrong; you tabulate, you publish a number. This worked for the AI we had until recently — systems where each interaction was independent, where context windows were short, where memory was a footnote.
That framing is breaking. The agents organisations are starting to deploy operate over hours, days, or weeks. They have memory. They take actions with persistent consequences. They interact with other agents and with humans whose own behaviour shifts in response. The interesting failures are no longer “the model said something wrong on prompt 47.” They are slower, distributed, and structurally invisible to per-turn validation.
Static benchmarks test reasoning. Persistent worlds test time-extended behaviour.
Persistent worlds are valuable as research environments for exactly the property that makes them dangerous to deploy carelessly: the failure mode is slow.
Three failure modes you cannot see in a single turn
Three governance gaps surface only when AI runs for more than one turn:
1. Memory drift. What the agent believed on day 1 may not be what it believes on day 30. Each individual update was small; the cumulative drift is significant. Per-turn validation never flagged anything because no single update was wrong. The drift is a property of the trajectory, not of any moment along it. By the time the system’s behaviour visibly diverges from its initial baseline, the deviation is the sum of hundreds of allowed steps — no one of which was the cause.
2. Capability scope-creep. An agent declared “may read user inbox” on day 1 might find itself, by day 14, also forwarding emails — because three intermediate plans extended the boundary one small step at a time. None of the steps individually crossed a guardrail; the aggregate did. The action surface widened without anyone deciding to widen it.
3. Audit reconstruction. When something visibly goes wrong on day 14, can you replay day 13 to find the cause? Can you prove which decision the agent made on day 7 — what inputs it had, what gates passed, what the audit trail actually says? If the runtime did not produce evidence as it ran, no amount of retrospective analysis will recover it. The record either exists or it does not.
These failure modes are not exotic. They show up in any AI system that operates over time with memory, with agency, and with non-trivial action surface. They are common. They are also nearly impossible to address with the safety primitives the field defaults to.
Guardrails validate outputs. They cannot see drift.
The dominant safety pattern today is the guardrail: a checker that runs at output time, asks whether the response violates a policy, and either releases or refuses. Guardrails have a real, narrow job, and they do it well. They catch bad single outputs.
They do not — cannot — catch drift.
Drift is not an output property. Guardrails cannot see it.
Drift lives in the relationship between turns, in the deviation between today’s behaviour and the established baseline, in the cumulative effect of small allowed steps. By the time a single output looks suspicious enough for a guardrail to refuse, the system has already drifted. The same logic applies to capability scope-creep, to multi-agent collusion, to audit reconstruction. They are properties of trajectories, of records, of cross-turn relationships. Guardrails are not designed to see them, and adding more guardrails does not change what they can see.
What runtime evidence actually looks like
A runtime evidence layer is a different primitive. Three concrete instruments:
Signed audit chain. Every governance decision the runtime makes — gate passed, gate refused, capability check, action released — produces a record. Each record is hash-chained to the previous one and cryptographically signed. The record is not a log; it is an artefact. If a record is modified after the fact, the recomputed hash diverges from the stored one. This is not an output property. It is a trajectory property, produced as the system runs.
Behavioural drift detection — coherence telemetry across turns. The runtime tracks summary signals about the system’s evolving state and compares them, over a sustained window, against a baseline. Sustained deviation triggers either a soft alert, a human-in-the-loop escalation, or in the most severe case a fail-closed shutdown. The drift signal is computed continuously; it is visible long before any single turn looks wrong on its own.
Capability-bounded action surface. What the agent is allowed to do is declared up front in a typed contract — not in a system prompt, not in a few lines at the top of the conversation. Bypassing the contract is a policy operation that produces an audit record, not a code path. The surface widens only when the deployer signs off on widening it; the runtime enforces the boundary even when the model does not.
These primitives do not replace the model. They do not replace the orchestrator. They do not replace guardrails. They sit alongside them and produce something different: evidence that the system was operating within its declared limits, recorded as it ran, verifiable later, immune to retrospective denial.
Where we are building toward
Persistent worlds need persistent governance. Not as a marketing line — as a structural argument. If your AI system runs for more than one turn, you need:
An audit chain that exists at run time, not at incident-review time.
A drift signal that is computed before any single output looks wrong.
A capability boundary that is a typed contract, not a prompt convention.
Evidence that is reviewable by someone who was not in the room.
These are not properties you add by writing a better system prompt or installing one more output filter. They are properties of the runtime, and they have to be designed in from the start.
This is the design space we have been building in. The work is open-source, deliberately scoped, and explicit about its limits. It is not a wrapper. It is not an agent framework. It is the audit and telemetry layer that sits above either, producing the evidence that long-horizon AI deployments are going to need — and that the systems being announced as next-generation AI laboratories will need most of all.
The research direction is clear. As AI moves from prompts into persistent worlds, we will need evidence-grade governance for agent state, memory, actions, and long-horizon behaviour.
Persistent worlds need persistent governance. That is where we are focused.