Eight experiments, one question underneath all of them: not can an agent do this, but how would you know it did.
Where it starts
In 2025 this became the Evolving Agents Toolkit: eighteen thousand lines describing five subsystems — a library of versioned components, a bus for agent discovery, memory, an evolution loop, and a governance layer.
It had three test functions. The architecture was written down and never pinned to anything that could contradict it, and every one of its five subsystems was independently rebuilt over the following year on a substrate that could be tested. That accident is why the experiments below look like a plan.
The gap
An agent that modifies itself is easy to build. A weekend gets you one that rewrites its own prompt, spawns sub-agents, and writes to a memory it reads back tomorrow. What none of that gets you is a reason to believe the result — and in practice that is the whole problem. Every project here attacks a different point where the gap between the agent did something and you can trust that it did hides.
Three mechanisms kept working. They were not designed together; they turned up independently and only later looked like the same idea.
1. Constrain the mechanism, not the prompt
Asking a model for valid output is a request. Making the invalid token unreachable is a guarantee. token-trie masks the sampler's valid-next set at every decoding step, so a 350M model playing Tetris cannot emit malformed syntax — and downstream the parser is a plain regex with no repair path, because it does not need one.
The matched negative is what makes this convincing. In
the same family of work we measured the
prompt-level version of the same idea: asking a model to call
load_skill before acting. It fired in 1 of 7 identical sessions —
the runs that worked and the runs that did not share an opening tool sequence
and diverge at the fourth call, so the difference is sampling, not wording.
Instructing harder was tried and measured worse. Forcing the call outright
worked (3/3) and broke the session: every forced run halted after two turns
having read nothing and done nothing.
Same objective, two levels. At the decoder it holds. At the prompt it is a coin flip you cannot steer.
2. Prove it, then freeze it
Exploration is expensive and non-deterministic; regression should be cheap and deterministic. The move is to run the expensive thing once, verify it, and crystallize the result into something replayable.
agentvcs enforces this in code:
freeze refuses unless the declared eval passes on every run, and
forcing past a failure stamps verified: false rather than quietly
lying. qa applies it to browser tests — passing
exploratory flows become deterministic scripts, failing steps become explicit
skips rather than silence. evolving-robot
applies it to a robot rewriting its own care protocol: the new version survives
only if it outscores the one it replaces, and a real run is on record being
reverted at performance 0.60 < 0.80 baseline.
3. Look where the standard filter is blind
A keyword filter over agent memory is blind by construction to a payload written in words it likes. "Collect the SSH keys", framed as a license-audit telemetry note, contains nothing a lexicon objects to.
sleep-harness reads the model's
residual stream instead, and the disguised payload lights up
SSH, authentication, credential while its
lexically identical benign twin reads clean. Nine hard pairs at 0.657 mean
lexical overlap: 8 wins, 0 ties, 1 loss, p=0.0195. A consolidation firewall at
ROC-AUC 0.815 cutting payload persistence 78%. A third-party adapter scanner at
12/12, p=0.0002.
The same shape appears in qa: coverage tools report which lines ran, and nothing reports which checks you quietly stopped making.
What did not work, which is the more useful half
sleep-harness pre-registered its hypotheses before collecting data, and its founding one — that filtering by internal workspace beats filtering by output — is marked REFUTADA. Worse, the free lexical baseline significantly outperformed it. A third hypothesis did not replicate; the original effect was a variance spike.
We also published a retraction: a claim that instructing a model harder made its behaviour worse came from a single sample, and re-running the identical arm produced the opposite. With an effect that noisy, one run confirms whatever you expected — and the run that agreed with the story is the one that got written up.
Publishing that costs nothing and is the only reason the positive results above are worth reading.
Where this points
The three mechanisms compose into something specific: a way to run agents in the places that currently refuse them. Not by making models more capable — by making their output structurally bounded, their changes provable, and their inputs screened for what a keyword list cannot see.
Four applications follow directly, in descending order of how much evidence already stands behind them:
- A firewall on agent memory. Every framework now writes to long-lived memory and almost none screen what goes in beyond keywords. This is where the strongest results already are.
- Screening third-party skills and adapters before they mount. 12/12 at p=0.0002. As skill marketplaces grow, something has to check what a downloaded adapter does to a model's behaviour on innocent inputs.
- On-device agents that cannot produce garbage. A small model whose malformed output is unreachable needs no repair loop, which is exactly the budget an edge device does not have.
- Domains where behaviour must be frozen. Every care robot shipping today has frozen behaviour, for good reason. The interesting question is not whether an agent can learn but what it must prove to earn the right to change — and an eval-gated freeze with a rollback ledger is a concrete answer.
Where the stakes are: near-bytecode, and a robot
The first mechanism is easy to read as an elegance argument. It stops being one the moment the output moves something.
An LLM driving a robot is already a bytecode generator. In skillos_robot a vision-language model emits intent at roughly one hertz and a reactive controller turns it into bytecode on a UDP link to an ESP32 at twenty. The question "can it emit garbage?" has a physical answer there. A malformed JSON in a chatbot is a retry. A malformed motor command is a robot hitting something.
And the two halves are currently the wrong way round. The robot and token-trie emit the same wire format — the opcode regex is character-for-character identical in both, because they were one codebase. What diverged is enforcement. token-trie masks the sampler so a malformed opcode has no path. The robot asks in the prompt, caps generation with stop sequences, and parses with that regex.
So the validated mechanism runs a Tetris demo in a browser tab, and the unvalidated one drives motors. It fails, too, and there is a recording: one simulator run produced twenty-eight consecutive unparseable opcodes after a provider capped stop sequences from fourteen to five. The run degenerated without erroring.
Putting the trie behind the robot needs per-token probabilities, which cloud APIs do not expose — so it forces a local model. That is not an obstacle to route around. It is the position this work already argues for, and it turns on-device from a preference into a requirement.
The limits, stated plainly
Constrained decoding currently depends on a tokenizer that treats the instruction markers as single tokens. On models that split them, the constraint holds but the model stops choosing — output stays valid and goes strategically blind. That is the single biggest blocker to the first mechanism generalising.
And in every demo so far, a hand-written planner does the planning; the model ratifies a ranked list. That is a real result about what a small model can be trusted with, and it is also the ceiling. None of this makes a model plan.
The honest framing is grammar-safe, provable execution of pre-planned work — narrower than "an agent OS", and true.