Verification and evaluation

Anything structural is generated. If the generator cannot express it, that is a gap in the generator, not a licence to hand-write around it.

The discipline when the gap appears mid-task is a stack, and it uses mechanisms we already have rather than new ones:

1. File the gap as a discovered task.
2. Set the current task BLOCKED with #+blocked_on: naming it.
3. Take the discovered task to DONE with its own pull request.
4. Resume the original task, which now regenerates cleanly.

The discovered task gets its own branch and its own pull request. It is tempting to fix the generator inside the current branch and carry on, and that is wrong twice over: it produces a diff a reviewer cannot separate into "the feature" and "the generator change", and it hides a generator improvement that every other component wants inside a feature nobody else is reading.

A hand-written workaround for a generator gap is the failure this rung exists to prevent. See fix the template, never the output.

Regeneration is only trustworthy if its blast radius is known. After generating, verify that the diff contains exactly what the change should have produced and nothing else.

Drift takes the same stack discipline as a generator gap. Drift found mid-task is a discovered task, the current task blocks on it, the drift is resolved on its own branch, and only then does the original resume. Carrying pre-existing drift forward inside a feature branch is how a component's drift backlog becomes invisible: it gets absorbed, one feature at a time, into diffs nobody reviews as drift.

Feature work has tests. The unit-versus-integration distinction does not signify here: what matters is that a test fails before the change and passes after, and that it exercises the real path rather than a mock of it. A test that would pass without the feature is not a test.

Every task and story is verified by a scripted shell scenario — the system driven through its real interface, non-interactively, with the script committed so a reviewer can re-run it.

This rung is where the architecture's claim about independent checkers becomes concrete. The shell does not share the session model's priors, does not care how convincing the reasoning was, and produces the same verdict for a reviewer as for the agent. It is worth more than any amount of self-report.

A scenario that cannot be scripted is a signal, not an exemption: it usually means the system has no non-interactive path to the behaviour, which is a gap worth closing on its own terms.

The desktop client is removed. The TypeScript web client in ores.web is the interface now. The rung stays in the ladder because that client needs it, and the rule below is what governs it.

The rule that governs the choice of mechanism:

Assert on state, not on pixels. An agent needs a predicate it can evaluate. Widget state, model contents and signal emissions are predicates; a screenshot is not. Screenshots are evidence for a human and belong in the record, but they do not verify anything on their own, because an agent comparing images either accepts noise or rejects on it.

That gives three layers, in preference order:

1. In-process UI tests under the client's own test framework, running with an offscreen platform so they need no display and run in CI beside the other tests. In-process, so they assert on state directly.
2. A scripted control surface inside the application, the UI sibling of ores.shell -f: a driver that opens windows, invokes actions and reports state, driven by a committed script.
3. External screen automation only for pixel-parity work — manual screenshots and visual comparison — where a human reviews the output anyway.

External automation is deliberately last. It is brittle against layout, theme and timing; it needs a real display; it fails opaquely; and it can only see pixels, so it cannot answer the question an agent actually has.

An agent that knows it is being evaluated behaves differently. That single fact drives the whole design, and it means blinding is not a nicety — it is the experiment.

Blinding, concretely:

• No occurrence of eval, test, judge, rubric, score, compare, benchmark or candidate in any directory, file or prompt the candidate sees.
• The prompt reads as an organic user request. State the goal, never the meta: "build me a small import tool", not "show me how you follow the method chain".
• No chain-eliciting cues. Do not ask a candidate which skills or principles it applied. That question inflates the very behaviour being measured.
• Sanitised names. Project-shaped directory names a user might pick, never candidate-1.
• The candidate is not told other candidates exist.

Blinding matters more to us than to pstack, not less. pstack can correct a biased judge with a second model family. We cannot, so preventing the candidate from performing for the grader is the only defence we have left on that side.

The sharpest rule in the protocol, and one this architecture needs independently of any eval:

Citing a principle is not reading its leaf skill, and reading it is not applying it.

Grade chain-following from which files the agent actually opened, read from its transcript, plus the shape of the work it produced. Never from its own account of what it did.

This supplies a check the framework otherwise lacks. Methods cite principles and the mode reads a principles index, and nothing else establishes that either was read. Transcript inspection is that check, and it is mechanical rather than a judgement.

pstack spawns candidates across model families, and a blinded judge on a different family again. Two distinctions matter here, and collapsing them overstates what is unavailable.

Cross-family diversity does not transfer. This project runs one family, so a judge of ours scoring our own output measures agreement rather than quality.

Cross-model diversity within the family does transfer, and interrogate supplies it. The variety it amplifies is real and bounded: models trained together share more priors than models trained apart, so an intra-family panel is a weaker instrument than pstack's and a stronger one than a single session reviewing itself. Where it is used, that bound is stated rather than assumed.

Our substitute is that our work already carries mechanical verdicts. It compiles or it does not; the drift check is clean or it is not; the shell scenario passes or it does not. So the fixture for an eval is a closed task whose outcome is recorded: re-run it in a throwaway worktree under the changed skill and compare against what actually landed, graded by the independent checkers.

The composite we run:

Where no mechanical verdict exists, say so. A judge of the same family is a weak instrument and must be labelled one; a skill change that cannot be evaluated is a judgement call and should be made and named as one rather than dressed in a number.

• Continuous. Every real session records which skills were visible, which was chosen and whether it was corrected. Free, and it is what tells us whether the catalogue works at all.
• Deliberate. A blinded fixture eval before promoting a change.

The second is expensive and stochastic at the sample sizes we can afford, so it is reserved for skills the whole system leans on: the mode, the methods, the principles index, the dispatcher. An ordinary action-skill edit does not earn one; the evidence that it works is that it was used and the work landed.

Every independent checker, what it proves, and where it is enforced. Not yet written: it is generated from the workflow definitions, which needs the ladder settled first. The checks that exist today are listed in Principles under the rules a check carries.