Six runs, two clean nos

A lens does not change what a coding agent gets right. I can say that now with nearly fourteen hundred trials behind it, which is more evidence than I had for most of what I believed in September. It does change what the agent does, reliably, and I spent the last of the budget finding out that those are two different sentences.

The question, for anyone who missed the previous entry: agent-lens turns a corpus into a skill, a fifteen-item checklist in RFC-2119 language with a citation on every line. The instrument scored its own output for shape and stability and had never weighed a single corpus against a real task. So I weighed one. The Rails Foundation’s Agents on Rails corpus is 22 tasks against Writebook, each with hidden verifiers, and it became the scale. Six runs between September 22 and October 7, three offline replays, about $158 in model spend. The per-run notes live in the public repo; this is the verdict.

The first no

A checklist of Rails idioms, shaped from the Rails Guides, moved pass rate by nothing. Haiku 4.5 went 13 to 14 of 44. A much stronger model went 75 to 77 of 110. I built a control with every API name rewritten as a description, in case the names were doing the work. Same score. I shaped a second lens from Writebook’s own source, so the checklist named the app’s actual conventions instead of the framework’s. Same score. The only content effect across all of it was that naming an API made the agent reach for that API slightly more often, and it ran both directions, and it did not pass more tasks.

That last sentence is the one I keep rereading. Evil Martians published a post on Hyperdrive in September with a stat I’d saved as the exact gap a lens should close: 58 of 63 tasks passed, but only 41% of the solutions reached for the Rails API the task was built around. Agents pass by routing around the framework. A lens that names the framework should fix that. And it did, a little, on the reach side. It just turned out that reach and pass are different axes, and the verifiers only see one of them. I’d bet on moving the wrong one.

The yes that lasted a week

A process lens is a different animal. I shaped one from the Rails Testing Guide, pointed at a single claim: every stated behaviour has a test. The strong model, left alone, wrote tests in 50 of 110 trials. With the process lens loaded, 89 to 93 of 110, in three separate arms, with three times the test lines and about one extra step per trial. That mechanism is rock solid. The lens changes what the agent does, every time.

Pass rate across those three arms came in at 84, 75, 82 against a baseline of 75. Pooled, 73% versus 68%, a z-statistic under one. Consistent in direction, not significant, and when I cut it by task it looked like it had a shape: all of the gain sat on the 14 tasks whose hidden checks a self-written test could actually reach, and none of it on the 7 that need instrumentation or assert something the task never said. That was a story I wanted to be true. I’d also drawn the line after looking at the data, which is the oldest way there is to find a pattern in noise.

So the sixth run was a pre-registration. The split written down before launch, the direction one-sided, the kill thresholds in the brief: a gain on the reachable tasks under five points, or the unreachable seven moving as much as the fourteen, and the lens gets downgraded to “behaviour only.” Ten attempts per task this time, both arms, 440 trials, $6.33, just under five hours. Then pool it with the earlier arms for fifteen per task.

Both kill criteria fired. On the reachable fourteen the process lens scored 80% to baseline’s 84% in run six alone, and 82% to 81% pooled. On the seven it couldn’t help, 52% to 50%. The seven moved as much as the fourteen, which is to say neither moved. Across every trial the strong model has run with the process lens loaded, 550 of them, it passes 73% against a baseline of 72%. The agent took more steps, cost a fifth more per trial, and got exactly as many tasks right. A week ago I put myself at 65% that the effect was real. The honest number now is whatever you’d give a coin.

Why the corpus is the ceiling, and why that’s not the whole story

Three tasks fail zero of fifteen in every arm of every run. I read the failing verifier checks and they are all the same kind of thing: an attachment that should have been purged and wasn’t, a variant that shouldn’t have been reprocessed and was, a cascade erase. Application behaviour at the edges. A checklist the agent reads cannot reach those. Two of the three were on my unreachable list, and I’d have told you that explained it, except the third was on the reachable one. The agent could have written the test. It writes a test for what it believes the behaviour is, and its belief is the bug. Reachable on paper is not reachable by an agent that’s confidently wrong.

Jed Schneider’s “After the tests pass” landed on me in September for reasons that had nothing to do with this. I’d written then that the pilot measured pass/fail per task, the pre-release half, and that API reach was the production-side observation I was missing. I had the frame right and the lesson backwards. The pilot’s own instrument was green by its own tests. Every lens was a faithful, well-formed, cited checklist. The tool did exactly what it says. What I’d never tested was whether a faithful checklist from this corpus could touch the failure mode at all. Six runs later: it can’t, and making the agent test its own beliefs doesn’t fix the beliefs.

The thing that reads the diff

I also wanted to know whether anything could gate a finished patch without running it. Three readers over 330 final diffs. A local decision model on the first 1,500 characters: 0.518 AUC, a coin. A hosted decision model with the full diff: 0.519, the same coin. A frontier generative model with the full diff: 0.629, climbing to 0.745 when the router happened to pick its strongest option. The fifteen lens items, scored by every reader, sat at or below chance. Whether these patches pass is not visible on the surface of the diff. If I want a submit gate, it’s a strong fixed model with the diff and the test output, and I’ve filed that as the next proposal.

Two things I’d have missed at k=5

Any loaded skill, regardless of what it says, dropped one specific task from four-of-five to one-of-five. Six arms, four different checklists, with and without the standing “internally verify” paragraph I assumed was the cause. The agent hoists a validation pass ahead of the real pass and the validation pass is wrong. At ten attempts the gap closed to five-of-ten against five-of-ten, so the effect is smaller than six arms made it look, and the lesson survives it anyway: loading a skill at all changes the agent’s shape, and content is the smaller lever.

And five attempts per task is not enough to say anything about a task. Between two arms running the identical lens on the identical model, one task went from four-of-five to zero-of-five. I’d written a per-task sign test in the run 4 notes. I withdrew it in run 5. Run 6 was the confirmation: the one task that still looks lens-shaped at fifteen attempts, 7 of 15 against 2, sits next to two that went the other way by nearly as much. Arm-level counts at a few hundred trials hold still. My confident little table was noise wearing a bar chart, and so, it turns out, was my reachability split.

What I was actually testing

The last draft of this entry ended with a plan. A second corpus, picked so the failing checks were ones a test could catch, and a line I wrote as the consolation prize: if the behaviour change didn’t convert there, I’d have learned the lens makes agents busier, not better. The survey for a second corpus came back empty. The candidates either hide the decisive check on purpose or don’t have exercises at all. So the test moved onto the corpus I had, with the split pre-registered, and the consolation prize is the finding. Busier, not better. It cost $6.33, which is the cheapest definite answer I’ve bought all year.

I wrote in the trilogy entry that I’m drawn to the hard part and the hard part is rarely the part that matters. Shaping a faithful checklist was the engineering, and it worked every time. The three weeks went to a question I hadn’t noticed I was asking, and the answer is that a checklist is the wrong shape for correctness, whatever the corpus. What survives for agent-lens is narrow and real: if you want an agent to do something, write tests, run the formatter, check the migration, a process lens will get it done four times in five. If you want it to be right, nothing you can hand it to read will move that needle, and the gate has to run the code.