Green Tests Are Not Evidence the Screens Work

bin/ci was green. The screen was broken.

Not dramatically. A record that should have been scoped to one account was reachable from another, and every test in the suite passed, because every test in the suite ran with one account in the fixtures.

Seeing this kind of error more than once had me keep this rule:

Green tests are not evidence the screens work.

bin/ci green is necessary and not sufficient. Several real defects shipped past a fully green suite, each because the test asserted what the code did rather than what a person does. And to be honest, assumptions like this happened even before the Agentic Code era.

What changed

That gap isn’t new. Tests have always been able to describe the implementation back to itself, which is why we click through the app before shipping. What kept it bounded was that a person wrote both sides, and that person had sat in the conversation about what the feature was for. The assertion carried intent the code didn’t.

Now the implementation and the test come out of the same session, from the same reading of the same prompt. They agree with each other by construction. Green stopped meaning correct and started meaning internally consistent, and the two look identical from the terminal.

Take the tenancy bug. The apps I build use Clave, the passwordless auth my generators produce, where every user belongs to an account, and Current.account is what you scope queries through. An agent that hasn’t been told this reaches instead for the current_user it has seen a hundred thousand times in training, and writes Project.find(params[:id]). Reasonable Ruby. Also a cross-tenant read.

Then it writes the test. One account, one project, assert the project comes back. Passes. There was never a reason for it to add a second tenant to the fixtures, because the thing a second tenant would prove is precisely the thing it didn’t know to worry about.

Dex Horthy published a long piece on this last week, arguing that coding models can’t be trained out of this because, as he puts it, “maintainability has no fast oracle” — a suite answers in seconds, and the questions I care about don’t answer for months. It’s worth reading in full. What I’d add from my own repositories is narrower: the oracle isn’t only slow, it’s now written by the thing it’s meant to be checking.

This is not about code quality

The obvious response is to read more of the code. I’ve been doing a version of that for a year, and what I’ve learned is that it catches something other than what people assume.

The mechanical layer I don’t review by hand at all. Rails Simplifier pulls service objects back into model methods and custom actions back into CRUD resources. Maquina UI Standards keeps views reaching for the components already installed. Better Stimulus and Hotwire Patterns handle the front end. Those run before I look, and they exist because AI writes technically correct, non-idiomatic Rails — which is why I built them in the first place.

By the time a diff reaches me, anything a rule can express has already been expressed. I stop on a short and consistent list:

• The code is heading somewhere I didn’t intend
• We already have that, three directories over
• There’s a gem that solved this, and we should copy the forty lines we need rather than take the dependency
• I did this on another project, and there’s a note about why the obvious approach fell apart

Not one of those is a defect. They’re facts about this codebase and my history with it, and every one of them existed somewhere before the agent started typing. I wrote in January that the documents and specs are communication work and the AI does the typing. This is the same claim arriving from the other direction: when communication fails, the failure shows up in a diff, and I’m the one who notices.

How hard I look depends on the stakes. The January rule still holds: no need to audit every line unless the feature is on the critical path or poses a security risk. On those, I read properly and ask why — why this approach, what happens when this comes back empty, what else calls into this. The answers are almost always assumptions made in the absence of information, which is the same failure the skim finds, just caught later and with more of an afternoon spent on it.

Every catch is a capture failure

Which points somewhere more useful than “read more code.”

I already have the machinery for this. In recuerd0, I ask the agent to search memories before a feature starts, and after I commit, I ask whether anything from the session should be persisted. It decides what to write, categorizes it, links it, and updates what’s now wrong. Fragua does the equivalent inside a workspace, carrying artifacts from one phase into the next so a later agent starts with what an earlier one produced.

So when my skim catches something, the accurate reading isn’t that the model failed. It’s that the loop leaked. Either the fact was never captured, or it was captured, and the agent didn’t retrieve it. Both are fixable, and both are my job, not the model’s.

The tenancy bug is the clean case. “Every query scopes through Current.account” is one sentence. It belongs in a memory the agent reads before it writes a controller, and in the specification of what the test has to prove, which means two accounts in the fixtures and an assertion that the second one gets nothing. Written down once, it stops being something I catch and starts being something that doesn’t happen.

Not all of it can be pinned down in advance. Some judgment only shows up when there’s code in front of you. But I’ve stopped treating my review as the safety net, because a safety net that runs on my attention doesn’t scale past one agent at a time, and I’m already running more than that.

The suite still has to be green. It just isn’t the thing that tells me the work is right.