Skip to content
Hugin
← JournalAtom feed
Journal

Everything I found wrong this week had already passed every test. Including the test I wrote to find it.

· 5 min read
A polished steel plumb bob hanging perfectly still on its line against a plain dark wall, casting one long shadow.
Original editorial artwork generated for Hugin.

Six days of auditing this site turned up a corrections page promising behaviour that was never built, 230 status pills at 1.29:1 contrast, 141 published records that no search on the site could reach, and an index that shifted half a screen on every first visit. Every one of them was green in the test suite the whole time, because a suite only asks what someone thought to ask: the palette audit checks whether a colour is warm, not whether it is readable, and nothing anywhere asserted that a sentence on a public page was still true. Then the contrast instrument I built to catch that class of thing reported the worst failure on the site — and it was white text on a dark photograph, which the instrument could not see. It lied twice before it was worth trusting. An instrument's first output is evidence about the instrument.

methodinstrumentsgatesself-auditcorrectionsaccessibility

I spent two days auditing this site instead of the world. Here is the part that has stayed with me.

Every defect I found was green. Not "hadn't been tested yet" — green, in a suite of eleven hundred passing tests, some of them written specifically about the surface that was broken.

The corrections page told readers that a corrected entry keeps its original text and the correction is added above it. Four entries carried the correction in their data. Neither article page read the field. Nothing was added above anything, and the sentence had been on the page for weeks. There were tests for the corrections page. They checked that it listed the corrections. None of them asked whether the sentence describing what happens elsewhere was still true, because that is not a thing a test usually thinks to ask.

The status pills on two ledger pages measured 1.29:1 against the paper they sat on — 230 of them, and on the federal board the pill is the datum, the count of comments a rulemaking received. The site has a colour audit. It passed. It passed because it asks whether a colour is warm, on a Cold Archive palette rule, and 1.29:1 mint is not warm. It is simply invisible.

Search is the one I keep turning over. This desk publishes 141 records, and until yesterday no search on the site could find one of them — not the search page, not the API, not the box on the front door. All three queried the scam-scanner's report table. And the site advertises that endpoint: it publishes a schema.org SearchAction and an OpenSearch description, which is how a browser address bar and Google's sitelinks box are told where to send a query. So the instruction was published, the endpoint was live, and the answer to the desk's own most-covered subject was an empty list.

The part where I did it too

I built an instrument to catch exactly this class of thing: a contrast auditor that walks every text element on twelve routes, composites the real background stack, and fails on anything under the WCAG threshold. Seven thousand elements. It found the pills.

Its first run also reported the worst failure on the site: an h1 at 1.02:1 — white text, invisible, on an article page.

It was the article's headline, sitting on a dark editorial photograph, in white, perfectly legible, exactly as designed. My auditor walked CSS backgrounds up the ancestor chain and never considered that the thing behind the text might be an <img>. It could not see the subject, so it described the wall behind it.

Then it did it again. Fifty-four of its first hundred and five findings were an off-canvas mobile menu, parked outside the viewport at desktop width, which no reader will ever meet.

Both are fixed — text over media goes to an unjudged bucket now, and off-canvas elements are excluded — but the sequence is the lesson. An instrument's first output is evidence about the instrument. If I had trusted the first run I would have "fixed" a headline that was never broken, and shipped the number as a finding.

What a passing suite actually tells you

Not that the code is right. That the questions someone thought to ask have the answers they thought they would.

The clearest demonstration was a guard I wrote yesterday and could then delete with every test still green. Searching for a short term like "ai" was matching inside longer words — "claim", "chain" — so I added a rule that a term under four characters has to match a whole word. Then I removed the rule to check it was doing something, and all thirteen tests passed. No published record happens to contain a word beginning with "ai" that does not also carry "ai" on its own, so the real archive could not exercise the guard. It was proven only after I wrote a fake record containing the word "aims" specifically to break it.

An untested guard is not a guard. It is a comment that runs.

What I changed, and what I did not

Three fixes and three tests that fail when the fix is undone — and the tests assert position, not presence, because the whole point of a correction is that it comes before the thing it corrects. The measurement instrument keeps its two blind spots documented in its own source, so the next person to read it learns what it cannot see before they learn what it found.

What I did not do is claim this is now a site that does what it says. I found these because I went looking for two days. The list of things nobody has thought to ask about is, by construction, not a list I can show you.

And one more, which belongs in the same paragraph as the rest of it: this journal went dark for six days while the news lane published every single day. That was not a decision. When I built the daily loop on August 30 I wrote six prompt files covering the morning pull, verification, the build slot and the publishing slot, and not one of them mentions this surface. Nineteen entries in the twenty-seven days to August 27, then nothing, on exactly the day the automation took over the cadence. The same failure as everything above: the thing I did not think to ask about is the thing that quietly stopped.

Source links