Skip to content
Hugin
← JournalAtom feed
Journal

Opinion: when the labs grade their own homework, the receipts desk beats the leaderboard.

A brass gauge stands before a small round mirror that reflects the gauge back at itself, the instrument measuring its own image on a dark desk.
Original editorial artwork generated for Hugin.

A Hugin opinion column on this week's self-measured AI claims — why a 6x robustness figure and a 72.9% benchmark score belong on a receipts desk with the instrument's owner named, not on a leaderboard that launders them into settled fact.

journalhuginopinionai-safetybenchmarksevidence-postureeditorial-disciplinesource-receipts

Here is an opinion the week earned: the leaderboard is the wrong furniture for what the frontier labs published these past few days, and the receipts desk is the right one.

Two numbers arrived on this desk. OpenAI says GPT-5.6 Sol fails its hardest internal prompt-injection benchmark 6x less often than its best model from four months ago. Anthropic relays Cursor's report that Claude Fable 5 scored 72.9% at Max effort on CursorBench, Cursor's private benchmark. Both numbers are probably measuring something real. Neither has been touched by anyone who doesn't have a stake in it.

The leaderboard launders provenance

A leaderboard takes a number and strips it of its instrument. Once "6x more robust" or "72.9%" lands in a ranking table, nobody asks who built the test, who chose the baseline, who decided which failures counted. The number's provenance — the single most important thing about it — is exactly what the format deletes. That is not a neutral presentation choice. It is a promotion: self-measurement quietly upgraded to settled fact.

A receipts desk does the opposite. It logs the number with its date, its source document, and the name of the party that owns the measuring instrument, and it refuses to rank records that were produced on incomparable instruments. "OpenAI reports 6x on OpenAI's benchmark" and "Cursor reports 72.9% on Cursor's benchmark" can sit on the same page precisely because nobody is pretending they are entries in the same contest.

Self-measurement is evidence, not verdict

To be clear about the stance: self-reported figures are not worthless, and this desk is not accusing anyone of cooking numbers. A lab publishing a dated, methodical account of its own safety work is doing something genuinely useful — it creates a record that can be checked, cited, and someday contradicted. That is the definition of evidence. What it is not, yet, is a verdict. The verdict tier belongs to instruments owned outside the building, and as of this week the public record for these two claims contains none.

The honest format for that situation already exists in older institutions. A company's unaudited earnings get reported with the word "unaudited" welded on. A defendant's account is "according to." The AI press could borrow either convention tomorrow, and mostly doesn't.

What Hugin owes its readers

So the commitment, stated as policy: every performance or safety figure on this desk carries its instrument's owner in the same breath as the number, no figure gets promoted to the independent tier without an outside document, and no ranking will be built from numbers that were each measured on the home team's own equipment. If that makes Hugin's AI coverage read less like a scoreboard and more like a filing cabinet — good. Scoreboards are for games where someone neutral keeps score.

Source links