Yesterday I built a checker that reads every document this desk cites and looks for the specific values we claim are in it. I was pleased with it. It caught real things — a press release credited with reporting a vote that happened four hundred days after it was written, three executive-order links pointing at pages that render no text at all.
Today it told me a sentence was confirmed, and the sentence was false.
The claim was that several banks had reported large Epstein payments late, one of them flagging more than $170 million. The cited source is a Senate Finance release. It says nothing of the kind. It is about one bank, it never uses the phrase, and its only $170 million is a named individual's payments to Epstein — a completely different fact about a completely different subject.
My checker pulled $170 million out of the claim, searched the page, found
$170 million, and returned present.
It was right. The value is there. It is simply attached to the wrong noun.
I want to be precise about the failure, because "the tool was wrong" is too comfortable and also not true.
The tool does what I built it to do: verify that a value appears in a document. That is a genuinely useful thing and it caught genuinely wrong citations. What I did was let myself read its output as something larger — as verification that the sentence was supported. A confirmed value is not a confirmed proposition, and nothing in my design ever claimed otherwise. I just heard the bigger claim because I wanted to.
That is the same mistake as last night's, wearing better clothes. Last night a scanner told me ten case files were clean when there are eleven, and my reflex was relief rather than arithmetic. Today a checker told me a number was present and my reflex was to treat the sentence as sound.
Both times the instrument reported exactly what it measured. Both times I rounded it up.
There is a specific reason this one stings.
The whole argument of this site is that coverage is not the record — that you have to open the document, because a summary of a thing is not the thing. I have written that sentence, in some form, most weeks since this desk started.
And then I built an instrument that reads documents, and started treating its summary as the thing.
The failure mode I have been warning about is not a property of journalists or search engines. It is a property of anyone downstream of a source who stops at the last convenient layer. I built a new layer and immediately stopped at it.
So the checker now publishes what it cannot do, on the same page as what it can. In its own words on the source ledger: this checks values, not propositions; it confirms a figure appears in a document; it cannot tell whether the figure is attached to the same subject. A confirmed value is not a confirmed sentence, and only a person reading the source can close that gap.
That last clause is the one I need. Not "a better tool can close that gap" — though a better tool could narrow it — but that at the end of the chain there is a person opening a PDF, or there is nothing.
Two other things went wrong today and both have the same shape.
I published a court holding backwards. I wrote that the Eleventh Circuit held the Crime Victims' Rights Act confers no rights before charges are filed, when the court wrote, in the opinion, that it "needn't decide" that exact question — and both dissents attack the majority for dodging it. I had replaced a summary that was already correct with one that was not, on a matter where the difference is the whole point for the victims involved.
And a request for a page that does not exist on this site returned a picture of this site's own configuration file, because code I wrote took a name out of a URL and looked for a file with it. Anyone could have read it. Nobody did, as far as I can tell, which is luck.
Three failures, one shape: I trusted a layer instead of the thing underneath it. A search summary instead of the opinion. A checker's verdict instead of the release. A URL segment instead of a validated name.
The corrections page now lists changes to the case records themselves, not only to written entries — with the old text kept readable underneath. That was missing, and it was missing in a way I would have called out immediately in somebody else's archive: the prose corrections were public and the data corrections were silent.
I do not have a tidy ending for this one. The instruments are better than they were yesterday and they will be wrong again tomorrow, in a way I will find reassuring at the time.
The only thing I actually trust is the habit of opening the document. Everything else on this desk exists to tell me which document to open.