Skip to content
Hugin
Back to NewsAtom feed
Hugin News

August 16: this desk's own pages stopped answering, and the alert named the wrong line.

4 min read
A long-exposure night photograph of a wide empty concrete underpass lit by overhead lamps, the roadway completely clear of traffic and carrying no light trails at all.
Original editorial artwork generated for Hugin.

For roughly a quarter of an hour today, every account profile page on this site returned an error instead of a record. The database behind them could not be reached. The automatic alert reported that the cause was the background call that writes visit logs — it was not. That call was the one part of the page handling the failure correctly, which is exactly why it filled the log. Two other reads, with no handling at all, took the page down.

huginnewsoutageincidentevidence-postureoperator-observationaccountabilitydisclosuremethodcorrections
0source receipts0source hosts4 minread timelinkedprimary source

This site publishes how long public bodies take to answer a documented request. Today its own pages answered nothing.

What happened

At 17:15 UTC an alert opened on the route that serves account profile pages — the /u/ addresses carrying what this desk has recorded about a named Reddit account. Over the following five minutes that route logged 380 failed requests. Its average across the preceding 24 hours was 2.

At the point the dashboard was read, fourteen minutes in, the route's error rate stood at 92.9% across 382 requests, and the curve had already fallen back toward the floor. Underneath it, the same panel counted roughly 1,500 timeouts against the Postgres endpoint the site reads from.

For that stretch, anyone who opened one of those pages got an error instead of a record. Not a partial page and not a stale one. Nothing.

The alert named the wrong line

The alert's generated summary said the failures came from a background visit-logging call that had not been handled gracefully by the function.

That call was handled. It sits inside a catch, and during the anomaly it caught and logged its own failure 324 times. That is precisely why it dominated the error log, and precisely why it looked like the cause. A line that survives to be logged 324 times is, by definition, a line that crashed nothing.

The page assembles by requesting five things at once and waiting for all of them together. Three of the five read the database. One of those three had been given a fallback at some point, with a comment describing this exact scenario. The other two had none. When the database went out of reach, those two threw — and one thrown error inside a set that is awaited together discards the whole set, and with it the page.

So the loudest evidence and the actual cause were two different lines. That is worth recording, because mistaking the first for the second is the failure this desk spends most of its time catching in other people's records.

What was changed

The guard now sits with the data rather than with the page. Every profile read returns its empty value on a database failure instead of throwing, so a read added next month is protected by default rather than by somebody remembering to protect it. That inconsistency — one read hardened, its two siblings not — was the whole defect.

The tempting version of this repair was to stop there and let each failed read quietly return nothing. It was rejected. On a site whose purpose is publishing what has been recorded against an account, an account with a long flag history would then render identically to an account with a clean one. A page that cannot reach its evidence must not present that as an absence of evidence.

So each surface now says which it is:

  • The profile page renders — the Reddit-sourced account details never depended on our database — carrying a notice that the archive could not be reached and that the empty sections below may not mean what they appear to mean.
  • The public API answers 503 rather than a well-formed record reporting zero flags, which anything consuming it would otherwise be entitled to cache as a clean account.

One thing should be stated plainly rather than implied: no reader was served a falsely clean record during this outage. The pages failed outright instead of emptying. The decision above governs what the repair does, not what happened today.

What is still open

The routine that confirms the database tables exist re-runs on every request until it succeeds once. During an outage it never succeeds, so every arriving request attempted the full check again. That is part of why the timeout count against the database is as high as it is — the site answered an unreachable database by asking it more often.

That is not fixed today. The obvious repair — remember the failure and stop retrying for a while — risks a genuinely new deployment never creating its tables at all, which is a worse failure than a noisy one. It is recorded here as open rather than left out.

Limits of this record

The alert opened at 17:15 UTC and the error curve had returned close to baseline by the time the dashboard was read. "Roughly a quarter of an hour" is an honest description of that shape; it is not a measured start and end, and this entry does not claim one.

Why the database endpoint became unreachable is not known here. What is known is that it was unreachable from this application, and that this application handled that badly.

The same dashboard panel also recorded around 1,200 timeouts against old.reddit.com in the window. Those are a separate matter and did not contribute to the failures described above — the code paths that read Reddit already degrade to a blank or partial account record, and they did. They are mentioned only so the panel is not quoted selectively.

Primary sourceHugin status