The power went out. Not gracefully — the whole rack, mid-write, the way a home lab loses power when the grid blinks and there is no UPS between the wall and the machines. When it came back, everything restarted at once: the database, Redis, the reverse proxy, the tunnel, a dozen containers racing to bind ports in whatever order systemd felt like that morning.

Most of it recovered on its own. Postgres did its write-ahead-log replay and came up intact. The tunnel dialled home. The application container went green on its health check within seconds. By every signal we normally trust, the service was back. It was not. Every request to the site died in an infinite redirect — 308, over and over, to the exact URL it had just come from — and the browser gave up with ERR_TOO_MANY_REDIRECTS.

The container was reporting perfect health while the site was completely unreachable. Both statements were true at once, and that is the whole story.

Green light, dead site

The first instinct after an unclean shutdown is to suspect the database, so that is where we looked, and that is where we wasted the first few minutes. The data was fine. The app process was up, the workers had booted, the pool had reconnected. docker ps showed the container as healthy.

That word is doing a lot of work, and it is worth being precise about what it actually means. A container health check runs inside the container, against localhost. Ours hit http://localhost:5000/health directly, and that endpoint answered 200 without complaint. So the health check was telling the honest truth: the application, spoken to directly, was perfectly well. What it could not see was everything that happens to a request before it reaches localhost — the tunnel, the proxy, the routing rules. The failure lived entirely in that gap.

A localhost health check cannot detect a routing failure. It proves the app is alive. It says nothing about whether anyone can actually reach it. After an incident that reshuffles your network layer, “healthy” is the least trustworthy word on the dashboard.

So we went to the edge and asked the question the health check could not. A single curl that follows redirects tells you immediately whether you are in a loop:

shell · follow the redirects
$ curl -sIL https://hexvault.co.uk/health | grep -iE '^HTTP|^location'

HTTP/2 308
location: https://hexvault.co.uk/health
HTTP/2 308
location: https://hexvault.co.uk/health
HTTP/2 308
location: https://hexvault.co.uk/health
# ...and so on, forever.

There it is, unmistakable: a permanent redirect whose destination is identical to its source. The server was answering every request by pointing at the request it had just received. Nothing downstream of that ever ran — not the app, not the health of it, nothing. The proxy was replying before the application ever saw the request.

Why the app looked guilty

Our application does have HTTPS-enforcement logic. Any zero-knowledge service should: if a request somehow arrives over plain HTTP, you want to bounce it to HTTPS before a single header is read. So the obvious suspect was our own middleware — a before_request hook deciding, wrongly, that a secure request was insecure and redirecting it.

It is a good theory and it was the wrong one, but chasing it is instructive, because the mechanism it describes is real — it was just happening one layer up. Here is the shape of an app-level version of the bug:

Python · the plausible innocent
if request.headers.get('X-Forwarded-Proto') == 'http':
    # Fine on its own. Catastrophic if something
    # upstream sets that header to 'http' on every
    # request — including the ones already on HTTPS.
    return redirect(request.url.replace('http://', 'https://'), code=308)

The lesson that transfers, wherever the redirect actually lives: a redirect loop is never really about the redirect. It is about a disagreement over what protocol the request arrived on. One layer is convinced the connection is insecure; another has already secured it. The redirect is just the symptom of that argument, repeated until the browser quits.

We had proxied traffic through the container network and confirmed the app itself did not loop when spoken to directly. That exonerated the application and pointed one layer out — to the thing sitting between the tunnel and the app, rewriting requests.

The actual cause, in six lines of YAML

The reverse proxy config had this on the entrypoint that the tunnel delivers to:

traefik.yml · the culprit
entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
          permanent: true   # ← 308, unconditionally

Read on its own it looks like good hygiene: redirect all plain HTTP to HTTPS. In most topologies it is exactly right. Ours is not most topologies. TLS is terminated at the edge by Cloudflare, and the tunnel delivers the request to the proxy over plain HTTP on that very entrypoint. So every request the tunnel forwarded hit a rule that said “this is HTTP, send it to HTTPS,” which produced a 308 back to the same URL, which came in through the tunnel again as HTTP, which hit the same rule. The loop was not a bug in the sense of a typo. Every line did exactly what it said. The lines simply could not all be true at once.

An HTTP→HTTPS redirect on the entrypoint your tunnel delivers to is always a loop. The edge already did the TLS. Telling the origin to redo it just sends the request back the way it came.

The fix was to delete the redirection block from that entrypoint. HTTPS is still enforced — at Cloudflare, which is the correct place to enforce it when Cloudflare is the layer terminating TLS. The origin’s job is to trust that and serve. The moment the block came out, the loop stopped and the site was back.

“Why now?” is the real question

Here is the part that matters more than the fix: that config had been sitting there working for weeks. The power cut did not introduce the bug. It exposed one that was already present and merely dormant, waiting for a restart to reshuffle the order in which containers claimed the network.

That is the pattern worth internalising from any incident that arrives on the back of a reboot. The reboot is rarely the cause. It is a fuzz test you did not write — it restarts everything in a fresh order, with fresh state, and surfaces every latent assumption about what comes up before what. A redirect that only loops when the tunnel attaches in a particular sequence. A certificate resolver that only matters once something forces a reload. These do not show up in normal operation because normal operation never disturbs the arrangement they quietly depend on.

The same outage surfaced a second one exactly like it: a block of TLS configuration that had been dropped under the assumption “the edge terminates TLS, we need no certificates here” — true for public traffic, and quietly false for the direct LAN routes that also bind port 443. Nobody noticed for days, because nothing had restarted to make the proxy reload its certificate store. Same species of bug: correct-looking, dormant, waiting for a reboot to become visible.

Cheap checks that would have caught it

Every one of these is a single command, and each maps directly to a failure this incident produced. They are worth wiring into a post-boot check so the machine tells you it is unwell before a user does:

  • Detect the loop, not just the status. A health check that follows redirects and flags when the destination equals the source would have gone red immediately, instead of the localhost check going green. curl -sIL and compare location to the requested URL.
  • Check the cert issuer, not just that TLS answers. A proxy serving its own self-signed default cert still completes a handshake. Assert the issuer is a real CA, not the fallback, or you will serve a browser warning and never know.
  • Verify containers are running, not merely present. “Created” and “Restarting” are not “Up.” An orphaned container quietly crash-looping in the background is easy to miss until it is the thing you needed.
  • Assert on config landmines directly. Grep the proxy config for a redirect on the tunnel’s entrypoint, and for the presence of the certificate resolver. Both of ours were greppable one-liners that would have failed a check the instant the config drifted.

We turned those into a script that runs a couple of minutes after every boot and again on a timer, and writes its verdict to the journal. It is not sophisticated. It just asks, from the outside, the questions the health check answers from the inside — which are the only questions that were ever in doubt.

What the reboot really tested

None of this was exotic. A redirect that should not have been on that entrypoint, a certificate block that should not have been removed, an orphaned container that should have been pinned to a known-good version instead of silently auto-updating into one that dropped a feature. Ordinary drift. What made it a two-hour evening rather than a two-minute fix was that all of it hid behind a green light, and the first honest signal — that a request could not survive the round trip — only appeared when we went and asked for it from the edge.

The uncomfortable version, which is the true one: the failure was in the config the whole time, and every dashboard we owned said healthy. A power cut did not break the site. It restarted the machine into a state that revealed the site had a latent way of breaking — and handed us, unasked, the exact fuzz test that found it. We wrote down the fix, wrote the checks, and bought a UPS. The checks were cheap. The green light was the expensive part.