S3, CloudFront, and a WebSocket that kept dying at exactly 60 seconds

The pattern looks simple from a distance: static frontend on S3, CloudFront in front of it for global delivery, and a backend behind that handling the parts of the app that need to be live, chat, notifications, presence, whatever needs a WebSocket instead of a request/response cycle. Every AWS reference architecture diagram makes this look like three boxes and two arrows. Wiring it together for real took a lot longer than the diagram suggests, mostly because S3 and a WebSocket connection want fundamentally different things from the network in front of them, and CloudFront has to be told, explicitly, how to give each of them what they need.

The actual conflict

S3 serves files. It has no concept of a persistent, bidirectional, stateful connection, because it isn't a compute environment at all. CloudFront sits in front of both the bucket and the real backend as the single entry point for the domain, which means the moment you add a WebSocket backend behind the same distribution, CloudFront needs a way to know that some requests are "give me this file" and others are "hold this connection open indefinitely and let both sides talk whenever they want." Point the whole domain at S3 and there's nowhere for the socket traffic to go. Add the backend as a second origin without being careful about how, and you either break the static site or end up with routing that happens to work by accident, which is worse, because it stops working the first time your path structure changes.

Getting the routing right

CloudFront resolves each request against an ordered list of cache behaviors, matched by path pattern, and the first one wins. My first pass left the ordering vague enough that WebSocket handshake requests were falling through to the default S3 behavior, which produced 403s and 404s that had nothing to do with permissions or missing files, S3 was just being asked for an object that was never going to exist because the request was never meant for it.

The fix was to stop being clever about it: one specific cache behavior for the WebSocket path prefix, pointed at the load balancer in front of the real-time backend, and the default /* behavior pointed at S3, with the specific prefix evaluated before the catch-all. Once the two traffic types had genuinely separate paths at the edge instead of overlapping ones, the routing errors stopped being routing errors and started being actual application errors, which is a much better class of problem to have.

The 60-second mystery

Connections kept dropping at almost exactly sixty seconds of quiet, which is the kind of suspiciously round number that tells you the cause isn't in your code, it's a default somewhere in the chain you haven't looked at yet. It turned out to be the Application Load Balancer's idle timeout, which defaults to 60 seconds and has nothing to do with whether the connection is healthy, only with whether bytes have moved recently. A WebSocket that's just sitting open, doing exactly what it's supposed to do, waiting for the next event, looks identical to a dead connection from the ALB's perspective once that clock runs out.

Raising the idle timeout helps (it goes up to 4000 seconds, not the 3600 I originally assumed before actually checking the docs), but that alone is just picking a bigger number to eventually hit again. The part that actually fixed it was adding a real application-level heartbeat, a ping from one side, a pong back, on an interval well inside whatever the shortest timeout anywhere in the path happens to be. It's a small amount of code and it removes an entire category of "why did this die" debugging, because now silence past a couple of missed heartbeats means something is actually wrong, not just that nobody had anything to say for a minute.

HTTPS and WSS, no exceptions

I tried testing against plain HTTP and WS early on to keep local iteration simple, and CloudFront doesn't allow that for anything it's fronting, it wants TLS the whole way through. So the whole path ended up HTTPS and WSS end to end, which is the right call anyway, just not the one I expected to make on day one for what was supposed to be a quick local test.

The sharper edge was the WebSocket handshake headers. Sec-WebSocket-Key, Sec-WebSocket-Version, Sec-WebSocket-Protocol, none of that reaches the backend unless the Origin Request Policy is explicitly told to forward it. Miss one and the handshake doesn't error loudly, it just fails, in a way that looks like a generic connection problem rather than "your CDN quietly dropped a header the protocol needs." That one cost real time, because every symptom pointed at the backend or the client, and the actual cause was a policy setting three layers away from either.

Scaling breaks the one assumption that made everything easy

Everything above works cleanly with one backend instance, and falls apart the moment there's more than one. A WebSocket is a connection to a specific process on a specific node. If a client is connected to node A and an event that matters to them gets published from node B, node A has no way to know that happened unless something bridges the two.

The bridge is Redis, used purely as a pub/sub backplane. Every backend node subscribes to the same channels, and publishing an event from any one node fans it out to every other node, which then pushes it down whatever local connections care about it. The client never knows or needs to know which node it's actually attached to. It's not a novel pattern, it's the same shape as a chat app's presence system or a multiplayer game's event bus, but it's easy to skip in early testing because a single-instance setup hides the whole problem until the day you actually need a second instance for load or availability.

Keeping the backend off the public internet entirely

The load balancer doesn't need to be publicly reachable at all if CloudFront is the only thing meant to talk to it. CloudFront's VPC origins feature lets the ALB live in a private subnet and be reached directly by CloudFront without ever getting a public endpoint, which is a cleaner answer than the older pattern of exposing the ALB publicly and trying to lock it down with custom header checks that CloudFront injects. It's a newer feature and it's a meaningfully better default than what came before it.

What actually changed my thinking

The real shift wasn't any single fix, it was treating the S3 site and the WebSocket backend as two branches of one routing tree instead of two separate deployments that happen to share a domain. Once the cache behaviors, the timeouts, and the header policy were all reasoned about as one system instead of "the frontend config" and "the backend config" bolted together, the whole thing stopped being fragile. It's still more moving parts than a plain static site, and every one of those parts, the ALB timeout, the heartbeat interval, the header policy, the Redis backplane, is a place a future change can quietly reintroduce one of these exact same failures if nobody remembers why it's there.