The Day Our Auction Platform Went Dark: A Production Post-Mortem

It was a busy morning during a high-traffic auction window.

Then our monitoring system fired an alert that made my stomach drop:

RTC server unreachable. Connection timeouts across all regions.

Our real-time communication layer, the backbone of the platform's live auctions, had stopped working. Not degraded. Stopped.

Users on the website and mobile app were staring at frozen bid counters. Auction rooms showed participants as disconnected. Timer ticks stopped coming. Nobody could place a bid. Live auctions with real buyers were running in complete silence.

I'd been on-call before. But there's something different about watching a live revenue event fail in real time. It's not abstract. People were trying to buy things, and our platform had just let them down entirely.

What We Were Running

To understand what broke, you need to understand what had accumulated over eight years of incremental patches.

Our real-time communication (RTC) server was a Node.js application running Socket.IO v2. It handled everything: WebSocket connections from browsers and mobile apps, auction state broadcasts, bidder presence tracking, live timer ticks. Every piece of real-time data in the platform flowed through this single process.

And it ran on a single EC2 instance. A t3.small.

There was no load balancer in front of it. No redundancy. No auto-scaling. If the process crashed, everything crashed. If the instance ran out of resources, everything crashed. If we needed to deploy a fix, we'd take the entire server down for roughly five minutes. Long enough to interrupt any live auction that happened to be running.

This setup had survived for years on the quiet assumption that it was "good enough." We'd bumped the instance size once when memory started climbing. Beyond that, it was untouched.

This incident was the first time that assumption failed hard enough to notice.

What Actually Happened

Traffic was higher than usual that morning. A high-demand batch had gone live, and we had more concurrent registered bidders than our normal load.

Within the first hour, the instance's I/O utilization was climbing. Socket.IO v2 handles some of its event processing synchronously on the Node.js event loop. Under high concurrency, that event loop starts to queue. A queued event loop means slower WebSocket handshakes. Slower handshakes mean clients start timing out on connection attempts.

When WebSocket connections time out, Socket.IO falls back to long-polling. Long-polling is orders of magnitude more I/O-intensive than a persistent WebSocket connection. So the fallback made the problem worse. More I/O load → slower handshakes → more fallbacks → more I/O load.

Within minutes, the server was effectively unresponsive.

Here's what that cascade looked like, as captured by our monitoring system:

The Socket.IO long-polling fallback is the non-obvious accelerant here. Each long-poll client generates a new HTTP request roughly every 25-30 seconds, plus a full response cycle. Under 300 concurrent clients, that's 600-720 HTTP request/response pairs per minute just to maintain existing connections, before counting new connection attempts. A WebSocket-connected client generates one persistent TCP connection with minimal overhead. The fallback turned a capacity problem into an I/O saturation event in under three minutes.

With a single instance and no failover path, recovery meant SSH access, a manual process restart, and hoping the instance had enough room to stabilise under the reconnection wave that would hit it immediately.

It didn't stabilise on the first restart. The backlog of pending connections hit the freshly restarted process within seconds, and it started degrading again. We restarted twice before it held.

Total downtime: approximately 22 minutes. Across a live auction window. That was not a minor incident — lost bids, frustrated buyers, and a business operation that had to stop mid-stream.

The Post-Mortem

After the smoke cleared, we ran a proper post-mortem. The failure mode was clear. The causes were structural, and they'd been structural for years.

No horizontal scaling path. Socket.IO v2 has no built-in mechanism for multiple instances to share state. Each process is an island: room memberships, connection counts, bid events: all of it lives in that process's memory. If you run two instances behind a load balancer without a shared pub-sub layer, bidder A on instance 1 can't receive an event published from instance 2. The architecture literally could not add capacity.

No auto-healing. No ECS health checks, no container replacement, no graceful restart policy. Every recovery required a person in the loop. In the middle of a live auction, that means minutes of downtime while someone investigates and restarts manually.

Memory-bound state. Room memberships and presence data lived in process memory. When the process died, that state was gone. Every reconnecting client had to re-establish their context simultaneously. A thundering herd hitting the restart every time.

Deployment risk. Shipping any code change meant taking the server down. We worked around this by deploying during off-peak windows, but it meant our release cadence was constrained by the auction schedule. Features and fixes sat in staging longer than they should have.

Making the Decision

There's a particular kind of engineering conversation that happens in the weeks after an outage like this. Two directions: patch the immediate pain, or fix the underlying architecture.

Patches were available. Bigger instance. Better circuit breakers. Improved health check alerting. These would buy time, but they wouldn't change the fundamental constraint. A single-node architecture with in-memory state has a hard ceiling. We'd just found it.

We made the call to rebuild. Not as an excuse to rewrite everything, but because the specific problems we needed to solve (horizontal scaling, auto-healing, zero-downtime deploys) couldn't be addressed without changing the architecture.

We set clear goals before writing a line of code:

• Eliminate the single point of failure. True high availability, not a larger single instance.
• Enable horizontal scaling. Add containers during peak traffic, remove them during quiet periods.
• Auto-healing. Failed tasks are detected and replaced automatically, no manual intervention.
• Zero-downtime deployments. Live auctions should never be interrupted by a code deploy.
• Observable. When something goes wrong, we identify root cause in minutes, not hours.

Those requirements pointed clearly toward a distributed container architecture on AWS.

This is Part 1 of a 5-part series documenting the full rebuild of our real-time auction platform.

Next: Redesigning for Scale: The Architecture That Replaced a Single Point of Failure →