The Decisions, Near-Disasters, and 35% Cost Savings from Rebuilding a Live Bidding Engine

Every architectural decision in this redesign came with a tradeoff. Some were clear-cut. Some were genuinely hard calls. And one decision in staging almost derailed the entire migration, not because of a bad architectural choice, but because of a configuration mistake we were very lucky to catch before it reached production.

This final article in the series covers the engineering decisions that shaped the architecture, the CI/CD pipeline that made zero-downtime deployments possible, a near-disaster in staging, and the actual numbers from the system that's been running live ever since.

Context: Part 5 and final in the series documenting our real-time auction platform rebuild. Full case study → | Start from the beginning →

Why Redis Pub/Sub, Not Kafka

The most common question about this architecture: "Why Redis for the pub-sub layer? Why not Kafka?"

It's a fair question. Kafka is robust, well-understood, and handles enormous volumes. But it's the wrong tool for this problem.

The job of the pub-sub layer in a Socket.IO system is low-latency event broadcast between server instances. When a bidder places a bid, that event needs to reach every connected client in under 100ms. The Redis adapter publishes to a Redis Pub/Sub channel; all subscribers receive it via an active TCP connection with no polling. Delivery is measured in single-digit milliseconds.

Kafka is optimised for durable event streaming: writing events to a persistent, partitioned, replayable log. The overhead of producer batching, consumer group coordination, and offset management adds latency that's incompatible with live auction event delivery. It's also substantially more operationally complex: brokers, KRaft, retention policies, consumer groups, partition rebalancing. That's a meaningful amount of infrastructure to own and operate.

Redis Pub/Sub is not durable. If a subscriber is disconnected when an event is published, it misses the message. That's fine here, Socket.IO handles reconnection and state re-sync at the application layer. We don't need replay; we need fast delivery to currently connected clients.

Redis also gave us shared state storage (presence data, session cache) in the same cluster. One dependency, two roles, simpler operational picture overall.

Why ECS Fargate, Not Kubernetes

Kubernetes was a reasonable alternative. More flexible, more portable, with a larger ecosystem. We chose ECS Fargate for three reasons:

Existing AWS infrastructure. Our team already runs significant workloads on AWS. ECS Fargate integrates natively with ALB, ECR, CloudWatch, IAM, and VPC networking. Setting up EKS would have required ingress controllers, new IAM roles, networking configuration, and operational expertise that didn't exist on the team.

Zero node management. With Fargate, we define task definitions and ECS manages the underlying compute. No EC2 nodes to patch, right-size, or monitor. For a team that's primarily product-focused, eliminating the operational surface of container nodes was the right trade.

Time to production. We were rebuilding a system that had just failed in production. The priority was getting a reliable architecture deployed, not building the most flexible possible platform. ECS Fargate reached production meaningfully faster than EKS would have.

The tradeoff: ECS is more constrained than Kubernetes. No custom operators, limited multi-tenancy patterns, smaller ecosystem for tooling. For our scale and team context, those constraints were acceptable.

CI/CD: GitHub Actions + ECR + Rolling Deployments

The deployment pipeline was a first-class concern from day one. The old system's five-minute downtime-per-deploy wasn't just an availability problem. It was a velocity problem. Deployments needed to be fearless.

Two optimisations that cut deployment time significantly:

Docker layer caching. We restructured the Dockerfile to separate dependency installation from application code. The node_modules layer is cached in ECR and reused unless package.json changes:

FROM node:20-alpine AS base
WORKDIR /app

# Layer 1: Dependencies, cached unless package.json changes
COPY package*.json ./
RUN npm ci --production

# Layer 2: Application code, rebuilt every deploy
COPY . .
RUN npm run build

EXPOSE 3000
CMD ["node", "dist/server.js"]

Parallelized tests. The test suite previously ran sequentially. We split it into three parallel GitHub Actions jobs: unit tests, integration tests, and socket lifecycle tests. Combined with layer caching, total deployment time dropped by roughly 50%.

The rolling deployment itself is handled by ECS. When a new task definition is registered, ECS launches new tasks with the updated image, waits for them to pass health checks, and then drains and stops old tasks. ECS keeps the minimum healthy count satisfied throughout. No auction downtime, no coordinating deploys around the auction schedule.

The Near-Disaster: Redis Connection Pool Exhaustion in Staging

This is the one that almost got us.

During our staging load test (we used k6 to simulate production-scale concurrent connections with realistic bid event patterns), the pub-sub layer started behaving erratically. Events were missing. Redis was responding slowly. The Fargate tasks reported healthy but were clearly under stress.

We added detailed Redis client metrics and the problem became visible: the pub-sub layer was opening a new Redis connection for every emitted event.

The root cause was a misconfigured Redis client. In some default configurations, the Redis client's connection pool size is effectively 1. Under load, the pub-sub layer was queuing operations, and the queue grew faster than Redis could service it. Within minutes of load test start, Redis connection limits were exhausted. The adapter wasn't failing loudly, it was silently dropping events while the containers looked fine from the outside.

The fix was explicit connection pool and keepalive configuration:

const pubClient = createClient({
  url: process.env.REDIS_URL,
  socket: {
    reconnectStrategy: (retries) => Math.min(retries * 50, 2000),
    keepAlive: 5000,
  },
});

const subClient = pubClient.duplicate();

// Verify connectivity on startup, not at first use
await Promise.all([pubClient.connect(), subClient.connect()]);

We also added Redis connection count as an explicit metric in our CloudWatch dashboard, with an alarm at 80% of the configured limit.

The lesson, and I can't stress this enough: connection limits are silent killers in distributed systems. They fail loudly under realistic load and invisibly in light test environments. A 10-connection integration test would never have surfaced this. We were lucky to have run a production-scale load test in staging before we shipped.

If you're integrating Redis into a high-concurrency system: test with production-scale connection counts before you go live, and monitor connection pool usage in production as a first-class metric.

Cost Analysis: What Reliability Actually Costs

The old setup was a single small EC2 instance. Simple, predictable, cheap.

The new architecture is more expensive. ECS Fargate tasks at minimum run size, ElastiCache with a replica node, ALB, and ECR storage cost meaningfully more at baseline, scaling further with load during peak windows.

That's a 3-4× increase. But the framing matters.

The production incident resulted in 22 minutes of downtime during a live auction window. That downtime had direct revenue impact: buyers who couldn't bid, auctions that ran without participation, operational cost of incident response and recovery. A single incident like that costs more than the annual delta between old and new infrastructure.

More concretely: Fargate auto-scaling means we pay for capacity when we need it and scale down during quiet periods. The old setup provisioned for peak all the time, a t3.small running at 5% utilisation during off-peak hours, still costing the same. When you measure cost per transaction rather than absolute monthly spend, Fargate's auto-scaling is more efficient.

Net result: approximately 35% reduction in AWS compute cost on a per-transaction basis compared to the always-on single-instance model.

Final Results

Since going live, the rebuilt platform has delivered:

The 40% reliability improvement is measured across connection stability, error rates, and deployment consistency. Not a single vanity metric. The zero post-launch outages is the one that actually matters most.

Key Takeaways

If I had to distill the entire series into five sentences:

1. Single-node architectures work until they don't. The failure mode is sudden and expensive. Build for horizontal scaling before you need it.

2. Redis + Socket.IO adapter is purpose-built for multi-instance WebSocket fan-out. It's operationally simple, subsecond in latency, and the right tool for this specific problem.

3. Sticky sessions are a transport mechanism, not an architectural dependency. They keep WebSocket connections stable; your containers should still be stateless.

4. Structured logging is infrastructure, not a debugging aid. Add it from day one. You will need it before you expect to.

5. Test under realistic concurrency before shipping. Connection limits, pool exhaustion, and thundering herd patterns only appear at scale. A light test environment tells you almost nothing about production behaviour.

This wraps up the series. Five articles covering the full engineering story of how we rebuilt a live production auction platform, from incident to architecture to the near-disaster we almost shipped.

The Bidding Engine Redesign case study on my portfolio has a condensed view of the entire project for a quicker read.

Series: Part 1: The Incident · Part 2: Architecture · Part 3: Redis + Socket.IO · Part 4: Resilience · Part 5: Tradeoffs & Lessons