What Happens When a Container Dies Mid-Auction: Resilience Engineering in Practice

A distributed system isn't more reliable just because it has more moving parts. It's more reliable because those parts are designed to fail gracefully and recover automatically.

The difference between a system that can scale horizontally and a system that's actually reliable is how it behaves when something goes wrong at 2 PM during a live auction. An extra container doesn't help if a crash still requires manual intervention. Redis pub-sub doesn't help if the Redis primary goes down and takes your broadcast layer with it.

This article covers how we engineered failure recovery into every layer of the new architecture: three specific failure scenarios, real recovery timelines, and the observability work that changed how we respond to incidents entirely.

Context: Part 4 of the series on rebuilding our real-time auction platform. See the Bidding Engine Redesign case study and Part 3: Redis + Socket.IO Scaling.

Multi-AZ Deployment: The Foundation

Before we talk about failure scenarios, the foundation: tasks are deployed across multiple AWS Availability Zones.

This means a single AZ outage (hardware failure, AWS maintenance, network partition) only affects the tasks in that zone. Tasks in the healthy AZ continue serving traffic without interruption. ECS automatically replaces failed tasks in the affected zone.

For a live auction platform, this distinction matters. An AZ failure without Multi-AZ deployment is total downtime. With it, it's a blip.

Failure Scenario 1: Container Crash Recovery

What happens: A Fargate task crashes mid-session. Maybe an unhandled exception, an OOM kill, or a host-level issue.

Timeline:

1. Task fails → ECS detects via container health check (interval: 10s, threshold: 3 failures = 30s detection)
2. ALB marks the target unhealthy → stops routing new connections to it
3. ALB deregisters the target → active connections on that task receive a TCP close
4. Socket.IO v4 clients detect the connection close → begin reconnection with exponential backoff
5. ECS schedules a replacement task → Fargate provisions and starts it
6. New task passes health checks → ALB registers it and begins routing traffic
7. Reconnecting clients land on surviving or new healthy tasks

Total recovery time from crash to replacement task registered: approximately 30 seconds.

The Socket.IO v4 reconnection algorithm handles client-side recovery cleanly. With exponential backoff and jitter, clients don't all hammer the new task simultaneously:

// Socket.IO v4 client reconnection configuration
const socket = io("wss://rtc.example.com", {
  reconnection: true,
  reconnectionDelay: 1000,
  reconnectionDelayMax: 5000,
  reconnectionAttempts: Infinity,
  randomizationFactor: 0.5, // distributes reconnection attempts to prevent thundering herd
});

The randomizationFactor is the part that prevents the thundering herd. Without jitter, every client whose connection dropped reconnects at exactly the same time, hitting the new task simultaneously and potentially taking it down too. With jitter, reconnection attempts spread across several seconds.

Failure Scenario 2: Zero-Downtime Rolling Deployment

What happens: We're shipping a new version of the RTC server. In the old architecture, this meant five minutes of downtime. In the new architecture, it means a rolling replacement with no service interruption.

The GitHub Actions workflow that drives this:

# .github/workflows/deploy.yml
name: Deploy RTC Server
on:
  push:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        suite: [unit, integration, socket]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '20' }
      - run: npm ci
      - run: npm run test:${{ matrix.suite }}

  deploy:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1
      - name: Login to ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@v2
      - name: Build and push (with layer cache)
        uses: docker/build-push-action@v5
        with:
          push: true
          tags: ${{ steps.login-ecr.outputs.registry }}/rtc-service:${{ github.sha }}
          cache-from: type=registry,ref=${{ steps.login-ecr.outputs.registry }}/rtc-service:cache
          cache-to: type=registry,ref=${{ steps.login-ecr.outputs.registry }}/rtc-service:cache,mode=max
      - name: Deploy to ECS
        run: |
          aws ecs update-service \
            --cluster production-cluster \
            --service rtc-service \
            --force-new-deployment \
            --region ap-southeast-2

The three test suites run in parallel via matrix:. The cache-from/cache-to pair on the Docker build enables ECR-backed layer caching, which is what cuts build time significantly on most deploys.

Timeline:

1. GitHub Actions publishes a new Docker image to ECR
2. ECS rolling deployment starts: ECS launches one new task with the updated image
3. New task starts and passes health checks → ALB routes new connections to it
4. ECS begins draining the old task: marks it as "draining" in the target group
5. ALB stops sending new connections to the draining task
6. Existing WebSocket connections on the draining task are held open for the drain timeout (60s)
7. After 60 seconds, ECS stops the old task, connections that didn't naturally close get a clean TCP close
8. Process repeats for each remaining old task until all are replaced

Result: no connection drops, no interrupted bid sessions, no auction downtime.

The connection drain window is the critical detail. Without it, ECS would stop the old container immediately, severing all active WebSocket connections simultaneously. This causes every bidder on that task to reconnect at once. With a 60-second drain window, clients finish their current interactions and the connection pool empties naturally before the task is stopped.

Failure Scenario 3: Redis Primary Failover

What happens: The Redis ElastiCache primary node in one AZ becomes unavailable.

AWS ElastiCache Multi-AZ is configured with automatic failover:

• A replica runs in a second AZ, continuously replicating from the primary
• If the primary becomes unavailable, ElastiCache promotes the replica
• A new replica is launched to restore the replicated pair
• DNS is updated to point to the new primary

Timeline: under 30 seconds from primary failure to pub-sub resuming.

During the failover window, new events won't propagate across containers. Events emitted during that window are delivered only to clients on the originating container. This is an acceptable tradeoff: a brief period of partial delivery is far better than complete platform unavailability.

The @socket.io/redis-adapter handles the reconnection transparently. When the Redis connection drops and comes back, it re-subscribes to the channels automatically.

Health Checks: The Mechanism Behind Auto-Healing

All three scenarios above depend on reliable health check configuration. We configured health checks at two layers:

ECS task health check (container-level):

{
  "command": ["CMD-SHELL", "node -e \"require('http').get('http://localhost:3000/health', (r) => { process.exit(r.statusCode === 200 ? 0 : 1); })\""],
  "interval": 10,
  "timeout": 5,
  "retries": 3,
  "startPeriod": 30
}

ALB target group health check:

Protocol: HTTP
Path: /health
Interval: 30 seconds
Healthy threshold: 2
Unhealthy threshold: 3
Timeout: 5 seconds

The health endpoint itself is minimal. It checks that the process is alive and Redis is reachable:

app.get("/health", async (req, res) => {
  try {
    await redisClient.ping();
    res.status(200).json({ status: "ok" });
  } catch {
    res.status(503).json({ status: "redis_unavailable" });
  }
});

If Redis is unreachable, the task reports unhealthy and the ALB stops routing to it. This prevents a degraded container from receiving traffic it can't handle properly.

Structured Logging: The Feature That Changed Incident Response

The resilience engineering above handles infrastructure failures. But the single highest-leverage change we made to incident response wasn't a failover strategy. It was structured logging.

Before this redesign, debugging a Socket.IO incident looked like this:

"Users can't bid." → SSH into the server → grep through unstructured log output → manually correlate connection IDs → realise the logs don't capture the event that actually matters → escalate

Average time to root cause identification: 45 minutes to several hours.

We added JSON-structured logging across every significant socket lifecycle event:

const log = (event: string, socketId: string, data: Record<string, unknown> = {}) => {
  console.log(JSON.stringify({
    timestamp: new Date().toISOString(),
    event,
    socketId,
    containerId: process.env.HOSTNAME, // ECS task ID
    ...data,
  }));
};

io.on("connection", (socket) => {
  log("socket.connect", socket.id, {
    userId: socket.handshake.auth.userId,
    clientIp: socket.handshake.address,
  });

  socket.on("auction:join", ({ auctionId }) => {
    log("auction.join", socket.id, { auctionId });
  });

  socket.on("bid:place", ({ auctionId, amount }) => {
    log("bid.place", socket.id, { auctionId, amount });
  });

  socket.on("disconnect", (reason) => {
    log("socket.disconnect", socket.id, { reason });
  });

  socket.on("error", (err: Error) => {
    log("socket.error", socket.id, { error: err.message, stack: err.stack });
  });
});

These logs feed into CloudWatch Logs with automatic indexing. A query that previously required SSH and grep now runs in the CloudWatch Logs Insights query editor:

fields @timestamp, event, socketId, auctionId, reason
| filter event in ["socket.disconnect", "socket.error"]
| filter auctionId = "AUCTION_ID_HERE"
| sort @timestamp desc
| limit 100

The containerId field is particularly useful in a multi-container deployment, it immediately tells you which Fargate task a connection was on, which is essential for correlating connection failures with specific task health events.

Average time to root cause identification after: under 10 minutes.

That's not a small improvement. That's the difference between an incident that disrupts an auction and one that gets resolved before anyone notices.

Next in the series: The Decisions, Near-Disasters, and 35% Cost Savings from Rebuilding a Live Bidding Engine →

Previous: Connecting a Fleet: Redis Pub/Sub and Socket.IO Scaling ←