Scaling Backend Systems for High Concurrency
How production systems handle 100K+ concurrent users without degrading latency or reliability.
Problem
Modern backend systems face traffic patterns that can spike by 10×–100× within minutes. A poorly designed system fails under load; a well-designed one scales transparently. This post documents the architectural decisions behind handling 100K+ concurrent users while maintaining sub-200ms p99 latency.
System Constraints
- Throughput: 100K concurrent users, ~50K requests/second at peak
- Latency: p99 < 200ms end-to-end
- Availability: 99.95% uptime SLA
- Data consistency: Eventual consistency acceptable for read-heavy paths
Architecture
Client Requests
↓
[CDN / Edge Cache]
↓
[Load Balancer] ← health checks, connection draining
↓
[API Gateway] ← rate limiting, auth, routing
↓
[Service Mesh] ← circuit breakers, retries
┌──────────────┬──────────────┐
│ Service A │ Service B │
└──────────────┴──────────────┘
↓ ↓
[Redis Cache] [Message Queue]
↓ ↓
[Primary DB] [Worker Services]
Key Technical Decisions
1. Stateless Services
Every service instance is stateless. Session data lives in Redis. This enables horizontal scaling with zero coordination.
2. Read-Through Cache
All hot data paths use a read-through cache pattern with Redis:
request → cache hit? → return cached data
↓ miss
→ fetch from DB
→ write to cache (TTL: 60s)
→ return data
This reduced database load by ~80% for read-heavy endpoints.
3. Event-Driven Writes
High-frequency write operations are decoupled via a message queue (SQS). The API layer acknowledges immediately; workers process asynchronously. This keeps API response times consistent under write spikes.
4. Connection Pooling
Each service maintains a pool of 10–20 database connections per instance. Without pooling, connection overhead at scale becomes the bottleneck.
Trade-offs
| Decision | Benefit | Cost |
|---|---|---|
| Eventual consistency | High write throughput | Stale reads possible |
| Async writes via queue | Consistent API latency | Delayed data visibility |
| Redis caching | Reduced DB load | Cache invalidation complexity |
| Stateless services | Easy horizontal scaling | External session storage required |
Results
- 60% reduction in p99 latency after Redis caching layer introduced
- Stable under 3× peak load in load tests (Locust, 150K concurrent users)
- Zero downtime deploys via blue/green deployment on ECS
- Database connection count reduced by 70% after connection pool tuning
→ Related: Reducing AWS Costs by 40% → Related: System Design Trade-offs Every Engineer Should Understand