Introduction

Every system design decision is a trade-off. There is no universally correct architecture-only contextually appropriate ones. This post covers the core trade-offs that appear repeatedly in production systems and the reasoning behind each choice.

1. Consistency vs Availability (CAP Theorem)

The CAP theorem states that a distributed system can guarantee at most two of three properties: Consistency, Availability, and Partition tolerance.

Since network partitions are a physical reality, the real choice is:

CP system: Return an error or wait when a partition occurs (consistent but potentially unavailable) AP system: Return potentially stale data (available but not always consistent)

When to choose CP

• Financial transactions
• Inventory systems where overselling is unacceptable
• Authentication systems

When to choose AP

• Social feeds (stale data is acceptable)
• Product catalog reads
• Analytics dashboards

Rule: Default to AP for read-heavy systems. Use CP only where business logic demands it.

2. Monolith vs Microservices

This is frequently framed as a technology decision. It is not-it is an organizational and operational decision.

Rule: Start with a modular monolith. Extract services when a module has clear ownership, independent scaling needs, and a dedicated team.

Premature microservices are an operational liability.

3. Synchronous vs Asynchronous Communication

Synchronous (REST/gRPC): The caller waits for a response. Asynchronous (queues/events): The caller fires and the system processes independently.

Use synchronous when

• The caller needs an immediate result (e.g., authentication check, payment authorization)
• The operation is fast and reliable

Use asynchronous when

• Processing time is variable or long (e.g., image resizing, email delivery)
• You want to decouple producers from consumers
• The operation can be retried without side effects

Trade-off: Async systems are more resilient to downstream failures but add operational complexity (dead letter queues, idempotency, observability).

4. Caching Strategy Trade-offs

Rule: Use read-through with TTL for most read-heavy workloads. Avoid write-behind unless you have strong cache durability guarantees.

5. SQL vs NoSQL

Not a technology preference-a data model and access pattern decision.

Use relational (PostgreSQL, MySQL) when:

• Data has complex relationships and joins
• ACID transactions are required
• Schema is relatively stable

Use document/NoSQL (DynamoDB, MongoDB) when:

• Access patterns are known and simple (single-entity reads/writes)
• Horizontal scaling is a priority
• Schema flexibility is needed

Rule: PostgreSQL handles most use cases. Reach for DynamoDB when you need single-digit millisecond reads at massive scale with predictable access patterns.

Insight

Good architecture is contextual, not absolute. The correct system design is the one that matches your:

1. Scale requirements (current and realistic near-term)
2. Team capacity to operate it
3. Business constraints (latency, consistency, cost)

Engineers who understand trade-offs ship systems that last. Engineers who chase patterns ship systems that collapse under real conditions.

→ Related: Scaling Backend Systems for High Concurrency → Related: Reducing AWS Costs by 40%