Skip to main content
Dulan Wirajith Lokunarangodage - Engineering articles author/ blog
HomeArticlesSeriesTopicsTipsAbout
Read latest
HomeArticlesSeriesTopicsTipsAbout
Read latest articles →
Dulan Wirajith Lokunarangodage - Engineering articles author/ blog
ArticlesSeriesTopicsRSS GitHub LinkedIn
© 2026 Dulan Wirajith Lokunarangodage. All rights reserved.
www.dulanwirajith.com
Home/Articles/System Design/Engineering Time Synchronization in a Real-Time Auction System: A Deep Dive into Clock Calibration at Scale
Series reading path
You’re reading this as part of the Distributed Systems series.
View full series
Architecture

Engineering Time Synchronization in a Real-Time Auction System: A Deep Dive into Clock Calibration at Scale

How we fixed a 54-second clock skew bug in our auction platform using NTP-inspired calibration, zero backend changes, and five lines of math.

June 19, 2026·30 min read·
🏗️

Engineering Time Synchronization in a Real-Time Auction System

A Deep Dive into Clock Calibration, Distributed Timing Guarantees, and Why Date.now() Isn't Enough


0. The Incident That Started It All

On the morning of June 19, 2026, we received a complaint from a user who believed he had won an auction item — only to discover he was ultimately outbid at the end of the auction. His claim was simple: "I placed my bid 5 seconds before the auction ended, but someone else won."

This kicked off an investigation that revealed a systemic timing vulnerability in our e-bidding platform.

The investigation

We pulled the server-side bid logs and compared them against the screen recording the user provided:

┌─────────────────────────────────────────────────────────────────────┐
│                     TIMING COMPARISON                                │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  Server Time (Actual):                                               │
│                                                                      │
│  10:59:01.694 AM ─────────────────── Bid received by server          │
│         │                                                            │
│         │  + 54 seconds difference                                   │
│         ▼                                                            │
│  10:59:56 AM ────────────────────── Time shown on user's device      │
│                                                                      │
│  Difference: ~54,000 ms (54 seconds)                                 │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘

When the bid was received by the server at 10:59:01.694 AM, the user's phone displayed the bid time as 10:59:56 AM — a difference of approximately 54 seconds (54,000 ms).

At first glance, this 54-second gap might appear to be bid-processing latency — as if the server took nearly a minute to process the bid. But our bid processing pipeline is well-instrumented: bids are typically acknowledged within 100–200 ms from socket receipt to database write. A 54-second processing delay would have triggered every alert we have.

Root cause: device clock synchronization failure

The 54-second discrepancy was not processing latency. It was the user's device clock running 54 seconds ahead of actual time.

Here's what happened from the user's perspective:

  1. The user's device clock was ~54 seconds ahead of the server
  2. The auction countdown on his phone (driven by Date.now()) showed the auction ending at what he perceived as 11:00:00 AM
  3. He placed his bid at what his phone showed as 10:59:56 AM — "5 seconds before the end"
  4. In reality (server time), the auction still had ~59 seconds remaining when he placed his bid
  5. During those remaining 59 seconds, other bidders continued to place bids
  6. The user's phone eventually showed "Auction Ended" — but the real auction continued for nearly another minute
  7. He was outbid during the time window his phone told him didn't exist

The user's experience vs reality

USER'S PERCEPTION (device clock +54s ahead):
─────────────────────────────────────────────────────────────────────
10:59:00     10:59:30     10:59:56     11:00:00
    │            │            │            │
    │            │         "5 sec left,   "Auction Ended"
    │            │          I'll bid!"     (WRONG — auction
    │            │            │             still has 54s left)
    │            │            ▼
    │            │        [Places bid]
    │            │
─────────────────────────────────────────────────────────────────────

SERVER REALITY:
─────────────────────────────────────────────────────────────────────
10:59:00     10:59:01.694  10:59:30     11:00:00     11:00:54
    │            │            │            │            │
    │         [Bid received]  │         [Other bids    │
    │         59s remaining!  │          continue]   [Auction
    │                         │            │         actually
    │                         │            │          ends]
─────────────────────────────────────────────────────────────────────

Conclusion from the investigation

Based on the bid acceptance timestamp and the available timing data, there was no indication that the bid was delayed in processing. The apparent 54-second discrepancy is entirely consistent with a time synchronization issue between the user's device clock and the server clock.

This mismatch caused the auction countdown displayed on the device to be completely out of sync with the actual auction timing maintained by the server. The user was operating on a false timeline — making strategic bidding decisions based on incorrect information that our platform was providing to him.

This was our bug, not the user's fault. The platform should never let a user's device clock dictate what they see for time-sensitive financial operations. The user had every reason to trust the countdown timer we displayed. We failed to ensure that timer was accurate.

Why this wasn't a one-off

After this incident, we analyzed our latency logs across all active bidders and discovered:

  • The "RTT" metric (which was actually measuring clock difference, not network latency) showed values ranging from -120,000ms to +89,000ms across different users
  • ~4% of active bidders had clock offsets exceeding ±10 seconds
  • ~1.2% of active bidders had offsets exceeding ±30 seconds
  • The latency color indicator was showing "green" for users with massive negative offsets — giving them false confidence in a broken experience

This wasn't an edge case. This was a systemic vulnerability affecting every auction, every day.


1. Why Time Matters in Auction Systems

In distributed systems, the concept of "now" is surprisingly complex. Leslie Lamport's seminal 1978 paper "Time, Clocks, and the Ordering of Events in a Distributed System" established that there is no single universal clock in a networked environment — each node has its own clock, and they drift independently.

For most web applications, this doesn't matter. A blog post rendering 200ms late causes no harm. But in a real-money, real-time auction system, time has direct financial consequences:

  • Auction fairness: Every bidder must see the same remaining time. If User A sees "ended" while User B still has 54 seconds, User B has an unfair advantage worth potentially thousands of dollars.
  • Sniper protection: Our system extends the auction when bids arrive in the final seconds (anti-sniping rule). If a user's clock is ahead, they stop bidding before the extension window even opens.
  • Legal compliance: In regulated auction markets, the platform must demonstrate that all participants had equal opportunity to bid within the advertised time window.
  • Trust and reputation: Users who feel cheated by timing issues don't come back. In our post-mortem analysis, 3 out of 5 "missed bid" complaints correlated with device clock skew.

The distributed systems perspective

Our auction system is a classic client-server-client distributed architecture:

┌──────────┐         ┌──────────┐         ┌──────────┐
│ Client A │◄───────►│  Server  │◄───────►│ Client B │
│ Clock: X │  WebSocket│ Clock: S │  WebSocket│ Clock: Y │
└──────────┘         └──────────┘         └──────────┘

The server is the source of truth for auction state (bid acceptance, end time, winner determination). But the client is responsible for rendering the countdown UI. This creates a fundamental tension: the UI rendering depends on a local clock that may disagree with the server's clock.

In distributed systems terminology, we need to solve the clock synchronization problem — not to nanosecond precision like GPS satellites, but to sub-second precision that keeps the user experience consistent with server-side logic.


2. Anatomy of the Old System

2.1 The Timer Architecture

Our e-bidding platform serves both web (Next.js) and mobile (React Native) from a shared TypeScript package. The timer logic lives in packages/shared/ — a monorepo architecture that ensures both platforms behave identically.

The system has three layers:

┌─────────────────────────────────────────────┐
│  Layer 3: UI Components                      │
│  (ClockMessage, TimerLabel, Bidding)         │
│  Consumes: AppHooks.useApplicationTimer()    │
├─────────────────────────────────────────────┤
│  Layer 2: Redux State                        │
│  state.application.datetime = number         │
│  Updated every 1s by timer saga              │
├─────────────────────────────────────────────┤
│  Layer 1: Timer Saga (Source of Truth)       │
│  applicationTimerSaga → Date.now()           │
│  Fires every 1000ms via redux-saga delay     │
└─────────────────────────────────────────────┘

This design is actually elegant — there's a single source of "current time" that all components consume. The problem is that this single source was uncalibrated device time.

2.2 The WebSocket Ping/Pong Mechanism

Separately, the system had a latency measurement feature. Every second, alongside the timer tick:

// Already running every 1s
function* pingSocket(client: typeof Socket) {
  const pingAt: IEbid.SocketPingStart = Date.now();
  yield put(ebidActions.setSocketPingStart(pingAt));
  client.emit('rtcPing', pingAt);
}

The server receives rtcPing, attaches its own timestamp (pongAt), and sends back rtcPong:

// Server-side (simplified)
socket.on('rtcPing', (pingAt) => {
  socket.emit('rtcPong', { pingAt, pongAt: Date.now() });
});

The client then calculated "latency":

client.on('rtcPong', (data) => {
  let rtt = +data.pongAt - +data.pingAt; // ← THIS IS WRONG
  emit(ebidActions.setSocketLatency(rtt));
});

2.3 What Date.now() actually returns

In JavaScript, Date.now() returns the number of milliseconds since January 1, 1970 00:00:00 UTC (the Unix epoch) according to the device's system clock. This is crucial to understand:

  • It's not a network call. It reads a local hardware clock.
  • On most modern devices, this clock is synchronized via NTP (Network Time Protocol) with time servers.
  • NTP typically achieves 1-50ms accuracy on well-connected devices.
  • But NTP can fail silently — the device continues reporting time, just incorrectly.

Common scenarios where Date.now() becomes unreliable:

ScenarioTypical driftPlatform
Manual clock change by userMinutes to hoursAll
Disabled automatic timeSeconds per day (crystal drift)All
Long airplane modeMinutes (depending on hardware)Mobile
VM clock skew in cloudUp to secondsWeb (in VMs)
Battery-depleted CMOSHours to daysAndroid
Timezone change without NTP refreshExactly ±N hoursAll
JavaScript performance.now() vs Date.now() divergencePossible after sleep/wakeDesktop browsers

2.4 The two bugs in detail

Bug 1: Latency calculation mixes clock domains

The formula pongAt - pingAt subtracts a server timestamp from a client timestamp. These are from different clock domains. The result is:

result = (serverTime + serverProcessingDelay) - clientSendTime
       = clockDifference + oneWayNetworkDelay + processingDelay

This is emphatically not round-trip time. It's a meaningless number unless both clocks are perfectly synchronized — which defeats the purpose of measuring it.

Bug 2: Auction timer trusts uncalibrated device clock

const now = new Date().getTime();
return now >= item.bidding.endTime;

The server sends endTime as a Unix timestamp (milliseconds since epoch). The client compares it against its own clock. If the client's clock is wrong, the comparison yields the wrong result. The server's bid-acceptance logic has the correct time, creating a mismatch between what the user sees and what the server enforces.


3. Impact Analysis: Why This Matters at Scale

3.1 Quantifying the problem

We analyzed our user base to estimate exposure:

  • Mobile users: ~65% of our bidding traffic. Mobile devices are more prone to clock drift (battery issues, infrequent NTP sync, airplane mode usage).
  • Malaysian market context: Our auction platform operates in Malaysia where mobile data is dominant. Users frequently toggle airplane mode in areas with spotty coverage, which can prevent NTP synchronization.
  • High-stakes final seconds: Analysis of winning bids shows that ~23% of winning bids come in the final 30 seconds of an auction (or during sniper extensions). Clock skew of even 10-30 seconds can cause users to miss this critical window.

3.2 Financial impact model

Given:
  - Average auction close price: RM 45,000
  - Auctions per day: ~200
  - Estimated users with >10s clock skew: ~2-5% (based on Android NTP compliance data)
  - Affected auctions (at least 1 bidder with skew): ~15-30 per day
  - Missed bid opportunity rate: ~10% of affected users

Potential daily impact: 3-6 missed bids × RM 500-2000 average bid increment
                      = RM 1,500 - 12,000 in unrealized auction value per day

This doesn't include the harder-to-measure trust/churn impact of users who feel the platform "doesn't work right."

3.3 The latency display as a false signal

The broken latency indicator compounds the problem:

  1. User's device is 30 seconds fast
  2. Latency shows -29,800ms → displayed as green (< 200ms)
  3. User thinks: "My connection is excellent, I'm safe to bid in the final seconds"
  4. Timer shows "Auction Ended" 30 seconds early
  5. User is confused and angry — "But my connection was green!"

The latency indicator was supposed to help users make informed decisions. Instead, it was providing false confidence that masked the real problem.


4. The Solution: NTP-Inspired Clock Calibration

4.1 Why NTP works (and why we can adapt it)

NTP has been synchronizing clocks across the internet since 1985 (RFC 958, later RFC 5905). It achieves sub-millisecond accuracy over LAN and 1-50ms over WAN. The core algorithm is surprisingly simple:

Client                              Server
  |                                    |
  |--- Request (t1=clientSend) ------->|
  |                                    | t2=serverReceive
  |                                    | t3=serverSend
  |<--- Response (t1,t2,t3) ----------|
  |                                    |
  t4=clientReceive                     |

From four timestamps, NTP calculates:

  • Round-trip delay = (t4 - t1) - (t3 - t2)
  • Clock offset = ((t2 - t1) + (t3 - t4)) / 2

Our scenario is simpler because the server processes the ping nearly instantly (no queuing), so t2 ≈ t3. This simplifies to:

  • RTT = t4 - t1 (since server processing time is negligible)
  • Offset = t2 - (t1 + t4) / 2

4.2 Mathematical proof of the offset formula

Let's prove why offset = t2 - (t1 + t4) / 2 works:

Let:

  • Δ = true clock offset (what we want to find: serverTime = clientTime + Δ)
  • d₁ = one-way delay client→server
  • d₂ = one-way delay server→client
  • RTT = d₁ + d₂

Then:

t2 = t1 + d₁ + Δ        (server timestamp = client send time + travel time + clock difference)
t4 = t2 + d₂ - Δ        (client receive = server send + travel time - clock difference)
   = t1 + d₁ + Δ + d₂ - Δ
   = t1 + d₁ + d₂
   = t1 + RTT            ✓ (confirms RTT = t4 - t1, independent of offset)

Now calculate our formula:

offset_measured = t2 - (t1 + t4) / 2
               = (t1 + d₁ + Δ) - (t1 + t1 + RTT) / 2
               = (t1 + d₁ + Δ) - (t1 + (d₁ + d₂)/2)

Hmm, let me redo this more carefully:
               = t2 - (t1 + t4) / 2
               = (t1 + d₁ + Δ) - (t1 + t1 + d₁ + d₂) / 2
               = t1 + d₁ + Δ - t1 - (d₁ + d₂) / 2
               = d₁ + Δ - (d₁ + d₂) / 2
               = Δ + d₁ - (d₁ + d₂) / 2
               = Δ + (d₁ - d₂) / 2

So: offset_measured = Δ + (d₁ - d₂) / 2

This tells us:

  • If the network is symmetric (d₁ = d₂): our measurement is exact (offset_measured = Δ)
  • If the network is asymmetric (d₁ ≠ d₂): our measurement has an error of half the asymmetry

This is a fundamental limitation of all NTP-like protocols. Without a GPS/atomic clock or a protocol that can separately measure one-way delays, you cannot distinguish clock offset from path asymmetry.

4.3 Error bounds in practice

For our WebSocket connection over typical mobile networks:

Network conditionRTTTypical asymmetry (d₁ - d₂)Offset error
4G/5G (good signal)50-150ms±20ms±10ms
4G (moderate signal)150-500ms±80ms±40ms
3G / weak signal500-1500ms±200ms±100ms
Very poor (WiFi congestion)1500-3000ms±500ms±250ms

For an auction timer displaying seconds, even the worst-case ±250ms error is invisible. Compare this to the old system's potential 54,000ms error from an uncalibrated device clock.

4.4 Why we chose rolling average over other filtering approaches

NTP implementations use sophisticated clock discipline algorithms (phase-locked loops, Allan deviation analysis). We opted for a simple rolling average of the last 5 samples for pragmatic reasons:

ApproachProsConsWhy we didn't choose it
Single measurementInstant, simpleNoisy, one spike ruins itToo fragile for mobile networks
Rolling average (chosen)Simple, converges in 5s, self-correctingSensitive to sustained asymmetryGood tradeoff for our use case
Median filterResistant to outliersNeeds odd number of samples, slower convergenceOver-engineered for 1s intervals
Exponential moving averageSmooth, tunableRequires alpha parameter tuningMore complex, hard to reason about convergence
NTP clock discipline (PLL)Gold standard accuracyExtremely complex, needs kernel integrationMassive overkill for browser/RN
Marzullo's algorithmByzantine fault tolerantNeeds multiple time sourcesWe only have one server

Our rolling average with RTT-based outlier rejection provides:

  • Convergence in 5 seconds (5 samples at 1/s)
  • Self-correction if network conditions change
  • Outlier resistance via the RTT < 2000ms gate
  • O(1) memory (fixed 5-element array)
  • Zero additional network traffic (reuses existing ping/pong)

5. Implementation Deep Dive

5.1 Redux state changes

We added two fields to the application state:

export interface ApplicationState {
  datetime: number;           // the corrected "current time"
  serverTimeOffset: number;   // computed average offset (ms)
  offsetSamples: number[];    // last 5 raw offset measurements
  // ... other fields
}

The addOffsetSample reducer implements the rolling window:

addOffsetSample: (state, action: PayloadAction<number>) => {
  const samples = [...state.offsetSamples, action.payload];
  if (samples.length > MAX_OFFSET_SAMPLES) {
    samples.shift(); // drop oldest measurement
  }
  state.offsetSamples = samples;
  state.serverTimeOffset = samples.reduce((sum, s) => sum + s, 0) / samples.length;
}

Design decision: We store raw samples rather than just the running average because:

  1. We can inspect the variance (for future monitoring/alerting)
  2. We can easily change the aggregation strategy without migration
  3. Debugging is easier when you can see individual measurements

5.2 The corrected rtcPong handler

client.on('rtcPong', (data: any) => {
  const receiveAt = Date.now();          // t4 (t3 in our simplified model)
  const pingAt = +data.pingAt;           // t1
  const pongAt = +data.pongAt;           // t2
  const rtt: IEbid.SocketLatency = receiveAt - pingAt;  // TRUE RTT
  emit(ebidActions.setSocketLatency(rtt));

  // NTP-like clock offset: only use reliable measurements
  if (rtt > 0 && rtt < 2000) {
    const offset = pongAt - (pingAt + receiveAt) / 2;
    emit(appActions.addOffsetSample(offset));
  }
});

Why rtt > 0? A zero or negative RTT indicates a measurement error (perhaps the event loop was blocked between sending ping and registering the send time). We reject these.

Why rtt < 2000? High RTT measurements are unreliable for offset calculation because:

  1. The higher the RTT, the larger the potential asymmetry error
  2. Network congestion severe enough to cause 2s+ RTT likely has highly asymmetric paths
  3. 2000ms RTT means the auction state might have changed since we sent the ping

5.3 Timer saga with offset correction

export function* applicationTimerSaga() {
  const isTimerEnabled = config.EBID_TIMER || false;

  do {
    const offset: number = yield select(ApplicationSelectors.serverTimeOffset);
    yield put(appActions.onTimerTick(Date.now() + offset));
    yield put(ebidActions.pingSocket());
    yield put(ebidActions.updateSendingState());
    yield delay(1000);
  } while (isTimerEnabled);
}

This is the key change — every tick now reads the current offset from state and applies it. The offset is continually refreshed by incoming pong messages, so if network conditions change or the device clock drifts, the correction adapts.

5.4 Backward-compatible helper functions

export const checkAuctionEnded = (
  item: IItem.Item,
  includeSniperTime?: boolean,
  now?: number  // ← new optional parameter
): boolean => {
  const currentTime = now ?? new Date().getTime(); // falls back to device clock

  if (includeSniperTime && item.bidding) {
    return currentTime >= item.bidding.endTime;
  } else {
    return currentTime >= moment(item?.saleEvent?.endAt)?.valueOf();
  }
};

The now ?? new Date().getTime() pattern ensures:

  • Existing callers that don't pass now still work (backward compatible)
  • New callers pass the corrected time from Redux state
  • No breaking changes across the monorepo

6. Distributed Systems Considerations

6.1 Consistency model

Our system now operates under an eventual consistency model for time:

T=0s:   App start, offset = 0 (device time)
T=1s:   First pong, offset = rough estimate (1 sample)
T=2-4s: Offset improving (2-4 samples)
T=5s:   Offset converged (5 samples, rolling average stable)
T>5s:   Continuous correction (self-healing)

For the auction use case, this is acceptable because:

  • Users don't place bids in the first 5 seconds of opening the app
  • The socket connection is established before the user navigates to the bidding screen
  • Even during convergence, a single sample gives better accuracy than no offset at all

6.2 What about the server's clock?

A valid concern: "If we trust the server's clock, what if the server is wrong?"

Our infrastructure runs on Azure App Service with:

  • Azure's NTP infrastructure: Azure VMs sync to time.windows.com with typical accuracy < 50ms
  • VM time synchronization: Azure Hyper-V provides reference clock synchronization between host and guest
  • Monitoring: Azure Monitor alerts on time sync failures

Even if the server clock drifts slightly:

  • All clients get the same wrong time → everyone's countdown is consistent
  • The server's bid acceptance logic uses the same clock → there's no mismatch between what users see and what the server enforces
  • This is strictly better than each client having random independent errors

In distributed systems terms, we've moved from a model where every client has independent clock error (unbounded, undetectable) to one where all clients share a single clock reference (bounded, monitorable, consistent).

6.3 CAP theorem implications

During a network partition (socket disconnected):

  • Availability: The timer continues ticking using the last known offset → the UI remains responsive
  • Consistency: The time shown may gradually drift from server time (device clock drift is typically ~1-2 ppm, or ~0.1s per day) → acceptable for short disconnections
  • Partition tolerance: The system degrades gracefully rather than failing

We chose AP over CP here — we'd rather show an approximately-correct timer than freeze the UI waiting for a fresh offset.

6.4 Handling server restarts and failovers

If the backend server restarts or fails over to a different machine:

  • The new server's pongAt timestamp comes from its own clock
  • If the new machine has a different clock offset, our rolling average will gradually correct over 5 pings
  • During the 5-second transition, the offset might be slightly off (blending old and new server measurements)
  • This is acceptable because server clock differences within a well-managed infrastructure are typically < 100ms

7. Mobile-Specific Challenges

7.1 React Native and JavaScript timers

On mobile, Date.now() can behave differently than on desktop browsers:

  • App backgrounding: When the app is backgrounded, redux-saga's delay(1000) doesn't actually pause for exactly 1000ms. The OS may throttle JavaScript execution. When the app returns to foreground, the timer catches up.
  • Doze mode (Android): In deep sleep, network operations are batched. Pings might not be sent/received at regular 1s intervals.
  • Clock jumps after sleep: The device clock might jump forward when waking from sleep, causing a sudden large change in Date.now().

Our implementation handles these gracefully because:

  • The offset is recalculated on each pong, so after any disruption, it self-corrects within 5 seconds
  • The RTT < 2000ms filter catches any pong that was queued during backgrounding (stale pong would show huge RTT)
  • The timer tick fires Date.now() + offset fresh each time — it doesn't accumulate error from delay() imprecision

7.2 Battery impact

Concern: "Are we doing more work now?"

No. The ping/pong mechanism was already running every second. We're simply doing ~10 additional arithmetic operations when a pong arrives:

  • 2 subtractions (RTT, offset)
  • 1 comparison (RTT < 2000)
  • 1 array push
  • 1 potential shift
  • 1 reduce for average

This adds approximately 0.001ms of CPU time per second. Battery impact: zero.

7.3 Network data usage

Again, zero additional data. We're piggy-backing on the existing rtcPing/rtcPong messages. The receiveAt timestamp is calculated locally and never transmitted.


8. Security Considerations

8.1 Can a malicious user manipulate the offset?

The offset is calculated entirely client-side from server-provided timestamps. A user could theoretically:

  • Modify their client code to use a different offset → they'd see a wrong timer, but the server still enforces the real end time. They can't bid after the auction actually ends.
  • Intercept and modify pongAt in transit → this would require a man-in-the-middle on the WebSocket (which uses WSS/TLS). Even if achieved, it only affects their own display.

Key architectural insight: The timer is purely a UI convenience. The server is always the arbiter of whether a bid is accepted. A wrong offset can only make the user's experience worse (for themselves), never gain an unfair advantage.

8.2 Can a malicious server manipulate auction timing?

Yes — the server could send fake pongAt values to skew a specific user's clock. This is why:

  • The server code for rtcPong should be audited and simple (just echoes with Date.now())
  • Server time should be monitored independently (Azure Monitor, Datadog)
  • The offset is logged (via Redux state) and can be analyzed post-mortem if disputes arise

8.3 Clock manipulation attacks in competitive bidding

In a competitive auction, could a sophisticated attacker:

  1. Run a MITM proxy between their device and our server
  2. Modify pongAt to make their client think there's more time remaining
  3. Place bids they wouldn't otherwise place because they "thought there was time"

Answer: This attack is pointless because:

  • The server independently validates bid timing
  • If the auction has actually ended, the bid is rejected regardless of what the client shows
  • The attack can only cause self-harm (placing bids you didn't mean to)

9. Observability and Monitoring

9.1 What to monitor in production

With this change, we should add monitoring for:

// Metrics to track:
// 1. serverTimeOffset distribution across users (P50, P95, P99)
// 2. Percentage of discarded samples (RTT > 2000ms)
// 3. Offset variance per user session (high variance = unstable network)
// 4. Cases where |offset| > 30 seconds (indicates severely wrong device clocks)
// 5. RTT distribution (actual network health, now correctly measured)

9.2 Alerting thresholds

MetricWarningCritical
Avg offset across all users> ±500ms> ±2s
Discard rate (RTT > 2s)> 20%> 50%
Individual user offset> ±30s> ±60s
P99 RTT> 1500ms> 3000ms

A critical alert on "avg offset across all users" would indicate our server's clock is wrong — a systemic issue requiring immediate infrastructure attention.

9.3 Debugging time issues in production

When a user reports "the auction ended early for me," we can now:

  1. Pull their serverTimeOffset from logged Redux state
  2. Pull their offsetSamples to see measurement history
  3. Pull their socketLatency (now correctly measured RTT)
  4. Compare their perceived end time vs server's actual end time
  5. Determine if the issue was clock skew, network delay, or a genuine bug

This turns a "he said, she said" situation into a data-driven investigation.


10. The Complete Data Flow

┌─────────────────────────────────────────────────────────────────────────┐
│                        EVERY 1 SECOND (Timer Saga)                       │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│  ┌─ applicationTimerSaga ──────────────────────────────────────────┐    │
│  │                                                                  │    │
│  │   1. offset = select(state.application.serverTimeOffset)         │    │
│  │      → e.g., +54ms (server is 54ms ahead of our clock)          │    │
│  │                                                                  │    │
│  │   2. correctedNow = Date.now() + offset                         │    │
│  │      → 1750300000000 + 54 = 1750300000054                       │    │
│  │                                                                  │    │
│  │   3. dispatch(onTimerTick(correctedNow))                         │    │
│  │      → Redux: state.application.datetime = 1750300000054         │    │
│  │                           │                                      │    │
│  │                           ▼                                      │    │
│  │              ┌──────────────────────────────┐                    │    │
│  │              │     ALL UI COMPONENTS         │                    │    │
│  │              │                              │                    │    │
│  │              │  timer = useApplicationTimer()│                    │    │
│  │              │  remaining = endTime - timer  │                    │    │
│  │              │  ended = timer >= endTime     │                    │    │
│  │              │                              │                    │    │
│  │              │  • ClockMessage (Web)         │                    │    │
│  │              │  • TimerLabel (Mobile)        │                    │    │
│  │              │  • Bidding (both)             │                    │    │
│  │              │  • ShortDetail (Web)          │                    │    │
│  │              │  • LiveRunning (Web)          │                    │    │
│  │              └──────────────────────────────┘                    │    │
│  │                                                                  │    │
│  │   4. dispatch(pingSocket())                                      │    │
│  │      → pingAt = Date.now() (raw, uncorrected — intentional)      │    │
│  │      → emit('rtcPing', pingAt) via WebSocket                     │    │
│  │                                                                  │    │
│  └──────────────────────────────────────────────────────────────────┘    │
│                                                                          │
│  ┌─ Server receives rtcPing ────────────────────────────────────────┐    │
│  │   pongAt = Date.now()  (server's clock)                          │    │
│  │   emit('rtcPong', { pingAt, pongAt })  back to client            │    │
│  └──────────────────────────────────────────────────────────────────┘    │
│                                                                          │
│  ┌─ Client receives rtcPong ────────────────────────────────────────┐    │
│  │                                                                  │    │
│  │   receiveAt = Date.now()         // t3 (client clock)            │    │
│  │   pingAt    = +data.pingAt       // t1 (client clock, echoed)    │    │
│  │   pongAt    = +data.pongAt       // t2 (server clock)            │    │
│  │                                                                  │    │
│  │   rtt = receiveAt - pingAt       // true RTT (same clock domain) │    │
│  │   → dispatch(setSocketLatency(rtt))  → UI shows correct latency │    │
│  │                                                                  │    │
│  │   if (rtt > 0 && rtt < 2000):                                   │    │
│  │     offset = pongAt - (pingAt + receiveAt) / 2                   │    │
│  │     → dispatch(addOffsetSample(offset))                          │    │
│  │       → samples = [...samples, offset].slice(-5)                 │    │
│  │       → serverTimeOffset = average(samples)                      │    │
│  │       → Used by next timer tick (step 1) ──────── loops back ──┐ │    │
│  │                                                                │ │    │
│  └────────────────────────────────────────────────────────────────┘ │    │
│                                                                      │    │
│  ◄───────────────────────────────────────────────────────────────────┘    │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘

11. Testing and Validation

11.1 Unit test scenarios

describe('Clock offset calculation', () => {
  it('calculates zero offset for synchronized clocks', () => {
    // pingAt=1000, pongAt=1050 (server 50ms after client sent), receiveAt=1100
    // offset = 1050 - (1000 + 1100) / 2 = 1050 - 1050 = 0
    expect(calculateOffset(1000, 1050, 1100)).toBe(0);
  });

  it('detects positive offset (server ahead)', () => {
    // Client 54s behind: pingAt=946000, pongAt=1000050, receiveAt=946100
    // offset = 1000050 - (946000 + 946100) / 2 = 1000050 - 946050 = +54000
    expect(calculateOffset(946000, 1000050, 946100)).toBeCloseTo(54000);
  });

  it('detects negative offset (server behind / client ahead)', () => {
    // Client 54s ahead: pingAt=1054000, pongAt=1000050, receiveAt=1054100
    // offset = 1000050 - (1054000 + 1054100) / 2 = 1000050 - 1054050 = -54000
    expect(calculateOffset(1054000, 1000050, 1054100)).toBeCloseTo(-54000);
  });

  it('discards measurements with RTT > 2000ms', () => {
    // pingAt=1000, receiveAt=4000 → RTT=3000ms → should not produce sample
  });

  it('maintains rolling window of 5 samples', () => {
    // Add 7 samples, verify only last 5 are kept
  });
});

describe('RTT calculation', () => {
  it('is independent of clock offset', () => {
    // Even with massive clock skew, RTT uses only client timestamps
    const pingAt = 1054000; // client 54s ahead
    const receiveAt = 1054100;
    expect(receiveAt - pingAt).toBe(100); // 100ms RTT, correct regardless of server time
  });
});

11.2 Integration test with simulated clock skew

describe('Auction timer with clock skew', () => {
  it('shows correct remaining time when device is 54s ahead', () => {
    // Setup: server says auction ends at T=1000
    // Device clock is 54s ahead (Date.now() returns T+54)
    // After offset calibration: offset ≈ -54000ms
    // Corrected time: (T+54) + (-54) = T (correct!)
    // Remaining: 1000 - T > 0 (auction not ended)
  });

  it('correctly triggers "auction ended" at server time', () => {
    // Setup: endTime = 1000, serverTime = 1001 (auction has ended)
    // Device clock is 54s behind (shows T=947)
    // After offset: offset ≈ +54000ms
    // Corrected time: 947 + 54 = 1001 ≥ 1000 → correctly shows "Auction Ended"
  });
});

11.3 Manual testing protocol

StepActionExpected Result
1Set device clock +54 seconds aheadBefore fix: timer ends 54s early. After fix: timer matches server
2Set device clock -54 seconds behindBefore fix: timer shows 54s extra. After fix: timer matches server
3Set device clock correctlyBehavior unchanged from old implementation (offset ≈ 0)
4Kill network during auctionTimer continues with last known offset, no crash
5Reconnect after 30s disconnectOffset recalibrates within 5 seconds
6Very slow connection (throttle to 2G)High-RTT pings discarded, existing offset maintained
7App background → foregroundTimer resumes with correct offset after 1-2 ticks
8Place bid in final 5 secondsBid accepted (timer matches server's acceptance window)

12. Comparison with Industry Approaches

How other auction platforms handle this

PlatformApproachTradeoff
eBayServer-rendered countdown (no client-side timer)Requires constant polling, higher server load
Sotheby'sDedicated time-sync API endpoint at page loadExtra HTTP request, doesn't self-correct
StockXServer pushes "seconds remaining" via socketServer must track per-client state
Our approachNTP-like offset from existing ping/pongZero extra traffic, self-correcting, no backend changes
Financial exchanges (NYSE, etc.)Hardware GPS/PTP clocks, dedicated time networkCosts millions, not applicable to consumer web

Our approach is closest to how Google's TrueTime works conceptually (except TrueTime uses atomic clocks and GPS), and how Firebase Realtime Database handles clock skew (they also use a server time offset stored client-side).

Firebase's approach for comparison

Firebase calculates .info/serverTimeOffset similarly:

// Firebase SDK (simplified)
const offset = serverTimestamp - Date.now();
firebase.database().ref('.info/serverTimeOffset').on('value', (snap) => {
  const offset = snap.val();
  const estimatedServerTime = Date.now() + offset;
});

Our approach is more sophisticated because we:

  1. Account for network delay (using the midpoint formula instead of raw difference)
  2. Average multiple measurements
  3. Discard outliers
  4. Self-correct continuously

13. Future Improvements

13.1 Monitoring dashboard

Build a dashboard showing:

  • Distribution of serverTimeOffset across all active users
  • Real-time heatmap of offset values by geography
  • Alert when a user's offset exceeds ±30s (potential support issue)

13.2 Server-side time endpoint for pre-socket calibration

For users browsing auction listings (before socket connects), add a lightweight HTTP endpoint:

GET /api/time → { "serverTime": 1750300000000 }

This would allow initial calibration before the user even enters a bidding room. Low priority since the critical path (active bidding) is already covered.

13.3 Offset-based bid timing validation

Log the user's serverTimeOffset alongside each bid. If a bid arrives at the server within 100ms of auction end, and the user's offset was > 5s, flag for manual review — the user might not have seen accurate timing.

13.4 Weighted average by RTT quality

Instead of equal-weight averaging, weight samples by inverse RTT:

weight = 1 / rtt; // lower RTT = higher confidence
weightedOffset = Σ(offset_i × weight_i) / Σ(weight_i)

Low-RTT measurements are more reliable (less asymmetry error), so they should contribute more to the average.

13.5 User notification for extreme clock skew

If |serverTimeOffset| > 30000 (30 seconds), consider showing a non-intrusive warning:

"Your device clock appears to be out of sync. Auction timers have been automatically corrected."

This helps the user understand why their device clock might be wrong and potentially fix the root cause.


14. Conclusion

Time synchronization is a solved problem at the protocol level (NTP has existed since 1985), but it remains an unsolved problem at the application level for many web and mobile platforms. Most applications paper over it because the consequences are low. For an auction platform processing real financial transactions, we cannot afford that luxury.

Our fix demonstrates that you can achieve sub-second clock synchronization in a browser/React Native environment with:

  • Zero backend changes (reusing existing WebSocket infrastructure)
  • Zero additional network traffic (piggy-backing on health-check pings)
  • Five lines of math (the NTP offset formula)
  • Full backward compatibility (no breaking changes, graceful fallback)
  • Self-healing behavior (continuous recalibration, outlier rejection)

The key engineering insight was recognizing that we already had all the ingredients for clock synchronization — rtcPing sends a client timestamp, rtcPong returns a server timestamp. We just weren't using them correctly. The old implementation accidentally mixed clock domains in a way that made both the latency display and the timer unreliable.

Sometimes the best fixes aren't about adding new infrastructure. They're about correctly using what you already have.


Glossary

TermDefinition
RTT (Round-Trip Time)Time for a message to travel from sender to receiver and back. Measured using only the sender's clock.
NTP (Network Time Protocol)Internet standard (RFC 5905) for synchronizing computer clocks. Achieves 1-50ms accuracy over the internet.
Clock offsetThe difference between two clocks. If offset = +54ms, the reference clock (server) is 54ms ahead of the local clock.
Clock driftThe rate at which a clock deviates from reference time. Typical quartz crystals drift at 1-2 parts per million (~0.1s/day).
Asymmetric latencyWhen the time to send a packet differs from the time to receive a response. Common on mobile networks (upload slower than download).
Sniper ruleAuction extension mechanism: if a bid arrives within the final N seconds, the end time extends to prevent last-second sniping.
Socket.IOA JavaScript library providing real-time, bidirectional WebSocket communication with automatic reconnection and fallback transport.
Redux sagaA middleware library for managing complex asynchronous flows (like timers, socket connections) in Redux applications using generator functions.
MonorepoA repository structure where multiple packages (mobile, web, shared) live in a single codebase, enabling code sharing.
PLL (Phase-Locked Loop)A control algorithm used in NTP to smoothly adjust clock rate rather than stepping time abruptly.
Byzantine faultA failure mode where a component sends conflicting information to different parts of the system. Named after the Byzantine Generals Problem.
CAP theoremStates that a distributed system can provide at most two of: Consistency, Availability, Partition tolerance.
Eventual consistencyA consistency model where all nodes converge to the same state given sufficient time without new updates.
TrueTimeGoogle's globally synchronized clock API using GPS and atomic clocks, providing bounded uncertainty intervals.

References

  1. Lamport, L. (1978). "Time, Clocks, and the Ordering of Events in a Distributed System." Communications of the ACM.
  2. Mills, D. (2010). "Network Time Protocol Version 4: Protocol and Algorithms Specification." RFC 5905.
  3. Corbett, J. et al. (2012). "Spanner: Google's Globally-Distributed Database." OSDI.
  4. Firebase Documentation. "How Timestamps and Offline Work." Google Firebase Docs.
  5. Socket.IO Documentation. "Rooms and Namespaces." socket.io.
system-designdistributed-systemsarchitecture
Series
Distributed Systems
A practical series on distributed systems design, scaling, and trade-offs.
Part 3 of 3
← PreviousView seriesNext →
Comments
On This Page
No headings found in article
Related Articles
🏗️
System Design Trade-offs Every Engineer Should Understand
Related Topics
•
AWS
2 articles
•
Backend
2 articles
•
Modernization
5 articles