Building a Real-Time Leaderboard That Doesn't Fall Over

TL;DR#

The question: how do you update a leaderboard in real time, for many concurrent players, without losing data or coupling every part of the system together?

The answer: split the system into independent services that never call each other directly — they only ever talk through a cache, a message queue, and a database. Each one can be restarted, scaled, or replaced without the others noticing.

Here's the full breakdown.

The Challenge#

Leaderboards look simple on the surface — store some scores, sort them, show the top N. In practice, one that has to work at real product scale has to answer several hard questions at once:

• Latency — when a score changes, how fast can we tell the player their new rank? Milliseconds matter; nobody wants to refresh to find out they moved up.
• Fan-out — one score update usually needs to trigger several independent things: merging into a global view, recording history for audit/analytics, and notifying anyone who just got overtaken. Should the service that accepted the write be responsible for all of that, synchronously, in the request path?
• Durability — what happens to a score update if a downstream consumer is down, restarting, or slow? Can we guarantee it's never silently lost?
• Scope — most real leaderboards aren't one flat list. Players expect regional rankings, a global view, and a friends-only view, often across daily/weekly/monthly periods, all at once.

Bolting all of this onto one service tends to produce a system where every feature is coupled to every other feature, and a slow or failing notification path can degrade the write path itself.

The Approach#

Instead of one service doing everything, the system is split into four independent processes that communicate only through shared infrastructure — a cache, a message broker, and a database — never through direct service-to-service calls:

This isn't an arbitrary split. Each service does exactly one job, and none of them know the others exist — they only know about the infrastructure between them.

How It Works#

1. A score update lands. A client posts a score delta for a user and region. The API writes it directly into a sorted-set structure in the cache layer — this is what makes rank lookups sub-millisecond, even under heavy write volume.

2. One event, fanned out three ways. The API publishes a single message to a fanout exchange on the message broker. The broker delivers an independent copy of that message to three separate queues — one per downstream consumer — with zero coupling between them. If the API had to call all three services directly and wait for each one, a single slow consumer would slow down every score update. With fanout, the write path is done the moment the event is published.

3. Three consumers do their jobs in parallel.

• The aggregator treats each message purely as an activity signal ("this region/period is live"). It always recomputes the global leaderboard from the cache on its own ticker (every few seconds), so it stays correct even if a message is dropped, delayed, or arrives out of order.
• The history writer inserts the event into a durable table. This table — not the queue, not the cache — is the actual source of truth. Message queues aren't a log; once a message is acknowledged, it's gone. The database is what survives.
• The notifier computes the player's new rank and, if it crossed a configurable threshold, publishes to a pub/sub channel. Notably, the notifier never touches a live connection to the client — it doesn't even know one exists.

4. The update reaches the player live. The API is the only process that owns live client connections. It subscribes to the pub/sub channel and forwards matching notifications straight to connected clients — no polling, no refresh.

flowchart TD
    Client["Game / Web Client"] -->|"POST /score/update"| API["API Service"]
    API -->|"instant write (ZINCRBY)"| Cache[("Sorted-Set Cache")]
    API -->|"publish 1 event"| Exchange{{"Fanout Exchange"}}
    Exchange -->|queue| Aggregator["Aggregator"]
    Exchange -->|queue| HistoryWriter["History Writer"]
    Exchange -->|queue| Notifier["Notifier"]
    Aggregator -->|"merge regional to global (~5s)"| Cache
    HistoryWriter -->|INSERT| DB[("Durable Event Log")]
    Notifier -->|"rank check + publish"| PubSub[("Pub/Sub Channel")]
    PubSub -->|subscribe| API
    API -.->|"live push"| Client

Data Model#

Three kinds of records carry the whole system:

• Users & relationships — identity and a lightweight social graph, which is what powers a friends-only leaderboard view.
• Score events — an append-only, replayable log of every score change, indexed by user and time period.
• Leaderboard snapshots — point-in-time top-N captures per scope (global / regional / friends), so "what did the leaderboard look like at time X" is always answerable after the fact.

The cache layer carries the hot path: one sorted set per region/period and one for global/period, plus a lightweight per-user score mirror so a friends-scope query can batch-fetch scores instead of hitting a sorted set once per friend.

API Surface#

What This Validated#

Running this system end to end, a few properties stood out directly:

• Independent failure — stopping the notifier mid-stream doesn't slow or block score writes, the aggregator, or history persistence. Restart it, and it just resumes consuming from its own queue.
• Correct under disorder — because the aggregator always recomputes from the cache rather than trusting message order, out-of-order or delayed events never produce a wrong global leaderboard.
• Real fan-out, visible in real time — one publish turns into three independent queue deliveries, ticking up in lockstep, observable live in the broker's own monitoring tools.
• Live client updates — a connected client receives a push the moment a score update crosses it into the top ranks, without a single poll.

Why This Matters Beyond a Demo#

The pattern here — decoupling through infrastructure instead of direct calls — is the same one a real multi-region production system would use. Nothing about the design changes if the aggregator, history writer, and notifier each became their own deployment in their own region tomorrow.

It applies anywhere "who's #1 right now" has to update in real time, at scale, without losing data: gaming leaderboards, learning-streak rankings, sales contest boards, fitness challenges, and similar live-ranking features.

Tech Stack#

Go · Redis (sorted sets + Pub/Sub) · RabbitMQ (fanout exchange) · MySQL · WebSockets

Key Takeaways#

1. Decouple through infrastructure, not direct calls. A message queue between services means a slow or failing consumer never blocks the write path.
2. Pick one source of truth, deliberately. A queue is a transport, not a log — durability has to live somewhere that's actually replayable.
3. Design consumers to tolerate disorder. Treating messages as "wake-up signals" rather than authoritative state makes a consumer correct by construction, not by careful sequencing.
4. Keep notification delivery ignorant of transport. The service that detects a change doesn't need to know anything about how it's delivered — that separation keeps both pieces simple.

Need a Leaderboard Built for Your Game or App?#

This is the exact architecture we reach for as a development agency whenever a client needs a live-ranking feature — a gaming leaderboard, a learning-app streak board, a sales contest board, or a fitness challenge ranking. If you're building a game or product that needs real-time rankings and don't want to find out the hard way that your leaderboard falls over under real traffic, this is a system Vaniworks builds as part of our web development work.

Book a free 30-minute call to talk through what your leaderboard needs to handle.