Did you ever stop to think that the databases powering your favorite online games might be, at their core, just elaborate lies? It’s a chilling thought, especially when you consider the colossal engineering effort that goes into making millions of concurrent players feel like they’re interacting in real-time. This isn’t just about slapping more servers online; it’s about wrestling with the fundamental nature of state, concurrency, and the ghosts in the machine that lurk within the database layer.
The folks behind what they call the “Treasure Hunt Engine” found this out the hard way. What started as a relatively straightforward Elixir cluster – nine beefy bare-metal nodes, 256GB RAM a pop, 64 cores each, with Redis for the quick stuff and PostgreSQL for the serious state – quickly devolved into a cascading failure scenario. Their initial architecture, a seemingly sensible loop of player actions hitting Redis, broadcasting to live clients via WebSockets, and then snapshotting to PostgreSQL every five seconds, held up admirably for a while.
Then the player count nudged past 150,000. Suddenly, PostgreSQL replication lag wasn’t a minor annoyance; it was an 8-second chasm. Adding more replicas didn’t fix it; it just shoved the problem downstream, making Redis replication the new bottleneck. Their PostgreSQL read replicas, meant to offload the primary, buckled under the weight of N+1 queries spewed out by the ORM, bringing a 32-core replica to its knees. Even Veltrix’s much-vaunted horizontal sharding introduced its own venom: a 20ms+ tail latency that bloomed as they split PostgreSQL into forty separate databases. Scaling wasn’t working; it was just picking a new way to fail.
And the Elixir nodes? At 210,000 players, they started screaming about beam.smp memory pressure. The culprit? A single process: the stateful GenServer responsible for each hunt. Picture this: a single hunt instance potentially holding a million players, with the GenServer keeping the entire player set in a map in its memory. When 3,000 concurrent hunts fired up, each with an average of 70 players, that GenServer memory ballooned to 12GB per node. They weren’t solving scalability; they were engaged in a desperate, losing battle with the garbage collector.
Their first instinct, naturally, was vertical scaling. Double the RAM to 512GB, cram in more cores. It offered a meager 12% reduction in GC pauses, hardly a victory. The hunt coordinator dashboard still froze solid when Redis flirted with 100,000 concurrent connections. The Redis instance itself, gorging on 320GB of RAM, was still single-threaded for persistence. Enabling AOF every second to try and shore up durability? That doubled disk I/O and introduced fsync stalls that brought the whole thing crashing down under load.
Next, they punted to Redis Cluster, sixteen shards deep. It handled the raw volume, sure, but their Phoenix app now had to fan out sixteen connections for every broadcast. The p99 latency? It leaped from a respectable 45ms to a cringe-worthy 180ms. And the kicker? If one shard decided to take a nap, the entire hunt coordinator UI would stutter. A cascading latency disaster born from distributed complexity.
Kafka entered the fray as a fan-out layer. Hunts were sharded across 200 partitions, with a consumer service, penned in Go, pushing updates to WebSocket clients. It eased the Redis load, but the Go consumers became memory black holes. Each one hoarded a 500MB buffer per hunt. Three thousand active hunts meant 1.5 terabytes of RAM just for these buffers. OOM errors again, this time on the consumer pods.
Finally, they gambled on PostgreSQL Citus. They sharded hunt tables by hunt_id, but the ORM’s love affair with joins turned into a full-blown tragedy. A single hunt coordinator view that used to zip by in 12ms now crawled for 2.3 seconds. Why? The Citus planner threw up its hands, admitting it couldn’t push the query down. It pulled all rows into a single worker and then performed the join in memory. This glorious experiment lasted a mere 72 hours before the rollback.
It’s a classic tale: each attempted fix solved one symptom while creating a new, often more virulent, disease. The documentation, as it so often does, failed to mention that scaling this particular beast meant trading raw latency for memory bloat, or sacrificing durability for fan-out complexity. They were, as they candidly admit, optimizing for the wrong axis. The real insight? Stop trying to scale a stateful monolith. Redesign the boundaries.
From Monolith to Microservices: The Rebuild
So, they blew up the monolith. Utterly. The engine was reborn as three distinct bounded contexts, each designed to do one thing and do it exceptionally well.
The Hunt State Service (HSS) is now a stateless Go beast. It’s liberated from holding any hunt state. Its job: ingest player actions via gRPC, validate them against hunt rules stored in PostgreSQL, and then spew immutable events to Kafka. Each event is a lean Protobuf package: hunt_id, player_id, action, version vector. Pure computation. Thirty replicas, humming behind an NLB, chugging through 46,000 events per second with p99 latencies under 20ms.
Then there’s the Hunt State Store (HSS) – a sharded PostgreSQL cluster using Citus, but with a crucial difference. ORM joins are dead. Every read and write is a simple, singular hunt_id lookup. Pre-calculated hunt scoreboards? They’re cached in Redis, but only as read-through caches. A miss? Recompute from the event log, not some ponderous ORM query. Sharding by hunt_id modulo 256 ensures even distribution, keeping each shard comfortably under 200GB, with a hot standby ready in the same AZ.
Finally, the Event Fan-out Service (EFS), another Go marvel. It subscribes to Kafka’s hunt.events topic and slaps updates onto WebSocket clients via NATS JetStream. NATS JetStream itself acts as a 24-hour event log, stored in memory-mapped files. Reconnect? EFS streams events from NATS, no state recomputation needed. Forty replicas of EFS handle 35,000 connections each. NATS JetStream serves up fan-out at p99 under 15ms, even with 300,000 clients reconnecting simultaneously.
The real architectural pivot, the one that seems to have unlocked this whole puzzle, was treating the hunt itself as an immutable event stream. This isn’t just a database strategy; it’s a philosophical shift. Instead of trying to mutate complex in-memory states that cascade failures, they’re now dealing with append-only logs. It’s a pattern we’ve seen bloom in domains like finance and IoT, where strict ordering and historical accuracy are paramount.
This shift mirrors, in a peculiar way, the evolution of version control systems. Git, for example, doesn’t let you undo in the traditional sense; it lets you create new commits that negate previous ones. The event stream approach is similar: you don’t change a player’s score; you record the action that results in a new score. This immutability drastically simplifies reasoning about state, makes debugging less of a dark art, and inherently builds in a strong audit trail.
Their journey is a stark reminder that sometimes, the most advanced scaling solution isn’t about faster hardware or more clever caching. It’s about fundamentally rethinking the boundaries of your services and embracing architectural patterns that treat data not as a mutable object to be constantly tweaked, but as a historical record of events. The lesson learned here isn’t just about databases; it’s about understanding the core nature of the problem you’re trying to solve.
What Went Wrong With the Old Architecture?
Their initial system, while elegant for small player counts, suffered from several critical flaws as it scaled:
- Stateful Monolith Bottlenecks: Core services (GenServers) held too much state (player sets in memory maps), leading to massive memory consumption and GC pressure under load.
- Database Churn and Lag: PostgreSQL struggled with replication lag, while read replicas were crippled by ORM-generated N+1 queries. Sharding introduced significant tail latency.
- Connection and Fan-out Overhead: Redis hit connection limits, and later, Redis Cluster introduced high fan-out latency. Kafka required massive buffering, leading to memory exhaustion.
- ORM Query Complexity: Joins across sharded databases (even with Citus) proved prohibitively slow and inefficient, often failing to push down operations as intended.
The New Architecture: Bounded Contexts and Event Streams
The rebuild hinges on three core components:
- Hunt State Service (HSS): Stateless computation, validating actions and emitting immutable events to Kafka.
- Hunt State Store (HSS): Sharded PostgreSQL for authoritative state, optimized for single
hunt_idlookups, with Redis for read-through caching. - Event Fan-out Service (EFS): Consumes Kafka events and pushes updates to clients via NATS JetStream, leveraging its persistent event logging for reconnections.
The key decision was treating the hunt as an immutable event stream, a departure from mutable state management that dramatically simplified scaling and reasoning about the system’s behavior.
Why Does This Matter for Developers?
This story is a potent cautionary tale for any developer working on distributed systems, especially those involving real-time user interactions. The temptation to shove more RAM into a problem or shard a monolithic database is ever-present. However, the “Treasure Hunt Engine” saga underscores that true scalability often lies in architectural decomposition and embracing patterns like event sourcing. Understanding when a system has become a “stateful monolith” – a single, tightly coupled unit where scaling one part exacerbates problems elsewhere – is a critical skill. The shift to stateless services, immutable event streams, and carefully chosen persistence layers demonstrates that the ‘how’ and ‘why’ of data flow and state management are far more impactful than sheer hardware power when pushing the limits of concurrent user loads. It’s a proof to the fact that sometimes, the most effective solution is not to optimize the existing structure, but to dismantle it and rebuild with fundamentally different principles.
What Can We Learn from This Database Nightmare?
Beyond the technical specifics, this narrative offers profound lessons:
- The Illusion of Scalability: Simply adding resources to a flawed architecture doesn’t create scalability; it just delays the inevitable, often making the eventual collapse more dramatic.
- Beware of ORMs in Distributed Systems: ORMs, while convenient for development, can become performance killers in sharded, distributed environments due to their abstraction over direct, optimized queries.
- Embrace Event-Driven Architecture: Treating state as an immutable stream of events is a powerful paradigm for building resilient and scalable systems, offering better debugging, auditing, and concurrency management.
- Rethink Boundaries: The most significant scaling breakthroughs often come from correctly identifying and enforcing service boundaries, moving away from monolithic designs towards well-defined, loosely coupled components.
🧬 Related Insights
- Read more: AI Psychosis Grips Developers—Gallup’s Gen Z Rage Signals What’s Next
- Read more: HarmonyOS @State Blind to Singleton Tweaks
Frequently Asked Questions
What was the main problem with the original treasure hunt engine’s database? The original engine’s databases, particularly PostgreSQL and Redis, couldn’t keep up with the increasing player count. This led to replication lag, connection bottlenecks, and ORM queries crippling read replicas, while sharding introduced unacceptable latency.
How did the new architecture solve the scaling issues? The new architecture decomposed the engine into stateless services and a sharded, optimized data store. By treating hunts as immutable event streams and eliminating complex ORM joins, the system achieved significantly higher throughput and lower latency.
Will this architectural shift impact game performance for players? Yes, significantly. The new architecture is designed to drastically reduce lag and improve responsiveness, ensuring a smoother and more reliable experience for a much larger player base.
