I mean it’s not a bank in that it doesn’t have all the audit requirements etc but it’s sorta a bank in that it’s keeping track of transactions and state for every player every time a mission completes, which is a transaction that they really don’t want to go wrong.
And at a guess that code is held together with spit and baling wire.
Napkin math: That centralized transactional account updates every time there’s an end mission payout and with 400k concurrent players at 4 players per mission, say average 20 minute length (some missions 2 min some 40) that’s thousands of transactions per minute.
Totally doable load for even a modest central server I’d think… Unless you’ve got optimization issues. Which I’d argue a central db that doesn’t shard out at scale is an optimization issue.
Rare’s Sea of Thieves has/had similar issues. They run all their game instance servers on Azure (from some deep divers looking at IPs the game communicates with) and can spin up/down as many as needed to handle as many players as needed (from some change logs a year or two ago).
That said, during times of especially high player counts like right after an update drops, there’s often issues when you initially launch the game and it tries to log you in. Gold payouts can also be delayed by a number of hours, but they’ve got some sort of queuing system in place so you don’t need to stay signed in until they show up. You can sign out and they can somehow apply while you aren’t playing.
I wonder what’s the scalability issue. I’ve never seen a system that can’t be fixed by throwing more horizontally scalable resources at it.
Given that when the servers are overloaded you don’t get rewards: They probably have a centralized database or some other centralization issue.
You can add to a cluster all you want but if you’re using a SQL monolith you’ll get locks.
If you only have 100k concurrent users you can probably get away with it.
At 450k they’re running into gridlock
Yeha, that’s what I can’t imagine. What part of their data architecture can’t be sharded?
user accounts? sessions? cache keys? profiles? graphical assets?
This isn’t a highly transactional bank with strong transactional guarantes.
Would be pretty cool if they explained the issue after fixing it.
I mean it’s not a bank in that it doesn’t have all the audit requirements etc but it’s sorta a bank in that it’s keeping track of transactions and state for every player every time a mission completes, which is a transaction that they really don’t want to go wrong.
And at a guess that code is held together with spit and baling wire.
Napkin math: That centralized transactional account updates every time there’s an end mission payout and with 400k concurrent players at 4 players per mission, say average 20 minute length (some missions 2 min some 40) that’s thousands of transactions per minute.
Totally doable load for even a modest central server I’d think… Unless you’ve got optimization issues. Which I’d argue a central db that doesn’t shard out at scale is an optimization issue.
Rare’s Sea of Thieves has/had similar issues. They run all their game instance servers on Azure (from some deep divers looking at IPs the game communicates with) and can spin up/down as many as needed to handle as many players as needed (from some change logs a year or two ago).
That said, during times of especially high player counts like right after an update drops, there’s often issues when you initially launch the game and it tries to log you in. Gold payouts can also be delayed by a number of hours, but they’ve got some sort of queuing system in place so you don’t need to stay signed in until they show up. You can sign out and they can somehow apply while you aren’t playing.