Wednesday
Oct222008
EVE Online Architecture
Wednesday, October 22, 2008 at 4:15AM Sorry, the content for this post apparently did not make the transition from the old HighScalability website, it's all messed up, but there's still a some useful content...
EVE Online is "The World's Largest Game Universe", a massively multiplayer online game (MMO) made by CCP. EVE Online's Architecture is unusual for a MMOG because it doesn't divide the player load among different servers or shards. Instead, the same cluster handles the entire EVE universe. It is an interesting to compare this with the Architecture of the Second Life Grid. How do they manage to scale?
Information Sources
- EVE Insider Dev Blog
- EVE Online FAQ
- Massively - Eve Evolved: Eve Online's Server Model and its discussion on Slashdot
- EVE Online Forums
- Massively Multiplayer Game Development 2
Platform
- Stackless Python used for both server and client game logic. It allows programmers to reap the benefits of thread-based programming without the performance and complexity problems associated with conventional threads.
- SQL Server
- Blade servers with SSDs for high IOPS
- Plans to use Infiniband interconnects for low latency networking
- Founded in 1997
- ~300K active users
- Up to 40K concurrent users
- Battles involving hundreds of ships
- 250M transactions per day
- Proxy Blades - These are the public facing segment of the EVE Cluster - they are responsible for taking player connections and establishing player communication within the rest of the cluster.
- SOL Blades - These are the workhorses of Tranquility. The cluster is divided across 90 - 100 SOL blades which run 2 nodes each. A node is the primarily CPU intensive EVE server process running on one core. There are some SOL blades dedicated to one busy solar systems such as Jita, Motsu and Saila.
- Database Cluster - This is the persistence layer of EVE Online. The running nodes interact heavily with the Database, and of course pretty much everything to do with the game lives here. Thanks to Solid-state drives, the database is able to keep up with the enormous I/O load that Tranquility generates.
- With innovative ideas MMO games can scale up to the hundreds of players in the same battle.
- SSDs will in fact bridge the gap huge performance gap between the memory and disks to some extent.
- Low latency Infiniband network interconnect will enable larger clusters.
Lessons Learned
- There are many interesting facts about the architecture of the EVE Online MMOG such as the use of Stackless Python and SSDs.
- Check out the information sources for detailed insights to the development and operation of the EVE Online game.
Reader Comments (10)
No, no, a thousand times no. If you want an experience that is remotely playable for the majority of people involved, we're looking at battles of 400 or 500 as an absolute maximum. Everything above that is unplayable and potentially node/game-crashing. I do not contest any of the other numbers though, all in all they have a monster of a cluster running the Eve-O universe.
I'm calling BS - we'd routinely crash the node with 60 - 100 ship battles.
Thank you for the feedback. I have updated the entry.
They've updated recently and they've managed to get a system to hold at a thousand ships, of course this doesn't mean that battles will be possible on this scale, it merely means that the system in question, we'll say Jita, will not crash. But 400-500 ship battles may not be so far fetched as we may once have believed. With the massive updates they've been doing at CCP it would not surprise me to see such battles taking place on a regular basis. Of course I won't be participating in them because I am a coward and I rather enjoy having my HIC in one piece.
Fly Safe.
yes and no..
I am the CEO of a Goonswarm corporation and take part in battles with over 1000 players in local most weeks. Since CCP launched "Stackless IO" as well as upgrading to 64-bit architecture on the serverside we have noticed serious improvements in server-generated lag. If we warn CCP 24 hrs in advance of a node being put under stress, they will reinforce that node for us and 99% of the time that fight will be more than playable.
You will also notice that you can undock in jita without waiting 10 mins to load grid, which wasn't the case a few months ago.
Even the highest end graphics card/CPU combinations will struggle with displaying and keeping track of thousands of individual objects on screen (ships, missiles, wrecks, capsules, fighters & drones), so large battles these days generally rely on the power of your PC as opposed to server responsiveness.
I can see how the server architecture could be applied to a first person game but the design challenges make it incredibly difficult to get right. It's not just a matter of a company buying the designs or engine from CCP, it would need to be completely gutted and redesigned for a first person perspective game. The games industry's major investors seem unwilling to risk their money on big changes like that, unfortunately, but it would make one hell of a game if someone managed it.
http://www.mustuniversity.com/Schools-Majors/Applied-Arts/Architecture.html">online architecture degree
Even the highest end graphics card/CPU combinations will struggle with displaying and keeping track of thousands of individual objects on screen (ships, missiles, wrecks, capsules, fighters & drones), so large battles these days generally rely on the power of your PC as opposed to server responsiveness. http://www.carrentaladvice.net/">car rental
I think anything over 500 ships is overkill anyway. I'm quite comfortable with the current limit as I tend to steer clear of the large scale smirmishes in any event.
Dave | http://www.pilkingtonselfcleaningglass.co.uk/where-to-use/glass-doors/index.html ">Glass Doors
If we warn CCP 24 hrs in advance of a node being put under stress, they will reinforce that node for us and 99% of the time that fight will be more than playable.http://www.livescores.com.sg/">Live Scores
Python is extremely slow... it's one of the slowest languages. It can be like 10X slower than C++. Some of the high-scale scientific libraries written in python actually use C++ under the hood for the math libraries, but I can't imagine that's the case here. As a hobbyist developer I still don't see why python could be used for such a large undertaking like this, especially since coding (IMO) seems like hell compared to more conventional languages.
Secondly, while not being completely educated on the matter, I wonder how they handle persistence across nodes...
To me it seems like you basically can't even do basic sanity checks for player input without first reading the original position from the database. You could assume their position for a single frame based on their position and last input, but then it's still being written to the persistent database on a node somewhere across the server rack meanwhile they're crossing a node line in-game and suddenly you're out of sync on their new node. They could probably even abuse that mis-sync with their cooldowns by repeatedly hopping over node lines. I'm assuming there's some sort of loading screen for each zone, in that case, which would probably fix this. Still, though, I wish there was a way to do it more smoothly so it could be integrated in game worlds more closely-knit than separate solar systems.. I'm working on something like this right now. if it was buttery smooth, then you could theoretically increase the resolution of your nodes until you CAN handle 2000 players at once, merely by scaling horizontally rather than pulling out your massive dedicated server every time like they use for Jita, but the cross-node communication would be insane, and I'm still trying to figure this out to avoid IO problems on the database