The PA Report - Planning for war: how the EVE Online servers deal with a 3,000 person battle

One of the largest battles in EVE Online history took place due to an accidental click, according to reports. The full details of why and how so many ships ended up battling each other is a fascinating read, but my question is what happens to the servers when something this crazy happens in the game. CCP, the creator of EVE Online, has detailed plans in place specifically for this sort of event.

“The customer service team (GMs) keep an eye out for gigantic fights like this,” CCP Veritas, one of the game’s developers, told the Report. “We’ve got a cluster status webpage that shows big red numbers when a node gets overloaded like it was by this fight, so it’s pretty easy to see what’s up.” It’s what happens next that is interesting.

The choices available

The most common response is to do nothing. Almost all battles create manageable loads on the servers, or they are over so quickly it doesn’t matter. Once it’s clear that something has to be done, however, the team has some options.

It’s hard to wrap your head around, but they sometimes move the in-game space itself. “We move other solar systems on the node away from the fight. This disconnects anyone in those systems temporarily, but spares them from the ongoing symptoms of being on an overloaded server,” Veritas explained. “It helps the fight system a little bit as well, especially if a reinforcement fleet is traveling through those other systems. This was done for the fight over the weekend, but is rare.”

They have an even larger weapon available to keep the servers running under extreme circumstances, but there are large trade-offs involved. “We’ve got a couple machines that are crazy-good hardware, well above what the rank and file of what the cluster is,” Veritas said. 

“This is the machine that systems get reinforced on when players request that. Unfortunately, the same thing above applies – anyone in the system when the move happens gets disconnected.  Because of this, it’s basically never done,” he continued. “In this case, the fight broke out because of a Titan put out of position by accident. Had we gone this route and moved the system, the Titan and his friends simply wouldn’t have logged back in, killing the fight. So, yeah, this just isn’t done.”

Past that, there’s very little they can do. The servers are always running at 100 percent, so it’s not a matter of throttling anything up. They do have a built-in mechanism for dealing with massive battles, however: They slow down time itself.

Time dilation

This is old hat for EVE Online veterans, but it’s an interesting way to deal with heavy player load and lag, while still keeping the game moving. Once server load reaches a certain point, the game automatically slows down time by certain increments to deal with the strain. Time was running at 10% speed during this 3,000-person battle, which is the maximum amount of time dilation possible. We’ve embedded a video to show what time dilation looks like in practice.

“It’s basically a very graceful way of handling ‘lag’ produced in these situations where other games would have their servers melt,” Veritas told us. “It actually slows down time in the system to make sure the server calls and responses are both carried through and done in the correct order. In this case, as people jumped in it slowed down gradually until it hit the cap at 10%, meaning a pretty slow experience, but one that is still meaningful from a game play and ‘tactical’ perspective.”

So instead of the servers breaking down, time dilation allows for even massive battles to play out, although at a slower pace. The result is that encounters like this, which incorporated thousands of players piloting high-value ships, are able to take place.

“For the HoneyBadgers, losses sustained included six Dreadnoughts, 11 Carriers, and one Supercarrier. The Clusters suffered far worse: 44 Dreadnoughts, 29 Carriers, five Supercarriers, and three Titans,” PC Gamer reported. “Ultimately, GoonSwarm leader The Mittani called the Battle of Asakai ‘a complete rout’ for the powerful Something Awful alliance. Estimated ISK (EVE’s in-game currency) cost in damages are still being calculated, but early totals reach beyond 700 billion for both sides combined.”

That’s around $24,000, depending on the value of ISK at any given time, and with that much at stake it’s important that battles take place on stable servers. Luckily, CCP has the tools to allow this sort of thing to happen. EVE Online continues to be a fascinating game.