open All Channels
seplocked EVE Information Portal
blankseplocked New Dev Blog: Fixing Lag: Drakes of Destiny (Part 1)
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: 1 [2] 3 4

Author Topic

Jason Edwards
Internet Tough Guy
Spreadsheets Online
Posted - 2010.12.10 17:42:00 - [31]
 

Originally by: Aika Achura
Quick Fix Solution?

1. Reduce N launcher mount points on ships to 1.
2. Add missile damage multiplier of N to affected ship hulls.
3. Reduce turret mount points and high slots as needed for balance.

It would probably be a bit boring for the pilot though.

That's what stacked guns did.

Durnin Stormbrow
Posted - 2010.12.10 17:44:00 - [32]
 

Edited by: Durnin Stormbrow on 10/12/2010 17:51:25
Originally by: Silen Boon
While I applaud any effort to reduce lag, my perception is that fleets will always increase to the size to the point where lag is used to tactic, where the lag favours one fleet over another.
Is there any evidence of the strategic use of lag and can anything be done about it?


I agree that given the current mechanics, fleet size is always going to chase the cap. Today, there is nothing that 500 players can do that 5,000 couldn't do faster if the node could handle it.

The advantage of lag is simple: 'We're already here and have the grid loaded. Since the node is lagging, you can't come here and expect to live." Defenders that are expecting to loose a fight in a bad way can try to overload the node to the point that it dies. They may still loose in the chaos that follows, but that chaos is a better opportunity than the coordinated ass beating they were already taking.

As for what to do about it, I think the answer there has to be more, smaller, distributed objectives and the concept of overkill.
From a game performance perspective, having 2500 invaders and 2500 defenders fighting on 50 separate grids scales MUCH better than 5,000 combatants on 1 grid.
By overkill, I mean that if an objective can be taken down with 10 dreads in one siege cycle, another 40 would be better spent on other objectives rather than dropping the 1st objective in 1/5th of the time and sitting idle for the rest of the siege cycle.

In terms of game mechanics:
If establishing sovereignty involved placing TCUs (Some new device, more like SBUs favoring the residents rather than the TCUs we have today) around the periphery of the system (not at POSes), and improving sovereignty involved many more TCUs, an invading fleet would have each of those TCUs as a separate objective.
If dreads return to being king of the hill for killing fixed targets, and the TCUs are small enough that using more than [x] dreds does not get the job done faster, then the invaders at each objective would be limited to [x] dreds + a few more for buffer and whatever support/defense fleet and reactionaries was allocated to them.
Defenders could choose to throw 1000 ships at defending a single objective, at the cost of ignoring every other objective being attacked.
Invading a well entrenched system could still take hours and involve 1000s of players, but the battles would be broken down into many skirmishes that are easier for the node to handle, rather than 1 massive battle.

As a bonus, smaller fleets would actually have objectives and 5-10 man cruisers roams might come to mean something if they can actually soften a system prior to an invasion.

Shasz
Angels of Anarchy
Posted - 2010.12.10 17:44:00 - [33]
 

Fantastic blog! I love that visual profiling tool too.

Thanks for sharing! I'm looking forward to part deux.

gfldex
Posted - 2010.12.10 17:59:00 - [34]
 

Originally by: Hawk TT

Destiny is (probably) the only C++ server-side code being used...So, no GIL, no Python.


I would bet on C instead of C++. And the block post shows that Destiny ain't the problem. It's the code that is called after the C part is done. As soon as you get a callback from the C part to the Python part (and those are the reason you want to have a nice language instead of typing your fingers bleedy in C), you get hit by the GIL again.

Megan Maynard
Minmatar
Navigators of the Abyss
Posted - 2010.12.10 18:10:00 - [35]
 

So you are saying we can blame Drakes for lag.

I'm totally ok with that.

PERMABAN MISSILE BOATS, THEY ARE RUINING OUR GAME EXPERIENCE!!!!!!!

Vincent Athena
Posted - 2010.12.10 18:32:00 - [36]
 

I do not think placing multiple objectives about a solar system will do much to lag. The entire solar system runs on one node, so 2000 people in system are processed by one node whether they are in the same grid or scattered.

Sure there will be a little improvement, some load comes from things that scale an N^2 of the objects in a grid. And player's fps would go up. In addition, many players would like to see the blob broken up. So multiple objectives would be a good move in any case.

But you know how sometimes you get a large amount of lag when there is no one around, or even in your solar system? Its a big fleet fight happening in a system that happens to be on the same node as you.

Scharde Alton
Aperture Ventures
Posted - 2010.12.10 18:33:00 - [37]
 

Full disclosure: I spent several years working on real-time game communication middleware over fixed and mobile internet.

You've probably already thought of this stuff, but I am an optimization nerd.

You're spending most of the server cycles doing 2 things:

1. "figuring out which clients need to be informed"

I suggest that you don't do this. Send the message to every client in the bubble and let them figure out whether or not it means a display update for them. This scales with the number of clients, and EVE is not a bandwidth hog (it's much more adversely affected by latency).

2. "Further investigation showed that this is mainly expensive due to the time required to serialise each update message."

This sounds to me like you're packing the same object over and over again into the packet buffer. If true, this can be significantly improved by serializing once and using scatter/gather I/O to let the NIC driver do that work by DMA instead of loading down the CPU with it.

Alternatively, this would be a ton of work but you could replace sending state objects with transactions.

Black Dranzer
Caldari
Posted - 2010.12.10 18:40:00 - [38]
 

Edited by: Black Dranzer on 10/12/2010 18:49:05
So wait. You're telling me the physics engine runs at a simulation speed of one FPS?

Okay, wow, this explains a lot. For one, it explains why "fly with your joystick" would be an absolutely terrible idea, even with astonishingly good interpolation/dead reckoning. Secondly, it explains why the "input delay" is so bad even given TCP/Server Authorization/Australia. Thirdly, it explains why missiles have a tendency to "double back" on their targets at the last second. Fourthly, it explains why everything seems to have a sort of "delayed but synced" feel to it. Fifthly, it explains why "warping in sync" actually happens in the frequency it does. Finally, it explains why pretty much nothing in Eve happens in a timescale of less than a second.

Thought off the top of my head: Do you think it's possible that the size of the timestep is partially responsible for slowing things down? I know it seems counterintuitive; In most situations increasing the step frequency would just bog things down further. But it doesn't seem impossible that a good portion of calculation time is burnt up in compensating for the "fuzzyness" of large timesteps. Alternatively, maybe it's in part the other way around; There might be computations that are done once a second that really don't have to be. Of course, variable sized timesteps for multiple interlocking things would be an enormous pain in the ass.

But I'll stop talking out of my ass now. Good blog! I very much look forward to the next one.

Edit: Wait, hang on. Hang on.

Please tell me you're not communicating the position of every missile to every player every second. Please tell me you're not doing that. You're not doing that, right? You are just sending the basic moving ship position data and then letting the client sort out the rest, right?

Lutz Major
Posted - 2010.12.10 18:48:00 - [39]
 

Originally by: Marcel Devereux
BeOS:: What's this?
Without googleing ... yes kids we have brains to memorize!

If it is the original BeOS, then it was introduced, what? 12-15 years ago. As - what else - Windows killer.

Back then it had already 64bit ability and was determined to do extreme multitasking. It performed better in Multimedia than any windows or linux back then.

I wonder if CCP really uses the OS from back then ...

Marcel Devereux
Aideron Robotics
Posted - 2010.12.10 18:54:00 - [40]
 

Edited by: Marcel Devereux on 10/12/2010 18:58:03
Originally by: Lutz Major
Originally by: Marcel Devereux
BeOS:: What's this?
Without googleing ... yes kids we have brains to memorize!

If it is the original BeOS, then it was introduced, what? 12-15 years ago. As - what else - Windows killer.

Back then it had already 64bit ability and was determined to do extreme multitasking. It performed better in Multimedia than any windows or linux back then.

I wonder if CCP really uses the OS from back then ...



I know what BeOS is (I actually worked at Be on BeOS and BeIA). I'm just curious to know what CCP named BeOS.

ElfeGER
Deep Core Mining Inc.
Posted - 2010.12.10 19:22:00 - [41]
 

congrats on finding a runtime profiler

Nareg Maxence
Gallente
Posted - 2010.12.10 19:33:00 - [42]
 

Very good blog. Massive props for the BTTF reference.

Gecko O'Bac
Deep Core Mining Inc.
Posted - 2010.12.10 19:38:00 - [43]
 

Originally by: ElfeGER
congrats on finding a runtime profiler


I guess your application to CCP is well on its way and you're about to become team lead for gridlock?

Damn smartasses.

That said: Tech **** blog. Me loves.

I'm just not sure I understood a thing... You said in the blog that "we can't do any asynchronous operations such as database queries or anything that might cause execution to yield." But... aren't asynchronous calls non blocking by default? I mean, that's how I usually use that term: I make an asynchronous request and go on doing my stuff... I guess you may want to avoid execution of call backs as well if your code is entirely single threaded... Perhaps that's what you meant?

Elsa Nietzsche
Posted - 2010.12.10 19:49:00 - [44]
 

Edited by: Elsa Nietzsche on 10/12/2010 19:49:47
how long do we have to wait for part 2?

gfldex
Posted - 2010.12.10 20:09:00 - [45]
 

Originally by: Elsa Nietzsche
Edited by: Elsa Nietzsche on 10/12/2010 19:49:47
how long do we have to wait for part 2?


Until it is written and signed of by a superior. What means neither of the two parties involved can provide any meaningful answer, because one depends on the other.

Epitrope
The Citadel Manufacturing and Trade Corporation
Posted - 2010.12.10 20:26:00 - [46]
 

Originally by: Black Dranzer
Thought off the top of my head: Do you think it's possible that the size of the timestep is partially responsible for slowing things down? I know it seems counterintuitive; In most situations increasing the step frequency would just bog things down further. But it doesn't seem impossible that a good portion of calculation time is burnt up in compensating for the "fuzzyness" of large timesteps. Alternatively, maybe it's in part the other way around; There might be computations that are done once a second that really don't have to be. Of course, variable sized timesteps for multiple interlocking things would be an enormous pain in the ass.


I actually came up with a similar idea a while ago, and I've always been curious about why EVE isn't set up this way. A node dies when it starts to consistently miss its heartbeat, right? And it misses its heartbeat when it can't do a tick's worth of computation in 1 second. So: under heavy load, why not up a tick to 2 seconds? Send out an update to all the clients, and make everything that happens on that node take twice as long. This, it seems to me, would scale to any number of players (so long as those players are patient)... if it takes longer than 2 seconds to do a tick, go up to 3 seconds/tick, and so on. I would think that most players would prefer a slower but working solar system than a faster broken one, and it means that a node would never have to die again.

Has this idea been considered? If so, why has it been rejected?

Gecko O'Bac
Deep Core Mining Inc.
Posted - 2010.12.10 20:36:00 - [47]
 

Originally by: Epitrope
Originally by: Black Dranzer
Thought off the top of my head: Do you think it's possible that the size of the timestep is partially responsible for slowing things down? I know it seems counterintuitive; In most situations increasing the step frequency would just bog things down further. But it doesn't seem impossible that a good portion of calculation time is burnt up in compensating for the "fuzzyness" of large timesteps. Alternatively, maybe it's in part the other way around; There might be computations that are done once a second that really don't have to be. Of course, variable sized timesteps for multiple interlocking things would be an enormous pain in the ass.


I actually came up with a similar idea a while ago, and I've always been curious about why EVE isn't set up this way. A node dies when it starts to consistently miss its heartbeat, right? And it misses its heartbeat when it can't do a tick's worth of computation in 1 second. So: under heavy load, why not up a tick to 2 seconds? Send out an update to all the clients, and make everything that happens on that node take twice as long. This, it seems to me, would scale to any number of players (so long as those players are patient)... if it takes longer than 2 seconds to do a tick, go up to 3 seconds/tick, and so on. I would think that most players would prefer a slower but working solar system than a faster broken one, and it means that a node would never have to die again.

Has this idea been considered? If so, why has it been rejected?


From what we know, I doubt this would solve the problem (unless there really are computations that are unneeded for a 1 s timestep). See, the problem is not exactly that if you don't complete the tasks in 1 second you miss the beat... It's that we're using 100% of the resources of the node (CPU first... Dunno if memory gets loaded as well... Perhaps jita is more of a high memory lower cpu usage kind), so the computations simply start being queued until the point where they simply timeout due to the amount of backlogged work. Augmenting the size of the timestep would perhaps let the server "live" longer (the hearbeat pulse though is sent 2 times per minute IIRC), but I doubt it'd make the lag any better.

Hoshi
Hedron Industries
Red Dwarf Racketeering Division
Posted - 2010.12.10 20:51:00 - [48]
 

Originally by: Scharde Alton

1. "figuring out which clients need to be informed"

I suggest that you don't do this. Send the message to every client in the bubble and let them figure out whether or not it means a display update for them. This scales with the number of clients, and EVE is not a bandwidth hog (it's much more adversely affected by latency).


Then you place too much trust in the client which will be hacked to display what is happening on the other side of the solarsystem. Who needs the scanner when you tell the client the exact position/heading/speed of every single object in the system.

Aineko Macx
Posted - 2010.12.10 21:15:00 - [49]
 

OMG they DO have a profiling tool Shocked

Sarcasm aside, this...
Quote:
2) Sending an update to each client identified in the step above. Further investigation showed that this is mainly expensive due to the time required to serialise each update message. (Serialising means converting a data structure in memory into a stream of bytes that can be sent over a network)

... sounds like an excellent candidate for parallel execution to me. The data set is static at this point, so spawn a bunch of worker threads and have them deal with serializing & sending it to clients in parallel. You could go further and cache the serialized data fragments that get sent repeatedly.

Scharde Alton
Aperture Ventures
Posted - 2010.12.10 21:16:00 - [50]
 

Originally by: Hoshi
Originally by: Scharde Alton

1. "figuring out which clients need to be informed"

I suggest that you don't do this. Send the message to every client in the bubble and let them figure out whether or not it means a display update for them. This scales with the number of clients, and EVE is not a bandwidth hog (it's much more adversely affected by latency).


Then you place too much trust in the client which will be hacked to display what is happening on the other side of the solarsystem. Who needs the scanner when you tell the client the exact position/heading/speed of every single object in the system.


You don't actually run this risk, because the clients in the same bubble not solarsystem. In other words, on grid and already available for you to see.

trjcquee
Posted - 2010.12.10 21:34:00 - [51]
 

Fascinating stuff. I should like to point out that once you decide what messages need to be sent to what clients, the task of serializing the data into packets is one that is ripe for parallelization. Another CPU core could take on this job in a separate thread. Unfortunately this is exactly where Python fails: the ability to use multiple CPU cores in parallel.

Stackless Python doesn't really help in this case, because the lightweight threads it uses are really only simulations of honest-to-goodness threads of execution. Those "threads" are not capbable of running on separate cores in parallel. In a way, you might say they're not even in the same ballpark. (*rimshot* thank you, I'll be here all week)

Good to see you folks finally waking up and smelling the coffee. Too bad you're kinda stuck with Python.

trjcquee
Posted - 2010.12.10 21:49:00 - [52]
 

Originally by: Kimiya Alhena
SendToClients seems like the sort of operation that would be spending most of its time waiting on cache misses and bus traffic, both of which should not prevent other threads from running, especially in a multicore environment.

Exactly. The limiting factor here is Python, which is pretty much bound to a single CPU core.

Antihrist Pripravnik
Scorpion Road Industry
Posted - 2010.12.10 21:51:00 - [53]
 

How about replacing missiles with some kind of "mines" that are launched from current missile launchers.

1) Launch a missile (mine) at a target
2) Calculate damage time delay based on positions and speed vectors of the targets
3) When the time is out, check the target's position and speed vector
4) If the target is in "range", detonate (before you mention "range" in a reply, see point "c)" in the next paragraph)
5) Apply damage reduction based on target's properties at the point of the explosion (speed, signature radius,...)

Of course some game mechanics have to change:
a) Remove or modify defenders. Remove is viable because no one use them anyway (except NPCs in missions). Modification can be done by introducing a new high slot module that can "remove" some of the new mines attached to the ship (keep the skill "Defenders" and apply it to performance of the new module)
b) Smartbombs can't hurt missiles any more, but seriously... who relies on the exact time of activation of a smartbomb for fighting against missiles (ecxept forum trolls ofc.)
c) Balance the new "mines" (missiles) so that they are doing roughly the same damage (or no damage) to the same type of targets as they are now

It's just some of my thoughts about the "ball spam" issue. Something like this might help, because it will remove missile balls from the game.

Disclaimer: This is just a few random thoughts, not an actual suggestion, cry, rage or anything like that. If I wanted to suggest this, I would post it in the Features & Ideas section of the forum. And yes, I am mainly a Caldari missile pilot.

Nice blog btw. Waiting for the second part Wink

Jas Dor
Minmatar
Posted - 2010.12.10 21:57:00 - [54]
 

Edited by: Jas Dor on 10/12/2010 21:57:41
Personally, I think there is only one answer to blobing, have shields "flare" and render the target invulnerable to everything but the highest damage hit in a tick. I suspect though that this would cause to much server overhead.

ArchenTheGreat
Caldari
Pulsar Nebulah
Army of Lovers.
Posted - 2010.12.10 22:17:00 - [55]
 

Originally by: Cyaxares II
ApplyPreStuff, AdditionsAndDeletions, ... are these actual method names from your server code? Laughing


Those are names of timers not functions.

Also: awesome blog, more techy stuff please.

Palovana
Caldari
Inner Fire Inc.
Posted - 2010.12.10 22:24:00 - [56]
 

Originally by: Aika Achura
Quick Fix Solution?

1. Reduce N launcher mount points on ships to 1.
2. Add missile damage multiplier of N to affected ship hulls.
3. Reduce turret mount points and high slots as needed for balance.

It would probably be a bit boring for the pilot though.


What they will be doing is making all missiles do 2x the damage but have only 1/2 the rate of fire - easiest way of taking out half of the balls in play.

Lord's Prophet
Totally Abstract
O X I D E
Posted - 2010.12.10 22:46:00 - [57]
 

Originally by: Palovana
What they will be doing is making all missiles do 2x the damage but have only 1/2 the rate of fire - easiest way of taking out half of the balls in play.
This has all kinds of other balance issues. Defenders will become much more viable if there are only half as many missiles in flight. More defender = more missiles = waste of time using this as a fix.

The other thing is cargo space. If you're doubling the damage per missile you effectively double the ammo capacity of all missile-based ships. This is important for long engagements such as POS bashing. If you then reduce their cargoholds, this will affect active-tanking setups which use cap boosters. You would have to change the volume that missiles take in cargo, which will in turn affect people who build missiles. What about the T2 BPO holders, suddenly each missile is twice as effective, so prices will go up? Are you going to change how many missiles they get per run, or how long it takes to make 1 run?

Also, as yet another consequence, this will affect overheating, as heat damage is only added at the end of the module's cycle, and if it's being hot between damage additions this is going to cause much more unpredictable heat distribution, which is another bad thing...


Basically, the 'make it more marauder-like' is a half-baked answer which causes more problems than it solves.

Hoshi
Hedron Industries
Red Dwarf Racketeering Division
Posted - 2010.12.10 23:21:00 - [58]
 

Originally by: Scharde Alton
Originally by: Hoshi
Originally by: Scharde Alton

1. "figuring out which clients need to be informed"

I suggest that you don't do this. Send the message to every client in the bubble and let them figure out whether or not it means a display update for them. This scales with the number of clients, and EVE is not a bandwidth hog (it's much more adversely affected by latency).


Then you place too much trust in the client which will be hacked to display what is happening on the other side of the solarsystem. Who needs the scanner when you tell the client the exact position/heading/speed of every single object in the system.


You don't actually run this risk, because the clients in the same bubble not solarsystem. In other words, on grid and already available for you to see.


From the description in this blog it doesn't sound like that. The "figuring out which clients need to be informed" step are most likely finding out which clients are inside the grid, otherwise it would be a pointless step as every client inside will always need to be informed destiny changes. There are no clients inside the grid that doesn't need to know when a missile is fired or when a ship changes direction or explodes.

Apollo Gabriel
Mercatoris
Etherium Cartel
Posted - 2010.12.10 23:37:00 - [59]
 

Originally by: Monkey M3n
Edited by: Monkey M3n on 10/12/2010 16:35:50
most mega drake blobs don't group the missiles, because there goal is to cause more lag. So 3 groups of launchers isn't really replicating the REAL situation of the game. KTHX


You are making eve worse, Please leave.

L2Read its good, you can move out of mom's basement.

Apollo

CCP Explorer

Posted - 2010.12.10 23:49:00 - [60]
 

Originally by: Firartix
I might be wrong, but according to what you said here and in the blog, the missiles only generate 2 events, client-server communication wise : the launch event, and the outofrange/destroyed/hit event
The turrets only have the hit part afaik

This makes the missiles only making 2x turret's lag ! Why is there such a diff between both ?
Turrets only incur damage tracking calculation cost. Missiles incur physics simulation and damage tracking calculation cost. The physics simulation is actually relatively cheap but constantly adding new objects into the simulation and removing them again is the part that adds up.


Pages: 1 [2] 3 4

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only