open All Channels
seplocked EVE Information Portal
blankseplocked New Dev Blog: Fixing Lag: Drakes of Destiny (Part 1)
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: 1 2 [3] 4

Author Topic

CCP Explorer

Posted - 2010.12.10 23:51:00 - [61]
 

Originally by: Black Dranzer
So wait. You're telling me the physics engine runs at a simulation speed of one FPS?
Yes, EVE ticks at 1 Hz.

CCP Explorer

Posted - 2010.12.10 23:53:00 - [62]
 

Originally by: Lutz Major
Originally by: Marcel Devereux
BeOS:: What's this?
Without googleing ... yes kids we have brains to memorize!

If it is the original BeOS, then it was introduced, what? 12-15 years ago. As - what else - Windows killer.

Back then it had already 64bit ability and was determined to do extreme multitasking. It performed better in Multimedia than any windows or linux back then.

I wonder if CCP really uses the OS from back then ...

BeOS stands for BlueOS Object. Maybe that answer doesn't help...

CCP Explorer

Posted - 2010.12.10 23:54:00 - [63]
 

Originally by: Elsa Nietzsche
how long do we have to wait for part 2?
It will be published tomorrow.

CCP Explorer

Posted - 2010.12.10 23:56:00 - [64]
 

Originally by: Hoshi
Originally by: Scharde Alton

1. "figuring out which clients need to be informed"

I suggest that you don't do this. Send the message to every client in the bubble and let them figure out whether or not it means a display update for them. This scales with the number of clients, and EVE is not a bandwidth hog (it's much more adversely affected by latency).
Then you place too much trust in the client which will be hacked to display what is happening on the other side of the solarsystem. Who needs the scanner when you tell the client the exact position/heading/speed of every single object in the system.
Exactly. We have to assume that we can't trust the client. The server must be authoritative and only send relevant information to each client.

Breaker77
Gallente
Reclamation Industries
Posted - 2010.12.11 00:04:00 - [65]
 

Edited by: Breaker77 on 11/12/2010 00:04:30
Originally by: CCP Zymurgist
lag's arch nemesis, Drakes


Wouldn't the easiest solution be to cut the shields in half and remove 4 mid slots?

I'm pretty sure that would end all drake use thus solving lag.


Epitrope
The Citadel Manufacturing and Trade Corporation
Posted - 2010.12.11 00:44:00 - [66]
 

Originally by: Gecko O'Bac
From what we know, I doubt this would solve the problem (unless there really are computations that are unneeded for a 1 s timestep). See, the problem is not exactly that if you don't complete the tasks in 1 second you miss the beat... It's that we're using 100% of the resources of the node (CPU first... Dunno if memory gets loaded as well... Perhaps jita is more of a high memory lower cpu usage kind), so the computations simply start being queued until the point where they simply timeout due to the amount of backlogged work. Augmenting the size of the timestep would perhaps let the server "live" longer (the hearbeat pulse though is sent 2 times per minute IIRC), but I doubt it'd make the lag any better.


Perhaps I explained poorly. I'm not suggesting changing only the tick size, but actually changing the simulation speed along with it. Under heavy load, run everything at 50% of normal, which gives twice as long to do the same work. Queued work wouldn't time out, modules would activate in a timely fashion, and in general the game would keep working instead of breaking. If that's still not time enough to do the work that needs to be done, run the simulation at 1/3 speed, or 1/4 speed, and so on.

You're right that this wouldn't address memory usage issues. Slowing things down to swap isn't a good solution, but as I understand it memory's not usually the problem, CPU is. I believe that this would be a way to degrade gracefully, which would make larger fleet fights more playable with the kind of lag issues like missiles that Gridlock is optimizing, both now and in the future.

Tonto Auri
Vhero' Multipurpose Corp
Posted - 2010.12.11 00:57:00 - [67]
 

Edited by: Tonto Auri on 11/12/2010 01:01:42
Originally by: Epitrope

Perhaps I explained poorly. I'm not suggesting changing only the tick size, but actually changing the simulation speed along with it. Under heavy load, run everything at 50% of normal, which gives twice as long to do the same work. Queued work wouldn't time out, modules would activate in a timely fashion, and in general the game would keep working instead of breaking. If that's still not time enough to do the work that needs to be done, run the simulation at 1/3 speed, or 1/4 speed, and so on.

You're right that this wouldn't address memory usage issues. Slowing things down to swap isn't a good solution, but as I understand it memory's not usually the problem, CPU is. I believe that this would be a way to degrade gracefully, which would make larger fleet fights more playable with the kind of lag issues like missiles that Gridlock is optimizing, both now and in the future.


Actually, you're speaking of increasing simulation speed. Not to leap into the future, but to reduce amount of iterations over bigger period.
To clarify, take an example of 10 min battle. Normally, it's 600 ticks of time. If Destiny fails to process all events inside the tick, then switch tick length to 2 seconds AND calculate all events that would happen in these 2 seconds, so do only 300 ticks in the same 10 minutes, but still process all the events that should happen.
As everyone saw in the blog, the time spent on actual calculation is minimal, so increasing tick length AND calculated period shouldn't cause major issues. The only problem, client engine must be ready for this speed stepping.
All this does not exclude the need to reduce unnecessary notifications sent to clients.

P.S.
@CCP: Have you considered reworking POS mess on the same modular ground as Outposts work now? Would as well create more pleasing environment handling containers et all - with one POS representing only one container instead of each module having it's own container...

Aurelia Valeron
Posted - 2010.12.11 01:05:00 - [68]
 

Originally by: CCP Explorer
Originally by: Firartix
I might be wrong, but according to what you said here and in the blog, the missiles only generate 2 events, client-server communication wise : the launch event, and the outofrange/destroyed/hit event
The turrets only have the hit part afaik

This makes the missiles only making 2x turret's lag ! Why is there such a diff between both ?
Turrets only incur damage tracking calculation cost. Missiles incur physics simulation and damage tracking calculation cost. The physics simulation is actually relatively cheap but constantly adding new objects into the simulation and removing them again is the part that adds up.


I assume you are using a preexisting pool of physics objects (missile objects, I suppose) which are recycled instead of allocating / destroying physics objects on the fly?

Epitrope
The Citadel Manufacturing and Trade Corporation
Posted - 2010.12.11 01:23:00 - [69]
 

Originally by: Tonto Auri
Actually, you're speaking of increasing simulation speed. Not to leap into the future, but to reduce amount of iterations over bigger period.
To clarify, take an example of 10 min battle. Normally, it's 600 ticks of time. If Destiny fails to process all events inside the tick, then switch tick length to 2 seconds AND calculate all events that would happen in these 2 seconds, so do only 300 ticks in the same 10 minutes, but still process all the events that should happen.
As everyone saw in the blog, the time spent on actual calculation is minimal, so increasing tick length AND calculated period shouldn't cause major issues. The only problem, client engine must be ready for this speed stepping.
All this does not exclude the need to reduce unnecessary notifications sent to clients.


I understand what I'm saying, and I believe I understand what you're saying, and they're not the same.

There will always be larger fleet fights than the server can handle. Right now, the pre-tick client updates step(s) are taking up most of the time. You're right that fewer, bigger ticks would allow the server to send out fewer, bigger updates, which would cause the clients to run at full speed but somewhat jerkier as updates are coming in half as quickly but with twice as much data. However, as optimization work continues, perhaps that section won't dominate the tick quite so much and the evolve step will take a larger percentage, in which case fewer bigger ticks won't work as well.

Your suggestion is, from the information made available to us, a good one. It lessens the lag problem as it exists right now. My suggestion of slowing down the simulation speed is more of a way to keep the game somewhat playable with any lag, current or future, regardless of cause or nature. It's a way to degrade more gracefully than the way high-load nodes degrade do now.

Daneel Trevize
Gallente
Posted - 2010.12.11 02:02:00 - [70]
 

Edited by: Daneel Trevize on 11/12/2010 02:30:06
Can you not effectively lock something quicker than a (or more specifically within a given) second/tick? Can you ever lock something that mwd+cloaks within the same second/tick that you ctrl+click to lock them with, say, a preactivated point/other mods?

I guess this also covers why you can cloak then mwd soon enough after and still get the speed boost..?

Infinion
Caldari
Awesome Corp
Posted - 2010.12.11 02:40:00 - [71]
 

Originally by: Daneel Trevize
Can you not lock something quicker than a second/tick? Can you ever lock something that mwd+cloaks within the same second/tick that you ctrl+click to lock them with a preactivated point/other mods?

I guess this also covers why you can cloak then mwd soon enough after and still get the speed boost..?


I think it also has to do with the question of: can you react fast enough to lock the target and have your client send a call to the server while waiting for your scan resolution all before the target simply makes a call to move and cloak? If it wasn't for scan resolutions, this would probably be possible.

as for cloaking and mwding, the server purposefully waits around 3-5 seconds before making module activation illegal, so I do not believe it is a limitation in the programming rather than an intended game mechanic to give cloakers an advantage at escaping.

Escoce
Gallente
Interstellar Tritanium
United Trade Syndicate
Posted - 2010.12.11 03:44:00 - [72]
 

I see no one has commented on the most important part of this post.

The reference to "Back to the Future"....I had a great laugh when I reached that part of the post.

Thanks :)

Patyrn Runner
Posted - 2010.12.11 06:12:00 - [73]
 

So cool, and you guys are hiring! Now if only I was twice as smart and had some qualifications... ugh

Rip Minner
Gallente
ARMITAGE Logistics Salvage and Industries
Posted - 2010.12.11 07:41:00 - [74]
 

The man with the Masterplan left me hangingugh Other then that very nice blog and keep up the awsome work guysugh

Louis deGuerre
Gallente
Malevolence.
Posted - 2010.12.11 10:07:00 - [75]
 

I want some GridLock's Special-Love !

Pol Hald
Posted - 2010.12.11 10:36:00 - [76]
 

Excellent devblog, thanks. Waiting for part two :)

Ab Tallen
The Alphabet Soup
Posted - 2010.12.11 10:41:00 - [77]
 

Originally by: CCP Explorer
Originally by: Hoshi
Originally by: Scharde Alton

1. "figuring out which clients need to be informed"

Then you place too much trust in the client which will be hacked to display what is happening on the other side of the solarsystem. Who needs the scanner when you tell the client the exact position/heading/speed of every single object in the system.
Exactly. We have to assume that we can't trust the client. The server must be authoritative and only send relevant information to each client.

Does anyone besides the actual target need to know any details of the missile's life anyway?

DmitryEKT
Clandestine.
Posted - 2010.12.11 12:15:00 - [78]
 

Originally by: Ab Tallen
Does anyone besides the actual target need to know any details of the missile's life anyway?
Missiles can be destroyed by AOE, so it has to tell everyone on grid, because a ship can for example be a warpin for a fleet member with smartbombs, etc, and therefore everyone has a need to be able to see the missiles.

Rufus Asgori
Posted - 2010.12.11 12:55:00 - [79]
 

Edited by: Rufus Asgori on 11/12/2010 12:57:00
well, some programmers talk:
As youŽre already talking bout observers i guess the related pattern is in some way involved,
as the task pretty much calls for it.
If the generation of the outputstream really is an issue you might consider giving all info to all observers,
increasing the stream but decreasing the numbers of different streams to one per "grid" an let the observers decide which info is meant for them?
There should be enough bandwidht avaiable throughout the eve playing world.
Like sending all moving objects and all bubble stuff to all clients and then requesting a small sh/ar/str speed and modules stream for each ship, giving data or the "oups, you are dead" stream back.
Might be a cool playground for some freaking visitor stuff also, as your data structure for objects sounds pretty stable, with all your bubble and sphere stuff.
damn iŽd like to see that code ugh

Rufus

Ariz Black
Posted - 2010.12.11 13:34:00 - [80]
 

If the main reason the node gets laggy, is that it has to figure out who to tell about the added/removed balls, why not just do something like:

1. find another node that's not doing much at the moment
2. send that node a list of all the events, and who's in which grid
3. have the idle node do the figuring out out of which info to send to who, and have it do that

this way the node the combat is taking place on can just keep track of the important stuff and the client notification is handled by a less busy node

dunno if it would work like that but hey makes sense to me, at least kinda

CCP Masterplan


C C P Alliance
Posted - 2010.12.11 14:24:00 - [81]
 

Thanks for all the reponses. To address some topics without mass quoting, I'll try to group things together

Quote:
On making SendToClients truly miltuthreaded

This isn't something we've ruled out. We do have some broadcast mechanisms where this could be slotted in.


Quote:
On merging all missile launchers to a single item

Well this is kind of what happens when you group your turrets. It does indeed provide a nice saving on the CPU cost of missiles (so please use it where possible!) Changing things to be more Marauder-like doesn't seem to be the way to go to me - it would ripple out to touch too many other areas

Quote:
On turning missiles from physical entities into effects

This is what we've done with Fighter Bombers (See CCP Chronotis' blog)
So, could we do this for all missiles? Well, yes. But there would be some drawbacks. Firstly it would require a lot of re-balancing: Defenders would become obselete (and there are a lot of NPCs that use them) Secondly we would lose the ability for creative tactics such as the smartbomb firewall. Thirdly I think that having missiles use a distinctly different mechanic from turrets keep things more interesting.

I think it is pretty cool that an interceptor can have a string of slow missiles trailing him, knowing as long as he keeps moving quickly , they will simply never catch up. The fact that the outcome is not determined until when (if?) the missile reaches a target is a good thing.
[This last paragraph is me-as-a-player. In turn this motivates me-as-a-dev to make missiles performant enough that we don't need to turn them into effects]

Quote:
On Destiny as C/C++?

It is C++. Python and C++ work nicely together, as we have ways of making the same underlying object appear as a native object in both languages at the same time. There are several other systems that are also implemented as C++ (or a Python/C++ mixture) - Trinity (the graphics engine) is probably the most obvious.

CCP Masterplan


C C P Alliance
Posted - 2010.12.11 14:24:00 - [82]
 

Edited by: CCP Masterplan on 11/12/2010 14:24:49
Quote:
On skipping the 'figure out which clients need to be informed' step, and just broadcast to everyone in the bubble?

This would indeed save some work, but as CCP Explorer pointed out, would also open up exloit potential. Imagine the case where you have a few cloaked and uncloaked ships on a gate, and a few more cloaked and uncloaked ships warp into that bubble. (Remember that if you are flying a cloaked ship, the updates you receive must include your own ship) Now, remembering the client is in the hands of the enemy, you should be able to imagine ways to exploit this if everyone had the same update.

Quote:
On 1Hz updates?

Yup. It has always been that way. This is why a form of direct-flight joystick control simply wouldn't work in EVE.
If you look are the tick rate of other games, from FPS up to MMO, there is roughly an inverse relationship between tick-rate and number of players that can gather together.

Quote:
On the BTTF reference

I couldn't miss up an opportunity to do this :) (BTW there are a few other more obscure references to other things scattered in this blog and the second half as well...)

Quote:
On slowing down time / increasing the tick period

This is something we've considered, along with a number of other bigger-scale changes that we've discussed. Some we quickly eliminated beacuse they wouldn't actually fix the issue. Some we've got on the board as potentials. Some we've even protoyped.
Looking specifically at the slowing-down-time idea, this has received quite some attention. Some things to think about: What things should be slowed down? (Shield recharge, rate-of-fire, movement, skill-training, market order duration, aggression times, siege cycles...) Suddenly when you have two clocks running at different rates, you have to look at every single mechanic that is in some way related to time. What about other solar systems on the same node - should they slow down too? If we try to let different systems on a node run at different rates, it means we potentially have to keep track of which system a particular function call is dealing with, in case it needs to query a timer.
I'm not saying all of this is impossible, simply that there is a lot more to it than you might initially think. (Take this as a sign that this is indeed an idea that has received attention)

Quote:
On why did I choose 3 launchers - why not 1 or 7?

Simply because it was a reasonable midpoint. Seven launchers would have made the before/after difference more significant, but wouldn't change the basic result. One launcher would have meant the missile load was smaller relative to everything else and I wouldn't have been able to so easily see where I needed to be focusing my attention.

Quote:
On part two - when?

First you will need to learn patience. And also where the F5 key is...

Dunpeal
Caldari
M'8'S
Posted - 2010.12.11 14:34:00 - [83]
 

Edited by: Dunpeal on 11/12/2010 14:47:57
Originally by: Rufus Asgori

If the generation of the outputstream really is an issue you might consider giving all info to all observers,



If you let the client decide if the update is relative to itself, wouldnt that allow for client hacking and purposefully making a client wich cannot die? I mean

Client gets shot at, server talks to it and says "Youve been shot and it killed you", client then responds, "dead ship was not me", in wich the server would then get a data stream back from a client that suposedly no longer takes part of the simulation wich continues to input data although the server says it is dead.

Even if the client does not directly affect the simulation it will continue to affect the speed of the simulation by constantly taxing the server with input that the server will have to identify as being true or false, if the server code where prepared to handle those bad calls, one hacked client would probably not do much but hundreds probably would, therefore making that solution actually a new problem.

Originally by: Ariz Black
If the main reason the node gets laggy, is that it has to figure out who to tell about the added/removed balls, why not just do something like:

1. find another node that's not doing much at the moment
2. send that node a list of all the events, and who's in which grid
3. have the idle node do the figuring out out of which info to send to who, and have it do that

this way the node the combat is taking place on can just keep track of the important stuff and the client notification is handled by a less busy node

dunno if it would work like that but hey makes sense to me, at least kinda


If i remenber correctly the reason for CCP to not do this is keeping things within sync.
Offloading those calculations would create overhead in order to send the stuff, and making sure the transfer, running of the code and completion would keep in sync with the next tic. After wich the first node would be receiving new input to be resolved, where if the first input was not already resolved sent and received by clients before new input is returned to the first node, the simulation would infact be invalid at that point because clients where sending invalid input because it did not take into consideration the last update.

For example a ship that got killed with the last update still send a damage input to the server, so that players could receive damage from already killed ships.

P.S: Please disregard if what i said does not make sense, only basic knowledge of programming here. Laughing

Deep1
Posted - 2010.12.11 14:35:00 - [84]
 

Lovely - so that is what you send the time on - keep it coming :-)

With all that time spending on Clientupdates why not move that OUT of Destiny ?

Have Destiny create a client cache/node that send updates to clients so that Destiny can take a break ( by just sending ONE update not 300 or 1000+).

Since this would be new code make it so it can be done on the fly.
make it a bit smart and let that part desite if a client need or can get the update ( eq cloaking ships etc).

Tell the Player client what update-node that handles his/her request on enter System or grid/bubble.

And to the more complex part:

1) On node load create one node of a small size ( say 30-50 max )
2) If more than max clients enters system tell Destiny - and get new Updatenode
3) The extra nodes are create of a larger max Size say 100-200.

All UpdateNodes must keep track of it current state so count time of one update cycle
At startup of the node a value of 10 is set.
If the update time is higher than 70 % and value is less than 20 - increment value by 1.
If the update time is lower than 70 % and value is more than 0 - decrement value by 1.

this would give a track over 10 ticks so we don't overreact on spikes - and it does not take into account any other process running on that node - just tells us what we need if the work is done in a god manner. what it don't take into accounts is that a update cycle may go dead ( take a long time to complete eq 10 mins or more ).

4) any extra update nodes should be created on the CPU with the lowest current use - i believe that you have that info.

Next Cleanup - it really don't make any sense that a fleet travel vs 20 system should leave 20 systems with 10 extra nodes when they are all back in station.

So i would do some cleanup every 5 mins.
If a system only has one node - exit ..
If there is One extra Updatenode and it has less than 10 % of max user AND there is room for it in the Start node - move users ( tell new Update node to them ) and Kill the extra node.
If there is more than one UpdateNode check nodes and if more than one has less than 30 %
users try to merge buy moving users to the newest node.


Deep1

Cpt Wisdoom
Posted - 2010.12.11 15:11:00 - [85]
 

Just fix Drake itself, make it balanced already. Balanced Drake = less popular Drake = less Drakes in space = less missiles fired = less lags! No more drake swarms = everybody happy!

I think that it is a best solution.

Nevryn Takis
Posted - 2010.12.11 15:23:00 - [86]
 

My first reaction on reading elements of this is that the object model for the ball is wrong.
Additionally if you're mixing C++ and python I'm assuimg that the C++ elements are bolted into the python kernal as library elements are are therefore directly callable as python functions.
Assuming this is the case why isn't each client that a ball may has an effect on, particuallary the source and destination, stored as a reference in each ball, this way no figuring out is required, a ball is simply processed and the client object tracks whether any further data updates a need, when all it's referenced balls have been updated it fires an asynchronous thread to perform the serialization and update. This also allows the handling of lag since you can skip sending of updates if the current send hasn't succeeded, meaning you don't serialize unnecessarily.

I may of course be completely wrong....

CCP Masterplan


C C P Alliance
Posted - 2010.12.11 15:26:00 - [87]
 


MotherMoon
Huang Yinglong
Posted - 2010.12.11 22:38:00 - [88]
 

Edited by: MotherMoon on 11/12/2010 22:42:29
Originally by: CCP Masterplan
Defenders would become obselete


LaughingLaughingLaughingLaughingLaughingLaughingLaughingLaughingLaughing

Quote:
Yup. It has always been that way. This is why a form of direct-flight joystick control simply wouldn't work in EVE.


Is there still plans to have asteroid "dungeons" in the future where there can be joystick cobat in shuttles?

I've heard devs talk about this for like 5 years, but never anything official. Would that even be possible?

Does incarna have to match the 1Hz of the in space part? I'm asking because I know they might end up running on the same nodes maybe? How does that sorta stuff work? can you just split the code? Or will incarna run client side as there isn't much to exploit?

CCP Explorer

Posted - 2010.12.12 09:02:00 - [89]
 

Incarna will tick at a higher frequency. We are refactoring the network code these days for that purpose.

EdwardNardella
Capital Construction Research
Posted - 2010.12.12 16:29:00 - [90]
 

Originally by: CCP Explorer
Incarna will tick at a higher frequency. We are refactoring the network code these days for that purpose.

Wait! What? Will the player be able to notice a difference?


Pages: 1 2 [3] 4

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only