open All Channels
seplocked EVE Information Portal
blankseplocked New Dev Blog: Fixing Lag: Drakes of Destiny (Part 2)
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: 1 [2] 3 4 5 6

Author Topic

Turix
Interstellar eXodus
BricK sQuAD.
Posted - 2010.12.11 16:41:00 - [31]
 

I must say, this is exactly the kind of information I love to read about.

Personally, being a software engineer myself, I do like the detail (Psuedo-Code + Graphs = Win!). Defiantly keep writing these types of blogs and if possible encourage other teams to do the same, with the same level of detail! It would be... awesome?

Salpun
Gallente
Paramount Commerce
Posted - 2010.12.11 16:42:00 - [32]
 

Thanks for the reply. Great blog.

To some reinforced means that you are using the jita super node. Is this the case? If not what performance are you expecting out of it the next time you use it on a major fleet fight? Also how many of those do you have now and when was the last time they where used out side of jita?

CCP Masterplan


C C P Alliance
Posted - 2010.12.11 16:47:00 - [33]
 

Originally by: Ebisu Kami
Did you guys notice that, according to the graphs, the "Missiles - Inventory and Damage"-part of the load increased by roughly 12% from old to new?

I was waiting for someone to spot that! During the first test, the node was running close to 100%. Therefore the launcher modules weren't necessarily cycling at their full rate for every shot (the drakes were on the edge of experiencing module lag)
Once the CPU was lowered, the launcher were able to cycle at their full rate all the time, and so were able to pump out more missiles during the 100 seconds. More missiles means more inventory operations, so absolute inventory load increases slightly.
CCP Veritas' earlier blog explores some of these artifacts that occur at the edge of performance limits (and beyond)

Arkady Sadik
Minmatar
Electus Matari
Posted - 2010.12.11 16:47:00 - [34]
 

I would like to thank CCP and the devs for writing these very technical devblogs. I'm a programmer myself and I really appreciate such a close look at the work of the programmers at CCP. It really gives a "oh my gods, I can so totally relate to that" feeling. Thank you! :-)

Ebisu Kami
Posted - 2010.12.11 16:49:00 - [35]
 

Originally by: CCP Masterplan
Originally by: Ebisu Kami
Did you guys notice that, according to the graphs, the "Missiles - Inventory and Damage"-part of the load increased by roughly 12% from old to new?

I was waiting for someone to spot that! During the first test, the node was running close to 100%. Therefore the launcher modules weren't necessarily cycling at their full rate for every shot (the drakes were on the edge of experiencing module lag)
Once the CPU was lowered, the launcher were able to cycle at their full rate all the time, and so were able to pump out more missiles during the 100 seconds. More missiles means more inventory operations, so absolute inventory load increases slightly.
CCP Veritas' earlier blog explores some of these artifacts that occur at the edge of performance limits (and beyond)


I see. Thanks for the reply and clarifying :)

Zaotome
Posted - 2010.12.11 16:51:00 - [36]
 

thank you! i really like detailed info + pics :)

Gripen
Posted - 2010.12.11 16:59:00 - [37]
 

What is the point of sending new update for every missile launched? Why can't you do it in the same manner as with turrets: send only "activate\deactivate launcher" updates with RoF of launcher specified so both client side and server side simulations could create new missile objects for themselves at specified interval.

CCP Explorer

Posted - 2010.12.11 17:02:00 - [38]
 

Originally by: Salpun
Thanks for the reply. Great blog.

To some reinforced means that you are using the jita super node. Is this the case?
Reinforced simply means that the solarsystem was placed alone on a dedicated node. Each blade in the TQ cluster has a 4 cores (and 32 GB RAM) and we therefore run 4 nodes on each blade.

Salpun
Gallente
Paramount Commerce
Posted - 2010.12.11 17:06:00 - [39]
 

Thanks for the reply but you dodged the hard part of the question. Laughing

Amy Garzan
Gallente
Posted - 2010.12.11 17:57:00 - [40]
 

Awesome to see the results of this work knowing it has been going on for quite a while and in multiple parts, and also to see how its linking together.

So when do we get to see pictures of the server move a few months back as promised?? Very Happy

static zero
Minmatar
Power of the Phoenix
Posted - 2010.12.11 18:13:00 - [41]
 

Originally by: CCP Masterplan
That's why I added the single-frame cache: If we need to look up some ball info for multiple observers, the first one will cause the value to be fully looked-up/calculated, and subsequent requests will pull the results from the cache. As I mentioned, this was more of a problem when one fleet warps into another. In these cases, I was seeing a cache hit:miss ratio of 50,000:1 (I'm talking about the ball-state cache here, not the CPU cache)



Oh! I glossed over that three-line paragraph. Embarassed That's a striking hit ratio, no pun intended. So the single-frame cache helps populate the ball-field list. Then that is used build an update for each observer of the ball-field.

Later, you mention that serialization of that data needs to be done only once, and can be done using that list. On your bar graph, that appeared to be the bigger savings (unless SendToClient implies some other work to be done).

It sounds to me like each observer no longer has ball-field adds/deletes built just for them. Instead, they are now "subscribed" to ball-field updates (with adjustments for corner cases). Makes a lot of sense.

All of this sounds like a multicast paradigm to me, but that's probably only because I'm a network guy. :)

Well done!

-static

Tester128
Fremen Sietch
DarkSide.
Posted - 2010.12.11 18:16:00 - [42]
 

nice blog, man
good work with optimization too
keep it up!

Daneel Trevize
Gallente
Posted - 2010.12.11 18:19:00 - [43]
 

Edited by: Daneel Trevize on 11/12/2010 18:20:50
Going to repeat myself from part 1, but, re: Destiny being 1Hz

Can you ever lock something that mwd+cloaks within the same second/tick that you ctrl+click to lock them with, say, a preactivated point/other mods?

An ewb
Posted - 2010.12.11 18:27:00 - [44]
 

If everything is happening with 1hz, but everyone in the node is experiencing increasing delay, wouldn't it be possible to slow down the simulation itself dynamically based on how much stress the node is under? Have a fluff piece like "Massive fleet formation in this area is causing temporal disturbances" and slow down the simulation cycle so everyone still gets fed the stuff that is happening, but everything just seems to be going slower if there's more stuff happening, instead of random dropoffs and blackscreenings and such. Is this just a dumb idea?

The Mach
Reikoku
Cascade Imminent
Posted - 2010.12.11 18:33:00 - [45]
 

Great Blog!! Can nVidia Physx be used to optimize this further, or at least shift the load to a currently unused graphics card?

Dr Laren
Posted - 2010.12.11 18:39:00 - [46]
 

Originally by: The Mach
Great Blog!! Can nVidia Physx be used to optimize this further, or at least shift the load to a currently unused graphics card?


Servers don't have graphic cards.

Daneel Trevize
Gallente
Posted - 2010.12.11 18:40:00 - [47]
 

Edited by: Daneel Trevize on 11/12/2010 18:40:18
They already blogged that the issue's pre-tick, not the crunching of any physics simulation calculations but just setup and preparing to disseminate the results of said calcs. Also, Physx can go DIAF. Open GPGPU less so.

Dunpeal
Caldari
M'8'S
Posted - 2010.12.11 18:51:00 - [48]
 

Edited by: Dunpeal on 11/12/2010 18:52:07
Originally by: An ewb
If everything is happening with 1hz, but everyone in the node is experiencing increasing delay, wouldn't it be possible to slow down the simulation itself dynamically based on how much stress the node is under? Have a fluff piece like "Massive fleet formation in this area is causing temporal disturbances" and slow down the simulation cycle so everyone still gets fed the stuff that is happening, but everything just seems to be going slower if there's more stuff happening, instead of random dropoffs and blackscreenings and such. Is this just a dumb idea?


No, it isn't, not because i say so, but because mastermind said so on the comment thread of the first part.
It has gotten lots of looking into that as a "solution" to lag, not to kill it but to allow the node to "breath".
Mainly why they havent already jumped head first into it seems that it is more complicated than it seems to implement, when a solar system starts to get purpusefully out of sync with the rest of the cluster alot of planing needs to be done to allow for that temporal distortion to happen and make sure the node plays nice with all the other nodes.

Nareg Maxence
Gallente
Posted - 2010.12.11 18:56:00 - [49]
 

I love your stuff!

Using a profiler can be a huge eye opener. I have done some profiling on a DB application. By figuring out exactly where the bottlenecks where I managed to get 80-90% speed increases on some routines.

You are definitely up there competing with PrismX as my favorite dev!

Carniflex
StarHunt
Fallout Project
Posted - 2010.12.11 19:15:00 - [50]
 

I think I have noticed in the past that 1 Hz tick of the nodes by experimenting with damage modules on ships. So a little clarification would make me a bit happier ugh

Say, I have weapon system that with 3 damage mods has RoF of 4.2 seconds and with 4 damage mods RoF of 3.95 seconds. If I can kill the target in one volley does this mean that in the first case I can reactivate my gun on next target at 5th second and on second case of 4th second or can I activate my weapon system mid tick as well at 4.2 seconds ? Or is RoF of 4.2 seconds only used when it takes several volleys to kill a target and in one volley situations the RoF is rounded up to nearest full second ?

Black Dranzer
Caldari
Posted - 2010.12.11 19:27:00 - [51]
 

Edited by: Black Dranzer on 11/12/2010 19:29:26
Originally by: Dev Blog
1: for every observer: # (ie a user's ship)
2: for every ball in the observer's bubble:
3: if ball was added/deleted to the bubble in the last tick:
4: gather up all the data required to tell the observer's client about this ball
Originally by: Dev Blog
step 4 always produces the same data for a given ball, regardless of who the observer is
Originally by: CCP, paraphrased
Let's perform redundant calculations inside a quadratic loop OH GOD WHY IS OUR CODE SO SLOW
Please visit your user settings to re-enable images.

Louis deGuerre
Gallente
Malevolence.
Posted - 2010.12.11 19:33:00 - [52]
 

Tech blogs make me feel all warm and fuzzy Razz

Daneel Trevize
Gallente
Posted - 2010.12.11 19:33:00 - [53]
 

Edited by: Daneel Trevize on 11/12/2010 19:34:34
Originally by: Black Dranzer
WHY WOULD YOU DO THAT?

Because premature optimization is evil, so first you get it to work, and then you have other things to do ugh

Black Dranzer
Caldari
Posted - 2010.12.11 19:39:00 - [54]
 

Edited by: Black Dranzer on 11/12/2010 19:49:35
Originally by: Daneel Trevize
Edited by: Daneel Trevize on 11/12/2010 19:34:34
Originally by: Black Dranzer
WHY WOULD YOU DO THAT?
Because premature optimization is evil
This particular situation does not support that theory.

It just worries me; For the longest time I assumed that the kind of stuff that was causing lag was just general scale errors that would only be solvable by extremely clever algorithms. But if that kind of stuff has been sitting in the code base for however many years? God only knows what else is in there that needs tidying up. I mean, look, I do not claim to be an elite programmer, but when I'm going through a large collection and calculating stuff based off of that collection, assuming I'm actually writing field code and not just some function test prototype stuff, the first thing I do is look for major redundancies.

But, hey, nobody's paying me however many dollars an hour to program, so there we go.

Chruker
Posted - 2010.12.11 20:09:00 - [55]
 

Not to rain on your parade, but I've been getting a laggy experience in ex. Agrallarier and other systems surrounding Auvergne. It seems to have been in the last couple of weeks and it has been with just around a 100 people in the systems.

The lag is especially noticable when salvaging and looting missions... it is not severe lag, but more of a 1-2 seconds delay on every action.

Daneel Trevize
Gallente
Posted - 2010.12.11 20:40:00 - [56]
 

Edited by: Daneel Trevize on 11/12/2010 20:41:47
Originally by: Black Dranzer
This particular situation does not support that theory.

It just worries me; For the longest time I assumed that the kind of stuff that was causing lag was just general scale errors that would only be solvable by extremely clever algorithms. But if that kind of stuff has been sitting in the code base for however many years? God only knows what else is in there that needs tidying up. I mean, look, I do not claim to be an elite programmer, but when I'm going through a large collection and calculating stuff based off of that collection, assuming I'm actually writing field code and not just some function test prototype stuff, the first thing I do is look for major redundancies.

But, hey, nobody's paying me however many dollars an hour to program, so there we go.
Sadly I learnt that in RL with budgeted time & money if you dare show most employers that something works at all, they'll have the idea it's done and there's far higher priorities, and I expect it goes doubly when developing a game who's progress is what the company depends on to keep bringing in the iskies, rather than in say, an institution that's 'too big to fail' and you would think could afford to adopt a policy of doing it right first time... Rolling Eyes

Incidentally my fave Java look & feel style is the one that makes everything look like building blueprints rather than a regular GUI.

Charles Javeroux
Gallente
INTERSTELLAR CREDIT
Posted - 2010.12.11 21:40:00 - [57]
 

This blog gave me inspiration...."Save New Eden, kill a missile ship TODAY!" Very Happy

Hylax Ciai
Cataclysm Enterprises
Ev0ke
Posted - 2010.12.11 22:07:00 - [58]
 

Edited by: Hylax Ciai on 11/12/2010 22:33:37
So, the main reason for missiles being items and not just effects, is that their damage is not applied instantly?

Also, seeing that you put a nested loop in there i just couldn't think of anything else than "Man. Today is so loopy." Laughing
Unfortunately i've seen just a few episodes of the series..

Ben Derindar
Dirty Deeds Corp.
Posted - 2010.12.11 22:53:00 - [59]
 

Excellent read, nice work.

EdwardNardella
Capital Construction Research
Posted - 2010.12.11 22:55:00 - [60]
 

This would be a worthy second half of the Christmas gift mentioned in the blog about learning skills.


Pages: 1 [2] 3 4 5 6

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only