open All Channels
seplocked EVE Information Portal
blankseplocked New Dev Blog: ing Lag: Module Lag - Why Not All Bugfixes Are A Good
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: 1 2 [3] 4 5 6 7

Author Topic

Luke S
Zeta Corp.
Posted - 2010.08.23 18:05:00 - [61]
 

Stupid question: Does module lag happen to all types of mods? Such as mining lasers?

DmitryEKT
Clandestine.
Posted - 2010.08.23 18:25:00 - [62]
 

Originally by: Kesper North
Can you explain why you are using Maelstroms as a laser platform? Very Happy
because, even after the projectile buff, a maelstrom with tachyons does more dps than a maelstrom with 1400mm's.

also, he said he was shooting for hours, so he probably didnt want something ammo-dependant. and matari ships are cool. so there ya go. i'd have taken abaddons myself...

Helios Black
Viziam
Posted - 2010.08.23 18:30:00 - [63]
 

Gotta say that I love the amount of blogs dedicated to informing us what's going on with fixing lag. Nice to hear about the technical aspects of things and know there's actually people out there who care about the issue and are working on it.

Thanks CCP!

Hykke
Free Imperial Vikings
Posted - 2010.08.23 18:39:00 - [64]
 

Great series of dev blogs recently, this one in particular impressed me.

Being a developer myself I can appreciate the efforts you guys are doing to hunt down those illusive bugs and performance issues causing lag.

My guess is that sometime in the far far past there was a bug caused by the effect system not agreeing that a module should be cycled. The developer who got this bug report, then looked over the code and decided that this bug was so rare that he would just write a quick workaround. After this workaround was implemented the bug disappeared, and the problem was "solved" ... only to reappear 1.5 million times later :-)

Experience has taught me that a quick workaround is often the wrong solution to a problem

Great to see that you are not trying to implement a quick fix, but are trying to get to the root of the issue.

Ki Rathos
Minmatar
196 MPAD
Posted - 2010.08.23 18:47:00 - [65]
 

I too was curious if there was a way to offload the effects processing to another node. If it is sounds like that might do the trick.

Think it would be cool too if you guys could develop some kind of tool that allows the thin client to say run on my comp and take requests from your central server, maybe have it connect into sisi.

Would leave it on while I am working, you guys would then have a veritable army of PC's ready to put tons of automated pew pew ships onto a node. Maybe once you guys add in some diagnostic tools. A system like that would allow you to mass test whenever you needed from actual client machines, sort of the end to end way to check on things.

Vaerah Vahrokha
Minmatar
Vahrokh Consulting
Posted - 2010.08.23 19:25:00 - [66]
 

The developer facial expression... almost repaid for all of the months worth of effort spent in asking for lag fixes.

CCP Shadow


C C P
C C P Alliance
Posted - 2010.08.23 19:43:00 - [67]
 

Off-topic posts removed. Stay classy. Cool

Grez
Neo Spartans
Laconian Syndicate
Posted - 2010.08.23 20:03:00 - [68]
 

Originally by: CCP Shadow
Off-topic posts removed. Stay classy. Cool


*grabs a glass of whiskey*

Will do, Sparky!

Thanks for the awesome dev blog!

Trebor Daehdoow
Gallente
Sane Industries Inc.
Posted - 2010.08.23 20:12:00 - [69]
 

Originally by: Vuk Lau
I will have liberty to say that I almost pooped all over Socratesz when I heard the famous "EVE is now, from a technical standpoint, in a better state than it has ever been." so my bet is that was pure example of nonsense


NDA or NDA, this is TMI IMHO.

Originally by: CCP Warlock
But let's just say, that there's a little samizdat activity in CCP at the moment to provide an appropriately inappropriate t-shirt in order to properly memorialise that particular entry into the great list of "They can't hit an elephant from ther...." remarks.


Remember to print extras for the CSM. I take an XL.

Seriously now, excellent blog, it gives a good view of how the debugging process really works. And kudos to Vuk Lau for bringing the video to the meeting.



Selnix
Gallente
North Eastern Swat
Pandemic Legion
Posted - 2010.08.23 20:23:00 - [70]
 

Originally by: CCP Veritas
In other words, I believe Dogma is doing stupid things, and I intend to beat the stupid out of it before considering giving it rocket boots.


Given the current state of Rockets, only a stupid Dogma would try to use such boots. Very Happy

Thx for a nice blog tho.

Amy Garzan
Gallente
Posted - 2010.08.23 20:39:00 - [71]
 

Yay! Keep up the good work and techie blogs. I love reading them.

Mashie Saldana
Minmatar
Veto Corp
Posted - 2010.08.23 20:52:00 - [72]
 

Originally by: Kesper North
Can you explain why you are using Maelstroms as a laser platform? Very Happy

I think it was mentioned elsewhere that it was civilian lasers. Shocked

Glyken Touchon
Gallente
Independent Alchemists
Posted - 2010.08.23 21:13:00 - [73]
 

Originally by: Mashie Saldana
Originally by: Kesper North
Can you explain why you are using Maelstroms as a laser platform? Very Happy

I think it was mentioned elsewhere that it was civilian lasers. Shocked


Probably because it has 8 turrets & they didn't want to risk hurting the titan

Rage Spear
Minmatar
Republic Military School
Posted - 2010.08.23 21:51:00 - [74]
 

Best blog for a long, long time.

Thanks for writing this and good luck with finding some way to let us cycle our guns AND do everything else instead of just one or the other.

wizard87
Posted - 2010.08.23 21:57:00 - [75]
 

Good blog, great pictures.

You're a good communicator CCP Veritas - you're analogies seem to make sense for the less technically inclined.

Hope you get a good appraisal as you appear to be one of the good eggs at CCP. Wink

Brigitte Helm
Minmatar
Native Freshfood
Posted - 2010.08.23 22:34:00 - [76]
 

Thank you for a well thought out dev blog that explains not only the problem but how you find what the true problem is. Plus great photo

One minor correction the actual quote is "They couldn't hit an elephant at this dist..." General John Sedgwick's final wordsLaughing

Xianthar
Vanishing Point.
The Initiative.
Posted - 2010.08.23 22:43:00 - [77]
 

surprised your using cooperative scheduling given the very dynamic workload, i figure you'd used preemptive scheduling or tasklets with some sort of tasklet priority assignment.

CCP Atropos

Posted - 2010.08.23 22:43:00 - [78]
 

Originally by: Hawk TT
Edited by: Hawk TT on 23/08/2010 16:55:18
Great Blog! Nice screenshots Twisted Evil

For those interested in the Python GIL (Global Interpetter Lock) drawbacks and in Python cooperative-multitasking:
David Beazley's - Understanding Python GIL - PyCon 2010

This is David Beazley's presentation @ PyCon 2010...who the f**k is David Beazley?!?! Evil or Very Mad - OK, Google him ConfusedSmile
The presentation is fun, it's informative, it shows how Python behaves on multiple-CPU cores (BAAAAD!) etc.
The presentaiton also covers the issues when threads release control prematurely Embarassed
And there is some hope - a new GIL is being developed, unfortunately for Python 3.2+, but still it could be ported to Stackless 2.7...More or less, it seems that GIL will survive for the time being and will not be "killed" any time soon...

Hopefully CCP is working on something on their own...GIL in Python = No Benefits form Multicore Processing

I watched this since I was unable to attend PyCon this year and I have to agree; a very informative and not too technical view of life with the GIL.

Zendoren
Aktaeon Industries
The Black Armada
Posted - 2010.08.24 00:57:00 - [79]
 

That presentation goes to show you that, more often then not, laziness is the cause of most problems in life.

Sort answer is, the python Dev's are lazy and do not want to make a proper interpreter for its code, and CCP does not want to get into the programing language inventing business!

Looks like CCP is stuck between a rock and a hard place on the threading front! EmbarassedLaughing

So, how is that code refactoring project coming? Was the 3rd time the charm? Twisted EvilLaughingTwisted Evil

Jinli mei
Gallente
Dreddit
Test Alliance Please Ignore
Posted - 2010.08.24 02:31:00 - [80]
 

Is CCP doing anything fancy and magical like replacing the GIL for Python with something less suck?

OmegaTwig
Firebird Squadron
Terra-Incognita
Posted - 2010.08.24 02:47:00 - [81]
 

Originally by: Luke S
Stupid question: Does module lag happen to all types of mods? Such as mining lasers?

If you can get everyone in Jita into one belt with enough Veldspar so that everyone can be mining at once, with drones and everything then yes i believe this will happen with ANYTHING that cycles and produces an effect...

BTW: Love the Devblog, +1 for pics... Gonna have to check more regularly for them because they are coming out more frequently!

FakeEdit: Found the RSS Feed, YAY!

Blazde
4S Corporation
Morsus Mihi
Posted - 2010.08.24 03:31:00 - [82]
 

Originally by: CCP Veritas
I've been busy profiling it under some basic scenarios and there's some low hanging fruit to be plucked in there. There's significant algorithmic optimizations to be made as well, and those follow you around regardless of what language you're in

This single line answers all my frustrations with the feedback we've had so far about lag because any mention of basic low-level code optimisations has been glaringly absent before now. Please, please crack on with this. I spend a lot of time optimising code and it's really not uncommon to get 10-20 even 100 times speedup in code that hasn't been agressively optimised before, even if it was written fairly well in the first place. I have something I call the '10x rule': if giving your code ten times more time to run gives some significant benefit (removes a bottleneck, gives results previously unreachable, puts it acceptably inside a performance limit, cures lag!, etc) then optimise the hell out of it before even considering any costly high-level redesigns. In Python (or any other high level language) this applies more than ever because the language makes it very easy for a coder to do very costly things without realising it, if they don't fully understand what's happening underneath the hood.

In this case it sounds like the Dogma code is *the* fleet-fight bottleneck (makes perfect sense, activating modules/applying effects in very large numbers is pretty much the definition of a fleet fight), and it hasn't been agressively optimised in the past - resisting temption to ask why - so this should yield great results. (Hell just send me the Dogma sources and I'll send em back with all no-brainer optimisations done Razz)


Regarding the Dogma yielding bug, was this introduced around October 2008? Ie. the exact time lag seemed to get much better. This was when fleet fight lag changed very dramatically from:
- Not being able to lock
- Warping/basic movement commands taking forever to initiate. You could 'queue up' warp commands, and often spent entire battles warping to the place you wanted to be at an hour earlier
- Modules taking forever to active

to:
- Warping/moving being extremely responsive
- Locking often being pretty good
- Ghost wrecks
- Modules activating but not cycling, and getting stuck

There were theories at the time that devs had purposely changed the priorities of various commands, and it was seen as a very good thing because being in the right place and being able to lock stuff but not kill it fast is much less frustrating than being in the wrong place all the time being killed by the single ceptor that was lucky enough to lock you.

Some datapoints from my logs:
2008 Jan to June
An attempt is underway to activate the module - 116 occurences
An attempt is underway to deactivate the module - 2 occurences
2009 Jan to June (I did play quite a bit during this period, lag was just that good)
activate - 0
deactivate - 3
2010 Jan to June
activate - 7
deactivate - 245 (mainly occurs when deactivating mods by clicking their mini-icon next to target)


Regarding stuck modules I still think the problem is clientside as well. The server may stall sending the expected message back but it is the client that stops you interacting with the mod when it's blinky-red. If you could interact with it you could send new deactivation requests (which do somehow work), as shown by the mini-icon workaround. It's broken in normal non-laggy gameplay too because you can't do stuff you want to with a deactivating module like:
- Right click on it to get it's show-info (eg. check your optimal/falloff)
- Check or change it's autorepeat/autoreload status
- Pre-heat it

Good engineers don't just fix a problem once afterall Very Happy so I think you should fix this clientside too.

Anyway good luck and thanks for the blog, I might just have a new favourite dev (favourite new dev at the very least).

Liol Wongster
Posted - 2010.08.24 03:31:00 - [83]
 

Originally by: Ki Rathos
.........

Think it would be cool too if you guys could develop some kind of tool that allows the thin client to say run on my comp and take requests from your central server, maybe have it connect into sisi.

Would leave it on while I am working, you guys would then have a veritable army of PC's ready to put tons of automated pew pew ships onto a node. Maybe once you guys add in some diagnostic tools. A system like that would allow you to mass test whenever you needed from actual client machines, sort of the end to end way to check on things.


+1 for this.
I would happily let this run for the hours I'm not able to be online and active myself.
The mass tests all happen at a time that is no good for me to log in with my chars anyway. Anyway I caould help, I'd do happily.

Vaal Erit
Science and Trade Institute
Posted - 2010.08.24 03:34:00 - [84]
 

It looks like the lag hunting team and the thin client are making good progress. From the sound of this dev blog we might be getting a decent improvement in lag situations with the next expansion. Go go CCP!

Thor Brawn
Posted - 2010.08.24 03:36:00 - [85]
 

Good work!!! Keep going and this game will become hotter than a Sandy beach with Victoria Secret models...

Well almost Razz

Whatever Dood
Posted - 2010.08.24 03:38:00 - [86]
 

Edited by: Whatever Dood on 24/08/2010 04:57:22
Edited by: Whatever Dood on 24/08/2010 03:41:42
Edited by: Whatever Dood on 24/08/2010 03:41:11
Originally by: Jinli mei
Is CCP doing anything fancy and magical like replacing the GIL for Python with something less suck?


I'd guess they plan on doing something like running a separate instance of the Python interpreter on each thread, each with completely separate GIL's, and then manage their thread interactions themselves. Edit: or rework the Python interpreter internals and remove it entirely.

I swear my IQ dropped 20 points watching that presentation. I have never seen anything so totally ******ed in my life. "This is a problem that's been solved in operating systems" - yeah, like two-three DECADES ago. "Don't use this talk to justify not using threads" - maybe to justify not using Python?

I had to watch the presentation here, original link doesn't play all the way through for me:

http://www.mefeedia.com/watch/28851035

Blazde
4S Corporation
Morsus Mihi
Posted - 2010.08.24 03:46:00 - [87]
 

Originally by: Axemaster
Actually, why not use a gpu to process this? They seem tailor-made for these kinds of massive parallel computing problems.

GPU computing is not like having dozens of multi-purpose CPUs at you disposal. It's more like having dozens of scientific calculators each with say 128 bytes of RAM and attached to a shared hard drive. You can load some pretty complex programs onto them but you need to spend time pushing buttons to give them data at the start and to read it back from their screens at the end. They can't access your EVE database, or do any other useful I/O. Unless you can find enough highly parallel math based work for them to do then you spend all you time interacting with the calculators and never get any useful work done yourself. You in this case being a high performance general purpose computer yourself so you're pretty damn good at doing the same things you're spending time telling the calculators to do. Infact you're much, much better, there's just a lot of them.

It's surprisingly difficult to even come up with problems that benefit from GPU computing.

Agrilad
Posted - 2010.08.24 03:51:00 - [88]
 

I really liked this dev blog. Almost need to make my manager read it, along with the rest of my team.

With 50.0% of my professional experience being in C# and the .Net framework. With the rest being T-SQL.

There is a very direct correlation in the problem you saw and the laziness of try..catch programming. Why bother to write good 'if' checks if you can just try..catch the statement and not worry about them? Every catch costs 17.5 ms (or did at that time on that server). Which is all fine and dandy in 10's and 100's of executions, but 1,000s? 10,000s?
Where did this story come from? Well at a very base level, you have to translate a sql cell into an object in code. Some programmer at some point thought that testing whether a datetime is null was best with a try catch. So every nullable datetime field that was null in any db call to build our objects was costing 17.5 ms. So change that to an if check for null, and a massive server performance bottleneck fixed. (That 17.5 ms is the cost if an error occured and the .net framework had to process the error to the catch.)

One of my pet peeves.

Oh and the one course of my comp sci education that I directly apply every day I work, and I code all the way from core framework to UI. Is Algorithmic Analysis. It is as important in filling a UI with data, or changing visible controls, as it is to processing data at a low level.

The 2nd course was Programming Language Design. Basically what the differences are between the current languages and how to use them optimally.

My next favorite was real time computing.

Noun Verber
Gallente
Posted - 2010.08.24 06:04:00 - [89]
 

For people still confused:

Converting the algorithm before fixing it will just allow it to do more stupid things in the same amount of time, but still need to be improved later (possibly by someone who will need to learn the stupid again), so it's best to do it early by someone who understands it.

Ban Doga
Posted - 2010.08.24 06:37:00 - [90]
 

Good riddance!


But now think about how long this issue was present and how long it took you to fix that (and how long before that you did not even try to fix it). IMO there are quite a handful more of those "beauties" hiding in EVE.

Maybe this little bug can show you the difference between your "we are excellent" and your players' "please become excellent again".


Pages: 1 2 [3] 4 5 6 7

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only