open All Channels
seplocked EVE Information Portal
blankseplocked New Dev Blog: The Long Lag
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: 1 [2] 3 4 5 6 7

Author Topic

Frug
Omega Wing
Snatch Victory
Posted - 2010.08.17 18:35:00 - [31]
 

That blog was hot.

Makes me aroused.

Dragon Greg
Posted - 2010.08.17 18:39:00 - [32]
 

Originally by: CCP Greyscale
Originally by: Dragon Greg
[...]

But ultimately you have to consider humans being mostly sheep and following combinatory paths of least resistances and social security mechanisms in dealing with objectives game design could come up with. Think of it like the perpetual race between the bullet and the armour.

I'm really curious what the perspective of game design is on this, or rather, of the people responsible for overall product development. This is after all something that can touch on game design and product development format, if not principles. And it is an angle on conflict management which touches on quite a lot more then "just pvp", at minimum by analogy of principle it has a potential to affect all forms of conflict in the simulation.



This is something we think about a lot. I've talked about my thoughts on this stuff with a previous CSM (3 or 4 I think?), and there's some of it in this blog.


Very Happy That blog is actually exactly what inspired my thought process here, I enjoyed that blog immensely. And appreciated its contents and the thoughts behind it. I'm aware of previous CSM topics, but since this does appear to be something that is an ongoing process I wonder whether this is also something that continues with the current CSM 5?

Amazing how multi disciplinary EVE is, as a lot more then just a game.

Any chance you would consider a follow up devblog on the topic? Cool
See, this touches on what many players call "the other half of it", the tech angle and the human/group angle.
Not necessarily for the nullsec angle, although I do think there is a definite element there of that niche serving as a niche that pushes many an envelope, so to speak. Just from an emergent behaviour angle, something which honestly could fit in quite nicely with both surveys, research programs, as well as marketing studies.

linedash
Garoun Investment Bank
Posted - 2010.08.17 18:41:00 - [33]
 

Edited by: linedash on 17/08/2010 18:41:17
Originally by: CCP Oveur
But the first step, and greatest future potential, is to get things to fully capitalize on multi-processor and multi-core. Like spanning a solar system over multiple cores.


Any chance on more detail on this in general?

Originally by: CCP Warlock
Medium term will involve recruiting more cpu by going to multi-core servers.


How come you haven't moved to multi-core machines before now? Surely this would have helped the CPU bound lag issues a lot? Or even just allowed for more cores available to share systems over.

It's always felt like this was CCP being a bit cheap by not doing this before. Honestly curious why you haven't.

CCP Oveur

Posted - 2010.08.17 18:45:00 - [34]
 

Originally by: Tippia
Originally by: CCP Oveur
Originally by: Tippia
…and my only question is: in this Fixing Lag series, are you ever going to address (or even just engage in some wild mass guessing) what it was in Dominion that made the system soil itself compared to how it worked in Apocrypha?
Might want to read the comments on the previous blog in this series, it has your answers.
It had the answer "yes, well… there was lag back then too", which is… kind of a non-answer. The question really remains: why did we see such a drastic increase in non-optimal situations from Dominion and onwards (because that's basically what you were saying — that the same fleet lag happened in Apoc when things the planets weren't aligned right, but this doesn't change that it started to happen far more often after Dominion was deployed)?

What were the steps in this (supposedly) gradual degradation? Why did they all flock together in that one patch? And how were they "differently laggy"? That alone opens up a slew of questions that need to be answered…

What thread are you reading? It had answers like this. Dominion isn't what introduced what you are experiencing today or back then. This is many issues which have been getting worse over time.

And Apocrypha most certainly had laggy situations.

I actually can't believe I'm debating that an expansion had lag while you say it didn't. You feel up to some press interviews?Laughing

Mashie Saldana
Minmatar
Veto Corp
Posted - 2010.08.17 18:46:00 - [35]
 

Originally by: linedash
How come you haven't moved to multi-core machines before now? Surely this would have helped the CPU bound lag issues a lot? Or even just allowed for more cores available to share systems over.

It's always felt like this was CCP being a bit cheap by not doing this before. Honestly curious why you haven't.

The nodes are already on multi core CPUs (2 x dual core per blade if I remember correctly), the problem is that the software can't use more than one core yet.

CCP Oveur

Posted - 2010.08.17 18:48:00 - [36]
 

Originally by: linedash
Edited by: linedash on 17/08/2010 18:41:17
Originally by: CCP Oveur
But the first step, and greatest future potential, is to get things to fully capitalize on multi-processor and multi-core. Like spanning a solar system over multiple cores.


Any chance on more detail on this in general?

Originally by: CCP Warlock
Medium term will involve recruiting more cpu by going to multi-core servers.


How come you haven't moved to multi-core machines before now? Surely this would have helped the CPU bound lag issues a lot? Or even just allowed for more cores available to share systems over.

It's always felt like this was CCP being a bit cheap by not doing this before. Honestly curious why you haven't.


All our servers are multi-processing multi-core machines. She's referring to the server code, which has a solar system bound to a single core today. Enabling a solar system to use two cores would obviously allow it to scale up more.

Wollari
Phoenix Industries
Wicked Nation
Posted - 2010.08.17 18:52:00 - [37]
 

Nice read. Hope to read more stuff like this, even it's huge wall of text. And the presentation i nice aswell.

Thanks.

CCP Warlock

Posted - 2010.08.17 19:01:00 - [38]
 

Originally by: Tippia
Originally by: CCP Oveur
Originally by: Tippia
…and my only question is: in this Fixing Lag series, are you ever going to address (or even just engage in some wild mass guessing) what it was in Dominion that made the system soil itself compared to how it worked in Apocrypha?
Might want to read the comments on the previous blog in this series, it has your answers.
It had the answer "yes, well… there was lag back then too", which is… kind of a non-answer. The question really remains: why did we see such a drastic increase in non-optimal situations from Dominion and onwards (because that's basically what you were saying — that the same fleet lag happened in Apoc when things the planets weren't aligned right, but this doesn't change that it started to happen far more often after Dominion was deployed)?

What were the steps in this (supposedly) gradual degradation? Why did they all flock together in that one patch? And how were they "differently laggy"? That alone opens up a slew of questions that need to be answered…


If we knew we would tell you (honest, gov!). At first we thought it was introduced with Dominion, but after some player comments in the forums, and some detective work by the GM's, the balance of suspicion moved to it being a pre-existing bug that something in Dominion made worse. These really are the nightmare kind of bugs, because as you say, the first question everybody asks when something like this happens, and usually the easiest way to track it down, is 'what changed?'.

Partly too I think, there has been a certain degree of confusion about 'lag', between general performance degradation - and agreed there's also some work we need to do there too - and the specific long lag bug where the jump never fully completes, yet the player is a valid target in the destination system.

Kazuo Ishiguro
House of Marbles
Posted - 2010.08.17 19:10:00 - [39]
 

In the presentation, a figure of 1 message per second is mentioned. How far can this be reduced while maintaining an acceptable game experience? During a fleet engagement, a 99% chance of 0.1/s probably beats an erratic 1/s occasionally dropping to 0.01/s, so has anyone looked into a possible trade-off here?

CCP Warlock

Posted - 2010.08.17 19:12:00 - [40]
 

Originally by: Kazuo Ishiguro
In the presentation, a figure of 1 message per second is mentioned. How far can this be reduced while maintaining an acceptable game experience? During a fleet engagement, a 99% chance of 0.1/s probably beats an erratic 1/s occasionally dropping to 0.01/s, so has anyone looked into a possible trade-off here?


That is indeed something we're thinking about.

Dezolf
Minmatar
DAX Action Stance
Posted - 2010.08.17 19:12:00 - [41]
 

Edited by: Dezolf on 17/08/2010 19:12:54
Wow, a coincidence indeed. An english/polish couple stopped outside my apartment, and came knocking on my door, asking if they could borrow a phone (being in Denmark, neither had any working signal on their cell), their car had a punctured tire and it was a rental, hence no extra tire. So I let them borrow mine, and since the emergency guys told them it'd be an hour, I invited them inside (it was raining cats and dogs)

So we talked a bit, and the husband tells me he was at a conference not too long ago with a lot of computer science geek-types (he had asked what I was studying), and mentioned that there was a ph.d. guy talking about "this computer game, I think it was called eeeeve..? I think it's icelandic or something"

At first, it was a coincidence that he had heard about eve (since I don't know a lot of people who know it) and now, this!

(edit: left out why they stopped to borrow a phone :P)

Master Akira
Shiva
Morsus Mihi
Posted - 2010.08.17 19:14:00 - [42]
 

I know that asking for a timeline for that "first step" of putting a solar system across multiple threads/cores might be too much to ask, but... can you tell us something about it, even if it's a very rough estimate? Note that this is unrelated to the "fixing teh lagz" timeline, as I understand that even if lag were to be automagically fixed tomorrow, you guys still need to move to a multithreaded architecture at some point...

Do you guys have at least some internal prototype of that? or that's still "on the plans"?

Tippia
Caldari
Sunshine and Lollipops
Posted - 2010.08.17 19:14:00 - [43]
 

Originally by: CCP Oveur
What thread are you reading? It had answers like this. Dominion isn't what introduced what you are experiencing today or back then. This is many issues which have been getting worse over time.
The one after that, which, on second thought, might have more to do with the lag hunting, than with the solutions… Embarassed

But yes, that one had better answers.
Quote:
I actually can't believe I'm debating that an expansion had lag while you say it didn't. You feel up to some press interviews?Laughing
Oh, it's not that it didn't have lag — just less, which in many cases was good enough… no wait, I didn't say that!! It wasn't good enough!!1 Razz

Obsidian Hawk
RONA Corporation
RONA Directorate
Posted - 2010.08.17 19:16:00 - [44]
 

Too much wall of text.

Needs more pretty pictures and graphs.


But still a great read, though i have to read it again to make sure I understand it.


Smelly Bait
Posted - 2010.08.17 19:19:00 - [45]
 

Edited by: Smelly Bait on 17/08/2010 19:20:43
Dear CCP warlock,

Im pretty sure fleet finder is the big lagger or is involved in it. Im really curious in a sis test without fleet finder


SolidHadden
Minmatar
Sphere Industries
Posted - 2010.08.17 19:24:00 - [46]
 

Dear CCP

Thank you for a detailed blog like this and the dev replies!

Solid

PS Did I understand it correct that Warlock is a woman? and she wrote and understood that blog?
date me? I'm good looking!

CCP Warlock

Posted - 2010.08.17 19:25:00 - [47]
 

Originally by: Master Akira
I know that asking for a timeline for that "first step" of putting a solar system across multiple threads/cores might be too much to ask, but... can you tell us something about it, even if it's a very rough estimate? Note that this is unrelated to the "fixing teh lagz" timeline, as I understand that even if lag were to be automagically fixed tomorrow, you guys still need to move to a multithreaded architecture at some point...

Do you guys have at least some internal prototype of that? or that's still "on the plans"?


No, sorry I can't. I don't like to make commitments of any kind until we are absolutely sure we can nail it to the wall, and there are a lot of uncertainties in this one - as well as the need, as earlier posters pointed out, of very carefully and thoroughly testing it out.

There is a lot of motivation to do this though, the very least of which is there are some very cool server toys, I mean advanced high performance multi-core systems it will let us play with.

Psihius
Caldari
Anarchist Dawn
U N K N O W N
Posted - 2010.08.17 19:39:00 - [48]
 

Edited by: Psihius on 17/08/2010 19:41:00
Edited by: Psihius on 17/08/2010 19:40:07
Originally by: CCP Warlock

There is a lot of motivation to do this though, the very least of which is there are some very cool server toys, I mean advanced high performance multi-core systems it will let us play with.

I know the feeling, I was crazy happy when I got my hands on 2x Quad Core Xeon based server with 8 GB of RAM and 3 15rpms SAS HDD's a few years ago.
I wish you find that bug fast and get to play with the "toys" Very Happy Twisted Evil

Master Akira
Shiva
Morsus Mihi
Posted - 2010.08.17 19:40:00 - [49]
 

Originally by: CCP Warlock
Originally by: Master Akira
I know that asking for a timeline for that "first step" of putting a solar system across multiple threads/cores might be too much to ask, but... can you tell us something about it, even if it's a very rough estimate? Note that this is unrelated to the "fixing teh lagz" timeline, as I understand that even if lag were to be automagically fixed tomorrow, you guys still need to move to a multithreaded architecture at some point...

Do you guys have at least some internal prototype of that? or that's still "on the plans"?


No, sorry I can't. I don't like to make commitments of any kind until we are absolutely sure we can nail it to the wall, and there are a lot of uncertainties in this one - as well as the need, as earlier posters pointed out, of very carefully and thoroughly testing it out.

There is a lot of motivation to do this though, the very least of which is there are some very cool server toys, I mean advanced high performance multi-core systems it will let us play with.


I figured that giving a timeline wasn't going to happen. I wasn't really asking for a release date of finished code, but more inquiring about the general state of that development.

So the question is, have you started doing internal prototypes and experiments on that regard? are you trying to break the toys already? Very Happy

Being such an interesting challenge, I can only imagine (and hope) that you are already moving in that direction with early experimental code...

Hamish Nuwen
Gallente
Escuadron Federal de Asalto
Posted - 2010.08.17 19:40:00 - [50]
 

Originally by: CCP Oveur
Originally by: Hamish Nuwen
In the long run, you should stop thinking.

This. Bliss.

Meanwhile, you can continue thinking about:
  • Weapon grouping at squadron/wing/fleet level

  • No cargohold (=database) dependencies in combat (ammunition bays, preloaded scripts, ...)

  • Maybe a explicit combat mode (you can't open assets, or market or fitting, any not related with combat) with messaging priority (obviously only those related to combat)

  • Isolate client input check (anti-cheat) code, common to all ships/players, from rest of the code (hey, this is a computing task), and scheduling in another processor/cores

  • When real-time is a must, ACID is not law (ex. NoSQL movement in Web-based apss)

  • More

Bomberlocks
Minmatar
CTRL-Q
Posted - 2010.08.17 19:49:00 - [51]
 

Originally by: CCP Warlock
...I had a pet theory that it was a TCP rate adaption issue, in conjunction with a system lock affecting multiple clients. No such luck. We narrowed it down a little after I accidentally left myself logged on while stuck overnight, and came in the next morning and found I was successfully jumped into the system...
Firstly, I'd like to thank you for a forthright, highly detailed blog that exhibits none of the defensiveness we've come to expect, and one that comes straight to the point without trying to complicate the issue by making it too bland for us armchair software engineering warriors to read.

Then, I'd like to say that (from my utterly clueless perspective) a networking transport layer problem was also high on my list of probable contributing circumstances. You mention TCP rate adaption. I had a certain hunch that the statefullness of the TCP layer could be causing a feedback loop due to lost packets forcing retransmits and thereby slowing the network down under high load. This could appreciably happen at any point where TCP is being used, or between any points, or combinations of them, such as external networking equipment, the database network handling code (you've had problems with this before if I remember correctly; the starvation issue) and the various software processes that use networking code themselves.

If I'm not mistaken, the realtime game "time" is done in ticks, and the various processes that handle system, ship, fleet, grid etc are all statefull. If the ticks are stateless, i.e. they drive the whole game as a master time system, or even only a node, could it not be that the events or changes of state that are supposed to be being triggered are not receiving the information they need in time (especially if time in game process is handled statefully), leading to corrupt state in various objects?

Please forgive me again for my amateurish attempt to make a mental picture of what is happening, but certain correlations of the "lag", like the empty system lag backwash, or the empty system gate issues make me wonder if it isn't a combination of low level high load effects causing or contributing to software process degradation?

This was what caused me to ask in my post in CCP Tanis' blog how error or failure tolerant the code is, or how errors were handled in stateful objects.

I would be most grateful if you could take a minute or two to correct me in my idiocy.

Tiger's Spirit
Caldari
Posted - 2010.08.17 19:51:00 - [52]
 

This is a big nothing raving wall of text. :D

CCP Oveur

Posted - 2010.08.17 19:54:00 - [53]
 

Originally by: Hamish Nuwen
Originally by: CCP Oveur
Originally by: Hamish Nuwen
In the long run, you should stop thinking.

This. Bliss.

Meanwhile, you can continue thinking about:
  • Weapon grouping at squadron/wing/fleet level

  • No cargohold (=database) dependencies in combat (ammunition bays, preloaded scripts, ...)

  • Maybe a explicit combat mode (you can't open assets, or market or fitting, any not related with combat) with messaging priority (obviously only those related to combat)

  • Isolate client input check (anti-cheat) code, common to all ships/players, from rest of the code (hey, this is a computing task), and scheduling in another processor/cores

  • When real-time is a must, ACID is not law (ex. NoSQL movement in Web-based apss)

  • More


You seriously do not want me to think about solving technical issues. Last time that happened unicorns died, dolphins left earth and war broke out in a galaxy far far away.

Bomberlocks
Minmatar
CTRL-Q
Posted - 2010.08.17 20:01:00 - [54]
 

Originally by: CCP Oveur
Originally by: Hamish Nuwen
Originally by: CCP Oveur
Originally by: Hamish Nuwen
In the long run, you should stop thinking.

This. Bliss.

Meanwhile, you can continue thinking about:
  • Weapon grouping at squadron/wing/fleet level

  • No cargohold (=database) dependencies in combat (ammunition bays, preloaded scripts, ...)

  • Maybe a explicit combat mode (you can't open assets, or market or fitting, any not related with combat) with messaging priority (obviously only those related to combat)

  • Isolate client input check (anti-cheat) code, common to all ships/players, from rest of the code (hey, this is a computing task), and scheduling in another processor/cores

  • When real-time is a must, ACID is not law (ex. NoSQL movement in Web-based apss)

  • More


You seriously do not want me to think about solving technical issues. Last time that happened unicorns died, dolphins left earth and war broke out in a galaxy far far away.
No, but he is making some very good suggestions pertaining to reducing the overall number of requests (along the lines of the fleet jump suggestion) and the number of processes that need to react under high load.

Pity about the unicorns. They're funky creatures.

LHA Tarawa
Posted - 2010.08.17 20:21:00 - [55]
 

I work for a small company you may have heard of... IBM. Well, we were just bought by them. Before IBM we were a little company you never heard of called Initiate Systems.

Long-story short, we take billions of records (could be medical records such as doctor visits, billing, perscription, insurance; could be government records such as tax records, dirver's licenses, birth records, tickets, police incident reports (crime reports); heck we even do bank transactions, phone calls, Point of Sale transations....)

Anyway, we take these billions of records and then run a probabilistic scoring and matching, highly configurable algorithm to link records that are really the same person, place, thing... We then build relationships between the objects...

Oh, sounds easy right? Wrong.

A new record comes in, we have score that record against the billions of records already in the system to see where it goes. Then we have to check to see if that new record has changed the relationship between the object that it belogs to and all other objects in teh system.

So, what's my point. Yeah, this stuff is hard.

We can tell the customer to get more servers to process transactions, but that just shifts the bottleneck to the database. So, we have them get faster disk arraies, more memory for cache, and better design their databse schema... Which puts the bottleneck on our ETL brokers that pre-process transations on the way in and out....

Then they get so fast that they are bringing in so many new records that the hash buskets are getting too large more frequently, so we make them redo their frequency counts and rebucket more often which... (okay, getting too technical here)

In short, there is always a bottleneck. When we fix one bottleneck, the next bottleneck it revealed.

AND, this is hard stuff. As mentioned in the dev blog.... trying to log performance against millions of transactions creates gigs of log data that can't be sifted and create their own performance issues.

There are some REALLY basic things you know to do, and assuming you've done those your performace is okay. Getting above "just okay" is where it gets really, really hard.


In the end, if performance really isn't what teh customer was hoping because they have too complicated of a matching and scoring algorithm, eventually we have to tell the customer, sorry, but you are simply going to have to limit the candidate set and the complexity of teh scoring algorithm....

EVE's equivilant would be to simply limit how many characters/ships/drones... whatever's are allowed to be on the same grid. Now, work that into a coheasive game design!

Guess we could go "turned based".... Instead of your guns firing every x seconds, they fire every x melee rounds.... Ugh, token ring instead of ethernet style communication... SUCK.

Callipygian Provocateur
Posted - 2010.08.17 20:24:00 - [56]
 

Excellent blog. Thanks for the fantastic information. It's very refreshing to get a glimpse of some of the efforts going on behind the scenes.

On the topic of game design and scaling, with the current EVE battle mechanics, the bigger blob wins. There is at least one option to combat blobs: area affect weapons. The doomsday devices seem to have been put in the game specifically for this reason. However, they were unbalanced. The few AoE weapons currently in the game don't really scale to large fleet fights. Designing weapons systems that do (and are balanced), clearly, is the purview of a different team, but I think they are the solution to the long-term problem of blobbing.

Again, keep up the awesome work.

Jason Edwards
Internet Tough Guy
Spreadsheets Online
Posted - 2010.08.17 20:31:00 - [57]
 

I hope you patented the idea.

Manaxus Stormwolf
Posted - 2010.08.17 20:31:00 - [58]
 

hey there

I am a comp sci grad student at University of Washington CSS dept. Can you please send me a link to your doctoral paper you are referencing in the article? I'd like to read it.

Thanks!

Malen Nenokal
The Nightshift
Posted - 2010.08.17 21:02:00 - [59]
 

Speaking of scaling the technical side of game design with fleet fights to combat lag. The talk about hierarchical and distributed topology for data communication made me think about the alternative of "Representative Democracy" to speed up the democratic process.

For example, I'm curious if the fleet formations talked about at fanfest 2009 will help in the way that weapon grouping did as far as sending condensed data calls instead of 8 at a time; or in the case of a fleet formation, 20-50 ships synced to single calls for things like movement, jumping, warping or even weapon cycling. Compensated by some sort of hefty bonus, of course. YARRRR!!

Swidgen
Posted - 2010.08.17 21:17:00 - [60]
 

Originally by: LHA Tarawa
... we take billions of records (could be medical records such as doctor visits, billing, perscription, insurance; could be government records such as tax records, dirver's licenses, birth records, tickets, police incident reports (crime reports); heck we even do bank transactions, phone calls, Point of Sale transations....)

Anyway, we take these billions of records and then run a probabilistic scoring and matching, highly configurable algorithm to link records that are really the same person, place, thing... We then build relationships between the objects...

OT, but that sounds illegal in many jurisdictions. If you ever come across any of MY personal records, I forbid you to use them for anything unless you pay me USD $1,000,000 first. You have been put on notice.

Aside from that, nice devblog, but doesn't seem to have any real-world relationship to Eve as it exists today. Would be perfect if the game's software and systems were being re-designed from square one today. Applicability to improving the current state of the game? Very little, I fear.


Pages: 1 [2] 3 4 5 6 7

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only