open All Channels
seplocked EVE Information Portal
blankseplocked New Dev Blog: The Long Lag
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: [1] 2 3 4 5 6 7

Author Topic

CCP Fallout

Posted - 2010.08.17 16:31:00 - [1]
 

Our "Fixing Lag" series continues with CCP Warlock's excellent look at the problems inherent with distributed real-time systems in her dev blog.

Mashie Saldana
Minmatar
Veto Corp
Posted - 2010.08.17 16:32:00 - [2]
 

Edited by: Mashie Saldana on 17/08/2010 16:45:56
Back at FanFest 2007 it was a lot of talk about HPC and Infiniband, whatever happened to that? Was it rejected as unsuitable for EVE or is it awaiting a rewrite of the core code?


Quote:
Probably the most frustrating part is that based on past experience, when we do find this issue (or issue) it will be something that, in retrospect, appears incredibly obvious and silly to have caused so much pain.

Now isn't that the truth regarding all intermittent, hard to locate isses?

Zarofdium
Caldari
The Illuminatii
Mildly Intoxicated
Posted - 2010.08.17 16:38:00 - [3]
 

It is reassuring to hear of the dedication to these vital topics. Fixing the choke hold on large scale fleet combat will continue to drive players to this game for years to come.

Joe SMASH
You Got A Purty Mouth
Posted - 2010.08.17 16:39:00 - [4]
 

Notify: Wall Of Text hits you for 56987 wrecking. Your eyes is out of your control.

<pounds on warp>

Shocked

CCP Warlock

Posted - 2010.08.17 16:46:00 - [5]
 

Originally by: Joe SMASH
Notify: Wall Of Text hits you for 56987 wrecking. Your eyes is out of your control.

<pounds on warp>

Shocked


You think this is bad, you should see some of the textbooks they make us read...

Joe SMASH
You Got A Purty Mouth
Posted - 2010.08.17 17:00:00 - [6]
 

Originally by: CCP Warlock
Originally by: Joe SMASH
Notify: Wall Of Text hits you for 56987 wrecking. Your eyes is out of your control.

<pounds on warp>

Shocked


You think this is bad, you should see some of the textbooks they make us read...


All joking aside. Thanks for keeping us informed and such about the lag monster. o7

Mynxee
Veto.
Veto Corp
Posted - 2010.08.17 17:03:00 - [7]
 

Thrilled to see this dev blog, thrilled to check it off as Delivered on the CSM's "CCP Deliverables List" from the Summit, and now actually gonna get a nice cup of coffee and read it. Thanks, Warlock.


Satsuki Kakuri
Amarr
Posted - 2010.08.17 17:07:00 - [8]
 

Quote:
Probably the most frustrating part is that based on past experience, when we do find this issue (or issue) it will be something that, in retrospect, appears incredibly obvious and silly to have caused so much pain.


Working on games myself, that is my bane more often then naught, and no one else on the entire internet has ever run into a similar problem. And then it turns out to be some stupid checkbox hidden away five menus deep, found four days later. Good luck.

CCP Oveur

Posted - 2010.08.17 17:09:00 - [9]
 

Originally by: Mynxee
Thrilled to see this dev blog, thrilled to check it off as Delivered on the CSM's "CCP Deliverables List" from the Summit, and now actually gonna get a nice cup of coffee and read it. Thanks, Warlock.



Why isn't that list in the Evelopedia?

Vir Hellnamin
Gradient
Electus Matari
Posted - 2010.08.17 17:12:00 - [10]
 

Any news on the 'why the Fleet Finder lags client'? (And, why then switching to the Broadcasts tab, closing whole Fleet Comms, and opening it up again from Neocom bar, fixes the damn update-flood? lag...)

Anyways, appreciate the Wall o' Blogs - even if no pretty pictures :( - nice to have something to read during coffee-break. :)

Myxx
Atropos Group
Posted - 2010.08.17 17:14:00 - [11]
 

Edited by: Myxx on 17/08/2010 17:14:47
Originally by: CCP Oveur
Originally by: Mynxee
Thrilled to see this dev blog, thrilled to check it off as Delivered on the CSM's "CCP Deliverables List" from the Summit, and now actually gonna get a nice cup of coffee and read it. Thanks, Warlock.



Why isn't that list in the Evelopedia?

OMG ITS OVEUR! Hes alive?!

Mynxee
Veto.
Veto Corp
Posted - 2010.08.17 17:17:00 - [12]
 

Edited by: Mynxee on 17/08/2010 18:09:14
Originally by: CCP Oveur
Originally by: Mynxee
Thrilled to see this dev blog, thrilled to check it off as Delivered on the CSM's "CCP Deliverables List" from the Summit, and now actually gonna get a nice cup of coffee and read it. Thanks, Warlock.



Why isn't that list in the Evelopedia?


A good question! Simple oversight, actually. I will add it shortly.
Update: Done.

Master Akira
Shiva
Morsus Mihi
Posted - 2010.08.17 17:17:00 - [13]
 

Good read. Keep the blogs coming! This community isn't afraid of technobabble.

(Also Oveur, that picture in your FB page /o\)

CCP Oveur

Posted - 2010.08.17 17:18:00 - [14]
 

Originally by: Mashie Saldana
Edited by: Mashie Saldana on 17/08/2010 16:45:56
Back at FanFest 2007 it was a lot of talk about HPC and Infiniband, whatever happened to that? Was it rejected as unsuitable for EVE or is it awaiting a rewrite of the core code?


Within "HPC" we've done quite a few things but Infiniband specifically is still being considered. It wouldn't give any huge benefits today just by switching to Infiniband (none even I think), it's more that it opens up new possibilities between physical machines with the associated software changes.

But the first step, and greatest future potential, is to get things to fully capitalize on multi-processor and multi-core. Like spanning a solar system over multiple cores.

Genya Arikaido
Posted - 2010.08.17 17:19:00 - [15]
 

Originally by: CCP Warlock
Originally by: Joe SMASH
Notify: Wall Of Text hits you for 56987 wrecking. Your eyes is out of your control.

<pounds on warp>

Shocked


You think this is bad, you should see some of the textbooks they make us read...


Is it bad that I realized after reading the above quote that I took for granted that I understood your blog and followed what you were saying? That said, amazing blog. Answered in great detail many of my questions and concerns. ♥ for the induced nerdgasm.... Very Happy

Ohai Oveur! Welcome to this blog comment thread. Looking forward to your troll-arse-kicking-1337-epic-skillz here too...

CCP Oveur

Posted - 2010.08.17 17:19:00 - [16]
 

Originally by: Master Akira
Good read. Keep the blogs coming! This community isn't afraid of technobabble.

(Also Oveur, that picture in your FB page /o\)


I'm single, I figured spandex could help.

Elsa Nietzsche
Posted - 2010.08.17 17:24:00 - [17]
 

Great dev blog. Would read again.
I sincerely hope CCP (management) has learned that while users crave features we demand quality. Not to be offensive, but I'll make it simple for you:
1. quality
2. features

When your users are mostly bothering you for features, it means they are happy about the quality of work you do. And as a software engineer myself, I think most of us would rather put out good quality work over a good quantity of work any day.
Regarding the 'beat your head against a wall looking for the root cause', HAHAHA, isn't that why we enter this field? I mean, I have a love/hate relationship with debugging. It's maddening to go through but nothing is more awesome than when you find the solution. Personally, I have a hard time not running around the office telling everyone "holy ****, look what i just solved!"

I think it'd be cool to have a developer sticky where they can post the issues they solved and we can cheer them on, but that's just me.

Finally, is CCP Warlock's dissertation published? Could I get a link/copy? I'd really like to read it.

Thanks for the awesome work guys. Keep it up!

Dragon Greg
Posted - 2010.08.17 17:29:00 - [18]
 

Edited by: Dragon Greg on 17/08/2010 17:36:05
Quote:
From time to time we also discuss scaling issues with game design, since that is the only place where some of these distributed scaling problems can be solved. The longer term view on fleet fight performance lag is that whilst we can and will maximize performance within any single server's area of space, we are going to have to continue to work on game design to somehow limit the number of people that can be simultaneously in that space.


If Warlock or Oveur could elaborate on that, would be appreciate. Are you considering distributed objectives as an iteration to Dominion? Or is this a line of thought towards applying constraints.

Simply put, is there a line of thought possibly being followed akin to "encouraging/requiring" or one of "enforcing/limiting".

I should elaborate I think. These days, common doctrine within the limitations imposed by the mechanism provided in Dominion, is the difference between conflict by engagement and conflict by numbers. Or as some call it "win by objective aka bring the second accounts and push the node to its limits consciously". Also known as the worst kind of sheep inspired blob warfare, but common doctrine. Blob to tax or kill the node and you win.

Blanket application of a principle inspired on enforce / limit seems as such .. unhealthy. But an angle of encourage / require seems to me like carrying a requirement of iterating on Dominion concepts, distributing objectives and requirements of presence and activity, etc.

So, the concern should be more clear.

kano donn
New Path
Posted - 2010.08.17 17:42:00 - [19]
 

From everyone that plays EVE, i would suspect there are a few people that know quite a bit about the tech involved with finding this issue.

Why not make the real nit and gritty technical problems public so that as a community, we may look and offer insight.

Think of it as a form of volunteer developers.

Helios Black
Viziam
Posted - 2010.08.17 17:45:00 - [20]
 

Typos and grammatical errors for the lose in that powerpoint... Proofread much?

Good info though, thanks.

Indeterminacy
THORN Syndicate
BricK sQuAD.
Posted - 2010.08.17 17:47:00 - [21]
 

Originally by: Dragon Greg
Edited by: Dragon Greg on 17/08/2010 17:36:05
Quote:
From time to time we also discuss scaling issues with game design, since that is the only place where some of these distributed scaling problems can be solved. The longer term view on fleet fight performance lag is that whilst we can and will maximize performance within any single server's area of space, we are going to have to continue to work on game design to somehow limit the number of people that can be simultaneously in that space.


If Warlock or Oveur could elaborate on that, would be appreciate. Are you considering distributed objectives as an iteration to Dominion? Or is this a line of thought towards applying constraints.

Simply put, is there a line of thought possibly being followed akin to "encouraging/requiring" or one of "enforcing/limiting".

I should elaborate I think. These days, common doctrine within the limitations imposed by the mechanism provided in Dominion, is the difference between conflict by engagement and conflict by numbers. Or as some call it "win by objective aka bring the second accounts and push the node to its limits consciously". Also known as the worst kind of sheep inspired blob warfare, but common doctrine. Blob to tax or kill the node and you win.

Blanket application of a principle inspired on enforce / limit seems as such .. unhealthy. But an angle of encourage / require seems to me like carrying a requirement of iterating on Dominion concepts, distributing objectives and requirements of presence and activity, etc.

So, the concern should be more clear.



Awfully fancy way of saying...

I hope you're going to change game properties such that fewer people are required in the same place at the same time to achieve a given objective. I'm sure many of us can think of examples where this could be done.

I would guess many hope you're not going to... "Notify: System limit of n players reached, jump request denied."

Tippia
Caldari
Sunshine and Lollipops
Posted - 2010.08.17 17:55:00 - [22]
 

…and my only question is: in this Fixing Lag series, are you ever going to address (or even just engage in some wild mass guessing) what it was in Dominion that made the system soil itself compared to how it worked in Apocrypha?

CCP Oveur

Posted - 2010.08.17 17:57:00 - [23]
 

Originally by: Tippia
…and my only question is: in this Fixing Lag series, are you ever going to address (or even just engage in some wild mass guessing) what it was in Dominion that made the system soil itself compared to how it worked in Apocrypha?

Might want to read the comments on the previous blog in this series, it has your answers.

CCP Warlock

Posted - 2010.08.17 17:58:00 - [24]
 

Originally by: Dragon Greg
Edited by: Dragon Greg on 17/08/2010 17:36:05
Quote:
From time to time we also discuss scaling issues with game design, since that is the only place where some of these distributed scaling problems can be solved. The longer term view on fleet fight performance lag is that whilst we can and will maximize performance within any single server's area of space, we are going to have to continue to work on game design to somehow limit the number of people that can be simultaneously in that space.


If Warlock or Oveur could elaborate on that, would be appreciate. Are you considering distributed objectives as an iteration to Dominion? Or is this a line of thought towards applying constraints.

Simply put, is there a line of thought possibly being followed akin to "encouraging/requiring" or one of "enforcing/limiting".

I should elaborate I think. These days, common doctrine within the limitations imposed by the mechanism provided in Dominion, is the difference between conflict by engagement and conflict by numbers. Or as some call it "win by objective aka bring the second accounts and push the node to its limits consciously". Also known as the worst kind of sheep inspired blob warfare, but common doctrine. Blob to tax or kill the node and you win.

Blanket application of a principle inspired on enforce / limit seems as such .. unhealthy. But an angle of encourage / require seems to me like carrying a requirement of iterating on Dominion concepts, distributing objectives and requirements of presence and activity, etc.

So, the concern should be more clear.



To be clear, while we can advise game design on their scaling constraints, it's their responsibility to design the game. (Well, apart from providing them with the occasional - 'don't even think of doing that to the cluster' moment.)

The real challenge is always to design the system itself so that it will scale, and also to provide a good game experience. Ideally the constraints, there are always some, are essentially part of the game experience and accepted as such. So to answer your question, we certainly don't like the times when we have to actually enforce hard limits, even if realistically there's no immediate alternative.

For the short term we are concentrating on improving server and cluster performance, and fixing the long lag problem. Medium term will involve recruiting more cpu by going to multi-core servers. Longer term redesigning fleet fights to scale indefinitely by changing the game mechanics to allow the cluster to distribute load arbitrarily in them is what I would personally describe as an interesting problem. A very interesting problem, if the goal is also to still deliver a good experience.


Dragon Greg
Posted - 2010.08.17 18:13:00 - [25]
 

Edited by: Dragon Greg on 17/08/2010 18:22:48

Originally by: CCP Warlock


To be clear, while we can advise game design on their scaling constraints, it's their responsibility to design the game. (Well, apart from providing them with the occasional - 'don't even think of doing that to the cluster' moment.)

The real challenge is always to design the system itself so that it will scale, and also to provide a good game experience. Ideally the constraints, there are always some, are essentially part of the game experience and accepted as such. So to answer your question, we certainly don't like the times when we have to actually enforce hard limits, even if realistically there's no immediate alternative.

For the short term we are concentrating on improving server and cluster performance, and fixing the long lag problem. Medium term will involve recruiting more cpu by going to multi-core servers. Longer term redesigning fleet fights to scale indefinitely by changing the game mechanics to allow the cluster to distribute load arbitrarily in them is what I would personally describe as an interesting problem. A very interesting problem, if the goal is also to still deliver a good experience.




That I understand, and while I agree it is interesting I would not describe it as a problem. More as a one step at a time challenge.

But this does touch on a game design level, where with all due respect there should be a near continuous communication between the various teams in order to observe user behavioral patterns. Look at it this way, the math is one factor, the technology another, but the social dynamic is as much of a constant (even if highly variable) as other elements involved in this. It leaves me wondering whether CCP has something akin to a resident sociologist to deal with this challenge of bridging the math / technology / human behavioral patterning.

I do think that there is no prospect for infinite scaling. Yes, there are various theoretical scenarios to consider. But ultimately you have to consider humans being mostly sheep and following combinatory paths of least resistances and social security mechanisms in dealing with objectives game design could come up with. Think of it like the perpetual race between the bullet and the armour.

I'm really curious what the perspective of game design is on this, or rather, of the people responsible for overall product development. This is after all something that can touch on game design and product development format, if not principles. And it is an angle on conflict management which touches on quite a lot more then "just pvp", at minimum by analogy of principle it has a potential to affect all forms of conflict in the simulation.

From a business perspective I can fully understand the timeline you sketch, don't misunderstand it, but from the prospect of a customer of this one product that leaves a bit of a perception gap. From their perspective, you would be looking at a parallel process, but I imagine resource constraints prevent this?

ElfeGER
Deep Core Mining Inc.
Posted - 2010.08.17 18:15:00 - [26]
 

might be worth looking at some tools like Scribe to improve the logging

Hamish Nuwen
Gallente
Escuadron Federal de Asalto
Posted - 2010.08.17 18:22:00 - [27]
 

Originally by: CCP Oveur
But the first step, and greatest future potential, is to get things to fully capitalize on multi-processor and multi-core. Like spanning a solar system over multiple cores.


In the long run, you should stop thinking in terms of "geography" events if you want a truly distributed system load. Consider the work in terms of "computing tasks" (you known, strong cohesion - loose coupling) and its CPU requeriments is the key.

Tippia
Caldari
Sunshine and Lollipops
Posted - 2010.08.17 18:25:00 - [28]
 

Originally by: CCP Oveur
Originally by: Tippia
…and my only question is: in this Fixing Lag series, are you ever going to address (or even just engage in some wild mass guessing) what it was in Dominion that made the system soil itself compared to how it worked in Apocrypha?
Might want to read the comments on the previous blog in this series, it has your answers.
It had the answer "yes, well… there was lag back then too", which is… kind of a non-answer. The question really remains: why did we see such a drastic increase in non-optimal situations from Dominion and onwards (because that's basically what you were saying — that the same fleet lag happened in Apoc when things the planets weren't aligned right, but this doesn't change that it started to happen far more often after Dominion was deployed)?

What were the steps in this (supposedly) gradual degradation? Why did they all flock together in that one patch? And how were they "differently laggy"? That alone opens up a slew of questions that need to be answered…

CCP Greyscale

Posted - 2010.08.17 18:30:00 - [29]
 

Originally by: Dragon Greg
[...]

But ultimately you have to consider humans being mostly sheep and following combinatory paths of least resistances and social security mechanisms in dealing with objectives game design could come up with. Think of it like the perpetual race between the bullet and the armour.

I'm really curious what the perspective of game design is on this, or rather, of the people responsible for overall product development. This is after all something that can touch on game design and product development format, if not principles. And it is an angle on conflict management which touches on quite a lot more then "just pvp", at minimum by analogy of principle it has a potential to affect all forms of conflict in the simulation.



This is something we think about a lot. I've talked about my thoughts on this stuff with a previous CSM (3 or 4 I think?), and there's some of it in this blog.

CCP Oveur

Posted - 2010.08.17 18:33:00 - [30]
 

Originally by: Hamish Nuwen
In the long run, you should stop thinking.

This. Bliss.


Pages: [1] 2 3 4 5 6 7

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only