open All Channels
seplocked EVE Information Portal
blankseplocked New Dev Blog: Well, that was a few people
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: 1 [2] 3

Author Topic

CCP Veritas

Posted - 2010.11.09 21:04:00 - [31]
 

Originally by: Jamus Gorrelius
So in R10 you have 900 players on a Reinforced Node


Just double checked, that system was not on a reinforced node.

Dalvor Coraneu
Posted - 2010.11.09 21:06:00 - [32]
 

Edited by: Dalvor Coraneu on 09/11/2010 21:37:07
*


Celeritas 5k
Connoisseurs of Candid Coitus
Posted - 2010.11.09 21:16:00 - [33]
 

ITT: Nerds who have never done anything more complicated than installing linux giving CCP advice on how to program their supercomputer.

Jamus Gorrelius
Macabre Votum
Morsus Mihi
Posted - 2010.11.09 21:21:00 - [34]
 

Originally by: CCP Veritas
Originally by: Jamus Gorrelius
So in R10 you have 900 players on a Reinforced Node


Just double checked, that system was not on a reinforced node.


Thank you for that i heard it was but why the tremendus lag and have we got some fixes comming soon that will take game back to before dominion?

And are they going to be released with this expansion or are we going to have to wait again, because with all the mass testing you have done we have seen absolutly no improvment with any patch.

Hawk TT
Caldari
Bulgarian Experienced Crackers
Posted - 2010.11.09 21:45:00 - [35]
 

It seems you've been away for a while...
Why don't you check the Dev Blog archives for lag-related blogs in the last couple of monhts? There are plenty of juicy bits & pieces coming...Twisted Evil

Dev Blogs - September 2010 page

Originally by: Jamus Gorrelius
Originally by: CCP Veritas
Originally by: Jamus Gorrelius
So in R10 you have 900 players on a Reinforced Node


Just double checked, that system was not on a reinforced node.


Thank you for that i heard it was but why the tremendus lag and have we got some fixes comming soon that will take game back to before dominion?

And are they going to be released with this expansion or are we going to have to wait again, because with all the mass testing you have done we have seen absolutly no improvment with any patch.

Andrea Griffin
Posted - 2010.11.09 21:57:00 - [36]
 

Originally by: Celeritas 5k
ITT: Nerds who have never done anything more complicated than installing linux giving CCP advice on how to program their supercomputer.
But... But... The Cloud! It can do anything! ANYTHING! Just wave your hands a little bit!

Seriously, though - whenever I hear someone roll out the all mighty magical cloud as a cure all I start to get violent.

Anyway, I'm really glad that progress is being made here. It's very exciting and I hope you're having some fun in the process (I have fun optimizing code, but I'm weird I guess). Definitely looking forward to some detailed dev blogs down the road!

Herschel Yamamoto
Agent-Orange
Nabaal Syndicate
Posted - 2010.11.09 23:18:00 - [37]
 

Originally by: Celeritas 5k
ITT: Nerds who have never done anything more complicated than installing linux giving CCP advice on how to program their supercomputer.


Hence why some of us are humble enough to phrase our know-it-all-ness in the form of a question Wink

Aldariandra
Gallente
MunsterMunch
The 0rphanage
Posted - 2010.11.09 23:38:00 - [38]
 

Originally by: Andrea Griffin
Originally by: Celeritas 5k
ITT: Nerds who have never done anything more complicated than installing linux giving CCP advice on how to program their supercomputer.
But... But... The Cloud! It can do anything! ANYTHING! Just wave your hands a little bit!

Seriously, though - whenever I hear someone roll out the all mighty magical cloud as a cure all I start to get violent.

Anyway, I'm really glad that progress is being made here. It's very exciting and I hope you're having some fun in the process (I have fun optimizing code, but I'm weird I guess). Definitely looking forward to some detailed dev blogs down the road!


This comment gave me pleasure and satisfaction, and a great inner feeling of agreement, and I wanted to support it by quoting it.


Zendoren
Aktaeon Industries
The Black Armada
Posted - 2010.11.10 01:29:00 - [39]
 

Originally by: CCP Veritas
Originally by: Daedalus II
Ok so ship killing is expensive.
What about trying to prepare that as much as possible before it actually happens?


I've spent the past couple weeks digging at why exactly ship death is so expensive. I don't want to get into the details of that quite yet, but I'm happy to report that I've made a significant optimization to the process which is due for mass testing on Thursday. Think I can wring more out of it too if what I've done proves safe.


So, was it an "Aha!" moment or was it a "that's not supposed to happen" moment?

Taedrin
Gallente
Kushan Industrial
Posted - 2010.11.10 02:57:00 - [40]
 

Originally by: Aldariandra
Originally by: Andrea Griffin
Originally by: Celeritas 5k
ITT: Nerds who have never done anything more complicated than installing linux giving CCP advice on how to program their supercomputer.
But... But... The Cloud! It can do anything! ANYTHING! Just wave your hands a little bit!

Seriously, though - whenever I hear someone roll out the all mighty magical cloud as a cure all I start to get violent.

Anyway, I'm really glad that progress is being made here. It's very exciting and I hope you're having some fun in the process (I have fun optimizing code, but I'm weird I guess). Definitely looking forward to some detailed dev blogs down the road!


This comment gave me pleasure and satisfaction, and a great inner feeling of agreement, and I wanted to support it by quoting it.




To be serious though, if CCP DID manage to make solar systems multi-threaded, it would be a HUGE step towards eliminating the lag monster forever. If they managed to make it embarrassingly parallel (impossible, I know), then CCP could just throw more hardware at the cluster to make things all better.

Too bad that this would probably require a complete rewrite of the server code, though. It would probably also require some core changes to gameplay mechanics too.

Imhothar Xarodit
Minmatar
Wolverine Solutions
Posted - 2010.11.10 04:50:00 - [41]
 

Edited by: Imhothar Xarodit on 10/11/2010 04:56:02
Originally by: CCP Veritas
Originally by: Kikki Di'je
<fancy multi-processing cloud foo>


As current, the Eve server process isn't even multi-threaded at the OS level. We need to get moving down that path, but this level of setup is well beyond what we currently can leverage. It's doubtful that this level of wide processing would be beneficial to our particular load characteristic either, as the communication overhead of splitting a simulation over multiple machines would likely be a tighter bottleneck than the computation.


  1. If you cannot use multiple OS-threads in your Eve server process, how about running multiple Eve server processes? With different tasks: like process 1 is the client proxy, process 2 does the high-priority simulation stuff, process 3 does kill mail generation and other secondary stuff. I guess you have at least dual-cores with Hyper Threading, giving you a minimum of 4 logical cores to work with. You could either pack each core with its own process, or leave up a core for administrative/logging tasks which would not hamper the remaining processes' performance. That might be an easier solution than making the single process multi-threaded (thanks to the GIL...). Pushing messages between processes isn't that expensive with pipes.

  2. Ever thought about utilizing GPGPU for stuff like the physics part? To give you a hint: IBM GPGPU racks. A bit of OpenCL can make N-body simulations go weeeeeee by several magnitudes!

  3. Does Destiny make use of SSE instructions? This might sound silly, but it can speed things up quite a bit.


Ashemi Darkhold
hirr
Morsus Mihi
Posted - 2010.11.10 04:59:00 - [42]
 

Edited by: Ashemi Darkhold on 10/11/2010 05:03:16
Originally by: CCP Veritas
Originally by: Jamus Gorrelius
So in R10 you have 900 players on a Reinforced Node


Just double checked, that system was not on a reinforced node.


For the record I filed the form for the reinforced node for R10 several hours before downtime.
Noteworthy though the notifying character section of the form was bugged and just read "Test 1 Char Test 1 Corporation Test 1 Alliance"

CCP Veritas

Posted - 2010.11.10 10:02:00 - [43]
 

Originally by: Zendoren
So, was it an "Aha!" moment or was it a "that's not supposed to happen" moment?


Pretty boring really. I set up some new profiling tools then self destructed a ship. Then added a bit more instrumentation to see exactly what all the load was and saw that a fair bit of it didn't actually need to be done. Upon looking at the relevant code, someone long long ago had this optimization described in a comment saying it should be done some day if the need was there. Tried it out, got a significant win, so put it in the deployment pipe.

Nothin' special. That's kinda what we do in Gridlock these days.

CCP Veritas

Posted - 2010.11.10 10:12:00 - [44]
 

Originally by: Imhothar Xarodit
If you cannot use multiple OS-threads in your Eve server process, how about running multiple Eve server processes?


It's not that we can't possibly use multiple OS-threads, it's that we currently don't. Doing so is on the roadmap, but is no small task.

Originally by: Imhothar Xarodit
Ever thought about utilizing GPGPU for stuff like the physics part?


This comes up in every lag thread. I have done GPU programming before, and am familiar with the applications it's good for. Right now, the load that we're seeing is not even close to being applicable to a GPU solution.

Originally by: Imhothar Xarodit
Does Destiny make use of SSE instructions? This might sound silly, but it can speed things up quite a bit.


You're assuming that the bulk of the Destiny load is doing heavy math. I haven't been looking deep at Destiny personally lately (CCP Masterplan has), but I don't think that's a safe assumption.

Venkul Mul
Gallente
Posted - 2010.11.10 10:31:00 - [45]
 

Originally by: Elsa Nietzsche
Is there any way non-critical actions could be queued up?
say wrecks, loot, KM creation etc all get written off to a queue that can be processed after the fight. there's really no need for all that to be generated in the heat of battle.


1) Define "after the fight". It is the moment the last shot is fired in system?
Let's make some example of unwanted after effects:

- player X suicide gank someone in high sec. CONCORD arrive, he is destroyed. After all that the battle end and the wreck spawn.
How many opportunistic looters would be there and how slim would be the chance for the suicide ganker alt/corpmate to loot the wreck?

- let's make it a wardec or low sec/0.0 situation, so no CONCORD. it is a "small" fight and your side is loosing or has no time to loot before enemy reinforcements arrive.
You want to deny the loot to the enemy but the wrecks haven't spawned. What you will do? Wait for them to spawn or leave?

- Mission runners: you delay loot creation till after the fight end. You have just killed the marauder utility and made mission even longer. Possibly you have even nerfed ninja looting/salvaging.


2) the data need to be stored somewhere in the meantime and sent to that somewhere. It is possible it will cost more resources than doing it immediately.


Quote:

personally I'd like to see dynamic system population limits. say if there's 100 people in a system and load is high, noone else can jump in until the load stabilizes, then a new limit is set at say 500, etc.
this would allow new fleets to jump in but only when the system can handle them. But i'm sure many will argue against it.



There is a problem there. You are denying reinforcements to one of the groups.

Let's say you jump in system with 100 ships and there are 20 defenders. They have 200 ships next system but can't even try to enter because the server require 3 minutes to clear the jump load.
You gift the attacking side with a 5:1 advantage for 3 minutes.

Same thing if in reverse, the system had 100 defender, you jump in with 200 ships, but only the first 50 pass as the other 150 are delayed for some minute.
Slaughter follow.

Those are very negative consequences.

Lumy
Minmatar
Sebiestor Tribe
Posted - 2010.11.10 10:40:00 - [46]
 

I wonder, how many ships in space can Destiny realistically handle? Any estimates? Not shooting and blowing up stuff, just flying around, changing speed vector, bumping each other and so.

I assume physics simulation will always be a single thread (?), but you could eventually move everything else to other threads/cores.

Lucy PewPew
Posted - 2010.11.10 11:17:00 - [47]
 

Originally by: Elsa Nietzsche

personally I'd like to see dynamic system population limits. say if there's 100 people in a system and load is high, noone else can jump in until the load stabilizes, then a new limit is set at say 500, etc.
this would allow new fleets to jump in but only when the system can handle them. But i'm sure many will argue against it.


A cunning FC could manipulate this to ensure that they maintained superiority in a system. We have many cunning FCs in EVE.

Horisontti
River Styx.
Legion of xXDEATHXx
Posted - 2010.11.10 11:17:00 - [48]
 

You've done pretty good job so far I think. Just some things need to be adjusted. I think the lag thing also leads to some gamebalance issues like remote reps being being very good in lag (no cap, no hurries to lock).

But it was very impressive to have 3300 people in one system, keep up the good work!!!

HeliosGal
Caldari
Posted - 2010.11.10 11:30:00 - [49]
 

NC russians and others please keep bashing each other over the head the game needs to push the limits and then some more

VE Vengeance
Posted - 2010.11.10 11:47:00 - [50]
 

This Devblog was a fun read thx!

So basicly the node was too busy with Destiny to "kill itself" by commiting to heavy-load events? Laughing

So the conclusion of this is, when you reach a huge amount of players in one system, the system ends in a stalemate situation where it's too busy with destiny to die.

Fleet Reimbursement
Posted - 2010.11.10 12:07:00 - [51]
 

I await the other promised blog of historical graphs and pretty pictures.

I just have one question I am sure you can throw this on the back of answering someone else. How suprised were the two GM's that were killed in the battle when they were both popped?

Muul Udonii
Minmatar
THORN Syndicate
BricK sQuAD.
Posted - 2010.11.10 12:11:00 - [52]
 

Question regarding the way the requests are handled:

I've recently discovered the Ctrl Alt Shift M window; and in the LXQ battle I had 14 outstanding requests when my client eventually disconnected.

If I had only sent requests when I had zero outstanding requests would these requests have been processed ahead of the backlog of requests from ther other 3241 pilots in the system?

I ask because in another large laggy battle I was making sure my outstanding requests were very low, and yet I seemed to have the one request I sent waiting a very long time before being activated. In other cases I was able to spam three or four requests at once and get them processed after about the same period of time; but very close together.

Blazde
4S Corporation
Morsus Mihi
Posted - 2010.11.10 12:42:00 - [53]
 

Originally by: CCP Veritas
Originally by: Imhothar Xarodit
Does Destiny make use of SSE instructions? This might sound silly, but it can speed things up quite a bit.


You're assuming that the bulk of the Destiny load is doing heavy math. I haven't been looking deep at Destiny personally lately (CCP Masterplan has), but I don't think that's a safe assumption.

There is a huge amount of math in there and it uses x87, client side at least. I've speculated in the past that it might be using SSE serverside and that difference is one of the remaining causes of desync (especially in 'warping titan into station' type scenarios), OR that even if it's x87 serverside then codepath differences and truncating/not-truncating 80 bit floating point vaules is causing desync. All the desync I've observed in the last year 'comes from' the server. I can't get clients to diverge during their simulation. That could be for plenty of reasons, but it backs up the theory a bit.

As long as Masterplan is looking into it could you pass this on to him? I detailed it better in a bugreport around february '09 with data showing desyncs in the least significant digits of FP values, and suggested switching to SSE since there aren't really any downsides, trying it is probably not much more than tweaking compiler options, it might fix a lot of desync and it might even give a bit of a speed increase.


Very nice blog anyway, neatly explained a few things I've often wondered about.

Originally by: CCP Veritas
... saw that a fair bit of it didn't actually need to be done ... significant win..

Nothin' special. That's kinda what we do in Gridlock these days.

Is there a Team Gridlock fanclub we can join yet? Cool

Ishina Fel
Caldari
Terra Incognita
Intrepid Crossing
Posted - 2010.11.10 12:43:00 - [54]
 

Sooo... did I understnd that right?

In heavy load situations, Destiny is the biggest CPU hog. But by reducing the CPU time consumed by Destiny, you allow the server to choke itself to death by way of processing too many cheap commands that later require heavy follow-up computing.

Therefore, giving the server more CPU time for combat simulations will make the lag worse?

Good god, I don't want your job Laughing *buys the Gridlocks a round of Quafe for consolation*

CCP Veritas

Posted - 2010.11.10 13:04:00 - [55]
 

Originally by: Ishina Fel
Therefore, giving the server more CPU time for combat simulations will make the lag worse?


You got it. The node really wanted to die, but simply didn't have enough rope to make it happen.

But we'll do what needs to be done. Today that means optimizing Destiny and ship death. After those are deployed, we'll have something else to jump on. Each individual step may not benefit the user experience directly, but the sum of them all will add up to a positive.

Ms Freak
Amarr
Immortalis Inc.
Shadow Cartel
Posted - 2010.11.10 13:33:00 - [56]
 

Edited by: Ms Freak on 10/11/2010 13:43:39
Erm.. Just a suggestion here but...

Kill mails, insurance mails etc. Why not just let those get processed at some later date/time?

Yes we all love the kill-mails but why not have a seperate server/process dedicated to it that is extremely low priority. The same as spawning wrecks - loose the ship, create the pod, leave the wreck till later.

I guess essentially what i'm saying is: Why not give everything a priority? Destiny has a high priority, mails need to be the lowest (because if they arrive after 2 hours it's not the end of the world) and wrecks could be somewhere in the middle?

Another thought just crossed my mind reading this: I bet things in system (such as POSes/timers/PI) need some loving too - these could be given lower priorities too?!

The idea around pre-calculating the dropped-items is a nice one - that would almost entirely remove the need for that calculation mid-fight.

LXQ was unplayable really but it sound like CCP know why and are working towards fixing it, which can only be good i guess.

edit:
thinking out loud here but:
assuming you had several "queues" of events catergorized by "priority" (a viable way of implementing the above suggestion) you could then allow each cycle of the server to spend processing time on each queue. So high priority items are given 50% of the cycle processing time, then medium 35%, then low the remaining 15%. This would allow scaling of the queues according to priority whilst still allowing tasks to be completed in a sensible time-frame.

The scale of the fight would then dictate how long it takes to clear all the queues?

Una Achura
Posted - 2010.11.10 15:21:00 - [57]
 

Edited by: Una Achura on 10/11/2010 15:22:03
Originally by: CCP Veritas

The node really wanted to die, but simply didn't have enough rope to make it happen.



LaughingLaughing

TornSoul
BIG
Gentlemen's Agreement
Posted - 2010.11.10 15:30:00 - [58]
 

Edited by: TornSoul on 10/11/2010 15:33:58
Originally by: Ms Freak
LXQ was unplayable

FWIW - During LXQ I was pretty much roaming in what we now jokingly call God Mode.
Barely any lag (2-6 secs) for the entire duration (until that last SBU did some wonky stuff to everyone there)

Every major battle since, it's been the reverse...

Just goes to show...

Ms Freak
Amarr
Immortalis Inc.
Shadow Cartel
Posted - 2010.11.10 15:42:00 - [59]
 

Originally by: TornSoul
Edited by: TornSoul on 10/11/2010 15:33:58
Originally by: Ms Freak
LXQ was unplayable

FWIW - During LXQ I was pretty much roaming in what we now jokingly call God Mode.
Barely any lag (2-6 secs) for the entire duration (until that last SBU did some wonky stuff to everyone there)

Every major battle since, it's been the reverse...

Just goes to show...



Don't get me wrong - the client was working fine, cameras, moving, wapring (to some extent anyway) but in terms of actually getting anything done (like pew pew) it was broke.

And yes, it's quite amazing how it can let 3000+ players fly around a system as long you don't shoot each other yet the second pew pew happens the server tries to hang it's self but is so stressed by the thought it actually keeps plugging away!! Very HappyVery Happy

Shanky McStabber
Merch Industrial
Goonswarm Federation
Posted - 2010.11.10 16:00:00 - [60]
 

Edited by: Shanky McStabber on 10/11/2010 16:00:14
Originally by: Ms Freak


The idea around pre-calculating the dropped-items is a nice one - that would almost entirely remove the need for that calculation mid-fight.



The problem with this idea is the server would then have to recalculate dropped items every time you fired a missile/gun, launched a drone, fired a bubble. You would quickly lose all the gains by the sheer number of small "adjustments" that occur during the fight.


Pages: 1 [2] 3

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only