open All Channels
seplocked EVE Information Portal
blankseplocked New Dev Blog: Well, that was a few people
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: [1] 2 3

Author Topic

CCP Fallout

Posted - 2010.11.09 15:52:00 - [1]
 

CCP Veritas' newest dev blog takes a look at two recent fleet fights, one the record 3,200+ fight, the other the 1,200 fight the next day, and goes into detail about the differences between the two fights. Read all about it here.

Razin
The xDEATHx Squadron
Legion of xXDEATHXx
Posted - 2010.11.09 16:01:00 - [2]
 

Edited by: Razin on 09/11/2010 16:02:01
Was stuck loading grid for hours after a titan bridge into LXQ. Either the node should have died or we should be able to cancel out of trying to load the destination system after some time. The way it went the GMs refused to move anyone involved even though they knew around 600 were not going to be able to load the system.

Hemmo Paskiainen
Gallente
Posted - 2010.11.09 16:19:00 - [3]
 


Haniblecter Teg
F.R.E.E. Explorer
The Initiative.
Posted - 2010.11.09 16:20:00 - [4]
 

Edited by: Haniblecter Teg on 09/11/2010 16:20:28
heh, and the 1200 goofs in azn who forgot to make a fleet fight form crashed theirs.

Letrange
Minmatar
Red Horizon Inc
Cascade Imminent
Posted - 2010.11.09 16:23:00 - [5]
 

With this in mind, FC's may want to start planning around no node deaths if they suspect the node will get re-enforced... Since CCP seems to have removed the ability of the player base to crash re-enforced nodes... Just saying...

I look forward to further fallout from this battle. Technologically speaking. Couldn't give a damn about the NC/Russian tiff.

Elsebeth Rhiannon
Minmatar
Gradient
Electus Matari
Posted - 2010.11.09 16:27:00 - [6]
 

Quote:
I spent an embarrassing portion of my evening excitedly informing my wife about how high the number in system had gone.

If it is any consolation, huge numbers of players also spent embarrassing amounts of their time reporting the same number to IRC, Facebook, various forums, and other interwebs. Wink

Bagehi
Association of Commonwealth Enterprises
Posted - 2010.11.09 16:31:00 - [7]
 

So, what you're saying is drake army fleets don't just kill the other fleet but can kill the servers as well?

Awesome.

Illwill Bill
Svea Crusaders
Posted - 2010.11.09 16:33:00 - [8]
 

Another high-quality log from Veritas, filled with more juicy tech ****.

Thumbs up!

Garr Anders
Minmatar
The Red Circle Inc.
Posted - 2010.11.09 16:51:00 - [9]
 

Quote:
Team Cobra Kai


Strike first, strike hard, no mercy, sir!

Twisted Evil

Kikki Di'je
Lay Low
Posted - 2010.11.09 16:58:00 - [10]
 

CCP Veritas,

Just a few questions to attempt to get a better understanding of my experiences vs the technical aspects.

Does Destiny have deadlines on DB processes? During the massive fleet fight, I was able to move my ship, warp, spin, and roll with about 1-2 second lag periods. Drones and any module activation well...it just never happened.

I don't know if you are running a RAC environment or not, but if processes are being held up it would seem that more processing power is needed. Any thoughts on moving to a private cloud using virtualization?

Daedalus II
Helios Research
Posted - 2010.11.09 17:05:00 - [11]
 

Ok so ship killing is expensive.
What about trying to prepare that as much as possible before it actually happens?

Like when you undock a ship with a new fitting you could calculate and save which modules would drop in the event of a destruction. (You also have to recalculate this each time the fitting changes, in space or in station). And every time you put something into the cargo it's also quickly randomized whether it would be dropped or not if the ship is destroyed.
You could also pre-build the killmail as much as possible, just filling in the involved parties and the damage they did afterwards.
The insurance payout and mail should also be possible to prepare in advance, or maybe let it wait for 30 minutes or so before being calculated as it isn't time critical.

I don't know what else you can do, but I'm sure there are plenty of things you could pre-calculate before it's actually needed to save valuable time when it happens.

CCP Veritas

Posted - 2010.11.09 17:07:00 - [12]
 

Originally by: Kikki Di'je
Does Destiny have deadlines on DB processes? During the massive fleet fight, I was able to move my ship, warp, spin, and roll with about 1-2 second lag periods. Drones and any module activation well...it just never happened.


This syncs up quite well with what I saw on the tech side. Everything to do with your ship moving around is Destiny, and given that it does everything possible to keep up with real time, you were still able to move around even under insane load. However, non-Destiny load, like drones and modules and, well, pretty much everything else got starved effectively completely.

Originally by: Kikki Di'je
I don't know if you are running a RAC environment or not, but if processes are being held up it would seem that more processing power is needed. Any thoughts on moving to a private cloud using virtualization?


I'm not sure exactly what you're asking here. Could you clarify a bit?

Herschel Yamamoto
Agent-Orange
Nabaal Syndicate
Posted - 2010.11.09 17:08:00 - [13]
 

My first thought here is that Destiny being a processing priority seems a bit odd. Given the choice, I think most players would gleefully prefer to play with low lag at say 1/4 speed than to play with multi-hour lag(which I observed in LXQ, it literally took over an hour to start cycling my guns at one point) at full speed. I know it'd probably require you to take a chainsaw to half the code base to change that, but it seems like a throttle function on the physics simulation might cure lag nicely. Am I just a hopeless optimist here, or could this actually be done?

Anyways, thanks for the info.

CCP Veritas

Posted - 2010.11.09 17:10:00 - [14]
 

Originally by: Daedalus II
Ok so ship killing is expensive.
What about trying to prepare that as much as possible before it actually happens?


I've spent the past couple weeks digging at why exactly ship death is so expensive. I don't want to get into the details of that quite yet, but I'm happy to report that I've made a significant optimization to the process which is due for mass testing on Thursday. Think I can wring more out of it too if what I've done proves safe.

Mashie Saldana
Minmatar
Veto Corp
Posted - 2010.11.09 17:13:00 - [15]
 

With 3242 players in system, how many destiny balls were tracked at any one time on average?

NeoFusion
Caldari
Freelancer Union
Unaffiliated
Posted - 2010.11.09 17:16:00 - [16]
 

Edited by: NeoFusion on 09/11/2010 17:17:22
Originally by: Daedalus II
Ok so ship killing is expensive.
What about trying to prepare that as much as possible before it actually happens?

Like when you undock a ship with a new fitting you could calculate and save which modules would drop in the event of a destruction. (You also have to recalculate this each time the fitting changes, in space or in station). And every time you put something into the cargo it's also quickly randomized whether it would be dropped or not if the ship is destroyed.
You could also pre-build the killmail as much as possible, just filling in the involved parties and the damage they did afterwards.
The insurance payout and mail should also be possible to prepare in advance, or maybe let it wait for 30 minutes or so before being calculated as it isn't time critical.

I don't know what else you can do, but I'm sure there are plenty of things you could pre-calculate before it's actually needed to save valuable time when it happens.


I like it, or alternatively you could calculate the drop when the wreck is actually opened, seems mad I know but I'm sure there must be some way of linking a wreck to the ship it was before it actually got popped. This way the majority of that juicy processing could happen well after the battle, and only if the victor gets the chance to loot the field.

Actually, just thought properly about this and realised it wouldn't work due to the killmail being delivered instantly (showing the dropped items, doh)

Chribba
Otherworld Enterprises
Otherworld Empire
Posted - 2010.11.09 17:25:00 - [17]
 

Progress, is good progress.

Daedalus II
Helios Research
Posted - 2010.11.09 17:26:00 - [18]
 

Originally by: Herschel Yamamoto
My first thought here is that Destiny being a processing priority seems a bit odd. Given the choice, I think most players would gleefully prefer to play with low lag at say 1/4 speed than to play with multi-hour lag(which I observed in LXQ, it literally took over an hour to start cycling my guns at one point) at full speed. I know it'd probably require you to take a chainsaw to half the code base to change that, but it seems like a throttle function on the physics simulation might cure lag nicely. Am I just a hopeless optimist here, or could this actually be done?

Anyways, thanks for the info.

That may or may not be possible depending on the implementation.

In my mind it should be pretty simple to skip every second tick for example, effectively halving the needed CPU time.

That however has it's own interesting effects, two that I can think of is bumping and missiles.

Imagine a bump that should have been registered in a certain tick, but that tick was skipped so the bump isn't registered until one tick later. This means the ships are closer together than they should have been and the resulting bump will be much more powerful. Or if the ships are sufficiently fast (or ticks sufficiently rare) they might even be able to fly through each other, never registering a bump at all.

Same with missiles; a missile is supposed to hit a target a certain tick, but the tick is skipped so the missile overshoot the target entirely. It then has to turn around and chase the ship again, maybe missing a second time, and eventually running out of fuel even though it should have hit.

Daedalus II
Helios Research
Posted - 2010.11.09 17:30:00 - [19]
 

Originally by: NeoFusion

Actually, just thought properly about this and realised it wouldn't work due to the killmail being delivered instantly (showing the dropped items, doh)

Good idea nevertheless Wink
Maybe you could delay the killmail 30 minutes as well, they aren't time critical either...
Then you calculate the drop either when someone loots it or after 30 minutes when the killmail is generated.

Bruce Destro
Posted - 2010.11.09 17:34:00 - [20]
 

although im very glad i read this, as i was present during the fight, the only reason i am is because there is an afk cloaky npc corp jerk in my system. we need something we can do about this. EMO R.A.G.E!

Kikki Di'je
Lay Low
Posted - 2010.11.09 17:44:00 - [21]
 

Originally by: CCP Veritas
Originally by: Kikki Di'je
Does Destiny have deadlines on DB processes? During the massive fleet fight, I was able to move my ship, warp, spin, and roll with about 1-2 second lag periods. Drones and any module activation well...it just never happened.


This syncs up quite well with what I saw on the tech side. Everything to do with your ship moving around is Destiny, and given that it does everything possible to keep up with real time, you were still able to move around even under insane load. However, non-Destiny load, like drones and modules and, well, pretty much everything else got starved effectively completely.

Originally by: Kikki Di'je
I don't know if you are running a RAC environment or not, but if processes are being held up it would seem that more processing power is needed. Any thoughts on moving to a private cloud using virtualization?


I'm not sure exactly what you're asking here. Could you clarify a bit?


I'm used to seeing Oracle RAC systems for high processing as when one box starts to get clogged with requests and threads. With RAC, a load balanced multi box approach spreads the threads and requests evenly, helping the processing power clearing some of the clog although this is a cost based approach since it requires more physical hardware.

With the private cloud, you can use unused resources of other physical components to do the processing of clogged requests. So in this example, we had a reinforced node for this particular fight, what about other nodes that might have only been using 15% of their resources? These resources could be used to help speed up the processes of all the requests. With virtualization you could use some resources for session changes for ship explosions and pods, and other resources for what I think Destiny is for - velocities, explosion radius's and the like. The remaining available resources could be used for DB threads to start or simple requests such as activating a module.

With the recent buzz around cloud computing, I was just wondering if CCP was going to create a private cloud (I.E. not outsourcing external resources for obvious reasons such as uptime) to pull processing power and provide it where it needs it. I have seen demonstrations of this being an on demand solution which makes me wonder, while fleet notification is nice, in the future, you may not need it as once you reach a certain % of load, your virtualized DB's can kick in, and begin processing the extra load, much like a RAC system.

CCP Veritas

Posted - 2010.11.09 17:51:00 - [22]
 

Originally by: Kikki Di'je
<fancy multi-processing cloud foo>


As current, the Eve server process isn't even multi-threaded at the OS level. We need to get moving down that path, but this level of setup is well beyond what we currently can leverage. It's doubtful that this level of wide processing would be beneficial to our particular load characteristic either, as the communication overhead of splitting a simulation over multiple machines would likely be a tighter bottleneck than the computation.

Indeterminacy
THORN Syndicate
BricK sQuAD.
Posted - 2010.11.09 17:58:00 - [23]
 

Edited by: Indeterminacy on 09/11/2010 18:06:00
Originally by: CCP Veritas
...you were still able to move around even under insane load...


This has not been my experience. I was in LXQ and more recently CYB with about 600 +/- in local on what I would imagine was an unreinforced node. Imagine, a spontaneous battle, I know right? Anyway...

In all cases (multiple engagements in LXQ over multiple days) navigation commands were severely delayed. In two cases, once in LXQ and in CYB navigation was impossible. Completely impossible.

In LXQ the mwd stuck on and my ship refused to turn, stop, etc resulting in death and destruction.

In CYB, after a point, I had no control over anything. Period. Outstanding operations mounted and mounted until I said "I have better things to do on a Sunday" and logged off.

Edit: In fact, in CYB, I was even unable to join a fleet (got an empty fleet window and no fleet-chat window).

Edit: Something that's quite frustrating about this is that some people seem to have little to no problems in these cases while others are entirely dead in the water. What might account for this?

Pirokobo
GoonWaffe
Goonswarm Federation
Posted - 2010.11.09 18:07:00 - [24]
 

in the beginning CCP made the capitals and the subcapitals and it was good. Then the did give the capitals the doomsday and the subcapitals dieed in scores. Thus did battleships become the norm for they alone could economically deal with tanking a dd.

Then ccp did taketh away the dd and subcaps rejoiced for HACs would no longer quail in fear of instant death.

Thus did the plague of HACs come upon the sniper battleship, and they colud not track.

And the batleships did flee and reship into drakes, being in measure greater than HACs.

But it was not good. For HACs and drakes must move to survive an drakes use missiles that cause the server great harm.

And ccp being neither wise nor seeing did plan to smite the drake and not the hacs that ended the golden age of sniper battleships that the servers did love.

Kikki Di'je
Lay Low
Posted - 2010.11.09 18:17:00 - [25]
 

Originally by: CCP Veritas
Originally by: Kikki Di'je
<fancy multi-processing cloud foo>


As current, the Eve server process isn't even multi-threaded at the OS level. We need to get moving down that path, but this level of setup is well beyond what we currently can leverage. It's doubtful that this level of wide processing would be beneficial to our particular load characteristic either, as the communication overhead of splitting a simulation over multiple machines would likely be a tighter bottleneck than the computation.


I suppose this falls into the realm which we all unfortunately fall into at some point of cost effective vs labor effective vs processing effectiveness? I know there were was some new flash based technology that came out last year that was supposed to be capable of 7m+ transactions per second but can only imagine the cost not to mention training, installation and management time needed as well as possible future replacements.

You do bring up a good point with the network being the bottleneck, but perhaps there could be some sort of leverage with fiber, security, and protocol usage to speed things up?

Thanks for the responses, interesting to see and hear how bottlenecks are resolved!

Vincent Athena
Posted - 2010.11.09 18:35:00 - [26]
 

I wonder if you could put destiny on one core and everything else on another. Might not be too much of a bottleneck to split the work that way.

You would still have to give the core running destiny priority memory and network access, but you might still have a net gain.

ivar R'dhak
Minmatar
Posted - 2010.11.09 19:35:00 - [27]
 

Originally by: CCP Veritas
Originally by: Daedalus II
Ok so ship killing is expensive.
What about trying to prepare that as much as possible before it actually happens?


I've spent the past couple weeks digging at why exactly ship death is so expensive. I don't want to get into the details of that quite yet, but I'm happy to report that I've made a significant optimization to the process which is due for mass testing on Thursday. Think I can wring more out of it too if what I've done proves safe.
Why are you piling the ship-death calculations on top of all the other battle stuff in the first place?

Can´t you generate that "wreck database point"(your ship death calculations) randomly on undocking and just keep updating it when the cargo state changes.
Or just "cheat" and by default make all consumables in cargo non recoverable on ship death. Thus there´s no need to update that "ship death data subset" in space at all.
Oh well, no idea how you do your database stuff but I hope I´m not talking to much büllsh!t there. Wink

mechtech
SRS Industries
SRS.
Posted - 2010.11.09 19:36:00 - [28]
 

Originally by: Vincent Athena
I wonder if you could put destiny on one core and everything else on another. Might not be too much of a bottleneck to split the work that way.

You would still have to give the core running destiny priority memory and network access, but you might still have a net gain.


This makes sense to me too. If destiny is the only real time system, why not offload the other non-time sensitive processing to other cores and processors? Maybe not other computers per say, but it seems very possible to do this in a multi-core/multi-processor server, where each processor has access to the same memory.

Elsa Nietzsche
Posted - 2010.11.09 19:36:00 - [29]
 

Is there any way non-critical actions could be queued up?
say wrecks, loot, KM creation etc all get written off to a queue that can be processed after the fight. there's really no need for all that to be generated in the heat of battle.
personally I'd like to see dynamic system population limits. say if there's 100 people in a system and load is high, noone else can jump in until the load stabilizes, then a new limit is set at say 500, etc.
this would allow new fleets to jump in but only when the system can handle them. But i'm sure many will argue against it.

i also like the idea of player presence takes precedence over player actions. this way if a server is overloaded, then no one dies as opposed to one team gets to slaughter the other who can't load grid.

can't wait to see more blogs on this exciting stuff
keep up the good work

Jamus Gorrelius
Macabre Votum
Morsus Mihi
Posted - 2010.11.09 20:47:00 - [30]
 

So in R10 you have 900 players on a Reinforced Node and it is just as laggy and UNPLAYABLE as if it had 1200 or even 3200.

Can we have some nifty excuses for that or isnt there any, Since dominion something snapped and i truly believe your working on it but you havnt got a clue how to fix it.

What makes me cry a little inside is that we had a extremly playable game with 1200 before dominion on non reinforced node yet here we nearly a year later and we still have no FIX or any improvments.



Pages: [1] 2 3

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only