open All Channels
seplocked EVE Information Portal
blankseplocked New Dev Blog: Missiles Hate My Hamsters The Followup
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: 1 [2] 3

Author Topic

Ariane VoxDei
Posted - 2011.03.22 23:45:00 - [31]
 

Nice result. It does seem like you mislabeled your graph.
Clearly, (you can spot it on some spikes), what you have on the blue line is not "change" but percentage of last (presumeably) last weeks use data.

Change would then be 100% minus the blue line, i.e. it has to be read upside down to get the change.
A particularly good demonstation is the downward spike in "before" (red curve) CPU use just before saturday, which gets matched by a sharp upspike on the "change" (blue curve).
However, the "after" (green curve) hardly moves, so if the blue was indeed the change, it should curve down, matching the decreasing difference between red at green curves at that point.

The really damning part, is ofcourse the trivial observation that adding the blue to the green clearly does not yield the red line.
That is, adding the 'change' to 'after' does not yield the 'before', or, if you prefer, subtracting the 'change' from the 'before' does not yield the 'after'.

That leads to a overly optimistic reading of the data. It is still very considerable, considering that inventory is only part of the load, but misleading non the less.
For the worst case it makes it look about 20 percentagepoints better than it is - which is pretty major.

Shame you didnt also put up the jita population curve next to those cpu curves.
Not just for a CPU/user case.
Also to get a picture of how hard the population correlates with the usage.
Not everyone is a active in-person trader. There are lurkers, spaiz (not just cargoscanners), spammers , marketbrowsers/bots, ganks and warfare campers, all causing wildly different usage patterns serverside.
Even location matters, just think of 4-4 (undock) grid, vs pretty much all other space in Jita, except perhaps the Perimiter gate.

Still, I must express my annoyance that station inventories are actually tied to the same node as the system they are in.
Ok, maybe I am fantasizing, I had hoped that station inventory was a separate animal - that could be shifted to another node/blade/whatever if needed for performance reasons. Not least of which, that would have made it rather easier to get a good idea of the change in its performance (as a minimum you could take the process time use). That hope was, along with a few others, shot down brutally by the last blog.

But truly, surprised how few metrics that you are able to track (or at least are telling us about).

TorTorden
Amarr
Posted - 2011.03.22 23:46:00 - [32]
 

Originally by: Soldarius
Originally by: Abramul
There's been some banter about staging a 2000-man fleet battle in Jita to let highsec-dwellers know what lag feels like. Would this be a problem?


^this. Let's all make a lo skill frigate alt, appoint a couple CEOs to make new corps and war dec in prep for it. EPIC

CCP Veritas, as a Caldari exclusive missile spammer, I have been following this subject (missiles, hamsters, etc.) since it started. When I saw the note for this follow-up on my log-in screen, I ran in here expecting great things like: "After implementing the recommendations CCP Masterplan put forth, we are pleased to say lag due to missile spammage has been greatly reduced."

Instead I got "After implementing the recommendations CCP Masterplan put forth, we have no idea if it did anything. But at least the servers didn't catch fire again. Hooray! Oh, and btw, Jita is now much improved. Charts/proof."

Let me put this forth. If you take your car to the auto tech because of poor engine performance, you expect that when you go to pick it up, that the tech checked to make sure it's been fixed before giving it back to you.

Just because it doesn't blow up doesn't mean that it is working as intended. In fact, according to CCP Veritas, he has no data at all a to whether or not their changes have made any effect. I find this unacceptable. CCP Masterplan had plenty of data. Why not perform those same tests again and compare the results to the previous results?

Leaving disappointed.


Could have almost gone completely the other way, braking the inventory system, with massive downtime ensuing
And ccp saying, "missiles are hard, we are removing them from space and just making it look like lasers."

CCP Veritas

Posted - 2011.03.22 23:47:00 - [33]
 

Originally by: Soldarius
Just because it doesn't blow up doesn't mean that it is working as intended. In fact, according to CCP Veritas, he has no data at all a to whether or not their changes have made any effect. I find this unacceptable. CCP Masterplan had plenty of data. Why not perform those same tests again and compare the results to the previous results?


I have plenty of data on these changes as well, it was included in the first blog on the subject as they're lab tests, as Masterplan's was. The intention behind this blog is to follow up saying deployment went well (as the changes had not been deployed when the first blog went out) and to talk about notable live effects.

We just don't have the kind of metric collection on live servers that would allow me to say "missiles used to take X% CPU and now they take Y%". That level of instrumentation would introduce significant load by itself, which goes against what we're trying to do here in the first place.

But, based on the lab tests and the uneventful deployment, I feel confident saying that missile spaming is far less impactful than it used to be.

Archestratidas
Posted - 2011.03.22 23:47:00 - [34]
 

Originally by: Abramul
There's been some banter about staging a 2000-man fleet battle in Jita to let highsec-dwellers know what lag feels like. Would this be a problem?


You know that piling thousands of people in the same system will cause a lot of lag, yet you continue to take part in those activities. You could do small gang pvp if you so desired, but apparently you prefer your lag fights and subsequent posting rights about all things LAAG.

Frankly, I think that's your problem as it doesn't affect me at all. Enjoy.

CCP Veritas

Posted - 2011.03.22 23:54:00 - [35]
 

Originally by: Ariane VoxDei
It does seem like you mislabeled your graph.


You assume I meant "Absolute Difference" where I actually meant "Relative Difference". I went through a whole lot of potential titles for the graph and they all had faults - I'm just happy you could sort out what I was trying to convey in the end~

Abramul
Gallente
StarFleet Enterprises
BricK sQuAD.
Posted - 2011.03.23 00:10:00 - [36]
 

On a less joking note, do you happen to have on-hand how the per-ship CPU use of a heavy missile drake compares to that of an 1400mm artillery Maelstrom or mega pulse Armageddon? The previous graph lists repeats per second, but that sounds like it's per group rather than per ship.

CCP Veritas

Posted - 2011.03.23 00:32:00 - [37]
 

Originally by: Abramul
On a less joking note, do you happen to have on-hand how the per-ship CPU use of a heavy missile drake compares to that of an 1400mm artillery Maelstrom or mega pulse Armageddon? The previous graph lists repeats per second, but that sounds like it's per group rather than per ship.


Too many variables there to give you a definitive answer. Skills/fittings/implants can cause changes in rate of fire, and the usage of weapon grouping has a profound effect as well (1 group of 8 guns is about 1/8th the load that 8 groups of 1 gun is, as one would expect).

So instead I'll walk you through how to figure it out yourself based on the data you have. We have # of firings per second to reach full load (on my now-deprecated test server), we want # of ships needed to reach the same load. So really, you just need to figure out how many repeats a given ship will generate, then divide the number I gave by it.

So if, for instance, you had a ship with 4 weapons that had a rate of fire of 20 seconds (that is, 3 shots per minute) post skills-implants-fitting. The pilot is a jerk and doesn't group the weapons. That ship is able to put out (4 * 1/20) = .2 repeats per second. If that type of weapon system were able to spam out 1000 repeats per second, then we're looking at (1000/.2) = 5000 ships to saturate.

There's a pile of assumptions under that all, the most egregious of them being that this in this hypothetical situation the absolute only thing going on is all those ships shooting one single target who is never going to die. But it does give you some idea of the relative cost between ship types.

Itan Aeriel
Posted - 2011.03.23 00:42:00 - [38]
 

i just watched 4 missles do zero point 90 degree turns so something isnt right

Freidrich Nietchize
Gallente
Capitalist Pigs Inc.
Posted - 2011.03.23 01:00:00 - [39]
 

I demand that the CEO take those responsible for this work out for a round of drinks! Great work guys. Glad to see CCP is committed to making this game better, not just in features but in functionality. Sexy programmer talk and graphs also helped.
Going forward, make sure your variables don't look like this :3

Abramul
Gallente
StarFleet Enterprises
BricK sQuAD.
Posted - 2011.03.23 01:09:00 - [40]
 

Well, for Drake vs. Maelstrom, looks like it comes out to 8.5:1 compared to 44:1 previously. Of course, you as mentioned that's not exactly the most precise or current result, but still interesting.

TheSpyInCorp
Posted - 2011.03.23 01:29:00 - [41]
 

Originally by: Soldarius
Originally by: Abramul
There's been some banter about staging a 2000-man fleet battle in Jita to let highsec-dwellers know what lag feels like. Would this be a problem?


^this. Let's all make a lo skill frigate alt, appoint a couple CEOs to make new corps and war dec in prep for it. EPIC

CCP Veritas, as a Caldari exclusive missile spammer, I have been following this subject (missiles, hamsters, etc.) since it started. When I saw the note for this follow-up on my log-in screen, I ran in here expecting great things like: "After implementing the recommendations CCP Masterplan put forth, we are pleased to say lag due to missile spammage has been greatly reduced."

Instead I got "After implementing the recommendations CCP Masterplan put forth, we have no idea if it did anything. But at least the servers didn't catch fire again. Hooray! Oh, and btw, Jita is now much improved. Charts/proof."

Let me put this forth. If you take your car to the auto tech because of poor engine performance, you expect that when you go to pick it up, that the tech checked to make sure it's been fixed before giving it back to you.

Just because it doesn't blow up doesn't mean that it is working as intended. In fact, according to CCP Veritas, he has no data at all a to whether or not their changes have made any effect. I find this unacceptable. CCP Masterplan had plenty of data. Why not perform those same tests again and compare the results to the previous results?

Leaving disappointed.


Saying this just tells me that you know nothing about data analysis and programming and you just want ccp to spit up an answer. Fixing code isn't at all like fixing a car's engine when you have millions of lines of code, and thousands of events happening at the same time that use that code, spread between dozens of server nodes. Why do you think CCP Veritas used jita, a dedicated node, as the most accurate data model? In your example, you knew what the problem was (poor engine performance) and exactly where the problem was coming from. In this case, ccp knows what the problem is, but there are a million places it could come from. So the only option to remedy the problem is to optimize it bit by bit, and this is one of those optimizations. Did you really think they [or anyone] could just fix lag as quickly as you're asking it to be?

So no, you have not been reading the dev blogs, you have been skimming the dev blogs and looking for a "we fixed it" answer because CCP Veritas already explained it perfectly:

Originally by: CCP Veritas


As for impact, I unfortunately can't point to any metric and say "this is because of the missile work". We've got a cluster-wide CPU per user metric that's good at telling us when we've done something good in general, but it's very noisy depending on what the player population happens to be doing at the time, so it takes many weeks of data to be able to establish a reasonable trendline. In the case of this change, there simply wasn't enough data before and after without other complicating factors to be able to say what the impact was.

I hoped I could capture load from popular missioning systems, which tend to be more consistent in load than most and also involve lots of missiles, but systems like that do not have a consistent mapping on the cluster. What I mean by that is that on any given day, the systems around any given mission hub could be grouped together in many sorts of ways, with any number of other systems thrown into a mix. Looking at the CPU usage of such nodes is going to be extremely noisy depending on what other solar systems got introduced into the mix.


get over yourself and your "expectations" and appreciate these optimizations as breakthroughs in the fight against lag.


Siiee
Recycled Heroes
Posted - 2011.03.23 01:54:00 - [42]
 

Originally by: Illectroculus Defined
17:29:53
In response to the Jita cap, can't you guys just set a cap on reinforced systems that everyone is aware of so everyone can make a judgement call on whether to engage or not so we don't have more "shoot the lagged out enemy" fights?



Originally by: Bagehi

The current system is being exploited. That's why there is so much annoyance. Whoever loads first wins because everyone else is lagged out. Tactics, ships, skills, nothing else matters right now.



There currently is a soft cap in place on every system, and I'm sure that those are involved regularly in massive fleet warfare know roughly of the point where a system starts to fall over. When has anyone ever downsized a fleet to fit within current rough system limits, and what makes anyone think that a hard cap would be treated any differently?

Meatypopsicle
Posted - 2011.03.23 02:02:00 - [43]
 

Originally by: Siiee

There currently is a soft cap in place on every system, and I'm sure that those are involved regularly in massive fleet warfare know roughly of the point where a system starts to fall over. When has anyone ever downsized a fleet to fit within current rough system limits, and what makes anyone think that a hard cap would be treated any differently?


You obviously haven't seen the "Everyone who is in a sub cap that is not a HIC log off now!!!!111" Alliance mails.

Sturmwolke
Posted - 2011.03.23 02:16:00 - [44]
 

Edited by: Sturmwolke on 23/03/2011 02:16:18
Originally by: CCP Veritas
I hoped I could capture load from popular missioning systems, which tend to be more consistent in load than most and also involve lots of missiles, but systems like that do not have a consistent mapping on the cluster. What I mean by that is that on any given day, the systems around any given mission hub could be grouped together in many sorts of ways, with any number of other systems thrown into a mix.

Well, if you turn off all the agents from assigning random missions across the constellation and then concentrate them into one system during the weekends, you might get something. Probably the best ones are L3 hubs with a high number of Drakes running around.

I take it it wasn't possible or the code team's pretty reluctant to do a temp hack on TQ? Smile

Ce'Nedra Wolf
Posted - 2011.03.23 02:30:00 - [45]
 

I feel so envious of you. I wish I was working on such problems. Makes me wonder why I don't find a job that would have me working on such performance issues. I enjoy them.

Patient 2428190
DEGRREE'Fo'FREE Internet Business School
Posted - 2011.03.23 03:07:00 - [46]
 

CCP 1: There is no data to disprove/prove what the actual effect of our changes to the missile systems

CCP 2: Who cares, post a dev blog proclaiming great victory. If you title it something about missiles and hamsters and then stick a graph on the bottom, players will assume you did good things.

CCP 1: HIFIVE BRO.

CCP 2: You still aren't joining the rest of the development staff on the Twilight game.

Kytayn
Gallente
Kronos TEchnologies
Posted - 2011.03.23 03:29:00 - [47]
 

Always interesting to read about the technical aspects of Eve. Unfortunately I can't load that graph. cdn1.eveonline.com is apparently totally blocked from my ISP (I tried supplying the IP address and got an unknown host error. The way I understand it, that CDN host tried to get away with traffic dumping.)

I wonder if that's why I my Eve client can't load portraits anymore.

AkJon Ferguson
JC Ferguson and Son Ltd
Ferguson Alliance
Posted - 2011.03.23 03:46:00 - [48]
 

To the extent you're actually reducing lag, GJ CCP.

To the extent you're lying about how awesome (and "challenging-path" choosing) CCP is or about how lag is caused primarily by player numbers as opposed to CCP-implemented changes, knock it off.

If CCP wants to start behaving ethically, that would be OMGWTFBBQAWESOME. Start by banning all macroers and botters. Follow up by tripling the size of the BFF team. Hire Mynxee as head of IA. Make Incarna optional. Stop acting like RMT weasels and embrace the concept of plex for vanity items ONLY. Get SOMEBODY to vet all your devblogs for basic things like HONESTY. For now, your reputation is slightly below that of the typical jita scammer.

It's really not that hard to maintain an 8-year old cash cow like EVE. If you've finally figured that out, I'm glad.

Xiaodown
Guiding Hand Social Club
Posted - 2011.03.23 03:54:00 - [49]
 

Quote:
a full regression test for EVE Online takes a sizeable team of QA many days to do, and even then it's likely something slips through the cracks, there's just so much you can do in this game.


No offense, but that's really bad agile. Your user story is not INVEST or you're lacking SOA or your test matrix needs work or something. I'm not sure exactly where that kind of problem lies, but that's... not a good thing.

Nobani
Merch Industrial
Goonswarm Federation
Posted - 2011.03.23 04:03:00 - [50]
 

Interesting note: there's a leak in the inventory items, which they are using downtime to clean up.

You can see this pretty clearly in the relative change graph -- when they switched from a O(n) access container to O(1) , the improvement increases linearly over the course of the day, and is reset each downtime. It looks like performance would have increased by ~40% near downtime by fixing this leak, without needing to change the data structure.

Galandil
Posted - 2011.03.23 05:35:00 - [51]
 

Originally by: CCP Veritas
Originally by: Mashie Saldana
What is the new Jita cap?


2000 currently. We expect there to be plenty of headroom above that, but we want to see each boundary along the way reached safely before we raise it more.


With the current changes. What is your estimated cap? or What is the current system population goal? I know we all have a wishful thinking of no cap in the system, but with the current technology, there has to be one.

I'm not looking for: "This is our goal and it will never change." I'm mainly looking for: "This is our current ballpark figure with the given technology we have."

Lykouleon
Wildly Inappropriate
Goonswarm Federation
Posted - 2011.03.23 06:13:00 - [52]
 

Not enough graphs :pout:

Mirei Jun
Right to Rule
THE UNTHINKABLES
Posted - 2011.03.23 06:13:00 - [53]
 

Nice read. It's boggling to think Eve has not used hashing up until now to handle inventory. With Eve getting so popular hopefully we don't see collisions of any kind for a while! :p

Damion Rayne
Gallente
Dark-Rising
Executive Outcomes
Posted - 2011.03.23 06:25:00 - [54]
 

Originally by: Patient 2428190
CCP 1: There is no data to disprove/prove what the actual effect of our changes to the missile systems

CCP 2: Who cares, post a dev blog proclaiming great victory. If you title it something about missiles and hamsters and then stick a graph on the bottom, players will assume you did good things.

CCP 1: HIFIVE BRO.

CCP 2: You still aren't joining the rest of the development staff on the Twilight game.



One: World of Darkness is not twilight, it's a completely different world and set of stories from White Wolf Entertainment.

Two: Obvious Troll is blatantly obvious, so go away.

Three: Those that feel the need to complain just for the sake of complaining shall never be happy with anything. So again, go away.

Antihrist Pripravnik
Scorpion Road Industry
Posted - 2011.03.23 08:02:00 - [55]
 

Edited by: Antihrist Pripravnik on 23/03/2011 08:04:12
The amount of effort invested in sorting out inventory problems and missile optimisations is not small. However, are you sure that you are "barking at the right tree"?

Frankly, the first thing that will be noticeable to actual players would be when this start to function as intended: http://wiki.eveonline.com/en/wiki/Logging_out.

"Logging in PvP combat
If you are in PvP combat, your ship will warp to a random point and remain there for 15 minutes, unless it's being warp scrambled, in which case it will be at the mercy of your enemies. If your ship is blown up after you log out, your pod will remain for another 15 minutes, which is plenty of time for a resourceful player to find it, using scanner probes.
"

Until then, you'll still have ships that die many hours (up to 9 ugh currently) after the people using them get kicked out of the server or loggs off because of the black screen.

Louis deGuerre
Gallente
Malevolence.
Posted - 2011.03.23 09:06:00 - [56]
 

Game, SET and match Wink

CCP Veritas

Posted - 2011.03.23 09:48:00 - [57]
 

Originally by: Galandil
With the current changes. What is your estimated cap?


My gut feeling is that we'd be able to pull off 2500-2750 people doing normal Jita-type things.

CCP Veritas

Posted - 2011.03.23 10:01:00 - [58]
 

Originally by: Antihrist Pripravnik
The amount of effort invested in sorting out inventory problems and missile optimisations is not small. However, are you sure that you are "barking at the right tree"?


We spend a fair bit of time asking this question of ourselves. There's more to how we decide what to work on than just what the worst symptom is - some estimation of difficulty and expectation of being able to deliver improvements in a reasonable amount of time has to be a part of it too. I'll be going into more detail on how we choose what to work on in my Fanfest presentation (Friday 4pm), so stay tuned for that~

Originally by: Antihrist Pripravnik
Frankly, the first thing that will be noticeable to actual players would be when this start to function as intended: http://wiki.eveonline.com/en/wiki/Logging_out.


That's one of the problems that's been in the back of our heads for some time now. Nearly impossible to just sit down and work directly on that problem - there's no clear starting point. It's like approaching a great problem in Physics - you rarely go directly for the end goal, but you work in the general neighborhood of the problem and, if you're lucky and paying attention, you might stumble on something great.

The savings in Jita is a good example of what I'm talking about, actually. We had been pondering what Jita's problem was for some time. The profiling slices we had from it looked perfectly reasonable, in that it was mostly inventory work, which makes sense given what people in Jita tend to do. So the problem just simmered in our minds for months. Then, while working on missile optimizations, I stumbled on the inventory data structures and immediately it was clear what Jita was spending its time doing.

So my point is, even if we're not working directly on the problem you see as the worst in the world, there's good reasons for that and we're working in the neighborhood of it, so serendipity may strike at any moment.

Abramul
Gallente
StarFleet Enterprises
BricK sQuAD.
Posted - 2011.03.23 13:30:00 - [59]
 

By the way, have you looked into whether suppressing Jita local (wormhole mode would probably be easiest) would improve performance? Probably nowhere near as effective as this, but depending on how you have things set up, might be an easy way to gain a few percent. Might save a bit of bandwidth too if you're transmitting portraits as images.

Dr Sheepbringer
Gallente
Halinallen veroparatiisi
Inglorious Carebears
Posted - 2011.03.23 13:30:00 - [60]
 

Originally by: CCP Veritas

So my point is, even if we're not working directly on the problem you see as the worst in the world, there's good reasons for that and we're working in the neighborhood of it, so serendipity may strike at any moment.


This is also why I like the idea of incarna being in the pipe. They are battling 3D-graphics in a very different manner and lets face it some stuff in those areas also cause LAG! One way or another.


Pages: 1 [2] 3

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only