open All Channels
seplocked EVE Information Portal
blankseplocked New Dev Blog: Fixing Lag: Well, this one doesn't really...
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: 1 [2] 3

Author Topic

Dav Varan
Posted - 2010.09.10 15:22:00 - [31]
 

Originally by: CCP GingerDude
Originally by: Voight Kampf
Nice blog. It still doesn't cover one question. Why we didn't have this lag BEFORE Dominion? If i understood your blog right this issue is independent from sov system etc and still we didn't encounter it in Apocrypha. How you can explain this?
I'm fairly sure that this issue was present before Dominion; you just needed bigger fleets to trigger it before. I can't really explain why the server pain-threshold dropped around Dominion conclusively, since all our metrics suggested that we should've been able to handle even more pilots then before. My pet theory regarding that today is actually the change in playstyles and fitting that we saw after dominion. Some weapons are much more CPU intensive than others and Dominion basically ushered in an era of much increased use of missiles, fighter bombers and smartbombs in fleet fights. These are known to be cpu heavy compared to many other weapons and modules, so a significant increase in the % of ff players using them could very well have moved the tipping point.



So smartbombs up for weapon grouping soon tm ?


Arkady Sadik
Minmatar
Electus Matari
Posted - 2010.09.10 15:40:00 - [32]
 

Please do not implement "fleet jump"; a taxi service to move large groups of people around EVE without them having to do anything at all is not good for the game, IMHO.

Yeay Fritg
Caldari
Confrerie de Kaedri
Cluster Of Rebirth
Posted - 2010.09.10 15:43:00 - [33]
 

Clear and direct.

Only one thing : Thank you !

Yeay

Vir Hellnamin
Gradient
Electus Matari
Posted - 2010.09.10 15:53:00 - [34]
 

Edited by: Vir Hellnamin on 10/09/2010 16:03:51
Edited by: Vir Hellnamin on 10/09/2010 16:02:30
Originally by: Arkady Sadik
Please do not implement "fleet jump"; a taxi service to move large groups of people around EVE without them having to do anything at all is not good for the game, IMHO.


Maybe something like FC-confirm for fleet jump after every pilot themselves has hit "jump" - like "FC, 31/101 (30%) of your fleet is ready to jump? go?"... (but, just don't make that denying access to rest of the UI like some pop-ups.) ... still, might be messy with Titan-bridging... (give longer cycle-time? I don't know...)

That way, you might get a single "jump-packet", but not FC-"taxiable" fleet. Also, everyone could then blame FC for lag and everything!!!! Laughing


BTW, would there be any chance to get some Dev reply/blog how you distribute solar-systems to a (one) node? And, some talks about have you thought of using some algorithms or patterns (cluster some nearby system under one node)? Cluster-up a system with station and near-by's in 0.0; move the lag-zone away from the last gate to second to last or further... well that would just deny all shoots, but most likely lessen the petitions for lost hulls. (wouldn't help with Titan-bridging) Or similar stuff?

Bagehi
Association of Commonwealth Enterprises
Posted - 2010.09.10 16:05:00 - [35]
 

It is possible that the increased use of stealth bombers at gates could be partly to blame. I don't remember it being the norm to have a couple bombers guarding entry gates like it is today. I would imagine the server load is pretty high when a large fleet jumps, immediately followed by a couple bombs being launched at the ships once they start appearing.

However, I will point out that this has happened in fleets that were moving via cyno chains, so there are no drones out, no missiles out, nothing shooting anyone, and it was highly unlikely that anything terribly exciting was happening on the node in a different system. Personal experience and all.

StealthSeeYa
H A V O C
Cascade Imminent
Posted - 2010.09.10 16:09:00 - [36]
 

Good job on your findings, just too bad it was that long to be worked on.

Countless petitions were made receiving the same answer.. logs show nothing (on server side).. resulting in lost of time, isk and the will to logon and play.

It's not like thousands of players were lying or not seeing clear..
Hoping you'll listen more to your player base next time.

CCP GingerDude

Posted - 2010.09.10 16:31:00 - [37]
 

Originally by: Vir Hellnamin
Edited by: Vir Hellnamin on 10/09/2010 16:03:51
Edited by: Vir Hellnamin on 10/09/2010 16:02:30

BTW, would there be any chance to get some Dev reply/blog how you distribute solar-systems to a (one) node? And, some talks about have you thought of using some algorithms or patterns (cluster some nearby system under one node)? Cluster-up a system with station and near-by's in 0.0; move the lag-zone away from the last gate to second to last or further... well that would just deny all shoots, but most likely lessen the petitions for lost hulls. (wouldn't help with Titan-bridging) Or similar stuff?


Not my call :) I can tell you though that both the process of determining which systems are hardened and the method of moving them to dedicated nodes are actively being worked on.


CCP GingerDude

Posted - 2010.09.10 16:41:00 - [38]
 

Originally by: StealthSeeYa
Good job on your findings, just too bad it was that long to be worked on.

Countless petitions were made receiving the same answer.. logs show nothing (on server side).. resulting in lost of time, isk and the will to logon and play.

It's not like thousands of players were lying or not seeing clear..
Hoping you'll listen more to your player base next time.


This was worked on from the getgo. I'll grant you that it did take way too long to deploy the change and I apologize for that, but this kind of change, seemingly minor, has huuuuge fallout potential, so testing and deployment is always going to be extrememly careful. At no point did we doubt that players were experiencing what the said, we just didn't have anything that would explain it or verify individual reports of it happening.

Rakshasa Taisab
Caldari
Sane Industries Inc.
Posted - 2010.09.10 17:01:00 - [39]
 

Originally by: CCP GingerDude
I'm fairly sure that this issue was present before Dominion; you just needed bigger fleets to trigger it before. I can't really explain why the server pain-threshold dropped around Dominion conclusively, since all our metrics suggested that we should've been able to handle even more pilots then before. My pet theory regarding that today is actually the change in playstyles and fitting that we saw after dominion. Some weapons are much more CPU intensive than others and Dominion basically ushered in an era of much increased use of missiles, fighter bombers and smartbombs in fleet fights. These are known to be cpu heavy compared to many other weapons and modules, so a significant increase in the % of ff players using them could very well have moved the tipping point.

DRAKE BLOB NERFED CCP!

Serena Greyskull
Rens 911
GoonSwarm
Posted - 2010.09.10 17:08:00 - [40]
 

Edited by: Serena Greyskull on 10/09/2010 17:11:04
So, do you still not know whats causing lag on small scale fights?

We were in a fight the other day where there was no more than 150 combatants (yes i know thats not too small scale but come on this should be handled easily and normally is), and the lag was so bad it took 70 of us 20 mins to kill a carrier. Module lag was so bad, that one would be permajammed by a pos, even if you warped out and warped back in, you would still be jammed.

I strongly suggest that you start talking to FC's of bigger alliances. I would be more than happy to fill in a questionnaire if this helps, but you would be better off actually TALKING to us, and asking what the conditions of the fight was, and how this affected our gameplay. You would be surprised about how much information we have concerning not only the fight itself, but the conditions leading up to a fight, and so on.

All we want is for the lag to be fixed so we can have ~goodfights~, but you have to talk to us more if you want information.

/rant off

Mynxee
Veto.
Veto Corp
Posted - 2010.09.10 17:11:00 - [41]
 

Edited by: Mynxee on 10/09/2010 17:28:20
Thanks for the ongoing communication! I love hearing about the technical challenges you face, how problems are tracked down, what fixes are identified/done, etc. But more than anything...it is really enjoyable to read the constructive and open dialog between devs and players on these technical topics. And along those lines...

Originally by: Serena Greyskull
I strongly suggest that you start talking to FC's of bigger alliances. I would be more than happy to fill in a questionnaire if this helps, but you would be better off actually TALKING to us, and asking what the conditions of the fight was, and how this affected our gameplay. You would be surprised about how much information we have concerning not only the fight itself, but the conditions leading up to a fight, and so on.

All we want is for the lag to be fixed so we can have ~goodfights~, but you have to talk to us more if you want information.


@CCP GingerDude: Has CCP ever considered conducting Focus Group discussions on voice comms with groups of 6-12 FCs from bigger alliances and some key devs from the lag team? It would give a really good opportunity to gather useful information and brainstorm. Conversation is often much more productive than collecting written data, because the volume of information that can be shared is so high. Besides that, the value of free association, potential for making synergystic connections, and just plain bouncing ideas off each other cannot be overrated.

I organized and conducted some focus groups like this during my CSM campaign and have done a lot of them in my professional life as well. If you want some ideas on how to conduct them in a manageable and productive way, just request access to the internal CSM forums and we can have a discussion there about it.

T'Amber
Garoun Investment Bank
Posted - 2010.09.10 17:18:00 - [42]
 

/me points at the ginger and skips back to the shadows to rotate and dribble




Serena Greyskull
Rens 911
GoonSwarm
Posted - 2010.09.10 17:26:00 - [43]
 

Originally by: Mynxee
Edited by: Mynxee on 10/09/2010 17:17:46
Edited by: Mynxee on 10/09/2010 17:16:15
Thanks for the ongoing communication! I love hearing about the technical challenges you face, how problems are tracked down, what fixes are identified/done, etc. But more than anything...it is really enjoyable to read the constructive and open dialog between devs and players on these technical topics. And along those lines...

Originally by: Serena Greyskull
I strongly suggest that you start talking to FC's of bigger alliances. I would be more than happy to fill in a questionnaire if this helps, but you would be better off actually TALKING to us, and asking what the conditions of the fight was, and how this affected our gameplay. You would be surprised about how much information we have concerning not only the fight itself, but the conditions leading up to a fight, and so on.

All we want is for the lag to be fixed so we can have ~goodfights~, but you have to talk to us more if you want information.


@CCP GingerDude: Has CCP ever considered conducting Focus Group discussions on voice comms with groups of 6-12 FCs from bigger alliances and some key devs from the lag team? It would give a really good opportunity to gather useful information and brainstorm. Conversation is often much more productive than collecting written data, because the volume of information that can be shared and the value of free association and bouncing ideas off each other cannot be overrated.

I organized and conducted some focus groups like this during my CSM campaign and have done a lot of them in my professional life as well. If you want some ideas on how to conduct them in a manageable and productive way, just request access to the internal CSM forums and we can have a discussion there about it.



tbh, even a big fat REPORT LAGGY SYSTEM BUTTON with a bunch of questions we can answer taped to our HUD would be a great improvement

CCP Warlock

Posted - 2010.09.10 17:32:00 - [44]
 

Originally by: Serena Greyskull
Edited by: Serena Greyskull on 10/09/2010 17:11:04
So, do you still not know whats causing lag on small scale fights?

We were in a fight the other day where there was no more than 150 combatants (yes i know thats not too small scale but come on this should be handled easily and normally is), and the lag was so bad it took 70 of us 20 mins to kill a carrier. Module lag was so bad, that one would be permajammed by a pos, even if you warped out and warped back in, you would still be jammed.

I strongly suggest that you start talking to FC's of bigger alliances. I would be more than happy to fill in a questionnaire if this helps, but you would be better off actually TALKING to us, and asking what the conditions of the fight was, and how this affected our gameplay. You would be surprised about how much information we have concerning not only the fight itself, but the conditions leading up to a fight, and so on.

All we want is for the lag to be fixed so we can have ~goodfights~, but you have to talk to us more if you want information.

/rant off


Can you let me know - either here or in game mail - approximately when, and which system this was in?

We are working our way down a list of a variety of theories about this - and it probably isn't a single cause - but reports of known instances are certainly helpful.


Ban Doga
Posted - 2010.09.10 18:04:00 - [45]
 

Originally by: CCP Warlock
Originally by: Serena Greyskull
Edited by: Serena Greyskull on 10/09/2010 17:11:04
So, do you still not know whats causing lag on small scale fights?

We were in a fight the other day where there was no more than 150 combatants (yes i know thats not too small scale but come on this should be handled easily and normally is), and the lag was so bad it took 70 of us 20 mins to kill a carrier. Module lag was so bad, that one would be permajammed by a pos, even if you warped out and warped back in, you would still be jammed.

I strongly suggest that you start talking to FC's of bigger alliances. I would be more than happy to fill in a questionnaire if this helps, but you would be better off actually TALKING to us, and asking what the conditions of the fight was, and how this affected our gameplay. You would be surprised about how much information we have concerning not only the fight itself, but the conditions leading up to a fight, and so on.

All we want is for the lag to be fixed so we can have ~goodfights~, but you have to talk to us more if you want information.

/rant off


Can you let me know - either here or in game mail - approximately when, and which system this was in?

We are working our way down a list of a variety of theories about this - and it probably isn't a single cause - but reports of known instances are certainly helpful.




That's what happens when your first line of customer support ignores complaints for months and keeps telling that "everything is working as it should":
You have to get back to your customers and ask for the information that they used to provide willingly and for free.

Seriously just browse through the petition system (or even the forums for that matter) and you will find dozens of "Hey, system XYZ just broken down, we are all stuck".

CCP GingerDude

Posted - 2010.09.10 18:34:00 - [46]
 

Originally by: Mynxee
Edited by: Mynxee on 10/09/2010 17:28:20

@CCP GingerDude: Has CCP ever considered conducting Focus Group discussions on voice comms with groups of 6-12 FCs from bigger alliances and some key devs from the lag team? It would give a really good opportunity to gather useful information and brainstorm. Conversation is often much more productive than collecting written data, because the volume of information that can be shared is so high. Besides that, the value of free association, potential for making synergystic connections, and just plain bouncing ideas off each other cannot be overrated.

I organized and conducted some focus groups like this during my CSM campaign and have done a lot of them in my professional life as well. If you want some ideas on how to conduct them in a manageable and productive way, just request access to the internal CSM forums and we can have a discussion there about it.



Thanks for the offer. We'll see what transpires. We are already in dialog with several FCs and other leaders who show up to our mass tests on Singularity. And I agree that their input is quite valuable. Feel free to join us :)

Serena Greyskull
Rens 911
GoonSwarm
Posted - 2010.09.10 18:38:00 - [47]
 

The fight happened on the 25/8/2010 and was in AXDX. I think the fight started around 20:00 eve time, but i cant be sure.

this is the battlereport:

https://www.pandemic-legion.com/killboard/view_battle.php?start_time=2010-08-25 20:55:00&end_time=2010-08-25 22:33:00&system=AXDX-F

The battlereport shows only 22 hostiles, the reason for this is we took virtually no losses the entire fight, therefore only people we killed showed up on the battlereport. Also, the battle lasted over 6 hours, so the 90 freindlies that you see on the BR, is because people joined and left the fleet over that time. My fleet numbers never exceeded 70.

Hostile numbers were around 80, although there was around 200 in local.

I cant tell you how long we spent killing those two carriers, but it was over 3 hours. This was down to both hostile carriers repping the ones we were trying to kill, but mostly due to lag.

It actually took us almost 3 hours to kill the cynojammer that was our main objective, even though by the end there was only around 50 people total on grid.

I cant actually get on eve atm because my main comp is down, but if you have any questions ill be reading this thread.

Serena Greyskull
Rens 911
GoonSwarm
Posted - 2010.09.10 18:47:00 - [48]
 

Originally by: CCP GingerDude
Originally by: Mynxee
Edited by: Mynxee on 10/09/2010 17:28:20

@CCP GingerDude: Has CCP ever considered conducting Focus Group discussions on voice comms with groups of 6-12 FCs from bigger alliances and some key devs from the lag team? It would give a really good opportunity to gather useful information and brainstorm. Conversation is often much more productive than collecting written data, because the volume of information that can be shared is so high. Besides that, the value of free association, potential for making synergystic connections, and just plain bouncing ideas off each other cannot be overrated.

I organized and conducted some focus groups like this during my CSM campaign and have done a lot of them in my professional life as well. If you want some ideas on how to conduct them in a manageable and productive way, just request access to the internal CSM forums and we can have a discussion there about it.



Thanks for the offer. We'll see what transpires. We are already in dialog with several FCs and other leaders who show up to our mass tests on Singularity. And I agree that their input is quite valuable. Feel free to join us :)


If I was to ask the majority of PL's Fc's why they dont turn up to these tests (which they dont), it would be because they spend most of their spare time playing eve, and FC'ing ops, which simply doesnt leave any time for mass testing.

If you want full cooperation, get hold of alliance leaders in game. They know who their best FC's are and they will give you the best information. A lot of these guys work in computers anwyay, and can give you very valuble info. Just dont expect them to answer evemails :P

Saying that i might turn up to your next mass test, i havent been on one yet.

Kayscha
Posted - 2010.09.10 19:05:00 - [49]
 

Originally by: Germaldi's sister
Edited by: Germaldi''s sister on 10/09/2010 12:57:48
Edited by: Germaldi''s sister on 10/09/2010 12:36:54
well what if instead of a dual state transfer system ye changed it to a tri state transfer system... state 1 originatining system state (ship jumps at gate) -> state 2 in hyperspace state (ship accelarates to hyperspeed and flies between gates in hyperspace) -> state 3 destination system (ship drops out of hyperspace within 15km of the destination gate/cyno.

During the hyperspace flight sequence the servers hand over the client info from origin to destination.

sort of like how warping from grid to grid works or even like how accel gates work in plex's. when u iniatate warp u cannot stop it by logging off... the same for jumping systems, the ship will continue to go to its destination then when it comes out of hyperspace the 60 second timer (if no aggro) or 15min timer (if aggro'd) starts. to prevent cheating using the logoffski trick



This.

Exactly what is CCP's stance on this (beyond saying "we don't want that")? I don't see how this could be severely exploited. Players would be inserted into the target system when that node is ready for them, no matter what the client did (or failed to do because it logged off or crashed).

...and you might encounter the Thargoid/Jovian Battle Fleet in hyperspace, too... (j/k)

DaiTengu
Gallente
GoonWaffe
Goonswarm Federation
Posted - 2010.09.10 19:32:00 - [50]
 

Edited by: DaiTengu on 10/09/2010 19:32:40
Originally by: Kayscha
Originally by: Germaldi's sister
Edited by: Germaldi''s sister on 10/09/2010 12:57:48
Edited by: Germaldi''s sister on 10/09/2010 12:36:54
well what if instead of a dual state transfer system ye changed it to a tri state transfer system... state 1 originatining system state (ship jumps at gate) -> state 2 in hyperspace state (ship accelarates to hyperspeed and flies between gates in hyperspace) -> state 3 destination system (ship drops out of hyperspace within 15km of the destination gate/cyno.

During the hyperspace flight sequence the servers hand over the client info from origin to destination.

sort of like how warping from grid to grid works or even like how accel gates work in plex's. when u iniatate warp u cannot stop it by logging off... the same for jumping systems, the ship will continue to go to its destination then when it comes out of hyperspace the 60 second timer (if no aggro) or 15min timer (if aggro'd) starts. to prevent cheating using the logoffski trick



This.

Exactly what is CCP's stance on this (beyond saying "we don't want that")? I don't see how this could be severely exploited. Players would be inserted into the target system when that node is ready for them, no matter what the client did (or failed to do because it logged off or crashed).

...and you might encounter the Thargoid/Jovian Battle Fleet in hyperspace, too... (j/k)


Due to the nature of programs, and the nature of humans (eg: tinkerers, hackers, etc.) this probably wouldn't work real well. You'd end up with some dudes stuck solely in hyperspace then.

Were there to be a glitch, where the receiving node crashes, an entire fleet could be 'lost' with no recourse to get out but petitions.

I'm sure there's other technical reasons for this as well. A third state may require another server, etc.

Gecko O'Bac
Deep Core Mining Inc.
Posted - 2010.09.10 20:19:00 - [51]
 

I realize it's not "the" lagfix but it's time to change the missile system to a more server and player friendly one... I wrote a thread about this months ago (though I realize it's hardly impressive material), and not only changing the system to one which is closer to how turret works (in terms of "to hit" calculations) would allow for a lower server load... It'd allow also for nicer missile graphics and, in particular, animated missile launchers.

I do understand that it's a big change but the server would benefit from it.

Bomberlocks
Minmatar
CTRL-Q
Posted - 2010.09.10 20:37:00 - [52]
 

Good blog. I was pretty sure it was something like this. Client server session state change corruption. This, I think also explains why you can get the effect of one single pilot clicking jump on a gate, the gate flashing, but the pilot staying exactly where he was and being informed that the next time he can jump is 2 minutes 54 seconds away.

Rn Bonnet
Sniggerdly
Pandemic Legion
Posted - 2010.09.10 20:46:00 - [53]
 

Quote:
If I was to ask the majority of PL's Fc's why they dont turn up to these tests (which they dont), it would be because they spend most of their spare time playing eve, and FC'ing ops, which simply doesnt leave any time for mass testing.

If you want full cooperation, get hold of alliance leaders in game. They know who their best FC's are and they will give you the best information. A lot of these guys work in computers anwyay, and can give you very valuble info. Just dont expect them to answer evemails :P


I think something like 1/2 of the PL leadership works in IT with a significant portion of them being actual programmers/network engineers so...

For example it has been well known for awhile (before this fix):
That staging jumps massively decreased the chance of Y-2 style bugs.
That coming in on separate grids also decreased this chance of happening.

Both of which jive with your explanation.


Something which might be nice is a category of petitions, or simply a game forum reserved for "hey CCP dev come look at this bug" we are experiencing right now. That way it would be possible to inform CCP of when such "exceptional" circumstances happen so they themselves can come look at it.

Rn Bonnet
Sniggerdly
Pandemic Legion
Posted - 2010.09.10 21:04:00 - [54]
 

Oh and the not loading grid bit has also been known for awhile.

However unlike the other, not loading grid does let you log and eventually disappear, also your system "colors" never change with the bug you just fixed, where as not loading grid at least gives you some state info IIRC.

CCP Explorer

Posted - 2010.09.10 21:25:00 - [55]
 

Originally by: Gecko O'Bac
I realize it's not "the" lagfix but it's time to change the missile system to a more server and player friendly one... I wrote a thread about this months ago (though I realize it's hardly impressive material), and not only changing the system to one which is closer to how turret works (in terms of "to hit" calculations) would allow for a lower server load... It'd allow also for nicer missile graphics and, in particular, animated missile launchers.

I do understand that it's a big change but the server would benefit from it.
Indeed it would and it's on our list.

Blazde
4S Corporation
Morsus Mihi
Posted - 2010.09.10 21:26:00 - [56]
 

They've been reproducing the not-loading-grid problem the last 2 weeks in masstesting, and seemed like maybe some successful testing of a fix yesterday.

Jim Luc
Caldari
Rule of Five
Vera Cruz Alliance
Posted - 2010.09.10 21:58:00 - [57]
 

Originally by: CCP Explorer
Originally by: Gecko O'Bac
I realize it's not "the" lagfix but it's time to change the missile system to a more server and player friendly one... I wrote a thread about this months ago (though I realize it's hardly impressive material), and not only changing the system to one which is closer to how turret works (in terms of "to hit" calculations) would allow for a lower server load... It'd allow also for nicer missile graphics and, in particular, animated missile launchers.

I do understand that it's a big change but the server would benefit from it.
Indeed it would and it's on our list.


Yes. I want animated missile launchers and tons of missile graphics YESTERDAY. My grouped missiles look strange when all fired at once. Twisted Evil

Zanes Shoubje
Posted - 2010.09.10 22:10:00 - [58]
 

Nice blog. Love the how stuff works stuff.

And did I understand it correctly?

Because of Caldari

Gecko O'Bac
Deep Core Mining Inc.
Posted - 2010.09.10 22:24:00 - [59]
 

Originally by: CCP Explorer
Originally by: Gecko O'Bac
I realize it's not "the" lagfix but it's time to change the missile system to a more server and player friendly one... I wrote a thread about this months ago (though I realize it's hardly impressive material), and not only changing the system to one which is closer to how turret works (in terms of "to hit" calculations) would allow for a lower server load... It'd allow also for nicer missile graphics and, in particular, animated missile launchers.

I do understand that it's a big change but the server would benefit from it.
Indeed it would and it's on our list.


You just went up 2 notches in my list of favorite CCP employees :D

Grozen
Caldari
Titan Core
Posted - 2010.09.10 23:07:00 - [60]
 

I wish to congratulate you for informing everyone about what is being done, most mmos even the biggest ones communication is very lacking and nobody goes in this much detail.Keep it up players like when you tickle em.Very Happy


Pages: 1 [2] 3

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only