open All Channels
seplocked EVE Information Portal
blankseplocked New Dev Blog: Fostering meaningful human interaction, through testing
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: [1] 2 3 4 5 6

Author Topic

CCP Fallout

Posted - 2010.08.16 17:22:00 - [1]
 

So, about those mass-tests CCP Tanis has been talking about for some time. Check out his new dev blog for all the details.

CCP Tanis

Posted - 2010.08.16 17:25:00 - [2]
 

First! :p

Seriously, now go read the blog, it's chock full of info. We'll be around for a bit after to help answer any lingering questions. Also be aware that there are still several more blogs to be put out that will delve deeper into various issues and the work we're doing behind the scenes.

AtlantisX
Posted - 2010.08.16 17:27:00 - [3]
 

Wow.

Big thanks for the update. You may not have fixed lag, but this shows what you've been up to.

Tis good to know.

Chribba
Otherworld Enterprises
Otherworld Empire
Posted - 2010.08.16 17:29:00 - [4]
 

Very nice Smile

bitter vetiranarian
Posted - 2010.08.16 18:03:00 - [5]
 

Quote:
Since I'm not one who likes excuses I'll just skip to the solution here. We've started working directly with the CSM members to ensure we're keeping open lines of communication, feedback, and information between CCP and the players of EVE about mass-tests, their impact, and their findings.

...

We have struck up a new commitment with the CSM to build a sustainable and open dialog between CCP and the players about the quality and performance of EVE.


People will be holding you to these statements just fyi.

Awaiting more dev blogs with mild interest.

Bomberlocks
Minmatar
CTRL-Q
Posted - 2010.08.16 18:18:00 - [6]
 

Edited by: Bomberlocks on 16/08/2010 18:20:07
This is about the clearest, most straightforward, honest dev-blog I've ever read. Kudos to you Tanis. If you guys can keep this clear, open comunication up, I think you'll get a lot of support from us, both in terms of test participation and sympathy.

And kudos also to the CSM for sticking through the uphill battle of getting through to you guys to show you what we were feeling.

Excellent!

Edit: Just to be clear about it: You're not out of the woods yet. We really do expect the communication to continue in this manner, free of PR bullsh*t and overzealous use of the word awesome. Razz

Bomberlocks
Minmatar
CTRL-Q
Posted - 2010.08.16 18:23:00 - [7]
 

I also have a question about the testing and earlier comments from CCP devs about dynamic allocation of server resources to nodes. How is this coming on and how do these tests influence that process?

Mynxee
Veto.
Veto Corp
Posted - 2010.08.16 18:28:00 - [8]
 

Wow, nice long dev blog AND with charts! I will read that later this afternoon when I have time to really digest it.

Originally by: CCP Tanis
I wanted to bring up the CSM again, before signing off on this blog. Mass-testing is not just about data, or testing fixes, it is about involving the EVE community in assessing the overall quality of EVE. We feel very strongly that EVE's players must be involved, at some level, in the discussion about the quality of the game. You folks are, after all, the ones who play it day in and day out. You spend your free hours in the universe which we've built, so you should always have a say in how good, or bad, you think that universe is working. Obviously, it isn't feasible for us to have one-on-one discussions with everyone, so we have to find some more workable middle ground. We believe we've found a very appropriate one in the CSM. These are the people whom you all elect to represent you to CCP. These are the people who will carry your issues, your gripes, and your kudos to us.

We have struck up a new commitment with the CSM to build a sustainable and open dialog between CCP and the players about the quality and performance of EVE. This means that the CSM will be able to bring concerns to us more readily and that we can, in return, work together to ensure that we effectively communicate about those issues with all of you. This isn't limited to just the current causes of lag, but any issue that may crop up later that makes EVE run poorly, or limits the ability for people to have amazing 1000+ fleet fight once again. We feel this is a very positive change and look forward to working more closely with the CSM towards more effective communication and better mass-tests.


I want to thank CCP Tanis for taking the initiative to engage the CSM. Much respect and admiration for that, Tanis. I for one appreciate your direct, no bullcrap approach Twisted Evil and look forward to ongoing CSM collaboration with you and your team. :)


Bomberlocks
Minmatar
CTRL-Q
Posted - 2010.08.16 18:28:00 - [9]
 

I wanted to edit my question above, but I think I'd rather make it another separate question:

Could you perhaps explain how the process that occurs when a session change occurs, such as jumping through a gate? I don't mean the tiny details, but just an overview of the client-server communication and an overview of what goes on at the server during this process?

Dacil Arandur
Habitual Euthanasia
Pandemic Legion
Posted - 2010.08.16 18:29:00 - [10]
 

Thank you for this straight-forward, information-filled and open devblog.
Fallout was right about this being an awesome couple weeks!

I'm excited to read more!

Nuadi
Posted - 2010.08.16 18:36:00 - [11]
 

Wow, that's a huge wall of text. Read it once... going to have to read again. Simply wanted to give my eyes a break and say Thank You.

Arous Drephius
Posted - 2010.08.16 18:38:00 - [12]
 

Quote:
we've now got a new "Singularity updater" tool that we think will go a long way to making everyone's lives easier here



Jack Gilligan
Caldari Provisions
Posted - 2010.08.16 18:46:00 - [13]
 

Nice update, but one thing I would urge you guys to do is to NOT put all your eggs in the CSM basket. As we've seen, CSM members have their OWN agendas oftentimes that do not match the goals of the player base as a whole. Which is all the more remarkable that they were pretty much unified in their "FIX THE DAMN LAG" message.

Don't use the CSM as an excuse to not take feedback from people who aren't in it.

Obsidian Hawk
RONA Corporation
RONA Directorate
Posted - 2010.08.16 18:47:00 - [14]
 

Someone buy Tanis a beer! Great blog man and answers a lot of questions.


Also, I hope you arent shy. You will be wearing a mankini and swimming in the pond/fountain by ccp hq when you lose your mass testing bet.

CCP Warlock

Posted - 2010.08.16 18:48:00 - [15]
 

I'll answer both at once:

Originally by: Bomberlocks
I also have a question about the testing and earlier comments from CCP devs about dynamic allocation of server resources to nodes. How is this coming on and how do these tests influence that process?


The mass tests generally don't have much influence on this. We monitor load during them, and use that information to look at particular systems if we see them misbehaving themselves. In general we try to load balance systems across nodes, and that's game systems as well as solar systems, but a lot of that is done manually after looking at cluster performance data. More dynamic load balancing is something we are looking at closely, but carefully. There can be some very interesting hysteresis effects if that isn't handled very carefully - the load gets moved off somewhere else, that server then gets overloaded, load gets moved back...


Originally by: Bomberlocks
I wanted to edit my question above, but I think I'd rather make it another separate question:

Could you perhaps explain how the process that occurs when a session change occurs, such as jumping through a gate? I don't mean the tiny details, but just an overview of the client-server communication and an overview of what goes on at the server during this process?


Quite a lot is the short answer. Once the client has requested a jump, the server has to arbitrate communication with all the sub-systems that are involved with that client, and make sure that they have all done what they individually need to do to transfer the client to the new system. Some of those systems will be running on the destination system, some elsewhere in the cluster, so there is a fair bit of inter-process communication involved. Since clients can individually be interacting with different systems - some clients are in Fleets for example, most aren't, it can get quite involved.

Minchurra
Caldari
Posted - 2010.08.16 18:49:00 - [16]
 

I've been to a number of tests, but I haven't seen one where you've allowed the use of bombs. In my experience when bombs start going off I get severe lag (non-graphical related)and its a very common occurance especially when lots of BS are involved.

Similarly, have you tried bridging a couple of hundred into a system with a sizeable number in local? It might be similar to entering through a gate, but I've lost count of the times I've locked up from entering a crowded system through a titan bridge.

CCP Tanis

Posted - 2010.08.16 18:54:00 - [17]
 

Originally by: Minchurra
I've been to a number of tests, but I haven't seen one where you've allowed the use of bombs. In my experience when bombs start going off I get severe lag (non-graphical related)and its a very common occurance especially when lots of BS are involved.

Similarly, have you tried bridging a couple of hundred into a system with a sizeable number in local? It might be similar to entering through a gate, but I've lost count of the times I've locked up from entering a crowded system through a titan bridge.


We've just started allowing bombs during mass-tests (as of the last test on Aug 5), for exactly this reason. It adds complications with people being incidentally podded, but clones work around that well enough. In addition, we've also been using titan bridges for the last few tests. I don't think we manage to get as full use out of them as we could, but we hope that by also including players as FC's this will improve over time.

TamiyaCowboy
Caldari
KRAKEN FLEET
Posted - 2010.08.16 18:55:00 - [18]
 

but we've now got a new "Singularity updater" tool that we think will go a long way to making everyone's lives easier here. This is just a simple executable that you'd download, run, and point at your working TQ client. The tool then takes care of copying the client and updating it to the correct Singularity build, automatically. It will also give you a new link to the client, which connects it directly to Singularity. All of this will be with just three easy clicks for the end-user. We're still working out some kinks, and trying to make it run a bit faster, but we hope to have it out in public use very soon.

so another new tool ? does this one sit behind a desk lol

lets cut the rubbish and go straight to the point.

you complain about testing and player not joining, at current the way we have to connect to SISI is a big fail. not everyone is computer savy. now installer tool lol, patch instaler had problems and sometimes still does, may i also point to boot.ini problem, i dont thing even 5,000 would trust installers.

here is what i think CCP should do,for a start lets just have a new lite client just for sisi, simple easy and not a huge mind boggling brain fart most of us none computer savy types have to go through to just get testserver up. it will boost player coverage in sisi 100 fold and make testing for you more like TQ. you want more players and more testing of large fleets maybe a new secondary client is in order.

the old way of joining SISI needs to change a secondary LITE CLIENT would solve this problem and the brain ache for those that wish to join in. no offence but i dont trust a new point to clinet type tool, last time an installer pointed to a TQ client you nearly wiped out 45,000+ pc's and mac'sLaughing

so what say CCP, move with the times give your players a second client for testing maybe i CBA to join and not spend hours downloading new tq clients when sis patching goes pete tong

something somethingdark
Posted - 2010.08.16 18:57:00 - [19]
 

XBOX HUGE wall of text
Actual content thats new and previously unknown : 4% (much like recent eve expansions)

anyways
here is an idea for a mass test
Load up an old Apocrypha client/server on sissi and then go compare logs

Stick Cult
Posted - 2010.08.16 18:58:00 - [20]
 

Edited by: Stick Cult on 16/08/2010 19:10:10
First off, nice blog. It's one thing to see the mass tests happening, it's another to actually see results, keep up the good work.

Second... this:
Originally by: Blog
We've been considering things like "Singularity rewards," giving game-time for participating in "x" number of tests

Oh god, YES. I would definitely, along with many others I'm sure, make time for the mass tests if this happened.
(I've said this before, but I want to put it where it will be read by devs..) But, I'd settle for less. Following the 2 day downtime, we got 100k skillpoint pools. What about this: for every mass test you get some amount of skillpoints (2-5 million) on your Singularity account. It's enough so that players won't leave TQ so they can go fly their titans on the test server, but enough to encourage participation. In the end, you'd also end up with more people on the test server, which ultimately leads to ~more testing~, which is always a good thing. Can a dev say "no this will never happen" so I can stop talking about it? Razz

Once again, thanks for the great blog.

CCP Warlock

Posted - 2010.08.16 19:01:00 - [21]
 

Originally by: something somethingdark
XBOX HUGE wall of text
Actual content thats new and previously unknown : 4% (much like recent eve expansions)

anyways
here is an idea for a mass test
Load up an old Apocrypha client/server on sissi and then go compare logs


Unfortunately if logs could solve this problem it would be fixed. My eyes bleed from looking at logs so much the last few months.

Bella Yar
Posted - 2010.08.16 19:02:00 - [22]
 

WHY IS CCP WASTING RESOURCES ON BLOGS WHEN THEY SHOULD BE FIXING TEH LAGZ!!!!!111111

Bagehi
Association of Commonwealth Enterprises
Posted - 2010.08.16 19:05:00 - [23]
 

Edited by: Bagehi on 16/08/2010 19:24:20
Excellent stuff.

Originally by: CCP Warlock
Quite a lot is the short answer. Once the client has requested a jump, the server has to arbitrate communication with all the sub-systems that are involved with that client, and make sure that they have all done what they individually need to do to transfer the client to the new system. Some of those systems will be running on the destination system, some elsewhere in the cluster, so there is a fair bit of inter-process communication involved. Since clients can individually be interacting with different systems - some clients are in Fleets for example, most aren't, it can get quite involved.


I remember a thread in Assembly Hall related to this and the black screens of death. Has there been discussion about the possibility of not spawning a ship/character into the system/grid they are moving to until their client is ready/can load? I admit a general lack of knowledge of how these systems work, but is it feasible to not spawn everything in until the client is ready to load?

Zendoren
Aktaeon Industries
The Black Armada
Posted - 2010.08.16 19:05:00 - [24]
 

So short synopsis of the blog is that:

1) Thanks for the data with mass-testing. Please, keep coming to mass-testing!
2) We still don't know what is going on with LAG but we fixed mod cycling! \o/ (that's if they turn on)
3) We will be lowering the barrier of entry for SiSi soon™
4) SiSi is getting a Hardware boost as we speak!
5) Stay tuned for more info.
6) Contact your local CSM member for more info. or questions


Charles37
Posted - 2010.08.16 19:05:00 - [25]
 

Edited by: Charles37 on 16/08/2010 19:09:15
I just wanted to thank CCP Tanis for this *fantastic* dev blog about the mass testing program that has been going on.

You mentioned that people stop showing up if they don't get any feedback on what kind of results are gleaned, and I think that this is spot on. I've attended 3-4 of the mass testing events so far, and while I could have made a few others, I didn't, in part because it was such a one sided event. 400+ people dedicate an hour or two of their time on the test server, which doesn't include time spent patching, writing impressions in the feedback thread, etc, and we never have any idea what changes are being tested, what data is being gathered, etc. It doesn't have to be a full featured blog like this, but having a place to announce what will be tested and the results of the test (could we hope for some more pretty graphs?) I think will motivate more people to help out with the mass testing effort. Instead of lumping all the mass testing results onto the SISI forum, have you considered making a separate webpage for them, where you could also post mini blogs talking about what's going to be looked at during the next mass test, results of the most recent one, etc etc?

When it comes to the issue of incentives/rewards for participating in mass testing, I feel very strongly that they should be SISI ONLY. Perhaps free skill points for each mass test attended (to also help with the fact that the mirror isn't always up to date) or a "Medal of Mass-Testing" with various ranks, ala the FW Insignias. A Red vs Blue style set up might be interesting if you find something meaningful to track on a website and for the two teams to compete over.


Edit:
Originally by: CCP Tanis
We've just started allowing bombs during mass-tests (as of the last test on Aug 5), for exactly this reason. It adds complications with people being incidentally podded, but clones work around that well enough. In addition, we've also been using titan bridges for the last few tests. I don't think we manage to get as full use out of them as we could, but we hope that by also including players as FC's this will improve over time.


I had assumed that you were limiting the use of Bombs/Smartbombs to try and isolate specific problems as much as possible?

Axemaster
Posted - 2010.08.16 19:06:00 - [26]
 

Edited by: Axemaster on 16/08/2010 19:30:10
Edited by: Axemaster on 16/08/2010 19:14:28
Good to see that CCP is finally realizing that the way forward is quality, not quantity. Eve is such a long term game that the recent strategy of "grab 10000 noobs for 3 weeks" was abortive from the start, simply because it is so directly opposed to reality. Only madmen would take a years-old playerbase for granted.

Thank you for stepping back from the brink.

Edit: Unfortunately the fact that this problem came up at all shows that whoever was directing things was SERIOUSLY disconnected from the game. Hopefully that will never happen again.

Edit2: In regards to load balancing/switching and the node reinforcement issue - I think you might be putting a bit too much emphasis on having it be "real time". I doubt anyone would complain if the game paused for 30-60 seconds while transferring the 1000 player battle to a stronger node, while showing a "node reinforcing" message. After all, it beats dying at the black screen.

mechtech
SRS Industries
SRS.
Posted - 2010.08.16 19:07:00 - [27]
 

Nice blog!

Bomberlocks
Minmatar
CTRL-Q
Posted - 2010.08.16 19:08:00 - [28]
 

Originally by: CCP Warlock
I'll answer both at once:

Originally by: Bomberlocks
I also have a question about the testing and earlier comments from CCP devs about dynamic allocation of server resources to nodes. How is this coming on and how do these tests influence that process?


The mass tests generally don't have much influence on this. We monitor load during them, and use that information to look at particular systems if we see them misbehaving themselves. In general we try to load balance systems across nodes, and that's game systems as well as solar systems, but a lot of that is done manually after looking at cluster performance data. More dynamic load balancing is something we are looking at closely, but carefully. There can be some very interesting hysteresis effects if that isn't handled very carefully - the load gets moved off somewhere else, that server then gets overloaded, load gets moved back...


Thank you very much for taking the time to answer my questions. I presume you have a lot of edge cases in this process that need very careful handling. I think a follow up question would be what kind of load prediction would you use to pre-empt server overloading?
Originally by: CCP Warlock

Originally by: Bomberlocks
I wanted to edit my question above, but I think I'd rather make it another separate question:

Could you perhaps explain how the process that occurs when a session change occurs, such as jumping through a gate? I don't mean the tiny details, but just an overview of the client-server communication and an overview of what goes on at the server during this process?


Quite a lot is the short answer. Once the client has requested a jump, the server has to arbitrate communication with all the sub-systems that are involved with that client, and make sure that they have all done what they individually need to do to transfer the client to the new system. Some of those systems will be running on the destination system, some elsewhere in the cluster, so there is a fair bit of inter-process communication involved. Since clients can individually be interacting with different systems - some clients are in Fleets for example, most aren't, it can get quite involved.

If it isn't too much trouble, could you tell me how error tolerant these processes are? I ask this in the context of the problem of, for example, a ship sticking on a gate, where the "jump session handler", if I may call it that, seems to think a jump has occurred whereas the process that handles the ship state seems to think it hasn't.

Zendoren
Aktaeon Industries
The Black Armada
Posted - 2010.08.16 19:14:00 - [29]
 

Edited by: Zendoren on 16/08/2010 19:23:34
One question.

Is the Server-side coded for multiprocessing to take advantage of multi-core CPUs? If so, has there been stability and regression testing done on this code to ensure that process jobs ore not executed multiple times by multiple cores?

Edit: Change nomenclature

Genya Arikaido
Posted - 2010.08.16 19:32:00 - [30]
 

Informative.

Quote:
Improvements to the Planetary Interaction UI, making it easier to use


DETAILS!! If PI gets rid of the clickfest, I may try it again.


My sole question is...what's being done about the star system resource allocation? This being where theoretically a single star system could consume all node resources in the entire cluster, moving the glass ceiling from the limits of a single node to the limits of the entire cluster's hardware. Naturally you'd want to impose some artificial limit, or only partially apply the dynamic reallocation of resources to underused systems, but if you could do this...fixing lag would be as simple as adding more blades.

From what I understand though, this isn't so much of a limitation of EVE's architecture as it is the design of the cluster's hardware architecture in the way it manages sessions between nodes. The primary problem being in that transferring a session from one node to another while in the same system would create a moment of "loading...please wait" while flying around, as the session data is not globally available due to access speed problems.

Please, correct the crap out of me if I'm wrong. Most of this is just what I've gathered from devblogs and presentations at Fanfests. Though I do have some direct education in HPC in the process of earning my CompSci Degree (with minor in astrophysics...no joke, I love this stuff). I just don't necessarily get all the hardware limitations of an HPC cluster design.


Pages: [1] 2 3 4 5 6

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only