| Author |
Topic |
CCP Fallout
 |
Posted - 2009.09.02 16:25:00 - [ 1]
CCP Lemur fills us in on the important role our volunteer bug hunters have in New Eden in his newest dev blog. If you're interested in learning how you can become part of this important team, you should read the blog! |
ISD BH Sabaoth
Amarr
ISD BH
|
Posted - 2009.09.02 16:57:00 - [ 2]
Blog is full of win! ;) |
GM Renril
 |
Posted - 2009.09.02 16:58:00 - [ 3]
I agree  |
Skandrannon22 |
Posted - 2009.09.02 17:00:00 - [ 4]
Are you guys bumping your own topic?
Love it!
and I appreciate all the folks who squash bugs, I'd like to become one, but then I'd have to shut off all outside activity, quit work, and play on free wifi under a bridge...
-Skan |
ISD Libertina
ISD STAR
|
Posted - 2009.09.02 17:36:00 - [ 5]
Edited by: ISD Libertina on 09/09/2009 17:33:42I see Mr Renril volunteers to write one about my team and why the GM's love us so much  |
Abrazzar |
Posted - 2009.09.02 18:06:00 - [ 6]
Wow. The second graph is quite sobering on the quality of bug reports you people get. Over half of them get trashed.  |
something somethingdark |
Posted - 2009.09.02 18:22:00 - [ 7]
yeah i wouldnt exactly call them thrashed if you write back to inquire for more details
what would be interesting is how many bugreports ACTUALY get trashed and on average how much back and forth there is between the people involved....
i very rarely have a bugreport dismissed right away(as in its already fixed internaly) ... but i often have to insist that this bug realy exists and i have to wait for example till somebody with close to my hardware comes along and tests it .... |
Jason Edwards
Internet Tough Guy
Spreadsheets Online |
Posted - 2009.09.02 19:11:00 - [ 8]
Blog is so fail. Bug hunters are good in theory. Horrible fail in practice.
Lets go through the process of a literal bug I reported. Which mirrors soooo many others.
I find a bug; not very hard...
I go... um what the hell just happened? That's wierd. How'd I do that. I repeat it. I boot into ubuntu and run eve. This particular one happened when the native linux client was legit still. I reproduce on there. This pretty much confirms it's a bug. I then boohooo in fd- local and see if anyone else has seen the bug. I get confirmations.
I bother to go through a billion different login screens which firefox doesnt like to save the password for. Finally get to the report. This particular bug was not graphics related. Also since the bug goes through both clients and linux native... linux has no dxdiag. It does have LSHW which is ListHardware. I just added the logserver. No crash happened. I looked at the log server and never really saw much.
I posted the method of my madness. I had a friend replicate it. Sometimes not an option because of the nature of the bug; potential for exploits PERHAPS.
First time it got filtered. AKA TRASHED. All they wanted was a dxdiag. I'd mentioned it was happening on linux also and had other people also were busted up. My misanthropic tendencies already pushed. So I post dxdiag. They come back with "i couldnt replicate". Too which I said I dont care even slightly anymore.
Bug then makes it to tranquility and my friend wonders if I'd reported it or not like I said I would.
So sad. How about a graph showing how many bugs made it to tranquility that were filtered by the bug hunters and obviously werent fixed thusly?
Also... since the whole process is so closed off. You could have 10 good bug hunters working on the same problem.
If the bug reporting mechanism was opened up such that. If I had a bug. I could then go find if another group ticketed it. When the bug hunter comes along to ask for a dxdiag or whatever. Other people can unfilter the bug by adding theirs and making the same reports. Adding details as such. Not to mention People can look there and try to replicate the bug themselves. Then when they do. They can give the info. Maybe the bug is a 1/1000 thing for people on sisi. Except when u get to tranquility. It's like 8/10 have the problem.
Take Price History Memory Leak from awhile back. How many people actually check price history on sisi. How about tq? Big difference.
Bug hunters are doing a good job considering; but just a few simply improvements and it could be oh so much better. Not to mention. You could have these unchanging or slowly changing things like dxdiag be things you can/should upload to your account to be able to post a bug.
Worst case scenario someone posts with an out of date dxdiag. The bug isnt related and someone else posts and does the job. |
Gnulpie
Minmatar
Miner Tech |
Posted - 2009.09.02 21:14:00 - [ 9]
Edited by: Gnulpie on 03/09/2009 09:03:27
Nice blog and interesting to read.
Thanks to all the bughunters out there, making Eve a better place \o/
However...
...the blog also shows how incredibly ineffective the current bug system is!
About 40%-60% of the bug reports are getting trashed, causing only lots of work overhead and giving no benefit to the bug hunting.
Another 15%-25% of the bugs are already reported also, causing even more overhead which is completely wasted.
So, the bughunters mostly either trash reports (with explaining in detail why) or close the reports because they report an already known bug.
I think it is more than time to streamline the whole process of bug reporting and bug hunting in such a way, that this tremendous overhead is reduced significantly.
Don't make the life of those brave bughunters more difficult by keeping a bad system, make it as easy as possible with a new, better bugreport system! |
Miyamoto Uroki
Caldari
Sarum Industries |
Posted - 2009.09.03 06:20:00 - [ 10]
What the guys above me said. Doesn't seem to be a very effective bug reporting system. Make it more public.
If you have a bad, big bug in your sisi version and cannot fix it until it would hit TQ, hell then the release shouldn't pass QA anyways. So no possible abuse because of known bugs on test server for the public. |
Leana Darkrider
Minmatar
Creatio -ex- nihilo
The Donkey Rollers |
Posted - 2009.09.03 07:29:00 - [ 11]
Good read!
Good to see that volunteers, becides CCP, are doing their very best to improve this game for everyone. |
Louis deGuerre
Gallente
Malevolence.
|
Posted - 2009.09.03 08:10:00 - [ 12]
Bughunting is a thankless (as seen by the whine poster on the first page) but vital job, which we should all be thankful someone is doing. Keep up the good work !  |
Token Prophets
Walking Ghost Phase
|
Posted - 2009.09.03 08:44:00 - [ 13]
Nice Job!
If only we had some Starship Troopers to kill those bugs once and for all! |
ISD BH Sabaoth
Amarr
ISD BH
|
Posted - 2009.09.03 09:36:00 - [ 14]
Thanks all, the BH’s do carry out a thankless task but they do it for the same reason you guys report the bugs, to make EVE a better place.
There are some good insights into the Filtering system, the current system is old and does need improvement and this is something that is being looked into and hopefully will be tackled in the not too distant future.
I would like to clarify about the statistics you view on the main dev blog page; the trashed reports are quite high as we do get a lot of no complete reports and people thinking the filtering system is actually a petition system. There are also a lot of instances where reports are sent back because further information is needed, the player perhaps doesn’t care about the bug anymore and doesn’t reply... in the end we have to trash it or the filter list will be choked up with extremely old and non-reproducible reports.
If many people report the same bug we do combine this into a ‘defect’ which collates all the experiences/information and reproducible steps from each report into a combined set which enables us and QA to quickly see different variants of the same bug.
I hope this information helps you understand us, what we do and what we can’t, we are only human and we try our best for you guys.
|
Lost Hamster
Hamster Holding Corp |
Posted - 2009.09.03 09:51:00 - [ 15]
I agree that bug hunting is a thankless job.
And I can agree with that, that the current bug reporting system is ineffective.
Would be much better when we could add information to each other bug reports. That would reduce the amount of separate bug reports about the same issue. |
Gnulpie
Minmatar
Miner Tech |
Posted - 2009.09.03 10:02:00 - [ 16]
Edited by: Gnulpie on 04/09/2009 16:40:17 Originally by: ISD BH Sabaoth
Thanks all, the BH’s do carry out a thankless task...
So let's change this and make all the BH carry out a thankful task! You and all the other bughunters have my thanks! It is a job in the background, tiresome maybe or frustrating sometimes, but without you guys, Eve would be a far worse place! So I say: Thank you very much! |
ISD BH Sabaoth
Amarr
ISD BH
|
Posted - 2009.09.03 10:03:00 - [ 17]
Edited by: ISD BH Sabaoth on 03/09/2009 10:05:27
Lost Hamster: That’s exactly what we do with the 'Defects' we will combine bug reports detailing the same bug into 1 single defect for clarification and collation of all relevant information. If this was left to players to do then you could find that bug reports that are not related get added to each other as players can’t check the bugs out as thoroughly as ourselves and QA can. |
Vir Hellnamin
Gradient
Electus Matari |
Posted - 2009.09.03 11:05:00 - [ 18]
Adding info to other bug-reports might make a mess, but just seeing or being able to browse other bug reports would help by enabling us to link/refer to same/similar thing experienced in other bug-reports. Also, would be nice if you could check which other cases are covered already...
(One of the best bug-reports I consider wasn't even "written" by me, I just gathered and updated info from few threads to it...) |
ISD BH Sabaoth
Amarr
ISD BH
|
Posted - 2009.09.03 11:20:00 - [ 19]
Originally by: Vir Hellnamin
Adding info to other bug-reports might make a mess, but just seeing or being able to browse other bug reports would help by enabling us to link/refer to same/similar thing experienced in other bug-reports. Also, would be nice if you could check which other cases are covered already...
(One of the best bug-reports I consider wasn't even "written" by me, I just gathered and updated info from few threads to it...)
You do make a very good point, the only major problem i can see if when we are dealing with 'exploit' reports. Some exploits bug reports are not even listed as exploits they may be tied into a certain mechanic which in part causes something unexpected to happen and results in an advantage through un-designed mechanics. If bug reports are publicly avaliable we have no control over people then being able to find exploits within these reports, now that being said I do actually have something in mind that we may or may not get around to. It is my hope that we will have system that gives people that report bugs some information on the current top reports/defects that are being tackled and also monthly statistical information. This is still in the planning stages and we may not have the resources to dedicate to it yet but I will be pushing for it. |
Nidhiesk |
Posted - 2009.09.03 11:22:00 - [ 20]
I don't mean to be a party breaker but why is it (lately) some features as been disabled after a big patch. For example. the epic mission arc agent was disabled after the 1.5 patch. It was said that in a week it would be fixed but just a couple of days after it was fixed. I have to admit, great work on the time you fixed it cause I thought it would be a nightmare to wait a week.
But like I said, why release a feature which was not tested enough (when I see the info and the timeline)...if it was tested enough in the first place. I did notice that while some patches ago.
I'm also wondering if users could of done somethng to help this and prevent this kind of thing to happen in the future. |
ISD BH Sabaoth
Amarr
ISD BH
|
Posted - 2009.09.03 11:31:00 - [ 21]
I can’t properly answer your question as that is a decision that QA would have made not the Bug Hunters, I can however give my opinion on what probably did happen.
All new features must get tested in various scenarios, some automated some not while old features get 'regression' tested to ensure they still work as intended, in cases like this with agents the tests went fine on the testing servers but there perhaps was some sort of variable/scenario on the live server that caused unexpected results. As any good developer CCP disabled it to limit any damage these undesirable effects could have caused and it was then worked on and re-tested under new conditions to ensure it was working as intended. In this particular case I can’t say if this is exactly what happened or not but it’s likely.
As for anything players can do to help stop this in the future, there is only one thing and that is for more players to actually test these new features and not just ship fittings for your next combat encounter ;) But really we are all human and these things do slip past no matter how many procedures we have in place, with continued support from the players by testing the features we will only get better at squashing these bugs. :) |
Pottsey
Enheduanni Foundation
|
Posted - 2009.09.03 11:42:00 - [ 22]
Edited by: Pottsey on 03/09/2009 11:43:23
ISD BH Sabaoth said “with continued support from the players by testing the features we will only get better at squashing these bugs. :)”
But you don’t support us in testing. Take the epic missions there was massive problems with helping test. Like we could only do the mission once if we could even access the missions in the fist place. No one could do real testing as we couldn’t repeat anything this was ignored time and time again by CCP.
I was very disappointed with the state of epic missions it was like no one even gave it a basic bug run, I understand one or two bugs but not that amount of major ones that made it to the live server. Why have things like shadows be broken for around a year now as well?
|
Nidhiesk |
Posted - 2009.09.03 13:09:00 - [ 23]
Originally by: ISD BH Sabaoth
I can’t properly answer your question as that is a decision that QA would have made not the Bug Hunters, I can however give my opinion on what probably did happen.
All new features must get tested in various scenarios, some automated some not while old features get 'regression' tested to ensure they still work as intended, in cases like this with agents the tests went fine on the testing servers but there perhaps was some sort of variable/scenario on the live server that caused unexpected results. As any good developer CCP disabled it to limit any damage these undesirable effects could have caused and it was then worked on and re-tested under new conditions to ensure it was working as intended. In this particular case I can’t say if this is exactly what happened or not but it’s likely.
As for anything players can do to help stop this in the future, there is only one thing and that is for more players to actually test these new features and not just ship fittings for your next combat encounter ;) But really we are all human and these things do slip past no matter how many procedures we have in place, with continued support from the players by testing the features we will only get better at squashing these bugs. :)
2 things: 1. shouldn't CCP make available the features of the next patch right away asap on SISI. I've noticed that they do but not long before the patch. Which leads me to WiS and other big awesome things that are about to come...some day. Release it now so we can start bug hunting and help you guys. the sooner the better in my opinion. 2. Like others said, make the bugs report public. its an awesome thing to do. Thats what Gentoo (linux distro) does as well as other program and things on the Internet and its a great tool for bug report. If a bug doesn't have all the info, others can complete it. |
Trebor Daehdoow
Gallente
Sane Industries Inc.
|
Posted - 2009.09.03 15:08:00 - [ 24]
Bug hunting is a thankless task, but sometimes the result of the current system is that potential bug-reporters become disillusioned.
A big part of the problem is that the bug report database is not publically searchable (ie: like bugzilla). When we run into an anomaly in the game, there is no way to tell if we are the first to find the bug, or the 10,000th. There is no way to add our perspective to an existing bug report. This leads to tons of extra, wasted effort on the part of the players and the BH team -- we're like blind men in a cave, each privy to only a bit of the puzzle.
The brutal fact of the matter is, if you want to get really good feedback on the bugs, you have to be willing to air your dirty laundry.
For example, I found a relationship between the number of times you look at market details and session-change delays on the Mac client. It's 100% replicable, and takes only 5 minutes to demonstrate (you can get it to where, after a gate jump, you're almost always in the disconnect warp when the client loads the new system).
If you want to check it out for yourself, have 40-50 items in your quickbar, and time a jump through a gate. Now select each item in turn and load the details. Every 10 times or so (loading details of items like Trit probably helps) jump through a gate, and watch the jump times creep up. Eventually, you'll get to the point where the session changes take so long that you're already "warping" to 1m km off the gate.
The point I'm trying to make here is that I have one perspective on this. It might be a Mac-only issue. It might only affect characters with specific trade skills. Maybe it only affects a particular region. If there was a public database of bugs, other players could test and replicate the bug, and narrow down the circumstances in which it occurs.
Another thing that is very frustrating is that when you jump through all the hoops, provide the logs, etc, often the bug stays in "not filtered" limbo forever. The issue above was reported: 7/7. I updated the report with all the logs requested and extensive test data *and* a step by step method of demonstrating the bug: 7/20. Retested and confirmed after 1.5 was deployed. In total, I probably spent 4-5 hours tracking down the circumstances.
Current status (since 7/20): "not filtered".
|
Trebor Daehdoow
Gallente
Sane Industries Inc.
|
Posted - 2009.09.03 15:21:00 - [ 25]
Originally by: ISD BH Sabaoth
Lost Hamster: That’s exactly what we do with the 'Defects' we will combine bug reports detailing the same bug into 1 single defect for clarification and collation of all relevant information. If this was left to players to do then you could find that bug reports that are not related get added to each other as players can’t check the bugs out as thoroughly as ourselves and QA can.
There are many of us, and few of you. Linus's Law states: "Given enough eyeballs, all bugs are shallow". With a public bug database, the job of the BH team is still to separate the signal from the noise. But because the users have access to more information, their reports will tend to have more signal. I would venture to point out that if EVE had had a public bug database, the whole free-moon-materials debacle would probably have been aborted at an early stage. People would have noticed that bug report and replicated it. |
el caido
School of Applied Knowledge |
Posted - 2009.09.03 17:57:00 - [ 26]
boot.ini!  |
Jason Edwards
Internet Tough Guy
Spreadsheets Online |
Posted - 2009.09.03 22:39:00 - [ 27]
Quote:
If many people report the same bug we do combine this into a ‘defect’ which collates all the experiences/information and reproducible steps from each report into a combined set which enables us and QA to quickly see different variants of the same bug.
http://en.wikipedia.org/wiki/Linus%27_LawLinus' Law according to Eric S. Raymond states that "given enough eyeballs, all bugs are shallow". More formally: "Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix will be obvious to someone." The rule was formulated and named by Eric S. Raymond in his essay "The Cathedral and the Bazaar". By making it so only the bug hunters and perhaps a handful of ccp see the collated problems. It may be very difficult to fix the problem. It also means the same old shallow problems get reported. As opposed to more detail because people can read the simple stuff posts already. Quote:
You do make a very good point, the only major problem i can see if when we are dealing with 'exploit' reports. Some exploits bug reports are not even listed as exploits they may be tied into a certain mechanic which in part causes something unexpected to happen and results in an advantage through un-designed mechanics.
The idea of public bugs is that exploits become public essentially. The strive to find/fix them before release to tq is great. Not to mention... when someone believes they are a victim of an exploit. They likely will search the bug reports first. Find something applicable and then test. Those who use exploits do so under risk of banning. Quote:
ISD BH Sabaoth said “with continued support from the players by testing the features we will only get better at squashing these bugs. :)” But you don’t support us in testing.
Thats the thing I'm seeing. BH are hard to get ahold of in general. We support them while completely in the dark and thusly going to be without aim. At the same time. If we need help... it's often impossible to get... even if we do get ahold of a bh. Also... with the opening up of bug reports... you can make bh itself much more attainable and less secretive. ex. I had a pos exploit id heard about on tq. Went on sisi to test it... but it took the destruction of a previous pos. Well ill tell you that a carrier + sentries sure takes alot of time. I did get a friend to come help... but ewww. Quote:
1. shouldn't CCP make available the features of the next patch right away asap on SISI. I've noticed that they do but not long before the patch. Which leads me to WiS and other big awesome things that are about to come...some day. Release it now so we can start bug hunting and help you guys. the sooner the better in my opinion.
Cathedral and bazaarBut by a year later, as Linux became widely visible, it was clear that something different and much healthier was going on there. Linus's open development policy was the very opposite of cathedral-building. Linux's Internet archives were burgeoning, multiple distributions were being floated. And all of this was driven by an unheard-of frequency of core system releases.
Linus was treating his users as co-developers in the most effective possible way:
7. Release early. Release often. And listen to your customers. |
Jason Edwards
Internet Tough Guy
Spreadsheets Online |
Posted - 2009.09.03 22:46:00 - [ 28]
Quote:
Bug hunting is a thankless task, but sometimes the result of the current system is that potential bug-reporters become disillusioned.
Not only potential but also current ones. Quote:
Another thing that is very frustrating is that when you jump through all the hoops, provide the logs, etc, often the bug stays in "not filtered" limbo forever.
Had that happen. Had one reaching about 1 month old. Then school started for me and months later it got filtered with... "dxdiag please. are your sound drivers up to date?" Quote:
I would venture to point out that if EVE had had a public bug database, the whole free-moon-materials debacle would probably have been aborted at an early stage. People would have noticed that bug report and replicated it.
Not to mention... how many others? |
Vir Hellnamin
Gradient
Electus Matari |
Posted - 2009.09.04 05:22:00 - [ 29]
Seems like someone already commented the problem of closed bug-DB and how much it does "good" against exploits... thanks, don't have to write that long reply. :) |
ISD BH Sabaoth
Amarr
ISD BH
|
Posted - 2009.09.04 09:02:00 - [ 30]
Edited by: ISD BH Sabaoth on 04/09/2009 09:35:58
I know a lot of you are quite into the idea of a public bug’s database and are using examples of open source operating systems that have worked well under this scenario. The problem here is that EVE isn’t just a piece of software, the game world is live and as such exploits can resonate throughout the game world for a long time causing immense damage, by making exploits public it will only increase the chance of damage to EVE.
When an exploit is reported (and confirmed i.e. reproducible) CCP is on it immediately. This doesn’t change with publicly posted exploits, someone still has to report it and we always give exploit reports top priority. I can see people get annoyed that their particular report doesn’t get answered within the first few days or sometimes weeks but as you can see from the graphs in the devblog we are dealing with insane numbers of reports and we must also prioritise them.
Everyone that has commented seems quite passionate about this and helping CCP and you have good idea but most of these aren’t as easy as you think, you want a public bug database so you can help more and have the info to do so. Well what I say to you is, read the bottom of the devblog and apply for ECAID where you will get the correct training and information/tools to do the job properly.
As far as finding it hard to contact a Bug Hunter, that may be the case but only because when they are online they are working on the filter list! However if you come on the Coldfront IRC network and join #eve-chaos you can catch us or the odd QA/Dev alternatively Bug Hunters are always on Sisi but once again they are busy testing so if they don’t have time to talk... it’s generally because they are in the middle of a test.
This devblog was to inform people but also to make people aware that we are recruiting and on the lookout for good applicants, don’t get disappointed if you don’t make it though out of 250 applicants we normally accept 10-20 Max as the job really isn’t as easy as you make originally think.
I hope to see some of your applications soon
|
All Channels
EVE Information Portal