open All Channels
seplocked EVE Information Portal
blankseplocked New Dev Blog: The Day the Items Disappeared
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: 1 [2] 3

Author Topic

Tamasan
Gallente
BLACK LABEL INC
Malum Exuro
Posted - 2010.05.14 00:07:00 - [31]
 

It has been my experience that those *profanity* stupid one line coding errors are the hardest to find and fix.

Kudos to CCP for finding the problem quickly, and also making a tough choice that was better for the players (avoiding rollback), even if more time consuming for all the GMs having to deal with the petitions.

Abdiel Kavash
Caldari
Paladin Order
Fidelas Constans
Posted - 2010.05.14 00:54:00 - [32]
 

Originally by: Lagruna Zegata
Always unfortunate to see these things happen to such a terrific game, especially to divert CCP's attention right before a much-awaited expansion. Kudos for explaining everything as thoroughly as possible.

Some semi-technical side questions: How come this new area of the junkyard needed to be cleaned? And why was a modified script necessary for this area?

~LZ


I am guessing that this "junkyard" area has separate divisions for items, ships, characters, and whatever else ends up there. One reason I see for this setup is being able to find stuff easier in case you need to restore something. And as you might imagine, the pool of deleted items fills up much faster than, say, deleted characters. But over time, even the second one starts taking up too much space and IDs and needs to be purged.

Keep in mind that this is only my best guess, the actual reason might or might not be similar to this.

Louis deGuerre
Gallente
Malevolence.
Posted - 2010.05.14 01:02:00 - [33]
 

I remember two years ago my company had a 'black week' and were forced to do a rollback due to dramatic multiple hardware failure.
We were at that time not ready for a catastrophe of such magnitude and it took us almost three days.
For an on-line service, that is high drama indeed. So I fully understand and support CCP's decision to not do a rollback.

And I'm just gonna say it, this will go down in eve history as 'Black Friday' Razz

Love these blogs ! Keep em coming ! Cool

Tres Farmer
Gallente Federation Intelligence Service
Posted - 2010.05.14 01:46:00 - [34]
 

Hm.. ID's which piled up since now and never got reasigned.. what could this be - T2 BPO Id's? Twisted Evil

Anyways.. if it affected market orders the most I would guess this part of the junkyard gets wiped the most (short behind the NPC junkyard Id's I guess) and reused those new Id's first.

Very nice Blog.. want to read more from the Red Button!
Is there by any chance a 'CCP Run for yer lives' or a 'CCP Were Screwed' on board? Laughing

Ms Michigan
Gallente
Aviation Professionals for EVE
Fusion Alliance
Posted - 2010.05.14 02:12:00 - [35]
 

So.... Not to be the "Glumy Gus" here but this whole answer seems good but not good enough...

Fact: Items went missing. Fact: You are leaving it up to players to NOTICE what went missing (if one or more of the 114K+ items was one of theirs). Fact: No ONE in EVE, sub the noob with only a mining laser and ship to his name, remembers all of their items in and around all of New Eden.

So.... my question is: How do I know if I am affected? and What is CCP doing about it besides telling us (which I am not knocking) but given the afore mentioned - seems a neglect? I mean - I know I would notice if my Titan went missing, but what about players that have so many items and log in infrequently. How do I know if that faction hardner I thought I had in Jita is gone or not?

Nadarius Chrome
Celestial Horizon Corp.
Posted - 2010.05.14 03:44:00 - [36]
 

As far as I understand, only items that underwent a transaction while there were no IDs available were affected. Items just sitting in your hangar unmolested would not have disappeared.

EG someone sold you something by buy order, or you delivered a manufacturing run, or stacked some items.

Diomedes Calypso
Aetolian Armada
Posted - 2010.05.14 03:57:00 - [37]
 

Not trying to troll here...I'm just a curious guy that likes to think of things from different angles and fill holes in my general models of different things I carry around in my mind.

Could someone give me a few sentences on why rol-backs would be a nightmare, especially in a game ? (I can see problems rolling back medical records or stock trades)

Sure, like real world stock trades, people would give up any gains (or recoup losses) that happened during the intervening time but that seems fairly workable in a game. Its like taking a goal of the scoreboard if had rolled onto the field during the play before it.


There would be winners and losers from the event, but there are also winners and losers if you don't roll back and given the choice to corrupted information and erased action the erased action seems most consitent with rules of sport ?

But there must be some techical issue that I don't get ? Is it that a roll back can't truly restore the past state ?

cpu939
Gallente
Volatile Nature
White Noise.
Posted - 2010.05.14 04:18:00 - [38]
 

how many junkyards are there? how many items are in each junkyards?
also what is the daily recycled average i.e. number of items cleared of the list?
how many of the dev team did it take to fix this problem?
i bet the mayday call went out for all the devs to come quick (sry bad pun)

Callipygian Provocateur
Posted - 2010.05.14 06:21:00 - [39]
 

Diomedes Calypso, as Red Button mentioned, a rollback would have taken a long time, probably hours considering it sounded like it would involve restoring a backup copy of the entire EVE database, which, I suspect, is on the order of TBs or even 10s of TB. Or, the possible alternative of 'walking backwards' through the transaction logs isn't something CCP has software in place to do, and could potentially cause further complications.

Database administration isn't clean, easy or sexy. Writing scripts to interact with a busy, concurrent DB isn't simple stuff. It sounds like the problem in the script was one of timing, where item IDs were flagged as available before they truly were from the DB's perspective, but only by the tiniest fraction of a second, which is why the error didn't occur on Singularity.

Anyway, thanks a ton for the frank, honest communication. I think CCP's transparency and openness with the players is a huge part of why this community is so supportive.

CCP Red Button

Posted - 2010.05.14 09:39:00 - [40]
 

Edited by: CCP Red Button on 14/05/2010 09:40:52
Edited by: CCP Red Button on 14/05/2010 09:40:23
Originally by: Diomedes Calypso
Could someone give me a few sentences on why rol-backs would be a nightmare, especially in a game ?


Fair question Diomedes. The reason why we consider rollbacks an absolute last resort thing is that by doing so, even by a short period, you are loosing some of EVE History. Imagine a big battle being fought and then suddenly it never happened, a successful mission completed and then it never was, that ingenious market trade.. etc. So we will essentially explore every other option and turn every stone before deciding to rollback, even if that would in some cases be the easiest option from a technical standpoint.

123GuessWho
Posted - 2010.05.14 09:48:00 - [41]
 

Originally by: CCP Red Button
Edited by: CCP Red Button on 14/05/2010 09:40:52
Edited by: CCP Red Button on 14/05/2010 09:40:23
Originally by: Diomedes Calypso
Could someone give me a few sentences on why rol-backs would be a nightmare, especially in a game ?


Fair question Diomedes. The reason why we consider rollbacks an absolute last resort thing is that by doing so, even by a short period, you are loosing some of EVE History. Imagine a big battle being fought and then suddenly it never happened, a successful mission completed and then it never was, that ingenious market trade.. etc. So we will essentially explore every other option and turn every stone before deciding to rollback, even if that would in some cases be the easiest option from a technical standpoint.


I remember this happening in 2008 during a MAX campaign, all titans, capitals and pos' where back up after a server crash. So that must have been a roll back

wizard87
Posted - 2010.05.14 09:51:00 - [42]
 

Disaster Recovery, Roll-backs, Backups, Failovers.

Those words bring back horrible memories of DR tests in a former life. As for Eve its about time you realised the disaster occurred about 3 or 4 years ago around the time you removed Escrow....since then its been one long FailOveur - time for a roll-back.

CCP Red Button

Posted - 2010.05.14 09:52:00 - [43]
 

Originally by: cpu939
how many junkyards are there? how many items are in each junkyards?
also what is the daily recycled average i.e. number of items cleared of the list?
how many of the dev team did it take to fix this problem?
i bet the mayday call went out for all the devs to come quick (sry bad pun)


So many questions and so few answers Smile

To tell you the truth I don't really know how many there are. There are several, I know that much, and the number of items in each varies depending on what it contains. The recycled average also varies depending on the activity levels in game and the frequency of cleanups.

There were quite a few people working on fixing this problem last Friday, not just DBAs and programmers but individuals from almost every part of the company, customer support, community, operations, engineering etc. Essentially when EVE has a problem, the whole of CCP drops whatever they are doing and comes running.

Crudamus
Fearless Bandits
Fearless Bears
Posted - 2010.05.14 09:54:00 - [44]
 

I have 10's of thousands of items. So are you saying that it is possible for me to be missing something and would not even know it because I didn't bother to look over my assets?. Like a ship, ammo, BM, mod or even my chars on an account I don't have active?

Cebraio
Posted - 2010.05.14 10:38:00 - [45]
 

Originally by: Crudamus
I have 10's of thousands of items. So are you saying that it is possible for me to be missing something and would not even know it because I didn't bother to look over my assets?. Like a ship, ammo, BM, mod or even my chars on an account I don't have active?


I was going to ask exactly that. Which item categories were affected? How are we supposed to find out if we lost something valuable?

CCP Red Button

Posted - 2010.05.14 10:49:00 - [46]
 

Crudamus, no existing items were affected by this error. This only affected a small portion of new items being created during the 60 minute time period the script was running. That includes market orders being completed, items stacked or unstacked, new characters created etc. Most of these we were able to fix during Fridays downtime and we are still working on finding and fixing the rest. So if you were active and logged on in that time period and think this might have affected you I would encourage you to send in a petition and the GM's will investigate.

clixoras
Posted - 2010.05.14 11:00:00 - [47]
 

Originally by: CCP Red Button
and we are still working on finding and fixing the rest. So if you were active and logged on in that time period and think this might have affected you I would encourage you to send in a petition and the GM's will investigate.


Are you saying that you are investigating ALL items that are missing? or ONLY the ones that got petitioned? Like some others i don't have a clue what could be missing (with the variables you gave ofcourse). Could be a few mil but also 1b.

Hawk TT
Caldari
Bulgarian Experienced Crackers
Posted - 2010.05.14 11:13:00 - [48]
 


Check out "Datacore Continuous Data Protection" ;-)


http://www.datacore.com/products/prod_Traveller.asp

Instant "time-machine" roll-backs on storage (SAN) level + other extras. Such solution would require virtualizing your RAMSANs, but it has also multiple other benefits. With the current "physical/fixed" SAN architecture you are pretty much "handcuffed"...

Shasz
Angels of Anarchy
Posted - 2010.05.14 11:41:00 - [49]
 

I love reading the details like this, thank you very much CCP for keeping us in the loop.

If I understand correctly, the reason CCP can't just search for and correct every issue is because much of it requires research into the claim and a judgement call.

Consider this example: You, the unsuspecting pilot, buy a new Dominix in Jita. That new ship needs an item id and grabs the next available one. Somewhere else a pilot took delivery of his freshly produced Navy Raven from the LP store, and it too needs an item id. Due to the bug in script, both pilots got the same item id for their ship. I'm guessing the last one to log out and/or save that ship id is going to be the one that "wins" and keeps his ship the next time he logs in. The database can't save both ships under the same id, it can only overwrite one with the other. Now that case is easily checked and verified - there will be a record of the purchase of both ships and the missing ship can be safely re-created. This could even be automated.

But what about creating a new item id by unstacking items? Say you had a stack of 3 unassembled Dominixes instead, and you peeled one off the stack to fit it up, and your item id collided with another new ship. There probably wouldn't be a record of you simply unstacking a ship, and someone would have to search a backup of the database to verify that you did indeed own 3 Domis in the recent past, and that you hadn't lost one in combat during the bug timeframe. Writing a script to verify all that just isn't reasonable, and each case has to be petitioned as well because there's no way to automatically determine something is missing.

HTH

Sable Moran
Gallente
Moran Light Industries
Posted - 2010.05.14 11:43:00 - [50]
 

To all the whiners..

CCP t0rfifrans has explained quite clearly how to determine if you're affected by this issue or not in this news item.

Read it, apply brainpower and.... profit!?!??

Vint Rotach
Posted - 2010.05.14 14:27:00 - [51]
 

+1

Open and honest.

Wish the 'collection of individuals' I work for could work with the same idea.

Sleeper King
Posted - 2010.05.14 18:58:00 - [52]
 

Edited by: Sleeper King on 14/05/2010 19:00:50
Originally by: Hawk TT

Check out "Datacore Continuous Data Protection" ;-)


http://www.datacore.com/products/prod_Traveller.asp

Instant "time-machine" roll-backs on storage (SAN) level + other extras. Such solution would require virtualizing your RAMSANs, but it has also multiple other benefits. With the current "physical/fixed" SAN architecture you are pretty much "handcuffed"...


Wait... Don't you have to be in the IT Alliance to make suggestions like this? (joke) Rolling Eyes
How do I drop in an application to the CCP Alliance in Jove Space pls? Cool

Also, thanks for making the best decision you could in the heat of the moment that ultimately benefited the largest number of players.

Edit - Id also like to seem some statistics on the number of fail scam petitions for items that were not lost.

I lost a Eurbus BPO btw...

Meno Theaetetus
Wildly Inappropriate
Wildly Inappropriate.
Posted - 2010.05.14 19:43:00 - [53]
 

Originally by: CCP Red Button
Edited by: CCP Red Button on 14/05/2010 09:40:52
Edited by: CCP Red Button on 14/05/2010 09:40:23
Originally by: Diomedes Calypso
Could someone give me a few sentences on why rol-backs would be a nightmare, especially in a game ?


Fair question Diomedes. The reason why we consider rollbacks an absolute last resort thing is that by doing so, even by a short period, you are loosing some of EVE History. Imagine a big battle being fought and then suddenly it never happened, a successful mission completed and then it never was, that ingenious market trade.. etc. So we will essentially explore every other option and turn every stone before deciding to rollback, even if that would in some cases be the easiest option from a technical standpoint.


Just a quick heads up to ccp and empire people who don't seem venture out of >=0.5, small rollbacks occur nearly weekly when nodes crash, one minute no ship, the next .. oh I'm back here in my ship again + my insurance.


Antihrist Pripravnik
Scorpion Road Industry
Posted - 2010.05.14 20:47:00 - [54]
 

+1 for the great devblog

Alain Kinsella
Minmatar
Posted - 2010.05.14 21:33:00 - [55]
 

Thanks for the update. As a Systems Admin (and backup admin, and a DR/BK 'point man') I sympathize completely.

Have you had a chance to look at faster replacements to RamSan? I hear FusionIO is doing some rather interesting things now, for 1/10th the price.

[Yes, possible heresy, but we have RamSan units at work and so far they've only hit their best potential with high-end SPARC servers driving them.]

--A_K

arjun
Viziam
Posted - 2010.05.15 06:51:00 - [56]
 

i petitioned for the probable loss of 1,9 b through the market. no answer yet. is there still a que the gms work on or is this thing "ad acta" ?

Nadarius Chrome
Celestial Horizon Corp.
Posted - 2010.05.16 22:17:00 - [57]
 

Originally by: CCP Red Button
So if you were active and logged on in that time period and think this might have affected you I would encourage you to send in a petition and the GM's will investigate.


So what you're saying is, anyone who cannot definitively rule out any loss whatsoever from this should lodge a petition?

Can a GM skim through my transactions and stuff and see easily what, if anything, disappeared?

If so, incoming petition.

Jennifer Christage
Token Industries
Posted - 2010.05.18 13:00:00 - [58]
 

So, my petition sent minutes after the problems started still hasn't been fully resolved and no reply from a GM for almost a week. What exactly am I expected to do for you to resolve this problem?

Rip Minner
Gallente
ARMITAGE Logistics Salvage and Industries
Posted - 2010.05.20 07:58:00 - [59]
 

Edited by: Rip Minner on 20/05/2010 08:01:11
haha I find it very funny that the guy that gets called in for the worst is Called Red Button. Perfect name for the job :)

Blitz0r
Minmatar
M. Corp
M. PIRE
Posted - 2010.05.20 13:29:00 - [60]
 

Thank you for a wonderful report!

Good luck chasing the errors and keep up the good work


Pages: 1 [2] 3

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only