open All Channels
seplocked EVE Information Portal
blankseplocked Tranquility Issues Resolved
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: [1] 2 3 4 5 6 7

Author Topic

CCP Wrangler

Posted - 2007.09.15 19:49:00 - [1]
 

This Monday it was brought to our attention that an alliance couldn't get Sovereignty 4 in their capital system even though they should. Looking into it we concluded that their capital station was a conquerable station which shouldn't be possible to set as a capital station. Fixing this required changing procedures in the database so attempting to set a conquerable station as capital station causes an error.

Unfortunately, due to unforeseen behavior in the server code a certain argument was passed to the procedure, which did not have anticipated default value, causing the procedure to error on every call from conquerable stations that were set as capital stations, of which there are 10 of on Tranquility.

While hotfixing this we failed to notice that the error flag code we copied was bad, that error condition can possibly have caused other crashes, in the sense that it did not close the transaction. This in turn led to that every call to this procedure would lock up a session to the database as it waited indefinitely for the transaction to be either rolled back or closed. The sessions locking up created a plethora of different locks all over the database, causing us to chase for a ghost of the problem. Eventually this led to a database fail-over which meant Tranquility crashed.

This oversight has been located in two other procedures as well, but none of them are called often enough on Tranquility to cause any real problems.

To sum it up, a series of unfortunate exceptional cases set up an environment on TQ wherein 10 stations on the entire server could repeatedly lock sessions in the DB until it had to do a failover to free them up. As a result of this, our hotfixing protocols are being revised to prevent this from happening again in the future.

CCP Atropos

Posted - 2007.09.15 19:51:00 - [2]
 

\o/

Larg Kellein
Caldari
Agony Unleashed
Agony Empire
Posted - 2007.09.15 19:52:00 - [3]
 

Woah... POS wars really are killing the game, who would have thought?

Nice catch, CCP dudes and dudettes.

CCP Prism X


Gallente
C C P
C C P Alliance
Posted - 2007.09.15 19:53:00 - [4]
 

Working on Saturdays is so much more giving when it results in something. Wink

Blackback Starkiel
EveArt Studios
Posted - 2007.09.15 19:56:00 - [5]
 

Edited by: Blackback Starkiel on 15/09/2007 19:56:41
Sounds like a very difficult bug to catch. Good job! Let's hope that does it then!
.
Griatch

CCP Mindstar

Posted - 2007.09.15 19:57:00 - [6]
 

Very HappyVery Happy

Shiner BockBeer
Go Go Gadget ForumPostingAlt
Posted - 2007.09.15 19:58:00 - [7]
 

Excellent work. Thanks for the update. :)

Deathdealerdoc
Caldari
Blood and Rust
Slacker Trade Federation Unlimited
Posted - 2007.09.15 20:00:00 - [8]
 

good job!!!!

Onderzoeker 2
Posted - 2007.09.15 20:01:00 - [9]
 

good job CCP Cool

TigerWoman
Amarr
The Circle
Blade.
Posted - 2007.09.15 20:03:00 - [10]
 

superb work!

i owe you a beer on the fanfest!

Pirate Tom
Posted - 2007.09.15 20:05:00 - [11]
 

Originally by: "CCP Wranglermuffin"
...wherein 10 stations on the entire server could repeatedly lock sessions in the DB until it had to do a failover to free them up...


So in a roundabout sort of way, it really was BoB and Goonfleet (et al) causing the crash!

Lance Fighter
Amarr
Posted - 2007.09.15 20:08:00 - [12]
 

*tinfoil-hattery* Bob is killing the servers!!!! noooO! nerf bob!
*/tinfoil-hattery*

Great job CCP, kudos

torswin
Society Of The Abattoir
Posted - 2007.09.15 20:11:00 - [13]
 

http://www.yonkis.com/mediaflash/1000000.swf

thankyou! (same as wrangler (i think) posted but without the fleshlight ad).

Price Watcher
Posted - 2007.09.15 20:11:00 - [14]
 

Well done.

CCP Wrangler

Posted - 2007.09.15 20:11:00 - [15]
 

Originally by: Pirate Tom
Originally by: "CCP Wranglermuffin"
...wherein 10 stations on the entire server could repeatedly lock sessions in the DB until it had to do a failover to free them up...
So in a roundabout sort of way, it really was BoB and Goonfleet (et al) causing the crash!
No, they were not involved. I'd also like to clarify that this problem was not caused by any Alliance, we messed up and not any player(s). Wink
Originally by: torswin
http://www.yonkis.com/mediaflash/1000000.swf

thankyou! (same as wrangler (i think) posted but without the fleshlight ad).
You people will never let me forget that, will you? RazzLaughing

CCP John Proctor

Posted - 2007.09.15 20:12:00 - [16]
 

I can go sleep now Sad

Dont go 72 hours with no sleep... you get sent to the hospital and have to answer emails with your black berry from the emergency room Sad


Pirate Tom
Posted - 2007.09.15 20:16:00 - [17]
 

Wrangler I was going for the obvious poke at the 'OMG BOB' and 'OMG GOONS' crowd. I know it was a code error. You explained that pretty clearly.

Claudia Jefferson
Posted - 2007.09.15 20:18:00 - [18]
 

Thats the best news all week ;)

CCP Wrangler

Posted - 2007.09.15 20:19:00 - [19]
 

Originally by: Pirate Tom
Wrangler I was going for the obvious poke at the 'OMG BOB' and 'OMG GOONS' crowd. I know it was a code error. You explained that pretty clearly.
I figured, I just wanted to make sure people understood that it was no Alliances fault that this happened. Wink

torswin
Society Of The Abattoir
Posted - 2007.09.15 20:20:00 - [20]
 

Originally by: CCP Wrangler

Originally by: torswin
http://www.yonkis.com/mediaflash/1000000.swf

thankyou! (same as wrangler (i think) posted but without the fleshlight ad).
You people will never let me forget that, will you? RazzLaughing

nevar forget! Laughing

Takk fyrir, CCP! Very Happy

Elmicker
Wreckless Abandon
Posted - 2007.09.15 20:20:00 - [21]
 


Peter VonThal
Raygun Technologies
Posted - 2007.09.15 20:21:00 - [22]
 

Nice! I can imagine it wasn't fun trying to hunt something like that down.

Drykor
Minmatar
Aperture Harmonics
K162
Posted - 2007.09.15 20:28:00 - [23]
 

good job

Sylia
Flirtacious
Posted - 2007.09.15 20:31:00 - [24]
 

Guess sharkbait is sleeping after a few sleepless nigths now eh :)

Grats CCP on getting it fixed though, we players appreciate it

If I can save some iskies I'll buy a sexy dev or 2 a beer at fanfest

Neroo Tal
Minmatar
Order of the Golden Dawn
Shades of Gray
Posted - 2007.09.15 20:32:00 - [25]
 

'grats on solving the problem CCP! Well done! Now, all of you go take a nice long deserved nap!

Ambart Valaina -= Becomes sometimes Fate is truly Divine!

Elseer Radak
Gallente
Project F3
TRUST Coalition
Posted - 2007.09.15 20:33:00 - [26]
 

well caught, sentients, well caught! such a subtle error, such awful damage!

\o/ Extra Vitoc for the hamsters!!! heck, extra Vitoc for Dev, QA and OPS \o/

now go home and get some sleep you all clearly deserve it!

Elseer R.



Reem Fairchild
Minmatar
Punic Corp.
Posted - 2007.09.15 20:34:00 - [27]
 

Good job, CCP!!!

CCP Gangleri


Minmatar
Posted - 2007.09.15 20:34:00 - [28]
 

Originally by: Sylia
Guess sharkbait is sleeping after a few sleepless nigths now eh :)

Grats CCP on getting it fixed though, we players appreciate it

If I can save some iskies I'll buy a sexy dev or 2 a beer at fanfest


All devs are sexy, you might want to start saving up.

Ex0101
Gallente
The Scope
Posted - 2007.09.15 20:34:00 - [29]
 

\o/ Nice work guys, thanks for the explanation

ViolenTUK
Gallente
Demolition Men
Posted - 2007.09.15 20:36:00 - [30]
 

Edited by: ViolenTUK on 15/09/2007 20:47:59
In this crash the entire "3asy company" Alliance was lost of the database. All of the alliance members were kicked from the alliance and 3asy company doesnt show in the rankings list. Please CCP give us back our alliance. Shocked


Pages: [1] 2 3 4 5 6 7

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only