Posted - 2007.09.15 19:49:00 - [1]

 

This Monday it was brought to our attention that an alliance couldn't get Sovereignty 4 in their capital system even though they should. Looking into it we concluded that their capital station was a conquerable station which shouldn't be possible to set as a capital station. Fixing this required changing procedures in the database so attempting to set a conquerable station as capital station causes an error.

Unfortunately, due to unforeseen behavior in the server code a certain argument was passed to the procedure, which did not have anticipated default value, causing the procedure to error on every call from conquerable stations that were set as capital stations, of which there are 10 of on Tranquility.

While hotfixing this we failed to notice that the error flag code we copied was bad, that error condition can possibly have caused other crashes, in the sense that it did not close the transaction. This in turn led to that every call to this procedure would lock up a session to the database as it waited indefinitely for the transaction to be either rolled back or closed. The sessions locking up created a plethora of different locks all over the database, causing us to chase for a ghost of the problem. Eventually this led to a database fail-over which meant Tranquility crashed.

This oversight has been located in two other procedures as well, but none of them are called often enough on Tranquility to cause any real problems.

To sum it up, a series of unfortunate exceptional cases set up an environment on TQ wherein 10 stations on the entire server could repeatedly lock sessions in the DB until it had to do a failover to free them up. As a result of this, our hotfixing protocols are being revised to prevent this from happening again in the future.

Posted - 2007.09.15 19:51:00 - [2]

 

\o/

Posted - 2007.09.15 19:53:00 - [4]

 

Working on Saturdays is so much more giving when it results in something.

Posted - 2007.09.15 19:57:00 - [6]

 

Posted - 2007.09.15 20:11:00 - [15]

 

Originally by: Pirate Tom

---

Originally by: "CCP Wranglermuffin"

---

...wherein 10 stations on the entire server could repeatedly lock sessions in the DB until it had to do a failover to free them up...

---

So in a roundabout sort of way, it really was BoB and Goonfleet (et al) causing the crash!

---

No, they were not involved. I'd also like to clarify that this problem was not caused by any Alliance, we messed up and not any player(s).

Originally by: torswin

---

http://www.yonkis.com/mediaflash/1000000.swf

thankyou! (same as wrangler (i think) posted but without the fleshlight ad).

---

You people will never let me forget that, will you?

Posted - 2007.09.15 20:12:00 - [16]

 

I can go sleep now

Dont go 72 hours with no sleep... you get sent to the hospital and have to answer emails with your black berry from the emergency room

Posted - 2007.09.15 20:19:00 - [19]

 

Originally by: Pirate Tom

---

Wrangler I was going for the obvious poke at the 'OMG BOB' and 'OMG GOONS' crowd. I know it was a code error. You explained that pretty clearly.

---

I figured, I just wanted to make sure people understood that it was no Alliances fault that this happened.

Posted - 2007.09.15 20:34:00 - [28]

 

Originally by: Sylia

---

Guess sharkbait is sleeping after a few sleepless nigths now eh :)

Grats CCP on getting it fixed though, we players appreciate it

If I can save some iskies I'll buy a sexy dev or 2 a beer at fanfest

---

All devs are sexy, you might want to start saving up.