open All Channels
seplocked EVE General Discussion
blankseplocked Why the need for downtime?
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: [1] 2

Author Topic

Artamon
Amarr
Celtic industries
Posted - 2004.04.28 12:37:00 - [1]
 

As the subject says, why does Eve need to be taken offline for at least an 1hr EVERY single night of the week (night in my timezone)? What exactly is it that CCP do during this hour (outside of the hardware upgrade occassions)?

A few MMOGs out there have a reset/downtime once a week, which is acceptable. But once a day?

Art...

I guess I question this because the DT just happens to occur at the time that I am able to play. So I'm selfish - sue me! Very Happy

Kin Oreyn
Anonymous Alcoholics
Wrath.
Posted - 2004.04.28 12:41:00 - [2]
 

heh... I know what you mean, I'm in Japan and dt happens at 8pmMad. It sure would be nice if they only had to do it once a week

GenocidalSpliff
Imperium Technologies
Posted - 2004.04.28 12:54:00 - [3]
 

Eve runs on a massive database, and to prevents things going wrong, its best to maintain it regularly and keep everything in check. 1hour out of a day is hardly a lot to lose, if it is to you, i would suggest you get a life :|

Im pritty sure this is the only game of its sort, catering for the amount of players it does, so its understandable that it must maintained regularly.

Lady Michelle
Cold North Industries
Posted - 2004.04.28 12:57:00 - [4]
 

Not speaking as a techie now, I had enough of that in previous thread.

But it's because they need some backups, and use the heavy clusters for UT servers for the programmers.

Killash Larz
Gallente
The Scope
Posted - 2004.04.28 13:02:00 - [5]
 



Quote:
But it's because they need some backups, and use the heavy clusters for UT servers for the programmers.


and to boot all the sad no lifers off the server who would otherwise be on EvE 24/7 (like me) Twisted Evil

Meister
Amarr
Imperial Shipment
Posted - 2004.04.28 13:04:00 - [6]
 

With daily downtime CCP get payed for no service every day for at least 1 hour (sometimes more). It is quite profitable too to have a daily downtime as no players taking up bandwidth = saving money = more in the pockets and over the years it adds up for each player. I doubt it is neccessary to have a daily downtime. If it is, then CCP have a very bad server architecture. Modern games should not require daily downtime and most games dont. For me it is also quite annoying to see the server down every day because I am not used to it.

RaTTuS
BIG
Gentlemen's Agreement
Posted - 2004.04.28 13:30:00 - [7]
 

I recoken they should keep it up for 24 hours then down for 1, that way the DT would move arround the clock ... and then it don't appear in the middle of my prime time playing...

Lachenlaud
Minmatar
Wolfhaven Productions
Posted - 2004.04.28 13:34:00 - [8]
 

Originally by: Meister
Modern games should not require daily downtime and most games dont.


And you know this how? How many 'Modern' games have you written? What programming languages can you code in? Ever write a game designed to handle 10,000+ people on a regular basis with as much detail as Eve has? Confused

I've done programming and know multiple hardware platforms.... frankly, the concept of what it would take to make a game like Eve is staggering to me. Shocked

I would love to see things from the back-end some time in order to get an idea of how in the blue-heck they did it all (not that that is gonna happen unless they start giving tours - might be worth the trip to Iceland to see it - or they could come out with a small video tour of their facility - that would be cool ::HINT::). Laughing

But, unless you've walked a few hundred kilometers (god I been playin eve too long) - please don't disparage the marvelous job that the CCP dev team has done in creating this game for our enjoyment in such a generic fashion. Wink

/\/\ajor Kudo's CCP Dev Guys! Cool


Rodge
Posted - 2004.04.28 13:37:00 - [9]
 

How often is it actually a whole hour anyway?

Saturday and Sunday is the only times it hits me, but it's very rare that the game is unavailable for more than a half hour.

A half hour or even an hour is no biggie, sometimes it's the only time during the day I actually leave the house at weekends Very Happy

Lachenlaud
Minmatar
Wolfhaven Productions
Posted - 2004.04.28 13:40:00 - [10]
 

Originally by: RaTTuS
I recoken they should keep it up for 24 hours then down for 1, that way the DT would move arround the clock ...


Now THAT is simply brilliant! Great Idea - all for it. Give everyone a taste of downtime. Razz

Oops... one potential problem though - then they'd have to do the same thing with their staff in the office. Shocked

Downtime is scheduled to happen in the middle of the work day at CCP isn't it? Would that affect the whole staff there? How would something like that fit in with actual system operation parameters?

Heck - does the staff even need to BE there for downtime or is it virtually automated? Question

Questions questions.... would love to hear some CCP answers to those. Cool

Tamur
Minmatar
Republic Military School
Posted - 2004.04.28 13:43:00 - [11]
 

Quote:
please don't disparage the marvelous job that the CCP dev team has done in creating this game for our enjoyment in such a generic fashion.


Well said that man.

Maybe Downtime should be once every 2 days?
But if things start failing then to keep the game as lag free and smooth as possible then it is most needed.
Which would you hate more, loosing 1 hour a day to server maintanence or loosing your Tempest due to server lag / crash?

Still, it has to be said that EVE is a major development in modern gameing and is a fantastic game to be a part of.

Kusami
Posted - 2004.04.28 13:54:00 - [12]
 

Well, now that have 3x the I/O on their new RAID array, in theory, since they aren't running anything else at the time, the back-ups should take 1/3 of the old times?

Question


VinkNut
Gallente
Evolution
Band of Brothers
Posted - 2004.04.28 14:05:00 - [13]
 

Originally by: Meister
With daily downtime CCP get payed for no service every day for at least 1 hour (sometimes more). It is quite profitable too to have a daily downtime as no players taking up bandwidth = saving money = more in the pockets and over the years it adds up for each player. I doubt it is neccessary to have a daily downtime. If it is, then CCP have a very bad server architecture. Modern games should not require daily downtime and most games dont. For me it is also quite annoying to see the server down every day because I am not used to it.


Typical Amarr comments, frankly. From a position of ignorance.

Just think about how many individual items there are out there to keep track of, how many you trash, the logs that are kept for petitions, etc etc. Most company systems/databases of THIS size and complexity run batch jobs on mainframes overnight, CCP are doing this usually within 20 or 30 MINUTES, at a time when the majority of people playing it (it is after all based in Europe) should either be a) at school b) at college or c) doing something OTHER than playing Eve.

When you've written a system that uses a Database to keep track of probably about 100 million entities then you can comment - until then I suggest you keep your ignorance to yourself.

:)

Lufio
Posted - 2004.04.28 14:05:00 - [14]
 

well, the downtime needed for the backup? doubt that, you could use standby databases to do such things like backups without creating additional load on the primary databases.

But I guess the EVE-Databases do have a heavy load of inserting, updating and deleting a real big heap of small data throughout the 23 hours online, slowly fragmenting the storage devices. That's why such OLTP(Online Transaction Processing)-Databases mostly need to 'reorganize' their data storages, most of them can do that while still providing the service, but such a process will ALWAYS slow down the database and take a lot longer when done with the database up and running.

And well, maybe the indizes for the markets need a daily rebuild, too, so that you don't have to wait for a few minutes when doing a region search.

MooKids
Caldari
The Graduates
Morsus Mihi
Posted - 2004.04.28 14:14:00 - [15]
 

Originally by: RaTTuS
I recoken they should keep it up for 24 hours then down for 1, that way the DT would move arround the clock ... and then it don't appear in the middle of my prime time playing...


That would actually be a bad idea from a medical point of view for the developers. If the downtime kept changing by an hour each day, the devs would need to constantly change their sleep schedule. This could lead to increased fatigue as they employee's bodies would be unable to handle the stress. As it is, it takes 5-7 days for the human body to adjust to a new sleep schedule, so having it change daily would hurt them physically.

Mon Palae
The Bastards
The Bastards.
Posted - 2004.04.28 14:34:00 - [16]
 

Edited by: Mon Palae on 28/04/2004 14:36:25
I seriously doubt backups are why the system goes down. It would almost certainly take more than an hour to backup the EVE databases. I would guess they have something like a Storage Area Network (SAN) running that effectively provides real-time backups to the system.

Databases are finicky things and CCP has a monster database on their hands. I am not a database administrator/operator but I am in IT and have hung with the guys who do run the datacenter plenty to pick up a few things (if only from a distant overview perspective).

- You need to recover disk space from the database on a regular basis. Even if a given data row is 'deleted' it cannot be removed from the system while it is still possibly visible to some other aspect of the system using it. Eventually you want to get rid of it though and the only sure way to do it is to shut down the database so you know nothing else is viewing that data and it can be tossed. This issues is particularly noticeable in heavily updated tables and I would imagine the EVE database is a heavily updated creature. If this is not done you can get infinite growth on your storage requirements so it is very necessary.

- You need to generate planner statistics on the database to generate good, efficient queries. A database-wide analyze function needs to be run generally once daily. While the database does not have to be down for this it is best at low usage times and CCP never really has that (even the low side of their logged user cycle sees heavy usage).

- You need to prevent transaction ID wraparound failures which the database guys I spoke to said is a very bad thing to haev happen. Every transaction in a database gets a transaction ID number. This tells the database what order things happen in. The transaction id has a limited size of 32-bits (or ~4 billion transactions). Once the database uses its allotment of 4 billion IDs it suffers wraparound where the counter wraps back to zero. At this point transactions that were in the past appear to be in the future so their output becomes invisible. This is a hugely bad thing to have happen.

Note all of the above applies to a SQL system and may not necessarily apply to whatever CCP is running. Note also that all of the above do not necessarily require the database to be brought down to make happen but they do seriously impact performance. In a company this is usually done in the middle of the night when traffic is low but again CCP cannot count on it. Overall it is best for them to bring it all down, do their thing and start up again.

Database maintenance is critical. Catastrophic failure on a database is not a pretty sight and somethign as complex as the CCP database such a threat always looms. They simply must be proactive and aggressive in their maintenance.

Amaxma
Amarr
Posted - 2004.04.28 14:40:00 - [17]
 

Agree with comments from peeps here who have some understanding of the incredible J-O-B that CCP has done...

Probably best to leave the admin scheduling to those who know what they're @, and in the meantime just enjoy a glimpse of thats going to be ubiqiuituous as TV within the decade Cool

Monty Burns
Brutor Tribe
Posted - 2004.04.28 14:42:00 - [18]
 

Edited by: Monty Burns on 28/04/2004 14:48:22
Originally by: Lachenlaud
.....<snip>......
Oops... one potential problem though - then they'd have to do the same thing with their staff in the office. Shocked

Downtime is scheduled to happen in the middle of the work day at CCP isn't it? Would that affect the whole staff there? How would something like that fit in with actual system operation parameters?
.....<snip>.....



Yeah, fair play to em ... that would mean its in the lunch hour, where most of us are in the pub for a swift couple. Now if that is not motivation to get rid of the short and very tolerable 20-30mins (average) DT then I don't know what is!

Seriously though, we get told time n time again these are SQL servers, well, I don't know a professional SQL Server system that cannot cluster. And, we have been told these are clusters so surely some form of failover is implemented meaning that at worst and it is very bad practice, they can down one of the mirrors and back that up. MS SQL will dump out backups of databases whilst they are online, so I guess its not MS SQL. As for the staff actually being there ... we can only guess until they tell us which SQL system they use.


Any ideas anyone?

TBH, best leave em to it ... its usualy only 20 to 30 mins, time for a shower Wink


Be lucky!

Lao Tzu
Hellenistic Capital
Posted - 2004.04.28 14:53:00 - [19]
 

afik the database is MS SQL running on win2k.
Neutral

Cortex Reaver
The 14K
Posted - 2004.04.28 14:56:00 - [20]
 

The downtime hits at an inconventient time for me as well. 7am isn't a time I normally try to play, but I do like to check-in to monitor my skill-training.

It's a necessary evil. I'm sure they'll work on reducing the frequency at some point.

-CR


Primer Xenius
The Collective
Against ALL Authorities
Posted - 2004.04.28 15:04:00 - [21]
 

Quote:
Why the need for downtime?


Hamsters got to eat too sometime Very Happy

CCP kieron

Posted - 2004.04.28 17:29:00 - [22]
 

A good portion of the downtime is used to clean up the databases used in the game. There are over 9 billion items in EVE right now and when even a small portion of these items are sold, trashed, recycled, etc., other items are created and the old ones need to be cleaned up.

The server logs are also checked for errors and issues. If these things were left to take care of once per week, the disks would quickly clutter up, lag would be a result, etc.

There is much more involved in the maint periods, but this is an idea of some of what happens behind the scenes.


Doc Brown
Gallente
Capital Construction Research
Pioneer Alliance
Posted - 2004.04.28 17:56:00 - [23]
 

Edited by: Doc Brown on 28/04/2004 17:57:47
Originally by: Primer Xenius
Quote:
Why the need for downtime?


Hamsters got to eat too sometime Very Happy



CCP is still using Hamster power!?!?

Ugh, why haven't they upgraded to the more efficient Gerbil Power?


Oori
Byson Inc
Posted - 2004.04.28 19:12:00 - [24]
 

Originally by: kieron

A good portion of the downtime is used to clean up the databases used in the game. There are over 9 billion items in EVE right now and when even a small portion of these items are sold, trashed, recycled, etc., other items are created and the old ones need to be cleaned up.

The server logs are also checked for errors and issues. If these things were left to take care of once per week, the disks would quickly clutter up, lag would be a result, etc.

There is much more involved in the maint periods, but this is an idea of some of what happens behind the scenes.




As someone who worked as a DBA for a number of years with large databases (100Gb+), I can completely understand CCP's need for downtime. Considering the amount of queries that hit the database, they surely want to make sure their indexes are optimized. If their databases weren't optimized regularly query performance would degrade exponentially over time - and if you think you get lagged now you can't even imagine what would happen with a poorly indexed database. Corruption checking also needs to occur and in the event of a problem, the server needs exclusive access to the database to fix (previously mentioned) problem. Database backups surely also occur at some point and I would hope they occur throughout the day and then fully during downtime. Considering the amount of data that gets accessed/changed on a daily basis, I'm shocked that they only need an hour of downtime. One would hope that with the increase in accounts over time (esp. with E&B users) that the downtime needed remains constant. Good planning has surely accomplished this.

I have a feeling that the database platform choice was very price motivated. A more expensive DB platform could alleviate some of the issues listed above, but who would want to pay 10x more to play the game (so that CCP could afford a more expensive DB platform)? I'd rather have an hour downtime a day than pay ~$140/mo/account!

Regards,
Oori

Side note: My experience is with Microsoft SQL - so I made the above assumptions based on such a platform. It would be interesting to know what database platform CCP was actually using...

Maya Rkell
Third Grade Ergonomics
Posted - 2004.04.28 19:28:00 - [25]
 

CCP did say (about a month ago?) they'd like to start trying 2-day runs.

Gib Goblin
Caldari
Phoenix Propulsion Labs
Firmus Ixion
Posted - 2004.04.28 20:38:00 - [26]
 

I NEED THE DOWNTIME !

When else would I eat & sleep ??? CoolShocked

Jimmeh
Posted - 2004.04.28 22:16:00 - [27]
 

Right! Powergamers need to sleep sometime.

Rixeh
Caldari
Caldari Provisions
Posted - 2004.04.29 00:28:00 - [28]
 

MS SQL on Win2K -- sure couldnt hurt to clear all the memory leaks. :) Yeah they could use Unix/Oracle, but I gotta say MSSQL is actually one of the better pieces of software they've released, ever. Pretty easy to use, has some nice features too.

In any event downtime doesnt bother me 'cause I'm at work by that time anyway ;)

Lufio
Posted - 2004.04.30 10:00:00 - [29]
 

Originally by: Rixeh
MS SQL on Win2K -- sure couldnt hurt to clear all the memory leaks. :) Yeah they could use Unix/Oracle, but I gotta say MSSQL is actually one of the better pieces of software they've released, ever. Pretty easy to use, has some nice features too.

In any event downtime doesnt bother me 'cause I'm at work by that time anyway ;)


well, it might be a philosophical matter, but I for my part would've choosen the Unix/Oracle way for a project like eve. I admit that MS SQL has nice graphical admin tools and stuff, but I just like Oracle more for speed, security and all the other possibilities it has to offer when you where to look in their excessive documentations. I doubt tho that a regular downtime could be avoided when using Oracle, even on an Real Application Cluster with Replication or Standby database and and and..., for reasons already stated in this thread (index rebuilds and all this maintenance stuff, which is just necessary from time to time)Confused

and well, working, too during downtime, and on weekends: lunchtime (one gotta have to eat something from time to time, too Laughing)

Lady Michelle
Cold North Industries
Posted - 2004.04.30 10:09:00 - [30]
 

You say "nice graphical interface", I twitch.

Give me my Debain (GNU/Linux) with MySQL and a mysql client terminal, along with vim, and I'm happy for several days ;)

Anyway, I can understand the downtimes.
Databases tends to get cluttered over time, with that many entries.

I can live with an hour of downtime every day.


Pages: [1] 2

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only