open All Channels
seplocked EVE Information Portal
blankseplocked New Dev Blog: 64 bits should be enough for everybody
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: [1] 2 3 4

Author Topic

CCP Zymurgist


Gallente
C C P
Posted - 2010.10.22 14:13:00 - [1]
 

CCP Creber Cattus brings us a dev blog on why we are updating the item database to 64 bits. You can read about it here.

Sgt Blade
Caldari
Save Yourself Inc.
Posted - 2010.10.22 14:30:00 - [2]
 

128 would have been better....Laughing

Yarrow Naritus
Caldari
Divisione Ricerca e Sviluppo
OWN Alliance
Posted - 2010.10.22 14:35:00 - [3]
 

Thanks for sharing. I like that ccp is open about the technologies used behind the scenes. Even though, even as an IT guy, I don't understand what half of it means.

Qoi
Exert Force
Posted - 2010.10.22 14:36:00 - [4]
 

Edited by: Qoi on 22/10/2010 14:38:18
Did even include something new (14h downtime, oh noes!).

The graphs are AWESOME Laughing

Jack Gilligan
Caldari Provisions
Posted - 2010.10.22 14:42:00 - [5]
 

Edited by: Jack Gilligan on 22/10/2010 14:44:29
Quote:
Logs that points to a specific item ID become invalid (so our logs show nothing!)


WOW, if this change fixes the "logs show nothing" problem it will be "Teh best expansion EVAH!"

Has any thought been given to the possible impact that retaining that many more item ID's may have on game performance?

Thordina
Posted - 2010.10.22 14:44:00 - [6]
 

Will the maximun number of item-stacks inside a personal-station-hangar will also rise ?
At the moment it is capped at 999.
I am asking because it has also to do with some kind of technical limitation of the "item-system" in a greater scale :)

Silen Boon
Posted - 2010.10.22 14:45:00 - [7]
 

There are some fairly big numbers flying around there, but I guess space is a fairly big place.

Makko Gray
Pheno-Tech Industries
Crimson Wings.
Posted - 2010.10.22 14:47:00 - [8]
 

Edited by: Makko Gray on 22/10/2010 14:55:20
Originally by: CCP Creber Cattus
...after the change it can go up to 9,223,372,036,854,775,807, which is 2^63 - 1 (yes 63 we almost never use one of the bits)...


That extra bit is used for signing, why not use unsigned ints and have twice as many? Or allow negative identities? As from what I remember you don't get the unsigned option on the sql data type.

On second thoughts I imagine there's old code or issues that would raise their heads somewhere. Best play safe and leave it. Laughing

That said I wonder how long it'll take for the title to go the way of it's apocryphal originator - maybe you should have gone straight for 128 bits and implemented GUIDs

Sylpheed Hurricane
Caldari
GRC-Quebec Tax Evaders
Posted - 2010.10.22 14:55:00 - [9]
 

Why not aim for 128 or even higher? Thus preventing the same update style in i don't know 6years from now on?

Wollari
Phoenix Industries
Wicked Nation
Posted - 2010.10.22 14:56:00 - [10]
 

Edited by: Wollari on 22/10/2010 15:01:05
Edited by: Wollari on 22/10/2010 14:59:17
Quote:
64 bits should be enough for everybody

I think I heared something similar about IPv4 (32bit) before they decided that the introduction of IPv6 (128bit) was proposed. Similar with the old school pc's :-) 256kb memory is enough for everything.

So is 64bit really enough? *joking*

I just hope that unsigned BIGINT in MySQL/SQLITE will be enough, otherwise many pages or 3rd party applications could have problems when their underlaying database system don't fully support 64bit integers.

I guess the change with the conversation to 64bit is planned together with Tyrannis 1.2, which CCP PrismX mentioned earlier in his 2nd API DevBlog.

http://www.eveonline.com/devblog.asp?a=blog&bid=798

Dusty Meg
Posted - 2010.10.22 14:58:00 - [11]
 

Edited by: Dusty Meg on 22/10/2010 14:59:15
Good to hear. Atleast now itll be easier to work out ids of items.

Pep Alo
Posted - 2010.10.22 15:04:00 - [12]
 


Tsabrock
Gallente
Circle of Friends
Posted - 2010.10.22 15:16:00 - [13]
 

Edited by: Tsabrock on 22/10/2010 15:19:56
I'd imagine the biggest reason they don't go straight to 128-bit storage is the storage space. The jump from 32 to 64 will double the storage space needed to store the database, whereas jumping to 128 bits would double it again.

And with larger database sizes, the slower it can be for backend utilities, backups, SiSi mirrors and so-on. Granted, this could all be addressed if there's a need to do so, but it'll take years and years and years before that need may actually arrive. 32 bits to 64 bits may only be doubling amount of bits in use, but the actual increase in available item ID's will increase a factor of roughly 4.29 billion.

I think we're safe with 64-bit for a while :)

DmitryEKT
Clandestine.
Posted - 2010.10.22 15:16:00 - [14]
 

please, please, PLEASE make a complete backup of the database before doing this.
seriously.
you know what i'm talking about.

Destination SkillQueue
Are We There Yet
Posted - 2010.10.22 15:28:00 - [15]
 

Originally by: DmitryEKT
please, please, PLEASE make a complete backup of the database before doing this.
seriously.
you know what i'm talking about.


I'm sure there is no need to do that. What could possibly go wrong?






PS. I'm well aware that something will go horribly wrong with the change and a smart person would do well to ignore my post.

kano donn
New Path
Posted - 2010.10.22 15:30:00 - [16]
 

wait..... the d-scan. how does this affect the max range on that?

do we get to now scan an entire system???

or are we still limited to that silly 2 bill km

Omir Kajil
Gallente
Armada Projection
Posted - 2010.10.22 15:37:00 - [17]
 

4.29 BILLION.
so let me get this straight. Lets assume for a moment that every single ID number right now had to be recycled daily. Which obviously it doesn't, but for simplicity sake and assuming ID usage will at least sorts increase, especially with Incarna, we'll say it's daily.

This means with the new 64 system, numbers will effectively never need to be recycled for the foreseeable future. As that is 4.29 BILLION days.

Which equates to roughly 117,000... Centuries.
Not years.
CENTURIES. One hundred and seventeen thousand centuries.
Or eleven million years. Is my math even remotely right CCP? this seems outrageous to me. In a good way of course Shocked

Katrina Bekers
Gallente
Fighters Squadron
Posted - 2010.10.22 15:46:00 - [18]
 

Originally by: Wollari
I just hope that unsigned BIGINT in MySQL/SQLITE will be enough

Yes

Jarjar
Dreddit
Test Alliance Please Ignore
Posted - 2010.10.22 15:50:00 - [19]
 

Edited by: Jarjar on 22/10/2010 15:54:09
Originally by: Omir Kajil
4.29 BILLION.
so let me get this straight. Lets assume for a moment that every single ID number right now had to be recycled daily. Which obviously it doesn't, but for simplicity sake and assuming ID usage will at least sorts increase, especially with Incarna, we'll say it's daily.

This means with the new 64 system, numbers will effectively never need to be recycled for the foreseeable future. As that is 4.29 BILLION days.

Which equates to roughly 117,000... Centuries.
Not years.
CENTURIES. One hundred and seventeen thousand centuries.
Or eleven million years. Is my math even remotely right CCP? this seems outrageous to me. In a good way of course Shocked

I haven't checked, but your math probably is correct.

2^64 is (2^32)^2.
2^32 = ~4.3 billion
2^64 = ~4.3 billion * ~4.3 billion

TL;DR: The new system will allow almost 4.3 billion times as many ingame items than what's currently possible. Should be enough for... ever, as far as PC gaming is concered. Unless everyone starts unstacking their minerals. Laughing

Aineko Macx
Posted - 2010.10.22 15:50:00 - [20]
 

Edited by: Aineko Macx on 22/10/2010 15:51:57
Originally by: Makko Gray
Or allow negative identities? As from what I remember you don't get the unsigned option on the sql data type.
Of course you do.

Originally by: Jack Gilligan
Quote:
Logs that points to a specific item ID become invalid (so our logs show nothing!)

WOW, if this change fixes the "logs show nothing" problem it will be "Teh best expansion EVAH!"

That would be a truly welcome side effect :)

Quote:
Has any thought been given to the possible impact that retaining that many more item ID's may have on game performance?

Just because you can have more IDs, it doesn't mean the number of items will actually explode. Stuff is still being destroyed, but IDs won't have to be recycled for a looong time. The impact on performance should be minimal. The actual integer math is not the bottleneck and processing 64bit numbers on CPUs with 64bit word width has no penalty. DB row sizes will increase marginally as ID columns will go from 4 to 8 bytes, but assuming rows with an average of hundreds of bytes or more, this is insignificant.

Rakshasa Taisab
Caldari
Sane Industries Inc.
Posted - 2010.10.22 15:54:00 - [21]
 

Edited by: Rakshasa Taisab on 22/10/2010 15:59:11
Originally by: kano donn
wait..... the d-scan. how does this affect the max range on that?

I don't know... How many hogwags per milli-joule of Library of Congress does the SQL server use per popcorn?

Originally by: Aineko Macx
... The actual integer math is not the bottleneck and processing 64bit numbers on CPUs with 64bit word width has no penalty. DB row sizes will increase marginally as ID columns will go from 4 to 8 bytes, but assuming rows with an average of hundreds of bytes or more, this is insignificant.

That is wrong; processing 64bit integers vs. 32 bit results in slower performance due to increased memory bandwidth and cache usage. The CPU core does however process them both in one cycle, yet most real-world applications of this sort are cache bound.

Soden Rah
Gallente
EVE University
Ivy League
Posted - 2010.10.22 16:02:00 - [22]
 

Edited by: Soden Rah on 22/10/2010 16:37:55
Originally by: Sylpheed Hurricane
Why not aim for 128 or even higher? Thus preventing the same update style in i don't know 6years from now on?


To need 128-bit inventory you would need to have 2^63 - 1 items in the database. at 64 bits each thats 73,786,976 Terabytes of data. A 1 TB server harddrive atm costs about 240 ish. so the harddrives would cost around 17,708,874,311 with 300,000 accounts thats around 59k per account. Then you would also need the added cost of the data servers, the refrigerated storage room to hold them and the electricity bill from the whole thing.
Now assumeing that the number of eve accounts grows at 5% per year, and each account generates 1 billion new items per year then the total number of items will exceed 2^63 -1 after appx 150 years.

edit: just for clarity, solve from n=1 to n ((2^63)-1)-SUM(300000*10^9*(1.05^n))>=0

CCP Creber Cattus

Posted - 2010.10.22 16:08:00 - [23]
 

Originally by: Jack Gilligan

Has any thought been given to the possible impact that retaining that many more item ID's may have on game performance?


Yes, we have be running lots of performance tests.
But to get your hopes down from "CCP logs will always show something after this change" we will not keep all deleted items in the database forever, we will still remove the records from the database after some time (which will probably be longer than we now have) to keep the size of the database down, but we will not reuse the item ids.

Originally by: Thordina

Will the maximun number of item-stacks inside a personal-station-hangar will also rise ?
At the moment it is capped at 999.


No this change has nothing to do with that.. sorry.

Originally by: Makko Gray

That extra bit is used for signing, why not use unsigned ints and have twice as many? Or allow negative identities? As from what I remember you don't get the unsigned option on the sql data type.


True, no unsigned types in SQL Server (except tinyint). We use the minus range in some cases for internal stuff (hence, the "we almost never use one of the bits" part Smile )

Originally by: DmitryEKT

please, please, PLEASE make a complete backup of the database before doing this.


what is a backup? Smile (yes we will take a full backup)

Originally by: Omir Kajil

....Or eleven million years. Is my math even remotely right CCP?


Correct, 64 bits will be enough for... forever :) (It will at least not be my problem when that time comes)

Javajunky
Posted - 2010.10.22 16:11:00 - [24]
 

Please for the love of god don't include this in your "Optional Patch" thing or whatever you folks were thinking there. Other than that looks good.

Java

Firartix
Sense of Serendipity
Echoes of Nowhere
Posted - 2010.10.22 16:11:00 - [25]
 

Edited by: Firartix on 22/10/2010 16:14:47
Everything removed, because all my questions got answered while i was posting.
I hate you D: !







..... keep up with the good work though ^^

Guilliman R
Gallente
Northstar Cabal
Important Internet Spaceship League
Posted - 2010.10.22 16:14:00 - [26]
 

Edited by: Guilliman R on 22/10/2010 16:16:33
Edit nm im st00pid


64 is double of 32 amagaed !Shocked

Aurora Robotnik
Caldari
Ghosts of CKSSA
Posted - 2010.10.22 16:15:00 - [27]
 

If it isn't broke, don't fix it.

Oh wait, hi CCP.

Gin Andtonic
Posted - 2010.10.22 16:19:00 - [28]
 

Really, just tell everyone there will be a day without EVE and take the full 24 hours. It will head off a lot of whining. If nothing goes wrong, hey, shorter than expected downtime, or just take the rest of the day to kick back and enjoy VIP mode. If things go completely balls-up, and the potential for ball elevation is nearly limitless with a change like this, you'll have the buffer to fix it without the player base screaming at you.

Also, warning. Give us plenty of warning, please.

Firartix
Sense of Serendipity
Echoes of Nowhere
Posted - 2010.10.22 16:23:00 - [29]
 

Remember the Galactic Terran Vasudan Alliance's motto : "Take chances, make mistakes!".....

More seriously, i think having CCP try to actually improve stuff is a great idea.
It somewhat show more activity IMO than in some game like WoW where they just add features, zones, and never make any kind of database breaking change :P

By the way - could we get some extra info about the performance changes?
I mean there won't be the whole junkyard system anymore so it'll probably affect stuff like, idk, say, missile firing, fighter-bombers, fleet-wide reloads, mining, whatever.

On a final note there's actually one of the 3 points i wanted to talk about in my previous point that wasn't explained...
Well, like, why use global IDs for just every item ?
I don't personally get the purpose of having extractors along missiles and fighters in the database... Sounds like a waste of IDs, inecrase in database size stuff, and moar to me (not like i actually know DBs so i guess it's wrong XD)


Myxx
Atropos Group
Posted - 2010.10.22 16:38:00 - [30]
 

Edited by: Myxx on 22/10/2010 16:41:29
ya'll should open up Sisi during the downtime for this for an armageddon day style event. Could be a nice way to test massive numbers of ship losses and item losses to see what live might be like.


Pages: [1] 2 3 4

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only