open All Channels
seplocked EVE Information Portal
blankseplocked New Dev Blog: The Return of the Data Export
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: 1 [2] 3

Author Topic

Matalino
Posted - 2008.07.31 16:15:00 - [31]
 

Originally by: CCP Prism X
Why is crpNPCCorporationTrades missing?
Well, it should really never have been included. This is basicaly meta-data and could allow people to get an advantage they couldn't otherwise get in game. Something like finding out what is being sold in regions they've never been in would be a good example. Finding the best possible trade routes through some datamining would be another. Wink
I'm sorry if the mismatch has wasted anyones time.
I can understand some of what you are saying, but as a counter point we have long past the point were NPC sales are in anyway secret, with resources such as Eve Central and the Item Database. (unless you are planning to remove it from there too?)

Removing that table from the current official data dump only provides an advantage to those more experienced with the NPC market and Data Dumps, who know to go to the the previous data dumps and how to update that information.

If you insist on providing such an advantage to us veterns, I guess we will need to accept it, but I think that it would be better for newer players and developers if you returned it to the official data dump, even if you flattened the supplyDemand field to contain a true/false value instead of the previous range of values that might be mined for additional information.

CCP Prism X


Gallente
C C P
C C P Alliance
Posted - 2008.07.31 16:54:00 - [32]
 

And in turn I can see where you are coming from but I don't agree with any imbalance between new and old players. The old dump is still out there and easy to find for anyone with an incentive. Obviously that statement is counter-productive to my original one but we don't keep on doing things just because there's a precedence for doing them. Especially not when we think they shouldn't be done. Wink
Besides, EVE is an MMO. We nerf stuff in MMOs. Even data exports. C'mon, I don't get to nerf that much in my position here.

As to a true/false values, that might happen. I'll try and jot it down on my notebook when I get to work and bring it up when the time is right, before the next export.

Matalino
Posted - 2008.07.31 17:27:00 - [33]
 

Originally by: CCP Prism X
Besides, EVE is an MMO. We nerf stuff in MMOs. Even data exports. C'mon, I don't get to nerf that much in my position here.
But we love you because you don't nerf stuff, only boost it. Laughing
Originally by: CCP Prism X
As to a true/false values, that might happen. I'll try and jot it down on my notebook when I get to work and bring it up when the time is right, before the next export.
Thank you for giving it another thought. ugh

Alvara
Kuiper Belt Industries
Empire Research
Posted - 2008.07.31 18:18:00 - [34]
 

Personally I struggled with the previous dumps and usually gave up long before I compiled them myself.

Now a sql backup file I'll have up and doing queries against it in 5 min or less. So thank you very much.

Now back to Application Verifier logs while I try and get my work at work vista certified.

Azazel Mordred
Minmatar
Cloak of Shadows
Posted - 2008.07.31 18:42:00 - [35]
 

As an aside - the table "ramCompletedStatuses" seems to be missing.

Dal Rath
Posted - 2008.07.31 19:19:00 - [36]
 

Quote:
EVElopedia ... will, among other things, include a new and improved Item Database.

...which probably explains why nobody at CCP has bothered to update the existing Item Database yet...

Whilst waiting for EVElopedia, which is coming SOON(tm), you can enjoy the Empyrian Age Item Database here NOW(tm)

Lord Fitz
Project Amargosa
Posted - 2008.07.31 19:24:00 - [37]
 

While I managed with the old format, the new one is a heap easier for me, but then MSSQL is my preferred format anyway. Like before it won't be too long before you find it in any format you like I imagine.

Epitrope
The Citadel Manufacturing and Trade Corporation
Posted - 2008.07.31 23:49:00 - [38]
 

First off, thanks to you and t20 for the updated export! As for crpNPCCorporationTrades:

Originally by: CCP Prism X
Something like finding out what is being sold in regions they've never been in would be a good example.

This information is available in game by looking at the trade tab of an NPC corporation's info window. It's tedious and inconvenient, and I have certainly made the argument in the past that hard things tend not to be done, but I would say it isn't secret information (although the supplyDemand column was). (Plus, as Matalino mentioned, the E-O Item Database has it in a slightly less inconvenient form.)

Originally by: CCP Prism X
As to a true/false values, that might happen. I'll try and jot it down on my notebook when I get to work and bring it up when the time is right, before the next export.


I for one would be happy with just the sell orders. Right now I use the Trinity version of the crpNPCCorporationTrades table by just selecting corporationID, typeID where supplyDemand > 0. Prefiltering the exported table like that would avoid showing even the possibility of trade routes (which would have to be double-checked anyway because of the aforementioned price fluctuation).

Kojebo
Posted - 2008.08.01 01:02:00 - [39]
 

What's the 'technical reason' for not implementing foreign keys?

Chruker
Posted - 2008.08.01 08:14:00 - [40]
 

The invControlTowerResourcePurposes table is empty

BigWhale
Gallente
Three WiseMen Association
Posted - 2008.08.01 10:26:00 - [41]
 

One question, is MS SQL unable to produce an ANSI SQL text dump?

Now, I'll have to rely to 3rd party to get a data dump or buy Windows license. :(

I, for one, _don't_ welcome our proprietary data dump overlords... :(

iulixxi
Caldari
EVE-RO
Imperial 0rder
Posted - 2008.08.01 10:43:00 - [42]
 

Edited by: iulixxi on 01/08/2008 10:45:34
MDB version here (36.9 M)

Iulian

Dronte
Minmatar
Quam Singulari
Posted - 2008.08.01 12:44:00 - [43]
 

Jeeeesus christ, stop whining about the format already!. Its not like you had to wait long, the community had released the first ANSI SQL, MySQL, PSQL etc. version without 2 hours!! of the initial release, and posted links to them, both in this thread and in the various threads on the tech lab forum and on #eve-dev on IRC.

So all this really just sounds to me like you are just whining, for the sake of whining. I doubt any of you would have had any problem if it was released in .sql files even though you couldnt have used them for one reason or the other (historicly, the official CCP dumps didnt import without modifications into MySQL fx.) and therefor there has always been community adaptations of the data dumps that users has had no problems with.

So please just stop whining, and be happy that CCP has managed to find a way, where we dont have to wait half a year on updated data dumps, but only a few hours till the first conversions hit the community scene. This stinks of being the standard "We oh so hate the Microsoft"

Chruker
Posted - 2008.08.01 14:06:00 - [44]
 

Originally by: Dronte
Jeeeesus christ, stop whining about the format already!. Its not like you had to wait long, the community had released the first ANSI SQL, MySQL, PSQL etc. version without 2 hours!! of the initial release, and posted links to them, both in this thread and in the various threads on the tech lab forum and on #eve-dev on IRC.

For me it is more the matter of there being a 3rd party involved (mostly what differencies that might appear between different conversions)

Originally by: Dronte
I doubt any of you would have had any problem if it was released in .sql files even though you couldnt have used them for one reason or the other (historicly, the official CCP dumps didnt import without modifications into MySQL fx.) and therefor there has always been community adaptations of the data dumps that users has had no problems with.

But at least the .sql files could be edited with simple search-n-replace to the point where your database engine of choice could digest them. You try doing that with the .bak crap

Originally by: Dronte
So please just stop whining, and be happy that CCP has managed to find a way, where we dont have to wait half a year on updated data dumps, but only a few hours till the first conversions hit the community scene.

Right, I'll believe that when I see it. They still have to assign somebody during the launch and I think that is the biggest issue.

And dont come here with some crap about this smells like a oh-microsoft-stinks. The issue is that they decided to go with a binary file that cant be as easily converted to something usefull as the .sql files could. So its a step in the wrong direction! The right step would have been to strive to provide something that would be directly importable by most database engine (be it standardized sql, csv, xml or similar).

FInally, I really wonder how much work they did on this 'procedure to export'. It just looks tooo similar to the old schema with the issues it has.

I had hoped they would have normalized the data and also split the typeActivityMAterial table up into the ramTypeMAterials and ramTypeRequirements

Jesho
Minmatar
Swedish Aerospace Inc
The Kadeshi
Posted - 2008.08.01 14:25:00 - [45]
 

Originally by: CCP Prism X
Why is crpNPCCorporationTrades missing?
Well, it should really never have been included. This is basicaly meta-data and could allow people to get an advantage they couldn't otherwise get in game. Something like finding out what is being sold in regions they've never been in would be a good example. Finding the best possible trade routes through some datamining would be another. Wink
I'm sorry if the mismatch has wasted anyones time.
You can already get the same type of data from the Item database at eve-online.com

all you have to do is to find the item your looking for and then go to the NPC tab, there you have the npc corps that sell this item.
when you know the corp that sells the item it's easy to find out where those corps have station(s).

Cesar Malari
Dark-Rising
IT Alliance
Posted - 2008.08.01 14:42:00 - [46]
 

Originally by: Kojebo
What's the 'technical reason' for not implementing foreign keys?

If I had to guess, it's not so much a "technical" reason as a "performance" reason. If you have the FKs on there, they are checked for each insert/update, which takes time. In case you hadn't noticed, database performance is a giant issue for this game, so they likely don't have FKs on the live database.

Now, as to why they don't add them to the export, well, that one I can't think of a reason for, other than that they don't want to bother doing it. Remember -- they're trying to get a nice, easy, repeatable process for themselves here....

Cesar Malari
Dark-Rising
IT Alliance
Posted - 2008.08.01 14:44:00 - [47]
 

Edited by: Cesar Malari on 01/08/2008 14:44:56
Originally by: BigWhale
One question, is MS SQL unable to produce an ANSI SQL text dump?

Natively? No. SQL Server doesn't ship with a tool like mysqldump.

However, there is such a tool available -- the Database Publishing Wizard. Point it at a DB and it builds SQL scripts for the schema and/or data. I think this tool is also included in VS 2008.

In fact, I was already kicked off that tool to dump the SQL out for my own MySQL export when I saw the community-provided ones were available, so never bothered to finish it.

Chruker
Posted - 2008.08.01 14:51:00 - [48]
 

Oh, and all the rendered icons are blurry/out-of-focus

Ami Nia
Posted - 2008.08.01 14:53:00 - [49]
 

Originally by: Dronte
Jeeeesus christ, stop whining about the format already!. Its not like you had to wait long, the community had released the first ANSI SQL, MySQL, PSQL etc. version without 2 hours!! of the initial release, and posted links to them, both in this thread and in the various threads on the tech lab forum and on #eve-dev on IRC.


I strongly disagree.
This is not an OS war or a MS hate cry. It's both a technical and a matter of principle.

To begin with, CCP is supposedly supporting EvE on multiple platforms. Nobody on ANY of the officially supported platforms should ever need to access player-translated data. It is CCP that has to first-handedly provide usable files for each platform. It is not a question of how much time passes between the official files release and the translated files availability. It is the fact that CCP is to do the translations and consider them formally official files (or better yet release a platform independent, human readable, structured file).

Second, each and every binary distribution of a data base is intrinsically an inferior distribution. Any sane data dump shall be in a pure text format, possibly understandable by humans to the extent of being editable with pure text tools (as in notepad). This is a well know fact in the computer science community. There are only two forces that can make a binary file superior: file size (for example for transmission over the net or for reduced storage requirements) and monopolistic format lockouts.

If the first case is the reason to have a binary format, then the solution is to use a text format and then compress it with a well known patent free algorithm and into a well known patent free file format. Like zip, gz or bz.

If the second case is the reason to have a binary format, then you are in fact admitting that you do not want any one (but those few you select) to be able to access the data.

Technically the format CCP chose to distribute the data dump is:
-) a great format for internal use if and only if the company is not mature enough to be open to move to a better platform (that is: MS SQL probably IS the better platform. Now. But there's no guarantee it will still be in 5 or 10 years. And CCP should be ready to adapt if things change).
-) the kind of format an expert of MS SQL will recommend if he/she was to single handedly decide. What format to use when publicly releasing data should never be a single person decision, but always a committee consensus. And the manager should make sure the committee is both technically savvy and as heterogeneous as possible.
-) a very bad format to use for releasing data that is officially destined to multiple platforms.

Whining about the chosen format is therefore absolutely pertinent.


Hel O'Ween
Men On A Mission
EVE Trade Consortium
Posted - 2008.08.01 15:25:00 - [50]
 

Originally by: Cesar Malari
Edited by: Cesar Malari on 01/08/2008 14:44:56
Originally by: BigWhale
One question, is MS SQL unable to produce an ANSI SQL text dump?

Natively? No. SQL Server doesn't ship with a tool like mysqldump.

However, there is such a tool available -- the Database Publishing Wizard. Point it at a DB and it builds SQL scripts for the schema and/or data. I think this tool is also included in VS 2008.


Also, SQL Server Management Studio Express (free download) is a useful tool for SQL Server Express 2005

Cesar Malari
Dark-Rising
IT Alliance
Posted - 2008.08.01 16:00:00 - [51]
 

Edited by: Cesar Malari on 01/08/2008 16:01:29
Originally by: Ami Nia
To begin with, CCP is supposedly supporting EvE on multiple platforms.

I don't believe CCP officially supports anything other than the EvE client on multiple platforms. The data dumps at this point exist as more of a "here, see what you can do with this" than an officially supported released product. Case in point: how long it took after the patch before we got this dump.

Would it be better if CCP supplied this data in a more open format? Yes. However, if CCP doesn't feel it's worth their time to do the conversion, I'd rather they gave us what they've given us than nothing at all. In fact, if I had to choose one of two directions for further work on the data dump: make the export more cross-platform or include more data in the export, I'd pick #2. The community can convert the data rather easily. We can't see the data that's not released.

Constructive suggestions are good (ie. the discussion over crpNPCCorporationTrades and the suggesting a more open format). Whining about the lack of 'official' support (I quote official here because none of this is officially supported -- try filing a petition about it) for our chosen platform mak may make them think we don't appreciate what they've given us. Which we do. We like having more data and more up-to-date data to play with.

BigWhale
Gallente
Three WiseMen Association
Posted - 2008.08.01 16:48:00 - [52]
 

Well, I wanted to point out (what was already mentioned), that data dumps would most commonly be in a human readable form.

I am sure that CCP had a very good reason why they made a binary dump in some obscure format. And I would present the same argument if they gave us Mysql InnoDB binary tables. It simply seems wrong to me. Binary files are a pain and now they are binary files tied to only one platform.

I think that this format was the easiest way of doing TheDataDump. :) I dont blame them for that. The whole community was whining about it. Well, I would be willing to wait a bit more... :)

Chainsaw Plankton
IDLE GUNS
IDLE EMPIRE
Posted - 2008.08.01 17:12:00 - [53]
 

Edited by: Chainsaw Plankton on 01/08/2008 17:12:50
Originally by: CCP Prism X
And in turn I can see where you are coming from but I don't agree with any imbalance between new and old players. The old dump is still out there and easy to find for anyone with an incentive. Obviously that statement is counter-productive to my original one but we don't keep on doing things just because there's a precedence for doing them. Especially not when we think they shouldn't be done. Wink
Besides, EVE is an MMO. We nerf stuff in MMOs. Even data exports. C'mon, I don't get to nerf that much in my position here.

As to a true/false values, that might happen. I'll try and jot it down on my notebook when I get to work and bring it up when the time is right, before the next export.


you are lucky I just got back from the bathroom. otherwise I would have just shat myself there Laughing

all I get to nerf are abaddons* Laughing

*soon to include more amarr ships, and various other ships as well. I just installed flash again, that and I need to figure out what other ships shoot. also an eve-files account so I can have multiple signatures Twisted Evil

DHU InMe
Gallente
Corsairs Inc.
The Spire Collective
Posted - 2008.08.01 22:34:00 - [54]
 

Originally by: Epitrope
First off, thanks to you and t20 for the updated export! As for crpNPCCorporationTrades:

Originally by: CCP Prism X
Something like finding out what is being sold in regions they've never been in would be a good example.

This information is available in game by looking at the trade tab of an NPC corporation's info window. It's tedious and inconvenient, and I have certainly made the argument in the past that hard things tend not to be done, but I would say it isn't secret information (although the supplyDemand column was). (Plus, as Matalino mentioned, the E-O Item Database has it in a slightly less inconvenient form.)

Originally by: CCP Prism X
As to a true/false values, that might happen. I'll try and jot it down on my notebook when I get to work and bring it up when the time is right, before the next export.


I for one would be happy with just the sell orders. Right now I use the Trinity version of the crpNPCCorporationTrades table by just selecting corporationID, typeID where supplyDemand > 0. Prefiltering the exported table like that would avoid showing even the possibility of trade routes (which would have to be double-checked anyway because of the aforementioned price fluctuation).



If we could filter that junk. Some corporation have very ridiculous long list.

I also wonder if it alway 100% acurate in game...

Eraggan Sadarr
Comply Or Die
Posted - 2008.08.02 10:04:00 - [55]
 

Thx for the dump Very Happy

I'm really glad you decided on a more friendly SQL Bak file so that we in the future might see some faster dumps Very Happy

Malrock
Caldari
Mea Culpa Enigma
Posted - 2008.08.02 22:07:00 - [56]
 

Edited by: Malrock on 02/08/2008 22:11:21
Originally by: CCP Prism X
Oh my me, we're still on page one. I must be slipping. At least I managed to include the good old OS wars discussion in this blog. Laughing

Why are you forcing me to use MS products!
We're not really. Obviously some format had to be used and it has pros and cons for anyone involved. Using the same environment as our internal development environment makes everything very easy to maintain as well as offering us a very robust procedure to get this out with our expansions. My original idea was to include a non-official export as well but it became apparent that the robustness would suffer. If this format doesn't cut it for you in any way you can at least be glad that this data will now be released with expansions (I imagine it will be held back until we're sure we don't need to hotfix any live static data) and that still means you'll get your export from third parties a whole lot faster than.. a year.
I see it as a win for everyone involved although people with no access to MS platforms might have to wait a tad longer than others. That being said I'd erally rather want to discuss the export and it's uses rather than go into the oldest discussion in the industry. Wink

Take care, I'll drop by after the weekend, if not sooner.


Reason is here & clear, and i for one fully support it, there are plenty of people out there who can make conversions to any format, the goal here is to have 1 official format coming off ccp AS FAST AS POSSIBLE, then let others do the 100000001 subformats used by anyone else.

And before you go flaming about OS this and that; lets be happy we get anything at all, cause there's nothing saying that ccp must do anything for third party dev's.

There is how ever one simple request. Math on production seems to be off no matter how i try to balance (specifically on t2 stuff), could we get say stored procedures in dump that have the official formulas in them for building, refining, etc ?

Chruker
Posted - 2008.08.03 09:03:00 - [57]
 

agtAgentOffers is also MIA, although I would expect the same excuse as with crpNPCCorporationTrade

Cosy
Posted - 2008.08.04 00:23:00 - [58]
 

is me or i dont see the overheat icon

Azazel Mordred
Minmatar
Cloak of Shadows
Posted - 2008.08.04 17:06:00 - [59]
 

Originally by: Cosy
is me or i dont see the overheat icon


Seems to be there - icon22_10

Althea Nar'agh
Aliastra
Posted - 2008.08.05 07:55:00 - [60]
 

Hi Prism,

First of all thanks for these items we really needed them :)

I have couple quick questions tho...

Am using 64x64 icons and have unpacked:
EmpyreanAge_1.0_img_icons.zip
EmpyreanAge_1.0_img_types_dx9.zip

Shiptypes / 32x32 : 232 files
Shiptypes / 64x64 : 346 files

What gieves? In plus 64_64/29266.png (Apotheosis) does not exist (but it's there in 32_32) so why the diffrence? and it's not the only one with diffrence, other example:

Icons / 16x16 : 1604 files
Icons / 32x32 : 1092 files
Icons / 64x64 : 964 files

And my second question, could we get an allianceToIcon mapping preaty please?

Thanks


Pages: 1 [2] 3

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only