open All Channels
seplocked EVE Information Portal
blankseplocked New Dev Blog: Power to the People...
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: 1 2 3 [4] 5 6 7

Author Topic

Arkady Sadik
Minmatar
Electus Matari
Posted - 2011.08.30 21:21:00 - [91]
 

Originally by: CCP Stillman
CSV is a very simple format.


Yes and no. The basic format is trivial, but it gets complicated when you add things like newlines or commas in data sets. Luckily, for this specific data dump, those specialties do not matter at all, so CSV here works exceptionally well. I'm writing this to make sure CSV isn't seen as a great solution for other data exports.

As long as the export only contains numbers and at most dates, CSV works. For anything more complex, use something else.

Quote:
Does that seem like a nice middle-ground?


Yes.

Untested, but:

PostgreSQL:

COPY markethistory FROM 'foo.csv' WITH CSV HEADER;

MySQL:

LOAD DATA INFILE 'foo.csv' INTO TABLE markethistory FIELDS TERMINATED BY ',' LINES TERMINATED BY '\r\n' IGNORE 1 LINES;

SQLite:

.separator ","
.import foo.csv markethistory

(There's even an SQLite module to use a CSV file directly as a table...)

CCP Stillman

Posted - 2011.08.30 21:23:00 - [92]
 

Edited by: CCP Stillman on 30/08/2011 21:23:53
Originally by: Eybium

I can confirm Arkady Sadik's observation is true for the SQL backup as well; the stationID field is actually populated with regionID.



Ooops. I must have forgotten to change the schema after I worked with the data on a stationID basis as opposed to regionID. My bad Embarassed

And yes, the stationID field IS the regionID.

Alain Kinsella
Minmatar
Posted - 2011.08.30 21:44:00 - [93]
 

Edited by: Alain Kinsella on 30/08/2011 21:47:35
Originally by: Arkady Sadik
Originally by: CCP Stillman
CSV is a very simple format.


Yes and no. The basic format is trivial, but it gets complicated when you add things like newlines or commas in data sets. Luckily, for this specific data dump, those specialties do not matter at all, so CSV here works exceptionally well. I'm writing this to make sure CSV isn't seen as a great solution for other data exports.

As long as the export only contains numbers and at most dates, CSV works. For anything more complex, use something else.


The CSV 'format' (of entries separated by commas) is also the default export/import format for Sybase BCP (Bulk-Copy). That handles multiple data types just fine, so long as you don't change the column list over time.

Common command: bcp marketdata in market_file.csv -c -Ufoo -Pfoo -SDB_FOO

I'd probably grab a copy of Sybase SQL Anywhere and use that instead of MSSQL or MySQL, simply due to using ASE and IQ regularly at work (will be adding RAP to that very soon).


@ Stillman - Not to put a wet blanket over the good news, but is this preceding any news of banning cache readers as bot activity? Skreegs has not yet published his expected devblog on what you will consider botting, and supplying market history data seems to be a step towards forcing market data to 'official' channels only (market dump button, and now this).

[Edit - Also, any chance this data dump can be done in OHLC format? (Open-High-Low-Close) Yeah, its a stretch, but I'm curious if you hold that kind of data historically.]

E man Industries
Posted - 2011.08.30 21:48:00 - [94]
 

I really do miss the QER.
Was fun to see the ships used and ship counts and other items not contained in this data dump. Will this data still be presented quarterly?

Micrurus Decoratus
Caldari
Posted - 2011.08.30 21:52:00 - [95]
 

Great news.

To be honest I can't see this being too useful to the majority if the delay is greater than 24 hours, as they can always go to eve-central or eve-marketdata for incomplete, but up-to-date data.

However, that's not to say it isn't useful at all. Even with a one week delay, this could prove useful in creating automated market research tools that work with surprising accuracy.

Salpun
Gallente
Paramount Commerce
Posted - 2011.08.30 21:55:00 - [96]
 

Originally by: E man Industries
I really do miss the QER.
Was fun to see the ships used and ship counts and other items not contained in this data dump. Will this data still be presented quarterly?


Lets get this one sorted firstLaughing
I got a feeling we will be seeing alot more data dumps like what we are working on now once a stable system is in place.

Snorre Sturlasson
Posted - 2011.08.30 22:00:00 - [97]
 

Originally by: CCP Stillman

So how about we throw a page on EVELopedia, where everybody can share how to import a CSV file into their database engine of choice. That means we can release a single format, and if people wish to import it into a database, or any other application one could think of, then they can head on over to EVELopedia and find out how to do it easily.

-Stillman


Nice strategy, because a db like postgresql has already a command to import data via CSV.

Cearain
Caldari
The IMPERIUM of LaZy NATION
Posted - 2011.08.30 22:18:00 - [98]
 

I'm not much for spreadsheets but I do trade a bit. Mostly by traveling and checking the prices for certain goods in the areas I visit. I suppose any variations where I used to make money will be immediately pounced on by those with spreadsheets pulled up on 12 different monitors.Rolling Eyes

I'm not sure trading will be in my future anymore.

But anyway, I'm not even sure how to read this. What are the item and system id numbers?

Adunh Slavy
Ammatar Trade Syndicate
Posted - 2011.08.30 22:22:00 - [99]
 

CSV - It is the lowest common format.

Max Kolonko
Caldari
Worm Nation
Ash Alliance
Posted - 2011.08.30 22:30:00 - [100]
 

Originally by: Adunh Slavy
CSV - It is the lowest common format.


So is XML

Rees Noturana
Red Rock Mining Company
Posted - 2011.08.30 23:13:00 - [101]
 

Originally by: Max Kolonko
Originally by: Adunh Slavy
CSV - It is the lowest common format.


So is XML


XML is fine but considering the simplicity of the output, a simple table, having a less verbose format like CSV can really save on downloading. With XML we'll have as much text wrapping the content as we have in content. I'd say CSV is faster to parse as well.

Both the size of the file and speed of parsing really add up with a large data dump.

Dr Mercy
Posted - 2011.08.30 23:29:00 - [102]
 

For me, there is no almost use in the data as it doesn't delineate between market activity from buy orders and sell orders.

Would you be able to provide the same data but give have a pair values for preceded by BUY and SELL to indicate buy or sell orders?

Arkady Sadik
Minmatar
Electus Matari
Posted - 2011.08.30 23:32:00 - [103]
 

Edited by: Arkady Sadik on 30/08/2011 23:38:41
Originally by: Rees Noturana
With XML we'll have as much text wrapping the content as we have in content.


tr -d '\r' < MarketData.csv | awk -F, 'BEGIN{print "<markethistory>"} NR > 1{print "<history><typeid>" $1 "</typeid><regionid>" $2 "</regionid><datetime>" $3 "</datetime><transactioncount>" $4 "</transactioncount><totalquantity>" $5 "</totalquantity><totalvalue>" $6 "</totalvalue><avgprice>" $7 "</avgprice><maxprice>" $8 "</maxprice><minprice>" $9 "</minprice><medianprice>" $10 "</medianprice></history>"} END{print "</markethistory>"}' > MarketData.xml

571M MarketData.csv
2.1G MarketData.xml

:-D

(This post is meant as humor. If you take it seriously, that's your fault. I don't care if the data is XML, CSV, JSON, YAML, or whatever other fancy format you can come up with, as long as it's text. Some more disclaimers: The data is so trivial that you do not need escaping, so the above works. Yes, you could make it a much simpler XML format by using attributes instead of subelements. XML gurus will stalk you at night, though. Compressed, the two files above are 151M (csv) and 203M (xml).)

Dr Mercy
Posted - 2011.08.30 23:39:00 - [104]
 

And as someone on FHC said "To inform the 5-year plan to enhance nullsec and related discussions about self-sufficiency, the data should include ALL regions, not just the top 5 trade regions"

Can you address this point please?

Jayar Rotuu
Posted - 2011.08.31 00:02:00 - [105]
 

CSV format please.

Agamenox
Posted - 2011.08.31 00:24:00 - [106]
 

Its a terrible mistake give this information to all pilots in game, CCP, must start to evolved and join Roleplaying, PvE and market. Create a new set of missions for industrial, or create a new agents, when the player finish the missions, he could enter to the "secret" market or "smuggler" database with all info.

Stop putting content without doing first a history and game development job.

Eve Mythology and History must grow up!

Vaerah Vahrokha
Minmatar
Vahrokh Consulting
Posted - 2011.08.31 00:27:00 - [107]
 

Edited by: Vaerah Vahrokha on 31/08/2011 00:27:09
In order to be of any worth, the data has to provide:

- OHLC data or at least HLC since EvE markets run 24h a day.
- Be max 1 day old.
- Be of 1 region (selectable), not some aggregate.


Now, I am providing this accurate in game OHLC data service since months:

Thread link on MD

The service provides data (even daily updates), graphs or even an Excel sheet with the data dump.

How can a simple end user be able to produce timely, accurate data (partly based on cache reading) with graphical output but CCP seems to just provide averages and other quite useless (for trading) numbers?

Orisa Medeem
Posted - 2011.08.31 01:08:00 - [108]
 

Originally by: CCP Stillman

I just want to address this one already.

From a technical point of view, this is completely impossible I'm afraid. The amount of data it takes to generate this sort of data is absolutely huge. There has to be a delay, because of that. And it has to be measured in days.

From a design point of view, there's also the concern about people getting a huge advantage from getting this sort of information.

So this leaves an interesting question I'd be curious to hear. Say that we developed an API for this, which would give you the data for any set of regionID and typeID for 7 days ago. How useful would that be?


Guys, please, don't implement a "one size fits all" delay for the information available. There is such a wide range of uses from those 6k item types that no matter which value is chosen some information won't be useful (too old) and some will have a negative impact (too up-to-date information easily available).

I suggest making those delays somewhat dynamic, for example:
- Once a month, calculate for each item type and region which delay it should have, using something like C/sqrt(T), C being a constant and T being the total isk traded in the previous month.
- fit each result in one of the following intervals [1m, 3w, 2w, 1w, 4d, 3d, 2d, 36h, 24h, 18h, 12h, 9h, 6h]. Default to one week if there is no previous month statistics (for the first run and for new item types created)

This way industrial types will have a handy way to check up-to-date mineral prices while keeping some room for merchant types to search for good deals.

Also, by properly choosing C you can solve the performance problem since only a fraction of the items would have their statistics calculated very often.

Monstress
Posted - 2011.08.31 01:09:00 - [109]
 

CSV would be the most ideal format. Many RDBMS provide a way to import CSV data into tables, for example: LOAD DATA (MySQL) or COPY (Postgres).

Anya Ohaya
Posted - 2011.08.31 01:39:00 - [110]
 

  • Definitely CSV format. Not only is it compatible with databases, but also spreadsheets and statistical software like R

  • I would love to get longer time series. You cant do proper analysis of seasonal patterns and trading day influences with less than three years of data. It'd be great to put some hard numbers on the effects of holidays etc on prices and volumes

Abis Cann
Caldari
Due North
Posted - 2011.08.31 01:39:00 - [111]
 

CSV, please. XML being a close second and SQL being a distant third. As to the delay, the shorter the better but I can understand if there needs to be a day or two for data processing.

I'd also like to add that those who worry that a short delay will give an "unfair" advantage to serious traders are absolutely correct that it will be an advantage but I strongly disagree that it would be unfair. If you're willing to learn about markets and economics and put in the time and effort to work with the data then you deserve the advantage. Period. The information will be available to all and it's your decision not to use it so don't complain that others will be less lazy than you and call it unfair. They put in the time, they should reap the rewards.

Moreover, I've heard people claim that a lot of things would kill trading before. The introduction of freighters (and, later, jump freighters in regards to 0.0 markets) was initially feared by some people as the death knell for trading. And, yet, here we are. The fact of the matter is that, unless regional markets approach perfect competition *and* with roughly the same prices server-wide *and* prices never change significantly, trading will be viable. I seriously doubt a daily data dump will create this near-impossible situation, especially since people have been using workarounds for this very purpose for years. Bots are also not an issue if the data is delayed more than 12 hours or so.

Finally, I'd like to say thank you for implementing this. It's been a long time in coming but it's something I've been wishing for since I started trading years ago and is therefore much appreciated.

GrayLensman
Posted - 2011.08.31 01:48:00 - [112]
 

While long overdue I compliment CCP on finally providing an export of near real-time order data. I wrote for my own personal use an application that draws from the data provided by EVE-Central. With a minimal set of alts across targeted regions it is possible to get 24-48 hrs worth of visibility to price differentials to make profit-making between regions for select items possible.

Format is flexible - I use XML against EVE-Central APIs, meaning that while CCP concerns of performance are legitimate - I am in fact self-selecting against a minimal set of item types - which helps with performance (eg. no I am not interested in every type, every region and every price). An export of every type across every region is NOT useful to most pilots - given that - a selective API is the MOST useful application of order exports.

For those pilots without programming skills - some form of CSV export is the most helpful. This format is importable to most users - I leave it to the CCP experts to determine how best to easily export that selectively into Excel, OpenOffice etc.

From a timing perspective - real-time is the best situation, but I am sympathetic to the performance concerns of CCP to a point. Either you provide multi-region, real-time, price comparison like the US stock market or you dont - the fact that an advanced civilization cant have visibility multi-regionaly across all of EVE is well - lazy. There isnt any logical/futuristic restriction that would prevent this - have you ever checked your planetary instllations lately? (e.g its multi-regionally accessible). Dont let real-life supposed restrictions prohibit what would be an expected access to all data no matter where you are in EVE.

Bottom-line this would be very helpful - making it near-real-time is acceptable because reacting to pricing opportunities is not easy once you see them, moving products to take advantage of them takes time. Format needs to be broad - we are a multi-platform pilot community - CCP needs to decide the most easily accessible format as well as those who want minimal data, vs. those who can take advantage of maximal data.


Avat Ore
Posted - 2011.08.31 01:59:00 - [113]
 

I like where this is headed.

CSV over MSSQL backup format; preferred would be xml. My personal preference is to avoid proprietary formats as much as possible.

Delay of 24 hours sounds reasonable. Let those that have their own tools continue to be rewarded for their efforts by allowing them to continue to obtain more up to date data.

Open-High-Low-Close (OHLC) would also be a great addition as mentioned by Alain Kinsella.

Mara Rinn
Posted - 2011.08.31 02:16:00 - [114]
 

Originally by: Hel O'Ween
Please, please do not release the data in CSV format. CSV has no strong data types nor is it locale aware. It's an ugly hack, the lowest common denominator ... from a couple of decades ago.


What do locales have to do with market data exported from EVE Online? The only concern is recognising that all timestamps are in EVE time (i.e.: GMT), and that the text is e.g.: Latin 1 or Unicode. That is to say, all the information in the file comes from the same "locale" and thus the details can be agreed outside the file itself. Unicode character encoding, EVE-time timestamps.

CSV is the lowest common denominator, you have that right. It's a dead-simple format which has its problems, but none of its shortcomings will impact on the utility of the information in the proposed market summary exports.

XML adds needless bulk to the text file. MSSQL backup file is inaccessible to those who aren't running MS SQL Server.

In this instance, CSV is the simplest tool that will get the job done. "Locale" support in XML and other data description languages add needless complexity to the data export, as does packaging the data in opaque formats such as MSSQL backups.

I work with the "official" static data export, but one that was exported from MSSQL into SQLite SQL, which I then processed to ANSI-compliant SQL and imported into PostgreSQL. If that data had been available as plain CSV I would have been a much happier programmer.

Mara Rinn
Posted - 2011.08.31 02:20:00 - [115]
 

For the purpose of budgetary pricing and trend analysis, three-day-old data is perfectly fine. In fact, a weekly dump produced at the same time each week would be excellent. Say during the Tuesday downtime, immediately after any patches are applied.

Anything younger than three days is treading dangerously close to "live" data. There needs to be some room in the game for players engaged in the market to carve an advantage out through their own effort.

The data of "active pilots" and "kills in this system" is up to an hour old, leaving plenty of scope for PvPers to find hot areas but still have to scout for kills. The same should apply to this market data, with the realisation that market PvP happens in a longer timescale than flying-in-space PvP.

Heritor Skoliya
Caldari
Skoliyan Industry
Posted - 2011.08.31 02:29:00 - [116]
 

Probably, best way is API-function that returns historic market data. IMHO.

Suki Okiwana
Posted - 2011.08.31 02:57:00 - [117]
 

CSV please, it is universal and easy to convert. As for frequency, a daily update seems most reasonable; leaves ample wiggle room for dedicated traders and good enough for casuals.

Thoraemond
Minmatar
Far Ranger
Posted - 2011.08.31 03:19:00 - [118]
 

On the basis of: "Ooo! That'd be neat." I'd like current information, if just to see what would happen in the third-party space.

However, on a creating-more-niches game-design basis, I'd prefer that "current" information about the game universe require connection via the game client... so I'd vote for a delay of 48 h and the elimination of cache-scraping capabilities.

Rasz Lin
Caldari
Uitraan Diversified Holdings Incorporated
Posted - 2011.08.31 04:57:00 - [119]
 

Edited by: Rasz Lin on 31/08/2011 05:05:18
Originally by: CCP Dr.EyjoG
[
As CCP_Stillman mentioned then there is a real technical challenge to have close to live data. That challenge will not be resolved soon so let's focus on the 24 hour+ option.



Perhaps CCP should consider hiring someone from eve-central or eve-metrics, as they have/had NO PROBLEM generating this data live 24/7 with no delay.


Originally by: Dr Mercy
For me, there is no almost use in the data as it doesn't delineate between market activity from buy orders and sell orders.

Would you be able to provide the same data but give have a pair values for preceded by BUY and SELL to indicate buy or sell orders?


Good point.

TheSmokingHertog
Posted - 2011.08.31 05:07:00 - [120]
 

Originally by: Meissa Anunthiel
Edited by: Meissa Anunthiel on 30/08/2011 15:45:46
Originally by: CCP Stillman
So this leaves an interesting question I'd be curious to hear. Say that we developed an API for this, which would give you the data for any set of regionID and typeID for 7 days ago. How useful would that be?


.. txt .. because the average price in a region is less important to most than the average price in a system, specifically trade hubs.


Agree with the trade hub info, but the region information is very welcome for inter-regional traders. Are those not to common in your view?


Pages: 1 2 3 [4] 5 6 7

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only