open All Channels
seplocked EVE Information Portal
blankseplocked New Dev Blog: Power to the People...
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: 1 [2] 3 4 5 6 7

Author Topic

Eve Industrialist
Posted - 2011.08.30 15:34:00 - [31]
 

Originally by: CCP Stillman
So this leaves an interesting question I'd be curious to hear. Say that we developed an API for this, which would give you the data for any set of regionID and typeID for 7 days ago. How useful would that be?


I would gladly use this... when eve central returns a host not found error or a zero price.. this would be my fall back data source. In fact I now cache eve central prices for this reason and warn users when they are old. Needed for odd items in non trade regions. I wil also default to this price when cached price from eve central is too old.

Vessper
Indicium Technologies
Hephaestus Forge Alliance
Posted - 2011.08.30 15:34:00 - [32]
 

Almost 6.5 million rows of data - I'll take that in database format. That said, it literally took me 2 minutes to export this from MSSQL to CSV format so I see no reason why CCP can't easily provide both to satisfy everyone.

As for timing, the more recent the better.

GizzyBoy
Posted - 2011.08.30 15:35:00 - [33]
 

FFFFFFFFFFFFFFFFFFFFFFFFFFFFFUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU

I just spent 3 LONG nights integrating 2 different data streams for current and pricing semi current pricing, and then you spit this out @#$##$

and that note, it looks like I'll be adding the next one in the next few days,

can we get some kind feed from eve? im just after avrg's & volume traded atm.


GizzyBoy
Posted - 2011.08.30 15:38:00 - [34]
 

FYI just zip the csv,

mssql is a pain, because each ones a diff format...

tho tables are nice, but if you flat file it it should be ok.
i guess you could do monthly or weekly packs? maybe even once a night 1/2 hour after dt for previous day?

Yohai Hakachi
Posted - 2011.08.30 15:39:00 - [35]
 

Before offering my .02 ISK on this, I would like to congratulate the CCP team, and Dr. Eyjólfur "Eyjo" Guđmundsson in particular for their initiative to make historic market transaction data available.

Formatwise, I am very much in favour of CSV, for the simple reason that it can be imported in just about any program that I know of/use, so these records could full well (and probably will if permission is obtained) be used during statistics/econometrics/economics course(s) etc... for illustration of various topics.

I do have one question however. Will only the data of individual transactions be made available, or will there also be a "daily index", with the average, median, max and min transaction price (on day basis) for the particular items in specific regions?

Yours kindly.

CCP Stillman

Posted - 2011.08.30 15:43:00 - [36]
 

Originally by: Yohai Hakachi

I do have one question however. Will only the data of individual transactions be made available, or will there also be a "daily index", with the average, median, max and min transaction price (on day basis) for the particular items in specific regions?



The data provided is basically the transaction count, total value, volume, min, max, median and average price based on:
- regionID (Top 5)
- date (1 years worth in the zip file)
- typeID (All types sold on the public market)

We won't be providing individual transaction, just the aggregated information contained in the files in the dev blog Smile


Meissa Anunthiel
Redshift Industrial
Rooks and Kings
Posted - 2011.08.30 15:45:00 - [37]
 

Edited by: Meissa Anunthiel on 30/08/2011 15:45:46
Originally by: CCP Stillman
So this leaves an interesting question I'd be curious to hear. Say that we developed an API for this, which would give you the data for any set of regionID and typeID for 7 days ago. How useful would that be?


Quite useful, except for traders.

But then again the more useful the data is for that group of people, the more of a competitive advantage they have over casual traders, and the easier it is to automate (ie, bot).

Anything less than 3 days trailing would be detrimental to casual trading.

And definitely don't provide individual transactions.

That said, 7 days is more than enough for inventors to get a trend on their stock values and whether they should invent something or not, it's also useful for killboards. If you're going for 7 days, I'd add systemID as well, because the average price in a region is less important to most than the average price in a system, specifically trade hubs.

Yohai Hakachi
Posted - 2011.08.30 15:50:00 - [38]
 

Originally by: CCP Stillman

The data provided is basically the transaction count, total value, volume, min, max, median and average price based on:
- regionID (Top 5)
- date (1 years worth in the zip file)
- typeID (All types sold on the public market)

We won't be providing individual transaction, just the aggregated information contained in the files in the dev blog Smile




Ahh, my mistake. Thank you for clearing that up.

malaire
Posted - 2011.08.30 15:55:00 - [39]
 

Originally by: CCP Stillman
So this leaves an interesting question I'd be curious to hear. Say that we developed an API for this, which would give you the data for any set of regionID and typeID for 7 days ago. How useful would that be?

I currently check 1-3 month history when deciding what items to trade in and this could help a lot to find good deals automatically (even if this only contains portion of needed data.)

However, as a programmer I could probably get this data myself quite easily using cache readers, so I don't really mind if you decide to drop this idea completely. (Less information for my competitors ...)

Rakshasa Taisab
Caldari
Sane Industries Inc.
Posted - 2011.08.30 15:56:00 - [40]
 

Originally by: Meissa Anunthiel
That said, 7 days is more than enough for inventors to get a trend on their stock values and whether they should invent something or not, it's also useful for killboards. If you're going for 7 days, I'd add systemID as well, because the average price in a region is less important to most than the average price in a system, specifically trade hubs.

If you want the trade hub price, you use the median price.

Rees Noturana
Red Rock Mining Company
Posted - 2011.08.30 15:59:00 - [41]
 

@CCP Stillman

What is different about the way third party services process market data compared to what you are thinking about doing that requires a delay of days to get at the data? Perhaps a different approach can be taken to streamline price computation.

For an API I'd like what eve-central.com provides but from the official API server. I don't need a historical dump of the last year. Maybe you can provide a regular canned data dump for the historical data. Just publish that somewhere on a regular basis for those that want to generate charts and do more analysis.

stoicfaux
Gallente
Posted - 2011.08.30 15:59:00 - [42]
 

Originally by: CCP Stillman

From a design point of view, there's also the concern about people getting a huge advantage from getting this sort of information.


People already have tools to get mostly up to date and/or historical information, so the "huge advantage" already exists. Making the "huge advantage" a standard feature would just let more people in on it.

Quote:
So this leaves an interesting question I'd be curious to hear. Say that we developed an API for this, which would give you the data for any set of regionID and typeID for 7 days ago. How useful would that be?


I think Arkady Sadik has the answer in post #30.


Wind Jammer
Minmatar
Molden Heath Software Company
Posted - 2011.08.30 16:14:00 - [43]
 

Originally by: CCP Stillman

So this leaves an interesting question I'd be curious to hear. Say that we developed an API for this, which would give you the data for any set of regionID and typeID for 7 days ago. How useful would that be?


Hello CCP Stillman,

I was asking for something along these lines at one of the Dev Track roundtables at Fanfest, so I'm very happy after reading the dev blog and forum posts. I'm the developer of EVE Production Mixer and a few other market analysis tools that aren't public.

I think there are two angles that people will come at this when looking for price data.

Firstly, (this is the case for me and EVE Production Mixer), I'm interested in broad averages to get realistic prices that reflect what a user can actually achieve on the market.

The other angle is people who are looking for 'special sale prices' or unique one-offs that are way outside the averages. To find profitable deals like this I guess you'd need data updates by the minute.

At the moment, I refresh price data from Eve-Central daily. I don't want to diss Eve-Central because they do a great job, but my main problem with Eve Central data is that very low volume items often don't have a price.

In your question about frequency and delay, I would like daily data, and the for the previous day (may be 2 days previous is ok). A good example why: earlier this week someone bought up the entire stock of Fernite Carbide from Jita. For a day, the price was 20% higher than it had. Thanks to the daily price updates, EVE Production Mixer showed a realistic price the next day. I wouldn't be so keen to see that price spike appear in the data a week or 2 weeks later.

Anyway, keep up the good work CCP )

Amsterdam Conversations
Posted - 2011.08.30 16:20:00 - [44]
 

Hey I couldn't be arsed to do the QEN, so here is your data, do it yourself.

Niraia
Seekers of a Silent Paradise
Posted - 2011.08.30 16:27:00 - [45]
 

Cool stuff. MSSQL backup is my prefered format *hides*

CCP Stillman

Posted - 2011.08.30 16:33:00 - [46]
 

Originally by: Wind Jammer
In your question about frequency and delay, I would like daily data, and the for the previous day (may be 2 days previous is ok). A good example why: earlier this week someone bought up the entire stock of Fernite Carbide from Jita. For a day, the price was 20% higher than it had. Thanks to the daily price updates, EVE Production Mixer showed a realistic price the next day. I wouldn't be so keen to see that price spike appear in the data a week or 2 weeks later.


We'll be providing the data on a per-day basis. The question just is, what sort of delay there is on the data :)

radix29
Unforeseen Consequences.
THE UNTHINKABLES
Posted - 2011.08.30 16:34:00 - [47]
 

MSSQL all the way.. btw you can download and install SQL Express for free.


Cypreion
Posted - 2011.08.30 16:39:00 - [48]
 

I think XML would be the easiest format for this kind of thing no?

CCP Dr.EyjoG

Posted - 2011.08.30 16:42:00 - [49]
 

Originally by: Amsterdam Conversations
Hey I couldn't be arsed to do the QEN, so here is your data, do it yourself.


Ahhh, you saw right through me, didn't you Confused

The issue of the time lag of the information is a very interesting one for us. From the discussion so far I gather that anything else than 7 days is a nice to have, anything from 24 hours hold to 7 days is interesting to have, and that less than 24 hour old would be awesome.

As CCP_Stillman mentioned then there is a real technical challenge to have close to live data. That challenge will not be resolved soon so let's focus on the 24 hour+ option.

Does anyone see a problem with providing 24 - 48 hour old data?

Yvan Ratamnim
Posted - 2011.08.30 16:46:00 - [50]
 

Edited by: Yvan Ratamnim on 30/08/2011 16:47:30
WAIT.. WHAT

too much processing to provide 1 hour delayed data? WTF IS GOING ON HERE

Its already available with short delays to an extent via sites like eve-central, eve-marketdata, jitanomics, etc, so its already being exported...

It doesnt have to be a 1 hour or 6 hour delayed from when i first grab it, it can be from set eve times, 0:00 6:00 12:00 18:00 dump it and anything in between is just cached responses...

I mean honestly all the data is already being exported by third parties "he hard way" so why not just do it like this and make everyone happy?

Rees Noturana
Red Rock Mining Company
Posted - 2011.08.30 16:57:00 - [51]
 

Originally by: radix29
MSSQL all the way.. btw you can download and install SQL Express for free.


Windows isn't the only operating system in the world. Data should be in an open format.

Jareck Hunter
Rubicon Legion
Posted - 2011.08.30 16:59:00 - [52]
 

Mhh what about the idea of providing different dumps with different delays for specific marketgroups.

Minerals maybe get a daiyly one, posparts/moonmins a weekly one or so?

Rees Noturana
Red Rock Mining Company
Posted - 2011.08.30 17:00:00 - [53]
 

I'm thinking there is a difference in perspective going on here. CCP is talking about providing us with historical data, which is nice to have. But, many of us just want a way to query prices for our trade and production models. If we can get near real time median prices from trade hubs and also get batch downloads of historical data I think you could satisfy most people.

Yes, I can live with 24-48 hours for computing production costs.

Max Kolonko
Caldari
Worm Nation
Ash Alliance
Posted - 2011.08.30 17:09:00 - [54]
 

Edited by: Max Kolonko on 30/08/2011 17:22:32
Originally by: CCP Stillman
So this leaves an interesting question I'd be curious to hear. Say that we developed an API for this, which would give you the data for any set of regionID and typeID for 7 days ago. How useful would that be?


Originally by: CCP Dr.EyjoG
As CCP_Stillman mentioned then there is a real technical challenge to have close to live data. That challenge will not be resolved soon so let's focus on the 24 hour+ option.

Does anyone see a problem with providing 24 - 48 hour old data?



I would love for it to be no more than 2-3 days old. I don't need it to be updated to the minute.

For the sake of traders competition I dont like it to be anything else (it would be great, but i gueass it would slowly kill casual, less tech savy traders)

Pls make it in API format so we can send something like:
- item ID(or rather list of ID's)
- Region
- Date (or list of Dates)
to limit our queries.

And pls make sure that Your API calls will work nice with Google Docs ImportXML function :)


PS: Great Job Guys

Durin Sarga
Posted - 2011.08.30 17:11:00 - [55]
 

Would love to have an API version of the historical data available on a daily basis, as we can already get from cache readers or see in the historical table provided in the EVE client.

I think daily granularity is enough for inquisitive minds, and a 1 day delay is reasonable.

Just my 2 ISK.

Thebriwan
LUX Uls Xystus
Posted - 2011.08.30 17:21:00 - [56]
 

This is awesome news!

I personally would like to have the data in an XML-format. Second best would be CSV.

I see two different points in this discussion:
a) producers are needing actual data for item _for_ production
b) traders and statistic freaks need massive data for _as much items as possible_

So I propose two different approaches:
a) make all items from the primary producer index available starting from the last downtime
b) all others can wait 2 or 3 downtimes I think


Deamos
Dev Null Development and Holdings
Posted - 2011.08.30 17:22:00 - [57]
 

+1 for API of market data

Cypreion
Posted - 2011.08.30 17:30:00 - [58]
 

XML and JSON are formats of choice sans-whole db dumps.

Iam Widdershins
Project Nemesis
Moar Tears
Posted - 2011.08.30 17:41:00 - [59]
 

Edited by: Iam Widdershins on 30/08/2011 17:42:52
I think this is an OK idea, but the people who currently make serious isk form trans-region market trading will detriment from an influx of overly-informed traders using this API.

I wager that not a whole lot of the people asking for almost-live market data in this thread are currently involved in market trading.

Please implement a substantial delay, 7 days or more.

Iam Widdershins
Project Nemesis
Moar Tears
Posted - 2011.08.30 17:45:00 - [60]
 

Originally by: Meissa Anunthiel
If you're going for 7 days, I'd add systemID as well, because the average price in a region is less important to most than the average price in a system, specifically trade hubs.


This would be TMI. Look at the data dump as it is now, then imagine that 70x bigger.


Pages: 1 [2] 3 4 5 6 7

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only