open All Channels
seplocked EVE Information Portal
blankseplocked New Dev Blog: Power to the People...
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: [1] 2 3 4 5 6 7

Author Topic

CCP Fallout

Posted - 2011.08.30 14:06:00 - [1]
 

Attention market addict! Dr.EyjoG's newest dev blog gives us information about a new initiative: providing historic market data to capsuleers. This is a test run, so please do share your feedback with both Research & Statistics and Team Sleeper Cell in this thread.

Gnulpie
Minmatar
Miner Tech
Posted - 2011.08.30 14:08:00 - [2]
 

Just awesome!

Squizz Caphinator
Woopatang
Posted - 2011.08.30 14:09:00 - [3]
 

yes! yes! yes!

Femaref
Armageddon Day
WE FORM VOLTRON
Posted - 2011.08.30 14:09:00 - [4]
 

omg, this is epic.

From the two, I prefer csv of course, but others would be better.

Lost Hamster
Hamster Holding Corp
Posted - 2011.08.30 14:11:00 - [5]
 

Nice one. Please keep the SQL version. I think that's the easiest to use.

Mors Magne
Astral Adventure
Posted - 2011.08.30 14:12:00 - [6]
 

Quote: "Fly safe but remember to destroy stuff - it is good for the EVE economy!"

Very Happy

Femaref
Armageddon Day
WE FORM VOLTRON
Posted - 2011.08.30 14:16:00 - [7]
 

Originally by: Lost Hamster
Nice one. Please keep the SQL version. I think that's the easiest to use.


It's not sql, it's mssql backup. Put another way: PITA.

Myxx
Atropos Group
Posted - 2011.08.30 14:17:00 - [8]
 

I prefer Csv.

is it possible to keep both for those of us who like one but not the other?

Arkady Sadik
Minmatar
Electus Matari
Posted - 2011.08.30 14:21:00 - [9]
 

Thank you for this data. It's very useful. So useful, in fact, that we already collect it :o)

When you view the market history in-game, the data is written to a cache file. As cache reading is legal, we have a program that reads the cache file and uploads that data to our server. We also have a JavaScript tool for the IGB that opens the market window for a number of "interesting" types. This used to be provided by eve-metrics, but they closed down. :-( We have most of the data from eve-metrics, with some item histories going back as far as May 2008.

What we use it for

We generate a 7-day "index" price for materials in the regions we operate in. We buy materials below that index, calculate our production cost based on that index, and sell products above that calculated production cost. This allows us to easily cooperate between multiple players (some buy, some sell) without complicated record keeping.

The resulting index is public and can be found here:
http://www.electusmatari.com/gmi/

What format would be useful?

Well, it ends up in an SQL table here. Getting it in SQL directly is good. But to translate .bak files into SQL requires some MS program, and for the official DB dump, we actually rely on a 3rd party (zofu for president) to translate it to PostgreSQL format. I suspect a lot of others use MySQL. MSSQL backup format is pretty much unusable for most people, and CSV is easier to translate into something sensible.

Have you considered providing an API interface instead of file dumps?

Chribba
Otherworld Enterprises
Otherworld Empire
Posted - 2011.08.30 14:22:00 - [10]
 

There we go.

Dierdra Vaal
Caldari
Veto.
Veto Corp
Posted - 2011.08.30 14:26:00 - [11]
 

Edited by: Dierdra Vaal on 30/08/2011 14:28:08
I think this is a great initiative. I hope the developers are also looking at services like eve-central which are providing this sort of information already, to get an idea of how people use this sort of data. :)

As such, is it possible that this will be upgraded to an API call? Where, similar to eve-central, I can query one or more itemIds (as well as location parameters, time parameters, etc) and get their up-to-date price statistics back?

rofflesausage
Posted - 2011.08.30 14:32:00 - [12]
 

Edited by: rofflesausage on 30/08/2011 14:32:34
Out of the two formats listed, CSV.

.bak files are a pain to work with in my experience.

Rees Noturana
Red Rock Mining Company
Posted - 2011.08.30 14:36:00 - [13]
 

Edited by: Rees Noturana on 30/08/2011 15:08:32
CSV means simpler access to a lot of different programming languages.

MSSQL backup format means evil MS land, angry, angry, flame war. I think we've had enough of those lately. I don't need the added hassle of converting the data into something simple and readable before I run my script on it. Open standards beats weird MS garbage any day.

edit:

The shorter the delay the better of course. I can understand not wanting to release today's or even yesterday's prices though. I think one week should be good enough to prevent prices from totally leveling out across the empires and providing a close enough average for producers to price their wares. We have trade alts for those that need exact pricing from a distance.

Meissa Anunthiel
Redshift Industrial
Rooks and Kings
Posted - 2011.08.30 14:39:00 - [14]
 

Wonderful,

This has been a long time coming but is very very welcome.

Jareck Hunter
Rubicon Legion
Posted - 2011.08.30 14:47:00 - [15]
 

About the Delay.
I think i'm not only speaking for myself here, but most of the time, i'm not interrested in the price of tritanium 3 months or 2 years ago, if i want to feed my sheets for production and other things, i want to know how much something cost today, maybe yesterday or last week, so i can work with those numbers.

I think sites like eve-metrics or eve-central, got used a lot cause they provided actuall data, that could be read out automaticly.

So an update every downtime for items that get traded a lot, every 2-3 days things that get traded sometimes and maybe every week for low transaction items could be a way to go.

If you could archive something like that, that would be nice.

Marcel Devereux
Aideron Robotics
Posted - 2011.08.30 14:50:00 - [16]
 

Any text format (CSV, JSON, XML, etc..) over a MSSQL backup. If you go with MSSQL backup we have to rely on someone with a copy of MSSQL to do the conversion for us. Seeing as you already have it please do it for us. Thanks!

Face ofBoe
Posted - 2011.08.30 15:01:00 - [17]
 

Anyone who can restore an mssql bak can just as easily consume a csv. Just provide the csv and we can take it from there.

Mechael
Minmatar
Helhest 1st Prospectors' Collective
Posted - 2011.08.30 15:04:00 - [18]
 

.csv format would be greatly preferred. MSSQL is a pain in the ass. I'd like to request JSON as I tend to code in node.js and mongodb, but that wasn't an option.

As to a reasonable interval, I'd like to request as often as possible. An API would be preferred, actually. Please, don't make us wait longer than one week between updates.

This will be a huge, huge help and very powerful tool. With what tools we players currently have, in order to get complete market data we have to have someone (or multiple people) manually look up every single item in every single region so that we can read the cache files. It's not very feasible, and very obviously a large headache.

CCP Stillman

Posted - 2011.08.30 15:08:00 - [19]
 

Originally by: Dierdra Vaal

As such, is it possible that this will be upgraded to an API call? Where, similar to eve-central, I can query one or more itemIds (as well as location parameters, time parameters, etc) and get their up-to-date price statistics back?

Originally by: Arkady Sadik

Have you considered providing an API interface instead of file dumps?



It's no coincidence that the API team has been working with Dr. EyjoG and his team on this. Our ultimate goal is to expose this through an API with more up to date data. This release is to gather feedback and find out how you guys can best benefit from a such potential API.

Avall
Posted - 2011.08.30 15:11:00 - [20]
 

This is awesome!

1) Any text format is preferable to .bak. Personally I prefer XML.

2) Please please please make this an API call, per item. This way it could be used in web apps or mobile apps live, without having to download a large file or a lot of data.

3) Delay should be in the range of hours, absolute maximum is one day. Otherwise the data won't be used very much since more up to date data is available from crowd sourced cache reading repositories like eve central and others.

Looking forward to integrating this into Leaky Capsule, my iPhone app (yes, that was a shameless plug :)

Dierdra Vaal
Caldari
Veto.
Veto Corp
Posted - 2011.08.30 15:16:00 - [21]
 

Edited by: Dierdra Vaal on 30/08/2011 15:20:31
Originally by: CCP Stillman
Originally by: Dierdra Vaal

As such, is it possible that this will be upgraded to an API call? Where, similar to eve-central, I can query one or more itemIds (as well as location parameters, time parameters, etc) and get their up-to-date price statistics back?

Originally by: Arkady Sadik

Have you considered providing an API interface instead of file dumps?



It's no coincidence that the API team has been working with Dr. EyjoG and his team on this. Our ultimate goal is to expose this through an API with more up to date data. This release is to gather feedback and find out how you guys can best benefit from a such potential API.


If you do I'll buy you so much beer at fanfest <3

For my project the eve-central service is sufficient. Min, Max, Avg and Median prices per item, while allowing filtering on location (system(s), region(s), etc) or time period and allowing me to query multiple itemIDs at once.

The biggest thing I'm missing is similar price statistics for faction and deadspace items (basically anything that cannot be traded on the market), but I suspect thats a whole different set of problems.

In general I think the more information you can give people the better it is. Eventually people might even come up with totally new things to do with it that we can't do at the moment.

Vanessa Vansen
Posted - 2011.08.30 15:21:00 - [22]
 

Edited by: Vanessa Vansen on 30/08/2011 15:21:35
About the format ...
please use the most general format, i.e. CSV over MSSQL backup

however, I would vote for xml

CCP Stillman

Posted - 2011.08.30 15:23:00 - [23]
 

Originally by: Avall
Delay should be in the range of hours, absolute maximum is one day. Otherwise the data won't be used very much since more up to date data is available from crowd sourced cache reading repositories like eve central and others.

I just want to address this one already.

From a technical point of view, this is completely impossible I'm afraid. The amount of data it takes to generate this sort of data is absolutely huge. There has to be a delay, because of that. And it has to be measured in days.

From a design point of view, there's also the concern about people getting a huge advantage from getting this sort of information.

So this leaves an interesting question I'd be curious to hear. Say that we developed an API for this, which would give you the data for any set of regionID and typeID for 7 days ago. How useful would that be?

Sgt Napalm
Veto.
Veto Corp
Posted - 2011.08.30 15:23:00 - [24]
 

How much will the subscription based license cost in Aurum?

Arkady Sadik
Minmatar
Electus Matari
Posted - 2011.08.30 15:24:00 - [25]
 

Well, as mentioned, I think the most interesting data is roughly the last 28 days at most. There are uses for older data, but they're rare. For example, I have used old market data in the past to figure out which patch introduced a change that caused a significant price changes (there's a nice bulge in Tritanium which lasted for a year and starts and ends at specific patches, say).

That is, most applications will be interested only in the last few days of data. Our own application would do a daily query of about 380 types (if this is API-queryable, maybe even all types with a marketgroupid) once per day to get the last 7 days of data for four regions, nothing else.

CCP Stillman

Posted - 2011.08.30 15:24:00 - [26]
 

Originally by: Dierdra Vaal

The biggest thing I'm missing is similar price statistics for faction and deadspace items (basically anything that cannot be traded on the market), but I suspect thats a whole different set of problems.


I'm afraid that's outside the scope of this project, and is a very difficult problem in general. Sorry.

Eve Industrialist
Posted - 2011.08.30 15:25:00 - [27]
 

Looking forward to this. Api similar in form to eve cental xml would be best. Csv is doable as well. My first instict was to say sql but then I realized I am not likely to keep it up to date as such. A delay or csv only release would keep the eve central guys in business... tho I agree a delay longer than a day on mineral prices makes this data not as good as eve central.

Dierdra Vaal
Caldari
Veto.
Veto Corp
Posted - 2011.08.30 15:25:00 - [28]
 

Edited by: Dierdra Vaal on 30/08/2011 15:28:35
Originally by: CCP Stillman
So this leaves an interesting question I'd be curious to hear. Say that we developed an API for this, which would give you the data for any set of regionID and typeID for 7 days ago. How useful would that be?


Given how quickly markets can change, I think there will be many cases where 7 days will be too long a delay. Could it be a daily updated thing?

As an added question (just to make sure), will an API call allow universe-wide statistics like eve-central? While it's great to be able to filter by region(s) or system(s), sometimes you just want to get the lowest/highest price in the entire universe. I'm asking this as it would be functionally a little different from the current market mechanics as they exist ingame.

Nahkep Narmelion
Gallente
CALIMA COLLABORATIVE
Atrox Urbanis Respublique Abundatia
Posted - 2011.08.30 15:26:00 - [29]
 

CSV is preferred, but the file is too big for Excel. I'll likely use it in SAS for analysis, and csv files can be directly exported into SAS. But my guess is not many people will have access to something as powerful as SAS, so if the data could be broken up it might be even more useful.

Arkady Sadik
Minmatar
Electus Matari
Posted - 2011.08.30 15:27:00 - [30]
 

Stop posting while I post! <.<

Originally by: CCP Stillman
So this leaves an interesting question I'd be curious to hear. Say that we developed an API for this, which would give you the data for any set of regionID and typeID for 7 days ago. How useful would that be?


For our uses, not useful at all. We can get up-to-date data with our existing toolchain, and we use the data from the last 7 days. Anything before that is interesting mostly for historical reasons, or for items that are traded very rarely.


Pages: [1] 2 3 4 5 6 7

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only