open All Channels
seplocked EVE Information Portal
blankseplocked New Dev Blog: Power to the People...
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: 1 2 3 4 [5] 6 7

Author Topic

Kossaw
H A V O C
Cascade Imminent
Posted - 2011.08.31 05:20:00 - [121]
 

Edited by: Kossaw on 31/08/2011 05:22:24
CSV please. Most web based applications use MySQL (or anything BUT MSSQL) converting MSSQL to CSV is a pain in the butt for most of us.

Several 3rd party sites are already pulling market data real time from the client cache and we can get up to date market pricing within an hour or so.

So really, you need to do the whole job properly and give us an API call with updates approximately every 60 minutes or so. The data doesn't have to be instant, just reasonably fresh.

Edit: Forgot to say... Awesome. Thanks !

malaire
Posted - 2011.08.31 06:54:00 - [122]
 

Edited by: malaire on 31/08/2011 07:16:16
Originally by: Cearain
But anyway, I'm not even sure how to read this. What are the item and system id numbers?


For TypeIDs, download
http://zofu.no-ip.de/inca10/inca10-mysql5-sql-v1/inca10-invTypes-mysql5-v1.sql.bz2
and for RegionIDs:
http://zofu.no-ip.de/inca10/inca10-mysql5-sql-v1/inca10-mapRegions-mysql5-v1.sql.bz2

Both are in SQL format, which is quite simple to parse if you just look at the datalines and ignore extra code at beginning/end.

Original thread about these datafiles:
Static Data Export and Image export for Incarna 1.0

edit: typo

Arkady Sadik
Minmatar
Electus Matari
Posted - 2011.08.31 07:10:00 - [123]
 

Slightly tangential, could someone explain to me the use of an "opening" value for this dump? (Genuine question :-))

Some people ask for OHLC values. In a 24h market like EVE's, C is not useful as it's the O of the next day. We already have H and L in this dump, leaves O. If I understand correctly, O would be the value of the first trade after 00:00 - as this is regardless of volume, what use is this value exactly?

Vaerah Vahrokha
Minmatar
Vahrokh Consulting
Posted - 2011.08.31 07:39:00 - [124]
 

Edited by: Vaerah Vahrokha on 31/08/2011 07:41:44
Originally by: Mara Rinn
For the purpose of budgetary pricing and trend analysis, three-day-old data is perfectly fine. In fact, a weekly dump produced at the same time each week would be excellent. Say during the Tuesday downtime, immediately after any patches are applied.

Anything younger than three days is treading dangerously close to "live" data. There needs to be some room in the game for players engaged in the market to carve an advantage out through their own effort.




While for industrialists and general "buy and hold" people this is sufficient, it's still half a swing too late data for traders who need to evaluate market sentiment in a timely manner.

Basically in EvE you get:

- 0.01 ISK station traders (similar to RL "scalpers").
- 1-7 day "traditional" traders (similar to RL swing traders).
- > 7 day buy and hold buyers (similar to RL stock holders).

The first type does not really need an hystorical data feed.
The second type needs it and needs it in OHLC format.
The third type does not need it a lot but if it was OHLC data they'd get much more information than just some averages.

With the available tools, any end user can self-create the data feature presented in this dev blog by spending 1 hour installing 2 tools and that data will be fresh and updated to the last day or less.


Originally by: Arkady Sadik
Slightly tangential, could someone explain to me the use of an "opening" value for this dump? (Genuine question :-))

Some people ask for OHLC values. In a 24h market like EVE's, C is not useful as it's the O of the next day. We already have H and L in this dump, leaves O. If I understand correctly, O would be the value of the first trade after 00:00 - as this is regardless of volume, what use is this value exactly?


HLC values are the "entry point", OHLC values would be very easily produced by CCP by copying (in the export) the last day close into this day open and would save 3rd party users the hassle of importing CSV data, adding the missing datum, re-exporting it and finally feed it into the graphing applications (that generally demand full OHLC data).

Daedalus II
Helios Research
Posted - 2011.08.31 07:45:00 - [125]
 

I don't know if this has been asked already but there is one thing I'd like to see very much:
A differentiation between sell and buy price!

The price we see now is some hodgepodge of both, isn't it? As we all know there is a difference between the sell and buy price, and that is quite important if you're a trader.
It annoys me that this isn't available in the ingame market either.

Vaerah Vahrokha
Minmatar
Vahrokh Consulting
Posted - 2011.08.31 07:50:00 - [126]
 

Edited by: Vaerah Vahrokha on 31/08/2011 07:53:10
Examples of how my own freely provided OHLC EvE markets The Forge Region data exports look like in RL finance graphing applications:


Please visit your user settings to re-enable images.

Please visit your user settings to re-enable images.


It's possible to see how using OHLC data unlocks valuable features like: candle stick bars, very easy support / resistance spotting, ability to use free time frames (daily candle sticks, weekly candle sticks and so on).

Furthermore, with OHLC you can finally use semi-professional or professional financial tools for much improved visualization. Those who wish may draw classic or proprietary indicators on EvE data and so on.

With OHLC basically a whole new universe of trading is open to you.

I have implemented a decent implementation of OHLC as a puny end user, please CCP do better than me.

Arkady Sadik
Minmatar
Electus Matari
Posted - 2011.08.31 07:51:00 - [127]
 

Originally by: Daedalus II
I don't know if this has been asked already but there is one thing I'd like to see very much:
A differentiation between sell and buy price!


This has been asked for a few times, but it does not make sense.

This data is market history data, not market order data. The market history stores transfers that actually happen. The way the EVE market works, for any transfer to happen, there has to be a buy order and a sell order at the same time. If you "buy from a sell order", the mechanic is that you create a buy order for that item at that price and the market fulfills that. Likewise selling to a buy order. (The only difference being no taxes for buying from a sell order right away.)

There is no difference between buy and sell orders in this data. It doesn't exist. Market history, not market orders. (Market window, "history" tab, not "orders" tab.)

Daedalus II
Helios Research
Posted - 2011.08.31 08:03:00 - [128]
 

Originally by: Arkady Sadik
(The only difference being no taxes for buying from a sell order right away.)

I understand what you're meaning, but as you say there is also difference, at least as taxes are concerned.

It has to be possible to see if a transaction originated from a sell order or a buy order? The tax system obviously knows the difference. Then you simply keep two indices, one for orders bought from a buy order and one for orders bought from a sell order.

Vaerah Vahrokha
Minmatar
Vahrokh Consulting
Posted - 2011.08.31 08:14:00 - [129]
 

Edited by: Vaerah Vahrokha on 31/08/2011 08:14:12
Originally by: Daedalus II
Originally by: Arkady Sadik
(The only difference being no taxes for buying from a sell order right away.)

I understand what you're meaning, but as you say there is also difference, at least as taxes are concerned.

It has to be possible to see if a transaction originated from a sell order or a buy order? The tax system obviously knows the difference. Then you simply keep two indices, one for orders bought from a buy order and one for orders bought from a sell order.


This is a market data dump, not a trading book data dump. I.e. you are thinking "EvE Central" while this is not that. Market happen when demand meets offer generating a trade. At that point of contact, the buy and ask became one and it's meaningless to keep track of both in a *market* data dump.

Daedalus II
Helios Research
Posted - 2011.08.31 08:21:00 - [130]
 

Originally by: Vaerah Vahrokha
Edited by: Vaerah Vahrokha on 31/08/2011 08:14:12
Originally by: Daedalus II
Originally by: Arkady Sadik
(The only difference being no taxes for buying from a sell order right away.)

I understand what you're meaning, but as you say there is also difference, at least as taxes are concerned.

It has to be possible to see if a transaction originated from a sell order or a buy order? The tax system obviously knows the difference. Then you simply keep two indices, one for orders bought from a buy order and one for orders bought from a sell order.


This is a market data dump, not a trading book data dump. I.e. you are thinking "EvE Central" while this is not that. Market happen when demand meets offer generating a trade. At that point of contact, the buy and ask became one and it's meaningless to keep track of both in a *market* data dump.

Right I got you.

So why don't we want a trading book data dump instead? It seems more useful? And if you just HAVE to have your market data dump instead, just slap the buy and sell indices together and you get that.

malaire
Posted - 2011.08.31 08:22:00 - [131]
 

Originally by: Alain Kinsella
@ Stillman - Not to put a wet blanket over the good news, but is this preceding any news of banning cache readers as bot activity? Skreegs has not yet published his expected devblog on what you will consider botting, and supplying market history data seems to be a step towards forcing market data to 'official' channels only (market dump button, and now this).

Now that would be really interesting. No more external applications showing which orders I need to update and to what price. (At least not as easily as before, since I would need to Export each item.)

Kris Hakomairos
Posted - 2011.08.31 08:33:00 - [132]
 

Originally by: Jareck Hunter
About the Delay.
I think i'm not only speaking for myself here, but most of the time, i'm not interrested in the price of tritanium 3 months or 2 years ago, if i want to feed my sheets for production and other things, i want to know how much something cost today, maybe yesterday or last week, so i can work with those numbers.

I think sites like eve-metrics or eve-central, got used a lot cause they provided actuall data, that could be read out automaticly.

So an update every downtime for items that get traded a lot, every 2-3 days things that get traded sometimes and maybe every week for low transaction items could be a way to go.

If you could archive something like that, that would be nice.

Perhaps a little naive.

Historical information allows you to assess long term recurring trends. For instance, if you know that historically a particular item has higher demand at a particular time of the year, you can ramp up production of that item before the demand spikes, frequently taking advantage of lower mineral prices, which means more profit for you. The in-game 1 year graph just isn't enough to determine whether a spike is a blip or an annual recurring trend - this historical information is extremely useful to savvy marketeers.

Gaetring Xana
Amarr
Unstable Reaction Inc.
Posted - 2011.08.31 08:47:00 - [133]
 

I'd say update it quarterly with ALL regions and systems. Perhaps more recent when some sort of API implimentation emerges.

Also, what about putting the data dump in a spreadsheet format instead of database? I know the two are similar but still..

Florestan Bronstein
24th Imperial Crusade
Posted - 2011.08.31 08:51:00 - [134]
 

me gusta

CCP Stillman

Posted - 2011.08.31 09:11:00 - [135]
 

Originally by: Rasz Lin
Edited by: Rasz Lin on 31/08/2011 05:05:18
Originally by: CCP Dr.EyjoG
[
As CCP_Stillman mentioned then there is a real technical challenge to have close to live data. That challenge will not be resolved soon so let's focus on the 24 hour+ option.



Perhaps CCP should consider hiring someone from eve-central or eve-metrics, as they have/had NO PROBLEM generating this data live 24/7 with no delay.



Services like EVE-Central are wonderful tools. But you have to understand that to generate this data, you have to correlate every single transaction in the game for any given day and any given time. They're based off orders, as opposed to the data here which is generated from transactions. There's a one to many relationship between orders and transactions. Because of that, and EVE-Central most likely doesn't have a 1hz frequency on updates from all markets and all items, the data we have to work with is many times larger.

I'd personally love to provide this sort of data live because it satisfies my interest in the data. But it's not feasible because of the severe performance impact it would have.

Erik CoolBreeze
Amarr
Posted - 2011.08.31 09:15:00 - [136]
 

make it an api call and give us hourly updates, else why bother with this? just go to eve-central instead.

CCP Stillman

Posted - 2011.08.31 09:15:00 - [137]
 

Originally by: Arkady Sadik
Originally by: Daedalus II
I don't know if this has been asked already but there is one thing I'd like to see very much:
A differentiation between sell and buy price!


This has been asked for a few times, but it does not make sense.

This data is market history data, not market order data. The market history stores transfers that actually happen. The way the EVE market works, for any transfer to happen, there has to be a buy order and a sell order at the same time. If you "buy from a sell order", the mechanic is that you create a buy order for that item at that price and the market fulfills that. Likewise selling to a buy order. (The only difference being no taxes for buying from a sell order right away.)

There is no difference between buy and sell orders in this data. It doesn't exist. Market history, not market orders. (Market window, "history" tab, not "orders" tab.)

This is the VERY IMPORTANT distinction that differentiates this data with what you might get on EVE-Central. Both are valuable, but they're vastly different in terms of quantity and thus the cost of working with the data on a computational basis.

malaire
Posted - 2011.08.31 09:22:00 - [138]
 

Edited by: malaire on 31/08/2011 09:34:52
Originally by: Erik CoolBreeze
make it an api call and give us hourly updates, else why bother with this? just go to eve-central instead.

If CCP where to provide such data, they would need to include all orders.

Eve-central doesn't do that. They most likely have one IGB-javascript-bot per region to fetch data into cache. That can fetch perhaps 30 items per minute, taking hours to fetch all items. So for any single item they only get one snapshot every few hours, and they miss all orders which are created and fulfilled between these snapshots.

edit: it seems they rely on their users botting for themselves, but the main point still stands, they will miss any orders not included in user-submitted market-order-snapshots.

Meissa Anunthiel
Redshift Industrial
Rooks and Kings
Posted - 2011.08.31 09:30:00 - [139]
 

Originally by: Marcel Devereux
Originally by: Meissa Anunthiel
Originally by: CCP Dr.EyjoG
Originally by: Amsterdam Conversations
Hey I couldn't be arsed to do the QEN, so here is your data, do it yourself.


Ahhh, you saw right through me, didn't you Confused

The issue of the time lag of the information is a very interesting one for us. From the discussion so far I gather that anything else than 7 days is a nice to have, anything from 24 hours hold to 7 days is interesting to have, and that less than 24 hour old would be awesome.

As CCP_Stillman mentioned then there is a real technical challenge to have close to live data. That challenge will not be resolved soon so let's focus on the 24 hour+ option.

Does anyone see a problem with providing 24 - 48 hour old data?


Makes it more difficult for some traders to profit by making it clearer where the opportunities are (thereby creating competition where there was little or none). Information is power when it comes to markets Eyjo, so the shorter the span, the bigger the damage.

Then again, it may also creates opportunities for enterprising individuals. For markets that have big transaction volume, 24 hours wouldn't be a big deal, for "more remote" regions, it would. Personally I think 3 days is more than enough, it gives enough information that people would have to move their arses and see what it's like "down there" right now.




And by bigger the damage you mean more tears. This is EVE. They will need to HTFU or do something else. Universal access to data and competition is good.



Provide the exact same information in game without a person having to travel between regions and I'd agree. Requiring people to implement databases, API calls and other such things is unreasonable. Personally I have no issue with it, I'm a developper, but not everyone is, and those people matter too...

This API/export differs from other APIs in this that it provides information that isn't readily available to people in-game, that's a big difference.

CCP Stillman

Posted - 2011.08.31 09:37:00 - [140]
 

Originally by: Meissa Anunthiel

Provide the exact same information in game without a person having to travel between regions and I'd agree. Requiring people to implement databases, API calls and other such things is unreasonable. Personally I have no issue with it, I'm a developper, but not everyone is, and those people matter too...

This API/export differs from other APIs in this that it provides information that isn't readily available to people in-game, that's a big difference.

Being a programmer and market dude myself, I can think of numerous ways to gain an edge off this data. But then again, as pointed out, there's already ways of scraping very similar data in-game, though less efficiently.

So I'd be very interested in hearing any arguments against provide data every, say 24 hours. Would that ruin it for the small guys, who can't do these sorts of tools?

malaire
Posted - 2011.08.31 09:51:00 - [141]
 

Originally by: Meissa Anunthiel
Provide the exact same information in game without a person having to travel between regions and I'd agree. Requiring people to implement databases, API calls and other such things is unreasonable. Personally I have no issue with it, I'm a developper, but not everyone is, and those people matter too...

This API/export differs from other APIs in this that it provides information that isn't readily available to people in-game, that's a big difference.

What about giving same data in-game?

For current region: all orders and full market history (as it is currently)
For all other regions: delayed market history

Someone could get confused about different market histories (full vs. delayed) but then EVE is full of things which confuse someone.

Daid Thiefsant
Posted - 2011.08.31 10:17:00 - [142]
 

While nice, I really miss the buy/sell price difference in this.

I live in nulsec, so I'm not a trader. Because we live deep in nulsec we need to fly in some stuff we need. Having reasonable accurate prices on those items is very useful to know, because then we'll know how much we'll spend before we fly to hisec. We can also see which loot is valuable and which is junk.

Aineko Macx
Posted - 2011.08.31 10:19:00 - [143]
 

I understand this is supposed to be data on the transactions only, however, some limited/aggregated data on orders and spread would be highly welcome. For this, I suggest the following indices:
- Average over the lowest sell orders equivalent to (for instance) 25-50% of the daily traded volume*
- Average over the highest buy orders equivalent to 25-50% of the daily traded volume*

* this could be the volume averaged over the last week

Dierdra Vaal
Caldari
Veto.
Veto Corp
Posted - 2011.08.31 10:20:00 - [144]
 

Edited by: Dierdra Vaal on 31/08/2011 10:32:00
Originally by: Marcel Devereux
This is EVE. They will need to HTFU or do something else. Universal access to data and competition is good.



I have to agree with this. The more information is available, the more interesting things people will come up with. We can already get reasonably up-to-date info through sites like eve-central so lets not pretend this sort of thing doesn't exist already. Having it done by CCP just means it's more reliable (both in data quality and security).

So I'm very much in favour of a 24h (best) or 48h (acceptable) delay. Die hard traders can still get to the good deals first with this delay while 3rd party developers can still get reasonably accurate prices.

Originally by: CCP Stillman
So I'd be very interested in hearing any arguments against provide data every, say 24 hours. Would that ruin it for the small guys, who can't do these sorts of tools?


I rather doubt it tbh. Judging by some of the posts in this thread and the availability of services like eve-central, the super serious traders already have all sorts of financial graphs at their disposal. This might make their data a little more reliable but it doesn't seem to give them anything they don't already have.

And yet, even with these trading behemoths, plenty of small time traders (who aren't quite as srs bsns) manage to flourish just fine.

On top of that, we've seen that quite quickly market analysis tools are made public. I'm sure some people have their own spreadsheets and tools, but there are enough developers in the community who will release their own tools (out of altruism, ego or in the hopes of getting ISK donations, etc). As such I don't think there is a real risk that releasing info at a 24h interval means a few super srs bsns traders will walk all over the rest.

And the benefit is obviously all sorts of interesting things the eve community will no doubt come up with, that might not be possible (or not relevant) if the delay was longer.

Callean Drevus
Caldari
Icosahedron Crafts and Shipping
Silent Infinity
Posted - 2011.08.31 10:27:00 - [145]
 

Originally by: Rasz Lin
Edited by: Rasz Lin on 31/08/2011 05:05:18
Originally by: CCP Dr.EyjoG
[
As CCP_Stillman mentioned then there is a real technical challenge to have close to live data. That challenge will not be resolved soon so let's focus on the 24 hour+ option.



Perhaps CCP should consider hiring someone from eve-central or eve-metrics, as they have/had NO PROBLEM generating this data live 24/7 with no delay.


I can tell you that this was only possible because of the low volume of uploads they received. I now do the same thing with EVE Marketeer, and it takes about 0.2 seconds to update the historical data for an item, so as long as I'm at less than 5 updates a second I'm ok (could be faster, it isn't optimized at all). Imagine doing this for the entirely of the EVE universe every time someone updates an order somewhere and it should give you an idea of the scope.

Originally by: CCP Stillman
So I'd be very interested in hearing any arguments against provide data every, say 24 hours. Would that ruin it for the small guys, who can't do these sorts of tools?


Not necessarily since a lot of us provide these tools for free. In fact, I think the best tools that are in use are free (but then again, I cannot see what tools are not available to the general public). The problem is mostly making people aware that they exist Rolling Eyes

hero honda
Posted - 2011.08.31 10:37:00 - [146]
 

can we get an open and close with that? (start of trading day = end of down time,end of trading day = beginning of down time)

hero honda
Posted - 2011.08.31 10:40:00 - [147]
 

BTW csv, I can turn csv into whatever format I want easy and parsing a csv is easy.

Alain Kinsella
Minmatar
Posted - 2011.08.31 10:59:00 - [148]
 

Originally by: malaire
Originally by: Alain Kinsella
@ Stillman - Not to put a wet blanket over the good news, but is this preceding any news of banning cache readers as bot activity? Skreegs has not yet published his expected devblog on what you will consider botting, and supplying market history data seems to be a step towards forcing market data to 'official' channels only (market dump button, and now this).

Now that would be really interesting. No more external applications showing which orders I need to update and to what price. (At least not as easily as before, since I would need to Export each item.)


I've seen it before (with SL and introduction of their LindeX(tm) exchange - my own little 'I was there' moment actually Laughing ).

As discussed here though, I think what we're agreeing on currently will not hurt the loss of cache reading (unlike what happened with LL). Would still like an answer though. Razz

Meissa Anunthiel
Redshift Industrial
Rooks and Kings
Posted - 2011.08.31 11:11:00 - [149]
 

Originally by: CCP Stillman
Originally by: Meissa Anunthiel

Provide the exact same information in game without a person having to travel between regions and I'd agree. Requiring people to implement databases, API calls and other such things is unreasonable. Personally I have no issue with it, I'm a developper, but not everyone is, and those people matter too...

This API/export differs from other APIs in this that it provides information that isn't readily available to people in-game, that's a big difference.

Being a programmer and market dude myself, I can think of numerous ways to gain an edge off this data. But then again, as pointed out, there's already ways of scraping very similar data in-game, though less efficiently.

So I'd be very interested in hearing any arguments against provide data every, say 24 hours. Would that ruin it for the small guys, who can't do these sorts of tools?


Available of "mass" data is always very interesting for those who have the means to data mine it, so there's a lot of arguments in favour of these exports (which I support fully).

Smaller/casual traders (which I think are more numerous than the "pros") can benefit in areas that are more "niche" and thus require less micromanaging such as (but not limited to):
- small volume items in trade hubs or missioning centers
- empire region trading (buying in Jita, selling in Amarr)
- risky business (lowsec/nullsec trading)

I know a few people who make quite good money buying in Rens and reselling in Hek or Teonusude for instance (6 jumps from Rens, but in another region).

These markets are discovered through investigation by moving around and checking prices of individual item, something dumps do away with. A long trail of data gives "investigation clues" so data miners still need to verify if the markets are still valid by actually moving there. Short trail provides that data without moving about by showing if the decrease in average/median price is very recent.

Someone with an automated data mining system can easily identify new markets if too many people come into his juicy one, but the trader without those tools will require a significant amount of manual investigation to find something, and by the time he does, that market may already be gone.

My argument is, in essence, that longer trail still requires manual checking, shorter trail does away with that need. The barrier between people without those tools and those with it will increase the shorter the trail (Look what sub 100ms data feeds have done to the stock market as an extreme of what I'm saying). And while I favour giving the people "clues" to investigate, I'm not sure about what full automation would do.

I'm sure Eyjo would love to have sub 100ms market feeds and full automation capability as a research project (I'd love it too, personally), but I'm thinking of the noobs here.

Di55y
Posted - 2011.08.31 11:21:00 - [150]
 

I think you, CCP, should think the other way round:

Deliver as fast as possible as extensive as possible. People will find a reasonable meaning for this data (1+ week/month old data will find its way into blogs, etc. newer data will make profit for nifty traders).

That said, you can start thinking about the how:
- API ?
- Balancing via "what items, regions, stations are interesting for most of the players" -> kind of hit counting / whats popular

So you essentially will have to make your hamsters run at maximum (ridiculous) speed and delay (exactness) will vary for different markets.

Result is:
- one will have almost instant access to tritanium (or ore) prices in jita
- oen will have to rely on jumpfreighter prices which might be a week old (or look it up ingame, via eve-central or co.)

After that, you can think about giving your hamsters +5 nano-leg implants.


Pages: 1 2 3 4 [5] 6 7

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only