open All Channels
seplocked EVE Technology Lab
blankseplocked Eve-MarketData.com - replacement for eve-metrics
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: first : previous : 1 2 [3] 4 5 6 7 8 9 ... : last (10)

Author Topic

Fovo
Timetravel Enterprises
Posted - 2011.01.07 15:16:00 - [61]
 

One thing I just realised, is I am doing system maintenance on my server this weekend and I am not sure how to temporarily disable the data feeds.

It's not an issue on my side really, but no need for your server to waste resources feeding me data when I am off line.

Recommendation. if you receive a 404, stop the feed until re-requested.

Fovo

Eve Industrialist
Posted - 2011.01.07 15:18:00 - [62]
 

I'm going to get behind this. I used eve central for pricing for my mfg app and just when I get price caching working correctly eve central goes awol. Having alternate price source would be good anyway. I used their xml api and yours is close enough.
This alt sits in jita and can easily pull prices for you on a daily basis or move to another trade hub. I may or may not be able to help with coding depending on code base and needs.

Kallahar
Posted - 2011.01.07 18:40:00 - [63]
 

Improved the automatic uploader, it no longer checks for T2 ship blueprints, and I removed about 50 items that the CCP export *says* are available on the market, but they actually aren't. It still will check items that often don't have any sellers in your region, but that's by design. There are now 6,133 items checked, which will take about 7 hours to scan the entire market for a region for one client.

I've also decided to discontinue the item.xml/json api. I think that 99% of the people using it are only getting the average price, so that they can update their killboard, etc. However it provides far more info, which is CPU intensive. Therefore, I'm going to kill off that program, and people will have to update their code to use one of these new API's:
- http://eve-marketdata.com/developers/item.php for one item lookup
- http://eve-marketdata.com/developers/items.php for multiple

I do plan on adding statistical data back, but it will be with a new URL. This way, people who only need the price won't be wasting CPU or bandwidth getting a bunch of extra data they don't need.

Kallahar

Greshick
Test Alliance Please Ignore
Posted - 2011.01.07 19:18:00 - [64]
 

Not sure if you have the server done or something but I just received a 503 error message when I tried to visited the site in my OoG browser.

Kallahar
Posted - 2011.01.07 19:51:00 - [65]
 

Nightly Data Dumps added (and all mysql exports) to http://eve-marketdata.com/developers/

I'll look into the 503 errors, for now just reload the page.

Kallahar

Greshick
Test Alliance Please Ignore
Posted - 2011.01.08 00:23:00 - [66]
 

Thanks for looking into the 503's. I like the work your doing so far and I have been trying to help keep at least Jita's data up to date. A thought we would be to allow filtering by sec status, so could only see in high sec or filter by station. The stations would be nice for selling and buying in Jita.

Droxlyn
Caldari
Posted - 2011.01.08 03:31:00 - [67]
 

I'm mostly looking for the 5% market volume buy effective buy price and the 5% market volume sell price, ideally for a specific system. (IE Jita or Amarr)

That way I can look up my costs for input materials and the profit for the easy sell. I sometimes average the two together to get the middle buy/sell price that should go fast but make me or cost me just a little.

Averages and Medians are not as interesting to me. (Nor are the extremes.)

Drox

Dharken Rhaal
Amarr
Posted - 2011.01.09 07:58:00 - [68]
 

I was using buy/maximum and sell/mininum limited by region (The Forge obviously) from EM. Either I dumb or there is no such possibility with eve-marketdata. "best guess" probably is good for guessing price at killboards, but when you need to decide what to produce/buy/sell - you need more. (tritanium - 3.95?.. definitely need something more precise)
Probably I could take it from historic feeds, but that's a lot of data instead of two simple values.
Can we have simple maxBuy and minSell taken from most recent upload with region as parameter, please? :)

Grimdahl
Posted - 2011.01.09 09:53:00 - [69]
 

Originally by: Kallahar
... I think that 99% of the people using it are only getting the average price, so that they can update their killboard, etc.



Well -- Evemeep and all it's users are the 1% left it seems.
If there are really this less people using this extra data -- why not provide a specific URL exactly for this use?
(Evemeep always requests only one item combined with region)


otto M8
Posted - 2011.01.09 20:07:00 - [70]
 

Originally by: Droxlyn
I'm mostly looking for the 5% market volume buy effective buy price and the 5% market volume sell price, ideally for a specific system. (IE Jita or Amarr)

That way I can look up my costs for input materials and the profit for the easy sell. I sometimes average the two together to get the middle buy/sell price that should go fast but make me or cost me just a little.

Averages and Medians are not as interesting to me. (Nor are the extremes.)

Drox


First, a tip of the hat and a thanks for taking on this project. I agree with Droxlyn (as a user who got used to EMs simulated prices) that a price on some volume of significance is valuable. A 5% of volume is a great comprise between too little and too much processing/calculation.

Second, as a follow up, what would be even more useful is a standard set of statistics such as you had up briefly. I am most interested in pulling buy/sell prices from one region for quantities that are relevent for me to compare with the corresponding buy/sell prices in other regions.

When faced with the demise EM I started working towards pulling minimum sell, maximum buy by regions for items of interest to me limited by quantities that are useful to me in decision making. For instance, a minimum price is not useful if someone is trying to quickly unload a handful of an item dropped as mission loot if I need 100k units of it. The price I need to pay for those is of much great interest. So I might ballpark a minimum price for 50k units on that item in a specific region.

With EM's simulated price at 5% of the volume on market I got lazy and found that useful enough.

I guess I'm also a little unclear about what the "best guess" price actually means. In a killboard context it is probably not that critical, in an industrial context it is. I'm assuming it is a trimmed/filtered min sell, max buy, but please correct my assumptions if they are incorrect.

Cheers

Ten Bulls
Minmatar
Posted - 2011.01.09 23:20:00 - [71]
 

Edited by: Ten Bulls on 10/01/2011 03:42:15
Edited by: Ten Bulls on 09/01/2011 23:23:21
He, site and tool looks promising, good stuff.

I just tried it out, i had some issues, ingame it was checking a lot more prices than where being uploaded from the tool, i stopped getting prices ingame and the uploaded continued uploading the backlog of prices for quite some time, however it stopped without uploading all of them. i.e. there are still files in the cache folder that ts not trying to upload.

I tried enabling verbose output but then the uploader "fails to respond" as windows puts it.

Actually, i did have a torrent going in the background, maybe that was slowing down my uploads and causing problems...

EDIT: verbose mode works after restarting.

vonDuck
Gallente
Knights Templar ordre de eve brethren mmx
Black Thorne Alliance
Posted - 2011.01.10 15:05:00 - [72]
 

Something I do is eliminate anything outside of two standard deviations to try to cull outliers from screwing up the prices. You do this by region to ensure you don't get a universe-wide minimum or maximum. You can apply this to volume as well as price, too.

Another thing I do is create a hub price in addition to the region price. I find the system with the most inventory in the market and call that an "item hub". This isn't always the traditional hub as concentrations of drops might occur nearer to the mission system as people try to dump stock and get to the next mission.

Another edit is to ensure the min transaction volume is either 1 or the batch quantity of the item. This avoids the volume market scheme where you buy a bunch of stuff for a great trade just to realize you must have a min volume that is equal to all of the inventory in the region.

Fovo
Timetravel Enterprises
Posted - 2011.01.10 22:47:00 - [73]
 

One thought for those interested in a specific result set.
If you are capable of writing the sql query for that specific result and willing to contribute, consider providing it with your request. If I were in Kallahar's shoes, that would make things for me easier..


But then again, Kallahar should probably give this a thumbs up first.

Fovo

Kari Trace
Caldari
Nox Imperium
Posted - 2011.01.11 04:04:00 - [74]
 

Awesome! Plan on my uploading often!

Luminocity
Posted - 2011.01.11 09:41:00 - [75]
 

This..
Originally by: otto M8
First, a tip of the hat and a thanks for taking on this project.


..and this
Originally by: Dharken Rhaal
I was using buy/maximum and sell/mininum limited by region (The Forge obviously) from EM. Either I dumb or there is no such possibility with eve-marketdata. "best guess" probably is good for guessing price at killboards, but when you need to decide what to produce/buy/sell - you need more. (tritanium - 3.95?.. definitely need something more precise)
Probably I could take it from historic feeds, but that's a lot of data instead of two simple values.
Can we have simple maxBuy and minSell taken from most recent upload with region as parameter, please? :)


Serene Python
Posted - 2011.01.11 10:59:00 - [76]
 

Originally by: Kallahar
**** HISTORY ADDED ****
Much requested, I finally got history added. Here's the quirk, people running the uploader have to have it on the price history tab in order for it to upload, so I've got all of The Forge done, but other regions will take a while to get data.

see http://eve-marketdata.com/developers/item_history.php for info, note that it's *not* eve-marketdata compatible because I forgot to save their formatting before they closed... heh.

Kallahar



Nice to see that movement finally made it in. Thanks

randomToon987
Posted - 2011.01.11 13:20:00 - [77]
 

Originally by: Kallahar

...snip...

I've also decided to discontinue the item.xml/json api. I think that 99% of the people using it are only getting the average price, so that they can update their killboard, etc. However it provides far more info, which is CPU intensive. Therefore, I'm going to kill off that program, and people will have to update their code to use one of these new API's:
- http://eve-marketdata.com/developers/item.php for one item lookup
- http://eve-marketdata.com/developers/items.php for multiple

I do plan on adding statistical data back, but it will be with a new URL. This way, people who only need the price won't be wasting CPU or bandwidth getting a bunch of extra data they don't need.

Kallahar



I'm having high hopes for this service, but removing the most comprehensive part of the api makes it useless for me, and many others.
We might not be the biggest group, but we're developers too, and when we write special solutions for the public or our corp to use we always mention the source of our market data, so a lot of people learn about sites such as eve-metrics(R.I.P.) or eve-central through third party tools.
Wouldn't caching extra statistical data be a better solution than removing it altogether?

On a somewhat related note - I'm sure that designing your own api gives you more freedom, and that's ok, but you should analyse eve-metrics api, and try to learn something from the way it was designed. For example - you're not using api version number, which may result in a lot of applications relaying on your site breaking without warning when you decide to change something, using api version number is an easy way to make sure that the data you provide and the application that processes it will work together just fine, and if the version numbers differ it's easy to shut down gracefully.

Hel O'Ween
Men On A Mission
EVE Trade Consortium
Posted - 2011.01.11 14:41:00 - [78]
 

Originally by: Kallahar

I've also decided to discontinue the item.xml/json api. I think that 99% of the people using it are only getting the average price, so that they can update their killboard, etc. However it provides far more info, which is CPU intensive. Therefore, I'm going to kill off that program, and people will have to update their code to use one of these new API's:
- http://eve-marketdata.com/developers/item.php for one item lookup
- http://eve-marketdata.com/developers/items.php for multiple



Not sure why you did that.

1) Two different URLs for the same purpose? This way you have to keep two scripts up tp date. Besides, you're already doing the parameter parsing with the second one (multiple items). The single item (single value) is just a special case of multiple items.

2) In terms of (output) format: is that the final result we're getting there already? Because that seems to be some kind of home-brewn result format. Are you planning to return something more standard (like the good old XML)?

Jensius Duo
Zor Industries
Pandemic Legion
Posted - 2011.01.11 17:48:00 - [79]
 

Great honourable effort!

Any chance to have a csv/xml with all the latest price data for every item, in order to be able to look up prices quickly?
Eve-Metrics had a latest-basic.csv of ~600kB:
type_id,buy_min,buy_max,buy_med,buy_avg,sell_min,sell_max,sell_med,sell_avg,buy_simulated,sell_simulated

Azazel Mordred
Minmatar
Cloak of Shadows
Posted - 2011.01.11 18:01:00 - [80]
 

I'd really like to implement this in a project I work on, but as others have mentioned, the missing min and max sell/buy values (per region and otherwise) is unfortunately a deal breaker for me Crying or Very sad

Kallahar
Posted - 2011.01.11 22:00:00 - [81]
 

Thanks everyone!

1) Added a new API method - item_orders - it gives you all the orders for a region, so it should be easy for those of you who want the statistics to grab that and calculate it yourself. If it's not adequate, let me know!
2) Upgrading the database server tonight, so everything should be much faster.
3) SQL queries - YES! If you give me a query it's much easier for me to implement it. See the developers tab under mysql table definitions for the DB layout.
4) XML/JSON formats: if you find any specific errors, let me know. Personally I don't use them. Keep in mind that I can no longer access the eve-metrics API, so I'm not sure exactly how they did it.

Kallahar

vonDuck
Gallente
Knights Templar ordre de eve brethren mmx
Black Thorne Alliance
Posted - 2011.01.12 05:05:00 - [82]
 

If anybody can explain what they want, I can write the query (or explain why it can't be done Wink ).

Luminocity
Posted - 2011.01.12 07:48:00 - [83]
 

This..
Originally by: Hel O'Ween
1) Two different URLs for the same purpose? This way you have to keep two scripts up tp date. Besides, you're already doing the parameter parsing with the second one (multiple items). The single item (single value) is just a special case of multiple items.

Just remove "item price" as "item prices" already accepts both single and multiple values (comma delimited) as the typeID parameter..

Also, good work on the new sell orders API, it's a nice addition..
Now if we could have something in between of the "item prices" (which is ideal for killboards due to being very simple and general) and the new "sell orders" (which is probably good for more in-depth analysis by being quite detailed).
Something which would return the latest lowest sell order and highest buy order prices for a specific region. (please see this feature from eve-central as an example)

Azazel Mordred
Minmatar
Cloak of Shadows
Posted - 2011.01.12 07:58:00 - [84]
 

This would be helpful if the DB can manage it... Perhaps it would be useful to archive expired orders in the buying/selling tables after a week or two, to keep things a bit faster.


select
min(b.price) as buy_min, max(b.price) as buy_max, avg(b.price) as buy_avg,
min(s.price) as sell_min, max(s.price) as sell_max, avg(s.price) as sell_avg
from eve_inv_types t
left join items_buying b
on b.type_id = t.type_id and b.region_id = ? and b.date_expires > current_timestamp
left join items_selling s
on s.type_id = t.type_id and s.region_id = ? and s.date_expires > current_timestamp
where
t.type_id = ?

Luminocity
Posted - 2011.01.12 08:16:00 - [85]
 

Edited by: Luminocity on 12/01/2011 08:18:00
Originally by: Azazel Mordred
This would be helpful if the DB can manage it... Perhaps it would be useful to archive expired orders in the buying/selling tables after a week or two, to keep things a bit faster.


select
t.type_id,
min(b.price) as buy_min, max(b.price) as buy_max, avg(b.price) as buy_avg,
min(s.price) as sell_min, max(s.price) as sell_max, avg(s.price) as sell_avg
from eve_inv_types t
left join items_buying b
on b.type_id = t.type_id and b.region_id = ? and b.date_expires > current_timestamp
left join items_selling s
on s.type_id = t.type_id and s.region_id = ? and s.date_expires > current_timestamp
where
t.type_id in (?)


Quick on the draw there Azazel, nice work :)
This is pretty much exactly what I had in mind. It might be good to have it accept multiple typeID values (I modified the code above) as the current "items price" method does (comma delimited) so there would be less calls to the method (though potentially more work done internally for each call). Maybe have it limited to a specific amount of accepted typeID's per each call?

Also API versioning is a good idea, I think. Especially as there are probably a lot of modifications in the early days of this project..
Originally by: randomToon987
On a somewhat related note - I'm sure that designing your own api gives you more freedom, and that's ok, but you should analyse eve-metrics api, and try to learn something from the way it was designed. For example - you're not using api version number, which may result in a lot of applications relaying on your site breaking without warning when you decide to change something, using api version number is an easy way to make sure that the data you provide and the application that processes it will work together just fine, and if the version numbers differ it's easy to shut down gracefully.

Luminocity
Posted - 2011.01.12 10:17:00 - [86]
 

On another note. It looks like your methods currently don't accept HTTP POST requests (which I would prefer for multiple reasons).
I don't know about the late eve-metrics but eve-central accepts both POST and GET requests, which would be nice in this case aswell..

Hel O'Ween
Men On A Mission
EVE Trade Consortium
Posted - 2011.01.12 12:18:00 - [87]
 

Originally by: Kallahar

4) XML/JSON formats: if you find any specific errors, let me know. Personally I don't use them. Keep in mind that I can no longer access the eve-metrics API, so I'm not sure exactly how they did it.



First, the announced items.php doesn't seem to exist: http://eve-marketdata.com/api/items.php?type_ids=34 results in a 404.

Whereas http://eve-marketdata.com/api/item.php?type_ids=34 works, but returns a format I'm nor familiar with:

\ntype_id\tbuy\tmin\tmax\tmedian\taverage\tstd_dev\[...]
34
buy 0.01 3.4 2.4 1.86218285147515 42462764.62665
sell 0.01 3.4 2 1.86218285147515 0.45284745607815


Content type of the above: text/plain

Ah, I see, there's item_prices.xml which just holds a "price"

Now I'm really confused. Why can't you return the same amount of information in the XML (I guess JSON has the same issue) like you do in the item.php?

I'd really appretiate your efforts and I'd love to add your API to EWA, but I guess I'll wait until the "dust has settled" with the API stuff and a stable solution/format has been established.

Kallahar
Posted - 2011.01.12 17:35:00 - [88]
 

Thanks everyone!

1) All API's now accept POST or GET
2) /api/item.* has been removed, it was too CPU intensive for what most people needed. Click on "developers" to see the other API options.
3) Expired orders are now removed within 10 minutes, or sooner if someone checks the market for that item.

Kallahar

Colgir
Posted - 2011.01.12 18:00:00 - [89]
 

Hello,

at first, to make this tool is a great idea and i will upload as much data as i can.
But i have found a little Problem: There is a whole region missing. :)

The errormsg: Error: region not found for region_name = Derelik


Fovo
Timetravel Enterprises
Posted - 2011.01.12 20:07:00 - [90]
 

If anyone has a chance and would be so kind, I posted a related question on the forums here.

I need to create a very specific xml format for an application i would like to use. I will be combining data feeds from eve-marketdata and the eve-api and want to create one page that can run the specific query on the items requested.

Thanks

Fovo


Pages: first : previous : 1 2 [3] 4 5 6 7 8 9 ... : last (10)

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only