open All Channels
seplocked EVE Technology Lab
blankseplocked Eve-MarketData.com - replacement for eve-metrics
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: first : previous : 1 [2] 3 4 5 6 7 8 9 ... : last (10)

Author Topic

Wyke Mossari
Gallente
Posted - 2010.12.28 20:47:00 - [31]
 

Originally by: Kallahar
I got api/item.xml and api/item.json working, but I'm not sure how to calculate the following:

- median
- kurtosis
- skew
- variance
- standard_deviation
- simulated

If anyone knows how to calculate these in mysql/php, please let me know and I'll add them in.

Kallahar




Droxlyn
Caldari
Posted - 2010.12.28 22:53:00 - [32]
 

The mimetype for the item.xml export needs to be application/xml or else GoogleDoc's ImportXML function fails it seems.

I'm sure the JSON should get similar love.

Thanks,
Drox

Sinuous Grace
Posted - 2010.12.29 18:09:00 - [33]
 

Kallahar,

Is there something us "non coders" can do to help this project? I am an IT director with access to massive computer resources.

Karn Velora
Posted - 2010.12.29 20:13:00 - [34]
 

I find EVE Metrics just as it's shutting down. Definitely a "Doh!"-moment. Luckily I also checked the threads and ended up here. A big thank you is in order, for deciding to take over the service now that metrics are shutting down.

HighlanderUK
Interstellar Business Machines Corp.
Posted - 2010.12.30 02:00:00 - [35]
 

Kallahar, well done on taking this on, as i feel we really do need an EM type service for all us traders/manufacturers. Many thanks.

Tell us what you need, and make it a truly versatile resource. I have the uploader working and done several 1000 prices, but when searching the site's market, all i get is ERROR:Search Term Is Required. Thought it was due to price not being uploaded yet, but checked and done an upload of some ships and no joy.

Any ideas?

Shingyoku
Minmatar
Griffin Capsuleers
Ad-Astra
Posted - 2010.12.30 05:40:00 - [36]
 

Edited by: Shingyoku on 30/12/2010 05:58:29
Any chance you might make an API out of this like eve central has and eve metrics had? if so I'd have to start getting at the battleclinic guys to switch to your API in evemon and eveHQ since that autouploader will definitely put you over the remaining competition.

Edit, oh and please add some sort of system to compete like eve central has when you can login and eve metrics has no matter what. I'd love to see what my accounts are churning out compared to everyone else :D

Hel O'Ween
Men On A Mission
EVE Trade Consortium
Posted - 2010.12.30 10:57:00 - [37]
 

Originally by: Shingyoku

Any chance you might make an API out of this like eve central has and eve metrics had?


He has already, that's what we're talking about with that MIME type stuff he needs to correct in the previous postings.

In theory it is similar to EM's API (so you only have to change the domain name for your query, i.e. Tritanium would be http://eve-marketdata.com/api/item.xml?type_ids=34.

Matthias Matici
Posted - 2010.12.30 11:07:00 - [38]
 

Kallahar,

Any chance you can make this open source? That way any coders of us out here can contribute and give this project a big boost.

HighlanderUK
Interstellar Business Machines Corp.
Posted - 2010.12.30 19:13:00 - [39]
 

Originally by: Shingyoku
Edit, oh and please add some sort of system to compete like eve central has when you can login and eve metrics has no matter what. I'd love to see what my accounts are churning out compared to everyone else :D


yep forgot about this, EM had this spot on, even if i didn't know what my credits were for :) - don't have it like EC now, and lock the leader-board.

vonDuck
Gallente
Knights Templar ordre de eve brethren mmx
Black Thorne Alliance
Posted - 2010.12.30 21:15:00 - [40]
 

Median is the middle value, not the average (or "mean"). You can calculate this via finding the middle value (sort and find the middle) or by taking a weighted mean.

I have just converted my eve-metrics fed database to eve-central (sadly) and I would be EXTREMELY HAPPY to see this project succeed. :D

If you point out which statistics you are still working on I can give you code to calculate them in SQL. If you want some other statistics I can provide those, too. I also have some filters that try to get rid of market scams from the data so that you don't have over-inflated prices that don't reflect reality. I also have a filter that isolates the solar system with the most quantity in the market (sort of like determining the market hub rather than assuming one for lower-traffic regions).

A little about myself: I'm an information management and analysis consultant and have been a technology/information consultant for over 25 years. Use and abuse me, my hobby is my job!

Ix Forres
Caldari
Righteous Chaps
Posted - 2010.12.31 01:09:00 - [41]
 

Originally by: Wyke Mossari
Originally by: Kallahar
I got api/item.xml and api/item.json working, but I'm not sure how to calculate the following:

- median
- kurtosis
- skew
- variance
- standard_deviation
- simulated

If anyone knows how to calculate these in mysql/php, please let me know and I'll add them in.

Kallahar





Truth be told we kinda cheated here; we use the Gnu Scientific Library for most of the numbercrunching in EM, which then give you those statistical methods for free as an added bonus to being insanely fast. Here's the code that makes those stats in Ruby; converting it should be trivial... (disclaimer: the code may be full of bugs, and certainly has some shoddy error handling in it. Provided as-is, etc, yada-yada, don't blame me if it eats your children, but it's literally copy-pasted from a production copy of EM) https://gist.github.com/d706eb44855baa45f4ee

Ix Forres
Caldari
Righteous Chaps
Posted - 2010.12.31 01:16:00 - [42]
 

Edited by: Ix Forres on 31/12/2010 01:18:31
Originally by: HighlanderUK
Originally by: Shingyoku
Edit, oh and please add some sort of system to compete like eve central has when you can login and eve metrics has no matter what. I'd love to see what my accounts are churning out compared to everyone else :D


yep forgot about this, EM had this spot on, even if i didn't know what my credits were for :) - don't have it like EC now, and lock the leader-board.


Obviously this used to be Magic and we kept it that way to stop people gaming the system, but what the heck...

Where 'statistics' represents the calculated statistics for that upload:

def calculate_credits(statistics, res, now)
last_upload = Time.parse(res['created_at'])
factor = (statistics[:inserted_orders] + statistics[:updated_orders] + statistics[:expired_orders]+ statistics[:sold_out_orders])/statistics[:active_orders].to_f
[(100*Math::E**(5.0*factor - 5) + (now-last_upload)/6048.0).ceil, 100].min
end


Makurid came up with this after a few versions and lots of pondering and plotting of graphs like this with some sample uploads from before when we added proper credit calcs: http://assets.talkunafraid.co.uk/2009/09/uploader_reward_metrics.png

Specifically the aim was to reward informationally dense uploads while also rewarding uploads that were temporally informative - stuff from areas where we'd not seen anything for a while, for example. The reward system promoted distribution of uploads from across the map rather than focusing exclusively on the trade hubs, which obviously produce good informationally dense uploads in terms of the amount of churn in a given upload dataset but which are for obvious reasons typically uploaded more frequently than regions in the middle of nowhere.

Fovo
Timetravel Enterprises
Posted - 2011.01.02 23:00:00 - [43]
 

ok perhaps this is a silly request. However I am going to make it anyhow as I am probably not the first person to think it. (Just the first brave or foolish enough to ask)

I have a server setup currently and it is utilizing the POS-Tracker 3 package. This being said, I already have a running server and mysql database.

I spend a lot of time in mysql databases, and do reasonably well with it. My challenge is converting that http post to an insert/update command for my DB.
I am not a programmer, nor do I know PHP to any extent other than what google teaches me.

Would it be possible for you to provide an example page that we could use to receive your http_post(s) with. I understand i will have to create the database and tables, add the connection credentials to the source code of the page you provide. This is just to help individuates like me spend less time googling "How to insert http Post data into MYSQL) and more time building my eve empire.

Thanks

Fovo

Fovo
Timetravel Enterprises
Posted - 2011.01.02 23:19:00 - [44]
 

Another idea to consider is looking at creating a public version of a google spreadsheet similar to the one below. This used eve-central metrics, but could be easily converted to eve-marketdata.

Live EVE Pricing Example

Fovo

Fovo
Timetravel Enterprises
Posted - 2011.01.03 21:39:00 - [45]
 

Edited by: Fovo on 06/01/2011 14:38:37
I have a testing version of a php page for collecting the http post data. However I am having issues registering for an account with the IGB. I have added entries for both http://www.eve-marketdata.com/* and http://eve-marketdata.com/* to the trusted site on my in-game browser however it is still telling me I need to "Trust this site".

Anyone with an idea, give me a shout. Otherwise I will keep banging my head on this wall till something gives...

Fovo


** UPDATE **

Looks like I had to add that specific page to the trusted sites (http://www.eve-marketdata.com/update_market.php) and not just (http://www.eve-marketdata.com/*)


Kallahar
Posted - 2011.01.05 19:28:00 - [46]
 

Back from vacation!

Content-type's: fixed, now return as application/xml and application/json. Regarding formatting, the goal is to have it the exact same as eve-metrics for easy porting for people.

Open Source: I don't want to manage a project, that's why I have the "live feed" option, you can run your own project using the same data I have. Ideally CCP would just open the market "live feed" to the API, but they have their own reasons for not doing that.

Historical data: yes, working on it today

IGB problems: can you email me or contact me in game (kallahar) with everything about your char/location/etc? Some pages check the char_id, some check the region_id from the IGB headers, so it's possible that in some cases they don't send them correctly... if I can reproduce the problem, I can fix it.

What's the most important API calls people want completed first?

Kallahar

Grimdahl
Posted - 2011.01.05 19:39:00 - [47]
 

Edited by: Grimdahl on 05/01/2011 19:39:29
Originally by: Kallahar

What's the most important API calls people want completed first?



No idea -- but Evemeep needs:
1.) item data per region (query: api/item.xml?type_ids=xx®ion_ids=yy)
2.) historical data

and welcome back :-)

James Bryant
Posted - 2011.01.05 20:41:00 - [48]
 

Edited by: James Bryant on 05/01/2011 20:42:48
Kallahar,

I applaud your efforts here. As a former dev of Eve Market Reports (if anyone remembers it, you'll remember it didn't last long), I'll throw out there some of the lessons I learned.

1.
Very, very quickly, you're going to end up with a truly impressive amount of data. Many Gb, in fact, and most likely more or less in a single or very few tables. Figuring out the most efficient queries to select this data is paramount. Any small scaling problems will very quickly become huge scaling problems. Utilize db indices to their fullest extent and build your schema very carefully.

2.
When you start getting numbers for your historical data, you will be needing to do a lot of math and queries per page. You really need a messaging and task system to automate creating tables of statistical and historical calculations (avg,sum,median,etc) on a daily or even hourly basis. There is no way to sustain doing all these calcs on a per page load basis (it will work in the beginning, but not next month). Cron is not ideal here, but if that's all you have, it can sorta work for a while, although I think that's what evemetrics 1.0 was doing and you all remember how slow that site could be.

3.
Build on a framework. A framework gives you a caching system, which will be a necessity. It also can handle user registration, logins, models, views, controllers, etc. By no means should you not use a caching system of some kind, at the very least. I've been using Yii, and I like it a lot, although it is very young and the documentation can be a bit spotty. Zend is good also.

4.
You won't be able to last on shared hosting for very long, if that's what you are on now. This type of site does a lot of heavy lifting, both in the foreground and in the background. Utilize AJAX as much as possible for slow data fetches that will unnecessarily delay page loads. Prepare for the time where a dedicated server will become necessary (it will become necessary).

I know this sounds like a lot of ballyhooing, but these are things to be aware of and are challenges that will arise faster than you expect. I've been there already. There is no shame in scaling functionality back from evemetrics. 1.0 was written without an awareness of these challenges, and needed to be rewritten from the ground up for 2.0, and was done so properly having encountered those problems with those strategies mentioned above.

Eve Market Reports failed ultimately because of a lack of time and a lack of preparation for the high software and hardware requirements of the project.

Kallahar
Posted - 2011.01.05 22:12:00 - [49]
 

Grimdahl - added regions, thanks, will continue working on history

James - thanks for the tips, I'm sure I'll be learning a lot as I go too :)

Kallahar

Serene Python
Posted - 2011.01.06 01:30:00 - [50]
 

Originally by: Kallahar

What's the most important API calls people want completed first?



I'd really like to see movement of an item in a region

Hel O'Ween
Men On A Mission
EVE Trade Consortium
Posted - 2011.01.06 11:42:00 - [51]
 

Originally by: Kallahar

Content-type's: fixed, now return as application/xml and application/json. Regarding formatting, the goal is to have it the exact same as eve-metrics for easy porting for people.



Thank you very much.

Another question: is there a limit for the number of items we can query with one call we need to be aware of? I.e. max 10/100/1000 different typeIDs in one query?

Fovo
Timetravel Enterprises
Posted - 2011.01.06 14:36:00 - [52]
 

Edited by: Fovo on 06/01/2011 14:40:00
First of all, Thank you for doing a great job Kallahar!

Now as a follow up to an earlier request I made, I have built a simple php page that collects and stores the data from the feeds to a mysql database.
Kallahar, if you would like to place a copy of it on your site for people to download freely, please let me know and I would be happy to provide it to you. Otherwise, you will have to evemail me and I will get it to you.. Perhaps if i looked into eve-files I could place it there, but on the developer page of eve-marketdata would seem to be the logical place for it..

Anyway.. THANK YOU .. THANK YOU .. THANK YOU .. THANK YOU .. THANK YOU

Great JOB!!!!

Fovo
Timetravel Enterprises
Posted - 2011.01.06 16:29:00 - [53]
 

FYI, The client source download link is broken.

Kallahar
Posted - 2011.01.06 19:14:00 - [54]
 

Features added:

On the item search page: Buy orders and limiting by region

Re # of type_ids - theoretically you could grab them all, let me know if you run into performance problems.

Re source code - fixed, thanks, and example code for the uploads posted.

Kallahar

Miko Tyrane
Posted - 2011.01.06 21:40:00 - [55]
 

Love the site! Couple of bugs and a feature request tho.

In the item.json, type_name & regions are both being sent back with no data, is this intended???

It doesn't look like there is any difference in the numbers for the sell and buy data that gets returned... I was running some tests with different item_ids and The Forge region_id and all of numbers were coming back the same between the sell and buy data for each item_id... Individual id requests produced the same result. Here's a sample of the output:

[{"type_name":"","type_id":"30374","global":{"sell":{"avg":215.885535028185,"min":11.13,"max":307.64,"median":307.18,"standard_deviation":124.57213077392}"buy":{"avg":215.885535028185,"min":11.13,"max":307.64,"median":307.18,"standard_deviation":124.57213077392},"last_upload":"2011-01-06T13:10:21-08:00","type_id":30374,"type_name":""},"regions":[]}]

See how the sell avg is the same as the buy avg, etc?



Would it be possible to add quantities for the max and min values to the data returned from the json? My thinking is that it is only really important on the max and min values because those are the important metrics for volume of sales. It could potentially be useful to have the total volume in each category too I guess. So that you know if an item is saturated in a particular market or in high demand. I'm thinking an output something like this:

[{"type_name":"","type_id":"30370","global":{"sell":{"avg":1347.34618869507,"min":23.07@200,"max":7200.01@11938,"median":6000,"standard_deviation":2957.7200982709,"total_volume":212304}"buy":{"avg":1347.34618869507,"min":23.07@200,"max":7200.01@11938,"median":6000,"standard_deviation":2957.7200982709,"total_volume":212304},"last_upload":"2011-01-06T13:17:20-08:00","type_id":30370,"type_name":""},"regions":[]}]

(note that I didn't change the values between the sell and buy data points simply because I just wanted the format to be consistent in the example)

Really appreciate all the work you're putting into this :)

Miko Tyrane
Posted - 2011.01.06 21:44:00 - [56]
 

Something else I just thought of, what about getting the data returned from the api for a specific system id? Like say I was only wanting to see the results of sell and buy orders in Jita?

Kallahar
Posted - 2011.01.06 21:57:00 - [57]
 

buy/sell being the same price: fixed, was a typo, heh
for item.xml/json I was keeping the same format at eve-metrics used, but I'm working on new API features which will have a new url.

Kallahar
Posted - 2011.01.06 23:57:00 - [58]
 

**** HISTORY ADDED ****
Much requested, I finally got history added. Here's the quirk, people running the uploader have to have it on the price history tab in order for it to upload, so I've got all of The Forge done, but other regions will take a while to get data.

see http://eve-marketdata.com/developers/item_history.php for info, note that it's *not* eve-marketdata compatible because I forgot to save their formatting before they closed... heh.

Kallahar

Haleigh Agius
Minmatar
Rubicon Legion
Huzzah Federation
Posted - 2011.01.07 06:29:00 - [59]
 

Edited by: Haleigh Agius on 07/01/2011 06:30:18
Is it intended that the uploader checks the same unavailable items over and over (up to 3 of 5 queries) instead of going through the whole list first? It would be nice if there was an option to prevent it from retrying (or limit to 1 retry), especially when scanning regions without trade hubs.

Victuri
Posted - 2011.01.07 10:03:00 - [60]
 

The Auto uploader is fantastic, would there ever be a way we could create custom lists to run ourselves?


Pages: first : previous : 1 [2] 3 4 5 6 7 8 9 ... : last (10)

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only