open All Channels
seplocked EVE Technology Lab
blankseplocked Eve-MarketData.com - replacement for eve-metrics
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: [1] 2 3 4 5 6 7 8 9 ... : last (10)

Author Topic

Kallahar
Posted - 2010.12.18 00:43:00 - [1]
 

I use eve-metrics nearly every day to check prices in other regions. I couldn't stand to see it die, so I've created a similar site.

http://eve-marketdata.com

It's clearly an homage to eve-metrics. Right now it's got the uploader client (please upload!), a market browser, and it shows all the sellers for each item. I can add in the market trend analysis if needed, but I think most people use eve-metrics for finding out the "correct" price for an item.

It also provides a "live" feed (not AMQP, but HTTP post), and (soon) will have nightly data dumps. My goal is to let people transition easily from eve-metrics to eve-marketdata.

The buggiest part currently is the uploader, if anyone would like to write a better one, please do. The current code is under "developers".

- Kallahar

Kallahar
Posted - 2010.12.18 01:08:00 - [2]
 

oh, and I forgot a HUGE thank you to eve-metrics for providing such a quality service for so many years. You'll be missed!

Kallahar

Serene Python
Posted - 2010.12.18 14:46:00 - [3]
 

Edited by: Serene Python on 18/12/2010 16:24:50
Not sure if I'm the only one no this boat, but I use EM for historical data... Think you implement things like that?

p.s.
I mainly used it for the movement data which, not sure how you'd implement this yourself but that's a big feature for me as well as the API for developing my own apps

If you guys can't figure out how to do the movement just let me know and I'll figure it out during my free time, which there isn't a lot of though I don't mind coding during it

Elaiam Sendara
Posted - 2010.12.18 15:17:00 - [4]
 

Looks like you have a good start. I hope the community gets behind this project.

Serene Python
Posted - 2010.12.18 16:33:00 - [5]
 

Edited by: Serene Python on 18/12/2010 16:45:50
Edited by: Serene Python on 18/12/2010 16:43:15
Bugs!
When I try to search something I'm being sent to pricecheck.php which results in a 404. Changed the form action value and tried again and got "Error: type_id is required".

Just out of curiosity... Did you code this all on a single line? Cause when I view page source in firefox and notepad++ it's a single line

For suggested uploads you should make it so we can view it from any browser because some people can't use the IGB(like me) as well as I wouldn't mind seeing what needs to be uploaded while I'm just messing about with other things in firefox/chrome

P3k1
Posted - 2010.12.20 00:02:00 - [6]
 

I'm interested in helping out if you need help on a project...

Haven't looked closely at uploader code, but i saw it was written in Python.

Thibault Etienne
Posted - 2010.12.20 21:57:00 - [7]
 

Edited by: Thibault Etienne on 20/12/2010 21:59:42
I cant code or anything But ill leave my ships idling overnight and upload the markets. Least I can do. Good luck with this mate.
Buyers lists would be nice tooVery Happy

Kallahar
Posted - 2010.12.20 21:58:00 - [8]
 

Ah good find, search bug fixed. Thanks!

On the html-on-one-line, I code everything in PHP, so all outputted HTML is inside a print() statement, so there are no newlines. The back-end php code is very well organized, formatted, and commented :)

Kallahar

Serene Python
Posted - 2010.12.20 23:49:00 - [9]
 

Originally by: Kallahar
Ah good find, search bug fixed. Thanks!

On the html-on-one-line, I code everything in PHP, so all outputted HTML is inside a print() statement, so there are no newlines. The back-end php code is very well organized, formatted, and commented :)

Kallahar



Ahh yes PHP coding... I've often had people bug me about my HTML code looking very horrid. You working on having some form of movement being implemented?

Johnathan Roark
Caldari
The Graduates
Morsus Mihi
Posted - 2010.12.21 05:28:00 - [10]
 

Originally by: Serene Python
Originally by: Kallahar
Ah good find, search bug fixed. Thanks!

On the html-on-one-line, I code everything in PHP, so all outputted HTML is inside a print() statement, so there are no newlines. The back-end php code is very well organized, formatted, and commented :)

Kallahar



Ahh yes PHP coding... I've often had people bug me about my HTML code looking very horrid. You working on having some form of movement being implemented?


I use dwoo, makes the php and html clean, plus it makes it easier to fix the display stuff.

Originally by: Kallahar
I use eve-metrics nearly every day to check prices in other regions. I couldn't stand to see it die, so I've created a similar site.

http://eve-marketdata.com

It's clearly an homage to eve-metrics. Right now it's got the uploader client (please upload!), a market browser, and it shows all the sellers for each item. I can add in the market trend analysis if needed, but I think most people use eve-metrics for finding out the "correct" price for an item.

It also provides a "live" feed (not AMQP, but HTTP post), and (soon) will have nightly data dumps. My goal is to let people transition easily from eve-metrics to eve-marketdata.

The buggiest part currently is the uploader, if anyone would like to write a better one, please do. The current code is under "developers".

- Kallahar


I commend you for taking this on, but my concern is what happens when you get tired of eve, the site, or the it gets too expensive? I would like to see a team maintain and write what ever replaces eve-metrics. I would even be interested in helping with such a project.

Serene Python
Posted - 2010.12.21 08:52:00 - [11]
 

Originally by: Johnathan Roark
Originally by: Serene Python
Originally by: Kallahar
Ah good find, search bug fixed. Thanks!

On the html-on-one-line, I code everything in PHP, so all outputted HTML is inside a print() statement, so there are no newlines. The back-end php code is very well organized, formatted, and commented :)

Kallahar



Ahh yes PHP coding... I've often had people bug me about my HTML code looking very horrid. You working on having some form of movement being implemented?


I use dwoo, makes the php and html clean, plus it makes it easier to fix the display stuff


Never heard of dwoo... I just use notepad++, my PHP code itself it typically good looking just if I got HTML it's very horrid looking typically. I prefer just typing everything rather than letting some program handle everything... Biggest pet peeve in school when we learned HTML... Final project had probably almost 1MB in useless crap being loaded/added to the code.

Hel O'Ween
Men On A Mission
EVE Trade Consortium
Posted - 2010.12.21 11:22:00 - [12]
 

I seem to be missing the feature that made EC and EM so popular: their own API for integration into other apps/services.

Are you planning on adding something alike?

Johnathan Roark
Caldari
The Graduates
Morsus Mihi
Posted - 2010.12.21 18:59:00 - [13]
 

Originally by: Serene Python
Originally by: Johnathan Roark
Originally by: Serene Python
Originally by: Kallahar
Ah good find, search bug fixed. Thanks!

On the html-on-one-line, I code everything in PHP, so all outputted HTML is inside a print() statement, so there are no newlines. The back-end php code is very well organized, formatted, and commented :)

Kallahar



Ahh yes PHP coding... I've often had people bug me about my HTML code looking very horrid. You working on having some form of movement being implemented?


I use dwoo, makes the php and html clean, plus it makes it easier to fix the display stuff


Never heard of dwoo... I just use notepad++, my PHP code itself it typically good looking just if I got HTML it's very horrid looking typically. I prefer just typing everything rather than letting some program handle everything... Biggest pet peeve in school when we learned HTML... Final project had probably almost 1MB in useless crap being loaded/added to the code.


Dwoo isn't a program, its a library used to separate HTML code from PHP code. Another library that does something similar is Smarty, but its a bit more bloated.

As far as programs, I write my PHP using a program called PHPDesigner for syntax highlighting and some suggestive words. Also use Notepad++ for somethings.

Project 69
Posted - 2010.12.22 13:24:00 - [14]
 

alot of my php-scrits were relying on eve-metrics price checks.

so if you need any help apart from hosting, please contact me

GraphisoDavinci
Posted - 2010.12.22 15:32:00 - [15]
 

Man i cant thank you enough! Hope this will stay!

Btw i think the primary goal now is autom scanning of the market like evemetrics does.
Doing it now with eve metrics and uploading to evemarketdata :)

If you need help in any way (not programming) let me know

Kamikazie
Amarr
Macabre Votum
Morsus Mihi
Posted - 2010.12.23 02:05:00 - [16]
 

Kallahar I can write PHP what ever, mail me in-game with what you need done. I would love to help on the project.

Lord Helghast
Posted - 2010.12.23 13:52:00 - [17]
 

Edited by: Lord Helghast on 23/12/2010 13:54:10
#1 code suggestion should be get the automatic market crawling page the eve-metrics had that thing was awesome, saved me from getting click-induced hypertension lol.... the ability to load a page in the ingame browser, and click "upload the needed from market for my region" was just too awesome.

Suggestion beyond that a solid uploader is a must and to just keep the site light weight, don't need anything fancy, graphs are nice but honestly not needed if people want that they can load the xml via an api and do the graphing and calculations themselves, evemetrics was nice but it was easy to see where certain things got bloated (nicely bloated but bloated) mail and personal market orders i'd say even aren't needed features for a market site... keep it light weight, easily updateable and with a solid fast API and your site will be GREAT!!!

historical data is important but precision of historical data is something you might consider, #1 drop all db data for same item within the same say 15 minutes if the age is < 1 month > 6 hours .... if its > 1 month < 2 months drop data under 30 minutes, so that you have only 30 minute updates at the 2nd month, on the third and back to 1 year i'd say drop everything other than hourly updates for each item per region...

this way you can keep your database light but still have useable historical data.

Also when you design the API make it smart allow the developers out their to only request updates from a last key time

so if i ask for historical data for say ITEM999 don't send me a dump of the entire month, let me say " I want ITEM999 pricehistory since 10:15pm yesterday" that way developers of apps can be smart and only request data from the server that they really need and not keep grabbing huge dumps of csv/xml files of data.

same thing for price check api, keep it small and simple, request = ITEM999, type = sellmax, date = recent, region forge ... response 1000isk.... you can have a price full item grab api that sends the sell and buy max min mean for forge for example, but make sure that when you do api that theirs a nice light weight fast way to just get a price that has low overhead and low bandwidth demand.


BTW an idea to reduce leechers, actually implement a credit system, uploading gives credits, more indemand items/regions give higher credit reward, and using the API for calls deducts credits.... and for those that don't want to be uploading for whatever reason or are just over-pulling data, allow them to make ISK donations for credit bundles.

Kallahar
Posted - 2010.12.23 22:51:00 - [18]
 

All excellent ideas, I get the feeling that the most overhead will be in the database. Here are some stats:
66 Regions
7,928 solarsystems
10,860 items available on the market (eve_inv_types.published=1)
So that means 716,760 region+items, and each one has a list of buyers and sellers.
102,168 uploads to eve-marketdata so far

I haven't really played the markets, is having historical data needed? Is it different than going in-game and looking at the market "price history" or is it just wanting that data from a different region?

On the open-source comments, I fully support the whole open source community, but I have no interest in being in charge of a project or having to deal with reviewing submitted code. So my best thoughts are to keep the "live feed" open to anyone (ie, have a single uploader that everyone can use), then everyone can implement their own ideas separately. I could also just open up the mysql database, but I have to pay cold hard cash for that+bandwidth...

Thanks for the interest everyone!

Kallahar

Serene Python
Posted - 2010.12.24 08:38:00 - [19]
 

Originally by: Kallahar
I haven't really played the markets, is having historical data needed? Is it different than going in-game and looking at the market "price history" or is it just wanting that data from a different region?
Kallahar


In terms of what I use it for... YES! One of my apps uses movement/price history grabbed from the API. Then there's people who either can't login to check prices or don't want to start eve up to just check prices. I've also been in that boat because I didn't want to login/load up eve.

Kallahar
Posted - 2010.12.25 18:01:00 - [20]
 

I got api/item.xml and api/item.json working, but I'm not sure how to calculate the following:

- median
- kurtosis
- skew
- variance
- standard_deviation
- simulated

If anyone knows how to calculate these in mysql/php, please let me know and I'll add them in.

Kallahar


Serene Python
Posted - 2010.12.25 19:42:00 - [21]
 

Edited by: Serene Python on 25/12/2010 20:23:19
Originally by: Kallahar
I got api/item.xml and api/item.json working, but I'm not sure how to calculate the following:

- median
- kurtosis
- skew
- variance
- standard_deviation
- simulated


If anyone knows how to calculate these in mysql/php, please let me know and I'll add them in.

Kallahar




Those all sound all to familiar from math class...

Median (The middle value?) would be something like

$query = mysql_query("SELECT * FROM eve_item_prices WHERE regionID = $_GET[regionID]")
$rows = mysql_num_rows($query)
$median = $rows/2


Don't go line for line with that just a rough guess, taking the total number rows in a region and just dividing by 2 to get the middle row. Just got $query[$median] to get the middle value of the prices. If you have an odd number of rows then just round the $median price.

Merry Christmas!

P.S. I'll throw some other code in here once I get back from Christmas dinner, unless some one else post the code

Nikolai Kondratiev
Sphere Design Inc.
Posted - 2010.12.25 21:53:00 - [22]
 

Originally by: Kallahar
I got api/item.xml and api/item.json working, but I'm not sure how to calculate the following:

- median
- kurtosis
- skew
- variance
- standard_deviation
- simulated

If anyone knows how to calculate these in mysql/php, please let me know and I'll add them in.

Kallahar



Median can be done easily in either PHP or MySQL. Here is some code I use in PHP to calculate it before inserting values in database :


sort($values);
$n = count($values);
$h = intval($n / 2);
if($n % 2 == 0)
$median = round(($values[$h] + $values[$h-1]) / 2);
else
$median = $values[$h];


EM's simulated was the highest price you would have to pay to buy 5% of sell orders IIRC. Probably best done in PHP too when you get new order list for an item.

And for all the stat stuff, a simple google search should give you some explanations on how to calculate it (and probably some code for it too)



Serene Python
Posted - 2010.12.25 23:19:00 - [23]
 

http://php.net/manual/en/function.stats-standard-deviation.php

Can't actually code... PHP has some configuration issue I can't be bothered to fix right now

woddel
Gallente
Canis Industries Ltd
Avaricious Cartel
Posted - 2010.12.26 14:32:00 - [24]
 

hia kallahar

thanks for making this project and hopefully an equal alternative to eve metrics.

one question : do you already have some documentation how your http post service works? meaning the format the data is posted in, variable names etc. that would be great.

also, i am looking forward to a pull api where one can request certain item prices for specific regions. i guess the most important would be 'give me price for tritanium in forge' for example. what would be great is not to simply return the average of all sell order, but try to eliminate all the pseudo-sell orders like shuttles for 100 mil isk - that would greatly falsen the result. in my app, i usually take the weighted average (price * amount) of the top 75% of sell orders and try to ignore orders where the price goes badly above average. weighted average reduces the deviation from like one guy selling a megathron for 50 mil when there are 50 others for 75 mil. the correct price would then be like around 74.5 mil.

last tipp as a developer for myself : keep it simple. do stuff one step at a time. folks (like me) will always want more features - be sure to have that stuff working you do at the moment until you move on the the next module or feature.

good luck and thanks again!

woddel

Hel O'Ween
Men On A Mission
EVE Trade Consortium
Posted - 2010.12.26 17:53:00 - [25]
 

Originally by: Kallahar
I got api/item.xml and api/item.json working, but I'm not sure how to calculate the following:

- median
- kurtosis
- skew
- variance
- standard_deviation
- simulated



More than you perhaps ever wanted to know about statistics: http://en.wikipedia.org/wiki/Statistics.

(Scroll to the bottom of the page for links to algorithms)

Kallahar
Posted - 2010.12.26 23:47:00 - [26]
 

Okay, got median and standard deviation added to the item api, I'll have to figure out how to add the pecl stats extensions to php to get kurtosis and the others working.

For the HTTP callback api, I added documentation for it, just click of "developers".

I'll be adding a simple "what's the "correct" price for an item soon, it will be the best-guess price you could buy/sell an item for in Jita, that should be the price everyone uses as "the price".

Kallahar

Serene Python
Posted - 2010.12.27 08:12:00 - [27]
 

Edited by: Serene Python on 27/12/2010 08:11:54
Originally by: Kallahar
Okay, got median and standard deviation added to the item api, I'll have to figure out how to add the pecl stats extensions to php to get kurtosis and the others working.

For the HTTP callback api, I added documentation for it, just click of "developers".

I'll be adding a simple "what's the "correct" price for an item soon, it will be the best-guess price you could buy/sell an item for in Jita, that should be the price everyone uses as "the price".

Kallahar



Do you any intention of putting in the historical movement/historical prices?

Hel O'Ween
Men On A Mission
EVE Trade Consortium
Posted - 2010.12.28 09:43:00 - [28]
 

Originally by: Kallahar
I got api/item.xml and api/item.json working [...]


Sort of, it seems. For me, the closing </xml> tag is missing.

The query (typeID 24702 is the Hurricane) http://eve-marketdata.com/api/item.xml?type_ids=24702 returns

<?xml version='1.0' encoding='utf-8' ?>
<evemetrics api_version='2.0'>
<!--
EVE MarketData Item API
Documentation is available at the same URI with a .html extension.
-->
<type id='24702'>
<name></name>
<global>
<last_upload>2010-12-26 13:51:46</last_upload>
<buy>
<minimum>2800</minimum>
<maximum>24110122.01</maximum>
<median>20170003</median>
<average>14267671.157727</average>

<standard_deviation>6673819.0621229</standard_deviation>
</buy>
<sell>
<minimum>22000000</minimum>
<maximum>19000000000</maximum>
<median>26250000</median>
<average>45719842.5672625</average>
<standard_deviation>1229254309.9302</standard_deviation>
</sell>
</global>
</type>

</evemetrics>


BTW, is there another place/way to provide feedback to you which you prefer over the EO forums?

Catari Taga
Centre Of Attention
Middle of Nowhere
Posted - 2010.12.28 13:53:00 - [29]
 

Edited by: Catari Taga on 28/12/2010 13:55:20
Originally by: Hel O'Ween
Originally by: Kallahar
I got api/item.xml and api/item.json working [...]


Sort of, it seems. For me, the closing </xml> tag is missing.

There is no closing </xml> tag in XML since there is no opening <xml> tag either (unless you make it part of the document structure which he didn't). The "<?xml..." part is the doctype definition which is not a block element.

But what is wrong is that the server sends incorrect content type text/html instead of text/xml.

Hel O'Ween
Men On A Mission
EVE Trade Consortium
Posted - 2010.12.28 16:11:00 - [30]
 

Embarassed

Erhm ... yes ... of course.

Obviously coffee level wasn't high enough this morning, when I wrote that ...


Pages: [1] 2 3 4 5 6 7 8 9 ... : last (10)

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only