open All Channels
seplocked EVE Technology Lab
blankseplocked - replacement for eve-metrics
This thread is older than 90 days and has been locked due to inactivity.

Pages: first : previous : 1 2 3 [4] 5 6 7 8 9 ... : last (10)

Author Topic

Posted - 2011.01.12 21:08:00 - [91]

colgir - huh, that's odd :) Fixed, derelik added, not sure why the import missed it, it's region_id 10000001 though.

/api/item_orders now accepts solarsystem_id and station_id as optional parameters. I decided to leave type_id as a single value though, I think it's more efficient for me to handle multiple requests, rather than multiple types. Let me know if this is a big deal for anyone though.


Posted - 2011.01.12 21:54:00 - [92]

Edited by: Ahhmyface on 12/01/2011 22:01:55
Hello. I'm new to Eve (and the api), and I'm interested in creating a tool that uses the eve api to analyze market trends and report back a single best "investment" for research/manufacturing. It's a little hard to do by hand, especially since values are changing regularly. The basic idea would be to:

1. Compute processing materials/time required to produce each item based on character skills
2. Examine market trends for each item and calculate the maximum net profit/time (perhaps absolute time, or active user management time... maybe i need a value function) based on material costs and projected final product value.
3. Split out the name of the item and the best place(s) to sell it

I require no UI, or dynamic api calls. I just download data, batch process, output. Is there available code for a simple task like this should I just use the EVE API directly?

Posted - 2011.01.13 00:55:00 - [93]

You can use the eve database dump to get the refine values (it tells you how much minerals you get/need) - make sure to take skills into account. Then you can use the market data api to get the prices for those items, then you'll be able to calculate which ones are "best".

So in summary, you can get the price from eve-marketdata, but the rest is up to you :)


Posted - 2011.01.13 16:54:00 - [94]

Perfect. I will get to work.

Posted - 2011.01.13 21:17:00 - [95]

New feature: Shopping Cart!

So you can enter a bunch of items (or paste in the fit from EFT), and it will show you the nearest system that has them all, and the price in Jita for reference.


Interstellar Business Machines Corp.
Posted - 2011.01.13 21:37:00 - [96]

Started using it tonight, love the automatic market checker although the uploader is a bit slow for me. I went through all the ores and it might be only half way through the logs 10 minutes later. I'm just wondering if I run the auto stuff in the morning for an hour, power off the PC and then when I get home again 9 hours later will it upload the logs and screw with prices if someone has updated the prices in the mean time?

Mediek Ordos
Association of Commonwealth Enterprises
Imperial 0rder
Posted - 2011.01.14 11:46:00 - [97]

A-C-E are happy to see this project and will participate. nice job

Posted - 2011.01.15 08:08:00 - [98]

Edited by: Colgir on 15/01/2011 10:27:31
Is it normal that the upload requires about 30 seconds, to upload ONE product?!
At the moment it is impossible for me to use the automatic uploader because the queue of the uploader is quite fast 'overflowing' and the tool reports an error and stops working.

Edit: A few error-messages i get:
OnNewFile: ...\8833.cache
skipping unknown key ('config', 'GetMap', 30003438)
No data parsed
OnNewFile: ...\e58a.cache
skipping unknown key ('marketProxy', 'GetCharOrders', 14...09)
No data parsed
FileChanged: 3 91aa.cache

After that it is queueing but doesn't upload anything.

Grintau Felintish
Royal Black Watch Highlanders
Warped Aggression
Posted - 2011.01.16 01:46:00 - [99]

Aw, the "EFT Import" feature doesn't seem to work for me on the Shopping Cart. I click that button and can't do anything. Using Chrome 8.0.552.237 ...

Ten Bulls
Posted - 2011.01.16 13:19:00 - [100]

Originally by: Colgir
Edited by: Colgir on 15/01/2011 10:27:31
Is it normal that the upload requires about 30 seconds, to upload ONE product?!
At the moment it is impossible for me to use the automatic uploader because the queue of the uploader is quite fast 'overflowing' and the tool reports an error and stops working.

Edit: A few error-messages i get:
OnNewFile: ...\8833.cache
skipping unknown key ('config', 'GetMap', 30003438)
No data parsed
OnNewFile: ...\e58a.cache
skipping unknown key ('marketProxy', 'GetCharOrders', 14...09)
No data parsed
FileChanged: 3 91aa.cache

After that it is queueing but doesn't upload anything.

Where you running two eve clients ?

I think the uploader only works for one client, ive had to restart windows to get it working again.

Posted - 2011.01.17 17:15:00 - [101]

I use only one eve-client.

Astromechanica Federatis
Posted - 2011.01.17 19:34:00 - [102]

Nice work Kallahar :)
I'm trying to work with your mysql dump, but these seems not very up to date ...
mysql_items_selling, the one I'm interesting in, is from January 7th ...
Do you plan to generate this sql more often ? Or maybe I've to work with your API if I want more precise data ?

Posted - 2011.01.17 21:17:00 - [103]

Okay, Huge update, and a bunch of critical bugs fixed.

Turns out, the reference uploader that eve-metrics used didn't correctly upload orders or history that were blank. So if you had one item for sale in a region, and then that item sold, the database would never get updated. Not a problem in The Forge, but you get out to 0.0 and it's very common.

In the process of fixing that, I also completely redesigned how data is uploaded. So now there's a lot less bandwidth wasted. Also, and this is huge, uploads went from an average of 2.5 seconds per upload to 0.002 seconds. That was why so many people were filling up their queue. I did about 3,000 market and history updates at about 4 per second and it didn't error once. It also displays in the log "x items queued", which I got up to 60 one time before it automatically cleared them out. So hopefully that takes care of 90% of the uploader errors. If you're getting any uploader errors please let me know, if I can duplicate the error then I can fix it!

The "live data" feed also changed, to fit the new uploader design. Click on Developers on the site for the new design, it now also uploads history data.

The daily data dumps weren't working - fixed

You can also check the Reports tab, which will give you things like region statistics if you're looking for which regions need what data uploaded. As usual, we need lots more uploaders!

I'm really excited at how much data we're accumulating, some stats:
695,000 items for sale
239,000 buying
4,795,000 history points (region+type+date)
60% of The Forge updated in the last 24h - AWESOME
17% Sinq Laison
24% The Citadel (that was me...)
28% Metropolis
337 users (about 50 active uploaders, but I just reset the stats...)

Keep up the good work, my current goal is to get as much of the market updated often as possible!


Posted - 2011.01.17 22:47:00 - [104]

Also fixed the bug where the automatic uploader would stop after the first 100 items. It was an IGB Javascript error, and I haven't figured out a way to catch those errors yet, there doesn't seem to be a debug console in the browser...


Padraig O'Healey
Posted - 2011.01.18 00:38:00 - [105]

Been running your automatic uploader. Some items I know I have scanned (say, Armor Plates for The Forge region) that show a last scan time tag of 14992 days ago.

Thanks for your effort.

Timetravel Enterprises
Posted - 2011.01.19 15:45:00 - [106]

I would like to import this data into a tool I use. however it requires a specific format for the generated XML. I have been working on this locally, but I cant seem to keep the data up to date (my server is sick and needs to be put down Embarassed)
If I provide you the php page, would you consider hosting it?

Posted - 2011.01.19 21:27:00 - [107]

The new updater works quite well. Good work!

Posted - 2011.01.19 23:42:00 - [108]

Growing pains! Thanks everyone for uploading data, the DB is now at 2 gigs! 718,000 sell orders, 244,000 buy orders, 8.2 million history points!

I had to redesign the server side code that handles uploads again, there were an average of 30 uploads per minute up until the incursion 1.2 update happened. I took the opportunity to move the upload handler from every-upload to a batch process, so now it processes all the uploads once per minute. MUCH more efficient.

btw I did three client updates for Incursion 1.2. One computer worked fine, the 2nd broke but the repair tool fixed it, and the 3rd (my main gaming computer) utterly choked and I had to reinstall from scratch, ugh.

Also, I had to remove the "live data" API feature. It's been replaced with a new API method called "recent uploads" which will give you a list of region+types that were recently updated, you can then query the regular APIs for just those items to get the current data. MUCH better than the old way!

These changes meant a lot of wide-ranging changes, so if you notice that any pages are erroring or giving odd data, please let me know! If you can't use the Shopping Cart, please send me your javascript debug log, I'm pretty sure Safari doesn't like some of my javascript, but I don't have any mac's to test with.

Thanks everyone!


Padraig O'Healey
Posted - 2011.01.20 07:12:00 - [109]

Volumes reported in the history section of price_check.php seem to be inconsistent with in-game data - or at the very least, have a hard ceiling on value?

With Forge selected, volume transacted on 01-16 is 2,147,483,647 units. The cached in-game value is. 44,375,314,723 units. I'm guessing that DB field doesn't store anything greater than 2^31? May be "working as intended", but something I wanted to indicate once I noticed it.

I am also continuing to notice that reports with recent updates (at least, I'm assuming recent since I'm fairly certain they are my uploads) have timetags showing the data was last updated 14,994 days ago. That timetag is 1/1/1970... is the datestamp being lost somewhere?

Thanks for your effort with this, I never got to use EVE-Metrics, but I wish I had, from my reading here & elsewhere. You seem to have a good effort going, and given EVE-Central's lack of updates, seem to be my best hope for pulling in reasonably accurate market data.

Posted - 2011.01.20 20:06:00 - [110]

Padraig: Good catch! Yep, it was storing the order volume as an integer. It's now stored as a bigint (64 bit). The prices are stored as doubles, so they weren't affected.

The only data affected was for Tritanium, Pyerite, and Mexallon. The Forge has been updated, but other regions need their history data uploaded.

The 14,000 days - which page shows that? It's supposed to say "no uploads".


Posted - 2011.01.20 21:47:00 - [111]

Edited by: PsyKzz on 20/01/2011 21:47:45

Just so that you know, this is part of the examples, this page gives an error.

<font color="#ff3333">
<b>Error:</b> SQLGetSet
<br>1054:Unknown column '5pct_sell_price' in 'field list'
<br><i>select 5pct_sell_price as val from eve_inv_types where type_id = '34'</i>

Posted - 2011.01.21 01:08:00 - [112]

PsyKzz - fixed! had a typo, sorry about that.

In a related note, I changed how those prices are calculated. It is now in line with how eve-central does it. Here's how it works:

- Look at the entire market in the universe.
- Find the total quantity for sale, and find 5% of that.
- Using the entire universe, buy up 5% of the market.
- Take the average price from that purchase.

The price is updated once a day.

This will result in (in general) slightly lower prices than the old values, but it eliminates a few outlier prices that were screwing up the old calculation. It shouldn't be possible for anyone to game the stats now, unless they really do buy up 5% of the market in the entire universe. The price could change early on as new regions get scanned, right now it's jita-heavy, but that's how the actual market is too :)


Padraig O'Healey
Posted - 2011.01.21 21:44:00 - [113]

Regarding the bogus timetags:®ion_id=10000002

This is not the only item its like this for.

The Nietzsche Followers
Shades of Gray
Posted - 2011.01.22 02:22:00 - [114]

Edited by: Azrakadar on 22/01/2011 04:20:52
Hey there!

I'm posting to show my support for your initiative and efforts :)

I couldn't code my way out of a paper bag (I'd love to learn, but don't have the time) - so I'm helping by uploading :)

You rock!


Now, time for a couple of questions ;)

- Will the "item prices" method ever support any parameters that narrow down the places used in the calculations of price? For example region_id, solarsystem_id, or (this would be ideal for my purposes YARRRR!!) station_id?

- Probably because of my slow internet connection, the automatic uploader tends to build large queues. Tonight, after having uploaded 1000+ items, it had a queue in the 50-somethings. While writing this post, I went to check again and it was around 40.

The thing is, I'm kinda worried that if I leave the uploader running for too long, I'll be left with a huge queue that will cause crashes. Is this possible? Do I need to worry? What happens if the queue goes, say, OVER 9000!!!?

Shirah Yuri
Posted - 2011.01.24 08:53:00 - [115]

Awesome improvements to the site, keep up the good work.

Just found one minor issue:

does not return upload_id and upload_type. Neither do xml and json exports. Thanks for adding this great feature. This will definitely be useful for many and help reduce bandwidth usage.

Posted - 2011.01.25 00:12:00 - [116]

- Bad time tags: fixed!
- Long upload queue: I'll be adding a feature soon so you can adjust the automatic uploader speed.
- api: Prices in different regions: Not on that api call, but you can use item_orders to get them all and then take your own average. The reason is that item_price uses a one-a-day cached value.
- api: recent_uploads - fixed! it was actually the documentation that was wrong... it doesn't return a unique ID any more. just the type/region, which you can then use to call api/item_orders.

Thanks everyone! I've got another "holy *** that's as cool as the shopping cart" feature soon!


Matterick Boon
Posted - 2011.01.25 08:47:00 - [117]

I've experienced this issue twice now. I often go to bed and just leave the auto uploader running. When I wake up the updater page is still active and changing the "Checking" to random items but the market is no longer changing, it's stuck on an item. I checked my uploader and it looks like it stopped working after around 5700 items scanned.

To correct this I have to close the eve browser and open it again. Closing the market and reopening it does nothing. So it appears to be some issue between the browser and client. I can try to repeat this but if there is some debugging info or log i can provide you for troubleshooting please let me know.

Posted - 2011.01.25 18:47:00 - [118]

Okay, added the upload rate feature.

There does seem to be a problem though, and I'm not sure if it's one that I can fix... The problem is that there's no javascript error log that I've found in the IGB...

It appears that if you call CCPEve.showMarketDetails(); again before the previous call finishes, it can crash the browser. Restarting the browser fixes it, but reloading the page does not. Unfortunately, the function doesn't have a return value, nor does it block, so as far as I can tell there's no way for javascript to know whether the previous call has completed or not. Slowing it down should fix it, but if the game lags then it might still get an error.

I'll do some more searching...


Posted - 2011.01.25 19:40:00 - [119]

Okay, scratch that idea, looks like it just has an internal rate limiter at about 3 seconds, then it just silently errors. So the maximum speed is now set at 1 update per three seconds, and you can slow it down if you want.

Pro tip: If you search the market for 'e', then use the keyboard to go through each item, you can do about 3 per second, and it updates 58% of the market (but you have to press the down arrow about 3500 times, heh.


Posted - 2011.01.26 11:19:00 - [120]

Just to encourage uploaders, it'd be great if we could get the uploader keys working so people can see how much they've uploaded, get credit for it (greatest uploaders for the past few days/week/month etc), and possibly the points system eve-metrics had for uploading items that haven't been uploaded for a while.

I've been leaving 2 or 3 clients open when I get a chance with alts in different regions, just using the tool to keep cycling through new items. I know there's a lot of people who will do the same if they get some kind of score for it. Scores matter :P

This probably seems like a less critical part of eve-marketdata to get up and running, but it's impact on the number of uploaders will hopefully be significant, so more data, hopefully meaning more useful data on the sight, leading to more users, leading to more uploaders...... :)

Pages: first : previous : 1 2 3 [4] 5 6 7 8 9 ... : last (10)

This thread is older than 90 days and has been locked due to inactivity.


The new forums are live

Please adjust your bookmarks to

These forums are archived and read-only