open All Channels
seplocked EVE Technology Lab
blankseplocked EVE Marketeer (beta) - (Market) Data Aggregation & Display
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: 1 [2] 3 4

Author Topic

Almir Kadric
Posted - 2011.08.02 10:58:00 - [31]
 

BTW the SystemDistances table you have, does it contain routes greater than 15-20 jumps, lets say systems 52 jumps apart?

And does it provide the actual route between those systems?

Callean Drevus
Caldari
Icosahedron Crafts and Shipping
Silent Infinity
Posted - 2011.08.02 11:08:00 - [32]
 

Originally by: Almir Kadric
BTW the SystemDistances table you have, does it contain routes greater than 15-20 jumps, lets say systems 52 jumps apart?

And does it provide the actual route between those systems?


AFAIK, it shows every route between all systems. I just blew up my database checking whether it got distances higher than 20 jumps, so I'd guess yes. It's a pretty hefty table with 27M rows though. That was the first one I tried to eliminate when I found out the query was slow.

Also, I don't know if you saw it, since I edited my previous post, and you replied so fast, but I had to edit the order statement to do the calculation again. Otherwise it wouldn't show me anything higher than 999 per jump. Any ideas?

Almir Kadric
Posted - 2011.08.02 11:23:00 - [33]
 

Edited by: Almir Kadric on 02/08/2011 11:28:43
Originally by: Callean Drevus
AFAIK, it shows every route between all systems. I just blew up my database checking whether it got distances higher than 20 jumps, so I'd guess yes. It's a pretty hefty table with 27M rows though. That was the first one I tried to eliminate when I found out the query was slow.



So it shows all routes and not just the shortest? That's how I did my original caching, but I've changed it since to only do shortest. Also I changed mine to only cache high sec systems since realistically not many would actually trade through low or null sec. And it shows the actual route, checked it against actually in game Auto Pilot so far identical. Only down side is i recently did the caching for 10-15 jump routes and it took 4 hours =_= currently figuring out a way to optimize it further, otherwise ill have to whip out my super computer in a box to do the cache once and for all then just save it somewhere.

Originally by: Callean Drevus
Also, I don't know if you saw it, since I edited my previous post, and you replied so fast, but I had to edit the order statement to do the calculation again. Otherwise it wouldn't show me anything higher than 999 per jump. Any ideas?


EDIT: Just read your previous post edit. I see. That's unusual. I tested it on mine again and it seems fine. Did you change anything? show me your full query and ill take a look.

Callean Drevus
Caldari
Icosahedron Crafts and Shipping
Silent Infinity
Posted - 2011.08.02 12:15:00 - [34]
 

Ok, never mind, I found out what was causing the problem. MySQL doesn't like to sort on formatted fields Rolling Eyes

There is actually quite a big problem with not having cached queries for me, as the data is updated every minute (changing the table, thus making caching useless). I guess I'll have to find a way around that, possibly just storing the result of the trades query in a different table for an hour or so.

Almir Kadric
Posted - 2011.08.02 13:38:00 - [35]
 

Originally by: Callean Drevus
Ok, never mind, I found out what was causing the problem. MySQL doesn't like to sort on formatted fields Rolling Eyes


LOL didn't know that

Originally by: Callean Drevus
There is actually quite a big problem with not having cached queries for me, as the data is updated every minute (changing the table, thus making caching useless). I guess I'll have to find a way around that, possibly just storing the result of the trades query in a different table for an hour or so.


I don't think that really changes the resultant time that much. Especially if you're adding data on the fly. The database engine should be normalizing keys for entered data as you add them. The only time it slows down on first run is when you rebuild the table.

And to my next point why is your data changing every minute or so? You should realistically only be updating data every hour, or rather so doing a few items at a time where it takes an hour to rebuild the entire set.

Callean Drevus
Caldari
Icosahedron Crafts and Shipping
Silent Infinity
Posted - 2011.08.02 14:13:00 - [36]
 

Originally by: Almir Kadric
I don't think that really changes the resultant time that much. Especially if you're adding data on the fly. The database engine should be normalizing keys for entered data as you add them. The only time it slows down on first run is when you rebuild the table.

And to my next point why is your data changing every minute or so? You should realistically only be updating data every hour, or rather so doing a few items at a time where it takes an hour to rebuild the entire set.


Well, I try to have my marketdata as realtime as possible, so when someone uploads something new, I want those orders in my database as fast as possible.

Of course, I'm not calculating the traderoutes again everytime I get new data, that would be database suicide :)
I just don't think the MySQL query cache likes it when I constantly add new data to mess up it's calculations. But I have not yet put serious research into that.

On a positive note, even after adding in all the other joins, the retrieval is still quite fast at 15-30 seconds.

I was just reminded of my ability to export my own database. So if you still need it, you can get the complete solarsystemdistances table from http://www.serial-experiments.com/solarsystemdistances.7z

Callean Drevus
Caldari
Icosahedron Crafts and Shipping
Silent Infinity
Posted - 2011.08.02 19:13:00 - [37]
 

Well... I don't know what to say. Today I was working on making the trading information a reality, when I found out that some orders which are shown do not actually exist.

I thought this was weird, and I thought I might be doing something wrong with the import, but everything seemed quite fine. After a bit of tinkering I found out that my orders were actually correctly inserted, only to be suddenly reverted to the wrong state after a little bit of time. How was this possible?

It seems eve marketdata exports nonexistent orders. They even do that after they've just received the RIGHT information. After all, I just sent it to them. What was happening, was that my uploader sent the data to my server, where it got processed, it also sent it to the eve marketdata server where it was (presumably) processed as well. After my own processing everything went perfectly well, but then my lovely server got to the point where it retrieved all new data from eve marketdata, and surely, overwrote all the correct data with the invalid data that eve marketdata was sending.

I've now made it so that all data I recently received cannot be overwritten by eve marketdata data, but it still leaves the question of whether there is more incorrect data being sent...

Almir Kadric
Posted - 2011.08.03 02:34:00 - [38]
 

Originally by: Callean Drevus
On a positive note, even after adding in all the other joins, the retrieval is still quite fast at 15-30 seconds.


Glad to hear it ^_^

Originally by: Callean Drevus
I was just reminded of my ability to export my own database. So if you still need it, you can get the complete solarsystemdistances table from http://www.serial-experiments.com/solarsystemdistances.7z


LOL thanks for this, ill check what it contains and decides what's better, the one i built or this one.

Originally by: Callean Drevus

Well... I don't know what to say. Today I was working on making the trading information a reality, when I found out that some orders which are shown do not actually exist.

I thought this was weird, and I thought I might be doing something wrong with the import, but everything seemed quite fine. After a bit of tinkering I found out that my orders were actually correctly inserted, only to be suddenly reverted to the wrong state after a little bit of time. How was this possible?

It seems eve marketdata exports nonexistent orders. They even do that after they've just received the RIGHT information. After all, I just sent it to them. What was happening, was that my uploader sent the data to my server, where it got processed, it also sent it to the eve marketdata server where it was (presumably) processed as well. After my own processing everything went perfectly well, but then my lovely server got to the point where it retrieved all new data from eve marketdata, and surely, overwrote all the correct data with the invalid data that eve marketdata was sending.

I've now made it so that all data I recently received cannot be overwritten by eve marketdata data, but it still leaves the question of whether there is more incorrect data being sent...



Hmm sounds like eve-marketdata is caching the information first then only allowing exports after a certain time period. For example even though they get data in real time. They probably have that in a table which is only for inserts. Then they have a separate table which is built every hour or what ever time period which is specifically for select statements, which makes sense since they can speed up those statements, not to mention if they run a first run SQL cache statement to have sql ready for fast reads and just hot-swapping the databases over when they're ready.

Best to send them an email to find out what they're up to ;p

Let me know if you find anything

Callean Drevus
Caldari
Icosahedron Crafts and Shipping
Silent Infinity
Posted - 2011.08.03 20:14:00 - [39]
 

For everyone who is interested. The market scanner of evemarketeer is now able to scan only items you enter manually, therefore scanning only the items that are of interest to you.

Domi Etymology
Posted - 2011.08.04 00:17:00 - [40]
 

With the default upload suggestions for the market scanner, could I suggest you take out the Tech 2 BPO's, since I'm 99.999999% sure they aren't sold on the market anymore ;)

Callean Drevus
Caldari
Icosahedron Crafts and Shipping
Silent Infinity
Posted - 2011.08.04 09:56:00 - [41]
 

It seems there IS a marketgroup for them though :P as that is what I am filtering by. I'll look into removing them ;)

Temuil
Posted - 2011.08.05 15:09:00 - [42]
 

I've been using this for a few days and liking it already.

I have a request if possible, when you search for an item and it shows the Regional Prices, can Molden Heath be added? Smile

Callean Drevus
Caldari
Icosahedron Crafts and Shipping
Silent Infinity
Posted - 2011.08.05 20:48:00 - [43]
 

Originally by: Temuil
I've been using this for a few days and liking it already.

I have a request if possible, when you search for an item and it shows the Regional Prices, can Molden Heath be added? Smile


Jup, that is a legacy part of the website from when I was only using it myself. The other regions have been added/will soon appear.

Callean Drevus
Caldari
Icosahedron Crafts and Shipping
Silent Infinity
Posted - 2011.08.05 21:01:00 - [44]
 

I almost forget that I have added a few new features. Very Happy

You can now look up traderoutes within and across a few empire regions (these are updated every hour, you can see when they will be updated, so if you check a few minutes after that, they should be fresh). Provided data is actually uploaded in the regions where you try to find trades :) Trades older than 24 hours cannot be found, as it makes no sense, they'll be fulfilled anyway (I think, please contradict me if I'm wrong).

For now, eve central orders are not taken into account when calculating trades, since any and all orders I got during testing were either fake or outdated (even if updated 5 minutes previously). Didn't seem like a good idea.

You can also create a list of items to check their total value (and export and save the list to be used again later). Eventually saved on server, for now, you'll have to create your own text file :) I used this one to check the total value of items in a contract. Until CCP releases the Contracts API.

Callean Drevus
Caldari
Icosahedron Crafts and Shipping
Silent Infinity
Posted - 2011.08.07 19:58:00 - [45]
 

I just updated evemarketeer with a feature that I think traders will love. It already exists on eve-marketdata of course, but I've streamlined the process a bit.

It involves checking whether your orders are really the lowest in the region. All updating of data is done via ajax, so the only thing you have to do is periodically check whether all the rows are still green.

Check it out at http://www.evemarketeer.com/trade/orders

Callean Drevus
Caldari
Icosahedron Crafts and Shipping
Silent Infinity
Posted - 2011.08.07 23:04:00 - [46]
 

In addition, I have an interesting image here, of what happens when you get your data from 3 different sources that for some idiotic reason do not align.

I'm talking about my own data, eve-marketdata and eve-central. Does anyone see a pattern here: http://www.evemarketeer.com/graph/tiny_history/10000043/20413/2?

Cleuseau Remos
Caldari
Adamas Anima
Posted - 2011.08.08 15:55:00 - [47]
 

Callean,

Can you add line terminators to your API XML? I'm trying to access data with an xpath and it is not working. My guru friend told me Xpath needs it.

Example would be: http://www.evemarketeer.com/api/jita_info/34/xml

Shows up great in a browser and I'm sure its fine as XML, but my Xpath parser needs line terminators.

TIA!

Callean Drevus
Caldari
Icosahedron Crafts and Shipping
Silent Infinity
Posted - 2011.08.09 00:38:00 - [48]
 

Originally by: Cleuseau Remos
Can you add line terminators to your API XML? I'm trying to access data with an xpath and it is not working. My guru friend told me Xpath needs it.

Example would be: http://www.evemarketeer.com/api/jita_info/34/xml

Shows up great in a browser and I'm sure its fine as XML, but my Xpath parser needs line terminators.


I couldn't leave this alone :P

Something still tells me line terminators were not the problem, though I would not be hesitant about adding them if it were really the case. I've been trying with google docs myself for a bit just now, I made a few tiny changes to how the data was presented, and it seems to work now (google caches the same URL for 2 hours, so you'll have to change your URL a bit (add/remove a type_id or something) to see any changes in success).

Almir Kadric
Posted - 2011.08.09 02:28:00 - [49]
 

Originally by: Callean Drevus

I couldn't leave this alone :P

Something still tells me line terminators were not the problem, though I would not be hesitant about adding them if it were really the case. I've been trying with google docs myself for a bit just now, I made a few tiny changes to how the data was presented, and it seems to work now (google caches the same URL for 2 hours, so you'll have to change your URL a bit (add/remove a type_id or something) to see any changes in success).


Aren't you using an inbuilt XML parser to crate the XML? If so it should automatically present the XML ready for xpath use. Not to mention its easier coding to add on the fly as you use objects and an export rather than string concatenation. Hence cleaner looking code.

Callean Drevus
Caldari
Icosahedron Crafts and Shipping
Silent Infinity
Posted - 2011.08.09 09:29:00 - [50]
 

Edited by: Callean Drevus on 09/08/2011 09:29:36
Originally by: Almir Kadric
Aren't you using an inbuilt XML parser to crate the XML? If so it should automatically present the XML ready for xpath use. Not to mention its easier coding to add on the fly as you use objects and an export rather than string concatenation. Hence cleaner looking code.


Actualy, I'm building the XML myself, since there'a always only a set of rows to display it I really didn't think about using the php DOM or XML functions. There are very little ways in which XML can not be ready for xpath use if it's well formatted. And given the depth of my XML, it's pretty much impossible to make any mistake.

What I was talking about when I said 'changes to the way data is presented' I meant made sure that a 'content-type: text/xml' header is sent, which google docs seems to like :)

PS: It is however, not impossible that I'll fall back to PHP lib functions if the output gets any more complex :)

Callean Drevus
Caldari
Icosahedron Crafts and Shipping
Silent Infinity
Posted - 2011.08.09 20:32:00 - [51]
 

People, I'd like to present my awesome feedback and prioritization form, allowing you to make your wishes known more easily. Please use it!

Domi Etymology
Posted - 2011.08.10 01:09:00 - [52]
 

Eve's randomly down at the moment, so can't test the feedback form in the game browser, but trying to use it in firefox gives me "javascript:$('#descr_5').toggle();" in the address bar and "[object Object]" on the web page. Did I derp?

Callean Drevus
Caldari
Icosahedron Crafts and Shipping
Silent Infinity
Posted - 2011.08.10 13:10:00 - [53]
 

Nope, you didn't derp, I did. Seems firefox doesn't like me putting javascript function in the href anymore. In any case, it should be fixed now. It's really unhandy if there's a bug in the feedback form :)

Domi Etymology
Posted - 2011.08.11 12:10:00 - [54]
 

Trying to search for prices on R.A.M.-_Weapon_Tech seems to be busted.

For little bugs like this, do you want us to post here, or on the feedback thing you made?

Turhan Bey
EVE University
Ivy League
Posted - 2011.08.11 15:20:00 - [55]
 

- Attempts to upload to EVE-MarketData even when checkbox is deselected. The best workaround I can find is to empty your EMD key, in which case it spits out an error for each item, but at least it doesn't hang while the EMD site is down.

Error: EVE Marketdata upload key must be set before upload!
Uploaded 38 in Sinq Laison
Error: EVE Marketdata upload key must be set before upload!
Uploaded 35 in Sinq Laison
Error: EVE Marketdata upload key must be set before upload!
Uploaded 34 in Sinq Laison
Error: EVE Marketdata upload key must be set before upload!
Uploaded 39 in Sinq Laison

- What exactly determines the amount of "credits" given in the Top Uploaders list?

Top Uploads, Turhan Bey: 174,723
Top First Uploads, Turhan Bey: 70,787
Top Credits, Turhan Bey: 1,601,930.60

Top Uploads, Temuil: 72,783
Top First Uploads, Temuil: 10,251
Top Credits, Temuil: 1,716,783.58

Callean Drevus
Caldari
Icosahedron Crafts and Shipping
Silent Infinity
Posted - 2011.08.11 17:36:00 - [56]
 

Edited by: Callean Drevus on 11/08/2011 17:55:14
Edited by: Callean Drevus on 11/08/2011 17:48:33
Originally by: Domi Etymology
Trying to search for prices on R.A.M.-_Weapon_Tech seems to be busted.

For little bugs like this, do you want us to post here, or on the feedback thing you made?


Both are fine, if you can put it on feedback I will not forget about if the amount of posting here will suddenly increase for one reason or another.
Dots in the name were causing problems. They have been fixed.

Originally by: Turhan Bey
- Attempts to upload to EVE-MarketData even when checkbox is deselected. The best workaround I can find is to empty your EMD key, in which case it spits out an error for each item, but at least it doesn't hang while the EMD site is down.

- What exactly determines the amount of "credits" given in the Top Uploaders list?


I see, this probably happened when you turned it off while the program was working. It was not updating a required variable correctly, leaving it in the previous state until restart.

In regards to the way the credits are calculated. I is based on how much orders have changed since the last upload, and on how much time has elapsed between then and now. It also takes into account whether there have been previous uploads for that item (hence the first upload thing), because if I did not, you would get MASSIVE amounts of credits when uploading information for an item for the first time (actually, I think that would grant you infinite credits Rolling Eyes).
What I think is that Temuil must have uploaded data that was very much changed since before, in regions that were very much outdated.

Turhan Bey
EVE University
Ivy League
Posted - 2011.08.11 20:27:00 - [57]
 

I've restarted the uploader (several times) after deselecting the checkbox. Even tried it one more time just now. It still believes that it is in the selected state.

Callean Drevus
Caldari
Icosahedron Crafts and Shipping
Silent Infinity
Posted - 2011.08.11 20:38:00 - [58]
 

Originally by: Turhan Bey
I've restarted the uploader (several times) after deselecting the checkbox. Even tried it one more time just now. It still believes that it is in the selected state.


Is the checkbox itself in the correct state after restarting? I'll have this fixed in the next uploader version.

Turhan Bey
EVE University
Ivy League
Posted - 2011.08.11 21:59:00 - [59]
 

Yep, the checkbox itself appears correct. But the uploader behaves as if it were still selected.

Turhan Bey
EVE University
Ivy League
Posted - 2011.08.12 16:20:00 - [60]
 

re: credits

I assume that you have an upper cap on the time for repeated submissions, probably along the lines of a few days or a week. Otherwise, remote locations that may be ignored for many months at a time would generate credits also at huge levels. I would recommend treating a first-time submission at this cap level, i.e. reporting an item for the first time is equivalent to being the first to report it in N weeks/months at whatever is your cap.

Otherwise, what you are essentially telling your users is that it is more valuable for them to not go out to new areas and scan them, but instead repeatedly scan existing areas. It is also not valuable to keep remote area prices up-to-date, and they instead should wait as long as possible before going back and submitting updated info.


Pages: 1 [2] 3 4

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only