open All Channels
seplocked EVE Technology Lab
blankseplocked Reverence - 100% compatible EVE cache library for Python
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: 1 2 3 4 5 [6] 7

Author Topic

Risingson
Posted - 2011.05.17 10:52:00 - [151]
 

Edited by: Risingson on 17/05/2011 10:52:29

i am using this: https://github.com/ntt/reverence/blob/master/examples/datadump.py

most files working ok but i get:
MethodCall.server.map.GetDeadspaceAgentsMap.EN ... NO HEADER (<type 'list'>)
MethodCall.server.map.GetDeadspaceComplexMap.EN ... NO HEADER (<type 'list'>)


unfortunately thats exactly what i am looking for.
help appreciated since i am a py n00b. ty!

Entity
X-Factor Industries
Synthetic Existence
Posted - 2011.05.17 11:07:00 - [152]
 

Originally by: Risingson
i am using this: https://github.com/ntt/reverence/blob/master/examples/datadump.py


Rolling Eyes

I mean the cache file causing the crash, silly...

Risingson
Posted - 2011.05.17 11:16:00 - [153]
 

Originally by: Entity
Originally by: Risingson
i am using this: https://github.com/ntt/reverence/blob/master/examples/datadump.py

I mean the cache file causing the crash, silly...


uhmm, confused by your answer... is it somehow possible to get the data out of those 2 cache files?

Entity
X-Factor Industries
Synthetic Existence
Posted - 2011.05.17 11:48:00 - [154]
 

Edited by: Entity on 17/05/2011 12:23:20
Originally by: Risingson
Originally by: Entity
Originally by: Risingson
i am using this: https://github.com/ntt/reverence/blob/master/examples/datadump.py

I mean the cache file causing the crash, silly...


uhmm, confused by your answer... is it somehow possible to get the data out of those 2 cache files?


Thanks for reminding me why I do not have a customer support job Laughing

I want that file. Put it somewhere so I can inspect it myself.


Risingson
Posted - 2011.05.17 12:07:00 - [155]
 

Edited by: Risingson on 17/05/2011 12:09:05
Originally by: Entity
Thanks for reminding me why I do not have a customer support job


hehe. sorry should have thought of that by myself.... 86b8.cache and 9554.cache, datadump.py

thank you!

Entity
X-Factor Industries
Synthetic Existence
Posted - 2011.05.17 12:18:00 - [156]
 

Edited by: Entity on 17/05/2011 12:23:48
Edited by: Entity on 17/05/2011 12:19:31
Originally by: Risingson
Edited by: Risingson on 17/05/2011 12:09:05
Originally by: Entity
Thanks for reminding me why I do not have a customer support job


hehe. sorry should have thought of that by myself.... 86b8.cache and 9554.cache, datadump.py

thank you!


Ah, yeah, they're not database tables, so datadump.py does not dump those.
You should write code yourself to deal with these.

do

obj = blue.marshal.Load(file)

then obj[1]['lret'] is a list of tuples it seems.
other than the systemID and name I don't know what the 2 other values in there are though.

Also, seems I got confused by your earlier post, sorry for that ^.^

Risingson
Posted - 2011.05.17 12:46:00 - [157]
 

Edited by: Risingson on 17/05/2011 13:23:10

ok i will try. i've never coded a single line of py though =)
thanks for looking into this!

edit:
from reverence import blue
dump='9554.cache'
f = open(dump, 'rb')
file = f.read()
f.close
obj = blue.marshal.Load(file)
w = open('9554.txt','w')
w.write(str(obj[1]['lret']))
w.close


worx =)

Mike'P
Posted - 2011.05.17 19:10:00 - [158]
 

Originally by: Entity


Interesting. Can you put this file somewhere I can download it?


Well, I can't give you that particular file as it has now changed and doesn't exhibit the problem. However, I can give you another file which is even more screwy - it reports 4 shared objects, but if you get the old hex dumper out, there is quite clearly not 4 lowball int32_t's at the end of the file.

Grab it here....

http://www.team59.net/66d6.cache

Entity
X-Factor Industries
Synthetic Existence
Posted - 2011.05.18 00:54:00 - [159]
 

Originally by: Mike'P
Originally by: Entity


Interesting. Can you put this file somewhere I can download it?


Well, I can't give you that particular file as it has now changed and doesn't exhibit the problem. However, I can give you another file which is even more screwy - it reports 4 shared objects, but if you get the old hex dumper out, there is quite clearly not 4 lowball int32_t's at the end of the file.

Grab it here....

http://www.team59.net/66d6.cache



hm, looking at that file, it shows 2 ints at the end, and the count (the int after the very first byte) also says 2, and it decodes without error here (XP 32bit) I'm confused now :)

Mike'P
Posted - 2011.05.18 12:30:00 - [160]
 

Hmm, I just checked that file and you are 100% right, of course! Somehow a running Eve must have overwritten that cache file whilst I was flailing around trying to deduce what was going on.

Anyway, if you try the link again :-

http://www.team59.net/66d6.cache

*that* file has a '04' as the first int after the first byte - and its a GetOrder... head and tail below...

C:\Documents and Settings\Administrator\Desktop\66d6.cache 18,391 bytes 18/05/2011 13:19:42

-0 -1 -2 -3 -4 -5 -6 -7 -8 -9 -A -B -C -D -E -F

00000000- 7E 04 00 00 00 2C 14 04 11 88 13 09 47 65 74 4F [~....,......GetO]
00000001- 72 64 65 72 73 2F 04 AB 96 98 00 05 74 60 16 05 [rders/......t`..]
......
0000047B- 72 65 74 1B 02 13 07 76 65 72 73 69 6F 6E 2F 08 [ret....version/.]
0000047C- A3 11 DE 1A BD 14 CC 01 13 05 72 75 6E 69 64 02 [..........runid.]
0000047D- 00 00 00 01 00 00 00 [....... ]

I'm on AMD 64 bit, Linux 2.6.32, Python 2.6.2, for the record.


Entity
X-Factor Industries
Synthetic Existence
Posted - 2011.05.18 17:17:00 - [161]
 

Originally by: Mike'P
Anyway, if you try the link again :-



Ok, I've checked that file and did some occult experiments.
I can safely say it's either corruption, a failed save, or a bug in EVE.
See, the map counter requires that many ints at the end of the file, because there will be that many distinct TYPE_REF tokens in the stream.
I think not even EVE itself will know how to decode files with such corrupted maps; there is strong evidence for this in blue.dll in the form of a string saying "Bogus map data in marshal stream".

There is no fix for this, but I will make it detect this case and throw a proper exception. (probably one saying "Bogus map data in marshal stream" Laughing)

I'll update Reverence Soon™.

Thank you for pointing out this issue :)

Mike'P
Posted - 2011.05.18 20:05:00 - [162]
 

Well depressingly, it seems to be happening rather often on my setup!

Admittedly, I'm probably nudging the envelope slightly as this is an Eve running on a Linux laptop using wine with the cache folder mounted via samba on another Linux box which is running the actual Reverence code.

But even so...

The annoyance is that the code I'm trying to use load's the entire cache folder - and one duff cache file causes the whole thing to fall over. I guess if this particular case is trapped properly I can manage that exception - thanks for looking into it.

I think I reported it more to shine a light. rather than expecting any direct help!

So thanks muchly - oh, and great library, BTW ;-)

Entity
X-Factor Industries
Synthetic Existence
Posted - 2011.05.25 17:04:00 - [163]
 

1.2.2 released.
The only change is the fix for the corrupt shared object map issue above, but updating doesn't hurt :)

Guido Calabresi
Posted - 2011.06.10 02:57:00 - [164]
 

I can confirm that the datadump.py script runs properly on Mac OS X with the (more or less) default python install if you manually configure the path to the EVE directory inside the application package.

Sullinae
Posted - 2011.06.23 20:44:00 - [165]
 

busted for Incarna, needs to be updated for the new cache formats
please update this great tool, I want to start making piles of isk with my trade finding script again!

Entity
X-Factor Industries
Synthetic Existence
Posted - 2011.06.23 22:16:00 - [166]
 

Originally by: Sullinae
busted for Incarna, needs to be updated for the new cache formats
please update this great tool, I want to start making piles of isk with my trade finding script again!


It'd be nice if you could be a tad more specific?

Sullinae
Posted - 2011.06.25 04:43:00 - [167]
 

Edited by: Sullinae on 25/06/2011 04:43:49
Ah, nevermind. Thought it had something to do with them changing the order ID from 16 to 32 bit ints.

It was just that they took something out of the DBRows... or renamed it:

Getting an error:
AttributeError: 'blue.DBRow' object has no attribute 'issued'

Seems to work other than that now.

Thanks for all your hard work btw, this library rocks my socks.

locJ
Posted - 2011.06.25 08:05:00 - [168]
 

Edited by: locJ on 25/06/2011 09:12:40
Edited by: locJ on 25/06/2011 08:08:02
I Found a Problem
Some market items are ignored. Specifically some subsystems, such as Legion defensive subsystems.

Im using Reverence 1.2.2 with an autodumper Script for Eve market Scanner (ems)

Other then this, great stuff. Keep up the good work

Edit: hmm, ok .. I may be wrong. I think the problem I described above is because I already visited those items before starting the autodumper. This would mean that a "cache-entry" only gets parsed once. Is there a way to clear/refresh the cache ?

Kandreath
Posted - 2011.06.25 11:01:00 - [169]
 

I've worked through a couple of issues but this one has me stumped. I can't see it covered here at all.

Using Python 2.7 (amd64) on Windows 7 (64bit)

I'm trying to compile from source as instructed for a win7-64 setup. I installed vc 2010 and couldn't get it to work. I then un-installed and found vc 2008 and installed that. That got me over the first error.

(I'm including the story in case it's significant).

C:\Users\[...]\Downloads\reverence-1.2.2>C:\Python27\python.exe setup.py install
running install
running build
running build_py
creating build\lib.win-amd64-2.7
creating build\lib.win-amd64-2.7\reverence
copying src\blue.py -> build\lib.win-amd64-2.7\reverence
copying src\cache.py -> build\lib.win-amd64-2.7\reverence
copying src\config.py -> build\lib.win-amd64-2.7\reverence
copying src\const.py -> build\lib.win-amd64-2.7\reverence
copying src\dbutil.py -> build\lib.win-amd64-2.7\reverence
copying src\embedfs.py -> build\lib.win-amd64-2.7\reverence
copying src\exceptions.py -> build\lib.win-amd64-2.7\reverence
copying src\objectCaching.py -> build\lib.win-amd64-2.7\reverence
copying src\strings.py -> build\lib.win-amd64-2.7\reverence
copying src\util.py -> build\lib.win-amd64-2.7\reverence
copying src\_os.py -> build\lib.win-amd64-2.7\reverence
copying src\__init__.py -> build\lib.win-amd64-2.7\reverence
running build_ext
building 'reverence._blue' extension
Traceback (most recent call last):
File "setup.py", line 63, in <module>
package_data = {"reverence": ['*.txt']},
File "C:\Python27\lib\distutils\core.py", line 152, in setup
dist.run_commands()
File "C:\Python27\lib\distutils\dist.py", line 953, in run_commands
self.run_command(cmd)
File "C:\Python27\lib\distutils\dist.py", line 972, in run_command
cmd_obj.run()
File "C:\Python27\lib\distutils\command\install.py", line 563, in run
self.run_command('build')
File "C:\Python27\lib\distutils\cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "C:\Python27\lib\distutils\dist.py", line 972, in run_command
cmd_obj.run()
File "C:\Python27\lib\distutils\command\build.py", line 127, in run
self.run_command(cmd_name)
File "C:\Python27\lib\distutils\cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "C:\Python27\lib\distutils\dist.py", line 972, in run_command
cmd_obj.run()
File "C:\Python27\lib\distutils\command\build_ext.py", line 340, in run
self.build_extensions()
File "C:\Python27\lib\distutils\command\build_ext.py", line 449, in build_exte
nsions
self.build_extension(ext)
File "C:\Python27\lib\distutils\command\build_ext.py", line 499, in build_exte
nsion
depends=ext.depends)
File "C:\Python27\lib\distutils\msvc9compiler.py", line 473, in compile
self.initialize()
File "C:\Python27\lib\distutils\msvc9compiler.py", line 383, in initialize
vc_env = query_vcvarsall(VERSION, plat_spec)
File "C:\Python27\lib\distutils\msvc9compiler.py", line 299, in query_vcvarsal
l
raise ValueError(str(list(result.keys())))
ValueError: [u'path']

Any Idea?

Entity
X-Factor Industries
Synthetic Existence
Posted - 2011.06.25 12:52:00 - [170]
 

Originally by: Kandreath

File "C:\Python27\lib\distutils\msvc9compiler.py", line 299, in query_vcvarsal
l
raise ValueError(str(list(result.keys())))
ValueError: [u'path']

Any Idea?


Hmmm, I looked around a bit, and only found:
http://stackoverflow.com/questions/4676728/value-error-trying-to-install-python-for-windows-extensions

Quote:
If you have a 64 bit Python installation:
Install "Microsoft Visual Studio 2008 Professional Edition" with the "X64 Compiler and Tools" option enabled.


I cannot test this myself yet as i'm using 32bit windows and too lazy to update. (I did buy a new laptop this week, however it's DOA, so so much for that idea)

Entity
X-Factor Industries
Synthetic Existence
Posted - 2011.06.25 12:57:00 - [171]
 

Edited by: Entity on 25/06/2011 22:23:09
Originally by: Sullinae
Getting an error:
AttributeError: 'blue.DBRow' object has no attribute 'issued'

Seems to work other than that now.


Keep in mind CCP can and does change the structure and contents of cache files whenever they feel like. Errors like these are nearly always indicative of such a change.

In this case, CCP simply renamed the "issued" column to "issueDate". Quite far from 'busted for incarna' ;)

Originally by: Sullinae
Thanks for all your hard work btw, this library rocks my socks.


Cheers :)

Kandreath
Posted - 2011.06.25 22:14:00 - [172]
 

Originally by: Entity
Originally by: Kandreath

File "C:\Python27\lib\distutils\msvc9compiler.py", line 299, in query_vcvarsal
l
raise ValueError(str(list(result.keys())))
ValueError: [u'path']

Any Idea?


Hmmm, I looked around a bit, and only found:
http://stackoverflow.com/questions/4676728/value-error-trying-to-install-python-for-windows-extensions

Quote:
If you have a 64 bit Python installation:
Install "Microsoft Visual Studio 2008 Professional Edition" with the "X64 Compiler and Tools" option enabled.


I cannot test this myself yet as i'm using 32bit windows and too lazy to update. (I did buy a new laptop this week, however it's DOA, so so much for that idea)



I tried to use the python uitls offered as an alternative in the stack overflow reply, but this didn't help.
Decided to use the 32bit Python 2.7 install as the 2008 professional edition has only a 90 day trial and I recon I'll want to compile this again after the 90 days. This seemed to do the trick and I'm up and running now.

Thanks for the quick reply it solved my problem.

Entity
X-Factor Industries
Synthetic Existence
Posted - 2011.07.27 01:35:00 - [173]
 

I've just committed a few crash bug fixes.
The fixes are for a reference count bug in virtual column names, and directly nested arbitrary-size containers being prematurely cut because of not clearing the TYPE_MARK state.

These bugs don't affect most (if any at all) users, so no new installers yet.

Note that Reverence is currently incompatible with the build on Singularity due to CCP deploying a shiny new bulkdata management system. The necessary changes have been done to make Reverence work with that, but I will not push these changes until TQ is updated, as things may still change.

Fena Echerie
Posted - 2011.07.29 10:28:00 - [174]
 

Hi,

I just downloaded your lib and tried datadump.py. I see lots of xml files (mostly bulk data) but the market data I want is missing.
Snip of market data related errors:
MethodCall.proxy.marketProxy.GetRegionBest.10000016L ... UNSUPPORTED (dbutil.RowDict)
MethodCall.proxy.marketProxy.GetMarketGroups ... FAILED: Unsupported type // I edited datadump.py to report error type

When looking for a solution I found your last post, saying the windows installers are out of date, so I tried to compile the sources myself. However:
> python setup.py build
<snip>
building 'reverence._blue' extension
error: Unable to find vcvarsall.bat

Since I have Visual Studio 10 Professional installed and already tried adding the location of vcvarsall.bat to the PATH variable, I'm kind of lost now. Any help is appreciated :)

btw: Great work! Reverence makes a much better and cleaner impression than libevecache, which is passing/returning std::vectors by value all the time...

Entity
X-Factor Industries
Synthetic Existence
Posted - 2011.07.29 13:16:00 - [175]
 

Originally by: Fena Echerie
Hi,

I just downloaded your lib and tried datadump.py. I see lots of xml files (mostly bulk data) but the market data I want is missing.
Snip of market data related errors:
MethodCall.proxy.marketProxy.GetRegionBest.10000016L ... UNSUPPORTED (dbutil.RowDict)
MethodCall.proxy.marketProxy.GetMarketGroups ... FAILED: Unsupported type // I edited datadump.py to report error type

When looking for a solution I found your last post, saying the windows installers are out of date, so I tried to compile the sources myself. However:
> python setup.py build
<snip>
building 'reverence._blue' extension
error: Unable to find vcvarsall.bat

Since I have Visual Studio 10 Professional installed and already tried adding the location of vcvarsall.bat to the PATH variable, I'm kind of lost now. Any help is appreciated :)

btw: Great work! Reverence makes a much better and cleaner impression than libevecache, which is passing/returning std::vectors by value all the time...



The dump script was made for bulkdata, not for dynamic stuff like market data. If you want to process market data you will have to write your own code for it (there's several examples in this thread iirc)

Also, to compile it you need VS9 I think. Python's build scripts are picky :)


CheckingAmarr
Posted - 2011.08.12 03:25:00 - [176]
 

I have an old script which looks at market orders and then does some calculations with item blueprints, specifically through looking up cfg.ramtypematerials values. I noticed on page 4 there was some discussion of how ccp had broken that, but was there any real resolution of the issue? How do I access blueprint info for items ingame now? I am terrible at eve and python and not sure of this, and my script is just erroring out with a "cache file not found: 9ba4.cache" error.

I realize this isn't a tech support thread but any comments or examples would be greatly appreciated. Thanks.

Theodor Bliesheimer
Integrated Astromechanics Corp.
Zeta Conglomerate
Posted - 2011.08.24 15:19:00 - [177]
 

Edited by: Theodor Bliesheimer on 25/08/2011 06:23:34
Originally by: Entity
Note that Reverence is currently incompatible with the build on Singularity due to CCP deploying a shiny new bulkdata management system. The necessary changes have been done to make Reverence work with that, but I will not push these changes until TQ is updated, as things may still change.


Could CCP have already released the change you mentioned? Suddenly my market script fails here:

if "GetOldPriceHistory" in key or "GetOrders" in key
...
...

[EDIT]: Mmm, apparently this was a transient error. Deleting the "cache" and "bulkdata" folders in the Eve root fixed it.

Uberfrau
Posted - 2011.08.24 19:56:00 - [178]
 

Edited by: Uberfrau on 24/08/2011 21:07:03
Originally by: Guido Calabresi
I can confirm that the datadump.py script runs properly on Mac OS X with the (more or less) default python install if you manually configure the path to the EVE directory inside the application package.


Can you tell me what you did to make this work? I've tried these:

/Applications/EVE\ Online.app/Contents/Resources/transgaming/c_drive/Program\ Files/CCP/EVE/
/Applications/EVE\ Online.app/Contents/Resources/transgaming/c_drive/Program\ Files/CCP/EVE

I tried symlinking eve in the root (/eve) to see if it was an issue with spaces in the path, but I still get "Could not determine EVE cache folder location."

This is running Incursion on Lion, 10.7.1.


EDIT: I made this change in cache.py:

I commented this out:
#cachepath = os.path.join(home, "Library/Preferences/EVE Online Preferences/p_drive/Local Settings/Application Data/CCP/EVE", cacheFolderName, "cache")

And replaced it with this:
cachepath = os.path.join(home, "Library/Application Support/EVE Online/p_drive/Local Settings/Application Data/CCP/EVE", cacheFolderName, "cache")

Works fine now.

Uberfrau
Posted - 2011.08.24 21:19:00 - [179]
 

Originally by: Manathera
Edited by: Manathera on 18/12/2010 21:13:13
I just discovered reverence and love it, thank you entity. So I thought I'd contribute to this by providing a bit of code I wrote. based on a sample posted by Cyaxares II on the first page. His code just gets the high, low and average price history. My sample gets the entire order and outputs it to a file that's as identical to the market export files as possable.
MarketReader
Please let me know of any errors as this is the first python code I've ever written.


I get a Segmentation fault: 11 :]

Entity
X-Factor Industries
Synthetic Existence
Posted - 2011.08.24 22:15:00 - [180]
 

Originally by: Uberfrau
Originally by: Manathera
Edited by: Manathera on 18/12/2010 21:13:13
I just discovered reverence and love it, thank you entity. So I thought I'd contribute to this by providing a bit of code I wrote. based on a sample posted by Cyaxares II on the first page. His code just gets the high, low and average price history. My sample gets the entire order and outputs it to a file that's as identical to the market export files as possable.
MarketReader
Please let me know of any errors as this is the first python code I've ever written.


I get a Segmentation fault: 11 :]


Quite possibly an issue that's been resolved already. If you can, checkout the repository and build from that. otherwise just wait for the new release (which should be when CCP switches over to the new bulkdata system)


Pages: 1 2 3 4 5 [6] 7

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only