open All Channels
seplocked EVE Technology Lab
blankseplocked Reverence - 100% compatible EVE cache library for Python
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: 1 2 3 4 [5] 6 7

Author Topic

Verite Rendition
Caldari
F.R.E.E. Explorer
EVE Animal Control
Posted - 2011.02.03 14:52:00 - [121]
 

Originally by: Catari Taga
Originally by: Verite Rendition
Originally by: Catari Taga
See MethodCall.server.allianceRegistry.GetRankedAlliances for the alliance list.
What's the trick to getting that to come out of the cache? Even after the client has accessed it, the datadump script isn't handing it over.

I had to look at Entity's example script for that (the things I do Wink). If you comment out lines 94 and 96 of that and it will dump it as MethodCall.server.allianceRegistry.GetRankedAlliances.0.sql.
That did it. Thanks, you're my hero!Cool

Schry Evolon
Sebiestor Tribe
Posted - 2011.02.04 00:54:00 - [122]
 

Originally by: Entity
If anyone wants to give me one of the files that fails to decode on the CFilterRowset... I don't seem to have any of them in my cache.


Assuming you are "ntt" of the GitHub, I have sent it to your email addy.

Entity
X-Factor Industries
Synthetic Existence
Posted - 2011.02.04 07:43:00 - [123]
 

Originally by: Schry Evolon
Originally by: Entity
If anyone wants to give me one of the files that fails to decode on the CFilterRowset... I don't seem to have any of them in my cache.


Assuming you are "ntt" of the GitHub, I have sent it to your email addy.


Hmm that one works here. I guess it's one of the smaller changes I haven't put into a release yet.

Schry Evolon
Sebiestor Tribe
Posted - 2011.02.04 21:07:00 - [124]
 

Originally by: Entity
Originally by: Schry Evolon
Originally by: Entity
If anyone wants to give me one of the files that fails to decode on the CFilterRowset... I don't seem to have any of them in my cache.


Assuming you are "ntt" of the GitHub, I have sent it to your email addy.


Hmm that one works here. I guess it's one of the smaller changes I haven't put into a release yet.


Emailed you again.

If I can help, I would like to assist and test the unpublished updates to see if the resolution is within.

Entity
X-Factor Industries
Synthetic Existence
Posted - 2011.02.07 09:19:00 - [125]
 

I've committed a potential fix for the CFilterRowset problem.
Update and build from source if you want to test it.

Note that I don't actually have any such files that contain a populated table, so I don't know if it will work on those.

Guess you'll find out soon enough ;)

Schry Evolon
Sebiestor Tribe
Posted - 2011.02.08 00:11:00 - [126]
 

Originally by: Entity
I've committed a potential fix for the CFilterRowset problem.
Update and build from source if you want to test it.

Note that I don't actually have any such files that contain a populated table, so I don't know if it will work on those.

Guess you'll find out soon enough ;)


Everything works in the new build, so far.
Very Happy

Verite Rendition
Caldari
F.R.E.E. Explorer
EVE Animal Control
Posted - 2011.02.08 04:59:00 - [127]
 

If it's of any use to you Entity, Reverence works fine on Serenity (China) once you patch the cache directory locations.

Default client location: C:\Program Files\Optic\EVE

Default cache directory location: ...\CCP\EVE\c_program_files_optic_eve_tel.eve.gtgame.com.cn

MachoNet version is 219. They're still on Apocrypha (6.14.179118).

Nikolai Kondratiev
Sphere Design Inc.
Posted - 2011.02.13 10:30:00 - [128]
 

Edited by: Nikolai Kondratiev on 15/02/2011 17:51:11
Originally by: Entity
I've committed a potential fix for the CFilterRowset problem.
Update and build from source if you want to test it.

Note that I don't actually have any such files that contain a populated table, so I don't know if it will work on those.

Guess you'll find out soon enough ;)


I tried that new build (pretty sure I compiled new build correctly, python n00b here) with some cache file that would cause the CFilterRowset error. But instead of returning an error, python (windows version 2.7) just crashes when loading CachedMethodCalls on a test script.

Any idea on what could cause the crash ?

Edit : might have been a bugged file, didn't get the error again, can now go back to kill my brain trying to extract some data.

Papa Yoru
Electric Monk inc
Redrum Fleet
Posted - 2011.02.13 16:22:00 - [129]
 

Not sure if this is helpful, but EVE-Centrals uploader also fails now with the CFilterRowset issue. It is definitely something new since Incursions patchs. The only workaround i can find is to clear my cache each time I encounter this.

Thart
U.K.R.A.I.N.E
SOLAR FLEET
Posted - 2011.03.10 10:18:00 - [130]
 

Edited by: Thart on 10/03/2011 10:37:25
Users of my program reports problem with cache reader based on Reverence.
Quote:
[error] Updating market orders (cache)...
[error] Error calling cache import tool. Code: 255


Quote:
Traceback (most recent call last):
File "evemarketordersfromcache.py", line 49, in <module>
AttributeError: 'module' object has no attribute 'StackSize'


I got this problem too.
Here is a two files from eve online cache whih cause a problem - link

Upd: Line 49 content:
Quote:
what,obj=blue.marshal.Load(open(filename,"rb").read())

Entity
X-Factor Industries
Synthetic Existence
Posted - 2011.03.10 12:00:00 - [131]
 

Originally by: Thart
Edited by: Thart on 10/03/2011 10:37:25
Users of my program reports problem with cache reader based on Reverence.
Quote:
[error] Updating market orders (cache)...
[error] Error calling cache import tool. Code: 255


Quote:
Traceback (most recent call last):
File "evemarketordersfromcache.py", line 49, in <module>
AttributeError: 'module' object has no attribute 'StackSize'



The changes to the inventory system in latest patch caused this. looking into it.

Entity
X-Factor Industries
Synthetic Existence
Posted - 2011.03.10 20:57:00 - [132]
 

I've pushed a fix for the inventory system changes. Update/rebuild your copy of Reverence from source to try it out. (No new installer of yet).

Thanks to Thart for providing test files :)

For the technical:
The fix was not trivial; CCP gave the DBRow subsystem a new feature that basically implements Python properties.

Papa Yoru
Electric Monk inc
Redrum Fleet
Posted - 2011.03.17 15:59:00 - [133]
 

Thanks for the update :)

Theodor Bliesheimer
Integrated Astromechanics Corp.
Zeta Conglomerate
Posted - 2011.03.19 00:31:00 - [134]
 

Any estimate when an updated installer will be released?

Entity
X-Factor Industries
Synthetic Existence
Posted - 2011.03.19 13:02:00 - [135]
 

Originally by: Theodor Bliesheimer
Any estimate when an updated installer will be released?


Reason I'm not doing this as frequently as I should (read: every build) is because it's a pain to manually upload files to github.
I'll upload new installers when I've finished figuring out their upload mechanics so I can script it :P

Zeta Zhul
Caldari
Preemptive Paranoia
Posted - 2011.03.21 04:09:00 - [136]
 

A couple things.

1. The sample market history script on pastie.com does a walk through all directories, subdirectories, etc and deletes every single file.

Is there actually a reason why it deletes every single file regardless of the extension or type?

Because it seems to me that being that the output is in fact a CSV file that perhaps not deleting every last thing in the directory tree might be a worthwhile consideration.

2. If you are planning on using that sample script be advised that if you output the CSV file to your "C:/" root directory that you will in fact be committing computer suicide.

sigh.

3. No I didn't output to the root directory. But I did output to a directory with some subdirectories that had stuff in it that I would rather have kept. Oh well. Sometimes necessity leads to refactoring crappy code.

Entity
X-Factor Industries
Synthetic Existence
Posted - 2011.03.21 11:47:00 - [137]
 

Originally by: Zeta Zhul
A couple things.

1. The sample market history script on pastie.com does a walk through all directories, subdirectories, etc and deletes every single file.

Is there actually a reason why it deletes every single file regardless of the extension or type?

Because it seems to me that being that the output is in fact a CSV file that perhaps not deleting every last thing in the directory tree might be a worthwhile consideration.

2. If you are planning on using that sample script be advised that if you output the CSV file to your "C:/" root directory that you will in fact be committing computer suicide.

sigh.

3. No I didn't output to the root directory. But I did output to a directory with some subdirectories that had stuff in it that I would rather have kept. Oh well. Sometimes necessity leads to refactoring crappy code.


Wrong thread?

Zeta Zhul
Caldari
Preemptive Paranoia
Posted - 2011.03.21 15:29:00 - [138]
 

Originally by: Entity
Originally by: Zeta Zhul
A couple things.

1. The sample market history script on pastie.com does a walk through all directories, subdirectories, etc and deletes every single file.

Is there actually a reason why it deletes every single file regardless of the extension or type?

Because it seems to me that being that the output is in fact a CSV file that perhaps not deleting every last thing in the directory tree might be a worthwhile consideration.

2. If you are planning on using that sample script be advised that if you output the CSV file to your "C:/" root directory that you will in fact be committing computer suicide.

sigh.

3. No I didn't output to the root directory. But I did output to a directory with some subdirectories that had stuff in it that I would rather have kept. Oh well. Sometimes necessity leads to refactoring crappy code.


Wrong thread?


No actually it's the right thread.

It's this script. I'm sure you're aware of it and probably familiar with it. I just wanted to warn people that the script, as is, will conduct a complete directory walk and delete every single file regardless of type. So if you're going to run this script you must ensure that the output directory doesn't have anything important *and* neither do any subdirectories.

Offending code block:
Quote:

print "Deleting old records... \n"
for root, dirs, files in os.walk(OUTPATH, topdown=False):
for name in files:
os.remove(os.path.join(root, name))



I will fully admit that I should have review the whole thing, line by line, before running it. But as the script merely outputs a CSV file it simply did not occur to me that "Deleting old records..." would actually involve deleting everything including old records and pretty much anything else lying around.

Alaya Carrier
Posted - 2011.04.18 10:00:00 - [139]
 

Edited by: Alaya Carrier on 18/04/2011 10:00:23
Is this project dead like libevecache seems to be?

It's easy for me to build the Python part but it's very messy for me to get VC in the computer where I have EvE and Python installed so I can't really build off the git source. Crying or Very sad

Entity
X-Factor Industries
Synthetic Existence
Posted - 2011.04.18 13:44:00 - [140]
 

Originally by: Alaya Carrier
Edited by: Alaya Carrier on 18/04/2011 10:00:23
Is this project dead like libevecache seems to be?

It's easy for me to build the Python part but it's very messy for me to get VC in the computer where I have EvE and Python installed so I can't really build off the git source. Crying or Very sad


It's not dead. Most of my projects simply require very little to no maintenance, even across patches.

Alaya Carrier
Posted - 2011.04.18 19:52:00 - [141]
 

Edited by: Alaya Carrier on 18/04/2011 19:59:48
Originally by: Entity
Originally by: Alaya Carrier
Edited by: Alaya Carrier on 18/04/2011 10:00:23
Is this project dead like libevecache seems to be?

It's easy for me to build the Python part but it's very messy for me to get VC in the computer where I have EvE and Python installed so I can't really build off the git source. Crying or Very sad


It's not dead. Most of my projects simply require very little to no maintenance, even across patches.


Well, I made a script that worked for a long time with the previous version. Then a recent EvE patch broke it and it'd show errors with syntax like:


for blah in NewPriceHistory:
for history_item in blah['lret']:
print format_history_row(history_item)


where it complained about blah['lret'] expecting an integer instead of a string on such associative array subscript.

Exact error:


Traceback (most recent call last):
File "C:\Python26\EvEDump.py", line 54, in <module>
for history_item in blah['lret']:
TypeError: string indices must be integers, not str



I uninstalled the library.
I downloaded the git current release, rebuilt, installed with no errors and...

... now I still get the same associative array error.


What gives?

Entity
X-Factor Industries
Synthetic Existence
Posted - 2011.04.19 21:00:00 - [142]
 

Edited by: Entity on 19/04/2011 21:00:45
Originally by: Alaya Carrier

Traceback (most recent call last):
File "C:\Python26\EvEDump.py", line 54, in <module>
for history_item in blah['lret']:
TypeError: string indices must be integers, not str



I uninstalled the library.
I downloaded the git current release, rebuilt, installed with no errors and...

... now I still get the same associative array error.

What gives?


I'd wager a guess that the structure of that object changed when they patched.
I do not control how CCP represents data in cache. This is not a bug.

Portaprince
Babylon Creations
Posted - 2011.04.22 17:58:00 - [143]
 

Awesome work!!! I now have a .NET version of your cache parser (dumper.exe) working. I will release some code soon. It's really just a wrapper for dumper.exe that allows it to be used in .NET. Named "DumpWrapper" of course! Simply point it at a file (.cache files are bestVery Happy) and it hands you back a datatable, xml file, or csv string of the content. I can only compile it for Windows, but someone else could probably make a version for mac and linux...should they desire to stoop to the level of .NET Twisted Evil
Of course you know this will open the door to more apps!! I've currently only implemented the marketdata portion, but don't see any reason the rest of dumper functionality couldn't be implemented.
Funny thing is I tried 4 different ways to get that data:
1) Read the memory from the OS and decode...(yeah, like i have the time to play low level memory games in .NETShocked)
2) Reverse engineer the cache file / Python code... (again, not so easy from .NET)
3) Intercept the OS window handle and try and pull the control caption (not possible, because it doesn't use OS controls)
4) 15 lines of code to create a process and execute dumper on it while capturing the stdout!!!Cool

Sverre Haakonson
Gallente
Posted - 2011.04.24 20:57:00 - [144]
 

Has someone a binary build for windows? Because of python 2.7 was build with vc2008 and I can only access vc2010 it will not work for me.


Entity
X-Factor Industries
Synthetic Existence
Posted - 2011.04.25 10:35:00 - [145]
 

Edited by: Entity on 25/04/2011 10:35:35
Originally by: Sverre Haakonson
Has someone a binary build for windows? Because of python 2.7 was build with vc2008 and I can only access vc2010 it will not work for me.




Originally by: Entity

DOWNLOAD
The project is available on github here:
http://github.com/ntt/reverence

There's a windows installer and source distribution in the downloads section.


Rolling Eyes

Sverre Haakonson
Gallente
Posted - 2011.04.25 22:49:00 - [146]
 

Originally by: Entity
Edited by: Entity on 25/04/2011 10:35:35
Originally by: Sverre Haakonson
Has someone a binary build for windows? Because of python 2.7 was build with vc2008 and I can only access vc2010 it will not work for me.




Originally by: Entity

DOWNLOAD
The project is available on github here:
http://github.com/ntt/reverence

There's a windows installer and source distribution in the downloads section.


Rolling Eyes


Yes you are right. Shame on me. ugh

Moto Akimoto
Posted - 2011.05.03 09:52:00 - [147]
 

Just FYI, I finally have Reverence running on my Mac OSx. Yay! I'm not a programmer so it took me a while to figure out.

Thanks Entity, this is good stuff!

Zeta Zhul
Caldari
Preemptive Paranoia
Posted - 2011.05.07 20:33:00 - [148]
 

Originally by: Moto Akimoto
Just FYI, I finally have Reverence running on my Mac OSx. Yay! I'm not a programmer so it took me a while to figure out.

Thanks Entity, this is good stuff!


Hey that's great!

BTW I have an updated python script that includes the typeid in the market history extract. What I do is run through a whole list of market items, then run the extract script and then import it all into a MySQL database. It is a lot simpler to have the typeid on every row rather than having to parse out the name of the file.

Let me know if you want it.

Mike'P
Posted - 2011.05.16 17:48:00 - [149]
 

Hi guys,

(although this will probably only make sense to Entity!)

I've been playing around with this on linux, and noticed that I would sometimes get a segmentation violation running LoadCacheFolder("CachedMethodCalls") from python.

I narrowed it down to one cache file (d03c.cache - it has the string 'charGetNewTransactions' in the header) and playing around with src/blue/marshal.c, I notice that it is something to do with the mapping of shared objects, and in particular, the shared_map used in the STORE/RESERVE_SLOT/UPDATE_SLOT macros in marshal.c.

The 'shared_map' table is an array of int32_t values, and is basically the last thing in the cache file. In marshal.c, the table is initialised by pointing to the last x * 4 bytes in the cache file, where x is the number of shared objects in the file (read from the file itself).

With a hex dumper, I was able to see that the tricky cache file didn't appear to have enough bytes at the end to make up a whole number of int32_t types - eg, where it should have had 8 bytes to give 2 x 4 byte integers, it only had six. Hence, share_map was being parked 2 bytes too early in the file, and was picking up an 0x6969 value (in my case) as the LSB, which made the value of shared_map[0] a large number, and hence made

shared_obj[shared_map[0]]

cause a seg fault.

Anyone else seen this ?

I've got a dirty little hack to work around what I think is going on, but as I don't understand the ins and outs of the data structures, I'm hesitant to push it out there as a solution.

Hopefully that makes sense to someone....

Entity
X-Factor Industries
Synthetic Existence
Posted - 2011.05.17 02:09:00 - [150]
 

Originally by: Mike'P
Hi guys,

(although this will probably only make sense to Entity!)



Interesting. Can you put this file somewhere I can download it?


Pages: 1 2 3 4 [5] 6 7

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only