open All Channels
seplocked EVE Technology Lab
blankseplocked Reverence - 100% compatible EVE cache library for Python
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: 1 [2] 3 4 5 6 7

Author Topic

Entity
X-Factor Industries
Synthetic Existence
Posted - 2010.05.15 13:58:00 - [31]
 

Edited by: Entity on 15/05/2010 13:59:03
Originally by: cinderbrood
Edited by: cinderbrood on 15/05/2010 04:38:11
Will we ever see a pure python version of this without the c/c++ extensions?

(I want to run this on IronPython) but alas they lack .pyd support.

Ie will we ever see a pure python version of your blue.pyd?

would open up cache support to the .net world as well. (albeit in a hacky manner)


Heh. It actually used to be pure Python, it was mostly a pet research project.
Worked fine, and I've used it like that for a long time, but there's a couple of problems with it:

  • I'd end up having to maintain the same thing in two different languages.

  • Memory usage goes through the roof because of Python data structure overhead.

  • Without acceleration (Psyco jit compiler) it is painfully slow.



cinderbrood
Caldari
Knights of Nii
The 20 Minuters
Posted - 2010.05.15 20:40:00 - [32]
 

Aww Damn it :P Speed wasnt an issue for me either ( Within reason i guess )
As for memory usage.. Yeah thats not going to get any better with IronPython i guess. Any chance of the old release being archived somewhere for those of us with pet projects themselfs that want to hack at it :P (Short of writing an entire .net version )

Writing my own cache reading libs was something im trying to avoid as its just an addon. not a required part of my little play toy.

Miss AmarrPriceCheck
Posted - 2010.05.26 21:42:00 - [33]
 

On a previously working script, I now have the following error post-Tyrannis:
Quote:

Traceback (most recent call last):
File "C:\Documents and Settings\User\Desktop\scriptname.py", line 54, in <module>
if str(cfg.evelocations.Get(key[2]).locationName) == "Lonetrek":
File "C:\Python26\lib\site-packages\reverence\config.py", line 338, in __get__
value = self.method(obj)
File "C:\Python26\lib\site-packages\reverence\config.py", line 606, in evelocations
self._loadfrombulk("config.StaticLocations", rs, hint=1)
File "C:\Python26\lib\site-packages\reverence\config.py", line 734, in _loadfrombulk
rs.lines.extend(obj.lines)
AttributeError: 'CRowset' object has no attribute 'lines'



Sorry the whitespace formatting in quoted error is obviously wrong, but you get the idea.

Kadesh Priestess
Scalding Chill
Posted - 2010.05.26 22:27:00 - [34]
 

Entity updated reverence to work with tyrannis - you need to get and install latest version from git or if you prefer installer - wait for an update. Git version works perfectly for me.

Jita Grrl
Posted - 2010.06.13 12:06:00 - [35]
 

Originally by: Entity
Edited by: Entity on 03/10/2009 02:09:45
  • 100% compatibility with all bulkdata, cache and settings files.



  • What exactly do you mean when you said settings files?

    I'm mainly looking to pull window sizes and positions from the cache. It looks like that data should be stored in the various core_char_XXXXXXXX.dat files, but I'm not certain.

    Is this something Reverence can do or am I barking up the wrong tree?

    Entity
    X-Factor Industries
    Synthetic Existence
    Posted - 2010.06.13 19:19:00 - [36]
     

    Originally by: Jita Grrl
    Originally by: Entity
    Edited by: Entity on 03/10/2009 02:09:45
  • 100% compatibility with all bulkdata, cache and settings files.



  • What exactly do you mean when you said settings files?

    I'm mainly looking to pull window sizes and positions from the cache. It looks like that data should be stored in the various core_char_XXXXXXXX.dat files, but I'm not certain.

    Is this something Reverence can do or am I barking up the wrong tree?


    Yeah it can do that, simply do something like

    obj = blue.marshal.Load(contents_of_settings_file)


    settings files are mostly just dictionary mappings.

    Entity
    X-Factor Industries
    Synthetic Existence
    Posted - 2010.06.17 13:35:00 - [37]
     

    Edited by: Entity on 17/06/2010 13:35:44
    A windows installer and source distribution for version 1.1.0 are now available (see OP for location) for your convenience.

    Version 1.1.0 fixes various bugs and other issues and is compatible with Tyrannis, and adds a couple of features.
    The most useful change being that DBRows now have a string representation, which is handy for debugging and checking out table contents.

    Check the commit history on github for more details.

    Note: Users already using the current version directly from the repository don't have to do anything.

    Veevi Fenikusu
    Minmatar
    Fenikusu Trade
    Posted - 2010.07.09 11:38:00 - [38]
     

    This library is truly amazing, great work!

    Veevi Fenikusu
    Minmatar
    Fenikusu Trade
    Posted - 2010.07.17 14:44:00 - [39]
     

    Do have a question about using this! If you want to use this to monitor checking market orders, should you first browse everything you want to import, then run the cachemgr stuff, run it each time the cache folder changes, or run it at certain intervals(I'm doing it like this now, but it got to run through the entire cache everytime, and it gets bigger...)

    Entity
    X-Factor Industries
    Synthetic Existence
    Posted - 2010.07.17 15:47:00 - [40]
     

    Originally by: Veevi Fenikusu
    Do have a question about using this! If you want to use this to monitor checking market orders, should you first browse everything you want to import, then run the cachemgr stuff, run it each time the cache folder changes, or run it at certain intervals(I'm doing it like this now, but it got to run through the entire cache everytime, and it gets bigger...)


    I would just monitor the folder for new files and only check those after examining the existing ones.
    Also, you can decode the files manually with blue.marshal.Load(data) instead of using cachemgr.
    A further optimization would be to just inspect the binary data and see whether it's a market order before loading/decoding it.

    Gogo cyno
    Posted - 2010.07.19 16:45:00 - [41]
     

    I was playing around with this and had a question about the cachemgr's GetCacheFileName method:

    Through time stamps and various other things, I know that a file (say 6047.cache) is the file for corp orders (key ('marketProxy', 'GetCorporationOrders', 1234) after loading the folder). How do I call GetCacheFileName (or thus any of the load cached file ones) to access that file?

    Entity
    X-Factor Industries
    Synthetic Existence
    Posted - 2010.07.20 12:06:00 - [42]
     

    Originally by: Gogo cyno
    I was playing around with this and had a question about the cachemgr's GetCacheFileName method:

    Through time stamps and various other things, I know that a file (say 6047.cache) is the file for corp orders (key ('marketProxy', 'GetCorporationOrders', 1234) after loading the folder). How do I call GetCacheFileName (or thus any of the load cached file ones) to access that file?


    GetCacheFileName is kind of for internal usage and manual loading (by loading the file and decoding with blue.marshal.Load). If you know the cache key of a file (the tuple you mentioned), you can do this if the file is in CachedMethodCalls:

    obj = cache.LoadCachedMethodCall( ('marketProxy', 'GetCorporationOrders', 1234) )


    and this if it's in CachedObjects:

    obj = cache.LoadCachedObject( ('marketProxy', 'GetCorporationOrders', 1234) )


    Note that LoadCacheFolder is a one-shot thing and it actually loads all the data at once, so it is very inefficient if you only use one object. The method is provided mostly for debugging/dumping.


    Gogo cyno
    Posted - 2010.07.20 15:29:00 - [43]
     

    Edited by: Gogo cyno on 20/07/2010 15:36:10
    Originally by: Entity

    obj = cache.LoadCachedMethodCall( ('marketProxy', 'GetCorporationOrders', 1234) )




    Alas, I was hoping I was just doing it wrong. Whenever I try to load a specific method call in the above manner, I get a No Such File or Directory error and the filename will be different than what I know the cache file is. For example,

    r = cachemgr.LoadCachedMethodCall( ('marketProxy', 'GetOrders', 10000002, 40) )


    will produce that error. I know that method call is there because I can access it through loading the entire folder, but trying to load it specifically does not work.

    cmc[('marketProxy', 'GetOrders', 10000002, 40)]


    will return the correct information (with
    cmc = cachemgr.LoadCacheFolder("CachedMethodCalls")
    )

    Saphira Khan
    The Wretched.
    Posted - 2010.07.30 16:44:00 - [44]
     

    I might have a simular problem.
    When i do the following:
    cmc = LoadCacheFolder("CachedMethodCalls")
    for key, obj in cmc.iteritems():
    filename = cachemgr.GetCacheFileName(key)

    it returns a filename but this file doesnt exist anywhere in my cache folders.
    Any idea how i can solve this?
    I ideally want to remove cache files after i used them so i wont have to parse / send them again after i've already processed them.

    Entity
    X-Factor Industries
    Synthetic Existence
    Posted - 2010.08.02 18:52:00 - [45]
     

    Originally by: Saphira Khan
    I might have a simular problem.
    When i do the following:
    cmc = LoadCacheFolder("CachedMethodCalls")
    for key, obj in cmc.iteritems():
    filename = cachemgr.GetCacheFileName(key)

    it returns a filename but this file doesnt exist anywhere in my cache folders.
    Any idea how i can solve this?
    I ideally want to remove cache files after i used them so i wont have to parse / send them again after i've already processed them.


    Hm it looks like the CachedMethodCalls folder uses a different hash key than the CachedObjects one which works perfectly. I'll look into it.

    Entity
    X-Factor Industries
    Synthetic Existence
    Posted - 2010.08.02 20:04:00 - [46]
     

    Originally by: Entity
    Originally by: Saphira Khan
    I might have a simular problem.
    When i do the following:
    cmc = LoadCacheFolder("CachedMethodCalls")
    for key, obj in cmc.iteritems():
    filename = cachemgr.GetCacheFileName(key)

    it returns a filename but this file doesnt exist anywhere in my cache folders.
    Any idea how i can solve this?
    I ideally want to remove cache files after i used them so i wont have to parse / send them again after i've already processed them.


    Hm it looks like the CachedMethodCalls folder uses a different hash key than the CachedObjects one which works perfectly. I'll look into it.



    Hm, it looks like it's actually a decoding problem, but it's so incredibly subtle I don't understand why it's failing.

    The problem: cPickle produces a different pickle string for two seemingly identical tuples.

    Example: cachedmethodcall for t2 expanders in Forge:

    ('marketProxy', 'GetOrders', 10000002, 1319)

    GetCacheFileName produces "986f.cache" for the key pulled from the cache file, but if you do GetCacheFileName(('marketProxy', 'GetOrders', 10000002, 1319)) manually, it produces the correct "f42f.cache".

    The only difference here are the objectIDs differing. Even comparing the key to the constant ('marketProxy', 'GetOrders', 10000002, 1319) produces True.

    This shouldn't really matter, but apparently it does.

    A workaround is to re-tuple the key by doing key = key[:] before using it.

    At the moment I have no idea why it behaves like that, and I'm tempted to use that workaround in the library.



    Gogo cyno
    Posted - 2010.08.02 20:38:00 - [47]
     

    Originally by: Entity

    GetCacheFileName produces "986f.cache" for the key pulled from the cache file, but if you do GetCacheFileName(('marketProxy', 'GetOrders', 10000002, 1319)) manually, it produces the correct "f42f.cache".




    >> cachemgr.GetCacheFileName(('marketProxy', 'GetOrders', 10000002, 1319))
    'd58e.cache'


    And yet another result. I tried using the workaround you mentioned with no success.

    Entity
    X-Factor Industries
    Synthetic Existence
    Posted - 2010.08.03 00:56:00 - [48]
     

    Originally by: Gogo cyno
    Originally by: Entity

    GetCacheFileName produces "986f.cache" for the key pulled from the cache file, but if you do GetCacheFileName(('marketProxy', 'GetOrders', 10000002, 1319)) manually, it produces the correct "f42f.cache".




    >> cachemgr.GetCacheFileName(('marketProxy', 'GetOrders', 10000002, 1319))
    'd58e.cache'


    And yet another result. I tried using the workaround you mentioned with no success.


    I get the distinct feeling cPickle has some undocumented behaviour.

    what does cPickle.dumps(('marketProxy', 'GetOrders', 10000002, 1319)) give?

    Entity
    X-Factor Industries
    Synthetic Existence
    Posted - 2010.08.03 01:35:00 - [49]
     

    oh wow. Found the cause of the problem.
    cPickle is not stable for given input depending on refcounts.

    Originally by: picklebug.py
    import cPickle

    reffed = "xKITTENSx"[1:-1]
    print repr(cPickle.dumps(reffed))

    print repr(cPickle.dumps("xKITTENSx"[1:-1]))



    Produces:

    F:\>picklebug.py
    "S'KITTENS'\np1\n."
    "S'KITTENS'\n."

    Note the extra "p1\n" in the 1st case.

    Very strange... Not sure how to solve this. seems to be a cPickle issue.

    More worryingly, since the method I use to generate these cache filenames is exactly identical to what EVE does, I wonder whether EVE would have the same problem in some cases.

    Gogo cyno
    Posted - 2010.08.03 14:30:00 - [50]
     

    Using pickle seems to produce consistent results:


    import pickle

    reffed = "xKITTENSx"[1:-1]
    print repr(pickle.dumps(reffed))

    print repr(pickle.dumps("xKITTENSx"[1:-1]))


    produces:


    "S'KITTENS'\np0\n."
    "S'KITTENS'\np0\n."

    Entity
    X-Factor Industries
    Synthetic Existence
    Posted - 2010.08.03 15:26:00 - [51]
     

    Yep, but not identical to cPickle.
    Consistency doesn't help if the string is still different, as the hash code will be different too.

    Entity
    X-Factor Industries
    Synthetic Existence
    Posted - 2010.08.03 18:09:00 - [52]
     

    Update

    I've found evidence of EVE also being affected by this, as I have found cache entries for calls to the same method, with different arguments, that exhibit the same pickling inconsistency:

    "(S'lookupSvc'\np1\nS'LookupOwners'\np2\nVX-Factor Ind\nI0\ntp3\n." <- only three PUTs
    "(S'lookupSvc'\np1\nS'LookupOwners'\np2\nVNTT\np3\nI0\ntp4\n." <- four PUTs

    So, it's pretty much impossible to know whether EVE held more than 1 reference to a string when it hashed the cache key, and as such there's no good way to solve this problem.

    Recursively collecting all elements of the key into a new list so that their reference count is going to be at least 2 before using GetCacheFileName() does seem to fix it for most of the cache entries, but there's still going to be a handful that fail to produce the correct hash.

    Try the following workaround, but note that you should only do this for the CachedMethodCalls folder.


    def collect(key):
    foo = []
    _collect((key,), foo.append)
    return foo

    def _collect(items, append):
    for element in items:
    if type(element) in (list, tuple):
    _collect(element, append)
    append(element)

    foo = collect(key)
    filename = cm.GetCacheFileName(key)



    Guess I'll be poking some people in CCP about this.

    Catari Taga
    Centre Of Attention
    Middle of Nowhere
    Posted - 2010.08.03 18:22:00 - [53]
     

    Originally by: Entity
    Guess I'll be poking some people in CCP about this.


    Please follow the normal procedure and explain the bug to a bughunter first. Laughing

    Entity
    X-Factor Industries
    Synthetic Existence
    Posted - 2010.08.03 18:55:00 - [54]
     

    Originally by: Catari Taga
    Originally by: Entity
    Guess I'll be poking some people in CCP about this.


    Please follow the normal procedure and explain the bug to a bughunter first. Laughing


    Shocked

    Tonto Auri
    Vhero' Multipurpose Corp
    Posted - 2010.08.03 20:22:00 - [55]
     

    Pam-pam-pam... This could likely be the case for "stuck" corporation assets listings.

    SmallBrain
    Posted - 2010.08.03 21:58:00 - [56]
     

    Originally by: Saphira Khan
    I might have a simular problem.
    When i do the following:
    cmc = LoadCacheFolder("CachedMethodCalls")
    for key, obj in cmc.iteritems():
    filename = cachemgr.GetCacheFileName(key)

    it returns a filename but this file doesnt exist anywhere in my cache folders.
    Any idea how i can solve this?
    I ideally want to remove cache files after i used them so i wont have to parse / send them again after i've already processed them.

    A bug still being a bug, the filename of course should exist, but I solved this differently. This snippet might help other people too:


    def LoadOurCacheFolder(cache, name):
    crap = {}
    if cache.machoVersion > 181 and name.lower() == "bulkdata":
    name = cache.bulkdatapath
    else:
    name = os.path.join(cache.machocachepath, name)
    for filename in glob.glob(os.path.join(name, "*.cache")):
    try:
    what, obj = blue.marshal.Load(open(filename, "rb").read())
    except IOError:
    continue
    obj['filename'] = filename
    crap[what] = obj
    return crap


    This assigns the filename to the object too. Then I just do:


    cmc = LoadOurCacheFolder(cache, "CachedMethodCalls")
    for key, obj in cmc.iteritems():
    os.unlink(obj['filename'])


    Well .. the unlink is done when I know for sure the data is posted, but you get the idea :) So now the globl always returns new files. And a nice side-effect: it keeps the cache-dirs relative empty, and keep it from consuming tons of MiB when you scan the complete market in a few regions ;)

    Entity
    X-Factor Industries
    Synthetic Existence
    Posted - 2010.08.03 22:21:00 - [57]
     

    yep, that works too.

    I've send off some mails to the right people so hopefully the bug is going to be fixed sometime Soon™ and i'll update Reverence :)

    Saphira Khan
    The Wretched.
    Posted - 2010.08.11 17:56:00 - [58]
     

    Thanks for the help all, will try this soon!

    Induc
    Amarr
    Posted - 2010.08.17 20:10:00 - [59]
     

    Now with Python 2.7 released as the latest stable version, is it possible if Reverence could be updated to support it?

    Entity
    X-Factor Industries
    Synthetic Existence
    Posted - 2010.08.18 12:32:00 - [60]
     

    Originally by: Induc
    Now with Python 2.7 released as the latest stable version, is it possible if Reverence could be updated to support it?


    Working on it. It compiles for 2.7 it seems. Will have to check EVE first once it's up, to see if the marshal format and cache key hashing are still the same.

    Update Soon™


    Pages: 1 [2] 3 4 5 6 7

    This thread is older than 90 days and has been locked due to inactivity.


     


    The new forums are live

    Please adjust your bookmarks to https://forums.eveonline.com

    These forums are archived and read-only