open All Channels
seplocked EVE Technology Lab
blankseplocked Alliance/Corp Standings export
 
This thread is older than 90 days and has been locked due to inactivity.


 
Author Topic

postal dude
Caldari
Thundercats
RAZOR Alliance
Posted - 2008.03.25 22:19:00 - [1]
 

As I understand there is no standings export via the API or the eve-online website.

I poked around in the eve-cache folders and stumbled over a file with the corp standings in it. Before I start to implement an app in C# to parse the file: Is it legal or an eula-violation? Has somebody already wrote such an app?

CCP Lingorm


C C P
Posted - 2008.03.26 08:51:00 - [2]
 

As long as you do not modify the file in question it is technically not against the EULA or TOS. It is only a cached version of a database method call.

If you do try to change it it will not actually effect the corp standings only the data YOU see in your client (which could cause problems for you).

In short : Yes you can do it, please don't change the file.

postal dude
Caldari
Thundercats
RAZOR Alliance
Posted - 2008.03.26 10:10:00 - [3]
 

Edited by: postal dude on 26/03/2008 10:11:16
Originally by: CCP Lingorm

In short : Yes you can do it, please don't change the file.



Thank you very much! It was easy to implement the file-format. The only problem is, that only the standings from your corp are cached, but thats good enough for me:

Please visit your user settings to re-enable images.

If anybody wants the source, Iam willing to upload it Wink

CCP Lingorm


C C P
Posted - 2008.03.26 10:28:00 - [4]
 

I wouldn't mind a copy.

Salvis Tallan
Gallente
Sparse Enterprises
Posted - 2008.03.26 11:27:00 - [5]
 

Originally by: CCP Lingorm
I wouldn't mind a copy.


QFT. I like that and would love a copy of the code.

Ix Forres
Caldari
Righteous Chaps
Posted - 2008.03.26 11:57:00 - [6]
 

Originally by: CCP Lingorm
As long as you do not modify the file in question it is technically not against the EULA or TOS. It is only a cached version of a database method call.

If you do try to change it it will not actually effect the corp standings only the data YOU see in your client (which could cause problems for you).

In short : Yes you can do it, please don't change the file.



Does this apply to all cache data? I'm thinking market data, if that's cached when browsed. Would make for a much more effective market report uploader for a site I'm working on... Smile

Pronas
Scoopex
Majesta Empire
Posted - 2008.03.26 11:57:00 - [7]
 

nice tools

would like a copy too (as well as source ;)


Vessper
Indicium Technologies
Hephaestus Forge Alliance
Posted - 2008.03.26 12:05:00 - [8]
 

Great idea.

I'd love a copy of the app and the source if available please!

CCP Lingorm


C C P
Posted - 2008.03.26 12:07:00 - [9]
 

As long as you do not modify the cache files then you are free to read them and write tools for them. Of course we do not support these tools (*grin*) and if we change the cache file structure of methods they may break, not that I see this happening but it is possible.

And to make perfectly clear. If you modify them then your client may break.

postal dude
Caldari
Thundercats
RAZOR Alliance
Posted - 2008.03.26 13:05:00 - [10]
 

Okay then, here it is. I release this under the GNU GPLv3.

Please note, this is only for reading the standings out of the cache, if you want to parse other cachefiles, this program is no use for you.

Usage: You have to point the app to your cache folder. It is somewhere in you application-data-folder (OS-specific). The folder you must browse to looks like this: "[...]\cache\MachoNet\87.237.38.200\189\CachedMethodCalls".
Then just press "Read Standings". Please note it reads the standings from the first standings-cache file it finds, so make sure there is no out-of-date cache file. (Is that even possible? I dont know if EVE deletes old files in there)

Have fun with it ;)

eve_cache_standings-gpl.zip

Zathi Shaitan
Illiteracy Combatants
Posted - 2008.03.26 15:46:00 - [11]
 

Someone please share a binary version...

Shinhan
Dreddit
Test Alliance Please Ignore
Posted - 2008.03.26 15:52:00 - [12]
 

Originally by: postal dude
Okay then, here it is. I release this under the GNU GPLv3.

Please note, this is only for reading the standings out of the cache, if you want to parse other cachefiles, this program is no use for you.

Usage: You have to point the app to your cache folder. It is somewhere in you application-data-folder (OS-specific). The folder you must browse to looks like this: "[...]\cache\MachoNet\87.237.38.200\189\CachedMethodCalls".
Then just press "Read Standings". Please note it reads the standings from the first standings-cache file it finds, so make sure there is no out-of-date cache file. (Is that even possible? I dont know if EVE deletes old files in there)

Have fun with it ;)

eve_cache_standings-gpl.zip



Where is the binary? Not everybody has Microsoft Visual Studio 9.

For everybody else, here's excerpt from some comments in the source code:
'0x15 0x02 0x04' is the header for the id=>standings lookup table
0x05 => Alliance; 0x06 => Corp
'0x15 0x03 0x04' is the header for the id=>name/type lookup table

How long did it took you to find the headers and do you have any tips on how to find the headers for other data?
I wanna see if the Market Price History is cached... :)

postal dude
Caldari
Thundercats
RAZOR Alliance
Posted - 2008.03.26 16:15:00 - [13]
 

Edited by: postal dude on 26/03/2008 18:42:03
Edited by: postal dude on 26/03/2008 16:15:06
Originally by: Shinhan

Where is the binary? Not everybody has Microsoft Visual Studio 9.


Here you go: EveCacheStandings.exe
(GNU GPLv3 applies here too.)

Originally by: Shinhan

How long did it took you to find the headers and do you have any tips on how to find the headers for other data?
I wanna see if the Market Price History is cached... :)


Well, it took me a few hours. I am working on a decoder for all the cachefiles at the moment. I cant tell you if the market price history is cached yet, but you can try and look at the cache yourself, a few words are plaintext in them.

Edit: I quickchecked for that and after a few minutes I found the item-id, but not the price nor the quantity. But was just a quick view over the file.

Cori4n
Caldari
principle of motion
Interstellar Alcohol Conglomerate
Posted - 2008.03.27 01:25:00 - [14]
 

Originally by: CCP Lingorm
As long as you do not modify the file in question it is technically not against the EULA or TOS. It is only a cached version of a database method call.

How about the logserver? :D

MrRx7
Amarr
Cutting Edge Incorporated
RAZOR Alliance
Posted - 2008.03.27 06:14:00 - [15]
 

as alway I love you work.

Amida Ta
German Mining and Manufacture Corp.
Posted - 2008.03.28 00:39:00 - [16]
 

Originally by: postal dude
As I understand there is no standings export via the API or the eve-online website.

I poked around in the eve-cache folders and stumbled over a file with the corp standings in it.


This approach is really interesting. I scanned through some of the cachefiles and tried to find patterns. It actually looks as if the cachefiles are some kind of object serialization code (likely Opcode/Stackbased) which means it should be possible to read them programmatically.
As far as I've seen this might make quite some of the live APIs mostly useless. In fact it seems you could get much more data from the cachefiles that from the API.

BTW: It seems most of the caches are extremely inefficiently implemented. No wonder Eve hammers the CPU...

Verite Rendition
Caldari
F.R.E.E. Explorer
EVE Animal Control
Posted - 2008.03.28 02:01:00 - [17]
 

Originally by: Amida Ta
Originally by: postal dude
As I understand there is no standings export via the API or the eve-online website.

I poked around in the eve-cache folders and stumbled over a file with the corp standings in it.


This approach is really interesting. I scanned through some of the cachefiles and tried to find patterns. It actually looks as if the cachefiles are some kind of object serialization code (likely Opcode/Stackbased) which means it should be possible to read them programmatically.
As far as I've seen this might make quite some of the live APIs mostly useless. In fact it seems you could get much more data from the cachefiles that from the API.

BTW: It seems most of the caches are extremely inefficiently implemented. No wonder Eve hammers the CPU...
The API is a stable platform that can be accessed without a client however, the cache files are not. Plus the moment you use a macro to automate logging in to EVE to collect and extract data, you're violating the EULA. Still, this can definitely be useful in certain cases.

Amida Ta
German Mining and Manufacture Corp.
Posted - 2008.03.28 08:19:00 - [18]
 

Originally by: postal dude

Well, it took me a few hours. I am working on a decoder for all the cachefiles at the moment. I cant tell you if the market price history is cached yet, but you can try and look at the cache yourself, a few words are plaintext in them.

Edit: I quickchecked for that and after a few minutes I found the item-id, but not the price nor the quantity. But was just a quick view over the file.

Did you make any advances?
I seem to understand quite a lot that is within the files (at least as long as lists are used), however the header (the whole Class/Methodcall/etc. stuff is still pretty obscure.

Did you alread find out what opcode &H11 is? I assumed it may be an empty ASCII-String, however it seems it is followed by another Byte which I could not find out...

Herio Mortis
Dark Nebula Academy
O X I D E
Posted - 2008.03.28 08:51:00 - [19]
 

Originally by: Amida Ta

Did you alread find out what opcode &H11 is? I assumed it may be an empty ASCII-String, however it seems it is followed by another Byte which I could not find out...


As far as I can tell that byte is an index into an array of strings. Seems to be used as a form of
compression for often used classnames etc.

Ssoraszh Tzarszh
Minmatar
The Royal Engineers
Posted - 2008.03.28 09:06:00 - [20]
 

Originally by: postal dude
<great stuff>
so make sure there is no out-of-date cache file. (Is that even possible? I dont know if EVE deletes old files in there)

Have fun with it ;)

eve_cache_standings-gpl.zip



Yes, that is possible, i just got a standings list from a corp i left a few months ago :D

Amida Ta
German Mining and Manufacture Corp.
Posted - 2008.03.30 11:24:00 - [21]
 

Anybody made any advances?
Would be especially interested in OpCodes:
&H2C (Objectstart?)
&H17 (Constructorcall?)
&H14 (List/Tuples?)
&H20
&H0D

Vessper
Indicium Technologies
Hephaestus Forge Alliance
Posted - 2008.03.30 12:03:00 - [22]
 

Originally by: Amida Ta
Anybody made any advances?
Would be especially interested in OpCodes:
&H2C (Objectstart?)
&H17 (Constructorcall?)
&H14 (List/Tuples?)
&H20
&H0D


Quoting from my list:

"0D" + len + string: string of length "len" followed by "string". len is byte, if len==255 then len followed of 4 bytes of longint len

"14" + len + data: Tuple of "len" elements followed by elements themselves.

"17" + a + b: Class "a" calling "b" method

Hope that's some help to you as I'm failing miserably at trying to decode these Confused

Fubar
Posted - 2008.03.30 14:27:00 - [23]
 

Originally by: Amida Ta

Did you alread find out what opcode &H11 is? I assumed it may be an empty ASCII-String, however it seems it is followed by another Byte which I could not find out...


Opcode &H11 seems to be a short cut for common names/tags.

This is the list I have compiled so far.

&H08 = bloodLineID
&H09 = capacity
&H0A = categoryID
&H15 = tickerID
&H18 = Description
&H22 = graphicsID
&H23 = groupID
&H2C = locationID
&H2D = locationName
&H3F = ownerID
&H41 = ownerName
&H42 = raceID
&H4A = typeID

Amida Ta
German Mining and Manufacture Corp.
Posted - 2008.04.01 21:49:00 - [24]
 

Here are some observations:

Byte 2E () = String (Unicode?)
Byte XX StringLen (if XX is FF another 4 bytes follow that specify the length)
XX Bytes String Chars

Byte 0E () = String empty (Unicode?)
Single Opcode

Byte 2F () = Datetime?
Byte XX Datalength (usually 08)
XX Bytes Date data

Byte 10 () = String (ASCII? ObjectName?)
Byte XX StringLen
XX Bytes String Chars

Byte 0F () = Char/String (ASCII? that is 1 char long)
Byte 1: Character

Byte 11 () = ASCII String? Link to static String Table?
1 Byte Index Data ?

Byte 01 () = Null/Empty?????
Single Opcode

Byte 02 () = String (ObjectName?)
Byte XX StringLen
XX Bytes String Chars

Byte 04 () = 4 Byte Int
4 Bytes Int Data

Byte 05 () = 2 Byte Int
2 Bytes Int Data

Byte 06 () = 1 Byte Int
1 Bytes Int Data

Byte 08 () = Int Value Fixed: 0 ??? (false)
Single Opcode

Byte 09 () = Int Value Fixed: 1 (true)
Single Opcode

Byte 28 () = ?
Single Opcode

Byte 0A () = 8 Byte Double?
8 Bytes Double Data

Byte 15 () = Datensatz? (Row?)
Byte XX Anzahl der folgenden Datensätze

Byte 16 () = Dataset
Byte XX Number of Dataset items following

Byte 2C () = Object-Start? Commit Data?
Single Opcode

11 43 steht einer Tabelle vor
11 2B folgt nach Tabelle
11 24 folgt nach Tabelle

11 3F = "name" ?

1B 01 ?

17 11 51 16 03 Tabellendefinition?

17

17 11 häufig

17 11 36 14 07 2C (2F 08 = Header Date) kommt zu Beginn fast überall vor

7E 00 00 00 00 Header?
Bytes 1-4: Version?


Byte 14 () = Methodcall ???? List?
Byte XX Number of Parameters/Definition ??
1. Parameter: Class
2. Parameter: Method
3+ Parameter: Arguments

Byte 17 = New Object/Constructor call
1. Param: Objectname
2. Param Methodcall (Byte 14)

16 01 62 2C ?

Header 1 : 7E 00 00 00 00
Objectstart: 2C
Methodstruct+Parameter/Objectname
Payloadheader:
17 11 36 14 07
Datetime (Updatetime?)
INT
Empty
INT
INT/Bool (Fixed: True)
20 2B FF alternative 0D FF 13 27

Illusori
Posted - 2008.04.02 21:36:00 - [25]
 

NIce work there Amida, seems to support what I've found,
using a file as follows:


00000000: 7e00 0000 002c 1403 100b 4d65 7468 6f64 ~....,....Method
00000010: 2043 616c 6c2e 0570 726f 7879 1404 1188 Call..proxy....
00000020: 1012 4765 744e 6577 5072 6963 6548 6973 ..GetNewPriceHis
00000030: 746f 7279 04c0 9698 0005 0667 1711 3614 tory.......g..6.
00000040: 072c 2f08 82e8 7152 258d c801 05d2 2201 .,/...qR%.....".
00000050: 04fd 460f 0009 2b89 7e00 0000 0023 2c25 ..F...+.~....#,%
00000060: 020e 6462 7574 696c 2e43 526f 7773 6574 ..dbutil.CRowset
00000070: 1601 222c 0214 626c 7565 2e44 4252 6f77 ..",..blue.DBRow
00000080: 4465 7363 7269 7074 6f72 2514 062c 100b Descriptor%..,..
00000090: 6869 7374 6f72 7944 6174 6506 402c 1008 historyDate.@,..
000000a0: 6c6f 7750 7269 6365 0606 2c10 0968 6967 lowPrice..,..hig
000000b0: 6850 7269 6365 0606 2c10 0861 7667 5072 hPrice..,..avgPr
000000c0: 6963 6506 062c 1006 766f 6c75 6d65 0614 ice..,..volume..
000000d0: 2c10 066f 7264 6572 7306 032d 2d11 242d ,..orders..--.$-
000000e0: 2d08 1403 100b 4d65 7468 6f64 2043 616c -.....Method Cal
000000f0: 6c2e 0570 726f 7879 1404 1188 1012 4765 l..proxy......Ge
00000100: 744e 6577 5072 6963 6548 6973 746f 7279 tNewPriceHistory
00000110: 04c0 9698 0005 0667 .......g


I've worked out the following meaning:


---- Header ----
?? # (7e 0000 0000) header?
key/value # (2c)
---------- block repeated at end ----------
[ # 3 (14 03)
'Method Call' # (10 0b ...)
'proxy' # (2e 05 ...)
[ # 4 (14 04)
?? # (11 88)
'GetNewPriceHistory' # (10 12 ...)
regionid:'Essence'
# (04 c0969800) # 009896c0 is essence regionid (10000064)
typeid:'capacitor control circuit II'
# (05 0667) # 6706 is typeid for t2 ccc (26374)
]
]
------------------ end block --------------
?? # (17 11 36)
[ # 7 (14 07)
key/value # (2c)
?? # (2f 08 82e8 7152 258d c801)
?? # (05 d222)
?? # (01)
?? # (04 fd46 0f00)
?? # (09)
?? # (2b89 7e00 0000 0023 2c25)

'dbutil.CRowset' # (02 0e ...)
?? # (16 01 22)
key/value # (2c)
'blue.DBRowDescriptor%' # (02 14 ...)
[ # 6 (14 06)
key/value # (2c)
'historyDate' # (10 0b ...)
40 # (06 40)
key/value # (2c)
'lowPrice' # (10 08 ...)
6 # (06 06)
key/value # (2c)
'highPrice' # (10 09 ...)
6 # (06 06)
key/value # (2c)
'avgPrice' # (10 08 ...)
6 # (06 06)
key/value # (2c)
'volume' # (10 06 ...)
14 # (06 14)
key/value # (2c)
'orders' # (10 06 ...)
3 # (06 03)
]
?? # (2d2d 1124 2d2d 08)

---------- block repeated at start ----------
[ # 3 (14 03)
'Method Call' # (10 0b ...)
'proxy' # (2e 05 ...)
[ # 4 (14 04)
?? # (11 88)
'GetNewPriceHistory' # (10 12 ...)
regionid:'Essence'
# (04 c0969800) # 009896c0 is essence regionid (10000064)
typeid:'capacitor control circuit II'
# (05 0667) # 6706 is typeid for t2 ccc (26374)
]
]
------------------ end block --------------


From that I'd basically come to the same conclusions as Amida, except that 0x2c to me appears to always indicate a key/value pair, or at least a two-element array.

I'm strapped for time at the moment, but hopefully this is useful for others to take things forwards.

'Sori

Pasion Helada
SPANI
T o r m e n t u m
Posted - 2008.05.13 16:01:00 - [26]
 

Edited by: Pasion Helada on 13/05/2008 16:01:41
all lengths are 1 byte and if byte == 255 another 4 follow.

16 is a dictionary
followed by len
( value / key ) * len times

2C is a tuple of 2 items
25 is a tuple of 1 item
24 is an empty tuple
26 is an empty sequence

1C is always followed by 4 bytes, but unknown meaning at the moment.


 

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only