open All Channels
seplocked EVE Technology Lab
blankseplocked Active API cache project?
 
This thread is older than 90 days and has been locked due to inactivity.


 
Author Topic

Wendi Watson
Organization Too Secret To Know
Posted - 2010.07.03 01:25:00 - [1]
 

I'm running EveMon, occasionally EveHQ, and recently installed Yapeal to support a eve project I'm working on. Now I'm running into a problem with caching issues with the Industry Jobs and Market Orders API.

I was looking for a API cache I can run locally so every program can get access to the data without getting the cache errors for these APIs, but the cache programs listed on EveDev are inactive or defunct. Any projects I missed?

I'm running a local server with Apache 2.2 and PHP 5.3.


Dragonaire
Caldari
Corax.
PURgE Alliance
Posted - 2010.07.04 00:11:00 - [2]
 

I believe EveMon will use the XML files that Yapeal gets from the API. You should be able to just tell it to put them there the only problem is Yapeal does use directories for each section which EveMon might not understand. Hopefully that will help.

Jercy Fravowitz
School of Applied Knowledge
Posted - 2010.07.04 08:01:00 - [3]
 

i am using this as a "central" cache for various (evemon, eft, eam, selfwritten stuff using EVEAPI, ...) API apps: http://zofu.no-ip.de/apicache.pl

setup is to have a webserver that feels responsible for the api.eve-online.com vhost, and runs the cgi script for every request, then point apps at it by means of local DNS override or hosts entry.

the requested api page and parameters are normalized (keys resorted, lowercased, optional elements that do not change functionality in practice removed) and used as the cache key (== filename). the cache is pretty lightweight and has few dependencies.

caching is done simply in /tmp/, you might wish to change that (or at least change permissions) if there are untrusted users on the host running the cache webserver.
there is no authentication towards the cache whatsoever since the api key is part of the cache key, so if you have the api key, you are allowed to get the data that api key gives access to.

there is nothing fancy about this cache, you could call it "inactive" if you want, but i really dont know what i should change about it at this point, it has been working just fine for a long time. for the last two years, there were only cosmetic and convenience changes. X)

one issue any api cache will have is how to deal with timestamps.
the api pages usualy have two timestamps: currentTime (ct) and cachedUntil (cu).
now, you can deal with that in an app two ways: either you calc the difference between cu-ct, and use that as an offset to the current local time. or you compare cu directly to current local time.
the cache doesnt touch the timestamps. this means for a page that has a one hour cu-ct, and an app with delta strategy requesting the page from cache 45 minutes after ct, it will get the impression it has to wait 60 minutes.
if the cache decides to "modify" the timestamps in some way to deal with the cachiness, this might improve the situation for delta-apps, but in exchange will wreak havoc with apps that use absolute stamps.
basicly i didnt find any solution there that will work for all apps, so i decided to not change the timestamps at all, since its the least dangerous route and gives reliable, though sometimes slightly slower than possible results.

Vishus Nehs
Posted - 2010.07.06 01:19:00 - [4]
 

@ Jercy Fravowitz

I'd like to see if I can get your perl script working, how did you setup the webserver to run the script when it receives the API calls from EveMon, EveHQ, EFT, etc?

Thanks



Jercy Fravowitz
School of Applied Knowledge
Posted - 2010.07.06 06:56:00 - [5]
 

Originally by: Vishus Nehs
I'd like to see if I can get your perl script working, how did you setup the webserver to run the script when it receives the API calls from EveMon, EveHQ, EFT, etc?

Multiple steps involved and none of the specifics i did will work for anyone else since the setup here is a bit speshul. X)

webserver setup:
  • add a named vhost api.eve-online.com, so the webserver feels responsible for answering to that name. you can make that a separate vhost, or add it to an existing one, it doesnt really matter unless you have namespace collisions from what follows next.

  • add the apicache cgi to that vhost somewhere, and check that it can run. for testing the cgi setup itself, this script is very convenient. just fiddle with the config until you can call it _without_extension_ and it produces some output like this.

  • make sure "path runover" works. i totally made that name up. but the point is that you can call http://zofu.no-ip.de/env/sheep/go/baa?forsure and the webserver will call the same env cgi installed in the last step and the script can get the additional url components from env. this should work as soon as the cgi works at all (the last step).

  • returning to the apicache part, add aliases for the apicache cgi that correspond to the first path element of the api pages you want to use. in my case, those are simply symlinks pointing to the cache cgi with names account, char, corp, eve, map.

  • there are alternative way to do this for sure. f.ex. most webservers allow you to map whole pathes or vhosts to one ressource. in apache mod_rewrite or errordocument 404 directive come to mind. i am using the symlinked approach since my apicache runs within a well-populated document root.

  • you may have to install Date::Parse module (from cpan) for the apicache to work. that is the only external dependency that comes to mind that is not part of a standard perl installation.


applications setup:
  • add a dns override for api.eve-online.com to point to the webservers ip, either by messing with your resolver (if you have one, you know how to do this) or editing the /etc/hosts on *nixoids, or google "windows hosts file" for how to do that on your flavor of windows.

  • note those edits have to be done on the host running the api-using application.
    that way, when evemon wants to talk to api.eve-online.com, it will really talk to your webserver instead.


bit of work, but the resulting setup is supposed to cover "all" api-using applications.
if you get odd results with apps i didnt name (meaning: i didnt try them yet) please ask, i will take a look at what they may or may not be doing different.

MaddMaxx2000
Posted - 2010.07.06 11:28:00 - [6]
 

Edited by: MaddMaxx2000 on 06/07/2010 11:28:36
Looking to duplicate your setup, are you running your script on Linux/Apache? Any specific versions required?



Jercy Fravowitz
School of Applied Knowledge
Posted - 2010.07.06 13:45:00 - [7]
 

Originally by: MaddMaxx2000
Looking to duplicate your setup, are you running your script on Linux/Apache? Any specific versions required?

linux yes, apache no. no standard webserver at all. thats why i described the process in what i hope to be enough detail to nail together a working setup with pretty much any webserver.

the env.pl cgi has the main point of giving you something to setup the webserver with. as in, a test for the cgi setup that doesnt involve the api and api clients.

Solo Drakban
GoonWaffe
Goonswarm Federation
Posted - 2010.07.06 15:57:00 - [8]
 

Not that anybody is going to trust it but there is an API proxy that GoonFleet put up for it's members. The URL is at http://api.goonfleet.com/ and it's meant to be a drop in replacement for api.eve-online.com (IE, http://api.eve-online.com/server/ServerStatus.xml.aspx == http://api.goonfleet.com/server/ServerStatus.xml.aspx).

There is a major important change from the standard EVE API that our API does:


  • Our proxy returns HTTP error codes as close to the error as possible. That means if you get a 202 API error (API key authentication failure) the proxy will return an HTTP 403 Forbidden response instead of an HTTP 200 OK with the XML error code as the payload.

  • Our proxy supports SSL using https://api.goonfleet.com/ instead of http://. This encrypts the transaction between your client and the proxy. Note that the communication between our API proxy and the CCP API server is still unencrypted but less likely to be intercepted unless somebody manages to tap into a backbone router.



Some benefits of the proxy include:

  • Our proxy sets proper Cache-Control headers for all requests based on the <cacheUntil> elements of the XML returns form the CCP API. This means you can setup a stock HTTP proxy such as squid between your applications and our proxy and it will work properly.

  • The proxy re-orders the request elements into a standard order to determine cache hits so ?userID=..&apiKey=..&characterID=.. is equal to ?&characterID=..&userID=..&apiKey=.. for the purposes of cache hits. (Some EVE API caches I've seen would treat those are two different hits).

  • The URI parameters (apiKey, userID, characterID, etc) are stored using a one-way hash (SHA-512). It is impossible to retrieve any data from the URI elements (like API key). The only place this information exists is in memory during the request. (I strip URI parameters out when writing to log as well so the access log just contains what .xml.aspx was called, not how it was called).

  • The XML payload is stored encrypted in cache via AES (with a 256-bit key size) and the decryption key information is stored (encrypted itself) outside of the web directory and only provided, decrypted, in memory to the PHP application as needed to decrypt cache

  • Caching is two-tiered in memory and database. Frequently hit requests will come from memory cache resulting in greatly sped up applications.

  • Also along the lines of security I am the only person with direct access to the application so it's not like every goon on the planet can log into a shell and get your bits.



Like I said, not like I expect anybody outside of GoonSwarm to trust this API proxy, but it's there for the general community to use if they want to. Just be aware that while I am not logging actual request parameters I am logging requests and I reserve the right to deny access to abusive IP addresses and/or throttle requests as necessary.

Ruziel
Minmatar
Twilight Military Industrial Complex
Posted - 2010.07.07 01:57:00 - [9]
 

Originally by: Solo Drakban
Wall o' text Razz


I too am building some web based tools for Eve, and this sounds like something I would be interested in. Any chance of getting the source to this cache to hack on?

In my search, I ran across http://vlad.off.net/eve/eproxy-php.txt which appears a bit out of date, but looks like it could easily be fixed up with the new API calls.

Solo Drakban
GoonWaffe
Goonswarm Federation
Posted - 2010.07.07 02:24:00 - [10]
 

Edited by: Solo Drakban on 07/07/2010 02:28:51
Edited by: Solo Drakban on 07/07/2010 02:27:26
Originally by: Ruziel
Originally by: Solo Drakban
Wall o' text Razz


I too am building some web based tools for Eve, and this sounds like something I would be interested in. Any chance of getting the source to this cache to hack on?


Unfortunately I cannot give out the source for our proxy at the moment as we are currently 'refactoring' it in another language due to scaling (and developer) issues with Ruby on Rails. We're also writing a REST interface on top of the proxy to abstract and simplify writing applications on top of the EVE API so our members can more rapidly develop 'cool ****' without having to worry about the complexities of merging the DB dump and the API data sources, or dealing with odd SQL, etc. This project (and probably are killboard for anybody who really needs an ultra-high performance KB) are probably going to be open sourced and when we do we'll announce it here.

Quote:
In my search, I ran across http://vlad.off.net/eve/eproxy-php.txt which appears a bit out of date, but looks like it could easily be fixed up with the new API calls.


Yea, this was also written by a goon. We ran this one for awhile but it shows it's lack of in-memory caching under high load (which you'll probably never run into, goons and all that). You can snag an updated version with some bug fixes, a little more recent API call dictionary and modifications (like using the cacheUntil in the XML rather than a static cache timer per page) from https://apps.goonfleet.com/eproxy-php-v2.txt

Estimated Prophet
Ye Olde Curiosity Shoppe and Trading Company
EVE Trade Consortium
Posted - 2010.07.10 10:06:00 - [11]
 

Originally by: Vishus Nehs

I'd like to see if I can get your perl script working, how did you setup the webserver to run the script when it receives the API calls from EveMon, EveHQ, EFT, etc?



Originally by: MaddMaxx2000
Edited by: MaddMaxx2000 on 06/07/2010 11:28:36
Looking to duplicate your setup, are you running your script on Linux/Apache? Any specific versions required?



I got it working with Apache2 on Linux by doing the following:

Creating a directory /var/www/api.eve-online.com and putting the script in there, and creating symbolic links for account, char, corp, eve, map, and server.

Creating a vhost with the following:

<VirtualHost *:80>
ServerName api.eve-online.com
ServerAlias api
DocumentRoot /var/www/api.eve-online.com
ErrorLog /var/log/apache2/error.log
CustomLog /var/log/apache2/access.log combined
ServerSignature On
<Directory /var/www/api.eve-online.com>
Options +ExecCGI
SetHandler cgi-script
</Directory>
</VirtualHost>

There may be options more to add to get it working properly, but these were sufficient for me.

Peter Powers
FinFleet
Raiden.
Posted - 2010.07.10 10:41:00 - [12]
 

Originally by: Solo Drakban
Like I said, not like I expect anybody outside of GoonSwarm to trust this API proxy, but it's there for the general community to use if they want to. Just be aware that while I am not logging actual request parameters I am logging requests and I reserve the right to deny access to abusive IP addresses and/or throttle requests as necessary.


ever thought of making the code opensource?

Tonto Auri
Vhero' Multipurpose Corp
Posted - 2010.07.10 22:49:00 - [13]
 

Originally by: Peter Powers
Originally by: Solo Drakban
Like I said, not like I expect anybody outside of GoonSwarm to trust this API proxy, but it's there for the general community to use if they want to. Just be aware that while I am not logging actual request parameters I am logging requests and I reserve the right to deny access to abusive IP addresses and/or throttle requests as necessary.


ever thought of making the code opensource?


Ever though of reading thread in a whole before replying to it? The answer is in there already.


 

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only