open All Channels
seplocked EVE Technology Lab
blankseplocked Yapeal PHP API library (revision 1203 branches/keys/)
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: [1] 2 3 4 5 6 7 8 9 ... : last (24)

Author Topic

Dragonaire
Caldari
Corax.
PURgE Alliance
Posted - 2008.10.22 17:56:00 - [1]
 

Edited by: Dragonaire on 04/09/2011 03:08:36
Yapeal has now been in development for nearly three years and proven to be useful for many people in their Eve API applications and websites. For a description of what it does check out the project home page at GoogleCode.

Yapeal has now moved out of what I considered 'Alpha' into 'Beta' stage of development. For people use to Mircosoft's way of doing things Yapeal is now Win98 vs Win3.11 or Win95 (It actually works for more than 5 minutes Twisted Evil)

Yapeal is licensed under GNU LGPL 3

HOME PAGE http://code.google.com/p/yapeal/

You can find my public API key on the home page as well.

I have a blog for Yapeal called Peeling Yapeal that everyone is invited to follow as well.

It's now recommended that everyone uses the downloads of the non-installer versions which are available at:
https://sourceforge.net/projects/yapeal/files/trunk/
Yapeal is also still available through the SVN at:
http://code.google.com/p/yapeal/source/checkout

For more information about Subversion see http://subversion.tigris.org/
Windows users may find http://tortoisesvn.tigris.org/ easier to use.

Please report any bugs you find in this thread or as as issue on the website.

You can contact me through the developer name from the project @ gmail.com

Dragonaire
Caldari
Corax.
PURgE Alliance
Posted - 2008.10.22 18:00:00 - [2]
 

Edited by: Dragonaire on 18/11/2008 01:40:47
Here an extend explanation of what Yapeal does since it a little different then the other PHP API libraries people are using.

It doesn't only grab the data from the API for you but also saves the new data into a MySQL database for you. It was made to be ran from a crontab job and works in the background to 'Auto-magically' as I call it, keep the database updated so all you have to do is work with the database tables not the API and XML. It's made to be set and forget once configured.

Some features:

  • Auto download of API data

  • Auto updating of database tables

  • Can be ran from crontab (preferred setup)

  • Can be run from command-line for debugging or special needs

  • Keeps track in the database of API cachedUntil times so it is API friendly even if ran every minute in crontab

  • Does optional local caching of XML file from API

  • Multiple corporations and multiple characters by just adding their information to database tables



Many other features but those are the main ones for now.

Dragonaire
Caldari
Corax.
PURgE Alliance
Posted - 2008.10.25 23:14:00 - [3]
 

New version 0.1.0 available sorry for the broken one had up before. Not that this one might not have bugs but it should at least try to run ;) Also added MySQL files to build Database to db-tools directory.

Rehtom Lamina
Hounds of Helll
Circle-Of-Two
Posted - 2008.10.27 14:46:00 - [4]
 

will download and have a play tonight.

Dragonaire
Caldari
Corax.
PURgE Alliance
Posted - 2008.10.29 00:11:00 - [5]
 

Edited by: Dragonaire on 02/11/2008 20:35:36
Ok been doing a lot of bug hunting and seem to have eve and corp API stuff working but still working on the char part. Needed to make a couple changes to the membertracking table so you'll need to drop and re-create it for everything to work. I'm not doing a release at this point so if you want the most current code need to get it from the subversion.

svn checkout http://yapeal.googlecode.com/svn/trunk/

Dragonaire
Caldari
Corax.
PURgE Alliance
Posted - 2008.11.02 20:59:00 - [6]
 

I've moved the project over to google code so it's easier for me to manage along with another project I'm working on now.

Dragonaire
Caldari
Corax.
PURgE Alliance
Posted - 2008.11.18 00:56:00 - [7]
 

Ok after over 200 revisions according to subversion I think I've killed all the previous bugs that made it unusable. It's got a lot of fixes to database code since I've changed over to using ADOdb (included) instead of MDB2. Yapeal is now fully PEAR free so it should be much easier to install and use.

Rehtom Lamina
Hounds of Helll
Circle-Of-Two
Posted - 2008.11.19 10:49:00 - [8]
 

Good work ugh

oUTSKIRTs
Posted - 2008.11.20 15:47:00 - [9]
 

This is really cool, almost everything seems to be working now, the things that doesn't work right now, did work at some point during the revisions, so i'm excited to see how this project will turn out. Right now my cron job is doing exactly what it is supposed to (exept for at the moment 4 tables not being updated due to missing indexes or upper/lowercase issues that i noticed you're hard at work on fixing).
Tables not updating is: fuel, starbasedetail, IndustryJobs and MarketOrders

I haven't checked this latest revision 228 out, yet, waiting for cache timer (so tired of that). Thank you for this, it is really really usefull to get rid of the xml problems and get right to work on a database instead. Awesome!

Dragonaire
Caldari
Corax.
PURgE Alliance
Posted - 2008.11.21 02:36:00 - [10]
 

Thanks I'm killing bugs as fast as I find them but going to need some help from someone with some of the ones you talked about because the alt I'm using doesn't own any POS so that might make it hard testing Smile

My original code that I extracted Yapeal from had all the eve/ and corp/ stuff I used working but with breaking it out into separate project, changing DB structure, and switching DB libraries many things ended up broken Razz

I decided to work on the char/ stuff first this time since I'd never got around to making it all work before since my other project was mostly corp stuff and usually once it's working for one it's easy to move changes into the other code. That only leaves the few things that's unique to each to fix if they are still broken.

I'm up to revision 249 at this time so look for some more changes to be pushed back into trunk when I'm sure they're work. I'm trying to leave trunk as the more stable branch now that it's starting to work and I don't feel like I'm losing ground like before with it Smile

Also a trick I use during debugging is to have the XML caching on and then delete the records in the CachedUntil table for main script 'eve-api-pull' and which even one I'm test like 'MarketOrder' then see if it works Very Happy or doesn't Sad It's absolute need when working with AssetList and that 23 hour timer Razz As long as it has a good cache file it doesn't try grabbing one from API and you can cheat on that to for testing by just changing the time in XML. Saves me a lot of 101 errors from API during testing and the server from some of the worse of my typos Rolling Eyes

oUTSKIRTs
Posted - 2008.11.21 12:40:00 - [11]
 

It's looking good, nice work!

I am doing some tests here and there, and I like the cache feature a lot, i was trying to get the whole deal to connect through an api-proxy - so that other people still have their eve-mon when i pull the data to this db, but it seems to fail horribly right away. It is probably the proxy stuff not doing it just right. Not really important at this point.

A few more things I have noticed, that I forgot to mention:

There is a table in Yapeal.sql called 'logo' - but the program is calling it as 'corplogo' when trying to access it. It works fine by just renaming the table in the sql.

The starbasedetails (and i think 'fuel' too) table is actually gone from yapeal.sql
None of them get any data imported any longer, it has been gone for a while, i think i still have the revision that was infact pulling these two tables allright a while back.. Somewhere.. :)

In the marketorders table, there is a field called "range", very nasty name for a database field - this is also the name of a mysql function, making it fail unless in backtics or maybe calling the field something else might be better... Some say they have tables called range without noticing the problem, really I think it would be better to call it something else. I can see how that might become slightly impractical to say the least, but looking at your code im pretty amazed, you know your stuff and im sure you'll think of something lol

Other than that -- my database is alive, it's Twisted Evil

Really close, keep it up .. Cool

Dragonaire
Caldari
Corax.
PURgE Alliance
Posted - 2008.11.21 17:50:00 - [12]
 

Glad you like it and thanks for the catch on the logo table but it was the PHP that was wrong. All the tables and fields are directly named from the fields in the API so when there's a difference on something look at Eve-Dev or in the XML for what it should be. So in the case of 'logo' it's now logo everywhere like it was meant to be.

As to the 'range' field yes it sucks and there's a couple others but that what CCP used so that what I use and I just have to make sure I use the back-ticks or let ADOdb take care of any quoting for stuff like that.

I drop the other POS stuff for now because I felt it needed done differently and didn't have time to mess with it right now and no way to test it but I think I'll have some friends from another project to help me out with that now.

My naming for all the main tables should be in CamelCase just like the API calls are and the "sub" tables as I call them follow the XML which seems to be camelCase. The fields are also in camelCase so far anyway so that's what I'm using. For all the backend administration tables they should follow the same naming as well and as I am getting around to it I'm converting all the naming of class, functions, methods, and variables to follow PEAR conventions with a few mods here and there. Example with the function named 'getAPIinfo' I should uppercase the second 'I' and had the 'pi' lowercase but just didn't look right to me and didn't want to re-name it either Smile

Hope this has been helpful on see where some of the changes I'm make are coming from when not smashing bugs as I find them Very Happy

I've never tested it with any of the proxies out there but did make it so that any that act just like the API should work fine after all what we get from CCP is all through a proxy already on their end. Even if other proxies only follow the API naming and return XML and normal HTTP errors it should work but it would probably lose some of the special caching extras I've built in to work with the main API.

oUTSKIRTs
Posted - 2008.11.21 18:15:00 - [13]
 

This is probably not the way to fix this, but..

Quote:

function makeMultiUpsert(array $data, array $params, $table, $dsn) {
// Need this so we can do quoting.
$con = connect($dsn);
$needsQuote = array('C', 'X', 'D', 'T');
// Build query sections
$insert = 'insert into ' . $table . ' (`';
$insert.= implode('`,`', array_keys($params)) . '`)';

... a little further down ...

// Loop thru and build update section.
$updates = array();
foreach($params as $k => $v) {
$updates[] = "`" . $k . "`" . '=values(' . "`" . $k . "`" . ')';
};



Function taken from the common_db file.

I just punched in some backticks around the fields, and well.. it worked.. I think it shouldn't break anything else :)
The table is updated, and it looks kinda ok to me.

Dragonaire
Caldari
Corax.
PURgE Alliance
Posted - 2008.11.21 21:47:00 - [14]
 

Never had any problem in that function with 'range' etc because MySQL is smart enough to know that you can't have it inside the insert (range) ... values() ... in duplicate key update range=values('range') statement that it's building and adding extra back-ticks might break it. What I need to do is change to using a prepare, execute instead but I haven't yet because I figured I'd run into trouble some where along the line between my code , ADOdb, and MySQL because of the on duplicate key update part. I didn't figure out a way to make it work with MDB2 but I should take another look at it now with ADOdb and see if it could work. It should be a bit faster to, although that doesn't seem to be much of a problem once I started using transactions for the larger updates of WalletJournal, etc. I've seen an up to 10x speed up since using the transactions. It has to do with DB not have to do as much locking it seems. There was even measurable differences down to 10 records level which is what others have posted on-line so that why you'll see I check for number of records in multipleUpsert() and use transactions for the larger ones now.

oUTSKIRTs
Posted - 2008.11.22 00:11:00 - [15]
 

Just checked out rev. 266 and it looks like it is doing the range thing correctly

Real good job

Dragonaire
Caldari
Corax.
PURgE Alliance
Posted - 2008.11.22 01:16:00 - [16]
 

Ok transfered all the fixes from char/ to corp/ and everything seems to be working. Can't test StarbaseList at this time so it's only one I really has any questions about. Did a complete drop and re-create on DB and everything works so it seems the code and SQL are in sync now finally. Still might have a few bugs but I've began to test it in crontab off and on now that it seem to be stable enough. It's not like I've ever had it caused problems just wouldn't always run right and filling up it's logs much faster then I could use while fixing problems Razz

oUTSKIRTs
Posted - 2008.11.22 13:09:00 - [17]
 

StarbaseList is updating correctly, you also got the certificates right first time around. Man I totally like this stuff :)

Could you use my cache files to test with?

Dragonaire
Caldari
Corax.
PURgE Alliance
Posted - 2008.11.23 01:28:00 - [18]
 

Yep certificates was an easy one to do and for now I'm going to keep try to fill in some of the missing sub-tables before I really try adding more to it. I've had it run now with crontab for over 12 hours and it's never glitch even though API did once or twice on me Wink Been trying it on another hosted site with someone else's info and had a few problems but think it's something not configured right on site and we'll have to wait a couple days until owner is back in town to make some changes. Thanks for the offer on the cache files we'll see how it goes and if I decide to get the POS stuff wrote back in (it's commented out right now) I might contact you about them if friend doesn't get back where I can use him as guinea pig first Evil or Very MadLaughing

Dragonaire
Caldari
Corax.
PURgE Alliance
Posted - 2008.11.23 19:55:00 - [19]
 

Ok did some looking around on prepare/execute in ADOdb and found that because mysql does things a little different they only emulate using prepare with it. Since that basically what I'm doing already I might as well use my code as their's Smile While I was looking up stuff on prepare/execute in the MySQL manual I also decided to take another look at the stuff on insert and update about the quoting and decide even though I was right that it worked without problems as is, you were right oUTSKIRTs that the field names should be quoted with the back-ticks to insure no future problems so I've added them now in the latest versions. Glad you said something that made me take another look at it. Have them in there should help cut off some possible mystery bugs if MySQL purser changes or they add something to API in the future that could break it.

oUTSKIRTs
Posted - 2008.11.24 16:48:00 - [20]
 

Thanks for continuing the work on this!

Just following up, the back-tick thing worked ok and look nice and solid too. However some of the changes from rev 274 and up must've somehow cracked the whole deal out, I'll leave my cron running hourly tho, incase it's another temp eve-api oddity, which i doubt actually, I think there is a little something somewhere...

What happens is this: Nothing Neutral - pretty much absolutely nothing that I can see

I cleared the cacheduntill table, but it doesn't even update/create the 'eve-api-pull' record, no logs created even with trace on and trace level above 0, trace set to 'all'. It'll just exit, after a short pause - like a second or so - it does SOMETHING that's for sure, and it doesn't exit with any error - no output at all. Rather odd.

The ./cache/* dirs are all writeable. I'll take a look later, will also 'restart' from the svn source with a fresh database and basically crawl the script a bit to see what's up incase you have no idea what this could be and that there could be something that didn't sync correctly (i *might* have a few local changes, but I'm almost 100% sure i have rolled all changes back each time before a sync)

I'm a little too busy the next few days to really go too much into it, but i'll be watching the svn closely :)

Dragonaire
Caldari
Corax.
PURgE Alliance
Posted - 2008.11.24 20:32:00 - [21]
 

Yeah sorry about that broke it for a couple there but 280-281 works again I just committed what I was working on to test it and had to fix it only took a few minutes but seems you grabbed it while still broken Wink Usually when I break stuff like that you'll start getting error and warning logs in cache/log/ as long is you have them turned on in ini which are on by default.

oUTSKIRTs
Posted - 2008.11.24 22:39:00 - [22]
 

A fresh svn got it back up and running just now Cool

Dragonaire
Caldari
Corax.
PURgE Alliance
Posted - 2008.11.29 20:09:00 - [23]
 

Just in case you hadn't noticed Yapeal doesn't have anything implemented to deal with the generation change that has happened with the wallet journal transaction Id.
You can read more about the problem in this thread:

Journal API RefID change

I'd kind of forgot about them doing this so hadn't done anything to work around the problem (really was hoping CCP would just move to a bigint) but since they don't seem to plan on do that I'll be looking at a way to work around their bug Razz Since I'm already using bigint for all ID type fields in the database it shouldn't be hard to work something out but main problem will be dealing with php5's lack of an native bigint. I'll update here as I decided on how to deal with it.

Grim
Amarr
Posted - 2008.11.30 05:10:00 - [24]
 

Very nice, major props. Coding an assets pricer and viewer. It is very interesting because I am getting 3 databases to work together as one, one from eve central, the mysql item dump, and the ui. So, I have to say it is quite a challenge.

- Grim

Dragonaire
Caldari
Corax.
PURgE Alliance
Posted - 2008.12.02 23:44:00 - [25]
 

Added a new incomplete RegisteredCharacterManagement class and some helper files. Plan on having a couple more for Corporation, and User plus maybe ones to manage groups/ collections of the above. Thought I'd get some feedback on wither someone out there thinks they might be useful and maybe be willing to do some of the testing for me when I finish it up Evil or Very MadLaughing

Still working on the WalletJournal generations stuff hope to have something to test in a couple days on that.

ThyShiva
Posted - 2008.12.16 23:19:00 - [26]
 

Hi, I downloaded the Yapeal code on the 1st Dec so will still have to look at you newest release.

Out the box and after configuring the yapeal.ini file, the alliancelist, conquerablestationlist, errorlist, reftypes and serverstatus tables were populated.

At first I only populated the registereduser table with my Eve details but my char data did not download. Only after reading the code did I realise that registeredcharacter table must have data in it too. There seems to be a step missing because the avg user will not have the information you need to populate registeredcharacter table.

I see youre looking for a tester, put my name on that list as your project lends itself nicely to my current distraction.

-Shiva

Dragonaire
Caldari
Corax.
PURgE Alliance
Posted - 2008.12.17 16:52:00 - [27]
 

Hi glad to hear someone else finds it useful Very Happy

The basic information need for the RegisteredCharacter table can be found with a call to /account/Characters which only needs the info from RegisteredUser to get. Setting up and 'registering' characters, corporations, or users is a frontend application function so that why I really haven't added anything in Yapeal for doing it. I have been playing around with some ideas for managing those tables which you can check out in RegisteredAccountManagement.class.php and RegisteredCharacterManagement.class.php. They are untested and just some ideas I've have on how to do it. I've done a little more work on some of the ideas in a branch but it's tied in with some other changes I've been looking at making and I'm not ready show it to others until I've worked on it some more.

For something that's more of a application framework you should look at another project I'm working on with some others at EMPA. It mostly in the planning/outline stage without any real code but it was a large part of the driving force behind the changes to the design of Yapeal.

Tanja Torture
Posted - 2008.12.18 13:49:00 - [28]
 

Hi Dragonaire

I've setup yapeal on my server. I use one database only (I only filled the first DNS and left the rest commented). After I inserted an entry for RegistredUser and RegistredCharacter. When I run the eve-api-pull.php through "php5 eve-api-pull.php", I get the following message:
Quote:

[2008-12-18 13:40:45.651] Tired of waiting! Getting eve-api-pull for 0
[2008-12-18 13:40:45.658] Tired of waiting! Getting CharacterSheet for 91xxxxx


But the charactersheet and all others tables except "ChachedUntil" stay empty. What I'm doing wrong? I use revision 312.

greets Tanja

ThyShiva
Posted - 2008.12.18 17:33:00 - [29]
 

Tanja have a look in the Yapeal_warning.log file in the \cache\log directory for any clues, also have a look inside the directory \cache\char for any xml files they often have a clue to the error too.

Dragonaire Thanks for the help and new classes Ill play with them over the next few days.
Im looking into asset management as part on my project and I have noticed that you are storing the station location of the asset but not storing what container/ship the asset is in.
You will need to store the container/ship ID the asset is in, either in a new field or by replacing locationID with the itemID of the container/ship. I would think a new field is the best way forward.

-Shiva

Dragonaire
Caldari
Corax.
PURgE Alliance
Posted - 2008.12.19 00:28:00 - [30]
 

Tanja Torture -
Make sure you set `isActive`=1 but from the error messages you give it seems you have. The 'Tired of waiting ...' can mean one of two things either you never tried get that API before so there's nothing in cachedUntil to compare with or it's now more than ~5 minutes pass when cachedUntil says that API expired. It's just a notice really that I setup to help keep track when the API is acting up and let you know it's doing something Wink and follow ThyShiva advise if that still doesn't seem to let you figure it out try turning on some of the trace stuff and look in cache/log/Yapeal_trace.log and see if you can find where it's getting to. I'll need that information to help narrow it down more.

ThyShiva -
You'll want to learn about 'nested sets' and how you use the `lft` and `rgt` fields to find what is inside what. Basic idea is if item is inside of something else the child's lft > parent's lft and child's rgt < parent's rgt. You can find more information about it in HierarchicalData which is from another project I'm working on with some other people.

If you or others are interested in help with that project you'd be more than welcome as we've seemed to have stalled a bit after the planning stage Sad We basically want to make a GPL'd modular API application framework that will bring all our varies other API projects together into one and then expand out from there. Like to see all the people working on API applications not be reinventing the wheel so much and get out something we can all use and contribute to so it can become truly useful for everyone.


Pages: [1] 2 3 4 5 6 7 8 9 ... : last (24)

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only