open All Channels
seplocked EVE Technology Lab
blankseplocked Yapeal PHP API library (revision 1203 branches/keys/)
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: first : previous : ... 5 6 7 8 [9] 10 11 12 13 ... : last (24)

Author Topic

Cory Sopapilla
Minmatar
Kiroshi Group
Posted - 2009.06.25 15:07:00 - [241]
 

Ok so I was building a sandbox server for API tool development and ran into the map.xml issue myself. If you put the CCP dump into a DB then add yapeal to the same DB, mapJumps is a duplicate table. So even from a fresh install, the 2 can't coexist unless you delete/rename mapJumps from CCP. It makes me worry about the next CCP data dump upgrade though. I'm going to end up losing mapJumps for yapeal most likely since I'll probably forget by then.

Any chance of looking into that and possibly renaming the yapeal mapJumps next build if this is confirmed?

Dragonaire
Caldari
Corax.
PURgE Alliance
Posted - 2009.06.25 15:37:00 - [242]
 

No need to rename it that why we added the prefix stuff to Yapeal so if this became a problem there was already a fix in place Wink When you run the instiller just make sure you use the prefix option when setting up the tables so for example all of them look like MyDB.YapealaccountCharacters etc.

Cory Sopapilla
Minmatar
Kiroshi Group
Posted - 2009.06.25 18:58:00 - [243]
 

Cool. As always, thanks for the quick response.

Barrington Farquharsen
The Arrow Project
Morsus Mihi
Posted - 2009.07.11 12:39:00 - [244]
 

I'm trying to pull CorporationSheet data for corps entered into the utilRegisteredCorporation table.

How would I go about editing your code to allow for the entry of an activeAPI (corpCorporationSheet) and a corporationID into this table so that it would update corpCorporationSheet accordingly?

Ideally, I'm not wanting to be limited to a single corp in this table (by restrictions such as characterID cannot be NULL and needing API keys for a character to be entered too). I would like to enter multiple corpIDs and pull the CorporationSheet for each one to be entered into the corpCorporationSheet table automatically.

I'm sure this would be possible within your code structure but just can't figure it out myself.

Great API library though Smile

Dragonaire
Caldari
Corax.
PURgE Alliance
Posted - 2009.07.11 14:01:00 - [245]
 

Barrington Farquharsen - What you'll need to do is add another row to the utilRegisteredCorporation table with a minimum of the isActive, corporationID, characterID, and activeAPI set with the information on the corp. ActiveAPI is a space separated list of the APIs you want it to get for that corp. You'll need a row in utilRegisteredCharacter that correlates with the characterID column. You'll also need to add a row in utilRegisteredUser that correlates to the userID in utilRegisteredCharacter and has the fullApiKey filled. You don't have to set isActive=1 except if you also want to get API data for the character. The characterID you have in utilRegisteredCorporation needs to have CEO or director level access in the corp for it to work correctly. That is because of requirements setup by CCP to access most of the corp APIs.

Hopefully the above get you started down the right track. Yapeal was design to support multiple users, chars, and corps as that was a requirement for it in EMPA and also for it to be more generally useful for other developers. The installer only let's you add one of each as it is only meant to be used for initial testing for people to try out Yapeal before developing their own applications with it. Look for some useful util classes to show up in EMPA to make managing the utilRegistered* tables easier in the near future as more of what is being worked on in the branches get merged back into the trunk on it.

Barrington Farquharsen
The Arrow Project
Morsus Mihi
Posted - 2009.07.11 14:33:00 - [246]
 

I follow that the 3 tables need to be populated to do what you say, but ideally I would only like the utilRegisteredCorporation table to be populated.

As far as I am aware, the CorporationSheet can be accessed without an API key -- it only requires a corpID. This is what I would like to do. Ideally all I would do is populate the utilRegisteredCorporation and the cron job populates the corpCorporationSheet table.

Sadly I'm not interested in Corp or Character specific information, only CorporationSheets, hence the need for something where a characterID or API key is not required.

Would this be possible within Yapeal?

Dragonaire
Caldari
Corax.
PURgE Alliance
Posted - 2009.07.13 17:43:00 - [247]
 

Can people get 'public' info about a corporation without being part of the corp? Yes

Could Yapeal be modified to not require you to have corp member info to access the API? Yes.

Do we plain on doing so? No, but it is open source so if someone wanted to branch it and do so they're welcome to do so and they would probably even get a few points from me on things that would need to be changed.

Something that might work but I haven't testing is to point the characterID at your character info for the other corps and see if it'll let you still pull the info from the API. The only thing you might run into is Yapeal may error out because of any missing stuff in the API info returned since you aren't a member in the corp and don't have access. Also depending on how the API works when Yapeal passes your UserID and API key it may give you an error about not being a member of the corp. You would just have to try it with all the logging turned on and see what happens but it might actually work Smile

Dragonaire
Caldari
Corax.
PURgE Alliance
Posted - 2009.07.16 15:18:00 - [248]
 

Hi everyone been working on some stuff in EMPA and that's shown me a few areas where I could be improved in Yapeal as well. Decided to bring over the class autoloader first as it's one of the easiest and required little in the way of changes. It's something I had thought about doing before but never got around to so when the need for one in EMPA came up I wrote it and decided I should port it back into Yapeal too.

In the process of adding the class autoloader I also made a couple small changes to the installer so it'll now cache the SQL it uses to build the tables. You can find the files in cache/. Some of you might find them interesting or even useful when move stuff between development and hosting servers. The ones it'll save if there's an error might also be useful during debugging for you and us Wink

Look for some more changes/improvements to show up as my work schedule and development on EMPA allows.

revision 803

Cory Sopapilla
Minmatar
Kiroshi Group
Posted - 2009.07.16 15:59:00 - [249]
 

Awesome. Thanks for the update. I'll check it out tonight.

Anything in the autoloader classes that would help in user registration of API info to populate the tables? I'm working on my own but don't want to reinvent the wheel if it's already done.

Thanks

Dragonaire
Caldari
Corax.
PURgE Alliance
Posted - 2009.07.16 17:04:00 - [250]
 

No this let's me get rid of most of the require_once statements that were scattered throughout the code. You can read more about what it effects here. I do have planned some changes that will effect some of the utilRegistered* tables and the ini file but not sure when they will be done. There will also be a few changes that shouldn't be visible to anyone using Yapeal but should improve it in several ways much like this change which doesn't really show up in anyway to the outside developers.

Al Coholick
Volition Cult
The Volition Cult
Posted - 2009.07.21 12:52:00 - [251]
 

Is anyone working on integrating Eve Central data into Yapeal?
If not, I'd like to give it a go.

Cory Sopapilla
Minmatar
Kiroshi Group
Posted - 2009.07.21 18:02:00 - [252]
 

I just added a table to my eve_db and pull the daily csv file in. I was going to work with the API pulls but that is WAY down on my list.

Al Coholick
Volition Cult
The Volition Cult
Posted - 2009.07.21 20:13:00 - [253]
 

I was actually thinking about using the Mail-push feature.
It should be fairly easy to set up a dedicated google email account (or whatever email you choose) and suck the posted mails in from there. Perfect for a cron task.

Also perhaps include an import function for the existing CSV data files.

Dragonaire
Caldari
Corax.
PURgE Alliance
Posted - 2009.07.22 13:24:00 - [254]
 

Hi there has been some talk about adding Eve-Central and/or other pricing data but the latest person to talk to me about it is very busy with a couple other projects at this time so not sure if/when he would get any code done. I had some code in the original project that Yapeal grow out of that worked with the CSV feed. You could try the e-mail approach as well if you think you really need minute by minute updated data but the extra overhead and complications getting and decoding the e-mail would probably outweigh the benefits if you don't need that.

If either or both of you would be interested in work on this let me know via my G-mail account from the project and I can add you to the project if you're interested in working on it in a branch. I could also dig up my old code for you to have a look at too.

Al Coholick
Volition Cult
The Volition Cult
Posted - 2009.07.27 13:29:00 - [255]
 

Well, I finally got around to working on the Eve Central data this weekend, and it's 95% finished.
After looking at the size of the dump files, I decided that the email approach probably wasn't a good idea.
I didn't realize just how much data was recorded by Eve Central.

Which brings me to the problem I've come across. There is just SO much data.
Over 4 million records just for the first 12 days of January.

I've not really used or studied the data yet, so I'm not sure if there are rows I should be updating (instead of inserting a new row).
I see some duplicated rows, probably because multiple people submitted the same data.
I'm thinking I should make OrderID an exclusive key, and update any duplicates.

Should I throw away data older than X days?

Any advice?

Dragonaire
Caldari
Corax.
PURgE Alliance
Posted - 2009.07.27 18:09:00 - [256]
 

OrderID does recycle every so often so might think of using it and the date/time of the order but you will still end up with dups. In my old code I used the daily CSV and sorted and got rid of any dups in it as I went then when I upserted it to the DB I let MySQL take care of dups between the dailies. How ever you do it make sure you use the one with the latest date/time or you can end up outdated data. Out of all the orders in each CSV I'd say probably 33% were dups with the main trade centers running closer to 75-90% dups at times.

In my old code I had it deleting any record older than a year and really anything older than 3 months is ancient history with Eve but just in case someone want to do some market trend stuff a year might be useful.

Hope that helps. Like I said if you are interested in having a look at my old code let me know it was really rough but did seem to work well.

Al Coholick
Volition Cult
The Volition Cult
Posted - 2009.07.27 18:22:00 - [257]
 

Edited by: Al Coholick on 27/07/2009 18:24:11
Here's what I decided to do.

There is a new INI variable called "days_back", and this is used to determine how many days of data to process.
Any dump files older than this will be ignored.

Also, duplicate rows (by OrderID) will be updated rather than inserted as a new row.

A preliminary version is ready. Since I don't have SVN write access, here is a direct download.

http://www.greygames.com/yapeal_with_EveCentral.zip


The install process is untested, but the SQL statement to create the new databases required are in the comments of eve_central.php.


There is a new section in Yapeal.ini for the Eve Central specific variables.
Be sure to change the values to fit your system.

-------------------------------------------
; Section for Eve Central settings. (required if Eve Central updates are enabled)
[EveCentral]

; Only go back X days. Ignore any dump files older than this.
days_back=10

; A writable (by crontab) directory for processing EVE Central downloads
temp_dir="/home/USERDIR/tmpFiles"

; Name of the executable used to unpack Eve Central's "gz" files
; Include the full path if necessary.
unpack_util="gunzip"
-------------------------------------------

There is a variable in eve_central.php named $maxProcessFiles which specifies how many dump files to process per cron run.
Be careful if you increase this value from its default of 1.
It takes about 3 minutes per dump file (on my web server, including the file download), and if the the cron job doesn't finish before its next scheduled run, errors will occur.
When doing the initial download and import of the data, you may want to increase the frequency of the cron task (10 minutes?), and leave $maxProcessFiles at 1.

One major thing still missing the the cacheUntil feature. So please ensure your cron task doesn't run too often after the initial downloads, to prevent hammering Eve Central.

P.S.
Dragonaire, please check your evemail.

EDIT:
Oops, I must have been typing this when you replied, Dragonaire.
Thanks for the tip about OrderID recycling. I'll have to do something about that.

Cory Sopapilla
Minmatar
Kiroshi Group
Posted - 2009.07.27 18:28:00 - [258]
 

Edited by: Cory Sopapilla on 27/07/2009 18:33:18
Al, you're in our public channel so gimme a shout ;) I've done alot of the work already and we have a corp project going for an API frontend for yapeal. I have all of the csv stuff working already so maybe we could compare notes.

Basically, I grab the entire daily dump and put in a temp table but only allow the newest by date for each orderID per regionID (there are items in jita reported hundreds of times per day). Then I generate a list of unique itemIDs per regionID and delete all matches (1 at a time) and replace with the new orderIDs for that itemID in the regionID. There's lots of other stuff like deleting any that expired, etc.

What I want to do ultimately (on top of this) is pull the xml for mineral prices from the eve-central API so I can get hourly prices on those as well as an option in the front end to request new prices on other things if they have recently sent a new dump to eve-central.

Phaedriel
Posted - 2009.07.31 16:10:00 - [259]
 

Hi Dragonaire,

I just set Yapeal up on my server after a little messing around with Cron tasks (getting paths etc). I'm now getting an error returned to me. Seems like the database tables are in lowercase and the reference to them in the code are in uppercase.

Is this something you've come across before? Can you see any issues with me just renaming the database tables to how they are referred?

E.g.

yapeal.utilCachedUntil
is inthe database as
yapeal.utilcacheduntil

Dragonaire
Caldari
Corax.
PURgE Alliance
Posted - 2009.07.31 17:11:00 - [260]
 

Would probably cause you issue when trying to use the installer to update. The normal setting for MySQL is mixed case so I'd look at it's settings on your server myself to see if I couldn't fix that. I do know on Windows they seem to use a forced lower setting but it doesn't seem to cause any problems since MySQL just silent converts everything for you so the mixed case doesn't matter. I may look into adding settings to the DB connections in ADOdb to set for mixed case but you're the first one to have noticed it outside of one of the developers using Windows and I don't think it cause him any problems but let me know if it is and I'll look into it some more. The reason for the mixed case is the match the API as closely as possible and CCP does use mixed case in a very specific way in their naming.

Phaedriel
Posted - 2009.07.31 21:30:00 - [261]
 

Okay, I manually changed the names. As you say, I think it's a windows issue. I installed it on my local server and pretty much just copied it to my live server and exported/imported the SQL (yea, bad habit but it works most of the time!).

Just wanting to confirm the the message I'm getting means that all is okay (if so I'll disable the message).

Notice: Yapeal version 803 (beta) 2009-07-16 15:31:27 in (path)/httpdocs/Yapeal/inc/common_backend.php on line 54

(I editted (path) for security reasons but it's the path to the file.

If this is all good then I have to move onto Phase 2 of my application (not for public release, designed with myself and close friends in mind, just a useful tool)

Johnathan Roark
Caldari
The Graduates
Morsus Mihi
Posted - 2009.07.31 23:05:00 - [262]
 

Originally by: Phaedriel
Okay, I manually changed the names. As you say, I think it's a windows issue. I installed it on my local server and pretty much just copied it to my live server and exported/imported the SQL (yea, bad habit but it works most of the time!).

Just wanting to confirm the the message I'm getting means that all is okay (if so I'll disable the message).

Notice: Yapeal version 803 (beta) 2009-07-16 15:31:27 in (path)/httpdocs/Yapeal/inc/common_backend.php on line 54

(I editted (path) for security reasons but it's the path to the file.

If this is all good then I have to move onto Phase 2 of my application (not for public release, designed with myself and close friends in mind, just a useful tool)


Thats normal

Dragonaire
Caldari
Corax.
PURgE Alliance
Posted - 2009.08.01 05:26:00 - [263]
 

Yeah once you're sure it working the notice messages can usually be turned off without any problems do leave on the warnings and errors though as they can be helpful when something goes wrong. You'll also want to make sure you didn't turn on the tracing which is off by default

Dragonaire
Caldari
Corax.
PURgE Alliance
Posted - 2009.08.03 17:30:00 - [264]
 

Ok just ran across something again and decided to add it into Yapeal. You can now view the syntax highlighted source code of any of the PHP files in a web browser. For example: http://localhost/path/to/yapeal/yapeal.php?viewSource=1 will return the highlighted source code for yapeal.php

This is mostly for show but thought I'd add it anyway Smile Developers or someone trying to learn PHP might find it useful.

Aedrine
Posted - 2009.08.17 14:39:00 - [265]
 

Hello,

Is it possible to disable the sql part of this API? I just want to manipulate the XML.

Thanks :)

Dragonaire
Caldari
Corax.
PURgE Alliance
Posted - 2009.08.17 14:41:00 - [266]
 

Aedrine if you just want the raw XML to start with why are you trying to use a library when you can just grab the XML directly from API if that's what you want Question

Cory Sopapilla
Minmatar
Kiroshi Group
Posted - 2009.08.17 17:05:00 - [267]
 

Edited by: Cory Sopapilla on 17/08/2009 17:08:03
Edited by: Cory Sopapilla on 17/08/2009 17:07:13
Aedrine,

http://wiki.eve-id.net/APIv2_Page_Index is where you want to start. Each of the listed API calls is in the format following:

http://api.eve-online.com/account/Characters.xml.aspx?userID=1234567&apiKey=ABC123

Just start with http://api.eve-online.com and end it with the desired URI example column and fill in the ?name= input arguments. The one above would give the raw xml listing of your 3 characters if you use your own userid and apiKey as an example.

Note the cache short/long/etc timers to see how often you can pull. Don't hit longs every 5 minutes or you may end up being blocked ;)

autoland2
Posted - 2009.08.20 15:28:00 - [268]
 

Hi,

Stupid question, and yes they do exist :-)

Can I modify yapeal to run from mysql instad of mysqli ?

I tracked down the 2 files in teh installtaion that refence the DB Driver type, but will it work.

The new server we have meets all requirements except the mysqli driver.

Dragonaire
Caldari
Corax.
PURgE Alliance
Posted - 2009.08.20 18:13:00 - [269]
 

If your server doesn't have mysqli I think you'll find there are also other things missing as it's a normal part of the standard build as defined by the PHP org that everyone seems to ignore Razz but as to you're question since Yapeal uses transactions for some things (most of the API table updates) which the mysql drive doesn't support you'll run into trouble. There are reasons that I've set the requirements to what they are and they really aren't that hard to meet given a relatively new and correctly configured server.

Nick Partridge
Posted - 2009.08.20 19:36:00 - [270]
 

Ok fair enough.
Trying to get answers out of our server host, who support all the features of the system (Yapeal) except that, and there telling us they dont use it and we cant have it :-( what a pain.


Pages: first : previous : ... 5 6 7 8 [9] 10 11 12 13 ... : last (24)

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only