open All Channels
seplocked EVE Technology Lab
blankseplocked Eve Api (php/mysql)
 
This thread is older than 90 days and has been locked due to inactivity.


 
Author Topic

Jack Lagoon
Posted - 2011.06.13 01:55:00 - [1]
 

My mysql database consists of 1500 api keys that I would like to query in a timely manner using php. I pull the entire list from the database, because I have to check if each members eve account is active, and then work my way down the list using an api status function I wrote to see if they are active. Is it possible to speed up the api calls because at the moment querying 1500 api keys is a bit time intensive to work through? Or is it just because I have to go through so many it will take a while. Any type of example code on fast querying of lots of api keys wouldn't hurt

Lutz Major
Posted - 2011.06.13 06:44:00 - [2]
 

As far as I know there is no mechanisms to query multiple API keys at once.

I'd guess it takes you at least half an hour to check all those keys, due to response times.

Have you considered to mark/remove completely invalid keys (i.e. the user created new ones)?
Also I'd mark inactive accounts and query them only once a day.

And of course multi-threading would significantly improve the time to check your keys (but don't ask me how this is implemented in PHP Very Happy)

Trenker
Posted - 2011.06.13 16:56:00 - [3]
 

Multi-threading or multi-processing is what came to my mind as well. PHP does not have this, but you can fake it:

use a script (invoked my cron), that spawns several child scripts using exec

eg.:

$cmd = 'php queryapi.php 100 200 &';
exec($cmd);


The queryapi.php script would then query the api with the keys of DB UID 100 to 200. Using the & puts the script in the background, otherwise your main script would be blocked. This way you can make several "threads", that query a slice of the queue cake.

There are several good tools / scripts / classes on the net for this, just ask google for "php multi processing" or something like that.

Tonto Auri
Vhero' Multipurpose Corp
Posted - 2011.06.13 21:03:00 - [4]
 

Originally by: Trenker
Multi-threading or multi-processing is what came to my mind as well. PHP does not have this

I question your statement on both multithreading and multiprocessing.
PHP is not a single standalone application, it have multiple ways of execution, and multiple ways to handle execution of subsequent jobs.

MrRx7
Amarr
Cutting Edge Incorporated
RAZOR Alliance
Posted - 2011.06.14 02:38:00 - [5]
 

Originally by: Tonto Auri
Originally by: Trenker
Multi-threading or multi-processing is what came to my mind as well. PHP does not have this

I question your statement on both multithreading and multiprocessing.
PHP is not a single standalone application, it have multiple ways of execution, and multiple ways to handle execution of subsequent jobs.


He is not incorrect, there is no build-in support for multi-threading within PHP. Your best bet would be parallel processing such as the following example : http://phplens.com/phpeverywhere/?q=node/view/254

When I was still running the Northern Coalition killboard I wrote a parallel processing script so I could pull kills from multiple feeds at the same time.

Dragonaire
Caldari
Corax.
PURgE Alliance
Posted - 2011.06.15 05:22:00 - [6]
 

http://www.php.net/manual/en/book.pcntl.php
Quote:
He is not incorrect, there is no build-in support for multi-threading within PHP.
Might try learning what your talking about before saying it doesn't have threading. It's not enable in CGI for good reasons but is available and very usable for CLI on most OSes (read not Windows).

Peter Powers
FinFleet
Raiden.
Posted - 2011.06.15 08:03:00 - [7]
 

Edited by: Peter Powers on 15/06/2011 08:04:25
Originally by: Dragonaire
Might try learning what your talking about before saying it doesn't have threading. It's not enable in CGI for good reasons but is available and very usable for CLI on most OSes (read not Windows).

and you want to read up on the difference between a process and a thread.

edit: quoted wrong part.

Tonto Auri
Vhero' Multipurpose Corp
Posted - 2011.06.15 11:23:00 - [8]
 

PHP FCGI server is a multithreaded application...

Peter Powers
FinFleet
Raiden.
Posted - 2011.06.15 12:43:00 - [9]
 

Originally by: Tonto Auri
PHP FCGI server is a multithreaded application...

no php-fcgi uses forked children, as does php-fpm.

Tonto Auri
Vhero' Multipurpose Corp
Posted - 2011.06.15 23:12:00 - [10]
 

Originally by: Peter Powers
Originally by: Tonto Auri
PHP FCGI server is a multithreaded application...

no php-fcgi uses forked children, as does php-fpm.

Under windows? Seriously?

Dragonaire
Caldari
Corax.
PURgE Alliance
Posted - 2011.06.16 04:34:00 - [11]
 

Sorry was running late for work and on my way out the door when I posted it Embarassed but the point is PHP isn't as limited as others would have you think it is.

cyclobs
Minmatar
Posted - 2011.06.16 04:52:00 - [12]
 

Originally by: Dragonaire
Sorry was running late for work and on my way out the door when I posted it Embarassed but the point is PHP isn't as limited as others would have you think it is.


i've seen a perfectly fine PHP IRC bot pulling weather by zip code and relaying it to an IRC channel (or even multiple channels)

Trenker
Posted - 2011.06.16 08:25:00 - [13]
 

I am aware of the different ways on how to do this, but an exec call is my choice for the following reasons:
  • You can call that "process" with different rights using "su -c 'php ...'"
  • You can actually put those processes into the background.
  • You can add a "nice -n 19", which is great if creating a lot of processes. This way you will be superfast when creating a lot of processes, yet will not kill your server, as those processes will go a little slower if a web request needs to be worked on.

But I'm always open for suggestions.

Dragonaire
Caldari
Corax.
PURgE Alliance
Posted - 2011.06.16 14:47:00 - [14]
 

http://www.php.net/manual/en/function.pcntl-setpriority.php Let's you set the priority as well. Also depending on host setup you might not have access to things like 'su' etc because of security. Since pcntl is part of PHP itself it also is made to over differences in setting things like that on different platforms. IE 'su' works on most Linux and BSD systems but might not on Mac or other Unix host because of lack of the command etc.

You can also run into problems if the hosting company IT department decide to use some really off the wall shell that isn't bash,sh. The whole idea with PHP just like with Java and other cross platform virtual environments is to let you ignore differences like this and get on with what you want to get done.

Use the wheel that's already been invented don't make your own just because you can. I've personally found many times when I think I can do it better and made my own code in the end I often find when going back to look at the other people's code they had actually solved the problem I'm having and it was my lack of understanding the problem and how to make it work that had me thinking they did it wrong Wink


 

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only