open All Channels
seplocked EVE Information Portal
blankseplocked New Dev Blog: API Incarnated
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: [1] 2 3 4

Author Topic

CCP Zymurgist


Gallente
C C P
Posted - 2011.06.17 15:54:00 - [1]
 

CCP Elerhino is here to tell us about the upcoming changes to the API for Incarna. You can read about the changes and how to test them in this dev blog.

SXYGeeK
Gallente
do you
Posted - 2011.06.17 16:00:00 - [2]
 

IBC.

Looks good, through I'm worried my EVE HQ is going to be dead right away from permission errors.

are all errors included in this throttle?

Georgiy Giggle
The Sith Syndicate
Posted - 2011.06.17 16:07:00 - [3]
 

Anyway, EVE MON works fine.

Abdiel Kavash
Caldari
Paladin Order
Fidelas Constans
Posted - 2011.06.17 16:09:00 - [4]
 

Quote:
So we're going to start with: if your IP passes 3 errors a minute your IP gets blocked for 3 minutes.


Now hold on for a second. I go to my enemy alliance's web site, try to register there thrice with gibberish for API, and deny them all access for 3 minutes? Possibly longer for repeated "offenses"?

Or do errors only mean malformed requests?

Black Madness
Minmatar
Natural Born Builders
United Corporations Of Modern Eve
Posted - 2011.06.17 16:12:00 - [5]
 

Edited by: Black Madness on 17/06/2011 16:11:56
Not bad, but i personally don't like permanent IP banning.

Serpents smile
Posted - 2011.06.17 16:12:00 - [6]
 

Quote:
We're also quite sure that you'll let us know if we're wrong, as you always so diligently do.


What, who, us? Shocked
Laughing

Chribba
Otherworld Enterprises
Otherworld Empire
Posted - 2011.06.17 16:16:00 - [7]
 

Edited by: Chribba on 17/06/2011 16:21:55
more stuff

edit+q:

So with "Smaller Data", so we'd need to double-check our previous active orders if they are not listed in the api it has been filled? Am I reading this correctly?

/c

MailDeadDrop
Rage and Terror
Posted - 2011.06.17 16:19:00 - [8]
 

Originally by: CCP Elerhino
MarketOrders will now only return a list of active and non-expired orders. This call was fetching way too much information from the datastore and we decided to alter the query in a simple way, filtering out all expired and processed orders.

A character can only have about 60 orders, correct? At any one time I'd guess an active trader could have that many expired/processed orders, too. I fail to see how that (120 orders) constitutes "fetching way too much" information. Could you elaborate on this a bit more, as I suspect that *recently* expired and processed orders have some value in the API. The tough part may be figuring out what constitutes "recently".

MDD

Squizz Caphinator
Woopatang
Posted - 2011.06.17 16:19:00 - [9]
 

Some clarification please.

What is the definition of an API Error?

If I'm pulling requests that are obviously malformed?

= or =

If I'm processing 1k Character APIs and 3 of them fail with a 2xx Authentication Error Code (for the first time), then I am blocked for 3 minutes?

da go
Posted - 2011.06.17 16:22:00 - [10]
 

Edited by: da go on 17/06/2011 16:22:23
TrollMode ON
Originally by: CCP Elerhino
we have been extremely busy little bees
In the old days devs were brosefs.

TrollMode OFF

Vessper
Indicium Technologies
Hephaestus Forge Alliance
Posted - 2011.06.17 16:23:00 - [11]
 

You really need to define EXACTLY what constitutes an error here. A lot of users will enter full API keys to obtain information on their characters but may not have roles in a corporation to successfully obtain corporate API data. In which case, the API returns an error code.


Shepard Book
Posted - 2011.06.17 16:23:00 - [12]
 

I like the proactive approach on this. Good job.

Entity
X-Factor Industries
Synthetic Existence
Posted - 2011.06.17 16:27:00 - [13]
 

I can just see services being griefed using that error throttle now by people providing incorrect data on purpose.

There is NO way to know in advance if certain calls will produce an error or not!
Also, high profile services with a lot of users will cause a lot more errors, even though it's probably only like 0.01% of the total requests.

I'd say, just change your API to better deal with errors instead of dumping the responsibility of never failing a request on the 3rd party developers.

Arkady Sadik
Minmatar
Electus Matari
Posted - 2011.06.17 16:31:00 - [14]
 

Thanks for the update.

About the error throttling: There is currently no error-free way to find out for a contact entry in the contacts list if it's a character, corporation or alliance. The only way to find out is "trial and error":

- If the ID is in the alliance list, it's an alliance (doh)
- If you get a valid CorporationSheet, it's a corp (if not: error)
- If you get a valid CharacterInfo, it's a character (if not: error)
- Else it's a disbanded alliance

Would it be possible to get a API call to identify those IDs, or include a typeID in the contacts API?

MailDeadDrop
Rage and Terror
Posted - 2011.06.17 16:33:00 - [15]
 

Originally by: Entity
high profile services with a lot of users will cause a lot more errors, even though it's probably only like 0.01% of [ their ] total requests.

Actually, I think that points to an improvement to the error throttling..

Setting aside for the moment what constitutes an error, it is obvious that CCP will be tracking requestor IP addresses along with an error count. Instead, can CCP track the address and error *rate*? So that if the error rate exceeds the allowed maximum, then the lockouts begin?

MDD

da go
Posted - 2011.06.17 16:36:00 - [16]
 

Quote:
The basic theory we used here is that if nobody is complaining then we're not strict enough
Translation:

Winers drive our development.

Abdiel Kavash
Caldari
Paladin Order
Fidelas Constans
Posted - 2011.06.17 16:44:00 - [17]
 

Edited by: Abdiel Kavash on 17/06/2011 16:46:21
Originally by: MailDeadDrop
Originally by: CCP Elerhino
MarketOrders will now only return a list of active and non-expired orders. This call was fetching way too much information from the datastore and we decided to alter the query in a simple way, filtering out all expired and processed orders.

A character can only have about 60 orders, correct? At any one time I'd guess an active trader could have that many expired/processed orders, too. I fail to see how that (120 orders) constitutes "fetching way too much" information. Could you elaborate on this a bit more, as I suspect that *recently* expired and processed orders have some value in the API. The tough part may be figuring out what constitutes "recently".

MDD


My skills currently allow about 130 outstanding orders on one character, and I am nowhere near maxed. Barely decent I'd say.

I'll just say that this will be a (minor) pain in the ass - seeing completed orders lets me know what goods I need to restock. Obviously I don't remember or keep track of 60-70 items I'm selling to see which one sold out. Now the only way to find out is continuously pulling the API and looking for disappearing orders.

Matalino
Posted - 2011.06.17 16:48:00 - [18]
 

Edited by: Matalino on 17/06/2011 17:03:54
Originally by: Vessper
You really need to define EXACTLY what constitutes an error here.
This definately needs clarification. The point that I am most concerned about is that error responses are currently the only way to detect if an account is active when using the limited API key. In order to minimize the impact on applications that use a large number of keys, we need to have a means of checking if a key is active without triggering the error counter.

Even an application that uses a small number of keys will probably be impacted by the timers if more than two keys are for inactive accounts. Everytime an application encounters two inactive accounts it must avoid querying any potentially inactive accounts for one minute in order to avoid the three minute lock out. This significantly increases the complexity of all but the most basic applications. If more than one application is running it would be completely impractical to avoid being locked out because of queries using inactive keys.

Originally by: Vessper
A lot of users will enter full API keys to obtain information on their characters but may not have roles in a corporation to successfully obtain corporate API data. In which case, the API returns an error code.
Corp roles can be detected by first querying the character sheet. The character sheet will indicate if the character has the roles required to access the corp API. There is no need for an application to rely on error responses to detect corp roles.

Originally by: MailDeadDrop
Setting aside for the moment what constitutes an error, it is obvious that CCP will be tracking requestor IP addresses along with an error count. Instead, can CCP track the address and error *rate*? So that if the error rate exceeds the allowed maximum, then the lockouts begin?
Good theory, bad practice. Applications that make few queries that are prone to errors over which they have no control will have a higher base error rate. In order to avoid triggering the lock out, the simplest "fix" for those applications would be to make lots of "safe" calls to the API to improve the ratio of good/bad API requests. This results in a needless increase in overall API traffic.

Entity
X-Factor Industries
Synthetic Existence
Posted - 2011.06.17 16:48:00 - [19]
 

Oh and let's not forget the fact CCP wants to us to cough up $99 for a commercial license, yet we'd still run the risk of being locked out of one of the most essential features?

Because we sent a few bogus requests?

REALLY?

Missy Sasha
Posted - 2011.06.17 16:53:00 - [20]
 

Sounds like I'm going to get banned for using evemon. Bad form.

Abdiel Kavash
Caldari
Paladin Order
Fidelas Constans
Posted - 2011.06.17 16:58:00 - [21]
 

Originally by: Matalino
Originally by: Vessper
A lot of users will enter full API keys to obtain information on their characters but may not have roles in a corporation to successfully obtain corporate API data. In which case, the API returns an error code.
Corp roles can be detected by first querying the character sheet. The character sheet will indicate if the character has the roles required to access the corp API. There is no need for an application to rely on error responses to detect corp roles.


Even so, it's a question of which takes less resources, both on the server and client side: calling one API method which might in rare cases throw an error (most likely misuse or people dropping roles), or calling two methods every time?

Sometimes "sorry, no results" is just as good response as getting the results themselves.

Matalino
Posted - 2011.06.17 17:08:00 - [22]
 

Edited by: Matalino on 17/06/2011 17:20:08
Originally by: Abdiel Kavash
Originally by: Matalino
Originally by: Vessper
A lot of users will enter full API keys to obtain information on their characters but may not have roles in a corporation to successfully obtain corporate API data. In which case, the API returns an error code.
Corp roles can be detected by first querying the character sheet. The character sheet will indicate if the character has the roles required to access the corp API. There is no need for an application to rely on error responses to detect corp roles.


Even so, it's a question of which takes less resources, both on the server and client side: calling one API method which might in rare cases throw an error (most likely misuse or people dropping roles), or calling two methods every time?

Sometimes "sorry, no results" is just as good response as getting the results themselves.
If the error rate is low (less than three errors per minute) then there is no problem with the lockout trigger and the application can continue to use errors to detect corp roles.

If the application is going to be making enough queries to trigger the lockout, then it is probably better to query the character sheet first to verify permissions.

While it adds complexity, a hybrid approach is also possible: directly query corp API's, if two errors have been detected, query character sheets to verify corp roles for any additional queries made during the next minute.

Andrea Griffin
Posted - 2011.06.17 17:11:00 - [23]
 

Edited by: Andrea Griffin on 17/06/2011 17:12:59
Originally by: Black Madness
Not bad, but i personally don't like permanent IP banning.
Same. For those of us whose IP address changes on a semi-regular basis, it would really suck to be assigned the address of some derp face who had earlier abused the API. Make it a few days or a week, but permanent doesn't sound all that great.

Jason Edwards
Internet Tough Guy
Spreadsheets Online
Posted - 2011.06.17 17:25:00 - [24]
 

So developers while trying to code cant make mistakes else they get banned?

Richard Stallman wrote a program that divides by zero.

Richard Stallman can make infinite loops end.

Matalino
Posted - 2011.06.17 17:28:00 - [25]
 

Edited by: Matalino on 17/06/2011 17:31:29
Originally by: Jason Edwards
So developers while trying to code cant make mistakes else they get banned?
They can make mistakes. They are just limited to one or two mistakes per minute. If a program triggers the lockout, it is not unreasonable to spend 3 minutes finding and correcting the problem, at which point the lockout will have expired. Bans are for those who spend more time locked out than not. I expect that there should be no problem with getting locked out a few time a day while developing a program.

Matalino
Posted - 2011.06.17 17:41:00 - [26]
 

Edited by: Matalino on 17/06/2011 17:44:26
Originally by: CCP Zymurgist
CCP Elerhino is here to tell us about the upcoming changes to the API for Incarna.
Can we please get some clarification on when this change is going live? I hope this is not going into effect for June 21. A single weekend is certainly not enough time for developers to adapt to any change let alone one so drastic. If the change must go live on that date, at least dial down the restriction to allow for a higher initial error rate until developers have a chance to code responses to the change.

CCP Stillman

Posted - 2011.06.17 17:56:00 - [27]
 

As the dev blog states, our intent is to keep a consistent level of service for the API.

It's not the intent to be cause legitimate applications grief at all. Quite the opposite, because we want to give legitimate users and applications the best possible experience, for which we need throttling to make sure that the API's performance doesn't significantly degrade in the event of unusual requests are made excessively by any one entity.

It is however a valid concern you all raise. And it's something we aim to not have become a problem through tweaking the settings so that legit users won't feel it. Based on this, we'll make sure that the initial threshold is high enough that legitimate developers don't get bit by this, as it's not the intent.

I might add that this is an extension to our already existing policy where we block IPs that cause excessive errors. Note that "permanent" bans are only given out in the most extreme cases. And if that occurs, you can file a petition and we will lift the ban on your IP if you promise to fix whatever caused excessive requests.

Again, the intent is not to cause anybody to be blocked. It's to ensure that applications do not negatively affect the API if they do not respect errors and the like. We will tweak the throttling to ensure that legitimate users aren't affected. I'll talk to Elerhino on Monday, and discuss about raising the throttling threshold, if you're concerned about it being too low. Because you shouldn't be.

Sino Sarn
THORN Syndicate
BricK sQuAD.
Posted - 2011.06.17 18:07:00 - [28]
 

Nerf supers, already.

Risingson
Posted - 2011.06.17 18:24:00 - [29]
 

there would have to be a way to validate a users userid/apikey combo without causing an error counting towards that throttle.


Assaj Ventress
Posted - 2011.06.17 18:31:00 - [30]
 

CCP Stillman, while you are here, can you please clarify what exactly is considered en errouneous request?


Pages: [1] 2 3 4

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only