open All Channels
seplocked EVE Information Portal
blankseplocked New Dev Blog: API Incarnated
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: 1 [2] 3 4

Author Topic

El'Niaga
Minmatar
Republic Military School
Posted - 2011.06.17 18:32:00 - [31]
 

Originally by: CCP Stillman
As the dev blog states, our intent is to keep a consistent level of service for the API.

It's not the intent to be cause legitimate applications grief at all. Quite the opposite, because we want to give legitimate users and applications the best possible experience, for which we need throttling to make sure that the API's performance doesn't significantly degrade in the event of unusual requests are made excessively by any one entity.

It is however a valid concern you all raise. And it's something we aim to not have become a problem through tweaking the settings so that legit users won't feel it. Based on this, we'll make sure that the initial threshold is high enough that legitimate developers don't get bit by this, as it's not the intent.

I might add that this is an extension to our already existing policy where we block IPs that cause excessive errors. Note that "permanent" bans are only given out in the most extreme cases. And if that occurs, you can file a petition and we will lift the ban on your IP if you promise to fix whatever caused excessive requests.

Again, the intent is not to cause anybody to be blocked. It's to ensure that applications do not negatively affect the API if they do not respect errors and the like. We will tweak the throttling to ensure that legitimate users aren't affected. I'll talk to Elerhino on Monday, and discuss about raising the throttling threshold, if you're concerned about it being too low. Because you shouldn't be.


Now don't take this the wrong way, but you know your corporation released a new forum without even the most basic security implemented (which they then had to take the forums down but once again have them in test.....waste of resources). So you have to realize most people are very skeptical of CCP these days.

Instead of Devs that love the game and want to put out new and exciting ships/mods etc as CCP was in the early days, we seem the last year or two to have bean counters in charge. Bean counters produce lackluster expansions because they are cheap. They slowly run games into a ground because they don't love a game they only see dollar signs. Take the bean counter syndrome to far and you face the 2005 collapse that SWG suffered with the NGE.

What you really need to do for the Winter expansion is at least 20 new ships of new roles (that's 5 per race...1 frig, 1 destroyer, 1 cruiser, 1 battlecruiser, 1 battleship, and 1 capital very doable), with mods etc to support them. This is a spaceship game, that hasn't had significant new ships in over 2 years. If you have any dwindling revenue streams etc, that is the source. You could use multiple entries from the various ship contests, just because the winner was guaranteed doesn't mean you can't use others to help cut down development time.

Bean counters are unimaginitive, and that's why they eventually run all games into the ground.

Vessper
Indicium Technologies
Hephaestus Forge Alliance
Posted - 2011.06.17 18:32:00 - [32]
 

Originally by: CCP Stillman
As the dev blog states, our intent is to keep a consistent level of service for the API.

It's not the intent to be cause legitimate applications grief at all. Quite the opposite, because we want to give legitimate users and applications the best possible experience, for which we need throttling to make sure that the API's performance doesn't significantly degrade in the event of unusual requests are made excessively by any one entity.

It is however a valid concern you all raise. And it's something we aim to not have become a problem through tweaking the settings so that legit users won't feel it. Based on this, we'll make sure that the initial threshold is high enough that legitimate developers don't get bit by this, as it's not the intent.

I might add that this is an extension to our already existing policy where we block IPs that cause excessive errors. Note that "permanent" bans are only given out in the most extreme cases. And if that occurs, you can file a petition and we will lift the ban on your IP if you promise to fix whatever caused excessive requests.

Again, the intent is not to cause anybody to be blocked. It's to ensure that applications do not negatively affect the API if they do not respect errors and the like. We will tweak the throttling to ensure that legitimate users aren't affected. I'll talk to Elerhino on Monday, and discuss about raising the throttling threshold, if you're concerned about it being too low. Because you shouldn't be.


We still need clarification as to what constitutes an "unusual request" under these new rules. Is it certain error codes? All error codes? Malformed requests? Without this information, it's going to be hard to know what you guys are getting at and therefore how to make our apps more compliant.

Also, is this something that will be deployed next week?

Azazel Mordred
Minmatar
Cloak of Shadows
Posted - 2011.06.17 18:46:00 - [33]
 

Cool, I'm all for some attempts to improve the API's performance, and blocking repeated erroneous requests seems like a pretty reasonable idea.

I'll just echo the concerns already noted though, about what conditions exactly add to the counter which results in an IP block.

In addition to the obvious "griefing" opportunities presented by providing false API information to someone's service, I would imagine blocking due to requests to a data set before it's cache expiry would constitute a blockable offence. Unfortunately I feel this may also be abused, or even more likely, sites and services ending up inadvertently blocking themselves because a user is perhaps trying to use multiple services, or has signed up on one which is doing regular polling causing the rest to fail.

It's easy for such services to reject that user's API for a while *after* receiving errors, but you have to try it to know the result.


I think there are way too many API calls which you have no way of knowing will fail, until they have been called and returned an error - by which time you've got yourself blocked!


Lumy
Minmatar
Sebiestor Tribe
Posted - 2011.06.17 19:23:00 - [34]
 

LOL, CCP CMS fail. I believe you meant

<eveapi version="2">
<currentTime>2011-06-21 13:18:52</currentTime>
<error code="904">Your IP address has been temporarily blocked because it is causing too many errors. See the cacheUntil
timestamp for when it will be opened again. IPs that continually cause a lot of errors in the API will be permanently banned, please
take measures to minimize problematic API calls from your application.</error>
<cachedUntil>2071-06-21 13:21:50</cachedUntil>
</eveapi>


instead of

  2011-06-21 13:18:52
Your IP address has been temporarily blocked because it is causing too many errors. See the cacheUntil timestamp for when it will be
opened again. IPs that continually cause a lot of errors in the API will be permanently banned, please take measures to minimize
problematic API calls from your application.
2071-06-21 13:21:50

Veldar Reku
Minmatar
Wu Xi Holdings
Posted - 2011.06.17 19:50:00 - [35]
 

Originally by: Chribba

So with "Smaller Data", so we'd need to double-check our previous active orders if they are not listed in the api it has been filled? Am I reading this correctly?
/c


Co-ordinate with wallet transactions to know for certain.

But generally yes, if the order is NOT expired and it is no longer active, then it must have been filled or canceled.

Lutz Major
Posted - 2011.06.17 20:00:00 - [36]
 

Originally by: Matalino
rants implying nobody can code
You do realise, that you can query several dozens of API calls in under a minute?
I'm quering all available APIs of my six characters and three corporations in about 12 seconds.
But I know, which roles every char has and know, when roles change.

With foreign chars, I can't know and would have to query for restricted/full API key (okay these won't change), query the corp, query different corp pages, ...

--------

The other thing about the only active MarketOrders. Wouldn't it be possible to have the inactive orders at least once in a call (or up to a week)? So we could determine, whether they were cancelled/fullfilled/expired? That might help for corporation orders, where everyone could cancel them.

Florestan Bronstein
24th Imperial Crusade
Posted - 2011.06.17 20:05:00 - [37]
 

Originally by: MailDeadDrop
A character can only have about 60 orders, correct?

more like 300

Lutz Major
Posted - 2011.06.17 20:12:00 - [38]
 

Originally by: Veldar Reku
Co-ordinate with wallet transactions to know for certain

I know, it's pretty far fetched, but:

Char A creates a corp sell order for ISK 8.20
Char A creates a corp sell order for ISK 8.50
...
Someone buys in bulk for ISK 8.9
...
Char B cancelles the second order for ISK 8.5 and creates a new order for ISK 9.0
...

You can't determine which order was filled and which was cancelled :(

Wollari
Phoenix Industries
Wicked Nation
Posted - 2011.06.17 20:50:00 - [39]
 

You really should take care of the blocking thing. If somebody don't like DOTLAN for example, he would be funny enough wrong api keys in my registration panel and block my IP for X amount of time, blocking all DOTLAN map/sov/alliance requests just because someone is abusing my user api section.

The only change for me would be to actually use 2 different IPs. 1 for user bull**** and user generated api requests and one for the more important database and site updates. So site updates won't get blocked when someone is abusing my user section.

hmmmmm

Wollari
Phoenix Industries
Wicked Nation
Posted - 2011.06.17 20:59:00 - [40]
 

When Incarna evolves, I hope we get Incarna related APIs and IGB Browser headers aswell (like position in the station, section, etc)

Epitrope
The Citadel Manufacturing and Trade Corporation
Posted - 2011.06.17 21:22:00 - [41]
 

Originally by: devblog
Smaller Data

MarketOrders will now only return a list of active and non-expired orders. This call was fetching way too much information from the datastore and we decided to alter the query in a simple way, filtering out all expired and processed orders. This is potentially a big change but we figured most people are calling this to watch their active orders and for historic information they get their transactions. We're also quite sure that you'll let us know if we're wrong, as you always so diligently do.

Originally by: eve-dev wiki
orderState
byte
Valid states: 0 = open/active, 1 = closed, 2 = expired (or fulfilled), 3 = canceled, 4 = pending, 5 = character deleted.


How do we find out about orders which are created and destroyed between API calls?

Matalino
Posted - 2011.06.17 21:42:00 - [42]
 

Edited by: Matalino on 17/06/2011 21:45:48
Originally by: Lutz Major
Originally by: Matalino
rants implying nobody can code
You do realise, that you can query several dozens of API calls in under a minute?
I'm quering all available APIs of my six characters and three corporations in about 12 seconds.
But I know, which roles every char has and know, when roles change.
I don't know where you get the impression that I was ranting or that I was implying that nobody can code.

Originally by: Lutz Major
Originally by: Veldar Reku
Co-ordinate with wallet transactions to know for certain

I know, it's pretty far fetched, but:

Char A creates a corp sell order for ISK 8.20
Char A creates a corp sell order for ISK 8.50
...
Someone buys in bulk for ISK 8.9
...
Char B cancelles the second order for ISK 8.5 and creates a new order for ISK 9.0
...

You can't determine which order was filled and which was cancelled :(

First, Char B cannot cancel an order created by Char A.

Second, you can determine which order was filled based on the fact that the lowest priced order must always be filled before the higher priced order. This rule will allow you to determine which order was cancelled in all but the most specialized situation.

Finally, does it really matter which of the two orders was cancelled, and which one was filled? Aside from idle curiosity, why would you want to have this information?
Originally by: Epitrope
How do we find out about orders which are created and destroyed between API calls?
The wallet transaction log will record the purchase of any items, and the wallet journal will record broker fees. If the order was created and cancelled without being filled at all, you will have no record of what was ordered, but you will still have a record of the broker fees. In both cases, this may present a challenge for those who correlate Market Order creation times with Broker Fee timestamps to account for those broker fees. Keeping some history of completed market orders would allow for easy accounting of broker fees.

Risingson
Posted - 2011.06.17 22:45:00 - [43]
 

Originally by: Wollari
When Incarna evolves, I hope we get Incarna related APIs and IGB Browser headers aswell (like position in the station, section, etc)

current Incursions API first please :)

Epitrope
The Citadel Manufacturing and Trade Corporation
Posted - 2011.06.17 23:20:00 - [44]
 

Originally by: Matalino
Finally, does it really matter which of the two orders was cancelled, and which one was filled? Aside from idle curiosity, why would you want to have this information?

...

If the order was created and cancelled without being filled at all, you will have no record of what was ordered, but you will still have a record of the broker fees.


We're giving examples of data that are being taken away by the change to the behavior of MarketOrders, in the hope that that change won't go through. You've given workarounds, and that's good, but workarounds shouldn't be necessary and don't address our concerns.

Doktor Csernus
Posted - 2011.06.17 23:41:00 - [45]
 

The problem is the user input....

Matalino
Posted - 2011.06.18 00:52:00 - [46]
 

Originally by: Epitrope
We're giving examples of data that are being taken away by the change to the behavior of MarketOrders, in the hope that that change won't go through. You've given workarounds, and that's good, but workarounds shouldn't be necessary and don't address our concerns.
CCP Elerhino stated that they wish to remove this extra data from the Market Orders API in order to reduce load on the server. If we want them to keep it, then we will likely need a reason that has more substance than "idle curiosity" or for the convenience of a few programmers. Discussing the issue in this thread has revealed a specific reason to keep that data. Without this data, it is impractical to match broker fees to market orders that are created and filled/cancelled within the cache timer. For this reason, I agree that CCP should leave completed orders on the API for at least a few hours.

Ranka Mei
Caldari
Posted - 2011.06.18 01:06:00 - [47]
 

This looks like another braindead idea on the part of CCP to me (like charging 3rd parties for use of the API: that was a real whopper).

Banning IP addresses of people using the API, however temporarily, is barking up the wrong tree, as no one using these 3rd party apps has any real influence over how the app itself handles API requests.

Again CCP appears to be targetting the wrong folks. It's almost like they're purposely sabotaging their own game.

Aineko Macx
Posted - 2011.06.18 04:54:00 - [48]
 

Originally by: CCP Stillman
stuff

I am intrigued you wrote so much text yet didn't answer the most direct questions in the thread.

Viktor Del'Grande
Gallente Forschung und Entwicklungscorp
Posted - 2011.06.18 05:55:00 - [49]
 

Dont fool around, you have lost most of your reputation to me. You are preparing for "premium api services". *Damned 'BizDev' Team*

Lenner Shakiel
Posted - 2011.06.18 06:56:00 - [50]
 

Can we see easier ways to access the API, maybe a email to option for applications that can extract from email or eve mail option for corporations requiring API for recruitment?

Consortium Agent
Posted - 2011.06.18 07:18:00 - [51]
 

Edited by: Consortium Agent on 18/06/2011 07:25:26
O M F G...

Somebody pinch me... please? I must be dreaming. Did CCP not just slap their d*cks in the face of their most ardent supporters - the development community - with some bullsh*t commercial license money grabbing scheme a few days ago?

And now you rush out a devblog a mere handful of days before a new release because it 'fell through the cracks' that tells us 'we've changed some stuff and, oh yeah, if you cause a lot of errors accessing our sh*tty API... you'll be auto-banned'.

How many of us developers that you just d*ck slapped do you honestly think even have the desire to now go rush and fix our applications days before the release of a new API that will penalize us if we don't? This is *exactly* the kind of BS support we've always gotten from CCP with the API... and now you want to charge us what amounts to another year subscription just to access it.

Edit: Oh yeah - almost forgot. Many have asked for clarification. As usual. We have to ask for clarification in the first place. But in either case - you never answer the questions. What *should* we consider an error, exactly? I'm sure you'll tell us one day... oh wait, nvm... right - we've always figured it out *by ourselves* before.

I have only two words left for you CCP... there's only two words that could possibly express how I feel about you, your game and all your stupid BS...

F*CK YOU!

Hel O'Ween
Men On A Mission
EVE Trade Consortium
Posted - 2011.06.18 08:48:00 - [52]
 

Edited by: Hel O''Ween on 18/06/2011 13:18:01
I don't like the change to the Market Orders API.

What about the following? You have this nifty little oderState flag in your database/API already. How about adding this as a paramater to the Market Order API query?

This way the dev has control over what data he receives. Those that only need the active orders could just add that parameter to their query. You could even go so far and go with orderState=0 (Active) as the default setting, if the orderState parameter isn't passed along in the query. That way you would benefit from reduced traffic from those apps that are "too lazy" to implement the change.

Parsing through three different APIs (Market Order, Wallet Journal, Wallet Transaction) sounds like a weird thing to do in order to find out which market order has expired and needs to be resupplied by the trader.

Bonus points for adding a parameter for issueDate (return orders issued on issueDate and later)

Re: Blocking IPs. We really, really need a dedicated API that lets us verify the provided API key vs. desired API.

[Added]
Quote:

[...] my apologies for how late you're getting this information [...]



Oh yeah, and I will totally spent the whole AT9 finals weekend digging through my code. Confused

[Edit:Typos, lots of 'em]
Note to self: Don't post before caffeine has started your brain up, otherwise terrible English will be the result.

Josef Huffenpuff
Caldari
Posted - 2011.06.18 11:06:00 - [53]
 

I just want to re-iterate what everybody else is asking here and we do not have a proper answer for ....

3 Errors a minute on average over a day, all day, every day is probably too much and sounds like a feasable way to cut down on excessive erronous API calls.

But...

Its really easy to generate three API errors in as many seconds when validating API keys on an SMF forum for example. However, we wont normally try again for at least an hour. We don't want to be banned for that even for 3 minutes because that's a major PITA and most software will keep hammering the server anyway until it gets a good answer.

clixor
Celluloid Gurus
Posted - 2011.06.18 11:11:00 - [54]
 

Originally by: Viktor Del'Grande
Dont fool around, you have lost most of your reputation to me. You are preparing for "premium api services". *Damned 'BizDev' Team*


This, it's not uncommon to use full API for a variety of applications. What's next, data limit per account? extra fees for handling data?

As we're talking about small amount of data per user, i can't escape the feeling the motivation for the change is somewhat completely different.

Vaerah Vahrokha
Minmatar
Vahrokh Consulting
Posted - 2011.06.18 11:42:00 - [55]
 

Originally by: "CCP Elerhino"

Smaller Data

MarketOrders will now only return a list of active and non-expired orders. This call was fetching way too much information from the datastore and we decided to alter the query in a simple way, filtering out all expired and processed orders. This is potentially a big change but we figured most people are calling this to watch their active orders and for historic information they get their transactions. We're also quite sure that you'll let us know if we're wrong, as you always so diligently do.



I don't know why I even bother providing feedback with the kick in the balls you just gave us with the new CCP "lollicense".

Anyway since I am an idiot, I am still going to provide an useful feedback: you are gooing to kill 70% of the applications using the market API.

Most people don't (just) want to know if their orders are active but want to gather statistics about their recent past performance over time.

If you remove the expired orders (and no option to fetch them in any way) then you will achieve the opposite of what you worked for: with no past history, the apps will need to be run continuously, overloading your API service.
With the history, they need to be run about once per month instead.

What do you prefer? Every day to fetch 160 orders or once a month to fetch 1000?

Daedalus II
Helios Research
Posted - 2011.06.18 12:49:00 - [56]
 

If this new developer license deal is being implemented, why not require the license key to be trasmitted in each api request? That way you will know exactly which program sends erroreneous requests or too many requests. You can then contact the deveolper of the program and tell them to fix it. If it isn't fixed within x amount of time you block all requests (from anyone) with that license key.

Hel O'Ween
Men On A Mission
EVE Trade Consortium
Posted - 2011.06.18 13:23:00 - [57]
 

Originally by: Daedalus II
If this new developer license deal is being implemented, why not require the license key to be trasmitted in each api request? That way you will know exactly which program sends erroreneous requests or too many requests. You can then contact the deveolper of the program and tell them to fix it.


No need for a developer key to do that.

Other and I already send HTTP headers ("X-Developer-Contact" <email address> and "X-Application-Info" <app name>, in my case) with each API request.

Doktor Csernus
Posted - 2011.06.18 15:29:00 - [58]
 

Edited by: Doktor Csernus on 18/06/2011 15:33:28
Originally by: Josef Huffenpuff
I just want to re-iterate what everybody else is asking here and we do not have a proper answer for ....

3 Errors a minute on average over a day, all day, every day is probably too much and sounds like a feasable way to cut down on excessive erronous API calls.

But...

Its really easy to generate three API errors in as many seconds when validating API keys on an SMF forum for example. However, we wont normally try again for at least an hour. We don't want to be banned for that even for 3 minutes because that's a major PITA and most software will keep hammering the server anyway until it gets a good answer.


Yea, I could use an answer for this question too.

As I've said earlier, its pretty easy to get banned via the API verify process if the user inpunts are incorrect. I can check if the User ID isnt NaN and if the API key is exactly 64 char long, but even if I do it... any false data with "132456" user id and 64 of 'A' letter would go thru a JS verifiy and come back from EVE API with auth error.

CCP, please clarify the "error" calls. All of them? Are you killing the private/corporation/alliance sites with third party api tools for not having license?

Xenofarion
Gallente
Swords Horses and Heavy Metal
Posted - 2011.06.18 16:20:00 - [59]
 

Edited by: Xenofarion on 18/06/2011 16:22:14
Originally by: CCP Stillman
Note that "permanent" bans are only given out in the most extreme cases.


But you do know that most ISPs use dynamic IPv4?

This means when someone is screwing up and gets a permanent ban, but I happen to get that IP in the future, my EVEmon won't work for a day and miraculously work again the next day when I have a new IP again?

And if someone was a terrible troll (what totally couldn't happen on the internet), he could get his IP banned by flooding the API deliberately with wrong calls, then fetching a new IP, get it banned, rinse repeat? (bots *cough*)

This would result in some nice "Wtf why isn't my [application XYZ] working any more?" posts.

It's not like you haven't been in the crosshair of a malicious group that has access to a relatively big Bot-net recently.. ugh

Marcel Devereux
Aideron Robotics
Posted - 2011.06.18 17:39:00 - [60]
 

Please be very clear on what errors will cause a temporary or permanent ban. If outdated keys will cause any kind of ban you will need to provide a way for sites to check to see if a set of keys are still valid. This API call would also not result in any kind of ban unless the request was malformed.


Pages: 1 [2] 3 4

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only