open All Channels
seplocked EVE Information Portal
blankseplocked New Dev Blog: API Incarnated
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: 1 2 [3] 4

Author Topic

Sherksilver
Posted - 2011.06.18 22:45:00 - [61]
 

Please, Please, Please GIVE US DETAILS ON WHAT AN ERROR REALLY IS! - I dunno about the rest of you - but I find it extremely difficult to protect against something - when I do not know what I need to protect against....

Helps us devs out here CCP.

CCP Stillman

Posted - 2011.06.18 23:20:00 - [62]
 

Originally by: Marcel Devereux
Please be very clear on what errors will cause a temporary or permanent ban. If outdated keys will cause any kind of ban you will need to provide a way for sites to check to see if a set of keys are still valid. This API call would also not result in any kind of ban unless the request was malformed.

Right now any error will count towards a temporary throttle.

And just to be clear, you have to be very persistent to get your IP blocked from the API. And should that occur, you just need to file a petition to get unblocked.

And you're completely right about the fact that if you want to validate a lot of keys at once, then this would be bad for you. Right now, we'll have to mitigate that by allow you to do more errors per minute. With the customizable API keys, we can make sure that the APIKeyInfo call, which gives you information about the key, doesn't actually throttle you.

It's weekend right now. Come Monday, I'll talk to him about raising the initial errors/minute threshold to something more sane, as I agree that initially 3 errors/minute is gonna cause a lot of trouble. We'll look into raising this considerably. Once I know something, I'll let you all know.

CCP Stillman

Posted - 2011.06.18 23:26:00 - [63]
 

Originally by: Xenofarion
Edited by: Xenofarion on 18/06/2011 16:22:14
Originally by: CCP Stillman
Note that "permanent" bans are only given out in the most extreme cases.


But you do know that most ISPs use dynamic IPv4?

This means when someone is screwing up and gets a permanent ban, but I happen to get that IP in the future, my EVEmon won't work for a day and miraculously work again the next day when I have a new IP again?

And if someone was a terrible troll (what totally couldn't happen on the internet), he could get his IP banned by flooding the API deliberately with wrong calls, then fetching a new IP, get it banned, rinse repeat? (bots *cough*)

This would result in some nice "Wtf why isn't my [application XYZ] working any more?" posts.

It's not like you haven't been in the crosshair of a malicious group that has access to a relatively big Bot-net recently.. ugh

Again, if your IP is blocked, all you have to do is just file a petition and we'll unblock your IP. But I hope you realize that at the end of the day, we have to make sure that we can provide a consistent quality of service. We have no way of identifying the user of a particular IP. As such, all we can do is block an IP which is causing a higher load on the API than we accept and if they're legitimate users, we just unblock them when they contact us.

CCP Stillman

Posted - 2011.06.18 23:31:00 - [64]
 

Originally by: Ranka Mei

Banning IP addresses of people using the API, however temporarily, is barking up the wrong tree, as no one using these 3rd party apps has any real influence over how the app itself handles API requests.

We've been employing a policy of blocking IPs that cause excessive errors already. And so far, of the handful of times we've had to block anybody, it has yet to be an actual end-user we've had to block. But rather, developers who didn't respect errors and did large amounts of errors.

Marcel Devereux
Aideron Robotics
Posted - 2011.06.19 03:29:00 - [65]
 

Originally by: CCP Stillman
Originally by: Marcel Devereux
Please be very clear on what errors will cause a temporary or permanent ban. If outdated keys will cause any kind of ban you will need to provide a way for sites to check to see if a set of keys are still valid. This API call would also not result in any kind of ban unless the request was malformed.

Right now any error will count towards a temporary throttle.

And just to be clear, you have to be very persistent to get your IP blocked from the API. And should that occur, you just need to file a petition to get unblocked.

And you're completely right about the fact that if you want to validate a lot of keys at once, then this would be bad for you. Right now, we'll have to mitigate that by allow you to do more errors per minute. With the customizable API keys, we can make sure that the APIKeyInfo call, which gives you information about the key, doesn't actually throttle you.

It's weekend right now. Come Monday, I'll talk to him about raising the initial errors/minute threshold to something more sane, as I agree that initially 3 errors/minute is gonna cause a lot of trouble. We'll look into raising this considerably. Once I know something, I'll let you all know.


Thanks for the clarification! The APIKeyInfo API will probably solve 99% of the concern in this thread. Would it be possible to hold off on this throttling until we have that?

Epitrope
The Citadel Manufacturing and Trade Corporation
Posted - 2011.06.19 08:37:00 - [66]
 

Originally by: CCP Stillman
Right now any error will count towards a temporary throttle.

Does this include error 520 "Unexpected failure accessing database" and other server-side errors? When there's a response delay and several calls all return an error at once, getting banned (especially from other, working API calls) is undesirable.

Vessper
Indicium Technologies
Hephaestus Forge Alliance
Posted - 2011.06.19 22:13:00 - [67]
 

We should have a way of verifying the userID, APIKey and the key type without it contributing towards our ban. At present, we have to rely on checking a particular API and look at the error response (if any) to decide the key type for use later on. This is far from ideal so if you're making the error throttling a priority, you should also be giving us a way to avoid making forced errors in our applications.

Sherksilver
Posted - 2011.06.20 00:21:00 - [68]
 

Would also be nice to have the Character API update quickly when character roles change. For some API queries that require specific roles, if those change (yet character API does not update quickly) an app that is trying to access say - POS information can quickly run into issues (as that currently is the only way I know of to determine access allowed without generating an error off the bat).


Azazel Mordred
Minmatar
Cloak of Shadows
Posted - 2011.06.20 05:12:00 - [69]
 

I could be wrong, but I believe the Factional Warfare stats calls are also something you HAVE to generate an error on before you know if the character is enlisted or not.

CCP Stillman

Posted - 2011.06.20 13:43:00 - [70]
 

For tomorrow's deployment we've upped number to 150 errors/minute before you get temporarily block for 3 minutes. I hope this helps. If not, we'll investigate other options for what we can do to mitigate the impact of throttling being in place.

Vandrion
Gallente
The Collective
B O R G
Posted - 2011.06.20 22:04:00 - [71]
 

Originally by: CCP Stillman
Originally by: Xenofarion
Edited by: Xenofarion on 18/06/2011 16:22:14
Originally by: CCP Stillman
Note that "permanent" bans are only given out in the most extreme cases.


But you do know that most ISPs use dynamic IPv4?

This means when someone is screwing up and gets a permanent ban, but I happen to get that IP in the future, my EVEmon won't work for a day and miraculously work again the next day when I have a new IP again?

And if someone was a terrible troll (what totally couldn't happen on the internet), he could get his IP banned by flooding the API deliberately with wrong calls, then fetching a new IP, get it banned, rinse repeat? (bots *cough*)

This would result in some nice "Wtf why isn't my [application XYZ] working any more?" posts.

It's not like you haven't been in the crosshair of a malicious group that has access to a relatively big Bot-net recently.. ugh

Again, if your IP is blocked, all you have to do is just file a petition and we'll unblock your IP. But I hope you realize that at the end of the day, we have to make sure that we can provide a consistent quality of service. We have no way of identifying the user of a particular IP. As such, all we can do is block an IP which is causing a higher load on the API than we accept and if they're legitimate users, we just unblock them when they contact us.


Clarification on these please:

What category should the petition be filed under?

Will the petition be answered in a timely manner? For clarification my definition of a timely manner for this would be 24 hours or less.

Qoi
Exert Force
Posted - 2011.06.21 11:48:00 - [72]
 

Edited by: Qoi on 21/06/2011 11:48:08
the error limit is now working at 150/minute instead of the original 3/minute, however the ban duration is not 3 minutes, but 150 minutes.

CCP Stillman

Posted - 2011.06.21 12:07:00 - [73]
 

Originally by: Qoi
Edited by: Qoi on 21/06/2011 11:48:08
the error limit is now working at 150/minute instead of the original 3/minute, however the ban duration is not 3 minutes, but 150 minutes.

The API wasn't supposed to be open yet. This will be fixed by the time it opens up Smile

LifeHatesMe
Posted - 2011.06.21 13:35:00 - [74]
 

Why not use a system similar to SPAMHAUS (email spam), block by ip temporarily. Block by subnet if it gets really excessive. Manual unblock requests get you unblocked. Even that is better than the current system which processes all errors.

Ikaef Giasep
Posted - 2011.06.21 19:36:00 - [75]
 

Edited by: Ikaef Giasep on 21/06/2011 22:52:25
Originally by: CCP Stillman
Originally by: Qoi
Edited by: Qoi on 21/06/2011 11:48:08
the error limit is now working at 150/minute instead of the original 3/minute, however the ban duration is not 3 minutes, but 150 minutes.

The API wasn't supposed to be open yet. This will be fixed by the time it opens up Smile


While the Server was down today, our corp killboard was querying the API once per hour. The IP of the Server got blocked by you guys. It has been unblocked meanwhile. Nonetheless it is Important that you fix the counting mechanism in such a way that a Service outage caused by CCP does not count towards the blocking count.

.....
Running API Mod V3.3 on Jun 21 2011 13:00


Downloading latest XML file for ...


Your IP address has been temporarily blocked because it is causing too many errors. See the cacheUntil timestamp for when it will be opened again. IPs that continually cause a lot of errors in the API will be permanently banned, please take measures to minimize problematic API calls from your application.

No kills added, 0 malformed, 0 ignored and 0 verified from feed: ika which contained 0 mails.


Time taken = 0.91952800750732 seconds.

Ikaef Giasep
Posted - 2011.06.21 22:32:00 - [76]
 

I sincerly request you to revert the MarketOrder API change immediately. By not providing expired orders anymore, market orders that were created and fulfilled between two API calls will go unnoticed.

The result would be that the MarketTransactions API call would return entries that cannot be related to any market order. Are we supposed to create our own arbitrary orderIDs for them to ensure referential data integrity? How do we know how many orders were created/fulfilled between two API calls if we had two unrelated market transactions? Are these two orders or a single order?

You guys screwed up by not following design principles like the "single responsibility principle" and "separation of concerns". The MarketOrder API call should be self-contained. It should not be required to use other API calls to understand it.

Again, please revert this change asap or provide a solution. Thank you.

Agnitio Reperio Augustus
Gallente
The Scope
Posted - 2011.06.22 13:13:00 - [77]
 

I have a few problems with Incarna. I try to download it, and it says there's a corrupt file or something. And THE WINDOW WON'T CLOSE! This happening to anyone else?

Judy BigShot
Big Shot - For the Bounty Hunters
Posted - 2011.06.23 11:50:00 - [78]
 

Originally by: Ikaef Giasep
I sincerly request you to revert the MarketOrder API change immediately. By not providing expired orders anymore, market orders that were created and fulfilled between two API calls will go unnoticed.

The result would be that the MarketTransactions API call would return entries that cannot be related to any market order. Are we supposed to create our own arbitrary orderIDs for them to ensure referential data integrity? How do we know how many orders were created/fulfilled between two API calls if we had two unrelated market transactions? Are these two orders or a single order?

You guys screwed up by not following design principles like the "single responsibility principle" and "separation of concerns". The MarketOrder API call should be self-contained. It should not be required to use other API calls to understand it.

Again, please revert this change asap or provide a solution. Thank you.


Please provide a solution for these problems, some trading tools have now, due to the change of the MarketOrder-API!
Yes, I could do workarounds and "approximations", but as a *real* software developer I prefer *clean* solutions.
How about an optional parameter for transmitting also cancelled/sold out orders?

Thart
U.K.R.A.I.N.E
SOLAR FLEET
Posted - 2011.06.23 12:40:00 - [79]
 

Originally by: Hel O'Ween
Edited by: Hel O''Ween on 18/06/2011 13:18:01
I don't like the change to the Market Orders API.

What about the following? You have this nifty little oderState flag in your database/API already. How about adding this as a paramater to the Market Order API query?

This way the dev has control over what data he receives. Those that only need the active orders could just add that parameter to their query. You could even go so far and go with orderState=0 (Active) as the default setting, if the orderState parameter isn't passed along in the query. That way you would benefit from reduced traffic from those apps that are "too lazy" to implement the change.

Parsing through three different APIs (Market Order, Wallet Journal, Wallet Transaction) sounds like a weird thing to do in order to find out which market order has expired and needs to be resupplied by the trader.

Bonus points for adding a parameter for issueDate (return orders issued on issueDate and later)

Signed!

Siiee
Recycled Heroes
Posted - 2011.06.23 13:43:00 - [80]
 

Thirded.

I've been using expired orders to track the most recent price I've payed for a set of items. Without them I can do client side caching but then I still have a chance of missing orders entirely if they get posted and filled between API pulls. I'm going to have to pull a lot more data down to check wallet transactions to get the same data. Think large orders being filled in small increments. Just a handful of mineral orders can fill pages of wallet transactions.

Desmont McCallock
Posted - 2011.06.23 19:17:00 - [81]
 

As representative of the EVEMon Dev Team, I would like to inform you that due to the change made in the Market Orders API call, EVEMon can no longer notify the user when an order gets fulfilled or expired (meaning that you managed to remove a key property of the feature).

Therefore on behalf of the EVEMon Dev Team and the EVEMon users, I protest and kindly request that CCP reverts the change.

Thebriwan
LUX Uls Xystus
Posted - 2011.06.23 22:00:00 - [82]
 

I personally am not interested in orders that have been filled, but I can see that traders need that information. But I can also see the problem that arises with the big data volumes that the old system generates.

So I have a proposal to make:

Just make it so that all orders are replied that are outstanding and the filled orders since the last api request for orders.

Tonto Auri
Vhero' Multipurpose Corp
Posted - 2011.06.24 00:28:00 - [83]
 

Quote:
we figured most people are calling this to watch their active orders

Oh, wat? I'm absolutely sure, that 100% (yes, 100%, not even a single one left aside) of people, who persistently monitoring their market orders, do so for only one reason: To monitor completed and expired orders.
If by some strange reason you would've removed active orders from API, people would've lived with such unpleasant, but somewhat usable information, I think, but as you made it now, you just removed the very reason for existence of this API call.

Quote:
We're also quite sure that you'll let us know if we're wrong, as you always so diligently do.

Yeah, we do. You could always trust us in that.

Thart
U.K.R.A.I.N.E
SOLAR FLEET
Posted - 2011.06.24 07:02:00 - [84]
 

Originally by: Thebriwan
Just make it so that all orders are replied that are outstanding and the filled orders since the last api request for orders.


There will be problems with multiple tools used the same API.
The better solution is to limit non-active orders in time. One week for instance.

SghnDubh
BattleClinic
Posted - 2011.06.24 15:17:00 - [85]
 


Um, CCP guys.

The API reports every ship type as '0' instead of the correct ship, which makes all KM's sucking through the API malformed.

I'm reporting this here b/c originally we thought we were victimized by this "forcing you pesky 3rd party devs to optimize your code as if you were huge corporations with tonnes of programmers on staff" blocking scheme. Ok that was maybe harsh but I'm posting on eve-o, aren't I? :D

So, we need some API love QUICKLY please!!!


.




Gambit Ki
Posted - 2011.06.27 04:50:00 - [86]
 

Edited by: Gambit Ki on 27/06/2011 04:50:33
I would also like to ask that the MarketOrders data be fixed. As many others have pointed out the filled order data is important when managed 150+ orders. Try scrolling through your order trying to find what has sold out.

I thought local caching might work but it is true that short sales may not show up inside the 1 hour window making the tracking a guess at best.

Please bring back the different orderStates even if you limit how much data is returned. I'm happy to cache as long as I get the correct data.

Thank you

Melina Octurian
101st Space Marine Force
Nulli Secunda
Posted - 2011.06.27 06:57:00 - [87]
 

My trade team seeds 400+ orders on behalf of corporation, and it is nearly impossible to determine if a particular order has expired/fulfilled unless we go line item by line item. Additionally, 3rd party API programs (such as EVEmon) are not showing corp orders.. only personal orders. A resolution on this matter would be greatly appreciated.

CCP Elerhino


Minmatar
C C P
C C P Alliance
Posted - 2011.06.27 10:02:00 - [88]
 

Edited by: CCP Elerhino on 27/06/2011 10:03:04
We've been looking into the MarketOrder issue but finding a solution that is both simple and optimal has proven difficult. The reason is the nature of market orders, specifically how dates are stored and how they change and how the data can be retrieved in a cost-effective way. Basically us API people are arguing for simple but the DB team understandably wants optimal. Right now it looks like we'll do paging, something like the wallet transaction API, we hope that will solve the problem even though it might cost you guys some extra work. I was hoping for something simpler, e.g. include a week or a month of expired orders in the list but, like I wrote, that solution is not optimal from a DB standpoint. We'll get a solution ready as soon as possible, hopefully within the next few weeks.

The shipType defect has been fixed, we'll deploy that this week. The defect was only in the API layer, the data is correct in the database and will be displayed correctly for older kills in the API after the fix is deployed so you will be able to correct the 0-values that you already have.

Gambit Ki
Posted - 2011.06.27 14:49:00 - [89]
 

Thank you for your response. Paging would work just fine. It's not really that hard to crawl transactions and merge the results to a long list.

Melina Octurian
101st Space Marine Force
Nulli Secunda
Posted - 2011.06.27 19:54:00 - [90]
 

Originally by: CCP Elerhino
Edited by: CCP Elerhino on 27/06/2011 10:03:04
We've been looking into the MarketOrder issue but finding a solution that is both simple and optimal has proven difficult. The reason is the nature of market orders, specifically how dates are stored and how they change and how the data can be retrieved in a cost-effective way. Basically us API people are arguing for simple but the DB team understandably wants optimal. Right now it looks like we'll do paging, something like the wallet transaction API, we hope that will solve the problem even though it might cost you guys some extra work. I was hoping for something simpler, e.g. include a week or a month of expired orders in the list but, like I wrote, that solution is not optimal from a DB standpoint. We'll get a solution ready as soon as possible, hopefully within the next few weeks.



Thanks for the response. I undoubtedly understand the need to slim down the DB storage and volume of data sent with API calls. However, I'm sure that a compromise can be struck in this regard. In terms of the market seed API calls, I think most active traders don't even need a full week of expired orders to be backlogged. Also, is there that much need for API data to be sent more frequently than hourly? I'm not very knowledgeable regarding the frequency of data sent, but would limiting said frequency possibly work toward that "optimal" solution you mentioned?



Pages: 1 2 [3] 4

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only