open All Channels
seplocked EVE Information Portal
blankseplocked New Dev blog: Contracts in the API
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: [1] 2

Author Topic

CCP Guard


C C P
C C P Alliance
Posted - 2011.07.29 14:41:00 - [1]
 

Please check out CCP Elerhino's new dev blog about the contract API features due at the end of summer.

And if you have questions for CCP Elerhino and friends about the contents of the blog, leave them right here in the thread as always.

-CCP Guard

Tork Norand
Mechanical Eagles Inc.
The Ancients.
Posted - 2011.07.29 15:12:00 - [2]
 

Edited by: Tork Norand on 29/07/2011 15:12:16
As a thought, why not move completed contracts and orders to a new database.

This would greatly reduce the amount of data that has to be looked up "live" and still retain a large amount for "after the fact". Then, you start with the live lookup when doing queries of any sort, and if you need to drill down, you can do a slower query against the stored data from the other database. The only time duplication would arise is when something sold between the two, and an identifier can be used to exclude what's already been viewed.

Just a thought.

Edit: Holy crap! I had an accidental "First" post. People must not like database stuff.

SencneS
Rebellion Against Big Irreversible Dinks
Posted - 2011.07.29 15:17:00 - [3]
 

Edited by: SencneS on 29/07/2011 15:34:33
If Corporate contracts have to be locked behind a Director API then that is OK.
If a lacky wants to see the corporate contracts he can login and look at them.

I am also not completely understanding how the items in the contract show up in the API.
The way it reads is you'll get ContractID from one API, and on another you'll have items listed according to ContractID?

Originally by: Dev Blog
The contract lists themselves will not include items, bids or notifications. We'll have separate pages for items and bids but most likely completely skip notifications. The item page will display a list of items for a specified contract, the information will contain things like type ID, quantity and whether the item is being asked for or is included.


I also slight agree with Tork Norand, having the users market and contracts combined together might be easier.
I don't think it's own database would solve issues with getting the data, but combining Market Orders and Contracts on the same API might be a good solution. For Contracts just add an extra column to this already light API pull, for "Order Assignment". You could then use the already existing "buy", "Sell" column and add "Contract".

If it's "contract" then it reads the extra argument column which contains the ContractID.

Make the users of applications that read that API sort out the Contracts from the Market Orders.

Edit:-
I realized after saying "Combine the Market with Contract" how would you value each line item. I'd make them the same price for the entire contract. So if you have 800 items and the total contract is 1B ISK, all 800 items are priced at 1B ISK. You have no way to know which item the seller values more, so all the line items have the same price, the total contract price. This is pretty easy to filter from an API Pulling application.

Note on the "What data do we want"

Bidding on Auctions - Just list it as a line item with a negative value according to your current bid. Mark it as outstanding, so you kind of need to add another column or use the original argument column and expand it a little.

Won Auctions - Keep for a week, negative value again shows you purchased those items on that contract at the contract price.

Selling Auctions - Current Bid price, and the items in the Auction.

Sold Auctions - Keep for a week, list the items and the price it sold for.

WTB Exchange - Same as Bidding Auctions, stays out there until it's either filled or expired. Only this time it's price is set.

WTS Exchange - Same as above, but positive price value.

Expired Contracts - Keep for a week, list items and price contract contained.

Active Courier - Just thought about this, so the API in Market Orders gets a little fatter, because you'll want the Destination, and Collateral in there. But really the Market Orders API is already kind of light in terms of data being sent.

Completed Courier - Completed, Reward is pretty much all I can think of.

Failed Courier - Failed, Collateral shown as negative value, List items held in contract.

Chribba
Otherworld Enterprises
Otherworld Empire
Posted - 2011.07.29 15:53:00 - [4]
 

Interesting Smile

/c

Dierdra Vaal
Caldari
Veto.
Veto Corp
Posted - 2011.07.29 16:44:00 - [5]
 

Good news! Contracts have long been requested as an api feature.

Marcel Devereux
Aideron Robotics
Posted - 2011.07.29 16:49:00 - [6]
 

I would love the ability to view historical data. Can you please provide a way to walk market orders, wallet transactions/journal, and contracts that are older than a month? This data doesn't change and you could write it out in month chunks to flat files and server it up. What do you say?

Zirse
Minmatar
Brutor Tribe
Posted - 2011.07.29 16:51:00 - [7]
 

Quote:
member of the corporation but in the API we're locking it behind a corporation key which only directors and CEOs can create. This is a kind of a security measure in the sense that it prevents members from accidentally giving corporate information to a 3rd party.


Now maybe I'm misunderstanding here, but knowing how meta EVE gets, I don't think it'll be long before anybody who wants access to the corporation data will have it. Is there a way to only send corporate data to API keys associated with that corporation?

CCP Elerhino


Minmatar
C C P
C C P Alliance
Posted - 2011.07.29 17:30:00 - [8]
 

Originally by: Zirse
Quote:
member of the corporation but in the API we're locking it behind a corporation key which only directors and CEOs can create. This is a kind of a security measure in the sense that it prevents members from accidentally giving corporate information to a 3rd party.


Now maybe I'm misunderstanding here, but knowing how meta EVE gets, I don't think it'll be long before anybody who wants access to the corporation data will have it. Is there a way to only send corporate data to API keys associated with that corporation?


That's exactly what it will do if we keep it like this, this data will only be available to corporate keys. Smile

Shepard Book
Posted - 2011.07.29 17:43:00 - [9]
 

Will we eventually have access to market and contract data in evegate?

CCP Elerhino


Minmatar
C C P
C C P Alliance
Posted - 2011.07.29 17:45:00 - [10]
 

Edited by: CCP Elerhino on 29/07/2011 17:47:36
Originally by: Marcel Devereux
I would love the ability to view historical data. Can you please provide a way to walk market orders, wallet transactions/journal, and contracts that are older than a month? This data doesn't change and you could write it out in month chunks to flat files and server it up. What do you say?


This is something we thought about doing instead of the get-latest approach we ended up with. A get-latest solution is what we wanted to do to begin with, the reason why we thought about doing a walking/paging solution was that the method we used in the market orders recently has holes in it. But fetching single items to fill the holes is a better solution because it's a very fast query (by ID) and for some items you might have to page through a mountain of items to get to the single one you need because the paged items would be ordered by creation date and the single items you need can get quite old.

But this is not final and is actually something we'd really like to see some discussion on here. Wink

Thebriwan
LUX Uls Xystus
Posted - 2011.07.29 19:51:00 - [11]
 

This is good news.

In my opinion the corp contract should be secured with corp - keys.

What I would like to know how you want to implement the listing of the contents of a contract. Like with containers?

And if you are looking into more functionality for the API - would it be possible to have the assets from a single station with a shorter timer? That would be fabulous!

Golden Gnu
Gallente
The Golden Gnu Corp
Posted - 2011.07.29 20:25:00 - [12]
 

I'm sure this will make a lot of people happy.
I'll let the smart guys/girls find your perfect solution ;)

I love all the work the API is getting...
It's awesome, you know, the API.

Durin Sarga
Posted - 2011.07.29 21:05:00 - [13]
 

Most of this looks good. I got a little confused on what exactly the API output would 'look' like. The devblog was short on graphs, charts, figures, etc. : )

What are the chances of being able to export market item historical data. The stuff we can find on the second page of the Market details window when we select Table instead of Graph. THAT information would be a HUGE +1.

My 2 ISK.

Durin

Johnathan Roark
Caldari
The Graduates
Morsus Mihi
Posted - 2011.07.29 23:05:00 - [14]
 

Sounds good, any information on what the calls might be? I want to look. :)

PS, I would like an employment history api.

Risingson
Posted - 2011.07.29 23:13:00 - [15]
 

I wish you would do an api delivering info on ongoing Icursions !

Andski
GoonWaffe
Goonswarm Federation
Posted - 2011.07.29 23:36:00 - [16]
 

I'm so glad that this is finally being implemented. MY SPREADSHEETS thank you.

Ydnari
Gallente
Estrale Frontiers
Posted - 2011.07.29 23:46:00 - [17]
 

Useful.

I hope the items part of this will include items inside courier contracts, will make it much easier to keep track of stuff in flight.

SirHarryPierce
Posted - 2011.07.30 03:53:00 - [18]
 

Great! As mentioned above: my spreadsheets will thank you :).

Ikaef Giasep
Posted - 2011.07.30 05:56:00 - [19]
 

Originally by: Tork Norand
Edited by: Tork Norand on 29/07/2011 15:12:16
As a thought, why not move completed contracts and orders to a new database.

This would greatly reduce the amount of data that has to be looked up "live" and still retain a large amount for "after the fact". Then, you start with the live lookup when doing queries of any sort, and if you need to drill down, you can do a slower query against the stored data from the other database. The only time duplication would arise is when something sold between the two, and an identifier can be used to exclude what's already been viewed.


This is a solution that would really help. Just constantly copy new data to a separate server once in while and have the API operate on it instead. Every major company I know uses such a mechanism (for reporting). An additional benefit would be that the database of the other system could be specifically designed for API purposes. This could help to speed up API access

Cyaxares II
Posted - 2011.07.30 07:14:00 - [20]
 

Edited by: Cyaxares II on 30/07/2011 07:48:11

got my hopes up when seeing the devblog title but was of course disappointed.

market orders don't need a history because journal & transactions log keep track of everything that happens.

guess CCP is now playing "corrupt a wish" with us - "yes, you can have that contracts API you have been asking for for years but we will change one little detail to make it useless"

maybe try some of your own "agile" methodology and read the forums to see what use cases people actually have in mind and then think about how to implement them. Not "oh they want a contracts API, let's create some API for contracts (that is easy to implement) and everything will be fine!"

edit: I hate having to do your job for you but here you go...

from my POV there are three major areas of application for a contracts API

(a) "I want an application that gives me an overview over my revenue, profits, most profitable items, ... for the last week/month/quarter/... . Part of my business relies on buying from and selling to contracts."
(b) "I have this guy's Full API key and want to check if he had any financial connections to hostiles, possibly undisclosed alts, ... in the recent to mid-term past"
(c) "Do I have to log in to renew my filled/expired contracts already?"

The proposed API does (c) well but (c) is the scenario that is centered around cheap convenience the most.
It fails completely at (b).
It is mostly useless when used for (a) which is imo the prime use-case.

It may be possible to work around the API to arrive at some sort of solution for (a) - but then we are at "I have to run this tool multiple times per day to create my own local contracts history. If I forget to run it for a few days my whole accounting for that time period will be screwed up."
But even then we still face the problem that only "me selling with WTS/Auction contracts" is covered. "Me selling to WTB" or "me buying from WTS" leaves no tracks.

"but having something is better than having nothing" - if that means that the topic "Contracts API" is dealt with for the next few years in the eyes of CCP, then no having a mostly useless solution is not better than having no solution at all.

Thebriwan
LUX Uls Xystus
Posted - 2011.07.30 09:27:00 - [21]
 

Originally by: Cyaxares II
A bunch of stuff


I really had have it with this "I am f*** angry at CCP because they do not exactly as I wish"-attitude.

Was is really necessary to write a half a posting with a whine and an attack at CCP?
Do you really think that this is doing your POV any good?
And last but not least do you really think that anyone can imagine all the things that people want to do with the API?

This dev blog is clearly stating why they can not do to much history.
AND you could have presented you POV in a friendly, constructive manner.
Then maybe there could have been some movement in you direction.

Abdiel Kavash
Caldari
Paladin Order
Fidelas Constans
Posted - 2011.07.30 11:47:00 - [22]
 

So, as far as I understand, this is for my contracts only? Meaning, I can't ask the API "show me all the contracts for Estamel's Modified Thingamajig in Jita"?

Desmont McCallock
Posted - 2011.07.30 11:58:00 - [23]
 

Originally by: Abdiel Kavash
So, as far as I understand, this is for my contracts only? Meaning, I can't ask the API "show me all the contracts for Estamel's Modified Thingamajig in Jita"?


Nope, it's not a contract finder.

Zaxix
Daisy Hill Puppy Farm
Posted - 2011.07.30 14:08:00 - [24]
 

Red Frog/Black Frog has been one of the long time proponents of the contract API. How will the API handle courier contracts? From our viewpoint, historical data is essential, since we can use that data in so many different ways (e.g., rewarding pilots for meeting benchmarks, rewarding clients for repeat business, figuring out the best placement of resources, etc etc.). Will we simply have to create our own database and log the data once the API goes live?

Still its good to see the Contracts API taking off!!!

Consortium Agent
Posted - 2011.07.30 15:13:00 - [25]
 

o/ CCP

Nice devblog and it raises one question:

Quote:
the datasets simply aren't optimized for this kind of a query


Why not optimize the datasets then? Doesn't this seem like the most logical approach to addressing performance problems instead of limiting what players can do because your dataset was poorly designed?

I know, I know... it's a lot of development work blah, blah, blah - there are ways around this. I presume your API dataset is a replicant of your core dataset in some fashion? If so, DTS is a wonderful thing - you could build new, optimized, tables and procedures and then just have DTS translate the replication from the 'old' core dataset to the new API tables.

Having said all of that, however, I personally have an interest in a contracts API and will take whatever you'll give us, frankly. I would prefer to be able to get finished contracts from the API for some period of time (say 2 days perhaps) as, from the perspective of checks and balances, it would make things easier, but I could live without them if I had to ;)

And one more question - would corporate contracts not fall under a corp key in the new API you're working on? Doesn't this, by its very nature, eliminate the possibility of a luser giving away corp details to people he shouldn't?

Good work... keep it up!

Cyaxares II
Posted - 2011.07.30 21:31:00 - [26]
 

Edited by: Cyaxares II on 30/07/2011 21:37:19
Originally by: Thebriwan
This dev blog is clearly stating why they can not do to much history.
AND you could have presented you POV in a friendly, constructive manner.
Then maybe there could have been some movement in you direction.

sure, I'll take the blame for preventing CCP from reworking this proposal into something useful (because they feel insulted by my unfriendly feedback).

Originally by: Thebriwan
And last but not least do you really think that anyone can imagine all the things that people want to do with the API?

People have been asking for a Contracts API for several years now - hardly a secret why they want one. There have been countless threads on this topic, proposals in F&I, it has been discussed by CSM and ranked fairly high in their crowd-sourcing ("One of the more common applications for the API is collecting data for book keeping of your wallet. However, the lack of contracts API means there is a large gap in the data.", " We need a full contracts API. Currently the only information you can get about contracts is from the Wallet, and contract related entries don't list the items contained within or even who you were interacting with. This is important because much of the expenditures and transport of goods in large alliances are through contracts, and adding the system will help greatly with auditing and stuff like that. "), ...

The only explanation how they could be ignorant to the reasons why people want that API so much would be if they never spend time on the S&I, MD & F&I forums and don't listen properly to the CSM.

I think CCP did decide to react to the latest CSM request by implementing a API that they can call "contracts API" but that is actually useless. Gets rid off one item in their backlog without causing them a lot of work. Nice & cheap success story about listening to player requests while actually ignoring them.

I also don't like how you label my post as "unconstructive" - it's not like I made these use-cases up (see quotes above), I took some care to wrap them into nice user-centric stories that don't foreclose the solution of the problem, I even laid out how imo nothing would be better than the proposed solution...

Marcel Devereux
Aideron Robotics
Posted - 2011.07.31 03:22:00 - [27]
 

Originally by: CCP Elerhino
Edited by: CCP Elerhino on 29/07/2011 17:47:36
Originally by: Marcel Devereux
I would love the ability to view historical data. Can you please provide a way to walk market orders, wallet transactions/journal, and contracts that are older than a month? This data doesn't change and you could write it out in month chunks to flat files and server it up. What do you say?


This is something we thought about doing instead of the get-latest approach we ended up with. A get-latest solution is what we wanted to do to begin with, the reason why we thought about doing a walking/paging solution was that the method we used in the market orders recently has holes in it. But fetching single items to fill the holes is a better solution because it's a very fast query (by ID) and for some items you might have to page through a mountain of items to get to the single one you need because the paged items would be ordered by creation date and the single items you need can get quite old.

But this is not final and is actually something we'd really like to see some discussion on here. Wink


I'm fine with a API that provides just the recent and out standing contracts as long as we can someone get at the older contracts. Is there a way to go from journal refID to contractID and look up single contracts that way? We would of course still need a way to get at historical journal entries so we could retrieve the refID of completed contracts.

Desmont McCallock
Posted - 2011.07.31 08:22:00 - [28]
 

I think that this line in the devblog answers your question.
Originally by: CCP Elerhino

Similar to the Market Orders, the Contract list will contain contracts that are outstanding or in progress, as well as recently issued contracts. We'll also have a separate page or a contractID parameter so that you can look up a single contract.


Thebriwan
LUX Uls Xystus
Posted - 2011.07.31 09:29:00 - [29]
 

Originally by: Cyaxares II
I think CCP did decide to react to the latest CSM request by implementing a API that they can call "contracts API" but that is actually useless. Gets rid off one item in their backlog without causing them a lot of work. Nice & cheap success story about listening to player requests while actually ignoring them.


I don't see where the contracts API would be useless.
For my use cases the way they described it, it would fit perfect.

And what you did not see is that they cannot give access to all contracts (or orders) because of the db stress this would generate.

That is called a compromise.

Originally by: Cyaxares II
I also don't like how you label my post as "unconstructive" - it's not like I made these use-cases up (see quotes above), I took some care to wrap them into nice user-centric stories that don't foreclose the solution of the problem, I even laid out how imo nothing would be better than the proposed solution...


In everything that is communicated the "how" is as important as the "what". You have your good and valid points brought into context with a rant - even if the dev can see your point it will be tainted. That's all I'm saying.

Matthew
Caldari
BloodStar Technologies
Posted - 2011.07.31 15:22:00 - [30]
 

So, if we take it as read that queries against contract/order end date are expensive, but those against start date are reasonable, and that this cannot easily be changed, then the proposed solution seems reasonable. Given this, there seem to be three key design parameters to consider:

Minimum Query Frequency

This is defined by the time window of recently issued orders included in the API response. If the API only includes the last weeks worth of recently issued orders, then the user must query the API at least once every not-quite-a-week to be sure of having seen all records.

Note that the IndividualContract API does not cover this issue, because even if it allows the specified contract to be retrieved, the user has no API method of knowing the missing contractIDs.

So, the question here would be, is it reasonable for a user to end up with holes in their records if they do not query the API at least once every week? What would be a reasonable minimum interval?


Start-Up History Pull

Long-term users of the API can build up a history. But where a new user starts to use the API, or a new API is introduced, the ability to retrieve older historical information is limited.

With the currently proposed API, the ability to build up a history would be limited by the time window of recently issued orders. In this case, you would only be guaranteed a complete history going back one week from the point at which you start to use the API.

This is where Marcel's suggestion would come in. Once a contract/order is completed the record no longer changes. So the API server could archive this information into some sort of static cache, rather than troubling TQ for it every time. We could then get three API calls:

CurrentContracts - as proposed in the dev-blog, non-expired items as well as items issued less than X time ago.

ArchivedContracts - All expired items, ordered by creation date, with a paging mechanism to go back 1+ month. (this would just require filtering by isExpired, rather than ordering by ExpiryDate, which is hopefully a less costly query)

IndividualContract - as proposed in the dev-blog, give a contractID and get that individual contract back.

The new user starting to use the API can then use ArchivedContracts to retrieve an initial "boot-strap" historic dataset. They can then keep it updated using CurrentContracts, filling in gaps with IndividualContract.

The other advantage of ArchivedContracts is that it would act as a failsafe against the Minimum Query Frequency issue above. If the user does miss the Minimum Query Frequency for some reason, they can use ArchivedContracts to recover the missing contract details.


Bulk vs Individual Update of Contract Status

This point is about the efficiency of retrieving updates via CurrentContracts vs having to make a lot of calls to IndividualContract.

For example, if CurrentContracts returns the most recent week of issued contracts, but the typical life of your contract is 10 days, then you are going to have to use IndividualContract to update the majority of your completed contracts, leading to a lot of spamming of the IndividualContract API. In this case it may be more efficient to extend the time interval returned by CurrentContracts.

This is probably something best determined by looking at the distribution of time-to-expiry on all contracts to determine a suitable minimum coverage of CurrentContracts.


Pages: [1] 2

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only