open All Channels
seplocked EVE Technology Lab
blankseplocked Journal Walking / order of rows
 
This thread is older than 90 days and has been locked due to inactivity.


 
Author Topic

realdognose
Caldari
Biotronics Inc.
The Initiative.
Posted - 2011.04.23 13:58:00 - [1]
 

Edited by: realdognose on 23/04/2011 14:36:39
Edited by: realdognose on 23/04/2011 14:34:58
Edited by: realdognose on 23/04/2011 14:34:35
Edited by: realdognose on 23/04/2011 14:06:48
Edited by: realdognose on 23/04/2011 14:06:03
Hello,

i'm not sure, if the journal walking is working as intended:

yesterday i realized, that the API-Server may return transactions in a random order sometimes.
"Random" in the meaning of : The query on the server side doesn't use an explicit ORDER BY.

Please visit your user settings to re-enable images.

Please visit your user settings to re-enable images.

So i filled a bug report, and get the answer that "this is by design, and the server doesn't waste time sorting anything, because an XML-Document don't need to be sorted at all"

i agree in the point about the XML result - but if the SQL-Database will return "wrong" (in the meaning of unsorted) data, the sorting of the XML doesnt matter at all,
because it is even not ensured, that the transactions, printed due to the limit clause are REALLY the first 100, and the id you need for journal walking is REALLY included...

querying data without any explicit ORDER BY will result in a not defined order for each call.

Take this example:
EVE has been reseted, and my transactions have the ids 1 to 15.
now i'm fetching the transactions - all on one page, and the server will return this:

15,14,13,12,6,11,10,9,8,7,5,4,3,2,1

All is good - i can sort them later on. But now imaging, the maximum rowCount is 5.
So actually i need to recall 2 times, using the fromID-Parameter.

first call results in: 15,14,13,12,6

now my script knows: the lowest id is id 6 - i need to use this for journal walking

second call results in : 5,4,3,2,1

outcome: one page of transactions have been missed, using the walking. My Script will never fetch them again,
because my internal "Journal-walking-id" is now set to 15...

Well, this of course is an easy example, but you can imagine, who much transactions may be missed, if you use a rowCount
of 100 or even more.

So how should we work with "journal walking" if we don't know, the id we should use?

i know that the wrong order of an un-ordered-query is very rare - but if it happens once, it may mess-up a complete database, trusting the data served from the
api server...

Can we please get the data sorted by id, so we can "trust" on the journal walking feature and therefore unburden the server instead of "hammering" maximum transactions per page every 20 minutes just to ensure we don't miss anything?

The Question is:
is it ENSURED, that fetching 1000 items per page (without fromID) will RETURN the newest 1000 transactions? (how can this be ensured without any sorting?)

best regards,
realdognose

realdognose
Caldari
Biotronics Inc.
The Initiative.
Posted - 2011.04.23 18:44:00 - [2]
 

okay, i continued talking with the mentioned bughunter, and he told me, that it is ensured to
receive the requested data - just not sorted.

So ids intended to be on page one will be "somewhere" on page 1. and therefore journal walking will
work as expected.

So - everything "seems" to be fine. Embarassed


 

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only