open All Channels
seplocked EVE General Discussion
blankseplocked EVE RNG abnormal streakage... now with 80k+ test datapoints !
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: 1 2 3 4 5 6 7 [8] 9

Author Topic

Akita T
Caldari Navy Volunteer Task Force
Posted - 2008.02.22 22:53:00 - [211]
 

Edited by: Akita T on 22/02/2008 23:14:20

Originally by: Roche Pso
Originally by: Akita T
For instance, if 1 means heads up and 0 heads down, then
010100101010011010011101010010010100101000000100101001001011100101001010
would be more or less random, but
010100101100000000000000001010011111111111111111010101001001010010010011
would definetely be a "huh?" moment (especially if repeated often in a longer chain of tries).


Why would the second series be a "huh?" moment but the first not? They both have ***EXACTLY*** the same probability of occuring

You are correct in saying these PRECISE sequences have the same probability of occuring, given a perfectly random 50-50 generation.
But, there's a whole lot more sequences that "look like" the first one compared to how many sequences there are that "look like" the second one, and they ALSO have the exact very same probability of occuring.
Therefore, yes, the second sequence (or better said, "a sequence looking like the second sequence") is less likely compared to the first sequence (i.e. "a sequence looking like the first sequence").

Originally by: Matthew
There are 194 unique damage values in the list - just as you predicted. Now, I've not familiarized myself enough to know the distribution I should be looking at. The main bulk seem to range from 299 occurances, to 384 occurances. When sorted in order of occurances, this chunk step up in frequency in a nice straight line, though the damages are "out of order". I suspect this is a manifestation of the different types of hit.
There are three values that stand out from this general pattern. 9.7 and 28.9 occur 166 and 137 times respectively. Again, not sure what the significance of that is within the damage calculation.
58.2 occurs 650 times. Given your assertion earlier that wrecking hits occur 1% of the time regardless of all other factors, and this is counting from a sample space of 65524 samples, this is not far off the 655 wrecking hits we would expect to observe.

The damage value we see is the rounded value of nominal damage (unrounded) * damage multiplier (possibly float = 0.49 + number_from_RNG in most cases), and the damage multiplier goes from *0.5 to *1.48, with the "bottom-most part" being an abnormal *3 multiplier (NOT the "top-most part", as you'd expect... generated random number and therefore damage multiplier, if arranged by value, goes either : wrecking, worst shot, average shot, best nonwrecking... or maybe even : best nonwrecking, average, worst nonwrecking, wrecking... and NOT best nonwrecking close to wreckings as you might expect intuitively).

You're getting more or less 300-380 values for each rounded number, but you're only getting 166/137 for the extremities (roughly 300 total), which is quite normal imho (shows that the distribution of generated random numbers is fairly uniform across 191 subsections of equal "width", and the other three sections of "not equal" width pretty close, proportionally speaking).

I went back and looked at raw damage multiplier on the gun used... it was x2.61995893362, and against the 0EM/20TH resists I was shooting at, with 5EM/3TH Standard S ammo, the "nominal" damage should have been 19.387696108788, with the actual expected damages between 9.693848054394 and 28.69379024100624, and a wrecking hit damage of 58.163088326364.
Since after rounding you only get 9.7 for numbers between 9.693848054394 and 9.75, but you get 9.8 for numbers between 9.75 and 9.85, this looks quite normal.

P.S. On second thought, maybe it's best if you normalize wrecking shots to a value of either 19.387696108788*0.49 (9.49997109330612) or 19.387696108788*0.495 (9.59690957385006)... so either 9.5 or 9.6.
I'd tend to prefer 9.6 since it's close to the "average" of those numbers in the bottom-most 1%, is where the actual multiplier would end up as being rounded most of the time if there would have been no "wreckings exceptions" (you'd also get more 9.7s though) and it gives you a nice "uncut" distribution on .1's

Akita T
Caldari Navy Volunteer Task Force
Posted - 2008.02.22 23:08:00 - [212]
 


Originally by: Monikerina
[...]
For invention n is not 2, success and failure. A number is pulled from an unknown set of possible outcomes ,n, that is then applied to a formula to determine success or failure.
How can we use this if we don't know the possible outcomes?

The way invention succeeds or fails should be very similar (if not identical) to how the wreckings selection works, but instead of using 0.01 (1%) as cutoff value, they use the actual "calculated invention chance" as cutoff value.

So, if these were invention tries instead of gun shots, ALL shots above and below a certain "damage value" (which would depend on actual/calculated invention chance, based on item used, skills of user, etc) would be either failures or successes.

Monikerina
Amarr
Macross Space Defense Squadron
The Conglomeration of Ill Advised Ideas
Posted - 2008.02.23 00:10:00 - [213]
 

Originally by: Akita T

Originally by: Monikerina
[...]
For invention n is not 2, success and failure. A number is pulled from an unknown set of possible outcomes ,n, that is then applied to a formula to determine success or failure.
How can we use this if we don't know the possible outcomes?

The way invention succeeds or fails should be very similar (if not identical) to how the wreckings selection works, but instead of using 0.01 (1%) as cutoff value, they use the actual "calculated invention chance" as cutoff value.

So, if these were invention tries instead of gun shots, ALL shots above and below a certain "damage value" (which would depend on actual/calculated invention chance, based on item used, skills of user, etc) would be either failures or successes.



And how does this relate to the Pearson's chi-square test which is the topic we're discussing.

Akita T
Caldari Navy Volunteer Task Force
Posted - 2008.02.23 00:31:00 - [214]
 

Edited by: Akita T on 23/02/2008 01:38:13

Ahem... well... was not necessarily directly referring to that specific test, I might have taken your quote a little bit out of context Embarassed
However, read further, as it should also apply to that too, since it's about the nature of the results we got from the raw data.
In this case, we have 194 possible outcomes, with one of them of 1% chance, 191 of roughly equal (assumed) chance, and two more with roughly half of the previous mentioned chance for those 191 others.
___

Well, we have a RNG that generates numbers in a very fine interval (most probably floats, between 0 and 1), but we can only "see" 194 distinct intervals (one large interval for floats between 0 and 0.01 as wreckings, two smaller intervals, one for floats slightly above 0.01 the other for those close to 1, and 191 "equally wide" intervals for the rest of the displayed rounded damages), and we don't have all the numbers generated by it, however we're fairly certain we do have a significantly large number of them (anything between 5% and 50% of them, depending on highest/lowest estimable SiSi activity that would require the RNG to "draw" a number).

First you "translate" the damage values you see into something closer to the original floats (you will have some slight anomalies around 0.015 and below, and 0.995 and above, and you only have 194 discreet values, so it's not perfect, but good enough for what we plan to use them for).

Then, you pick an arbitrary invention chance, like, say, 50% (we'd be re-doing this for several other values of base invention chance).
Whatever goes below 0.5 (in our initially chosen 50% base chance example) you mark down as "0" (fail), whatever goes above 0.4 you mark as "1" (success).
THEN you analyze whatever it is you think should be analyzable from that sequence.
THEN you pick another base chance and start all over again (40%, 25%, whatever, you name it).

Essentially, we're turning this quality-of-hits data log into an equivalent invention run with several fixed/known invention chances.
For instance, we could then use the "longest expected run" criteria to look for abnormally large streaks given our roughly 80k tries (or at least the first 65k we can easily manipulate via Excel).
As mentioned and linked earlier in the thread, the formula for "longest expected runs" (formula #15 inside the link) is log [base 1/probability] out of trials*(1-probability).
This goes either way, at least within some reasonable margin of error, for both runs of successes (1s/heads) and runs of failures (0s/tails).

If we can notice CONSISTENTLY longer streaks as longest expected for certain values of base chance, we know we have a problem, at least with those particular base invention chances, if nothing else.
And why ? Because this is the most likely way EVE actually uses the RNG output to begin with.

Semkhet
Dark Tornado
Ethereal Dawn
Posted - 2008.02.23 12:31:00 - [215]
 

Edited by: Semkhet on 23/02/2008 12:44:36
Originally by: Akita T

If we can notice CONSISTENTLY longer streaks as longest expected for certain values of base chance, we know we have a problem, at least with those particular base invention chances, if nothing else.
And why ? Because this is the most likely way EVE actually uses the RNG output to begin with.



First, let me stress that I have great respect for your propension to investigate obscure aspects of EvE.

As soon CCP stated the intention of implementing invention, I redirected the training of an industrial/research alt specifically in the opportune direction. Since them, that alt has performed myriads of invention jobs in the most various fields, and skills-wise, is approaching 50 mil sp mainly in carebear stuff.

Considering that in any probability-based process, the highest probable outcome is failure, my SUBJECTIVE impression is that success is artificially clustered instead of being randomly distributed.

I certainly don't deny the existence of genuine streaks. However I noticed that very few of my batches of 10 invention runs contain only 1 successful job. It's often either no success, or many successive successful jobs even when using the worst chance decryptors.

Since I set the jobs of my batches almost all at the same time successively, and I also retrieve them successively, a pertinent suspicion would point to some form of time-related variable having great influence to the overall result of a given batch.

Now the problem that you have, is that statistics tools are UNABLE to differenciate a true random from a good quality pseudo-random sequence, unless you can feed the appropriate statistical instrument with a sequence whose length extends BEYOND the period of the PRNG. In the current case, you neither have access to a raw sequence, nor can you define which sequence length would have a good chance to extend beyond the period of a specific PRNG you don't even know the formulation to start with...

Hence even if you could identify with reasonable precision that the results of probability processes within EvE seem flawed (which is my intuition), it still would remain very difficult to identify the culprit within all the stages and intermediary transformations which might be involved. In a nutshell, with the limited and degraded data at your disposal, I doubt you could reverse-engineer the transformation chain.

That's why, with all due respect, I consider spending time on this subject as somewhat futile.

Akita T
Caldari Navy Volunteer Task Force
Posted - 2008.02.23 13:28:00 - [216]
 

Well, I'm not exactly trying to find out the exact problem, I'm simply trying to show to the guys from CCP that there is something fishy with either the RNG itself, or the way they use its output in the game right now... so that they could be perhaps persuaded to investigate further (being in a much better position to find out the actual problem, if any).

Semkhet
Dark Tornado
Ethereal Dawn
Posted - 2008.02.23 14:06:00 - [217]
 

Originally by: Akita T
Well, I'm not exactly trying to find out the exact problem, I'm simply trying to show to the guys from CCP that there is something fishy with either the RNG itself, or the way they use its output in the game right now... so that they could be perhaps persuaded to investigate further (being in a much better position to find out the actual problem, if any).



Commendable intention. However since lack of reaction is something as tangible as a formal reaction, you can consider symptomatic that no CCP-originated intervention shows up in your thread after 8 pages...

But like the old say: as long there is life, there is hope Wink

MotherMoon
Huang Yinglong
Posted - 2008.02.23 15:16:00 - [218]
 

Edited by: MotherMoon on 23/02/2008 15:18:24
Edited by: MotherMoon on 23/02/2008 15:18:04
Originally by: Akita T
Edited by: Akita T on 16/02/2008 22:37:18

I hate to repeat myself, but have you taken a look at the data Etho Demerzel posted ?
He made roughly 900 attempts in identical conditions, getting roughly 50-50 successes-failures.
For practical purposes, we might as well assume he's working with a base 50% chance.
This is just ONE person that DID collect the data.

In this series, he encountered AT LEAST 3 streaks of 16 failures, 5 streaks of 8 failures, and 8 streaks of 6 successes.
The probability that just ONE of his 16-failure streaks would be happening at all is already low enough (aprox 1.37%) in a series of 900 tries, I'll give you that, so no big deal, right ?
And the number of people doing invention is pretty high, so again no big deal, right ?
But he had not one, but three of those 16-streaks.
Ok, it might not be "such" a big deal either, and he could be the lucky (or unlucky, however you put it) person to have both recorded his data AND gotten "anomalous results" that should only happend only about 1 in a half million times somebody tried something like this.

I mean, seriously, THAT post of his alone should be ample PROOF that there's a very high probability the particular way EVE uses the RNG numbers is indeed broken, causing excessive streaks for some individuals, a lot more as it should Twisted Evil
Hey, what can I say, can't get more finger-pointy while still remaining sincere.



I think you missed the point.

the dev stated that lets say he took 60 attemps.

and EVERY time he got a kill mission dispite it being 5%.

well truth is the RNG didn't make 60 in a row it did this


00001101100100101011010111010010101011010111
0100100101010101000100101011
01011101100110110010010101010110101101010110

see what I mean? the RNG generates numbers for all players, and unless he was able to start every mission without having other players starting missions the his results show he got unlucky, but the number generator COULD still be being random and one player is just hitting the bad spots because it's not per player but server wide.

thus the devs says you would really need a conterite way to collect out side of your gut.

Akita T
Caldari Navy Volunteer Task Force
Posted - 2008.02.23 18:28:00 - [219]
 

Originally by: MotherMoon
I think you missed the point.
[...]see what I mean?

*sigh* Yes, I see what you mean, but no, I didn't miss the point.
We have had this kind of discussion several times over in this thread, be a dear and read it where it's already explained in several places and in several ways, since I hate repeating myself, and ran out of ways to say the same thing in different words.
Short version : yes, your example is a possibility, but a very, very unlikely one... it would be more likely to have a broken/unfair/biasked/streaky RNG rather than that being the actual sample of a true random number generator.

Vinchester
Posted - 2008.02.23 21:48:00 - [220]
 

I have read all this, and this is what i have to say:

lolwut?

Akita T
Caldari Navy Volunteer Task Force
Posted - 2008.02.23 23:26:00 - [221]
 

So, in other words, it doesn't bother you at all that the basic mechanism behind a lot of "events" in EVE (from invention failure/success, ECM jamming success/failure, weapon damage, scan failure/success/accuracy or mission assignments, to rare belt rate spawns or even exploration results) might have some pretty "uncommon" streaks ?
Pardon me if I return your "lolwut?" then...

Ter Fordal
Posted - 2008.02.24 13:42:00 - [222]
 

Thanks for the data Akita. One standard test is to take pairs of numbers seperated by varying distances in the sequence and check for correlations. You can also do this graphically which is what i did, plotting up to 32k pairs seperated by 2,3 and 4 steps in the sequence. It's far from conclusive but heres an example graph for 10,000 pairs along with a plot of the frequency of each damage type for all 65000 samples, i stripped out all the wrecking shots and the highest and lowest damage values:

http://dl.eve-files.com/media/corp/TerFordal/rand.JPG

Any strong bias towards producing consecutive high or low numbers should show up as a clustering of points towards one region of the graph, if they are evenly spread then any number is equally likely to follow any other number. Having checked a fair few they look pretty good to me. Of course we are not getting the raw data from the RNG but with about 190 distinct damage values and 65000 samples we are getting some reasonable precision, effectively we are probably testing the 7 highest bits of the RNG output.

Ter Fordal
Posted - 2008.02.24 13:49:00 - [223]
 

Just a thought, would you get more distinct damage values if you were doing bigger damage? seems like its just rounded to the nearest 0.1, hence the low frequencies for the highest and lowest damage values.

Semkhet
Dark Tornado
Ethereal Dawn
Posted - 2008.02.24 14:18:00 - [224]
 

Edited by: Semkhet on 24/02/2008 14:18:17
Originally by: Ter Fordal
Thanks for the data Akita. One standard test is to take pairs of numbers seperated by varying distances in the sequence and check for correlations. You can also do this graphically which is what i did, plotting up to 32k pairs seperated by 2,3 and 4 steps in the sequence. It's far from conclusive but heres an example graph for 10,000 pairs along with a plot of the frequency of each damage type for all 65000 samples, i stripped out all the wrecking shots and the highest and lowest damage values:

http://dl.eve-files.com/media/corp/TerFordal/rand.JPG

Any strong bias towards producing consecutive high or low numbers should show up as a clustering of points towards one region of the graph, if they are evenly spread then any number is equally likely to follow any other number. Having checked a fair few they look pretty good to me. Of course we are not getting the raw data from the RNG but with about 190 distinct damage values and 65000 samples we are getting some reasonable precision, effectively we are probably testing the 7 highest bits of the RNG output.


That's a Mickey Mouse test followed by a baseless assumption. Any mildly decent PRNG spread its specific pattern over extremely long sequences of numbers. Maybe you should check the period duration of simple PRNG's to get a clue what you are talking about. A friendly advise: if the above post represents the best stand of your knowledge on the matter, don't attempt to apply for a position in cryptanalysis unless you want to be ridiculed... Laughing

Counterparty
Posted - 2008.02.24 15:19:00 - [225]
 

You can't resist hinting you are some kind of crypto god. Then you claim analyzing this RNG is beneath you and pointless.

Until you start posting results, prepare to be ridiculed.

Semkhet
Dark Tornado
Ethereal Dawn
Posted - 2008.02.24 15:40:00 - [226]
 

Edited by: Semkhet on 24/02/2008 15:50:20
Originally by: Counterparty
You can't resist hinting you are some kind of crypto god. Then you claim analyzing this RNG is beneath you and pointless.

Until you start posting results, prepare to be ridiculed.


Nah, I'm just someone who had to work with this professionally, applying classic & well documented knowledge you do not seem familiar with, or you wouldn't have posted the way you did Laughing

So you can go on with your smacktard rethoric, fact of the matter is that either you comprehend the mechanics behing PNRG's & TRNG's and if you support or counter an argument you do it technically, either you go loose your time in CAOD Wink

Akita T
Caldari Navy Volunteer Task Force
Posted - 2008.02.24 15:48:00 - [227]
 

Originally by: Ter Fordal
Just a thought, would you get more distinct damage values if you were doing bigger damage? seems like its just rounded to the nearest 0.1, hence the low frequencies for the highest and lowest damage values.

Yes, damage is always displayed rounded to one decimal, and if I was to use some bigger damage guns, I'd get a "finer grain"... but that would also mean a longer refire rate, so more time to get the same number of results, and less of the raw RNG output (percentually speaking), since others are more likely to call on the RNG while I run these tests.

However, since the main issue was to "research for unnatural streaks" of success/failure for a certain success percentage (more or less arbitrarily chosen), the "grain" we have right now is already more than adequate already.
Remember, we're NOT trying to find out wether the RNG is a perfect RNG (it can't be), we're merely trying to discover wether there are some unusual numbers of longer-as-normally-expectable clusters of "below a certain value" or "above a certain value"... you know, test the validity of the "if I deliver my invention jobs fast in succession, I get mostly successes or mostly failures regardless of actual chance to invent" complaint ?

Matthew
Caldari
BloodStar Technologies
Posted - 2008.02.24 16:42:00 - [228]
 

Originally by: Semkhet
Nah, I'm just someone who had to work with this professionally, applying classic & well documented knowledge you do not seem familiar with, or you wouldn't have posted the way you did Laughing


I would certainly hope that when you are working with it professionally, that you act in a rather more, dare I say it, professional way. By all means explain why people are wrong, but berating them for being wrong is completely unnecessary.

Now, as to the correlation test.

Given that, in your own words, "a pertinent suspicion would point to some form of time-related variable having great influence to the overall result of a given batch.", testing for a correlation over time is an entirely pertinent thing to do. Though admittedly it would need to be rather more wide-ranging than the analysis Ter Fordal performed.

As you have already pointed out, we are not testing the PRNG directly, but the final output of a probability process involving several stages, any one of which may be generating the time-related variable. Therefore, the correlation may in fact have nothing at all to do with the PRNG, and therefore the period of the PRNG is potentially irrelevant.

Given that our suspicions have been raised based on a sample set much smaller than the likely period of the PRNG, it is not unreasonable to think that a correlation may manifest itself within sample sets of this smaller size. It is certainly worth doing, if only to eliminate that possibility.

Originally by: Akita T
test the validity of the "if I deliver my invention jobs fast in succession, I get mostly successes or mostly failures regardless of actual chance to invent" complaint ?


Of course, the tricky thing is that the weapons fire data we have can only go so far in doing that. If we find a fault in the weapons fire data, that's great, we can get that, and other processes that rely on the same broken stage, to be looked at.

But if we find nothing wrong with the weapons fire data, that doesn't eliminate there being something wrong with the invention results, because they don't share the same calculation path completely.

If we have to go back to the invention data, what would be useful in testing the particular hypothesis you state there, is data which records the delivery batches, as well as the individual results, though it would really need to have all the batches the same size to be useful for what I have in mind.

The number of successes per batch of a given size will itself be a random variable, that we can define from the probability of success, assuming that they were completely independent events. We can then compare this to the empirical results. If there is a time-related bias leading to jobs being delivered in the same batch having a higher probability of all coming out the same, then this should show up in the two extremes having a much higher frequency of occurance than the probability distribution would suggest.

Akita T
Caldari Navy Volunteer Task Force
Posted - 2008.02.24 18:59:00 - [229]
 

It WOULD help a lot if somebody from CCP would tell us how they compute success/failure based on the numbers spit out by the RNG... if it's a simple "above/below chance cutoff" (chance cutoff being a float from 0 to 1 too), or if they do something fancier, like select certain bits from the float the RNG spits out and construct a separate method of determining failure/success.
If it's the first, then the fire-data might be just as good as having a whole lot of invention runs... if not, we actually have to do some mass invention runs (which would certainly take a lot longer to get).

Semkhet
Dark Tornado
Ethereal Dawn
Posted - 2008.02.25 01:06:00 - [230]
 

Edited by: Semkhet on 25/02/2008 07:28:53

Took a quick look.

1) The sequence is composed by floats, where all numbers between "9.7" and "28.9" with precision of 1 decimal are represented. Adding the wrecking value, we get a symbolic alphabet of 194 tokens. (It's always handy to transform things into a symbolic alphabet so you can map the original values to unsigned bytes or words which are much easier to process in multi-pass analysis).

2) Count & map the number of occurrences for each symbol. 2 singularities are identified. The first number and the last one seem to be artificially forced negatively. They are the only ones to appear respectively only 166 and 137 times, while the average per number is 337. Wrecking shots peak at 650 (1%), almost the double than any other "normal" occurrence, as mentioned earlier by Akita.

3) From the 194 occurrences (1 per symbol obviously), we find that the total number of different occurrences is only 62, what is strange for an alphabet of 194 symbols. Each symbol should have a 1/194 chance to appear at each of the 65524 cycles. So we could expect the distribution to be somewhat less uniform. Some symbols could appear 5 times, other 1000. Instead everything seems nicely clustered above or under the average and despite having the theoretical possibility to see a different & unique number of occurrences for each symbol, we get a distribution using only 62 of the 194 degrees of freedom...

4) In good PRNG's, you should not be able to correlate the value emitted with its occurrence. And even less correlate adjacent values part of a progression with occurrences. But some PRNG's feed back by recycling grossly part of the last result in the generation of the next number. Let's check out. Sorting the list in ascending order, 1 column with the emitted values, one column with their related occurrence.

5) 4 singularities are identified. Values 21.7 & 21.8 have the same occurrence of 326. Values 22.5 & 22.6 have the same occurrence of 321, Values 26.9 & 27 have the same occurrence of 341. Values 28.6 & 28.7 have the same occurrence of 331.

6) Let's check the occurrences. Computing how many occurrences of the same type occur among the 194 occurrences.

326 appears 5 times.
321 appears 4 times.
341 appear 3 times.
331 Appears 6 times.

So, what is the probability that 21.7 & 21.8 are mapped to 2 out of 5 occurrences of the same kind, using 65524 generations of an entity which can take 194 different values, and subdivided in 62 unique types of occurrences (for a total of 194 occurrences) ?

What is the probability that 22.5 & 22.6 are mapped to 2 out of 4 occurrences among 194 ?

What is the probability that 26.9 & 27 are mapped to 2 out of 3 occurrences among 194 ?

What is the probability that 28.6 & 28.7 are mapped to 2 out of 6 occurrences among 194 ?

Besides, if you compute the theoretical average probability for a symbol to appear 1/194 = 0.5181 %, you take the ascending sorted list and compute the positive or negative deviation for each symbol in relation to the real occurrence, and then simply round every positive value to 1 and every negative to 0, you will see that out of the 194 results, only 51 are isolated results, all the rest are clustered in sequences of 2 or more of the same kind, in one case going up to 6 successive "1". Streaks.

It is possible that the process CCP uses to extract and present the data renders it garbled, interleaving numbers issued from different generations, but this only blur correlations with timelines. However, the fact that the distribution of frequency only uses 32% of the maximum available scale could point to the extraction process simply rejecting any value not fitting within the boundaries of the defined scope.

Bottom line, it's symptomatic from CCP's PRNG quality that despite being so primitive, the simple checks I used are able to identify & expose incoherences which would be extremely difficult to find in TRN sequences. Took me more time to write this post than perform the checks..

Akita T
Caldari Navy Volunteer Task Force
Posted - 2008.02.25 09:02:00 - [231]
 

Edited by: Akita T on 25/02/2008 09:19:08


Ok, just for the record, I have "normalized" the entire shooting data log.
EVE_RNG_normalized_v3.zip

Contents is EVE_RNG_normalized_v3.xls
Inside, there's 80486 values, on 5 columns (4x20k, 5th is the remaining 486), formatted as PERCENTAGES, displaying with 2 decimal points even if only the first is somewhat relevant to begin with.
You can easily convert that to display as 0.xxx if you want to, the actual "normalized" value is still in there.

Wrecking shots were "normalized" as 0% (but actually cover UNKNOWN values between 0% and aprox 1%).
Everything else was normalized as a percentage of ACTUAL nominal damage, so you might see some funny numbers.

This should make it a bit easier to work with the data, and more of the data, not just the initial 65k "processed" values.
EDIT : Missed one of "Rasta we"'s shots in the previous version, removed it and reordered remaining data.

Zoi Opia
Opia Investments
Posted - 2008.02.25 09:33:00 - [232]
 

Originally by: Semkhet
Besides, if you compute the theoretical average probability for a symbol to appear 1/194 = 0.5181 %, you take the ascending sorted list and compute the positive or negative deviation for each symbol in relation to the real occurrence, and then simply round every positive value to 1 and every negative to 0, you will see that out of the 194 results, only 51 are isolated results, all the rest are clustered in sequences of 2 or more of the same kind, in one case going up to 6 successive "1". Streaks.


Streaks? Doesn't your analysis only show that the PRNG CCP is using is actually more "fair and balanced" than your ideal of a true RNG would be? Which is exactly what one would expect from a PNRG-algorithm designed to give out an uniform distribution. I don't (yet) see any indication that the occurrence of number A in the generated sequence would in any way influence the occurrence of number B.

Akita T
Caldari Navy Volunteer Task Force
Posted - 2008.02.25 09:51:00 - [233]
 

Edited by: Akita T on 25/02/2008 09:58:58

Ok, so, I took the data from above, ignored streaks that might occur between columns (so at the 20k datapoint cutoff), and calculated the "longest streak" found for several probabilities...

PROBABILITY : 50% success
Longest expected succes streak = lg(80 000) = 16.2877124
Longest success streak : 16
Longest failure streak : 14

PROBABILITY : 36.78% success (aprox 1/e)
Longest expected succes streak = ln(80 000) = 11.2897819
Longest success streak : 10
Longest failure streak : 28

PROBABILITY : 10% success
Longest expected succes streak = log(80 000) = 4.90308999
Longest success streak : 4
Longest failure streak : 90
___

Meh, looks pretty normal, to be honest.
In other words, IF EVE uses the simple "cutoff" method, AND always pulls a new random number for invention success/failure, there should be no "unnatural" streaks occuring... at least, not on SiSi.
The only remaining thing to be checked then is wether the code that pulls the result is as it should be (but I suppose it's been checked)... and us, do more actual invention tests, not just shooting data.

Mistral Sud
Minmatar
Black Box Corp.
Posted - 2008.02.25 10:22:00 - [234]
 

yeah for me it is borked as well!:)

well after getting a drone mission 3 times in a row as well, i thought about this!

why not making a table?
instead of rolling the dice every time u accept a mission, roll the dice just ONCE for a fixed number of missions!
lets say 100 missions and then fill the table with 12% mission A, 14% mission B, etc etc.
that would lead to a 100 % probability of having every mission in said fixed size!
it would be easy to implement an algorithm to remove the doubles, by just checking the table, and it would be much more fairer in my opinion!

it would be nice to see this too for the probability of getting faction loot in extravaganzas as well:P
that always annoyed me when i look at the stuff my mates takes out of that missions compared with my luck:P



Apocryphai
Caldari
State Protectorate
Posted - 2008.02.25 10:26:00 - [235]
 

Phew, well that was a long read. Read the whole thread and there's some interesting points being raised but so far no hard data to indicate the OP's thesis is correct.

None of the (limited) analysis anyone has done of this 80k dataset shows any convincing sign of streaking that significantly deviates from the "norm" (which I think has been poorly defined here really).

I won't rehash the main problems with this data that Matthew in particular has pointed out, but here's another take on the whole matter:

1) Humans are pattern-recognition devices. It's how our brains work. We are highly attuned to patterns, it's the basis of our visual system, our memories and our language skills. Pattern recognition in one form or another is central to artistic methods, mathematical methods and scientific methods.

2) As a result of these pattern recognition abilities that are so crucial to our functioning we are also extremely good at finding patterns even where none exist. One simple demonstration of this is the Rorschach inkblot test (look it up if you're not familiar with it). we have an in-built predisposition to see patterns.

3) Thus, in any random or pseudo-random event human beings tend to see patterns and to spot things that "look abnormal" to us.

4) Given the limitations of our ability to sample from EVE's RNG and given the vast number of expressions of the RNG that EVE players see every day when conduction normal business in EVE, Occams Razor would suggst that the most likely explanation for perceived streakiness of the RNG is, in fact, our human over-propensity to see patterns where none really exist.

I don't think that the problem described in this thread is actually a mathematical or RNG-problem at all, I think it's a psychological issue caused by human perceptions.

Semkhet
Dark Tornado
Ethereal Dawn
Posted - 2008.02.25 10:56:00 - [236]
 

Edited by: Semkhet on 25/02/2008 11:30:53
Originally by: Zoi Opia
Streaks? Doesn't your analysis only show that the PRNG CCP is using is actually more "fair and balanced" than your ideal of a true RNG would be? Which is exactly what one would expect from a PNRG-algorithm designed to give out an uniform distribution. I don't (yet) see any indication that the occurrence of number A in the generated sequence would in any way influence the occurrence of number B.


In a true random sequence, taking the principle of transforming all the possible values mapping the lowest to 1 and the highest to 193 (leaving wrecking shots aside), each value has 1 in 193 chances (or 0.5181 %) to come out at each cycle. That it CAN come out does NOT mean that it WILL necessarily come out.

Hence each of the 194 symbols could have a different frequency since it could appear any number of times in the sequence. Instead, only 62 different frequencies show up. Now if you take a superficial look at these frequencies, arranging them related to the ascending order of the values, we find the artifacts mentioned in my last post.

If the value in itself holds no relation whatsoever to its frequency (what should usually happen in a TRN sequence), and given that we have 62 different frequencies among 194 (meaning that each value has 1 in 62 chances to fall in a given frequency), can you explain:

- What is the probability that the frequency "326", from its 5 occurrences among the 194 positions, two are assigned to the values 21.7 & 21.8 ?

- What is the probability that the frequency "321", from its 4 occurrences among the 194 positions, two are assigned to the values 22.5 & 22.6 ?

- What is the probability that the frequency "341", from its 3 occurrences among the 194 positions, two are assigned to the values 26.9 & 27 ?

- What is the probability that the frequency "331", from its 6 occurrences among the 194 positions, two are assigned to the values 28.6 & 28.7 ?

What is the combinatorial probability of these 4 patterns ?
There are more artifacts:

- Frequency "312" only occurs 2 times among the 194 positions. Related to values differing only by 1 decimal: 15.7 & 15.9. Coincidence ?

- Frequency "360" only occurs 3 times among the 194 positions. Two of the occurrences are related to values differing only by 1 decimal: 27.8 & 28. Coincidence ?

- Frequency "349" only occurs 6 times among the 194 positions. Two of the occurrences are related to values differing only by 1 decimal: 25.3 & 25.5. Coincidence ?

What is the combinatorial probability of these three patterns ?

Other fishy patterns:

14.3 343
14.4 342
14.5 344

22.2 335
22.3 334
22.4 336

What is the combinatorial probability that two groups of frequencies mapped to a value which should not bear any relation, suddenly show up the same pattern if ordered by their related ascending values, taking into account that taken separately, each of these frequencies has a probability of 1 in 62 ?

I neither have the time nor the interest to dig any further, but my intuition is that the entropy of the output of the PRNG is further degraded either by an aggressive promotion method to fill up the intended final scale, or any value which does not correspond to the intended final scale is simply filtered out. A good PRNG specifically programmed to output directly values within the intended boundaries would make better use of the 194 possible frequencies rather than the 62 we are observing.

As result the output is degraded by brute force and artificially clustered just above the average frequency of 337 (65524/194), that's why if you compute the deviation for each frequency vs 337, and assign a threshold to 337 to compute a 0 for negative and a 1 for positive deviations, you get a graph showing the "oscillation" of the output, which unfortunately displays a surplus of consecutive positive deviations WHEN ordering the deviations by the ascending values they are related with. Which means BIAS.

Semkhet
Dark Tornado
Ethereal Dawn
Posted - 2008.02.25 11:24:00 - [237]
 

Originally by: Apocryphai
Phew, well that was a long read. Read the whole thread and there's some interesting points being raised but so far no hard data to indicate the OP's thesis is correct.

None of the (limited) analysis anyone has done of this 80k dataset shows any convincing sign of streaking that significantly deviates from the "norm" (which I think has been poorly defined here really).

I won't rehash the main problems with this data that Matthew in particular has pointed out, but here's another take on the whole matter:

1) Humans are pattern-recognition devices. It's how our brains work. We are highly attuned to patterns, it's the basis of our visual system, our memories and our language skills. Pattern recognition in one form or another is central to artistic methods, mathematical methods and scientific methods.

2) As a result of these pattern recognition abilities that are so crucial to our functioning we are also extremely good at finding patterns even where none exist. One simple demonstration of this is the Rorschach inkblot test (look it up if you're not familiar with it). we have an in-built predisposition to see patterns.

3) Thus, in any random or pseudo-random event human beings tend to see patterns and to spot things that "look abnormal" to us.

4) Given the limitations of our ability to sample from EVE's RNG and given the vast number of expressions of the RNG that EVE players see every day when conduction normal business in EVE, Occams Razor would suggst that the most likely explanation for perceived streakiness of the RNG is, in fact, our human over-propensity to see patterns where none really exist.

I don't think that the problem described in this thread is actually a mathematical or RNG-problem at all, I think it's a psychological issue caused by human perceptions.


Most humans are extremely good at recognizing patterns, but often only in a very limited set of fields: those which are essential for their daily life interaction. Among them and as most developed: facial recognition and character recognition.

TASHT HYW OTSM EPPOLE ARUDENTSDN EHTSE WDORS.

Words are processed as distinct entities, rather than absolute sequences of letters. But this comes with routine. The same routine which makes that chess players, rather than being uber-intelligent beings, are above all individuals which have managed to integrate chess-board pattern recognition in the same area which processes facial recognition.

Some people can be classified as "visual mathematicians" and are good at discerning hidden patterns. Does not mean that what they are seeing is indeed genuine.

However, deterministic vs true random systems show rather opposite properties. And we are still talking about cpu-generated stuff here.

As stated before, there's a limit to what statistical instruments might distinguish. Absence of proof is not proof of absence Wink

ry ry
Heroes.
Merciless.
Posted - 2008.02.25 13:09:00 - [238]
 

err, the RNG isn't random, because the results aren't what you expected?

* mumbles something about drunken men, statistics and lampposts *

Zhecao Vai
Ultrapolite Socialites
GoonSwarm
Posted - 2008.02.25 13:42:00 - [239]
 

I saw your turret damage formula:

Quote:
x = random value between 0 and 1
z is the "To-Hit Chance", a value between 0 (0%) and 1 (100%)
z = ((1/2)^((((Transv/(Range*Tracking))*(Sig_Res/Sig_Rad))^2) +((max(0,Range-Optimal))/Falloff)^2))
If {x<=0.01} Then {quality = 3} else {quality = x + 0.49}
If {x<z} Then {damage = quality * base damage}


What makes you think X is a "random value between 0 and 1?" When you test it out, you see streaks and funny frequencies for results based on X.

So, isn't the conclusion just that there are other factors biasing X? It looks from Semkhet's post that something is getting rounded off somewhere, perhaps, or there's some kind of clustering around different desired damage multipliers.

As far as I can tell, your theory doesn't match the data, and you're complaining that it's a bug in the data.

Counterparty
Posted - 2008.02.25 13:58:00 - [240]
 

Edited by: Counterparty on 25/02/2008 14:01:35
Originally by: Semkhet

Other fishy patterns


Yep totally borked.


Here some frequency numbers from the full data set. 80486 samples over range of 194 possible damage values spread from 9.7 to 28.9 plus 58.2. The frequency oddballs are the high, low and wrecking value. Dmg 9.7 has freq 205, dmg 28.9 has freq 166 and dmg 58.2 has freq 805.

Removing the oddballs leaves 79310 samples over range of 191 values spread from dmg 9.8 to dmg 28.8. There are 66 frequency values ranging from highest, freq is 470 (dmg 18.7), to lowest, freq 361 (dmg 28.2).

dmg 21.7 and 21.8 still have same freqency (410). That is the only occurrence of sequential values at the same frequency in dataset.

Two occurrences seperate by two occur: dmg 11.0 and 11.2 have freq 417. dmg 14.5 and 14.7 have freq 429.



Here is the full frequency of frequencies table
freq of freq | freq | dmg values at that frequency
7 424 10.2 10.9 12.3 14.3 20.9 23.5 24.7
7 420 10.0 10.6 12.8 13.5 14.1 25.3 27.5
7 419 11.9 14.9 21.4 22.9 24.0 24.4 27.7
7 417 11.0 11.2 14.0 16.3 20.1 24.9 26.5
6 437 10.4 13.0 16.0 17.3 19.7 27.8
6 410 17.0 21.1 21.7 21.8 22.5 26.1
6 407 14.4 15.0 15.6 18.2 22.2 26.2
6 393 11.1 12.1 13.8 23.6 24.5 25.9
5 416 12.7 15.5 16.4 20.5 26.3
5 415 10.7 16.2 23.2 28.1 28.8
5 405 16.8 24.3 27.3 27.6 28.7
5 404 13.2 14.2 19.8 24.8 26.8
4 432 10.8 15.4 25.5 28.5
4 431 18.5 24.2 24.6 28.0
4 412 11.6 22.6 25.6 27.1
4 411 11.7 20.6 22.3 27.0
4 402 10.5 11.4 13.3 28.6
4 396 9.8 19.1 20.3 26.7
3 447 17.1 17.9 22.0
3 438 13.7 17.7 23.0
3 430 18.3 21.3 23.8
3 429 14.5 14.7 16.1
3 426 14.8 15.8 21.9
3 425 16.5 22.4 24.1
3 421 13.9 18.6 22.8
3 418 12.6 16.7 25.4
3 406 12.9 17.5 26.0
3 403 13.1 13.4 22.1
3 401 18.9 20.7 25.8
3 400 10.3 18.1 26.6
3 394 12.4 21.6 27.4
3 386 11.3 19.4 23.7
3 384 12.5 21.2 28.4
2 450 12.0 23.1
2 449 21.0 21.5
2 441 11.5 17.4
2 440 15.1 15.3
2 436 17.6 19.3
2 435 19.0 23.3
2 427 14.6 16.9
2 422 13.6 17.2
2 414 23.4 25.0
2 413 26.4 26.9
2 408 12.2 19.6
2 399 19.2 20.2
2 397 18.0 19.5
2 395 15.9 16.6
2 390 20.4 20.8
2 387 15.7 18.8
2 385 11.8 27.9
1 470 18.7
1 463 27.2
1 459 25.7
1 452 9.9
1 448 10.1
1 445 17.8
1 442 23.9
1 439 18.4
1 434 22.7
1 398 28.3
1 392 25.2
1 391 15.2
1 388 19.9
1 382 20.0
1 368 25.1
1 361 28.2


Pages: 1 2 3 4 5 6 7 [8] 9

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only