Posts Tagged ‘statistics’

BEYES AND BENFORD

2010-10-21

I was reminded today of a couple of ideas that had (and still have) a profound impact on my thinking, and my approach to life.

The two concepts are from the field of numbers — statistics, odds, likelihood, chance.  It’s nothing to be afraid of; it’s not mathematics — nor even arithmetic.

The first is Benford’s Law.

I love this.  It is also known as “The First Digit Law”. In lists of numbers from almost all real-life sources, the first digit is 1 almost a third of the time, and larger digits occur as the leading/ first digit with ever  lower frequency, until the point where nine occurs less than once time in twenty as the first digit.

Benford’s Law applies well to a wide variety of data sets, like electricity bills, street addresses, stocks and shares, population and death rates, lengths of rivers, physical and mathematical constants, and processes described by power laws (which are very common in nature).

Now, this is pretty amazing.  Well, it was to me anyway; it is somewhat counter-intuitive after all.  Yet it works. The first digit is likely to be the number “1” in over 30% of the cases, not because of mathematical theory, or classical ideas, but as a result of real-world evidence.  The real world is imperfect, and far from the purity of academia.

The second idea is Bayes’s Theorem.

What Tom Bayes did was link two things when it comes to probability.  This is somewhat important in that people are either innocent or guilty, something works or doesn’t, and so forth.  Bayes is often important for understanding gambles and gamblers.

The key idea is that the probability of an event A given an event B. And this is exactly the way the mind works —  in a coin toss (for example), there’s a 50 % chance of a head or tail result according to classical statistics, but the coin is not perfect and the bias becomes more evident with each toss, so the results are actually linked, and the odds change (and the bias becomes more obvious) with each result, until the outcome is predictable; nothing is pure and unbiased in the classical mathematical sense.

The notion that each piece of evidence in a court of law, has an effect on the jury’s verdict, that guilt or innocence depends on each and every witnesses’ testimony in turn, is amazing.  Human nature cannot wait until the end to decide on guilt, it works like Bayes’s, becoming more and more guilty or increasingly guilty depending on the evidence as it arrives.  So classical statistics suggests that each coin has a fifty per cent chance of turning up, while Bayes — like us — reckons it’s been tails too long, and it’s time for a heads!

§