Monday, September 03, 2007

Defending oneself against misuse of statistics

A little while ago, I came across a review of the five questions that Darryl Huff ("How to lie with statistics") suggested one should use to defend oneself "against statistics", via Mind Hacks. In addition to reviewing the five points, Tom at idiolect provides a commentary which provides some interesting and relevant additional points. The five questions I quote below, you can find the commentary here:
1. Who says so? Can we suspect deliberate or unconscious bias in the originator of the statistic? Huff recommends looking for an “O.K Name” - e.g. a university - as some slim promise of reliability. Second to this he recommends being careful to distinguish the originator of the ‘data’ from the originator of the conclusion or intepretation.
2. How Does He Know? Is the sample biased? representative? large enough?
3. What’s Missing? Statistics given without a measure of realiability are ‘not to be taken very seriously’. What is the relevant base rates / appropriate comparison figure? Do averages disguide important variations?
4. Did somebody change the subject? E.g. More reported cases are not the same as more cases, what people say they do (or will do) is not the same as what they actually do (or will do), association (correlation) is not causation.
5. Does it make sense? Is the figure spuriously accuracy? Convert percentages to real numbers and convert real numbers to percentages, compare both with your intuitions.

For me, one of the most interesting observations (and ensuing comment) made is concerned with credibility of the author who uses statistics (point one above). But I'm not going to recreate the discussion here (especially given some of the posts in the current Encephalon), read the post at idiolect.

No comments: