Skip to main content

Charlie Gibbons's Library tagged science →  View Popular

26 Apr 09

PrawfsBlawg: More Problems With Hypothesis Testing (From a More Technical Perspective)

  • The confidence level is set in advance. The resulting p-value is a random variable. To adjust the level in response to the p-value is an improper ex post move. So if the pre-set level of significance is 95% and the resulting p-value is 0.0001, the proper response is to say "The results are significant at the 95% level (p = 0.0001)." To readjust the claim to "The results are significant at the 99% level (since) p = 0.0001" is simply incorrect.

    Thus: there should only be one star per table, at whatever level the analyst sets in advance.

  • And so it is disappointing that rarely if ever do we see a paper wrestle with the proper level of significance, such as by asking whether this is a case where a false negative is better or worse than a false positive. After all, it is not always clear that we are best served by the conservatism of the 95% confidence interval. A false positive may be worse than a false negative in criminal law ("better 10 guilty men go free..."), but a false negative may be worse in some medical situations, such as whether a particular pill works.

PrawfsBlawg: A Misguided Philosophy of Science

  • Susan Haack provides a nice way to think about how we should approach knowledge production in the social sciences. She equates knowledge to a crossword puzzle. A single word, all alone in the grid, is not well-supported (especially if, as most scientific questions likely are, it is a Saturday NY Times). If 35-down is a seven-letter word for "Oceanographers' references," SEAMAPS fits, but we might not be all that confident it is right. But then we look at 46-across, which starts on the M, and we think that MATED could be the right answer for "Like shoes and socks;" we become a bit more sure that SEAMAPS is right. And our faith in MATED grows when we realize that 47-down, which starts on the D, is probably "DOSE," since the clue is "Recommended intake." So DOSE provides warrant for MATED, which in turn warrants SEAMAPS. Once the whole puzzle is filled, we're pretty sure we're right. We could be completely wrong--I've certainly erased entire quadrants of a crossword puzzle before--but the odds are low.
  • I attended a conference several years ago in which one of the speakers said that if we added up every paper that said "factor x is responsible for y% of the crime drop in the 1990s," we would see that we've explained about 250% of the crime decline. Everyone laughed but thought little of it. But this is a huge problem: the words in the crossword puzzle are not crossing properly. Something, somewhere, is wrong, but by taking a corpuscular view of empirical evidence, we miss it.
  • 1 more annotations...
1 - 12 of 12
Showing 20▼ items per page

Selected Tags

Related Tags

Sponsored Links

Ads by Google

Diigo is about better ways to research, share and collaborate on information. Learn more »

Join Diigo