The problem with p-values is how they’re used ∗

• Main Author: Andrew Gelman
• Other Authors: The Pennsylvania State University CiteSeerX Archives
• Format: Text
• Language: English
• Published: 2013
• Subjects: Greenland
• Online Access: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.300.9053; http://www.stat.columbia.edu/~gelman/research/published/murtaugh.pdf

I agree with Murtaugh (and also with Greenland and Poole, 2013, who make similar points from a Bayesian perspective) that with simple inference for linear models, p-values are mathematically equivalent to confidence intervals and other data reductions, there should be no strong reason to prefer one method to another. In that sense, my problem is not with p-values but in how they are used and interpreted. Based on my own readings and experiences (not in ecology but in a range of social and environmental sciences), I feel that p-values and hypothesis testing have led to much scientific confusion, but by researchers treating non-significant results as zero and significant results as real. I have, on occasion, successfully used p-values and hypothesis testing in my own work, and in other settings I have reported p-values (or, equivalently, confidence intervals) in ways that I believe have done no harm, as a way to convey uncertainty about an estimate (Gelman, 2013). In many other cases, however, I believe that null hypothesis testing has led to the publication of serious mistakes, perhaps notoriously in the paper by Bem (2011), who claimed evidence for extrasensory perception (ESP) based on a series of statistically significant results. But, as researchers in medicine and psychology such as Ioannidis (2005), Simmons, Nelson, and Simonsohn (2011)