Are all significant P values created equal? The analogy between diagnostic tests and clinical research

W. S. Browner and T. B. Newman

Just as diagnostic tests are most helpful in light of the clinical presentation, statistical tests are most useful in the context of scientific knowledge. Knowing the specificity and sensitivity of a diagnostic test is necessary, but insufficient: the clinician must also estimate the prior probability of the disease. In the same way, knowing the P value and power, or the confidence interval, for the results of a research study is necessary but insufficient: the reader must estimate the prior probability that the research hypothesis is true. Just as a positive diagnostic test does not mean that a patient has the disease, especially if the clinical picture suggests otherwise, a significant P value does not mean that a research hypothesis is correct, especially if it is inconsistent with current knowledge. Powerful studies are like sensitive tests in that they can be especially useful when the results are negative. Very low P values are like very specific tests; both result in few false-positive results due to chance. This Bayesian approach can clarify much of the confusion surrounding the use and interpretation of statistical tests.

THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES

Genetics of Osteoporosis: From Population Association to Individualized Prognosis of Fracture
Nguyen
IBMS BoneKEy 2008;5:212-221.
FULL TEXT  

Contradictions in Highly Cited Clinical Research
West
JAMA 2005;294:2695-2695.
FULL TEXT  

Prior convictions: bayesian approaches to the analysis and interpretation of clinical megatrials
Diamond and Kaul
J Am Coll Cardiol 2004;43:1929-1939.
ABSTRACT | FULL TEXT  

In STI interventions, size matters
Crosby and Rothenberg
Sex. Transm. Infect. 2004;80:82-85.
FULL TEXT  

Assessing the Probability That a Positive Report is False: An Approach for Molecular Epidemiology Studies
Wacholder et al.
JNCI J Natl Cancer Inst 2004;96:434-442.
ABSTRACT | FULL TEXT  

Sample Size Estimation: How Many Individuals Should Be Studied?
Eng
Radiology 2003;227:309-313.
ABSTRACT | FULL TEXT  

Hypothesis Testing I: Proportions
Zou et al.
Radiology 2003;226:609-613.
ABSTRACT | FULL TEXT  

Do statins confer early benefit after acute coronary syndromes? The results from FLORIDA
Varosy and Waters
Eur Heart J 2002;23:1893-1896.
 

Risk of Specific Birth Defects in Relation to Chlorination and the Amount of Natural Organic Matter in the Water Supply
Hwang et al.
Am J Epidemiol 2002;156:374-382.
ABSTRACT | FULL TEXT  

Counterpoint: Bias from Population Stratification Is Not a Major Threat to the Validity of Conclusions from Epidemiological Studies of Common Polymorphisms and Cancer
Wacholder et al.
Cancer Epidemiol. Biomarkers Prev. 2002;11:513-520.
FULL TEXT  

Clinical trial end points: On the road to nowhere?
Holloway and Dick
Neurology 2002;58:679-686.
ABSTRACT | FULL TEXT  

Decision support and safety of clinical environments
Morris
Qual Saf Health Care 2002;11:69-75.
ABSTRACT | FULL TEXT  

Human Carcinogenesis, the New Genetics, and the Practicing Pathologist
Murphy
INT J SURG PATHOL 2002;10:1-5.
ABSTRACT  

Sifting the evidence--what's wrong with significance tests?
Sterne et al.
ptjournal 2001;81:1464-1469.
FULL TEXT  

Sifting the evidence{---}what's wrong with significance tests? Another comment on the role of statistical methods
Sterne et al.
BMJ 2001;322:226-231.
FULL TEXT  

Toward Evidence-Based Medical Statistics. 1: The P Value Fallacy
Goodman
ANN INTERN MED 1999;130:995-1004.
ABSTRACT | FULL TEXT  

Talking about Treatment: The Language of Populations and the Language of Individuals
Steiner
ANN INTERN MED 1999;130:618-622.
FULL TEXT  

Correspondence
Baer et al.
J Geriatr Psychiatry Neurol 1991;4:122-123.
 

Interpretation of the Correlation Coefficient: A Basic Review
Taylor
Journal of Diagnostic Medical Sonography 1990;6:35-39.
ABSTRACT  

Selected Topics in Statistical Analysis of Clinical Research
Baer and Ahern
J Geriatr Psychiatry Neurol 1989;2:34-40.
ABSTRACT