This Article  • Full text  • PDF  • Correction  • Send to a friend  • Save in My Folder  • Save to citation manager  • Permissions  Citing Articles  • Citation map  • Citing articles on HighWire  • Citing articles on Web of Science (537)  • Contact me when this article is cited  Related Content  • Related articles  • Similar articles in JAMA  Topic Collections  • Quality of Care  • Quality of Care, Other  • Statistics and Research Methods  • Alert me on articles by topic  Social Bookmarking   What's this?

Jeroen G. Lijmer, MD; Ben Willem Mol, MD, PhD; Siem Heisterkamp, PhD; Gouke J. Bonsel, MD, PhD; Martin H. Prins, MD, PhD; Jan H. P. van der Meulen, MD, PhD; Patrick M. M. Bossuyt, PhD

JAMA. 1999;282:1061-1066.

Context  The literature contains a large number of potential biases in the evaluation of diagnostic tests. Strict application of appropriate methodological criteria would invalidate the clinical application of most study results.

Objective  To empirically determine the quantitative effect of study design shortcomings on estimates of diagnostic accuracy.

Design and Setting  Observational study of the methodological features of 184 original studies evaluating 218 diagnostic tests. Meta-analyses on diagnostic tests were identified through a systematic search of the literature using MEDLINE, EMBASE, and DARE databases and the Cochrane Library (1996-1997). Associations between study characteristics and estimates of diagnostic accuracy were evaluated with a regression model.

Main Outcome Measures  Relative diagnostic odds ratio (RDOR), which compared the diagnostic odds ratios of studies of a given test that lacked a particular methodological feature with those without the corresponding shortcomings in design.

Results  Fifteen (6.8%) of 218 evaluations met all 8 criteria; 64 (30%) met 6 or more. Studies evaluating tests in a diseased population and a separate control group overestimated the diagnostic performance compared with studies that used a clinical population (RDOR, 3.0; 95% confidence interval [CI], 2.0-4.5). Studies in which different reference tests were used for positive and negative results of the test under study overestimated the diagnostic performance compared with studies using a single reference test for all patients (RDOR, 2.2; 95% CI, 1.5-3.3). Diagnostic performance was also overestimated when the reference test was interpreted with knowledge of the test result (RDOR, 1.3; 95% CI, 1.0-1.9), when no criteria for the test were described (RDOR, 1.7; 95% CI, 1.1-2.5), and when no description of the population under study was provided (RDOR, 1.4; 95% CI, 1.1-1.7).

Conclusion  These data provide empirical evidence that diagnostic studies with methodological shortcomings may overestimate the accuracy of a diagnostic test, particularly those including nonrepresentative patients or applying different reference standards.

Author Affiliations: Department of Clinical Epidemiology and Biostatistics, Academic Medical Center, University of Amsterdam, Amsterdam, the Netherlands.

CiteULike    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter     What's this?

RELATED ARTICLES

The Hazards of Scoring the Quality of Clinical Trials for Meta-analysis
Peter Jüni, Anne Witschi, Ralph Bloch, and Matthias Egger
JAMA. 1999;282(11):1054-1060.
ABSTRACT | FULL TEXT  

Measuring the Quality of Trials: The Quality of Quality Scales
Jesse A. Berlin and Drummond Rennie
JAMA. 1999;282(11):1083-1085.
EXTRACT | FULL TEXT  

September 15, 1999
JAMA. 1999;282(11):1099-1100.
EXTRACT | FULL TEXT  

THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES

Integration of evidence from multiple meta-analyses: a primer on umbrella reviews, treatment networks and multiple treatments meta-analyses
Ioannidis
CMAJ 2009;181:488-493.
FULL TEXT  

Matrix metalloproteinase levels in acute aortic dissection, acute pancreatitis and other abdominal pain
Wen et al.
Emerg. Med. J. 2009;26:715-718.
ABSTRACT | FULL TEXT  

Proposals for a Phased Evaluation of Medical Tests
Lijmer et al.
Med Decis Making 2009;29:E13-E21.
ABSTRACT  

Decision-Analytic Modeling to Evaluate Benefits and Harms of Medical Tests: Uses and Limitations
Trikalinos et al.
Med Decis Making 2009;29:E22-E29.
ABSTRACT  

Hydrosonography of the Gastrointestinal Tract
Maconi et al.
Am. J. Roentgenol. 2009;193:700-708.
ABSTRACT | FULL TEXT  

Excluding venous thromboembolism using point of care D-dimer tests in outpatients: a diagnostic meta-analysis
Geersing et al.
BMJ 2009;339:b2990-b2990.
ABSTRACT | FULL TEXT  

Declines in Erythrocyte Sedimentation Rates in Patients with Rheumatoid Arthritis Over the Second Half of the 20th Century
ABELSON et al.
The Journal of Rheumatology 2009;36:1596-1599.
ABSTRACT | FULL TEXT  

Erythrocyte Sedimentation Rate, C-Reactive Protein, or Rheumatoid Factor Are Normal at Presentation in 35%-45% of Patients with Rheumatoid Arthritis Seen Between 1980 and 2004: Analyses from Finland and the United States
SOKKA and PINCUS
The Journal of Rheumatology 2009;36:1387-1390.
ABSTRACT | FULL TEXT  

Value of history-taking in syncope patients: in whom to suspect long QT syndrome?
Colman et al.
Europace 2009;11:937-943.
ABSTRACT | FULL TEXT  

Bivariate meta-analysis of sensitivity and specificity of radiographers' plain radiograph reporting in clinical practice
BREALEY et al.
Br. J. Radiol. 2009;82:600-604.
ABSTRACT | FULL TEXT  

Yield of Diagnostic Tests for Celiac Disease in Individuals With Symptoms Suggestive of Irritable Bowel Syndrome: Systematic Review and Meta-analysis
Ford et al.
Arch Intern Med 2009;169:651-658.
ABSTRACT | FULL TEXT  

Overinterpretation of Clinical Applicability in Molecular Diagnostic Research
Lumbreras et al.
Clin. Chem. 2009;55:786-794.
ABSTRACT | FULL TEXT  

Does Thromboelastography Predict Postoperative Thromboembolic Events? A Systematic Review of the Literature
Dai et al.
Anesth. Analg. 2009;108:734-742.
ABSTRACT | FULL TEXT  

Positive Predictive Value of Specific Mammographic Findings according to Reader and Patient Variables
Venkatesan et al.
Radiology 2009;250:648-657.
ABSTRACT | FULL TEXT  

Achieving Quality in Cardiovascular Imaging II: proceedings from the Second American College of Cardiology -- Duke University Medical Center Think Tank on Quality in Cardiovascular Imaging.
Douglas et al.
J Am Coll Cardiol Img 2009;2:231-240.
ABSTRACT | FULL TEXT  

Performance of Purified Antigens for Serodiagnosis of Pulmonary Tuberculosis: a Meta-Analysis
Steingart et al.
CVI 2009;16:260-276.
ABSTRACT | FULL TEXT  

Diagnostic Performance of Computed Tomography Angiography in Peripheral Arterial Disease: A Systematic Review and Meta-analysis
Met et al.
JAMA 2009;301:415-424.
ABSTRACT | FULL TEXT  

Systematic Reviews of Diagnostic Test Accuracy
Leeflang et al.
ANN INTERN MED 2008;149:889-897.
ABSTRACT | FULL TEXT  

When Should a New Test Become the Current Reference Standard?
Glasziou et al.
ANN INTERN MED 2008;149:816-821.
ABSTRACT | FULL TEXT  

GRADE: assessing the quality of evidence for diagnostic recommendations
Schunemann et al.
Evid. Based Med. 2008;13:162-163.
FULL TEXT  

Effect of deep vein thrombosis on ischaemia-modified albumin levels
Mentese et al.
Emerg. Med. J. 2008;25:811-814.
ABSTRACT | FULL TEXT  

Screening for Colorectal Cancer: A Targeted, Updated Systematic Review for the U.S. Preventive Services Task Force
Whitlock et al.
ANN INTERN MED 2008;149:638-658.
ABSTRACT | FULL TEXT  

GenoType MTBDR assays for the diagnosis of multidrug-resistant tuberculosis: a meta-analysis
Ling et al.
Eur Respir J 2008;32:1165-1174.
ABSTRACT | FULL TEXT  

Poor Performance of Universal Sample Processing Method for Diagnosis of Pulmonary Tuberculosis by Smear Microscopy and Culture in Uganda
Cattamanchi et al.
J. Clin. Microbiol. 2008;46:3325-3329.
ABSTRACT | FULL TEXT  

Spectrum bias--why clinicians need to be cautious when applying diagnostic test studies
Willis
Fam Pract 2008;25:390-396.
ABSTRACT | FULL TEXT  

Acute Appendicitis: Meta-Analysis of Diagnostic Performance of CT and Graded Compression US Related to Prevalence of Disease
van Randen et al.
Radiology 2008;249:97-106.
ABSTRACT | FULL TEXT  

Evidence-based diagnosis
Whiting et al.
J Health Serv Res Policy 2008;13:57-63.
ABSTRACT | FULL TEXT  

STARD Statement: Still Room for Improvement in the Reporting of Diagnostic Accuracy Studies
Bossuyt
Radiology 2008;248:713-714.
FULL TEXT  

Quality of Reporting of Diagnostic Accuracy Studies: No Change Since STARD Statement Publication--Before-and-after Study
Wilczynski
Radiology 2008;248:817-823.
ABSTRACT | FULL TEXT  

A Guide to Staging in Lung Cancer, but Potentially Misleading without Attention to the Details
Tournoy et al.
Am. J. Respir. Crit. Care Med. 2008;178:209-210.
FULL TEXT  

Grading quality of evidence and strength of recommendations for diagnostic tests and strategies
Schunemann et al.
BMJ 2008;336:1106-1110.
FULL TEXT  

Accuracy of mean arterial pressure and blood pressure measurements in predicting pre-eclampsia: systematic review and meta-analysis
Cnossen et al.
BMJ 2008;336:1117-1120.
ABSTRACT | FULL TEXT  

Balance Impairment Not Predictive of Falls in Geriatric Rehabilitation Wards
Haines et al.
Journals of Gerontology Series A: Biological Sciences and Medical Sciences 2008;63:523-528.
ABSTRACT | FULL TEXT  

Inflammatory Bowel Disease Diagnosed with US, MR, Scintigraphy, and CT: Meta-analysis of Prospective Studies
Horsthuis et al.
Radiology 2008;247:64-79.
ABSTRACT | FULL TEXT  

Diagnostic accuracy of tests for lymph node status in primary cervical cancer: a systematic review and meta-analysis
Selman et al.
CMAJ 2008;178:855-862.
ABSTRACT | FULL TEXT  

Use of uterine artery Doppler ultrasonography to predict pre-eclampsia and intrauterine growth restriction: a systematic review and bivariable meta-analysis
Cnossen et al.
CMAJ 2008;178:701-711.
ABSTRACT | FULL TEXT  

The accuracy of multivariate models predicting ovarian reserve and pregnancy after in vitro fertilization: a meta-analysis
Verhagen et al.
Hum Reprod Update 2008;14:95-100.
ABSTRACT | FULL TEXT  

What is the role of cerebrospinal fluid ferritin in the diagnosis of subarachnoid haemorrhage in computed tomography-negative patients?
Watson et al.
Ann Clin Biochem 2008;45:189-192.
ABSTRACT | FULL TEXT  

Physical examination tests of the shoulder: a systematic review with meta-analysis of individual tests
Hegedus et al.
Br. J. Sports. Med. 2008;42:80-92.
ABSTRACT | FULL TEXT  

Detection of latent tuberculosis in immunosuppressed patients with autoimmune diseases: performance of a Mycobacterium tuberculosis antigen-specific interferon {gamma} assay
Matulis et al.
Ann Rheum Dis 2008;67:84-90.
ABSTRACT | FULL TEXT  

The value of clinical features in the diagnosis of acute pulmonary embolism: systematic review and meta-analysis
West et al.
QJM 2007;100:763-769.
ABSTRACT | FULL TEXT  

Meta-Analysis of MR Imaging in the Diagnosis of Breast Lesions
Peters et al.
Radiology 2007;246:116-124.
ABSTRACT | FULL TEXT  

Optical Coherence Tomography versus Stereoscopic Fundus Photography or Biomicroscopy for Diagnosing Diabetic Macular Edema: A Systematic Review
Virgili et al.
IOVS 2007;48:4963-4973.
ABSTRACT | FULL TEXT  

A systematic review of commercial serological antibody detection tests for the diagnosis of extrapulmonary tuberculosis
Steingart et al.
Postgrad. Med. J. 2007;83:705-712.
ABSTRACT | FULL TEXT  

Accuracy of electrocardiography in diagnosis of left ventricular hypertrophy in arterial hypertension: systematic review
Pewsner et al.
BMJ 2007;335:711-711.
ABSTRACT | FULL TEXT  

Head-to-head comparison of multislice computed tomography and exercise electrocardiography for diagnosis of coronary artery disease
Dewey et al.
Eur Heart J 2007;28:2485-2490.
ABSTRACT | FULL TEXT  

A systematic review of commercial serological antibody detection tests for the diagnosis of extrapulmonary tuberculosis
Steingart et al.
Thorax 2007;62:911-918.
ABSTRACT | FULL TEXT  

Evidence-Based Medicine, Part 3. An Introduction to Critical Appraisal of Articles on Diagnosis
Schranz and Dunn
JAOA: Journal of the American Osteopathic Association 2007;107:304-309.
ABSTRACT | FULL TEXT  

Are you reading what we are reading? The effect of who interprets medical images on estimates of diagnostic test accuracy in systematic reviews
Brealey and Westwood
Br. J. Radiol. 2007;80:674-677.
ABSTRACT | FULL TEXT  

Impact of Renal Disease on Natriuretic Peptide Testing for Diagnosing Decompensated Heart Failure and Predicting Mortality
deFilippi et al.
Clin. Chem. 2007;53:1511-1519.
ABSTRACT | FULL TEXT  

Technology Assessment in Radiology: Putting the Evidence in Evidence-based Radiology
Hollingworth and Jarvik
Radiology 2007;244:31-38.
ABSTRACT | FULL TEXT  

Design-Related Bias in Hospital Fall Risk Screening Tool Predictive Accuracy Evaluations: Systematic Review and Meta-Analysis
Haines et al.
Journals of Gerontology Series A: Biological Sciences and Medical Sciences 2007;62:664-672.
ABSTRACT | FULL TEXT  

Evidence of reference standard related bias in studies of plain radiograph reading performance: a meta-regression
Brealey et al.
Br. J. Radiol. 2007;80:406-413.
ABSTRACT | FULL TEXT  

Accuracy of pulse oximetry in screening for congenital heart disease in asymptomatic newborns: a systematic review
Thangaratinam et al.
Arch. Dis. Child. Fetal Neonatal Ed. 2007;92:F176-F180.
ABSTRACT | FULL TEXT  

First Notification of Positive Blood Cultures and the High Accuracy of the Gram Stain Report
Sogaard et al.
J. Clin. Microbiol. 2007;45:1113-1117.
ABSTRACT | FULL TEXT  

A unification of models for meta-analysis of diagnostic accuracy studies
Harbord et al.
Biostatistics 2007;8:239-251.
ABSTRACT | FULL TEXT  

How should diagnostic tests be evaluated in glaucoma?
Medeiros
Br J Ophthalmol 2007;91:273-274.
FULL TEXT  

Quality of Reporting of Cancer Prognostic Marker Studies: Association With Reported Prognostic Effect
Kyzas et al.
JNCI J Natl Cancer Inst 2007;99:236-243.
ABSTRACT | FULL TEXT  

A Systematic Review of the Diagnosis of Occupational Asthma
Beach et al.
Chest 2007;131:569-578.
ABSTRACT | FULL TEXT  

Increased Diagnostic Information and Understanding Disease: Uncertainty in the Diagnosis of Developmental Hip Dysplasia
Roposch and Wright
Radiology 2007;242:355-359.
ABSTRACT | FULL TEXT  

Impact of Adjustment for Quality on Results of Metaanalyses of Diagnostic Accuracy
Leeflang et al.
Clin. Chem. 2007;53:164-172.
ABSTRACT | FULL TEXT  

The Effects of Study Design and Spectrum Bias on the Evaluation of Diagnostic Accuracy of Confocal Scanning Laser Ophthalmoscopy in Glaucoma
Medeiros et al.
IOVS 2007;48:214-222.
ABSTRACT | FULL TEXT  

Inositol Phosphoglycan P-Type in Preeclampsia: A Novel Marker?
Williams et al.
Hypertension 2007;49:84-89.
ABSTRACT | FULL TEXT  

A Prospective Study of the Prevalence of Primary Aldosteronism in 1,125 Hypertensive Patients
Rossi et al.
J Am Coll Cardiol 2006;48:2293-2300.
ABSTRACT | FULL TEXT  

Developmental Dysplasia of the Hip: Quality of Reporting of Diagnostic Accuracy for US
Roposch et al.
Radiology 2006;241:854-860.
ABSTRACT | FULL TEXT  

Achieving Quality in Cardiovascular Imaging: Proceedings From the American College of Cardiology-Duke University Medical Center Think Tank on Quality in Cardiovascular Imaging
Douglas et al.
J Am Coll Cardiol 2006;48:2141-2151.
FULL TEXT  

US or CT for Diagnosis of Appendicitis in Children and Adults? A Meta-Analysis
Doria et al.
Radiology 2006;241:83-94.
ABSTRACT | FULL TEXT  

The quality of diagnostic accuracy studies since the STARD statement: has it improved?
Smidt et al.
Neurology 2006;67:792-797.
ABSTRACT | FULL TEXT  

Bridging the gaps in evidence based diagnosis.
Straus
BMJ 2006;333:405-406.
FULL TEXT  

Systematic reviews of diagnostic tests in cancer: review of methods and reporting
Mallett et al.
BMJ 2006;333:413.
ABSTRACT | FULL TEXT  

Meta-analysis of plethysmography and rheography in the diagnosis of deep vein thrombosis.
Locker et al.
Emerg. Med. J. 2006;23:630-635.
ABSTRACT | FULL TEXT  

Meta-analysis of diagnostic and screening test accuracy evaluations: methodologic primer.
Gatsonis and Paliwal
Am. J. Roentgenol. 2006;187:271-281.
ABSTRACT | FULL TEXT  

The quality of reporting of diagnostic accuracy studies in glaucoma using the heidelberg retina tomograph.
Shunmugam and Azuara-Blanco
IOVS 2006;47:2317-2323.
ABSTRACT | FULL TEXT  

Multidetector computed tomography for acute pulmonary embolism.
Stein et al.
NEJM 2006;354:2317-2327.
ABSTRACT | FULL TEXT  

Low back pain investigations and prognosis: a review
Refshauge and Maher
Br. J. Sports. Med. 2006;40:494-498.
ABSTRACT | FULL TEXT  

Accuracy of B-type natriuretic peptide levels in the diagnosis of left ventricular dysfunction and heart failure: A systematic review
Latour-Perez et al.
Eur J Heart Fail 2006;8:390-399.
ABSTRACT | FULL TEXT  

Accuracy of B-type natriuretic Peptide tests to exclude congestive heart failure: systematic review of test accuracy studies.
Battaglia et al.
Arch Intern Med 2006;166:1073-1080.
ABSTRACT | FULL TEXT  

Can the Clinical History Distinguish Between Organic and Functional Dyspepsia?
Moayyedi et al.
JAMA 2006;295:1566-1576.
ABSTRACT | FULL TEXT  

Rapid Tests for Group B Streptococcus Colonization in Laboring Women: A Systematic Review
Honest et al.
Pediatrics 2006;117:1055-1066.
ABSTRACT | FULL TEXT  

Urine Telomerase and Bladder Cancer Detection
Weikert et al.
JAMA 2006;295:998-999.
FULL TEXT  

Bias in Research Studies
Sica
Radiology 2006;238:780-789.
ABSTRACT | FULL TEXT  

Evidence of bias and variation in diagnostic accuracy studies.
Rutjes et al.
CMAJ 2006;174:469-476.
ABSTRACT | FULL TEXT  

Sources of bias in diagnostic accuracy studies and the diagnostic process.
Furukawa and Guyatt
CMAJ 2006;174:481-482.
FULL TEXT  

Assessing diagnostic accuracy and the clinical value of positron emission tomography imaging in patients with solitary pulmonary nodules (SNAP)
Kymes et al.
Clin Trials 2006;3:31-42.
ABSTRACT  

What is the impact of change in diagnostic test method on surveillance data trends in Chlamydia trachomatis infection?
Burckhardt et al.
Sex. Transm. Infect. 2006;82:24-30.
ABSTRACT | FULL TEXT  

Coronary Angiography With Multislice Computed Tomography
Dewey
JAMA 2005;294:2298-2298.
FULL TEXT  

Coronary Angiography With Multislice Computed Tomography--Reply
Hoffmann
JAMA 2005;294:2298-2299.
FULL TEXT