Why Most Published Research Findings Are False

ALLIANCE FOR HUMAN RESEARCH PROTECTION (AHRP)
Promoting Openness, Full Disclosure, and Accountability www.ahrp.org

An article published in PLoS Medical (an Open Access Peer Reviewed Medical Journal) by an epidemiologist who holds academic positions on both sides of the Atlantic--at the University of Ioannina (Greece) and Tufts University (Massachusetts)-- knocks the socks off much of what passes the peer review process of medical journals.

Dr. John Ioannidis examines, what he says are the key factors that have resulted in mostly false, published medical research reports:

"There is increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims."

Dr. Ioannidis notes that the probability that a research finding is indeed true depends on "the prior probability of it being true (before doing the study), the statistical power of the study, and the level of statistical significance." He notes the high rate of nonreplication--a sure sign of the findings being false.

"What is less well appreciated is that bias and the extent of repeated independent testing by different teams of investigators around the globe may further distort this picture and may lead to even smaller probabilities of the research findings being indeed true....Several independent teams may be addressing the same sets of research questions. As research efforts are globalized, it is practically the rule that several research teams, often dozens of them, may probe the same or similar questions. Unfortunately, in some areas, the prevailing mentality until now has been to focus on isolated discoveries by single teams and interpret research experiments in isolation. An increasing number of questions have at least one study claiming a research finding, and this receives unilateral attention. The probability that at least one study, among several done on the same question, claims a statistically significant research finding is easy to estimate."

Dr. Ioannidis defines bias as "the combination of various design, data, analysis, and presentation factors that tend to produce research findings when they should not be produced.

Let u be the proportion of probed analyses that would not have been "research findings," but nevertheless end up presented and reported as such, because of bias."

He differentiates bias from chance variability "that causes some findings to be false by chance even though the study design, data, analysis, and presentation are perfect."

"Bias can entail manipulation in the analysis or reporting of findings. Selective or distorted reporting is a typical form of such bias."

Dr. Ioannidis' observation that "claimed research findings may often be simply accurate measures of the prevailing bias," is especially applicable to psychiatry.

The focus of a forthcoming presentation by Dr. David Healy, "Psychopharmacology in Turmoil:A Scientific or Ethical Crisis?" is bias, faulty data analysis, and manipulated clinical trial finding reports as manifested in the psychiatric literature. In his lecture, at Columbia University Medical Center (October 20), Dr. Healy will demonstrate how current clinical practice guidelines that purport to be "evidence-based" are not not based on scientifically valid evidence at all.

For those who are unaware of Dr. Healy's contribution to the credibility crisis in psychiatry: his analysis of previously undisclosed company data from SSRI drug trials contradicted the published reports about these trials. His findings of a drug-induced suicide risk, challenged the mindset and prescribing practices of the psychiatric establishment in the UK, Canada, Australia, and the US. By bringing the undisclosed hazards to public notice, the debate about the efficacy and safety of SSRs--and the validity of the process by which they were tested--reached a crescendo.

Families whose children became suicidal after being prescribed an SSRI--some becoming casualties of drug-induced suicide--came to Washington from coast to coast to testify before two FDA advisory committee hearings (in February and September, 2004). Their compelling testimonies coupled with independent analyses replicating Dr. Healy's, resulted in mandatory Black Box warnings about the drugs posing a twofold risk of suicidal behavior for adolescents.

"Psychopharmacology in Turmoil:A Scientific or Ethical Crisis?"
Date: Thursday, October 20
Time: 12:30 - 2:00
Location: Columbia Presbyterian Hospital --622 West 168th Street 10th floor - Irving Conference Room

Directions from Penn Station: take the #1 train (red line) uptown or the A train (blue line) uptown to 168th street. At 168 Street station (#1 train) take the elevator up to 168th and Broadway A train: just go up the stairs to 168th and Broadway
>From trains: walk west on 168th street 1/2 block to circular driveway:
Presbyterian Hospital 622 west 168th street
get a pass from the guard. take elevator behind the guard desk to the 10th floor go through the doors to the Irving Conference Room.

After the lecture there will be aboout 40 minutes for Q & A

Contact: Vera Hassner Sharav
212-595-8974


http://medicine.plosjournals.org/perlserv/?request=get-document&doi=10.1371/journal.pmed.0020124

Why Most Published Research Findings Are False
John P. A. Ioannidis

Summary
There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.

John P. A. Ioannidis is in the Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece, and Institute for Clinical Research and Health Policy Studies, Department of Medicine, Tufts-New England Medical Center, Tufts University School of Medicine, Boston, Massachusetts, United States of America. E-mail: jioannid@cc.uoi.gr