64% of Psychology Experiments Fail Replication Test

An international team of experts found that of 100 studies published in top-ranking journals in 2008, the results of just 25 percent of social psychology experiments and half of cognitive studies could be replicated in independent trials.

In the investigation, conducted in response to rising concerns over the reliability of psychological research, 270 scientists on five continents repeated various experiments.

Brian Nosek, a professor of psychology who led the study at the University of Virginia, said of the findings: “There is no doubt that I would have loved for the effects to be more reproducible. … I am disappointed, in the sense that I think we can do better.

“The key caution that an average reader should take away is [that] any one study is not going to be the last word,” he added. “Science is a process of uncertainty reduction, and no one study is almost ever a definitive result on its own.”

Guardian science editor Ian Sample writes:

All of the experiments the scientists repeated appeared in top ranking journals in 2008 and fell into two broad categories, namely cognitive and social psychology. Cognitive psychology is concerned with basic operations of the mind, and studies tend to look at areas such as perception, attention and memory. Social psychology looks at more social issues, such as self esteem, identity, prejudice and how people interact.

In the investigation, a whopping 75% of the social psychology experiments were not replicated, meaning that the originally reported findings vanished when other scientists repeated the experiments. Half of the cognitive psychology studies failed the same test. Details are published in the journal Science.

Even when scientists could replicate original findings, the sizes of the effects they found were on average half as big as reported first time around. That could be due to scientists leaving out data that undermined their hypotheses, and by journals accepting only the strongest claims for publication. …

[Marcus Munafo, a co-author on the study and professor of psychology at Bristol University] said that the problem of poor reproducibility is exacerbated by the way modern science works. “If I want to get promoted or get a grant, I need to be writing lots of papers. But writing lots of papers and doing lots of small experiments isn’t the way to get one really robust right answer,” he said. “What it takes to be a successful academic is not necessarily that well aligned with what it takes to be a good scientist.”

Read more here.

— Posted by Alexander Reed Kelly.