When Social Science Fails Itself

Yesterday, The New York Times reported on a study suggesting that less than half of research findings published in prominent psychology journals could be confirmed upon replication. The stunned, and at times stupefied, reactions from many readers show the public lacks a fundamental understanding of the scientific process.

The authors of "Estimating the reproducibility of psychological science," published in Science, investigated 100 manuscripts published in three leading psychology journal. Their goal was to test the veracity of the findings by replicating the original procedures; only 36 percent of findings in the original studies remained significant upon replication.

To many, these inaccuracies represent a damning failure for social sciences. I would argue they actually embody a triumph for the process of science, but at the same time point to a glaring problem in the process of publishing scientific research.

The public views publication as the pinnacle of research, believing that if it makes it to print then it must be fact. Therefore, when studies fail the test of replication, public confidence in science is shaken. And it is shaken. 

But that's largely because of a misconception that a single study is enough to establish research findings as facts. In truth, science is an exercise in consensus. When we are able to establish that findings hold true over time and across varied situations, we build a reliable body of knowledge that becomes the basis for scientific understanding. However, when replication fails to yield support for a particular finding, it is dismissed and we move forward with different ideas. Or at least that's how it's supposed to work.

In truth, the problem lies not with the scientific method, but instead with the publish-or-perish environment of academia. Tenure and promotion are based largely on one's ability to publish original research, and to publish it often. Problematically, we take the notion of "original" a bit too literally. As the authors of the Science piece put it:

Reproducibility is not well understood because the incentives for individual scientists prioritize novelty over replication. Innovation is the engine of discovery and is vital for a productive, effective scientific enterprise. However, innovative ideas become old news fast. Journal reviewers and editors may dismiss a new test of a published idea as unoriginal. The claim that "we already know this" belies the uncertainty of scientific evidence.

Put simply, replication is a necessary component science, but it's not sexy, so it's hard to publish. And since academics must publish to survive, they don't replicate studies often.

Perhaps the saddest part of this indictment is that it's our own damn fault. Most reputable social scientific journals are peer reviewed, meaning that we have the power to rectify a problem we know exists simply by changing our review policies.

So why don't we? I thought The New York Times was spot on there:

The act of double-checking another scientist's work has been divisive. Many senior researchers resent the idea that an outsider, typically a younger scientist, with less expertise, would critique work that often has taken years of study to pull off.

Certainly some -- but not all -- senior researchers feel this way, and I find it shameful. The whole premise of scientific inquiry is that no person or idea is above reproach. To quote Albert Einstein, "The important thing is not to stop questioning." 

Consequently, when Einstein first published papers on the photoelectric effect, Brownian motion, and special relativity -- the last of which shook the very foundations of Newtonian physics -- he was only in his mid-20s, very much a junior scholar. But how much could a 26-year-old possibly know anyway?