When Social Science Fails Itself

Yesterday, The New York Times reported on a study suggesting that less than half of research findings published in prominent psychology journals could be confirmed upon replication. The stunned, and at times stupefied, reactions from many readers show the public lacks a fundamental understanding of the scientific process.

The authors of "Estimating the reproducibility of psychological science," published in Science, investigated 100 manuscripts published in three leading psychology journal. Their goal was to test the veracity of the findings by replicating the original procedures; only 36 percent of findings in the original studies remained significant upon replication.

To many, these inaccuracies represent a damning failure for social sciences. I would argue they actually embody a triumph for the process of science, but at the same time point to a glaring problem in the process of publishing scientific research.

The public views publication as the pinnacle of research, believing that if it makes it to print then it must be fact. Therefore, when studies fail the test of replication, public confidence in science is shaken. And it is shaken. 

But that's largely because of a misconception that a single study is enough to establish research findings as facts. In truth, science is an exercise in consensus. When we are able to establish that findings hold true over time and across varied situations, we build a reliable body of knowledge that becomes the basis for scientific understanding. However, when replication fails to yield support for a particular finding, it is dismissed and we move forward with different ideas. Or at least that's how it's supposed to work.

In truth, the problem lies not with the scientific method, but instead with the publish-or-perish environment of academia. Tenure and promotion are based largely on one's ability to publish original research, and to publish it often. Problematically, we take the notion of "original" a bit too literally. As the authors of the Science piece put it:

Reproducibility is not well understood because the incentives for individual scientists prioritize novelty over replication. Innovation is the engine of discovery and is vital for a productive, effective scientific enterprise. However, innovative ideas become old news fast. Journal reviewers and editors may dismiss a new test of a published idea as unoriginal. The claim that "we already know this" belies the uncertainty of scientific evidence.

Put simply, replication is a necessary component science, but it's not sexy, so it's hard to publish. And since academics must publish to survive, they don't replicate studies often.

Perhaps the saddest part of this indictment is that it's our own damn fault. Most reputable social scientific journals are peer reviewed, meaning that we have the power to rectify a problem we know exists simply by changing our review policies.

So why don't we? I thought The New York Times was spot on there:

The act of double-checking another scientist's work has been divisive. Many senior researchers resent the idea that an outsider, typically a younger scientist, with less expertise, would critique work that often has taken years of study to pull off.

Certainly some -- but not all -- senior researchers feel this way, and I find it shameful. The whole premise of scientific inquiry is that no person or idea is above reproach. To quote Albert Einstein, "The important thing is not to stop questioning." 

Consequently, when Einstein first published papers on the photoelectric effect, Brownian motion, and special relativity -- the last of which shook the very foundations of Newtonian physics -- he was only in his mid-20s, very much a junior scholar. But how much could a 26-year-old possibly know anyway?

Jon Stewart: The Walter Cronkite of a Generation

Stewart at the 2008 USO-Metro
Merit Awards

Yeah, I said it. Well, more specifically Lewis Black said it:

Weirdly enough, [Jon Stewart] was, on a certain level, the Walter Cronkite of his generation. He was the trusted source. People trusted what came out of his mouth. What made Cronkite that and what makes Jon that is a certain kind of honesty that is related through the medium, and the one thing I learned from television is that it doesn't lie.

While Lewis Black may have stolen my headline in his Entertainment Weekly interview, and while I agree with the gist of his argument, I think there's more to the Stewart-Cronkite comparison. Specifically, I would argue temperance, sincerity, accuracy, and depth are the chief contributors to Stewart's legacy.

The guttural reaction to the Stewart-Cronkite comparison is easy to sum up: Stewart is a comedian, Cronkite is a newsman, so how can you compare the two? Stewart was more than a comedian. He was able to elevate to higher levels of credibility than his comedic counterparts due largely to his temperate nature.

Stewart speaks sternly, but without the rage of a Lewis Black. Stewart hits with biting satire, but refrains from the obtuse and deadpan tendencies of his protege, Stephen Colbert. And Stewart has always been more influential than the late night hosts of network television because he was never afraid to say something of substance or to alienate segments of his would-be audience. But, while Stewart is certainly left-leaning, he avoids playing the outraged -- and often off-putting -- liberal we see in Bill Maher.

In fact, in 2010 Maher very publicly criticized Stewart for taking it too easy on Republicans. I happen to agree with Maher's assessment that, if we were to dole out points for craziness among the two major parties, the Republicans would win in a landslide that would make the Reagan-Mondale election look like a nail-biter.

The brilliance of Stewart was that he never took the bait. While he criticizes the right with greater frequency than the left -- and at times more harshly -- Stewart lacks the smugness that Maher constantly displays. There is a real sense that Stewart is wrestling with the same political non sequiturs as his viewers, where Maher appears stubbornly entrenched in a preset ideology.

In many ways, the ongoing struggle of Stewart's personal politics fuels his quest for accuracy in reporting. Love him or hate him, you have to admire that Stewart works incredibly hard to accurately represent the facts, despite that, as a comedian, he has no professional responsibility to do so. Even more impressive to me, he freely admits factual errors that he makes, perhaps most notably in his mischaracterization of Dante Parker's death as the result of police shooting rather than a drug overdose.

In that same segment, Stewart described of The Daily Show as a "media counter-errorism" program and the difficulties of working within such confines. Predictably, Fox News dismissed Stewart's genuine effort to raise issues of police brutality and militarization for a more simplistic narrative: Jon Stewart hates police officers. Fox News anchor Brian Kilmeade went so far as to imply that Stewart was at best unfeeling toward police officers who die in the line of duty. Stewart's response to Kilmeade was characteristic of the complexity and nuance he attempts to bring to reporting:

You can truly grieve for every officer who has been lost in the line of duty in this country and still be troubled by cases of police overreach. Those two ideas are not mutually exclusive. You can have great regard for law enforcement and still want them to be held to high standards.

More than anything, that nuance, that context, and that depth which Stewart brings to important issues make him an effectual newsman. I don't watch television news anymore. My main source for news is The New York Times. Largely this is because I can't remember the last time I watched a nightly network newscast or a 24-hour cable news program and walked away feeling informed.

Part of that is a function of a fragmented media environment. It's difficult for anyone to have the gravitas and impact of a Cronkite because news audiences just aren't that large anymore. Also, a key aspect of news is that it is, well, new. Being the outlet to break a story matters, and the continuous nature of the news cycle in out digital environment puts an even higher premium on being first. Sadly, the drive to be first often leads to speculation, and even worse, misinformation. Unfortunately, many traditional news outlets aren't as responsible about correcting their factual errors as The Daily Show.

Another factor in the decline of TV news is a lack of resources. There was a time when networks had reporters on the ground, actually gathering information. Now, more often than not, on-location reporters are there just to say they are there -- and to get a good background shot, not to provide background on the story. That's one reason the "on location" reports from The Daily Show correspondents are so funny: we all know they're not on location, and we all know it doesn't matter.

What's worse, I all-too-frequently see anchors reading tweets from viewers as if they were field reports. I just start throwing things at the screen. Man-on-the-street interviews were always sorry excuses for news coverage, and viewer tweets are no different. I don't care what @hawkeyewoman9834 (or Kathy, 33, from Iowa) thinks about the administration's new immigration policy, though as an unemployed dental hygienist and mother of three, I can see why CNN sought out her expert opinion.

Institutional forces aside, much of the blame lies with the men and women in the television news business. The news media is supposed to be our watchdog, which means calling bullshit on institutional powers and their leaders. Somehow that core function of the news has been relabeled as subjective opinion, or even worse, partisanship. Good journalists have been scared into a position of reporting facts with no context, leaving relatively uninformed and often unqualified viewers to interpret what they see and hear. Bad journalists have devolved into pure punditry, where ideology dictates the facts rather than being informed by them.

This wasn't always the case. In 1954, Edward R. Murrow unabashedly took on Joseph McCarthy. He pointed out the contradictions made by the senator and demonstrated that, even if the Red Scare were not an overreaction, the conduct of the House Un-American Activities Committee was itself, ironically, un-American. Was that a partisan attack on a Republican senator or simply a statement of what was and a reasoned argument that it ought not be? Currently, the Edward R. Murrow Award is among the most prestigious honors given in recognition of outstanding electronic journalism.

Following the Tet Offensive in 1968, Cronkite said the following concerning the Vietnam War:

To say that we are closer to victory is to believe, in the face of the evidence, the optimists who have been wrong in the past. To suggest we are on the edge of defeat is to yield to unreasonable pessimism. To say that we are mired in stalemate seems the only realistic, yet unsatisfactory conclusion. On the off chance that military and political analysts are right, in the next few months we must test the enemy's intentions, in case this is indeed his last big gasp before negotiation. But it is increasingly clear to this reporter that the only rational way out then will be to negotiate, not as victors, but as an honorable people who lived up to their pledge to defend democracy, and did the best they could.

Cronkite traveled to Vietnam, gathered information first-hand, analyzed it, and developed a reasoned argument that America was in the midst of an unwinnable war. Was Cronkite acting as an unpatriotic communist unsupportive of U.S. troops, or a responsible journalist seeking to inform his audience as best he could? It's worth noting that Walter Cronkite hosted The CBS Evening News for another 13 years after that report, retiring as one of the most respected journalists and revered public figures in U.S. history.

Like Murrow and Cronkite, Stewart isn't afraid to call bullshit. He isn't afraid of controversy. He isn't afraid of reasoned editorializing. In short, he isn't afraid to inform his viewers, which sadly can't be said of many "real" television journalists. What's more, he makes you laugh, if only to keep you from crying. In the broader American zeitgeist, Stewart's legacy won't even remotely challenge that of Cronkite, but for his generation and for mine, Stewart may be remembered as the most trusted man in America.