Here’s a Textbook Criticism of Econometrics Gone Bad

A couple of weeks ago I linked to a study that suggested the anti-overdose drug Nalexone might, on net, actually cause more opioid deaths. I was skeptical because the data seemed to show no real pattern. The authors drew a dashed red line at the point where Nalexone was legalized, and sure enough, it looked like opioid deaths jumped after the new laws were passed. But the line seemed entirely arbitrary.

Now Andrew Gelman has weighed in. He has pretty much the same criticisms I did, but he also has the statistical chops to be taken seriously. However, he also links to Richard Border, who doesn’t say a word but lets the numbers speak for themselves:

This is a textbook criticism of statistics based on too few data points. As Border shows, you can put the cutoff line anywhere and it looks like there’s a big jump in opioid deaths. But that doesn’t mean anything. The error bars are enormous, and the before-and-after “trends” are meaningless.

I’d add that this also shows the value of simply using common sense. If you remove all the lines and the error bars and simply look at the scatterplot, there’s nothing there. That should make you skeptical from the very start. It’s true that occasionally you can tease real results out of data like this, but you’d better be damn sure of what you’re doing and you’d better check it ten different ways. The authors of this study didn’t do that.

TIME IS RUNNING OUT!

We have an ambitious $350,000 online fundraising goal this month and it's truly crunch time: About 15 percent of our yearly online giving usually comes in during the final week of the year, and in "No Cute Headlines or Manipulative BS," we explain why we simply can't afford to come up short right now.

The bottom line: Corporations and powerful people with deep pockets will never sustain the type of journalism Mother Jones exists to do. And advertising or profit-driven ownership groups will never make time-intensive, in-depth reporting viable.

That's why donations big and small make up 74 percent of our budget this year. There is no backup to keep us going, no alternate revenue source, no secret benefactor. If readers don’t donate, we won’t be here. It's that simple.

And if you can help us out with a donation right now, all online gifts will be matched thanks to an incredibly generous matching gift pledge.

payment methods

TIME IS RUNNING OUT!

We have an ambitious $350,000 online fundraising goal this month and it's truly crunch time: About 15 percent of our yearly online giving usually comes in during the final week of the year, and in "No Cute Headlines or Manipulative BS," we explain why we simply can't afford to come up short right now.

The bottom line: Corporations and powerful people with deep pockets will never sustain the type of journalism Mother Jones exists to do. And advertising or profit-driven ownership groups will never make time-intensive, in-depth reporting viable.

That's why donations big and small make up 74 percent of our budget this year. There is no backup to keep us going, no alternate revenue source, no secret benefactor. If readers don’t donate, we won’t be here. It's that simple.

And if you can help us out with a donation right now, all online gifts will be matched thanks to an incredibly generous matching gift pledge.

payment methods

We Recommend

Latest

Sign up for our free newsletter

Subscribe to the Mother Jones Daily to have our top stories delivered directly to your inbox.

Get our award-winning magazine

Save big on a full year of investigations, ideas, and insights.

Subscribe

Support our journalism

Help Mother Jones' reporters dig deep with a tax-deductible donation.

Donate