How Good Are Our COVID-19 Models?

Fight disinformation: Sign up for the free Mother Jones Daily newsletter and follow the news that matters.

Over the weekend I posted an estimate from Kevin Systrom of how well states are doing in fighting COVID-19. I warned that I couldn’t judge its value independently, and sure enough, one of Tyler Cowen’s correspondents says it’s no good:

Estimating R from noisy real-world data when you don’t know the underlying model is fundamentally difficult….

There’s more at the link, and I have nothing special to say about this particular critique. However, I do have a couple of more general comments.

First, the Systrom estimate is based on case counts, and there’s no question that the reliability of case counts is very poor. Systrom compared case counts only within a 7-day period, and if you assume that the errors don’t change much over such a short period, then perhaps his resulting estimate isn’t too terrible. That said, no matter what statistical wizardry you apply to case counts, you’ll never escape the underlying crappiness of the data.

Second, don’t obsess over the fact that Systrom is estimating R. The underlying data is case counts, and you can do anything you want to them. If you want something that looks sort of R-like, you can do it by applying the appropriate transformations. If you want something that looks different, you can do that too. You’re better off thinking of Systrom’s results not as, literally, R, but as a derivation of “something related to the spread of coronavirus.” But remember that no matter what the derivation is, it’s just a different way of looking at case counts.

Third, I’m skeptical of the demand for an “underlying model” before you do anything else. You may have noticed that there are lots of models of the spread of COVID-19 and they’re all over the map. One of the reasons for this is simple: if you create a model de novo and then fit it to the data, you can get practically any result you want. Obviously experts have some insight into which models are better than others, but this is still a new virus with a lot of unknowns. And very small changes in model specifications can lead to surprisingly large changes in projections. Right now, an awful lot of the model inputs are assumptions with very little empirical backup because COVID-19 is so new, so you should take nearly any model projection with a big grain of salt.

This is where I am right now. Maybe I’m wrong and the models are better than I think. But watching the modelers apply massive amounts of math to fundamentally unreliable data has made me skeptical. This is why, for example, the only thing I show in my charts is daily deaths on a 6-day rolling average. You need to smooth the daily data in some way, but I don’t want to pretend like I’m showing you anything much more than just the basic numbers. And the mortality numbers, although far from perfect, are a whole lot better than case counts.

FOLLOW THE MONEY

Corporations and billionaires don’t fund journalism like ours that exists to shake things up. Instead, support from readers allows Mother Jones to call it like it is without fear, favor, or false equivalence.

And right now, a longtime friend of Mother Jones has pledged an incredibly generous gift to inspire—and double—giving from online readers. That's huge! Because you can see that our fall fundraising drive is well behind the $325,000 we need to raise. So if you agree that in-depth, fiercely independent journalism matters right now, please support our work and help us raise the money it takes to keep Mother Jones charging hard. Your gift, and all online donations up to $94,000 total, will be matched and go twice as far—but only until the November 9 deadline.

$400,000 to go: Please help us pick up the pace!

payment methods

FOLLOW THE MONEY

Corporations and billionaires don’t fund journalism like ours that exists to shake things up. Instead, support from readers allows Mother Jones to call it like it is without fear, favor, or false equivalence.

And right now, a longtime friend of Mother Jones has pledged an incredibly generous gift to inspire—and double—giving from online readers. That's huge! Because you can see that our fall fundraising drive is well behind the $325,000 we need to raise. So if you agree that in-depth, fiercely independent journalism matters right now, please support our work and help us raise the money it takes to keep Mother Jones charging hard. Your gift, and all online donations up $94,000 total, will be matched and go twice as far—but only until the November 9 deadline.

$400,000 to go: Please help us pick up the pace!

payment methods

We Recommend

Latest

Sign up for our free newsletter

Subscribe to the Mother Jones Daily to have our top stories delivered directly to your inbox.

Get our award-winning magazine

Save big on a full year of investigations, ideas, and insights.

Subscribe

Support our journalism

Help Mother Jones' reporters dig deep with a tax-deductible donation.

Donate