'All Models Are Wrong But Some Are Useful': What COVID-19 Predictions Can And Can't Tell Us
As the COVID-19 pandemic rages on, many are wondering when it will be over. And it's easy to look to predictive modeling to provide clarity on that.
But modeling isn't always as straightforward as one might hope.
“All models are wrong but some are useful,” said Dr. Jonathan Samet, quoting a famous statistical axiom. He’s a pulmonary physician, epidemiologist and dean of the Colorado School of Public Health.
He also helped put together the COVID-19 infection, hospitalization and death models released by the Colorado Department of Public Health and Environment this week.
The department released three models put together by Dr. Samet and a team of researchers from the University of Colorado School of Medicine at the CU Anschutz Medical Campus, CU Boulder and CU Denver. The latest and most detailed thus far was made public Monday night.
The predictions show how high the peak of infections could be and when that peak could happen. But, the numbers and dates change depending on how effective social distancing efforts have been in the state since March 26, the date the stay-at-home order began.
If social distancing is 60% effective in reducing contact between people, the peak would come in mid-August at 64,613 infections. At 40% efficacy, the peak number of infections more than doubles, but it would happen in June.
Without any social distancing, the model predicts the peak would come next month but result in 222,643 infections.
On the other hand, with 80% effective social distancing since the stay-at-home order, the model actually suggests the peak would have come last week at around 2,000 people infected. We are well beyond that point now though - there are already 5,429 confirmed cases in the state.
The model doesn’t just predict infections. It also looks at deaths and hospitalizations. A social distancing efficacy of 50% or less could result in between 13 and 73,000 deaths and very overwhelmed hospitals.
Different groups, using different methods, can produce wildly different predictions. The state’s own models, produced by the same team, disagree with previous versions of themselves sometimes - even though they were released publicly just a day apart.
Even the latest model doesn’t have just one answer to when this will be over and how bad it will be. But that’s not the purpose of these models, not at this point at least, Dr. Samet said.
They exist to provide decision-makers, like the governor or local health officials, with a look at the different ways this crisis could play out in various circumstances.
“We use models as tools to say ‘well if we implement the kind of social distancing, for example, we have in place now, what are the consequences? When do they show up?’” he said. “Hopefully, not too long from now, we’ll be able to say ‘well, what happens if we begin to lighten up on social distancing.’”
As more data becomes available and the team adjusts its methods, the models will likely become more and more effective. They’re hoping to have a better idea of exactly how much contact Coloradans are actually having with each other in the next few days, which will narrow things down.
Another model by the Institute for Health Metrics and Evaluation has been getting a lot of traction. It’s been used by a lot of news media across the country, including internally at KUNC. Colorado’s projections weren’t publically available until this weekend, so there weren’t many options to look at.
The IHME model does not include peak infection projections. But it does look at hospital resources, and it says that Colorado’s needs for that should have peaked Monday. The state doesn’t think that is quite right, as state epidemiologist Rachel Herlihy noted in a press conference.
“We feel like the data that is being used to populate the model that’s being developed by the school of public health, we feel like, is probably the best model for Colorado right now,” she said. “Because it’s using assumptions that we feel to be true for Colorado. It’s using Colorado data.”
The models have to make assumptions and have certain limitations. The state’s model, for example, assumes that everyone who needs an ICU bed but doesn’t get one will die. It also doesn’t quite account for virus transmission in hospitals.
But the IHME model does something very important with its assumptions. Right at the top of the model’s webpage, it says “projections assuming full social distancing.”
The state’s model doesn’t go over 80% social distancing efficacy, because they feel like anything higher wouldn’t be reasonable.
“Efforts were focused on where we felt, most plausibly, what degree of social distancing we could most plausibly achieve,” Dr. Samet said.
As much as the models don’t always align in terms of numbers and dates, they do all share the same overarching message: social distancing makes a huge difference in how long and how hard the epidemic will hit Colorado.
“We’re in this for a longer run than anybody wants,” Dr. Samet said. “Then it’s going to be up to public health agencies, the governor to set policies that will safely allow us to begin (to) return to normal.”