Now together: the most trusted covid-19 model is a whole

Each week, each team not only submits a point forecast that predicts the outcome of a single number (for example, 500 people will die in a week). They also submitted probabilistic forecasts to quantify the uncertainty by estimating the number of cases or the likelihood of deaths. The interval or range is getting narrower and narrower, and the central forecast is the goal. For example, the model might predict that there is a 90% probability of seeing 100 to 500 people die, a 50% probability of seeing 300 to 400 people, and a 10% probability of seeing 350 to 360 people.

“It’s like a bullseye, getting more focused,” Reich said.

Funk added: “The clearer your definition of the target, the less likely you are to hit it.” This is a good balance, because arbitrarily wide predictions are correct but also useless. “It should be as precise as possible,” Funk said, “and also give the correct answer.”

When collating and evaluating all individual models, the integration attempts to optimize their information and mitigate their shortcomings. The result is a probabilistic forecast, statistical average or “median forecast”. In essence, this is a consensus, a finer calibration of uncertainty, and therefore more realistic. All uncertain factors are averaged during washing.

The Reich Lab’s research focused on the predicted number of deaths and evaluated approximately 200,000 predictions from mid-May to late December 2020 (updated analysis of predictions for the other four months will be added soon), and found that a single model’s The performance is very changeable. One week model may be accurate, next week it may be far away. However, as the author writes, “When combining the predictions of all teams, the set shows the best overall probability accuracy.”

Ashleigh Tuite, an epidemiologist at the Dalarana School of Public Health at the University of Toronto, said that these ensemble exercises not only help improve predictions, but also help people trust the model. “One lesson of integrated modeling is that no model is perfect,” Tuite said. “Even bands sometimes miss important things. Generally speaking, it is difficult for models to predict the inflection point-the peak, or whether things suddenly start to accelerate or decelerate.”

The use of integrated modeling is not unique to the pandemic. In fact, we use probability ensemble forecasts every day when we search for the weather on Google and notice that there is a 90% chance of precipitation. It is the gold standard for weather and climate prediction.

Tilmann Gneiting, a computational statistician at the Heidelberg Institute of Theory and Karlsruhe Institute of Technology in Germany, said: “This is a real success story, and it’s the path of the past about three years.” Gneiting said Before the collection, the weather forecast uses a single numerical model to produce deterministic weather forecasts in its original form. The model is “overconfident and extremely unreliable” (weather forecasters are aware of this problem and place the original results in Follow-up statistical analysis By the 1960s, a fairly reliable precipitation forecast probability had been produced).

However, Gneiting pointed out that the analogy between infectious diseases and weather forecasts has its limitations. On the one hand, the probability of precipitation does not change with human behavior-rain, umbrella or no umbrella-and the trajectory of the pandemic will respond to our preventive measures.

Prediction during a pandemic is a system affected by a feedback loop. “Models are not oracles,” said Alessandro Vespignani, a computational epidemiologist at Northeastern University and contributor to the Integration Center, who studies complex networks and the spread of infectious diseases, focusing on driving feedback Mechanism of the “technological society” system. “Any model provides an answer conditioned on certain assumptions.”

When people deal with the model’s predictions, their subsequent behavior changes can overturn assumptions, change disease dynamics and make predictions inaccurate. In this way, modeling can become a “self-destructing prophecy.”

There are other factors that may exacerbate this uncertainty: seasonality, variation, vaccine availability or vaccination; and policy changes, such as the CDC’s rapid decision to unmask. Justin Lesler, an epidemiologist at the Bloomberg School of Public Health at Johns Hopkins University, said: “These are huge unknowns. If you really want to capture the uncertainty of the future, it will really limit you. Words.” COVID-19 Forecast Center.

The overall study of death predictions observed that as the model predicts the future farther, the accuracy will decrease and the uncertainty will increase-the error of looking forward 4 weeks is about twice that of 1 week (4 weeks is Considered to be a meaningful short-term limit) forecast; within a 20-week time frame, the error is about 5 times).

“It is fair to discuss when it is valid and when it is invalid.”

Johannes Brahe

But evaluating the quality of the model (including all issues) is an important secondary goal of the forecasting center. And this is easy to do, because short-term forecasts will soon face the reality of daily statistics as a measure of their success.

Most researchers carefully distinguish this type of “prediction model”, aiming to make clear and verifiable predictions about the future, which can only be achieved in the short term; compared with “scenario models”, exploring “hypothetical” assumptions, Plot lines that may develop in the mid- to long-term future (because scenario models are not predictions, they should not be retrospectively evaluated based on reality).

During a pandemic, the key focus is often on models that predict severe errors.Johannes Bracher, a biostatistician at the Heidelberg Institute of Theory and Karlsruhe Institute of Technology, said: “Although longer-term hypothetical forecasts are difficult to evaluate, we should not shy away from comparing short-term forecasts with reality.” Who coordinates one? German and Polish hub, And provide suggestions for European hubs. “When things work and when they don’t work, the debate is fair,” he said. But a wise debate needs to recognize and consider the limitations and intent of the model (sometimes the most fierce critics are those who mistake the situational model for the predictive model).

Similarly, when the prediction in any given situation proves to be particularly tricky, the modeler should say so. “If we learn one thing, it is that even in the short term, cases are extremely difficult to model,” Brahe said. “Death is a more lagging indicator and easier to predict.”

In April, some European models were too pessimistic and missed the sudden reduction in cases. Then there was a public debate on the accuracy and reliability of the epidemic model. Bracher weighed in on Twitter and asked: “These models (not uncommon) are wrong. Is this surprising? After a year of pandemic, I would say: No.” He said that this makes the models show them The level of certainty or uncertainty, and their taking a realistic stance on unpredictable conditions and future developments becomes more important. “Modelers need to communicate uncertainty, but it should not be considered a failure,” Brahe said.

Trust certain models more than others

As the frequently quoted statistical aphorism says, “All models are wrong, but some are useful.” But as Bracher pointed out, “If you adopt an integrated model approach, in a sense, you are saying All models are useful, and each model has some contribution”-although some models may provide more information or be more reliable than others.

Observing this fluctuation prompted Reich and others to try to “train” ensemble models—that is, as Reich explained, “build algorithms to teach ensemble to trust some models more than others, and to understand which models are accurate Combinations can work together.” Bracher’s team is now contributing a mini-integration, built only from models that have been performing well in the past, amplifying the clearest signals.

“The biggest question is, can we improve?” Reich said. “The original method is that simple. It seems that there must be a way to improve the simple average of all these models.” However, so far, it has proved to be more difficult than expected-small improvements seem feasible, but obvious This improvement is almost impossible.

In addition to the weekly glimpses, an additional tool to improve our overall view of the pandemic is to use these “scenario modeling” to observe further over a four to six month time frame.Last December, driven by the surge in cases and the imminent availability of vaccines, Lessler and its collaborators launched COVID-19 Scenario Modeling Center, Negotiate with the Centers for Disease Control and Prevention.