Researchers find substantial uncertainties in Covid-19 pandemic simulations

Computer modelling to forecast Covid-19 mortality contains significant uncertainty in its predictions, according to an international study led by researchers at UCL and CWI. Their article was published in Nature Computational Science on 22 February.

Publication date
22 Feb 2021

Computer modelling to forecast Covid-19 mortality contains significant uncertainty in its predictions, according to an international study led by researchers at UCL and CWI in the Netherlands. Their article ‘The Impact of Uncertainty on Predictions of the CovidSim Epidemiological Code’, which was earlier announced in a Nature news item, was published in Nature Computational Science on Monday evening 22 February 2021.

While in a physical experiment it is common practice to provide error bars along with  the measured values,  the predictions from a computer model often lack a measure of uncertainty. This is despite the fact that such models undeniably are uncertain, and are used in high-level decision making. The international research team argues that computational predictions without error bars can paint a very incomplete picture, which they demonstrated in a recent study with a  computer model used for evaluating COVID-19 intervention scenarios.

This study was done within the VECMA project, a European Union Horizon 2020 research and innovation programme. At CWI, researchers Wouter Edeling and Daan Crommelin from the Scientific Computing group were involved. Edeling, first author of the article, and a specialist in quantifying modelling uncertainties, made part of the software for the VECMA toolkit. This was used to couple a uncertainty quantification (UQ) technique to the well-known epidemiological ‘CovidSim’ model from Neil Ferguson of Imperial College in the UK and data from the UK from 2020.

‘Curse of dimensionality’

Edeling explained: “For models with a high number of parameters like CovidSim, it is very difficult to study which effect uncertainties in the input parameters have on uncertainties in the output. Having many parameters means that the computational costs will be inordinately high – often referred to as the ‘curse of dimensionality’. We investigate how to do the computations as efficiently as possible, by finding out which parameters matter most for the output uncertainties. By focusing on these parameters it becomes possible to make good probabilistic predictions, which can be used by governments for their decisions.”  

The new methods are very effective. In testing the robustness of CovidSim the research team found that, although the code contained 940 parameters, 60 were important and, of those, only 19 dominated the variance in the output predictions. Half of the overall variation in their results was down to just three of the 940 input parameters: the disease’s latency period, the delay in an infected person self-isolating, and the effectiveness of social distancing. Whereas the latency period is a biological parameter, the other two (and quite a few others which were influential) are related to the intervention scenarios and human behavior. While they represent a difficult modeling task, unlike the biological aspects, these parameters (and the phenomena which they model), can be influenced by governmental policy.


CWI research on uncertainty quantification

Daan Crommelin, group leader of CWI’s Scientific Computing group and professor at UvA , says: "It is important to know about the uncertainties of computer simulations and model predictions. Not only for simulations of the pandemic but also for example in environmental studies, weather and climate modeling, or economics. It may be tempting to focus on a single prediction or a single simulation outcome, but it can also be very relevant to see how wide the range of possible outcomes is."

Partners

The study was carried out by researchers affiliated with CWI, University College London (UCL), the University of Amsterdam, Brunel University London, and the Poznan Supercomputing and Networking Centre in Poland. The work was funded as part of the European Union Horizon 2020 research and innovation programme (VECMA project #800925, www.vecma.eu)

The VECMA Consortium in 2019 in Amsterdam. Picture: VECMA.
The VECMA Consortium in 2019 in Amsterdam. Picture: VECMA.

More information

About CWI

Founded in 1946, Centrum Wiskunde & Informatica (CWI) is the national research institute for mathematics and computer science in the Netherlands. It is located at Amsterdam Science Park and is part of the Netherlands Organisation for Scientific Research (NWO). The institute is internationally renowned. Over 150 researchers conduct pioneering research and share their acquired knowledge with society. Over 30 researchers are also employed as professors at universities. The institute has generated twenty-seven spin-off companies.