From mortgage approvals to running the electricity grid, algorithms support people in making complex decisions. In practice, an algorithm analyses past data and current signals to recommend an action, like approving a loan, schedule maintenance, or balance energy supply and demand. When the data are reliable and representative, such systems can be faster and more consistent than ad‑hoc human judgement. But when the data are noisy or biased, the recommendations can be wrong in systematic ways, with real‑world consequences for households, companies, and critical infrastructure.
Wild data and brittle models
Modern datasets are increasingly 'wild'. They are scraped from the web, aggregated across sources, and full of quirks: missing values, measurement errors, duplicates, biases, and extreme outliers (rare, high‑impact “black swan” events). A model tuned on tidy historical data can look excellent on paper yet fail once deployed.
This is related to the 'optimizer’s curse' in mathematical optimization: when choosing among many options based on uncertain estimates, the option that looks best on paper often disappoints in practice. In other words, a plan that seems optimal in theory can underperform in reality because it has inadvertently 'overfit' to noise or lucky coincidences in the data.
Planning for reality, not just theory
“Today’s decision algorithms are powerful, but they often assume the world is tidier than it actually is,” says Bart Van Parys. “This project aims to prepare decision‑making methods for an era in which data are much wilder. Robust optimization methods have tried to do so by following the maxim ‘plan for the worst, hope for the best.’ Yet in practice they remain fragile when working with wild data. For instance, they plan for the worst only insofar as the data contain no errors or significant outliers. Robust optimization has also been criticized for being impractically conservative. In typical scenarios where the worst case does not occur, they are overly cautious.”
What the project will do
This project takes on a long-standing challenge to develop decision tools that offer rigorous performance guarantees even when working with wild data, and reduce conservatism by explicitly balancing worst and typical performance. The project brings together ideas from statistics to understand the impact of the challenging aspects of wild data, and computer science to study performance beyond merely the worst case.
Environment and collaboration
“CWI offers a highly conducive environment, as it houses several world‑renowned scholars on topics that closely intersect with my research, such as Peter Grünwald (safe statistics) and Guido Schäfer (algorithms with predictions),” says Van Parys. “The Netherlands punches far above its weight in mathematical optimization and data science. This project will further strengthen its standing in the strategically important task of developing mathematical algorithms that actually improve our lives.”
About Bart Van Parys
Bart Van Parys joined the Stochastics group of CWI as a senior researcher in 2024, relocating from MIT where he was a junior faculty member. His research focuses on distributionally robust optimization and data‑driven decision‑making. He has pioneered techniques in robust optimization and published in leading journals including Management Science, Annals of Statistics, and Mathematical Programming.
About the Vidi grant
An NWO Vidi grant is a funding instrument of the Dutch Research Council (NWO) within the NWO Talent Programme. The Vidi grant funds innovative scientific research and enables researchers to establish a research group and develop an innovative line of inquiry.
