Workshop on Modern Applications of Control Theory and Reinforcement Learning

Diederik M. Roijers is the academic liaison for AI research for the City of Amsterdam, and a member of the AI Research team. His team aims to improve the lives of the citizens of Amsterdam through AI research.

Next to his work at the City of Amsterdam, he is a senior researcher at the AI lab at the Vrije Universiteit Brussel (VUB), and currently supervising PhD students.

His main research interests are urban AI, reinforcement learning, planning, multi-agent systems, and multi-objective decision making. Click for more information of his tutorials or book for an introduction to multi-objective models and methods for multi-agent systems, RL and planning, or his publication page for information about his latest research. Other interests his are game theory, machine learning, robotics, e-tutoring systems, and education.

He obtained his PhD at the University of Amsterdam under the supervision of Shimon Whiteson and Frans Oliehoek. Click here for more information or his PhD thesis for further details. After his PhD, he worked on social robotics in the TERESA project, at the Department of Computer Science of the University of Oxford; as an FWO Postdoctoral Fellow on Multi-objective Reinforcement learning with Guarantees at the Vrije Universiteit Brussel; as assistant professor at the Vrije Universiteit Amsterdam; and as senior lecturer and researcher at the Institute of ICT and the Microsystems Technology Research Group where he worked on efficient AI for microproccessing systems and sensor data.

Elena Rovenskaya is the IIASA Advancing Systems Analysis (ASA) Program Director. She is also a research scholar at the Optimal Control Department of the Faculty of Computational Mathematics and Cybernetics, Lomonosov Moscow State University, Russia (on-leave). Her scientific interests lie in the fields of optimization, decision science, and mathematical modeling of complex socio-environmental systems.

Dr. Rovenskaya graduated in 2003 from the Faculty of Physics, Lomonosov Moscow State University, Russia. She received her PhD in 2006 at the Faculty of Computational Mathematics and Cybernetics, Lomonosov Moscow State University, Russia. In her PhD dissertation, Dr. Rovenskaya developed a new numerical method for solving a broad class of non-convex optimization problems.

In 2005, Dr. Rovenskaya participated in the IIASA Young Scientists Summer Program (YSSP). She continued to collaborate with the former Dynamic Systems (DYN) Program from 2006 to 2010, and later, from 2011 to 2012 with its successor, the Advanced Systems Analysis (ASA) Program at IIASA. From 2013 to 2020, Dr. Rovenskaya served as ASA Program Director and from 2019 to 2020, she was also appointed in the capacity of Acting IIASA Evolution and Ecology Program Director.

Dr. Rovenskaya was appointed Advancing Systems Analysis (ASA) Program Director from January 2021 as the institute moved to a new program structure. Currently, the new ASA Program includes 85+ scientists and aims to identify, develop, and deploy new systems-analytical methods, tools, and data that address the most pressing global sustainability challenges with greater agility, and help find solutions to those challenges that are both realistic and appropriate.

Herke van Hoof is associate professor at the University of Amsterdam. Before that, Herke van Hoof was a postdoc at McGill University in Montreal, Canada, where he worked with Professors Joelle Pineau, Dave Meger, and Gregory Dudek. He obtained his PhD at TU Darmstadt, Germany, under the supervision of Professor Jan Peters, where he graduated in November 2016. Herke got his bachelor and master degrees in Artificial Intelligence at the University of Groningen in the Netherlands. His group works on various aspects of modular reinforcement learning. To address the low data efficiency of reinforcement learning from scratch, we investigate topics like using (symbolic) prior knowledge, modularity, and transferring knowledge between tasks.

Talk details

Title: Reinforcement learning for real-world network infrastructure

Most current reinforcement learning research is done in the context of games and other simulated domains. However, leveraging the impressive results from these domains to make real-world impact requires tackling additional challenges. These challenges include handling structured state- and action spaces, providing safe, robust and scalable solutions even from modest datasets, and explicitly considering how a RL agent will interact with human collaborators. The AI4REALNET project focuses on such challenges in the context of sequential decision making in real-world critical infrastructure, such as power grids, train scheduling, and air traffic management. In this talk, I will discuss how the AI4REALNET project approaches these issues, and technical advances in these areas by our team.

Marcel van Gerven is Professor of Artificial Cognitive Systems and Principal Investigator in the Department of Machine Learning and Neural Computing of the Donders Institute for Brain, Cognition and Behaviour. Prof. van Gerven is an expert in machine learning and neuromorphic computing. His work ranges from understanding the computational mechanisms of learning, inference and control in natural and artificial systems to the development of new AI technology with applications in e.g. neuroscience, neurotechnology, healthcare and smart industry. Prof. van Gerven is recipient of several grants at the intersection of AI and neuroscience, such as Dutch Vidi, Crossover, Perspective and Gravitation grants as well as EU HBP and FET grants. He also received the Radboud Science Award for his scientific work. Prof. van Gerven is cofounder of Radboud AI and directs one of the European ELLIS units as part of the European Excellence Network in Machine Learning. He also contributes to the Healthy Data program, which aims to make AI accessible in healthcare, and is director of an Innovation Centre in AI for semiconductor manufacturing. Through his work, he aims to bridge the gap between natural and artificial intelligence and contribute to the development of sustainable AI solutions that make a positive impact in science, industry and society.

Marta Kwiatkowska is Professor of Computing Systems and Fellow of Trinity College, University of Oxford, and Associate Head of MPLS. Prior to this she was Professor in the School of Computer Science at the University of Birmingham, Lecturer at the University of Leicester and Assistant Professor at the Jagiellonian University in Cracow, Poland. She holds a BSc/MSc in Computer Science from the Jagiellonian University, MA from Oxford and a PhD from the University of Leicester. In 2014 she was awarded an honorary doctorate from KTH Royal Institute of Technology in Stockholm.

Marta Kwiatkowska spearheaded the development of probabilistic and quantitative methods in verification on the international scene and is currently working on safety and robustness for machine learning and AI. She led the development of the PRISM model checker, the leading software tool in the area and widely used for research and teaching and winner of the HVC 2016 Award. Applications of probabilistic model checking have spanned communication and security protocols, nanotechnology designs, power management, game theory, planning and systems biology, with genuine flaws found and corrected in real-world protocols. Kwiatkowska gave the Milner Lecture in 2012 in recognition of "excellent and original theoretical work which has a perceived significance for practical computing". She is the first female winner of the 2018 Royal Society Milner Award and Lecture, see her lecture here, and won the BCS Lovelace Medal in 2019. Marta Kwiatkowska was invited to give keynotes at the LICS 2003, ESEC/FSE 2007 and 2019, ETAPS/FASE 2011, ATVA 2013, ICALP 2016, CAV 2017, CONCUR 2019 and UbiComp 2019 conferences.

She is a Fellow of the Royal Society, Fellow of ACM, member of Academia Europea, Fellow of EATCS, Fellow of the BCS and Fellow of Polish Society of Arts & Sciences Abroad. She serves on editorial boards of several journals, including Information and Computation, Formal Methods in System Design, Logical Methods in Computer Science, Science of Computer Programming and Royal Society Open Science journal. Kwiatkowska's research has been supported by grant funding from EPSRC, ERC, EU, DARPA and Microsoft Research Cambridge, including two prestigious ERC Advanced Grants, VERIWARE ("From software verification to everyware verification") and FUN2MODEL ("From FUNction-based TO MOdel-based automated probabilistic reasoning for DEep Learning"), and the EPSRC Programme Grant on Mobile Autonomy.

Sander Bohté developed computational models to understand information processing in neural networks. He strongly believes that neural networks and computational neuroscience models should "compute"; the challenge is to develop insights from neuroscience into usefully computing neural networks, and to bring machine learning insights into models of how neurons in the brain compute. Sander is particularly focused on continuous-time information processing, where time is an explicit dimension of the problem domain. This includes networks of spiking neurons, predictive coding, interactive neural cognition, supervised neural learning, and (biologically plausible) deep reinforcement learning methods. Sander is also a part-time full professor of Computational Neuroscience at the University of Amsterdam, with the Swammerdam Institute of Life Sciences, and an honorary full-professor of Bio-inspired Neural Networks at the Rijksuniversiteit Groningen.

Sander Keemink is fascinated by how neurons and networks (in brains and machines) encode information and perform computations. While it is now possible to train and build highly effective networks, this does not mean we understand them. This hampers our interpretation of what neural networks are really doing, which is not something desirable for a technology seeing such increasingly widespread use. His research focus is therefore always to find the core underlying principles of network function (currently mainly in spiking networks, but also more generally).

Timm Faulwasser has studied Engineering Cybernetics at the University of Stuttgart, with majors in systems and control and philosophy. From 2008 until 2012 he was a member of the International Max Planck Research School for Analysis, Design and Optimization in Chemical and Biochemical Process Engineering Magdeburg. In 2012 he obtained his PhD from the Department of Electrical Engineering and Information Engineering, Otto-von-Guericke-University Magdeburg, Germany. From 2013 to 2016 he was with the Laboratoire d’Automatique, École Polytechnique Fédérale de Lausanne (EPFL), Switzerland. He was leading the Optimization and Control Group at the Institute for Automation and Applied Informatics at Karlsruhe Institute of Technology (KIT) in 2015-2019. From 2019 to 2024 he held the professorship for energy efficiency in the Department of Electrical Engineering and Information Technology, TU Dortmund University, Germany. Since April 2024 he leads the Institute of Control Systems at Hamburg University of Technology, 21073 Hamburg, Germany. Currently, he serves as associate editor for the IEEE Transactions on Automatic Control, the IEEE Control System Letters, as well as Mathematics of Control Systems and Signals. Dr. Faulwasser received the 2021-2023 Automatica Paper Prize.  His research interests include optimal and predictive control of nonlinear systems and networks.

Wolfram Barfuss is the Argelander (Tenure-Track) Assistant Professor of the Transdisciplinary Research Area (TRA) Sustainable Futures and based at the Center for Development Research (ZEF) at the University of Bonn, Germany. He obtained his doctoral degree in theoretical physics of complex systems from the Potsdam Institute for Climate Impact Research and the Humboldt University Berlin (2019). Before joining the University of Bonn in 2023, he was a postdoctoral scientist at the Tübingen AI Center at the University of Tübingen (2021-2023), the School of Mathematics at the University of Leeds (2020-2021), and the Max Planck Insitute for Mathematics in the Sciences in Leipzig (2019-2020). His research centers around the question, "Are we smart enough for the good life?" To answer this question, the BarfussLab develops formal models of collective reinforcement learning dynamics to better understand how, in complex environments, individual decisions become collective action for a sustainable future.

Talk details

Title: Collective Reinforcement Learning Dynamics for Sustainability Economics

Cooperation at scale is critical for achieving a sustainable future for humanity. However, achieving collective, cooperative behavior—in which intelligent actors in complex environments jointly improve their well-being—remains poorly understood. Complex systems science (CSS) provides a rich understanding of collective phenomena, the evolution of cooperation, and the institutions that can sustain both. Yet, much of the theory in this area fails to fully consider individual-level complexity and environmental context—mainly for the sake of tractability and because it has not been clear how to do so rigorously. These elements are well captured in multiagent reinforcement learning (MARL), which has recently focused on cooperative (artificial) intelligence. However, typical MARL simulations can be computationally expensive and challenging to interpret. In this presentation, I propose that bridging CSS and MARL offers new directions. By investigating the non-linear dynamics of collective reinforcement learning, we can better understand how, in complex environments, individual decisions become collective action for a sustainable future.

09:15 - 09:45: Registration and tea/coffee

09:45 - 10:00: Welcome

10:00 - 11:00: Marta Kwiatkowska

11:00 - 11:30: Break

11:30 - 12:10: Diederik Roijers - A Plea for User-Centred RL

12:10 - 12:50: Herke van Hoof - Reinforcement learning for real-world network infrastructure

12:50 - 14:00: Lunch and Poster session

14:00 - 15:00: Elena Rovenskaya

15:00 - 15:30: Break

15:30 - 17:00: Panel Discussion with Workshop Speakers

17:00 - 18:30: Discussion over drinks and bites

09:30 - 10:00: Registration and tea/coffee

10:00 - 11:00: Timm Faulwasser

11:00 - 11:30: Break

11:30 - 12:10: Marcel van Gerven

12:10 - 13:30: Lunch and Poster session

13:30 - 14:30: Wolfram Barfuss - Collective Reinforcement Learning Dynamics for Sustainability Economics

14:30 - 15:00: Break

15:00 - 15:40: Sander Keemink

15:40 - 16:20: Sander Bohte

16:20 - 17:30: Discussion over drinks