IMP-MARL: Infrastructure Management Planning via MARL

Pascal Leroy, Pablo G. Morato, Jonathan Pisane, Athanasios Kolios, Damien Ernst.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.
https://hdl.handle.net/2268/304451

Intelligent agents, trained through reinforcement learning (RL), excel in complex decision-making tasks across various fields such as games, autonomous driving, healthcare, and nuclear fusion. Multi-agent reinforcement learning (MARL) has gained attraction, notably in cooperative scenarios. To advance cooperative MARL, benchmark environments such as the StarCraft Multi-Agent Challenge and MaMuJoCo have been developed. While these environments contribute to cooperative MARL methods development, challenges persist in real-world applications.

Infrastructure Management Planning (IMP) is a modern application addressing current societal and environmental concerns. In IMP, agents plan inspections, repairs or retrofits to prevent failures in structures like bridges or wind turbines. The goal is to minimise the risk of system failures by considering the probability of failure and the associated consequences. An overarching representation of an infrastructure management planning (IMP) problem is presented in Figure 1.

Figure 1.  Overarching representation of an infrastructure management planning (IMP) problem.

The system deals with uncertainties in the damage of components by using a probability distribution called damage probability. This distribution changes over time based on a deterioration process and decisions made for each component. These decisions include doing nothing, inspecting, or repairing a component. To help make this complex process more efficient, the IMP-MARL framework is introduced.  It’s an open-source suite of environments where each agent manages a specific component. Agents aim to reduce inspection and maintenance costs while cooperating to minimise system failure risk. The framework is economically beneficial but also contributes to sustainability and societal well-being. For example, preventing the failure of a wind turbine ensures a steady electricity supply.

To evaluate the effectiveness of cooperative methods in IMP problems, the study benchmarks several state-of-the-art methods.  These methods are compared against expert-based heuristic policies in various IMP environments, including those related to offshore wind structural systems. The findings suggest that while cooperative methods can outperform heuristic policies, their performance may vary in different scenarios, highlighting the need for further research in cooperative MARL with many agents. In summary, the contributions of this work include introducing the IMP-MARL framework, conducting extensive benchmarking of cooperative MARL methods, and providing insights for both machine learning and reliability engineering communities.  The study’s source code is publicly available for transparency and to foster future comparisons.