TwinLoop: Simulation-in-the-Loop Digital Twins for Online Multi-Agent Reinforcement Learning

TL;DR

TwinLoop introduces a digital twin framework for online multi-agent reinforcement learning, enabling faster adaptation to changing conditions by simulating and improving policies before updating real agents.

Contribution

The paper presents TwinLoop, a novel simulation-in-the-loop digital twin approach that accelerates policy adaptation in multi-agent systems during context shifts.

Findings

Digital twins improve adaptation speed after context shifts.

TwinLoop reduces online trial-and-error in multi-agent reinforcement learning.

Evaluation shows effectiveness in vehicular edge computing scenarios.

Abstract

Decentralised online learning enables runtime adaptation in cyber-physical multi-agent systems, but when operating conditions change, learned policies often require substantial trial-and-error interaction before recovering performance. To address this, we propose TwinLoop, a simulation-in-the-loop digital twin framework for online multi-agent reinforcement learning. When a context shift occurs, the digital twin is triggered to reconstruct the current system state, initialise from the latest agent policies, and perform accelerated policy improvement with simulation what-if analysis before synchronising updated parameters back to the agents in the physical system. We evaluate TwinLoop in a vehicular edge computing task-offloading scenario with changing workload and infrastructure conditions. The results suggest that digital twins can improve post-shift adaptation efficiency and reduce reliance on costly online trial-and-error.