Optimal sequential decision-making for error propagation mitigation in digital twins

TL;DR

This paper develops and compares optimal decision-making frameworks, including MDP and POMDP, for mitigating error propagation in digital twins, validated through simulations and reinforcement learning benchmarks.

Contribution

It introduces a novel MDP/POMDP-based approach for error mitigation in digital twins, extending prior HMM-based regime inference with decision process modeling.

Findings

MDP policy achieves highest cumulative reward and nominal operation time.

POMDP recovers about 95% of MDP performance under observation noise.

Reinforcement learning algorithms effectively learn policies without explicit models.

Abstract

Here, we explore the problem of error propagation mitigation in modular digital twins as a sequential decision process. Building on a companion study that used a Hidden Markov Model (HMM) to infer latent error regimes from surrogate-physics residuals, we develop a Markov Decision Process (MDP) in which the inferred regimes serve as states, corrective interventions serve as actions, and a scalar reward that takes into consideration the cost-benefit tradeoff between system fidelity and maintenance expense. The baseline transition matrix is extracted from the HMM-learned parameters. We then extend the formulation to a Partially Observable MDP (POMDP) that accounts for the imperfect nature of regime classification by maintaining a belief distribution updated via Bayesian filtering, with the HMM confusion matrix serving as the observation model. Both formulations are solved via dynamic programming and validated through Gillespie stochastic simulation. We then benchmark two model-free reinforcement learning algorithms, Q-learning and REINFORCE, to assess whether effective policies can be learned without explicit model knowledge. A systematic comparison of different intervention policies demonstrates that the MDP policy achieves the highest cumulative reward and fraction of time in nominal operation, while the POMDP recovers approximately 95\% of MDP performance under realistic observation noise. Sensitivity analyses across observation quality, repair probability, and discount factor confirm the robustness of these conclusions, and the major gaps in the policy hierarchy are statistically significant at $p < 0.001$. The gap between MDP and POMDP performance quantifies the value of information providing a principled criterion for investing in improved classification accuracy.