Meta-Cognitive Reinforcement Learning with Self-Doubt and Recovery

TL;DR

This paper introduces a meta-cognitive reinforcement learning framework that enables agents to assess and regulate their learning process based on internal reliability signals, improving robustness against reward corruption.

Contribution

It proposes a novel meta-trust variable driven by VPES to modulate learning and enable recovery, enhancing robustness in noisy environments.

Findings

Higher average returns in reward-corrupted benchmarks

Significant reduction in late-stage training failures

Effective self-regulation of learning process

Abstract

Robust reinforcement learning methods typically focus on suppressing unreliable experiences or corrupted rewards, but they lack the ability to reason about the reliability of their own learning process. As a result, such methods often either overreact to noise by becoming overly conservative or fail catastrophically when uncertainty accumulates. In this work, we propose a meta-cognitive reinforcement learning framework that enables an agent to assess, regulate, and recover its learning behavior based on internally estimated reliability signals. The proposed method introduces a meta-trust variable driven by Value Prediction Error Stability (VPES), which modulates learning dynamics via fail-safe regulation and gradual trust recovery. Experiments on continuous-control benchmarks with reward corruption demonstrate that recovery-enabled meta-cognitive control achieves higher average returns and significantly reduces late-stage training failures compared to strong robustness baselines.