Learning Can Converge Stably to the Wrong Belief under Latent Reliability

TL;DR

This paper introduces a framework that uses learning dynamics to infer feedback reliability, helping algorithms avoid converging to incorrect solutions when feedback signals are unreliable or biased.

Contribution

The paper proposes the Monitor-Trust-Regulator (MTR) framework that infers reliability from learning trajectories and adjusts updates accordingly, improving robustness in unreliable feedback scenarios.

Findings

Standard algorithms often converge to incorrect solutions under latent unreliability.

Trust-modulated systems reduce bias accumulation and improve recovery.

Learning dynamics can serve as a source of information about feedback reliability.

Abstract

Learning systems are typically optimized by minimizing loss or maximizing reward, assuming that improvements in these signals reflect progress toward the true objective. However, when feedback reliability is unobservable, this assumption can fail, and learning algorithms may converge stably to incorrect solutions. This failure arises because single-step feedback does not reveal whether an experience is informative or persistently biased. When information is aggregated over learning trajectories, however, systematic differences between reliable and unreliable regimes can emerge. We propose a Monitor-Trust-Regulator (MTR) framework that infers reliability from learning dynamics and modulates updates through a slow-timescale trust variable. Across reinforcement learning and supervised learning settings, standard algorithms exhibit stable optimization behavior while learning incorrect solutions under latent unreliability, whereas trust-modulated systems reduce bias accumulation and improve recovery. These results suggest that learning dynamics are not only optimization traces but also a source of information about feedback reliability.