Do Not Imitate, Reinforce: Iterative Classification via Belief Refinement

TL;DR

The paper introduces Reinforced Iterative Classification (RIC), a reinforcement learning-based method that iteratively refines predictions, offering adaptive computation, improved calibration, and matching accuracy of traditional supervised models.

Contribution

RIC replaces single-pass imitation with RL, enabling iterative prediction refinement, adaptive computation, and better calibration in classification tasks.

Findings

RIC matches supervised accuracy on image benchmarks.

RIC provides an anytime classifier with adaptive computation.

RIC improves calibration over traditional models.

Abstract

Standard supervised classification trains models to imitate the exact labels provided by a perfect oracle. This imitation happens in a single pass, restricting the model to a fixed compute budget even when inputs vary in complexity. Moreover, the rigid training objective forces the model to express absolute certainty on its training data, resulting in overconfident predictions during evaluation. We propose Reinforced Iterative Classification (RIC), which replaces the imitative objective with Reinforcement Learning (RL). RIC deploys a recurrent agent that iteratively updates a predictive distribution over classes, receiving reward for stepwise improvement in prediction quality. The value function provides a natural halting criterion by estimating the remaining scope for improvement. We prove that the iterative formulation recovers the same optimal predictions as cross-entropy while yielding an anytime classifier. On image classification benchmarks, RIC matches the accuracy of supervised baselines with improved calibration and learns to allocate computation adaptively across inputs.