Bounded-Rationality, Hedging, and Generalization

TL;DR

This paper models learning as a bounded-rational decision process, analyzing how learners balance fitting data and hedging against distortion, with implications for understanding generalization.

Contribution

It introduces a framework linking bounded rationality, information geometry, and generalization, providing methods to recover hedging behavior from black-box responses.

Findings

Derived tradeoff curves between training loss and sample dependence.

Showed how to recover hedge and curves from black-box responses.

Connected regularizer geometry to information-theoretic measures like KL.

Abstract

A learner does not only fit data; it also determines how strongly the training sample may shape its output and how much distortion it can hedge. We study this relation as a bounded-rational decision problem whose primitive object is the induced channel from samples to outputs. The learner's response law determines which changes in this channel are cheap or costly, and therefore induces both a lower tradeoff curve between training loss and sample dependence and a matched upper certificate curve. When the response law is represented by an $f$-divergence regularizer, these curves live in the regularizer's native information geometry, with KL as the special case corresponding to Shannon mutual information. We show how the hedge and the two curves can be recovered from black-box behavior by observing responses to scaled losses and local loss perturbations. In learning, population loss is empirical loss plus the distortion induced by the particular training sample. The recovered hedge gives a practical certificate when it covers that distortion. Thus generalization is treated as a testable hedging property of the learner's own response law.