Rate-Distortion Signatures of Generalization and Information Trade-offs

TL;DR

This paper introduces a rate-distortion framework to analyze and compare how biological and artificial vision systems balance accuracy and robustness, revealing fundamental differences in their generalization behaviors.

Contribution

It proposes a novel geometric signature-based approach to characterize and compare the generalization trade-offs of different vision systems using rate-distortion theory.

Findings

Humans exhibit smoother, more flexible accuracy-robustness trade-offs.

Deep networks operate in steeper, more brittle regimes even at similar accuracy levels.

Robustness training shifts the rate-distortion signatures, but not always towards human-like behavior.

Abstract

Generalization to novel visual conditions remains a central challenge for both human and machine vision, yet standard robustness metrics offer limited insight into how systems trade accuracy for robustness. We introduce a rate-distortion-theoretic framework that treats stimulus-response behavior as an effective communication channel, derives rate-distortion (RD) frontiers from confusion matrices, and summarizes each system with two interpretable geometric signatures - slope ($\beta$) and curvature ($\kappa$) - which capture the marginal cost and abruptness of accuracy-robustness trade-offs. Applying this framework to human psychophysics and 18 deep vision models under controlled image perturbations, we compare generalization geometry across model architectures and training regimes. We find that both biological and artificial systems follow a common lossy-compression principle but occupy systematically different regions of RD space. In particular, humans exhibit smoother, more flexible trade-offs, whereas modern deep networks operate in steeper and more brittle regimes even at matched accuracy. Across training regimes, robustness training induces systematic but dissociable shifts in beta/kappa, revealing cases where improved robustness or accuracy does not translate into more human-like generalization geometry. These results demonstrate that RD geometry provides a compact, model-agnostic lens for comparing generalization behavior across systems beyond standard accuracy-based metrics.