Representation Convergence: Mutual Distillation is Secretly a Form of Regularization

TL;DR

This paper reveals that mutual distillation in reinforcement learning acts as a regularizer, improving policy robustness and generalization by fostering invariant representations, supported by theoretical proofs and empirical evidence.

Contribution

It provides the first theoretical proof linking policy robustness to generalization and empirically shows mutual distillation promotes invariant representations.

Findings

Mutual distillation enhances policy robustness.

Invariant representations emerge spontaneously.

Improved generalization performance is observed.

Abstract

In this paper, we argue that mutual distillation between reinforcement learning policies serves as an implicit regularization, preventing them from overfitting to irrelevant features. We highlight two separate contributions: (i) Theoretically, for the first time, we prove that enhancing the policy robustness to irrelevant features leads to improved generalization performance. (ii) Empirically, we demonstrate that mutual distillation between policies contributes to such robustness, enabling the spontaneous emergence of invariant representations over pixel inputs. Ultimately, we do not claim to achieve state-of-the-art performance but rather focus on uncovering the underlying principles of generalization and deepening our understanding of its mechanisms.