A Generalization Theory based on Independent and Task-Identically Distributed Assumption

TL;DR

This paper introduces a new generalization theory based on the ITID assumption that incorporates task properties, leading to better understanding and improvement of model generalization in image classification.

Contribution

It proposes the ITID assumption for generalization analysis, derives a new bound emphasizing hypothesis invariance, and develops a practical invariance enhancement algorithm.

Findings

The ITID assumption is validated as reasonable through experiments.

The invariance enhancement algorithm improves generalization performance.

The new theory better explains generalization phenomena in practical tasks.

Abstract

Existing generalization theories analyze the generalization performance mainly based on the model complexity and training process. The ignorance of the task properties, which results from the widely used IID assumption, makes these theories fail to interpret many generalization phenomena or guide practical learning tasks. In this paper, we propose a new Independent and Task-Identically Distributed (ITID) assumption, to consider the task properties into the data generating process. The derived generalization bound based on the ITID assumption identifies the significance of hypothesis invariance in guaranteeing generalization performance. Based on the new bound, we introduce a practical invariance enhancement algorithm from the perspective of modifying data distributions. Finally, we verify the algorithm and theorems in the context of image classification task on both toy and real-world datasets. The experimental results demonstrate the reasonableness of the ITID assumption and the effectiveness of new generalization theory in improving practical generalization performance.