A Novel Data-Dependent Learning Paradigm for Large Hypothesis Classes

TL;DR

This paper introduces a new data-dependent learning paradigm for large hypothesis classes that leverages empirical data more effectively without relying heavily on prior assumptions, enhancing generalization in various learning scenarios.

Contribution

It proposes a novel learning framework that reduces dependence on prior assumptions and improves generalization for large model classes.

Findings

Demonstrates improved generalization under multiple common assumptions

Reduces the need for prior knowledge of assumption parameters

Applicable to diverse learning scenarios such as clustering and contrastive learning

Abstract

We address the general task of learning with a set of candidate models that is too large to have a uniform convergence of empirical estimates to true losses. While the common approach to such challenges is SRM (or regularization) based learning algorithms, we propose a novel learning paradigm that relies on stronger incorporation of empirical data and requires less algorithmic decisions to be based on prior assumptions. We analyze the generalization capabilities of our approach and demonstrate its merits in several common learning assumptions, including similarity of close points, clustering of the domain into highly label-homogeneous regions, Lipschitzness assumptions of the labeling rule, and contrastive learning assumptions. Our approach allows utilizing such assumptions without the need to know their true parameters a priori.