Provable Weak-to-Strong Generalization via Benign Overfitting

TL;DR

This paper investigates how a strong student model can achieve successful generalization even when supervised by a weak teacher providing pseudolabels that are nearly random, revealing two distinct phases of learning.

Contribution

It provides a theoretical analysis of weak-to-strong generalization in overparameterized models with Gaussian data, identifying conditions for successful learning despite weak supervision.

Findings

Identifies two asymptotic phases: successful generalization and random guessing.

Proves a tight lower tail inequality for the maximum of correlated Gaussians.

Highlights the importance of logits in multilabel weak supervision.

Abstract

The classic teacher-student model in machine learning posits that a strong teacher supervises a weak student to improve the student's capabilities. We instead consider the inverted situation, where a weak teacher supervises a strong student with imperfect pseudolabels. This paradigm was recently brought forth by Burns et al.'23 and termed \emph{weak-to-strong generalization}. We theoretically investigate weak-to-strong generalization for binary and multilabel classification in a stylized overparameterized spiked covariance model with Gaussian covariates where the weak teacher's pseudolabels are asymptotically like random guessing. Under these assumptions, we provably identify two asymptotic phases of the strong student's generalization after weak supervision: (1) successful generalization and (2) random guessing. Our techniques should eventually extend to weak-to-strong multiclass classification. Towards doing so, we prove a tight lower tail inequality for the maximum of correlated Gaussians, which may be of independent interest. Understanding the multilabel setting reinforces the value of using logits for weak supervision when they are available.