Task Shift: From Classification to Regression in Overparameterized Linear Models

TL;DR

This paper explores how overparameterized linear models can adapt from classification training to regression testing, revealing conditions under which task shift is possible or impossible, and proposing a simple postprocessing method for effective transfer.

Contribution

It provides a theoretical analysis of task shift from classification to regression in linear models, including impossibility results and a novel asymptotic recovery algorithm.

Findings

Task shift is impossible without regression data in zero-shot setting.

A simple postprocessing algorithm asymptotically recovers the true predictor with limited data.

Minimum-norm interpolators exhibit structured attenuation enabling transfer under certain conditions.

Abstract

Modern machine learning methods have recently demonstrated remarkable capability to generalize under task shift, where latent knowledge is transferred to a different, often more difficult, task under a similar data distribution. We investigate this phenomenon in an overparameterized linear regression setting where the task shifts from classification during training to regression during evaluation. In the zero-shot case, wherein no regression data is available, we prove that task shift is impossible in both sparse signal and random signal models for any Gaussian covariate distribution. In the few-shot case, wherein limited regression data is available, we propose a simple postprocessing algorithm which asymptotically recovers the ground-truth predictor. Our analysis leverages a fine-grained characterization of individual parameters arising from minimum-norm interpolation which may be of independent interest. Our results show that while minimum-norm interpolators for classification cannot transfer to regression a priori, they experience surprisingly structured attenuation which enables successful task shift with limited additional data.