A step towards understanding why classification helps regression

TL;DR

This paper investigates why combining classification loss with regression improves results, especially for imbalanced data, by formalizing the relationship between the two and validating findings on multiple datasets.

Contribution

It provides a formal explanation for the benefits of adding classification loss to regression, particularly in imbalanced data scenarios, supported by empirical validation.

Findings

Adding classification loss benefits imbalanced regression tasks.

The effect is most pronounced with imbalanced data.

Findings hold across multiple datasets and tasks.

Abstract

A number of computer vision deep regression approaches report improved results when adding a classification loss to the regression loss. Here, we explore why this is useful in practice and when it is beneficial. To do so, we start from precisely controlled dataset variations and data samplings and find that the effect of adding a classification loss is the most pronounced for regression with imbalanced data. We explain these empirical findings by formalizing the relation between the balanced and imbalanced regression losses. Finally, we show that our findings hold on two real imbalanced image datasets for depth estimation (NYUD2-DIR), and age estimation (IMDB-WIKI-DIR), and on the problem of imbalanced video progress prediction (Breakfast). Our main takeaway is: for a regression task, if the data sampling is imbalanced, then add a classification loss.