Neural Multivariate Regression: Qualitative Insights from the Unconstrained Feature Model

TL;DR

This paper uses the Unconstrained Feature Model to analyze neural multivariate regression, revealing how multi-task learning and target normalization strategies influence training performance in deep neural networks.

Contribution

It provides the first qualitative insights into neural multivariate regression using the UFM, demonstrating how multi-task models and target normalization affect training loss.

Findings

Multi-task models outperform single-task models with similar regularization.

Whitening and normalizing targets reduce training MSE when variance is below one.

Empirical results confirm UFM predictions about training performance improvements.

Abstract

The Unconstrained Feature Model (UFM) is a mathematical framework that enables closed-form approximations for minimal training loss and related performance measures in deep neural networks (DNNs). This paper leverages the UFM to provide qualitative insights into neural multivariate regression, a critical task in imitation learning, robotics, and reinforcement learning. Specifically, we address two key questions: (1) How do multi-task models compare to multiple single-task models in terms of training performance? (2) Can whitening and normalizing regression targets improve training performance? The UFM theory predicts that multi-task models achieve strictly smaller training MSE than multiple single-task models when the same or stronger regularization is applied to the latter, and our empirical results confirm these findings. Regarding whitening and normalizing regression targets, the UFM theory predicts that they reduce training MSE when the average variance across the target dimensions is less than one, and our empirical results once again confirm these findings. These findings highlight the UFM as a powerful framework for deriving actionable insights into DNN design and data pre-processing strategies.