Multi-Observation Regression

TL;DR

This paper investigates multi-observation loss functions for regression, proposing algorithms with statistical guarantees, evaluating their empirical performance, and highlighting challenges and benefits in different data dimensionalities.

Contribution

It introduces four algorithms for multi-observation regression, providing theoretical guarantees and empirical evaluations, and compares their effectiveness to traditional single-observation methods.

Findings

Algorithms with statistical guarantees outperform others in low dimensions.

Empirical results show practical viability of proposed algorithms.

Lower bounds reveal intrinsic difficulty in high-dimensional settings.

Abstract

Recent work introduced loss functions which measure the error of a prediction based on multiple simultaneous observations or outcomes. In this paper, we explore the theoretical and practical questions that arise when using such multi-observation losses for regression on data sets of $(x,y)$ pairs. When a loss depends on only one observation, the average empirical loss decomposes by applying the loss to each pair, but for the multi-observation case, empirical loss is not even well-defined, and the possibility of statistical guarantees is unclear without several $(x,y)$ pairs with exactly the same $x$ value. We propose four algorithms formalizing the concept of empirical risk minimization for this problem, two of which have statistical guarantees in settings allowing both slow and fast convergence rates, but which are out-performed empirically by the other two. Empirical results demonstrate practicality of these algorithms in low-dimensional settings, while lower bounds demonstrate intrinsic difficulty in higher dimensions. Finally, we demonstrate the potential benefit of the algorithms over natural baselines that use traditional single-observation losses via both lower bounds and simulations.