Evaluating High-Order Predictive Distributions in Deep Learning

TL;DR

This paper investigates the challenges of evaluating high-order predictive distributions in high-dimensional deep learning, proposing dyadic sampling as an efficient solution for distinguishing model uncertainty in complex scenarios.

Contribution

It introduces dyadic sampling, a novel method for assessing predictive distributions in high-dimensional settings, overcoming limitations of existing low-order approaches.

Findings

Dyadic sampling effectively distinguishes agents in high-dimensional data.

The method works for both simple and complex models.

High-order predictive distribution assessment becomes impractical as input dimension increases.

Abstract

Most work on supervised learning research has focused on marginal predictions. In decision problems, joint predictive distributions are essential for good performance. Previous work has developed methods for assessing low-order predictive distributions with inputs sampled i.i.d. from the testing distribution. With low-dimensional inputs, these methods distinguish agents that effectively estimate uncertainty from those that do not. We establish that the predictive distribution order required for such differentiation increases greatly with input dimension, rendering these methods impractical. To accommodate high-dimensional inputs, we introduce \textit{dyadic sampling}, which focuses on predictive distributions associated with random \textit{pairs} of inputs. We demonstrate that this approach efficiently distinguishes agents in high-dimensional examples involving simple logistic regression as well as complex synthetic and empirical data.