Efficient estimation of the ANOVA mean dimension, with an application to neural net classification

TL;DR

This paper compares methods for efficiently estimating the mean dimension of high-dimensional functions, demonstrating their application to neural network classifiers and identifying the most effective approach for different function types.

Contribution

It introduces and compares leave-one-out methods for estimating the mean dimension, highlighting the most efficient algorithms for additive, multiplicative, and neural network functions.

Findings

Winding stairs is most efficient for neural network mean dimension estimation.

Inputs to the softmax layer have mean dimensions between 1.35 and 2.0.

Method performance varies with function type and distribution kurtosis.

Abstract

The mean dimension of a black box function of $d$ variables is a convenient way to summarize the extent to which it is dominated by high or low order interactions. It is expressed in terms of $2^d-1$ variance components but it can be written as the sum of $d$ Sobol' indices that can be estimated by leave one out methods. We compare the variance of these leave one out methods: a Gibbs sampler called winding stairs, a radial sampler that changes each variable one at a time from a baseline, and a naive sampler that never reuses function evaluations and so costs about double the other methods. For an additive function the radial and winding stairs are most efficient. For a multiplicative function the naive method can easily be most efficient if the factors have high kurtosis. As an illustration we consider the mean dimension of a neural network classifier of digits from the MNIST data set. The classifier is a function of $784$ pixels. For that problem, winding stairs is the best algorithm. We find that inputs to the final softmax layer have mean dimensions ranging from $1.35$ to $2.0$.