Binary Hypothesis Testing for Softmax Models and Leverage Score Models

TL;DR

This paper investigates the sample complexity of binary hypothesis testing for softmax and leverage score models, establishing asymptotic bounds and drawing analogies between these models in machine learning and linear algebra.

Contribution

It introduces the first analysis of sample complexity for hypothesis testing in softmax models and relates these results to leverage score models, providing new theoretical insights.

Findings

Sample complexity is asymptotically O(ε^{-2}) for softmax models.

Similar bounds are derived for leverage score models.

Establishes an analogy between softmax and leverage score models.

Abstract

Softmax distributions are widely used in machine learning, including Large Language Models (LLMs), where the attention unit uses softmax distributions. We abstract the attention unit as the softmax model, where given a vector input, the model produces an output drawn from the softmax distribution (which depends on the vector input). We consider the fundamental problem of binary hypothesis testing in the setting of softmax models. That is, given an unknown softmax model, which is known to be one of the two given softmax models, how many queries are needed to determine which one is the truth? We show that the sample complexity is asymptotically $O(\epsilon^{-2})$ where $\epsilon$ is a certain distance between the parameters of the models. Furthermore, we draw an analogy between the softmax model and the leverage score model, an important tool for algorithm design in linear algebra and graph theory. The leverage score model, on a high level, is a model which, given a vector input, produces an output drawn from a distribution dependent on the input. We obtain similar results for the binary hypothesis testing problem for leverage score models.