Quantifying Uncertainty in Error Consistency: Towards Reliable Behavioral Comparison of Classifiers

TL;DR

This paper enhances the measurement of error consistency (EC) in ML model benchmarking by introducing confidence intervals via bootstrapping and a model relating EC to response copying, enabling more reliable behavioral comparisons.

Contribution

It proposes a method to compute confidence intervals for EC and introduces a model linking EC to response copying, improving the reliability of behavioral benchmarking in ML.

Findings

Many reported differences between deep vision models and humans are statistically insignificant.

The new methodology allows for more conclusive and reliable behavioral comparisons.

Researchers can now design experiments with sufficient power to detect true behavioral differences.

Abstract

Benchmarking models is a key factor for the rapid progress in machine learning (ML) research. Thus, further progress depends on improving benchmarking metrics. A standard metric to measure the behavioral alignment between ML models and human observers is error consistency (EC). EC allows for more fine-grained comparisons of behavior than other metrics such as accuracy, and has been used in the influential Brain-Score benchmark to rank different DNNs by their behavioral consistency with humans. Previously, EC values have been reported without confidence intervals. However, empirically measured EC values are typically noisy -- thus, without confidence intervals, valid benchmarking conclusions are problematic. Here we improve on standard EC in two ways: First, we show how to obtain confidence intervals for EC using a bootstrapping technique, allowing us to derive significance tests for EC. Second, we propose a new computational model relating the EC between two classifiers to the implicit probability that one of them copies responses from the other. This view of EC allows us to give practical guidance to scientists regarding the number of trials required for sufficiently powerful, conclusive experiments. Finally, we use our methodology to revisit popular NeuroAI-results. We find that while the general trend of behavioral differences between humans and machines holds up to scrutiny, many reported differences between deep vision models are statistically insignificant. Our methodology enables researchers to design adequately powered experiments that can reliably detect behavioral differences between models, providing a foundation for more rigorous benchmarking of behavioral alignment.