Fair Comparison: Quantifying Variance in Resultsfor Fine-grained Visual Categorization

TL;DR

This paper investigates the variability in fine-grained visual categorization results, emphasizing the importance of considering variance and per-class performance alongside average accuracy for more reliable model evaluation.

Contribution

It quantifies the extent of performance variance in FGVC models and highlights the need for comprehensive metrics beyond average accuracy.

Findings

Significant variation exists across models and class distributions.

Per-class performance varies notably even among similar models.

Certain techniques can reduce variance in FGVC results.

Abstract

For the task of image classification, researchers work arduously to develop the next state-of-the-art (SOTA) model, each bench-marking their own performance against that of their predecessors and of their peers. Unfortunately, the metric used most frequently to describe a model's performance, average categorization accuracy, is often used in isolation. As the number of classes increases, such as in fine-grained visual categorization (FGVC), the amount of information conveyed by average accuracy alone dwindles. While its most glaring weakness is its failure to describe the model's performance on a class-by-class basis, average accuracy also fails to describe how performance may vary from one trained model of the same architecture, on the same dataset, to another (both averaged across all categories and at the per-class level). We first demonstrate the magnitude of these variations across models and across class distributions based on attributes of the data, comparing results on different visual domains and different per-class image distributions, including long-tailed distributions and few-shot subsets. We then analyze the impact various FGVC methods have on overall and per-class variance. From this analysis, we both highlight the importance of reporting and comparing methods based on information beyond overall accuracy, as well as point out techniques that mitigate variance in FGVC results.