On Guiding Visual Attention with Language Specification

TL;DR

This paper proposes a method to guide visual attention in image classification using language specifications, improving accuracy and fairness by constraining attention to task-relevant features, especially in biased or noisy datasets.

Contribution

It introduces a novel approach to supervise spatial attention with language grounding from pretrained models, enhancing fine-grained classification and fairness in noisy data environments.

Findings

3-15% improvement in worst-group accuracy

41-45% relative fairness metric improvements

Effective in biased and noisy datasets

Abstract

While real world challenges typically define visual categories with language words or phrases, most visual classification methods define categories with numerical indices. However, the language specification of the classes provides an especially useful prior for biased and noisy datasets, where it can help disambiguate what features are task-relevant. Recently, large-scale multimodal models have been shown to recognize a wide variety of high-level concepts from a language specification even without additional image training data, but they are often unable to distinguish classes for more fine-grained tasks. CNNs, in contrast, can extract subtle image features that are required for fine-grained discrimination, but will overfit to any bias or noise in datasets. Our insight is to use high-level language specification as advice for constraining the classification evidence to task-relevant features, instead of distractors. To do this, we ground task-relevant words or phrases with attention maps from a pretrained large-scale model. We then use this grounding to supervise a classifier's spatial attention away from distracting context. We show that supervising spatial attention in this way improves performance on classification tasks with biased and noisy data, including about 3-15% worst-group accuracy improvements and 41-45% relative improvements on fairness metrics.