Lost in Context: The Influence of Context on Feature Attribution Methods for Object Recognition

TL;DR

This paper explores how context influences object recognition models by analyzing feature attribution methods, revealing that models rely more on object volume than context, and that context changes impact performance more than perturbations.

Contribution

It introduces a comprehensive analysis of context influence on deep neural networks in object recognition using feature attribution techniques and curated datasets.

Findings

Models focus more on object volume than context volume.

Dependence on context remains stable across modifications.

Context change affects model performance more than perturbations.

Abstract

Contextual information plays a critical role in object recognition models within computer vision, where changes in context can significantly affect accuracy, underscoring models' dependence on contextual cues. This study investigates how context manipulation influences both model accuracy and feature attribution, providing insights into the reliance of object recognition models on contextual information as understood through the lens of feature attribution methods. We employ a range of feature attribution techniques to decipher the reliance of deep neural networks on context in object recognition tasks. Using the ImageNet-9 and our curated ImageNet-CS datasets, we conduct experiments to evaluate the impact of contextual variations, analyzed through feature attribution methods. Our findings reveal several key insights: (a) Correctly classified images predominantly emphasize object volume attribution over context volume attribution. (b) The dependence on context remains relatively stable across different context modifications, irrespective of classification accuracy. (c) Context change exerts a more pronounced effect on model performance than Context perturbations. (d) Surprisingly, context attribution in `no-information' scenarios is non-trivial. Our research moves beyond traditional methods by assessing the implications of broad-level modifications on object recognition, either in the object or its context.