Understanding the Dependence of Perception Model Competency on Regions in an Image

TL;DR

This paper investigates methods to identify regions in images that cause perception models to perform poorly, aiming to improve model interpretability and decision-making in real-world applications.

Contribution

It introduces five novel techniques for pinpointing image regions linked to low model competency, with a focus on gradients and reconstruction loss methods.

Findings

Competency gradients and reconstruction loss effectively identify low-competency regions.

These methods are computationally efficient and accurate in detecting unfamiliar or unseen objects.

The approaches facilitate better understanding of model failures in complex environments.

Abstract

While deep neural network (DNN)-based perception models are useful for many applications, these models are black boxes and their outputs are not yet well understood. To confidently enable a real-world, decision-making system to utilize such a perception model without human intervention, we must enable the system to reason about the perception model's level of competency and respond appropriately when the model is incompetent. In order for the system to make an intelligent decision about the appropriate action when the model is incompetent, it would be useful for the system to understand why the model is incompetent. We explore five novel methods for identifying regions in the input image contributing to low model competency, which we refer to as image cropping, segment masking, pixel perturbation, competency gradients, and reconstruction loss. We assess the ability of these five methods to identify unfamiliar objects, recognize regions associated with unseen classes, and identify unexplored areas in an environment. We find that the competency gradients and reconstruction loss methods show great promise in identifying regions associated with low model competency, particularly when aspects of the image that are unfamiliar to the perception model are causing this reduction in competency. Both of these methods boast low computation times and high levels of accuracy in detecting image regions that are unfamiliar to the model, allowing them to provide potential utility in decision-making pipelines. The code for reproducing our methods and results is available on GitHub: https://github.com/sarapohland/explainable-competency.