Inverting and Understanding Object Detectors

TL;DR

This paper introduces an inversion-based method to visualize and interpret modern object detectors, revealing their reliance on different features, learned object co-occurrences, and size-dependent cues, thereby enhancing understanding of their decision processes.

Contribution

It develops a novel layout inversion technique to generate synthetic images recognized by detectors, providing new insights into detector behavior and feature reliance.

Findings

Detectors use different features for classification and regression.

Detectors learn canonical co-occurrence motifs.

Size variation affects visual cues used by detectors.

Abstract

As a core problem in computer vision, the performance of object detection has improved drastically in the past few years. Despite their impressive performance, object detectors suffer from a lack of interpretability. Visualization techniques have been developed and widely applied to introspect the decisions made by other kinds of deep learning models; however, visualizing object detectors has been underexplored. In this paper, we propose using inversion as a primary tool to understand modern object detectors and develop an optimization-based approach to layout inversion, allowing us to generate synthetic images recognized by trained detectors as containing a desired configuration of objects. We reveal intriguing properties of detectors by applying our layout inversion technique to a variety of modern object detectors, and further investigate them via validation experiments: they rely on qualitatively different features for classification and regression; they learn canonical motifs of commonly co-occurring objects; they use diff erent visual cues to recognize objects of varying sizes. We hope our insights can help practitioners improve object detectors.