Are These the Same Apple? Comparing Images Based on Object Intrinsics

TL;DR

This paper introduces a new image similarity metric based on intrinsic object properties, addressing the challenge of recognizing objects under varying extrinsic conditions across diverse categories.

Contribution

It extends re-identification techniques to general objects, proposing a benchmark dataset and demonstrating that combining deep features with foreground filtering effectively measures intrinsic similarity.

Findings

Deep features with foreground filtering outperform existing methods

The CUTE dataset enables benchmarking of intrinsic object similarity

Proposed approach improves re-identification and object recognition tasks

Abstract

The human visual system can effortlessly recognize an object under different extrinsic factors such as lighting, object poses, and background, yet current computer vision systems often struggle with these variations. An important step to understanding and improving artificial vision systems is to measure image similarity purely based on intrinsic object properties that define object identity. This problem has been studied in the computer vision literature as re-identification, though mostly restricted to specific object categories such as people and cars. We propose to extend it to general object categories, exploring an image similarity metric based on object intrinsics. To benchmark such measurements, we collect the Common paired objects Under differenT Extrinsics (CUTE) dataset of $18,000$ images of $180$ objects under different extrinsic factors such as lighting, poses, and imaging conditions. While existing methods such as LPIPS and CLIP scores do not measure object intrinsics well, we find that combining deep features learned from contrastive self-supervised learning with foreground filtering is a simple yet effective approach to approximating the similarity. We conduct an extensive survey of pre-trained features and foreground extraction methods to arrive at a strong baseline that best measures intrinsic object-centric image similarity among current methods. Finally, we demonstrate that our approach can aid in downstream applications such as acting as an analog for human subjects and improving generalizable re-identification. Please see our project website at https://s-tian.github.io/projects/cute/ for visualizations of the data and demos of our metric.