Are They the Same Picture? Adapting Concept Bottleneck Models for Human-AI Collaboration in Image Retrieval

TL;DR

This paper introduces CHAIR, a human-in-the-loop model based on concept bottlenecks that improves image retrieval by enabling human correction of intermediate concepts, leading to better embeddings and retrieval performance.

Contribution

The paper adapts the Concept Bottleneck Model to allow human correction of concepts in image retrieval, enhancing interpretability and performance in human-AI collaboration.

Findings

CHAIR outperforms similar models on retrieval metrics without external intervention

Human intervention further improves retrieval performance

The method demonstrates effective human-AI complementarity

Abstract

Image retrieval plays a pivotal role in applications from wildlife conservation to healthcare, for finding individual animals or relevant images to aid diagnosis. Although deep learning techniques for image retrieval have advanced significantly, their imperfect real-world performance often necessitates including human expertise. Human-in-the-loop approaches typically rely on humans completing the task independently and then combining their opinions with an AI model in various ways, as these models offer very little interpretability or \textit{correctability}. To allow humans to intervene in the AI model instead, thereby saving human time and effort, we adapt the Concept Bottleneck Model (CBM) and propose \texttt{CHAIR}. \texttt{CHAIR} (a) enables humans to correct intermediate concepts, which helps \textit{improve} embeddings generated, and (b) allows for flexible levels of intervention that accommodate varying levels of human expertise for better retrieval. To show the efficacy of \texttt{CHAIR}, we demonstrate that our method performs better than similar models on image retrieval metrics without any external intervention. Furthermore, we also showcase how human intervention helps further improve retrieval performance, thereby achieving human-AI complementarity.