PCNN: Probable-Class Nearest-Neighbor Explanations Improve Fine-Grained Image Classification Accuracy for AIs and Humans

TL;DR

This paper introduces PCNN, a method that uses probable-class nearest neighbors and an image comparator to enhance the accuracy of fine-grained image classification and improve human-AI decision-making.

Contribution

It presents a novel approach leveraging nearest neighbors and image comparison to refine predictions of a pretrained classifier, boosting accuracy and interpretability.

Findings

Improves fine-grained classification accuracy on multiple datasets.

Reduces human over-reliance on AI decisions in user studies.

Enhances decision accuracy for both AI and humans.

Abstract

Nearest neighbors (NN) are traditionally used to compute final decisions, e.g., in Support Vector Machines or k-NN classifiers, and to provide users with explanations for the model's decision. In this paper, we show a novel utility of nearest neighbors: To improve predictions of a frozen, pretrained image classifier C. We leverage an image comparator S that (1) compares the input image with NN images from the top-K most probable classes given by C; and (2) uses scores from S to weight the confidence scores of C to refine predictions. Our method consistently improves fine-grained image classification accuracy on CUB-200, Cars-196, and Dogs-120. Also, a human study finds that showing users our probable-class nearest neighbors (PCNN) reduces over-reliance on AI, thus improving their decision accuracy over prior work which only shows only the most-probable (top-1) class examples.