Case-Enhanced Vision Transformer: Improving Explanations of Image Similarity with a ViT-based Similarity Metric

TL;DR

This paper introduces CEViT, a Vision Transformer-based similarity metric that enhances explainability in image similarity assessments and maintains competitive classification accuracy.

Contribution

The paper proposes CEViT, a novel case-enhanced ViT model that improves interpretability of image similarity and integrates with k-NN classification.

Findings

CEViT achieves accuracy comparable to state-of-the-art models.

CEViT provides case-influenced explanations of image similarity.

Preliminary results show promising explainability improvements.

Abstract

This short paper presents preliminary research on the Case-Enhanced Vision Transformer (CEViT), a similarity measurement method aimed at improving the explainability of similarity assessments for image data. Initial experimental results suggest that integrating CEViT into k-Nearest Neighbor (k-NN) classification yields classification accuracy comparable to state-of-the-art computer vision models, while adding capabilities for illustrating differences between classes. CEViT explanations can be influenced by prior cases, to illustrate aspects of similarity relevant to those cases.