Revisiting a kNN-based Image Classification System with High-capacity Storage

TL;DR

This paper proposes a kNN-based image classification system that stores knowledge externally, allowing updates without fine-tuning, interpretability of inference, and avoiding catastrophic forgetting, achieving competitive accuracy on ImageNet and CIFAR-100.

Contribution

It introduces an external knowledge storage system for image classification that updates via database modifications, not model fine-tuning, enhancing interpretability and incremental learning.

Findings

Achieves 79.8% top-1 accuracy on ImageNet without fine-tuning.

Attains 90.8% accuracy on Split CIFAR-100 in incremental learning.

Provides interpretability by analyzing neighborhood samples in kNN.

Abstract

In existing image classification systems that use deep neural networks, the knowledge needed for image classification is implicitly stored in model parameters. If users want to update this knowledge, then they need to fine-tune the model parameters. Moreover, users cannot verify the validity of inference results or evaluate the contribution of knowledge to the results. In this paper, we investigate a system that stores knowledge for image classification, such as image feature maps, labels, and original images, not in model parameters but in external high-capacity storage. Our system refers to the storage like a database when classifying input images. To increase knowledge, our system updates the database instead of fine-tuning model parameters, which avoids catastrophic forgetting in incremental learning scenarios. We revisit a kNN (k-Nearest Neighbor) classifier and employ it in our system. By analyzing the neighborhood samples referred by the kNN algorithm, we can interpret how knowledge learned in the past is used for inference results. Our system achieves 79.8% top-1 accuracy on the ImageNet dataset without fine-tuning model parameters after pretraining, and 90.8% accuracy on the Split CIFAR-100 dataset in the task incremental learning setting.