Visual Recognition with Deep Nearest Centroids

TL;DR

This paper introduces Deep Nearest Centroids (DNC), a nonparametric, explainable network for large-scale visual recognition that leverages class sub-centroids for classification, improving transparency and transferability.

Contribution

DNC revisits the classic Nearest Centroids classifier, integrating it into deep networks for improved explainability, transferability, and performance in image and pixel recognition tasks.

Findings

DNC outperforms parametric models on CIFAR-10 and ImageNet.

DNC enhances pixel recognition accuracy on ADE20K and Cityscapes.

DNC offers ad-hoc explainability by using actual training images as sub-centroids.

Abstract

We devise deep nearest centroids (DNC), a conceptually elegant yet surprisingly effective network for large-scale visual recognition, by revisiting Nearest Centroids, one of the most classic and simple classifiers. Current deep models learn the classifier in a fully parametric manner, ignoring the latent data structure and lacking simplicity and explainability. DNC instead conducts nonparametric, case-based reasoning; it utilizes sub-centroids of training samples to describe class distributions and clearly explains the classification as the proximity of test data and the class sub-centroids in the feature space. Due to the distance-based nature, the network output dimensionality is flexible, and all the learnable parameters are only for data embedding. That means all the knowledge learnt for ImageNet classification can be completely transferred for pixel recognition learning, under the "pre-training and fine-tuning" paradigm. Apart from its nested simplicity and intuitive decision-making mechanism, DNC can even possess ad-hoc explainability when the sub-centroids are selected as actual training images that humans can view and inspect. Compared with parametric counterparts, DNC performs better on image classification (CIFAR-10, ImageNet) and greatly boots pixel recognition (ADE20K, Cityscapes), with improved transparency and fewer learnable parameters, using various network architectures (ResNet, Swin) and segmentation models (FCN, DeepLabV3, Swin). We feel this work brings fundamental insights into related fields.