Rosetta Neurons: Mining the Common Units in a Model Zoo

TL;DR

This paper uncovers common neural features called 'Rosetta Neurons' shared across diverse vision models, revealing inherent visual concepts learned without labels and enabling model translation and manipulation.

Contribution

It introduces an algorithm to identify Rosetta Neurons across multiple models, demonstrating shared representations across architectures, tasks, and supervision types.

Findings

Shared visual concepts are embedded in models regardless of architecture or supervision.

Rosetta Neurons enable cross-model translation and manipulation without retraining.

Generative models allow visualization of shared concepts.

Abstract

Do different neural networks, trained for various vision tasks, share some common representations? In this paper, we demonstrate the existence of common features we call "Rosetta Neurons" across a range of models with different architectures, different tasks (generative and discriminative), and different types of supervision (class-supervised, text-supervised, self-supervised). We present an algorithm for mining a dictionary of Rosetta Neurons across several popular vision models: Class Supervised-ResNet50, DINO-ResNet50, DINO-ViT, MAE, CLIP-ResNet50, BigGAN, StyleGAN-2, StyleGAN-XL. Our findings suggest that certain visual concepts and structures are inherently embedded in the natural world and can be learned by different models regardless of the specific task or architecture, and without the use of semantic labels. We can visualize shared concepts directly due to generative models included in our analysis. The Rosetta Neurons facilitate model-to-model translation enabling various inversion-based manipulations, including cross-class alignments, shifting, zooming, and more, without the need for specialized training.