Visual DNA: Representing and Comparing Images using Distributions of Neuron Activations

TL;DR

This paper introduces Distributions of Neuron Activations (DNAs), a novel, compact method for representing and comparing datasets in computer vision by analyzing neuron activation distributions in a frozen feature extractor.

Contribution

The paper proposes DNAs as a flexible, dataset-agnostic tool for dataset comparison, enabling granular, customizable distance measurements in a compact form.

Findings

DNAs effectively differentiate datasets in various tasks.

DNAs are highly compact, less than 15MB for large datasets.

Demonstrated applicability across synthetic, real, and diverse datasets.

Abstract

Selecting appropriate datasets is critical in modern computer vision. However, no general-purpose tools exist to evaluate the extent to which two datasets differ. For this, we propose representing images - and by extension datasets - using Distributions of Neuron Activations (DNAs). DNAs fit distributions, such as histograms or Gaussians, to activations of neurons in a pre-trained feature extractor through which we pass the image(s) to represent. This extractor is frozen for all datasets, and we rely on its generally expressive power in feature space. By comparing two DNAs, we can evaluate the extent to which two datasets differ with granular control over the comparison attributes of interest, providing the ability to customise the way distances are measured to suit the requirements of the task at hand. Furthermore, DNAs are compact, representing datasets of any size with less than 15 megabytes. We demonstrate the value of DNAs by evaluating their applicability on several tasks, including conditional dataset comparison, synthetic image evaluation, and transfer learning, and across diverse datasets, ranging from synthetic cat images to celebrity faces and urban driving scenes.