Exploring the Interchangeability of CNN Embedding Spaces

TL;DR

This paper demonstrates that CNN embedding spaces are largely interchangeable across different architectures and training datasets, enabling performance preservation when transforming embeddings between models for recognition tasks.

Contribution

It introduces empirical methods to map CNN embedding spaces, showing their interchangeability for image classification and face recognition tasks across various architectures.

Findings

Embedding spaces are linearly mappable across CNNs.

Performance is preserved when transforming embeddings between models.

Mappings can be derived from backend layer weights or estimated empirically.

Abstract

CNN feature spaces can be linearly mapped and consequently are often interchangeable. This equivalence holds across variations in architectures, training datasets, and network tasks. Specifically, we mapped between 10 image-classification CNNs and between 4 facial-recognition CNNs. When image embeddings generated by one CNN are transformed into embeddings corresponding to the feature space of a second CNN trained on the same task, their respective image classification or face verification performance is largely preserved. For CNNs trained to the same classes and sharing a common backend-logit (soft-max) architecture, a linear-mapping may always be calculated directly from the backend layer weights. However, the case of a closed-set analysis with perfect knowledge of classifiers is limiting. Therefore, empirical methods of estimating mappings are presented for both the closed-set image classification task and the open-set task of face recognition. The results presented expose the essentially interchangeable nature of CNNs embeddings for two important and common recognition tasks. The implications are far-reaching, suggesting an underlying commonality between representations learned by networks designed and trained for a common task. One practical implication is that face embeddings from some commonly used CNNs can be compared using these mappings.