Exploring the Sharpened Cosine Similarity

TL;DR

This paper investigates the use of Sharpened Cosine Similarity as a replacement for convolutional layers in CNNs, analyzing its performance, interpretability, and robustness on CIFAR-10.

Contribution

It provides the first comprehensive empirical analysis of SCS in CNNs, exploring its behavior, interpretability, and robustness compared to traditional convolution.

Findings

SCS may not significantly improve accuracy

SCS can lead to more interpretable features

SCS may slightly enhance adversarial robustness

Abstract

Convolutional layers have long served as the primary workhorse for image classification. Recently, an alternative to convolution was proposed using the Sharpened Cosine Similarity (SCS), which in theory may serve as a better feature detector. While multiple sources report promising results, there has not been to date a full-scale empirical analysis of neural network performance using these new layers. In our work, we explore SCS's parameter behavior and potential as a drop-in replacement for convolutions in multiple CNN architectures benchmarked on CIFAR-10. We find that while SCS may not yield significant increases in accuracy, it may learn more interpretable representations. We also find that, in some circumstances, SCS may confer a slight increase in adversarial robustness.