Feature Map Similarity Reduction in Convolutional Neural Networks

TL;DR

This paper introduces the Convolutional Similarity method to reduce feature map redundancy in CNNs, demonstrating improved accuracy, faster convergence, and smaller models, addressing limitations of kernel orthogonality approaches.

Contribution

It proposes a novel similarity reduction technique that independently minimizes feature map redundancy, enhancing CNN efficiency and performance.

Findings

Reduces feature map similarity effectively

Improves classification accuracy

Enables smaller models with comparable performance

Abstract

It has been observed that Convolutional Neural Networks (CNNs) suffer from redundancy in feature maps, leading to inefficient capacity utilization. Efforts to address this issue have largely focused on kernel orthogonality method. In this work, we theoretically and empirically demonstrate that kernel orthogonality does not necessarily lead to a reduction in feature map redundancy. Based on this analysis, we propose the Convolutional Similarity method to reduce feature map similarity, independently of the CNN's input. The Convolutional Similarity can be minimized as either a regularization term or an iterative initialization method. Experimental results show that minimizing Convolutional Similarity not only improves classification accuracy but also accelerates convergence. Furthermore, our method enables the use of significantly smaller models to achieve the same level of performance, promoting a more efficient use of model capacity. Future work will focus on coupling the iterative initialization method with the optimization momentum term and examining the method's impact on generative frameworks.