Dynamic Channel Selection in Self-Supervised Learning

TL;DR

This paper explores dynamic channel selection in self-supervised learning models to reduce computational costs while maintaining performance, adapting supervised channel selection methods to self-supervised networks across multiple datasets.

Contribution

It demonstrates that a standard channel selection method can be effectively applied to self-supervised models, achieving significant FLOPs reduction without performance loss.

Findings

Comparable accuracy with reduced FLOPs across datasets

Effective application of supervised channel selection to self-supervised models

Significant computational savings in image classification tasks

Abstract

Whilst computer vision models built using self-supervised approaches are now commonplace, some important questions remain. Do self-supervised models learn highly redundant channel features? What if a self-supervised network could dynamically select the important channels and get rid of the unnecessary ones? Currently, convnets pre-trained with self-supervision have obtained comparable performance on downstream tasks in comparison to their supervised counterparts in computer vision. However, there are drawbacks to self-supervised models including their large numbers of parameters, computationally expensive training strategies and a clear need for faster inference on downstream tasks. In this work, our goal is to address the latter by studying how a standard channel selection method developed for supervised learning can be applied to networks trained with self-supervision. We validate our findings on a range of target budgets $t_{d}$ for channel computation on image classification task across different datasets, specifically CIFAR-10, CIFAR-100, and ImageNet-100, obtaining comparable performance to that of the original network when selecting all channels but at a significant reduction in computation reported in terms of FLOPs.