Unsupervised Representation Learning for Binary Networks by Joint Classifier Learning

TL;DR

This paper introduces BURN, a self-supervised learning method for binary networks that enhances their accuracy and deployability on resource-limited devices by jointly training classifiers and feature extractors.

Contribution

The paper proposes a novel self-supervised training approach for binary networks using a moving target and feature similarity loss, improving performance over existing methods.

Findings

BURN outperforms existing self-supervised baselines on five downstream tasks.

BURN sometimes surpasses supervised pretraining in accuracy.

The method is effective across multiple datasets and tasks.

Abstract

Self-supervised learning is a promising unsupervised learning framework that has achieved success with large floating point networks. But such networks are not readily deployable to edge devices. To accelerate deployment of models with the benefit of unsupervised representation learning to such resource limited devices for various downstream tasks, we propose a self-supervised learning method for binary networks that uses a moving target network. In particular, we propose to jointly train a randomly initialized classifier, attached to a pretrained floating point feature extractor, with a binary network. Additionally, we propose a feature similarity loss, a dynamic loss balancing and modified multi-stage training to further improve the accuracy, and call our method BURN. Our empirical validations over five downstream tasks using seven datasets show that BURN outperforms self-supervised baselines for binary networks and sometimes outperforms supervised pretraining. Code is availabe at https://github.com/naver-ai/burn.