Randomized Quantization: A Generic Augmentation for Data Agnostic Self-supervised Learning

TL;DR

This paper introduces a novel data augmentation method for self-supervised learning that leverages randomized channel-wise quantization, improving performance across multiple data modalities without relying on modality-specific techniques.

Contribution

The authors propose a generic augmentation technique using randomized channel quantization based on precision redundancy, applicable across diverse data types.

Findings

Outperforms existing generic augmentation methods

Achieves comparable results to modality-specific augmentations

Effective across vision, audio, and 3D point cloud data

Abstract

Self-supervised representation learning follows a paradigm of withholding some part of the data and tasking the network to predict it from the remaining part. Among many techniques, data augmentation lies at the core for creating the information gap. Towards this end, masking has emerged as a generic and powerful tool where content is withheld along the sequential dimension, e.g., spatial in images, temporal in audio, and syntactic in language. In this paper, we explore the orthogonal channel dimension for generic data augmentation by exploiting precision redundancy. The data for each channel is quantized through a non-uniform quantizer, with the quantized value sampled randomly within randomly sampled quantization bins. From another perspective, quantization is analogous to channel-wise masking, as it removes the information within each bin, but preserves the information across bins. Our approach significantly surpasses existing generic data augmentation methods, while showing on par performance against modality-specific augmentations. We comprehensively evaluate our approach on vision, audio, 3D point clouds, as well as the DABS benchmark which is comprised of various data modalities. The code is available at https: //github.com/microsoft/random_quantize.