Local Manifold Augmentation for Multiview Semantic Consistency

TL;DR

This paper introduces local manifold augmentation (LMA), a novel data augmentation technique that generates diverse, semantically consistent views by modeling data variations, significantly improving self-supervised learning performance and robustness across multiple datasets.

Contribution

The paper proposes LMA, an instance-conditioned generator that captures local data variations to produce diverse augmentations, enhancing multiview semantic consistency in self-supervised learning.

Findings

LMA improves accuracy on CIFAR, STL10, and ImageNet benchmarks.

LMA enhances invariance to viewpoint, pose, and illumination.

LMA increases robustness to distribution shifts like ImageNet-V2 and Sketch.

Abstract

Multiview self-supervised representation learning roots in exploring semantic consistency across data of complex intra-class variation. Such variation is not directly accessible and therefore simulated by data augmentations. However, commonly adopted augmentations are handcrafted and limited to simple geometrical and color changes, which are unable to cover the abundant intra-class variation. In this paper, we propose to extract the underlying data variation from datasets and construct a novel augmentation operator, named local manifold augmentation (LMA). LMA is achieved by training an instance-conditioned generator to fit the distribution on the local manifold of data and sampling multiview data using it. LMA shows the ability to create an infinite number of data views, preserve semantics, and simulate complicated variations in object pose, viewpoint, lighting condition, background etc. Experiments show that with LMA integrated, self-supervised learning methods such as MoCov2 and SimSiam gain consistent improvement on prevalent benchmarks including CIFAR10, CIFAR100, STL10, ImageNet100, and ImageNet. Furthermore, LMA leads to representations that obtain more significant invariance to the viewpoint, object pose, and illumination changes and stronger robustness to various real distribution shifts reflected by ImageNet-V2, ImageNet-R, ImageNet Sketch etc.