Adversarial Momentum-Contrastive Pre-Training

TL;DR

This paper introduces an adversarial momentum contrastive pre-training method that enhances feature learning efficiency with smaller batches and fewer epochs, outperforming existing methods and some supervised defenses.

Contribution

It proposes a novel adversarial momentum contrastive learning approach using dual memory banks, reducing computational requirements while improving feature robustness.

Findings

Achieves superior performance with smaller batch sizes and fewer epochs.

Outperforms previous adversarial pre-training models.

Outperforms some state-of-the-art supervised defensive methods.

Abstract

Recently proposed adversarial self-supervised learning methods usually require big batches and long training epochs to extract robust features, which will bring heavy computational overhead on platforms with limited resources. In order to help the network learn more powerful feature representations in smaller batches and fewer epochs, this paper proposes a novel adversarial momentum contrastive learning method, which introduces two memory banks corresponding to clean samples and adversarial samples, respectively. These memory banks can be dynamically incorporated into the training process to track invariant features among historical mini-batches. Compared with the previous adversarial pre-training model, our method achieves superior performance with smaller batch size and less training epochs. In addition, the model outperforms some state-of-the-art supervised defensive methods on multiple benchmark datasets after being fine-tuned on downstream classification tasks.