SeA: Semantic Adversarial Augmentation for Last Layer Features from Unsupervised Representation Learning

TL;DR

SeA introduces a semantic adversarial augmentation technique in feature space to enhance fixed deep features for classification, outperforming baseline features and rivaling fine-tuning in efficiency.

Contribution

The paper proposes a novel semantic adversarial augmentation method in feature space that improves fixed deep features for classification tasks.

Findings

SeA improves deep feature performance by 2% on average across 11 tasks.

SeA achieves comparable results to fine-tuning on 6 out of 11 tasks.

The method is more efficient than traditional fine-tuning.

Abstract

Deep features extracted from certain layers of a pre-trained deep model show superior performance over the conventional hand-crafted features. Compared with fine-tuning or linear probing that can explore diverse augmentations, \eg, random crop/flipping, in the original input space, the appropriate augmentations for learning with fixed deep features are more challenging and have been less investigated, which degenerates the performance. To unleash the potential of fixed deep features, we propose a novel semantic adversarial augmentation (SeA) in the feature space for optimization. Concretely, the adversarial direction implied by the gradient will be projected to a subspace spanned by other examples to preserve the semantic information. Then, deep features will be perturbed with the semantic direction, and augmented features will be applied to learn the classifier. Experiments are conducted on $11$ benchmark downstream classification tasks with $4$ popular pre-trained models. Our method is $2\%$ better than the deep features without SeA on average. Moreover, compared to the expensive fine-tuning that is expected to give good performance, SeA shows a comparable performance on $6$ out of $11$ tasks, demonstrating the effectiveness of our proposal in addition to its efficiency. Code is available at \url{https://github.com/idstcv/SeA}.