TextManiA: Enriching Visual Feature by Text-driven Manifold Augmentation

TL;DR

TextManiA introduces a novel method that leverages pre-trained language models to semantically enrich visual features, improving performance especially in imbalanced and scarce data scenarios.

Contribution

The paper presents a new text-driven manifold augmentation technique that transfers language model representations to enhance visual feature spaces without requiring visual training data.

Findings

Effective in class imbalance scenarios

Enhances visual features with semantic information

Compatible with label mix-based methods

Abstract

We propose TextManiA, a text-driven manifold augmentation method that semantically enriches visual feature spaces, regardless of class distribution. TextManiA augments visual data with intra-class semantic perturbation by exploiting easy-to-understand visually mimetic words, i.e., attributes. This work is built on an interesting hypothesis that general language models, e.g., BERT and GPT, encompass visual information to some extent, even without training on visual training data. Given the hypothesis, TextManiA transfers pre-trained text representation obtained from a well-established large language encoder to a target visual feature space being learned. Our extensive analysis hints that the language encoder indeed encompasses visual information at least useful to augment visual representation. Our experiments demonstrate that TextManiA is particularly powerful in scarce samples with class imbalance as well as even distribution. We also show compatibility with the label mix-based approaches in evenly distributed scarce data.