LARE: Latent Augmentation using Regional Embedding with Vision-Language Model

TL;DR

LARE introduces a regional embedding approach for vision-language models, enabling effective data augmentation across unseen domains and improving image classification accuracy in diverse conditions.

Contribution

The paper proposes LARE, a novel regional embedding method that enhances vision-language models' domain adaptation and classification performance through latent region sampling.

Findings

LARE outperforms previous models on three benchmark datasets.

LARE demonstrates robustness with limited and imbalanced data.

LARE effectively generalizes to unseen domains.

Abstract

In recent years, considerable research has been conducted on vision-language models that handle both image and text data; these models are being applied to diverse downstream tasks, such as "image-related chat," "image recognition by instruction," and "answering visual questions." Vision-language models (VLMs), such as Contrastive Language-Image Pre-training (CLIP), are also high-performance image classifiers that are being developed into domain adaptation methods that can utilize language information to extend into unseen domains. However, because these VLMs embed images as a single point in a unified embedding space, there is room for improvement in the classification accuracy. Therefore, in this study, we proposed the Latent Augmentation using Regional Embedding (LARE), which embeds the image as a region in the unified embedding space learned by the VLM. By sampling the augmented image embeddings from within this latent region, LARE enables data augmentation to various unseen domains, not just to specific unseen domains. LARE achieves robust image classification for domains in and out using augmented image embeddings to fine-tune VLMs. We demonstrate that LARE outperforms previous fine-tuning models in terms of image classification accuracy on three benchmarks. We also demonstrate that LARE is a more robust and general model that is valid under multiple conditions, such as unseen domains, small amounts of data, and imbalanced data.