Dizygotic Conditional Variational AutoEncoder for Multi-Modal and Partial Modality Absent Few-Shot Learning

TL;DR

This paper introduces DCVAE, a novel multi-modal data augmentation method using paired CVAEs to improve few-shot learning, especially in scenarios with partial modality absence, achieving state-of-the-art results.

Contribution

The paper proposes a new multi-modal data augmentation approach with paired CVAEs that enhances feature diversity and quality for few-shot learning tasks.

Findings

Achieves state-of-the-art performance on miniImageNet, CIFAR-FS, and CUB datasets.

Effective in scenarios with partial modality absence.

Demonstrates improved feature diversity and synthesis quality.

Abstract

Data augmentation is a powerful technique for improving the performance of the few-shot classification task. It generates more samples as supplements, and then this task can be transformed into a common supervised learning issue for solution. However, most mainstream data augmentation based approaches only consider the single modality information, which leads to the low diversity and quality of generated features. In this paper, we present a novel multi-modal data augmentation approach named Dizygotic Conditional Variational AutoEncoder (DCVAE) for addressing the aforementioned issue. DCVAE conducts feature synthesis via pairing two Conditional Variational AutoEncoders (CVAEs) with the same seed but different modality conditions in a dizygotic symbiosis manner. Subsequently, the generated features of two CVAEs are adaptively combined to yield the final feature, which can be converted back into its paired conditions while ensuring these conditions are consistent with the original conditions not only in representation but also in function. DCVAE essentially provides a new idea of data augmentation in various multi-modal scenarios by exploiting the complement of different modality prior information. Extensive experimental results demonstrate our work achieves state-of-the-art performances on miniImageNet, CIFAR-FS and CUB datasets, and is able to work well in the partial modality absence case.