Rethinking Data Augmentation for Robust Visual Question Answering

TL;DR

This paper introduces KDDAug, a knowledge distillation-based data augmentation method for visual question answering that generates pseudo answers for diverse image-question pairs, improving robustness and generalization across models and datasets.

Contribution

It proposes a novel, model-agnostic data augmentation strategy using knowledge distillation to generate pseudo answers, relaxing pairing constraints and enhancing VQA model robustness.

Findings

KDDAug improves VQA performance across multiple benchmarks.

It enhances robustness to out-of-distribution data.

The method is effective with various backbone architectures.

Abstract

Data Augmentation (DA) -- generating extra training samples beyond original training set -- has been widely-used in today's unbiased VQA models to mitigate the language biases. Current mainstream DA strategies are synthetic-based methods, which synthesize new samples by either editing some visual regions/words, or re-generating them from scratch. However, these synthetic samples are always unnatural and error-prone. To avoid this issue, a recent DA work composes new augmented samples by randomly pairing pristine images and other human-written questions. Unfortunately, to guarantee augmented samples have reasonable ground-truth answers, they manually design a set of heuristic rules for several question types, which extremely limits its generalization abilities. To this end, we propose a new Knowledge Distillation based Data Augmentation for VQA, dubbed KDDAug. Specifically, we first relax the requirements of reasonable image-question pairs, which can be easily applied to any question types. Then, we design a knowledge distillation (KD) based answer assignment to generate pseudo answers for all composed image-question pairs, which are robust to both in-domain and out-of-distribution settings. Since KDDAug is a model-agnostic DA strategy, it can be seamlessly incorporated into any VQA architectures. Extensive ablation studies on multiple backbones and benchmarks have demonstrated the effectiveness and generalization abilities of KDDAug.