Learn 3D VQA Better with Active Selection and Reannotation

TL;DR

This paper introduces a multi-turn active learning approach for 3D Visual Question Answering that effectively identifies and reannotates misleading data, improving model performance while reducing training costs.

Contribution

It proposes a novel active learning strategy that uses semantic uncertainty and reannotation to address misleading labels in 3D VQA datasets, enhancing training efficiency.

Findings

Improved 3D VQA accuracy with less training data

Halved training costs for high-accuracy models

Effective identification and correction of misleading annotations

Abstract

3D Visual Question Answering (3D VQA) is crucial for enabling models to perceive the physical world and perform spatial reasoning. In 3D VQA, the free-form nature of answers often leads to improper annotations that can confuse or mislead models when training on the entire dataset. While other text generation tasks can mitigate this issue by learning on large-scale datasets, the scarcity of 3D scene data enlarges the negative effect of misleading annotations. Although active learning strategies can select valuable instances for training, they fail to identify and resolve misleading labels, which the oracle inevitably provides in practice. To address this issue, we propose a multi-turn interactive active learning strategy. This strategy selects data based on models' semantic uncertainty to form a solid knowledge foundation more effectively and actively requests reannotation from an oracle to resolve potentially misleading labels. For uncertainty assessment, we utilize a variance-based metric that takes semantic relationships between terms into consideration, thus avoiding the uniform inter-class similarity assumption of previous assessment metrics. Extensive experiments exhibit better model performance and a substantial reduction in training costs, with a halving of training costs for achieving relatively high accuracy. The code is available at https://github.com/fz-zsl/AQuA.