S3C: Semi-Supervised VQA Natural Language Explanation via Self-Critical Learning

TL;DR

This paper introduces S3C, a semi-supervised learning approach for VQA natural language explanations that improves logical consistency and achieves state-of-the-art results without relying heavily on human-annotated explanations.

Contribution

The paper proposes a novel semi-supervised framework with self-critical learning to generate more faithful and consistent explanations for VQA models, reducing dependence on costly annotations.

Findings

Achieves state-of-the-art performance on VQA-NLE datasets.

Improves logical consistency between answers and rationales.

Effectively leverages unlabeled data through semi-supervised learning.

Abstract

VQA Natural Language Explanation (VQA-NLE) task aims to explain the decision-making process of VQA models in natural language. Unlike traditional attention or gradient analysis, free-text rationales can be easier to understand and gain users' trust. Existing methods mostly use post-hoc or self-rationalization models to obtain a plausible explanation. However, these frameworks are bottlenecked by the following challenges: 1) the reasoning process cannot be faithfully responded to and suffer from the problem of logical inconsistency. 2) Human-annotated explanations are expensive and time-consuming to collect. In this paper, we propose a new Semi-Supervised VQA-NLE via Self-Critical Learning (S3C), which evaluates the candidate explanations by answering rewards to improve the logical consistency between answers and rationales. With a semi-supervised learning framework, the S3C can benefit from a tremendous amount of samples without human-annotated explanations. A large number of automatic measures and human evaluations all show the effectiveness of our method. Meanwhile, the framework achieves a new state-of-the-art performance on the two VQA-NLE datasets.