Cognitive Visual Commonsense Reasoning Using Dynamic Working Memory

TL;DR

This paper introduces a dynamic working memory model for visual commonsense reasoning, enhancing inference accuracy and interpretability in VCR tasks by storing and utilizing accumulated prior knowledge.

Contribution

The paper presents a novel dynamic working memory approach that improves generalization and interpretability in visual commonsense reasoning tasks.

Findings

Significant performance improvements on the VCR benchmark dataset.

Enhanced interpretability of reasoning process.

Effective utilization of prior knowledge in inference.

Abstract

Visual Commonsense Reasoning (VCR) predicts an answer with corresponding rationale, given a question-image input. VCR is a recently introduced visual scene understanding task with a wide range of applications, including visual question answering, automated vehicle systems, and clinical decision support. Previous approaches to solving the VCR task generally rely on pre-training or exploiting memory with long dependency relationship encoded models. However, these approaches suffer from a lack of generalizability and prior knowledge. In this paper we propose a dynamic working memory based cognitive VCR network, which stores accumulated commonsense between sentences to provide prior knowledge for inference. Extensive experiments show that the proposed model yields significant improvements over existing methods on the benchmark VCR dataset. Moreover, the proposed model provides intuitive interpretation into visual commonsense reasoning. A Python implementation of our mechanism is publicly available at https://github.com/tanjatang/DMVCR