Improving Commonsense in Vision-Language Models via Knowledge Graph Riddles

TL;DR

This paper introduces DANCE, a scalable data augmentation method using knowledge graph linearization to enhance commonsense reasoning in vision-language models, validated by a new diagnostic benchmark.

Contribution

We propose DANCE, a novel data augmentation technique leveraging knowledge graphs to improve commonsense in VL models without additional dataset collection.

Findings

DANCE significantly improves commonsense reasoning in VL models.

DANCE maintains performance on standard retrieval tasks.

A new retrieval-based benchmark evaluates commonsense capabilities.

Abstract

This paper focuses on analyzing and improving the commonsense ability of recent popular vision-language (VL) models. Despite the great success, we observe that existing VL-models still lack commonsense knowledge/reasoning ability (e.g., "Lemons are sour"), which is a vital component towards artificial general intelligence. Through our analysis, we find one important reason is that existing large-scale VL datasets do not contain much commonsense knowledge, which motivates us to improve the commonsense of VL-models from the data perspective. Rather than collecting a new VL training dataset, we propose a more scalable strategy, i.e., "Data Augmentation with kNowledge graph linearization for CommonsensE capability" (DANCE). It can be viewed as one type of data augmentation technique, which can inject commonsense knowledge into existing VL datasets on the fly during training. More specifically, we leverage the commonsense knowledge graph (e.g., ConceptNet) and create variants of text description in VL datasets via bidirectional sub-graph sequentialization. For better commonsense evaluation, we further propose the first retrieval-based commonsense diagnostic benchmark. By conducting extensive experiments on some representative VL-models, we demonstrate that our DANCE technique is able to significantly improve the commonsense ability while maintaining the performance on vanilla retrieval tasks. The code and data are available at https://github.com/pleaseconnectwifi/DANCE