Retrieve, Caption, Generate: Visual Grounding for Enhancing Commonsense in Text Generation Models

TL;DR

This paper introduces VisCTG, a multimodal approach that uses image captioning to improve commonsense reasoning and text generation quality in Transformer models like BART and T5.

Contribution

It proposes a novel multimodal method that leverages image grounding to enhance commonsense and fluency in concept-to-text generation tasks.

Findings

Significant performance improvement over baseline models.

Enhanced commonsense reasoning and fluency in generated text.

Addresses issues of poor specificity in baseline outputs.

Abstract

We investigate the use of multimodal information contained in images as an effective method for enhancing the commonsense of Transformer models for text generation. We perform experiments using BART and T5 on concept-to-text generation, specifically the task of generative commonsense reasoning, or CommonGen. We call our approach VisCTG: Visually Grounded Concept-to-Text Generation. VisCTG involves captioning images representing appropriate everyday scenarios, and using these captions to enrich and steer the generation process. Comprehensive evaluation and analysis demonstrate that VisCTG noticeably improves model performance while successfully addressing several issues of the baseline generations, including poor commonsense, fluency, and specificity.