Enhancing Zero-shot Commonsense Reasoning by Integrating Visual Knowledge via Machine Imagination

TL;DR

This paper introduces Imagine, a zero-shot commonsense reasoning framework that integrates visual signals generated by machines to overcome textual reporting biases, significantly improving reasoning performance.

Contribution

The paper presents a novel framework that incorporates machine-generated visual signals into PLMs, enhancing zero-shot commonsense reasoning beyond existing methods.

Findings

Outperforms existing zero-shot approaches

Surpasses advanced large language models in benchmarks

Effectively mitigates reporting bias in reasoning

Abstract

Recent advancements in zero-shot commonsense reasoning have empowered Pre-trained Language Models (PLMs) to acquire extensive commonsense knowledge without requiring task-specific fine-tuning. Despite this progress, these models frequently suffer from limitations caused by human reporting biases inherent in textual knowledge, leading to understanding discrepancies between machines and humans. To bridge this gap, we introduce an additional modality to enrich the reasoning capabilities of PLMs. We propose Imagine (Machine Imagination-based Reasoning), a novel zero-shot commonsense reasoning framework that supplements textual inputs with visual signals from machine-generated images. Specifically, we enhance PLMs with the ability to imagine by embedding an image generator directly into the reasoning pipeline. To facilitate effective utilization of this imagined visual context, we construct synthetic datasets designed to emulate visual question-answering scenarios. Through comprehensive evaluations on multiple commonsense reasoning benchmarks, we demonstrate that Imagine substantially outperforms existing zero-shot approaches and even surpasses advanced large language models. These results underscore the capability of machine imagination to mitigate reporting bias and significantly enhance the generalization ability of commonsense reasoning models