ZEBRA: Zero-Shot Example-Based Retrieval Augmentation for Commonsense Question Answering

TL;DR

ZEBRA is a zero-shot framework that enhances commonsense question answering by combining retrieval, reasoning, and introspection without additional training, leading to improved accuracy and interpretability.

Contribution

ZEBRA introduces a novel zero-shot approach that integrates retrieval and reasoning to improve commonsense QA without extra training or templates.

Findings

Outperforms previous knowledge integration methods

Achieves up to 4.5 points accuracy improvement

Effective across 8 benchmarks

Abstract

Current Large Language Models (LLMs) have shown strong reasoning capabilities in commonsense question answering benchmarks, but the process underlying their success remains largely opaque. As a consequence, recent approaches have equipped LLMs with mechanisms for knowledge retrieval, reasoning and introspection, not only to improve their capabilities but also to enhance the interpretability of their outputs. However, these methods require additional training, hand-crafted templates or human-written explanations. To address these issues, we introduce ZEBRA, a zero-shot question answering framework that combines retrieval, case-based reasoning and introspection and dispenses with the need for additional training of the LLM. Given an input question, ZEBRA retrieves relevant question-knowledge pairs from a knowledge base and generates new knowledge by reasoning over the relationships in these pairs. This generated knowledge is then used to answer the input question, improving the model's performance and interpretability. We evaluate our approach across 8 well-established commonsense reasoning benchmarks, demonstrating that ZEBRA consistently outperforms strong LLMs and previous knowledge integration approaches, achieving an average accuracy improvement of up to 4.5 points.