Rationale-Augmented Ensembles in Language Models

TL;DR

This paper introduces rationale-augmented ensembles for language models, improving multi-step reasoning performance and interpretability by sampling rationales in a unified framework applicable to various NLP tasks.

Contribution

It proposes a general rationale-augmented ensemble framework that enhances robustness and performance over manual prompt engineering methods.

Findings

Rationale-augmented ensembles outperform existing prompting methods.

The approach improves interpretability of model predictions.

Framework is applicable to diverse NLP tasks.

Abstract

Recent research has shown that rationales, or step-by-step chains of thought, can be used to improve performance in multi-step reasoning tasks. We reconsider rationale-augmented prompting for few-shot in-context learning, where (input -> output) prompts are expanded to (input, rationale -> output) prompts. For rationale-augmented prompting we demonstrate how existing approaches, which rely on manual prompt engineering, are subject to sub-optimal rationales that may harm performance. To mitigate this brittleness, we propose a unified framework of rationale-augmented ensembles, where we identify rationale sampling in the output space as the key component to robustly improve performance. This framework is general and can easily be extended to common natural language processing tasks, even those that do not traditionally leverage intermediate steps, such as question answering, word sense disambiguation, and sentiment analysis. We demonstrate that rationale-augmented ensembles achieve more accurate and interpretable results than existing prompting approaches--including standard prompting without rationales and rationale-based chain-of-thought prompting--while simultaneously improving interpretability of model predictions through the associated rationales.