Reward Engineering for Generating Semi-structured Explanation

TL;DR

This paper introduces a reward engineering approach using reinforcement learning to improve the generation of semi-structured explanations in language models, surpassing previous methods on key benchmarks.

Contribution

It presents a novel reward engineering technique in reinforcement learning that enhances semi-structured explanation generation, addressing limitations of supervised fine-tuning.

Findings

Achieves state-of-the-art results on ExplaGraph and COPA-SSE benchmarks.

Demonstrates the effectiveness of reward aggregation methods in RL for explanation tasks.

Highlights potential of RL for future semi-structured explanation generation research.

Abstract

Semi-structured explanation depicts the implicit process of a reasoner with an explicit representation. This explanation highlights how available information in a specific query is utilised and supplemented with information a reasoner produces from its internal weights towards generating an answer. Despite the recent improvements in generative capabilities of language models, producing structured explanations to verify a model's true reasoning capabilities remains a challenge. This issue is particularly pronounced for not-so-large LMs (e.g., FLAN-T5-XXL). In this work, we first underscore the limitations of supervised fine-tuning (SFT) in tackling this challenge, and then introduce a carefully crafted reward engineering method in reinforcement learning (RL) to better address this problem. We investigate multiple reward aggregation methods and provide a detailed discussion which sheds light on the promising potential of RL for future research. Our proposed method on two semi-structured explanation generation benchmarks (ExplaGraph and COPA-SSE) achieves new state-of-the-art results.