Reasoning Circuits: Few-shot Multihop Question Generation with Structured Rationales

TL;DR

This paper introduces a structured rationale framework for multi-hop question generation that improves performance and control in low-shot settings using modest model sizes, reducing the need for extensive rationale annotation.

Contribution

The work presents a new multi-step rationale schema and a low-supervision approach for multi-hop question generation, enabling better control and performance with smaller models.

Findings

Improved question quality with structured rationales in low-shot regimes

Enhanced control over question difficulty through rationale schema

Better automatic and human evaluation results compared to baselines

Abstract

Multi-hop Question Generation is the task of generating questions which require the reader to reason over and combine information spread across multiple passages using several reasoning steps. Chain-of-thought rationale generation has been shown to improve performance on multi-step reasoning tasks and make model predictions more interpretable. However, few-shot performance gains from including rationales have been largely observed only in +100B language models, and otherwise require large scale manual rationale annotation. In this work, we introduce a new framework for applying chain-of-thought inspired structured rationale generation to multi-hop question generation under a very low supervision regime (8- to 128-shot). We propose to annotate a small number of examples following our proposed multi-step rationale schema, treating each reasoning step as a separate task to be performed by a generative language model. We show that our framework leads to improved control over the difficulty of the generated questions and better performance compared to baselines trained without rationales, both on automatic evaluation metrics and in human evaluation. Importantly, we show that this is achievable with a modest model size.