Generating Diverse Hypotheses for Inductive Reasoning

TL;DR

This paper introduces Mixture of Concepts (MoC), a novel method for generating diverse, high-quality hypotheses in inductive reasoning with large language models, overcoming limitations of temperature-based diversity control.

Contribution

The paper proposes MoC, a new approach inspired by human reasoning, to improve hypothesis diversity and quality in LLM-based inductive reasoning tasks.

Findings

MoC outperforms standard IID sampling in diversity and accuracy.

Increasing temperature leads to text degeneration, limiting diversity.

MoC achieves significant improvements on inductive reasoning benchmarks.

Abstract

Inductive reasoning - the process of inferring general rules from a small number of observations - is a fundamental aspect of human intelligence. Recent works suggest that large language models (LLMs) can engage in inductive reasoning by sampling multiple hypotheses about the rules and selecting the one that best explains the observations. However, due to the IID sampling, semantically redundant hypotheses are frequently generated, leading to significant wastage of compute. In this paper, we 1) demonstrate that increasing the temperature to enhance the diversity is limited due to text degeneration issue, and 2) propose a novel method to improve the diversity while maintaining text quality. We first analyze the effect of increasing the temperature parameter, which is regarded as the LLM's diversity control, on IID hypotheses. Our analysis shows that as temperature rises, diversity and accuracy of hypotheses increase up to a certain point, but this trend saturates due to text degeneration. To generate hypotheses that are more semantically diverse and of higher quality, we propose a novel approach inspired by human inductive reasoning, which we call Mixture of Concepts (MoC). When applied to several inductive reasoning benchmarks, MoC demonstrated significant performance improvements compared to standard IID sampling and other approaches.