Dipper: Diversity in Prompts for Producing Large Language Model Ensembles in Reasoning tasks

TL;DR

Dipper introduces a training-free ensemble method that uses diverse prompts in parallel to improve reasoning performance of smaller LLMs, outperforming larger models on benchmarks.

Contribution

Dipper is a novel framework that transforms a single LLM into an effective inference-time ensemble using diverse prompts, without additional training.

Findings

Significant performance improvements on reasoning benchmarks.

A 3-model DIPPER ensemble outperforms a larger 7B model.

Parallel prompting enhances reasoning capabilities.

Abstract

Large Language Models (LLMs), particularly smaller variants, still struggle with complex reasoning tasks. While inference-time prompting can guide reasoning, existing methods often rely on sequential queries. Ensemble approaches offer a promising path to performance gains, especially given recent batch inference speed-ups. This work introduces DIPPER, a novel, training-free framework that transforms a single LLM into an effective inference-time ensemble. By feeding the model an optimized and diverse set of prompts in parallel, DIPPER elicits varied reasoning paths, leading to performance gains. We empirically demonstrate significant improvements on reasoning benchmarks, such as MATH, where a DIPPER ensemble of three Qwen2-MATH-1.5B instances (via parallel prompting of a single model) outperforms a larger 7B model.