Weaker LLMs' Opinions Also Matter: Mixture of Opinions Enhances LLM's Mathematical Reasoning

TL;DR

This paper introduces a post-training method that combines opinions from weaker LLMs to improve the mathematical reasoning abilities of a stronger LLM, demonstrating a 5% performance boost on benchmarks.

Contribution

It proposes a novel mixture of opinions (MoO) approach that leverages weaker LLMs' outputs to enhance a stronger LLM's reasoning capabilities.

Findings

MoO improves mathematical reasoning by 5% on average.

Incorporating diverse perspectives benefits LLM reasoning.

MoO outperforms standard fine-tuning and prompting methods.

Abstract

Recent advances in Large Language Models (LLMs) have raised interest in their formal reasoning capabilities, particularly in mathematics. While closed LLMs like GPT-4 perform well on mathematical benchmarks, e.g., GSM8K, it remains unclear whether small to medium-sized open LLMs can achieve similar performance, questioning their reliability. To close this gap, we propose a post-training approach leveraging a mixture of opinions (MoO) from weaker ancillary LLMs to enhance a (relatively) stronger LLM's reasoning. For that, each post-training sample is augmented with Chain-of-Thought (CoT) reasoning steps and answers from ancillary LLMs, enabling the main LLM to learn from diverse perspectives. We compare MoO with standard supervised fine-tuning (SFT), few-shot prompting, and the Mixture of Agents (MoA) method on mathematical reasoning benchmarks. Our results show that incorporating weaker LLMs' opinions improves mathematical reasoning by an average of 5%, highlighting the value of diverse perspectives in reasoning tasks.