Key-Point-Driven Mathematical Reasoning Distillation of Large Language Model

TL;DR

This paper introduces Key-Point-Driven Mathematical Reasoning Distillation (KPDD), a method to improve small language models' mathematical reasoning by breaking down problems into stages, significantly reducing errors and achieving state-of-the-art results.

Contribution

The paper proposes KPDD, a novel distillation approach that enhances small language models' reasoning by focusing on key points and structured problem decomposition, addressing semantic errors.

Findings

KPDD-CoT improves reasoning capabilities.

KPDD-PoT achieves state-of-the-art performance.

Method reduces misunderstanding errors in SLMs.

Abstract

Large Language Models (LLMs) have demonstrated exceptional proficiency in mathematical reasoning tasks due to their extensive parameter counts and training on vast datasets. Despite these capabilities, deploying LLMs is hindered by their computational demands. Distilling LLM mathematical reasoning into Smaller Language Models (SLMs) has emerged as a solution to this challenge, although these smaller models often suffer from errors in calculation and semantic understanding. Prior work has proposed Program-of-Thought Distillation (PoTD) to avoid calculation error. To further address semantic understanding errors, we propose Key-Point-Driven Mathematical Reasoning Distillation (KPDD). KPDD enhances the reasoning performance of SLMs by breaking down the problem-solving process into three stages: Core Question Extraction, Problem-Solving Information Extraction, and Step-by-Step Solution. This method is further divided into KPDD-CoT, which generates Chain-of-Thought rationales, and KPDD-PoT, which creates Program-of-Thought rationales. The experiment results show that KPDD-CoT significantly improves reasoning abilities, while KPDD-PoT achieves state-of-the-art performance in mathematical reasoning tasks. Our approach effectively mitigates misunderstanding errors, advancing the deployment of efficient and capable SLMs.