The Karp Dataset

TL;DR

The paper introduces the Karp dataset, a collection of NP-completeness reduction proofs designed to evaluate and improve the mathematical reasoning abilities of Large Language Models, with experiments showing fine-tuning benefits.

Contribution

It presents the first dataset of detailed NP-completeness proofs for training and benchmarking LLMs' reasoning skills, spanning various difficulty levels.

Findings

Fine-tuning with the Karp dataset enhances LLM reasoning capacity.

State-of-the-art models show improved performance after training on Karp.

The dataset covers a range of difficulty levels from undergraduate to academic research.

Abstract

Understanding the mathematical reasoning capabilities of Large Language Models (LLMs) is a central topic in the study of artificial intelligence. This new domain necessitates the creation of datasets of reasoning tasks for both training and benchmarking the performance of LLMs. To this end, we introduce the Karp dataset: The first dataset composed of detailed proofs of NP-completeness reductions. The reductions vary in difficulty, ranging from simple exercises of undergraduate courses to more challenging reductions from academic papers. We compare the performance of state-of-the-art models on this task and demonstrate the effect of fine-tuning with the Karp dataset on reasoning capacity.