Complexity-aware fine-tuning

TL;DR

This paper introduces a complexity-aware fine-tuning method for large language models that selectively applies reasoning to complex data, improving efficiency and accuracy over standard approaches.

Contribution

It proposes a novel approach that uses entropy to identify complex data, enabling more efficient fine-tuning with less data and better performance.

Findings

Outperforms standard supervised fine-tuning in accuracy.

Uses 81% less data than traditional methods.

Effectively distinguishes data complexity with entropy.

Abstract

General-purpose Large Language Models (LLMs) are frequently fine-tuned through supervised fine-tuning (SFT) to enhance performance in specific domains. Better results can be achieved by distilling the chain-of-thought of a larger model at the cost of numerous expensive calls and a much greater amount of data. We propose a novel blueprint for efficient fine-tuning that uses reasoning only for complex data identified by entropy. Specifically, across three small open models ($\approx 3B$) we split the training data into complexity categories by a single token answer entropy (ROC AUC $0.73$), fine-tune large language models (LLMs) via SFT and distillation, and show that our pipeline significantly outperforms the standard SFT approach ($0.58$ vs $0.45$ average accuracy) and outperforms the distillation approach ($0.58$ vs $0.56$ average accuracy) while using $81\%$ less data.