Step-by-Step Mastery: Enhancing Soft Constraint Following Ability of Large Language Models

TL;DR

This paper introduces a new training pipeline that combines dataset construction, preference optimization, and curriculum learning to significantly improve large language models' ability to follow multiple soft constraints in instructions.

Contribution

It presents a novel combination of dataset generation, DPO training, and curriculum learning specifically designed to enhance soft constraint adherence in LLMs.

Findings

Improved soft constraint following accuracy in LLMs.

Effective curriculum learning based on constraint complexity.

Publicly available datasets and code for further research.

Abstract

It is crucial for large language models (LLMs) to follow instructions that involve multiple constraints. However, it is an unexplored area to enhance LLMs' ability to follow soft constraints. To bridge the gap, we initially design a pipeline to construct datasets with high-quality outputs automatically. Additionally, to fully utilize the positive and negative samples generated during the data construction process, we choose Direct Preference Optimization (DPO) as the training method. Furthermore, taking into account the difficulty of soft constraints indicated by the number of constraints, we design a curriculum learning training paradigm based on the constraint quantity. We experimentally evaluate the effectiveness of our methods in improving LLMs' soft constraint following ability and analyze the factors driving the improvements.The datasets and code are publicly available at https://github.com/Rainier-rq/FollowSoftConstraint.