MuSC: Improving Complex Instruction Following with Multi-granularity Self-Contrastive Training

TL;DR

MuSC introduces a novel training framework that enhances complex instruction-following in large language models by leveraging multi-granularity self-contrastive training, eliminating the need for stronger models like GPT-4.

Contribution

It proposes a multi-granularity self-contrastive training method that improves instruction alignment without relying on advanced models, using both coarse and fine granularity techniques.

Findings

Significant improvement on complex instruction-following benchmarks

Outperforms previous self-alignment methods

Effective on open-sourced models

Abstract

Complex instruction-following with elaborate constraints is imperative for Large Language Models (LLMs). While existing methods have constructed data for complex instruction alignment, they all rely on a more advanced model, especially GPT-4, limiting their application. In this paper, we propose a Multi-granularity Self-Contrastive Training (MuSC) framework, to improve the complex instruction alignment without relying on a stronger model. Our method is conducted on both coarse and fine granularity. On coarse-granularity, we construct constraint-aware preference data based on instruction decomposition and recombination. On fine-granularity, we perform token-aware preference optimization with dynamic token-level supervision. Our method is evaluated on open-sourced models, and experiment results show our method achieves significant improvement on both complex and general instruction-following benchmarks, surpassing previous self-alignment methods.