System Message Generation for User Preferences using Open-Source Models

TL;DR

This paper introduces SysGen, a pipeline that generates effective system messages for LLMs using existing datasets, significantly improving conversation quality and adaptability in user interactions.

Contribution

The paper presents a novel method for generating system messages from existing datasets, enhancing LLM responses without requiring manually annotated system prompts.

Findings

Improved performance on Multifacet and SysBench benchmarks.

Significant gains in short, early-stage conversations.

Enhanced diversity and structure in system messages improve adaptability.

Abstract

System messages play a crucial role in interactions with large language models (LLMs), often serving as prompts to initiate conversations. Through system messages, users can assign specific roles, perform intended tasks, incorporate background information, and specify various output formats and communication styles. Despite such versatility, publicly available datasets often lack system messages and are subject to strict license constraints in industrial applications. Moreover, manually annotating system messages that align with user instructions is resource-intensive. In light of these challenges, we introduce SysGen, a pipeline for generating system messages that better align assistant responses with user instructions using existing supervised fine-tuning datasets that lack system messages. Training open-source models on SysGen data yields substantial improvements in both single-turn (Multifacet) and multi-turn (SysBench) conversation benchmarks. Notably, our method shows strong gains in shorter conversations, suggesting that it enhances early-stage interaction effectiveness. Our qualitative analysis further emphasizes the value of diverse and structured system messages in improving LLM adaptability across varied user scenarios.