Self-Directed Synthetic Dialogues and Revisions Technical Report

TL;DR

This paper introduces SDSD, a dataset of self-guided multi-turn conversations generated by open models, aiming to enhance instruction-following fine-tuning and synthetic data diversity.

Contribution

The paper presents a novel dataset of self-directed synthetic dialogues and explores revision principles to improve synthetic preference data for open models.

Findings

Created a multi-model synthetic dialogue dataset with guided conversations.

Incorporated principles from Constitutional AI for conversation revisions.

Encourages further research in multi-turn synthetic data and open model applications.

Abstract

Synthetic data has become an important tool in the fine-tuning of language models to follow instructions and solve complex problems. Nevertheless, the majority of open data to date is often lacking multi-turn data and collected on closed models, limiting progress on advancing open fine-tuning methods. We introduce Self Directed Synthetic Dialogues (SDSD), an experimental dataset consisting of guided conversations of language models talking to themselves. The dataset consists of multi-turn conversations generated with DBRX, Llama 2 70B, and Mistral Large, all instructed to follow a conversation plan generated prior to the conversation. We also explore including principles from Constitutional AI and other related works to create synthetic preference data via revisions to the final conversation turn. We hope this work encourages further exploration in multi-turn data and the use of open models for expanding the impact of synthetic data.