DeepThink: Aligning Language Models with Domain-Specific User Intents

TL;DR

DeepThink introduces a novel framework for generating high-quality instructions to better align language models with real user intents in domain-specific QA, leading to significant performance improvements.

Contribution

The paper presents DeepThink, a new method that generates realistic instructions by simulating user questions and conversations, improving domain-specific LLM adaptation.

Findings

Achieves 7.92% performance improvement over baseline methods.

Enhances relevance, completeness, clarity, accuracy, and actionability of answers.

Effective in advertising domain QA tasks.

Abstract

Supervised fine-tuning with synthesized instructions has been a common practice for adapting LLMs to domain-specific QA tasks. However, the synthesized instructions deviate from real user questions and expected answers. This study proposes a novel framework called DeepThink to generate high-quality instructions. DeepThink first generates a few seed questions to mimic actual user questions, simulates conversations to uncover the hidden user needs, and refines the answer by conversational contexts and the retrieved documents for more comprehensive answers. Experiments demonstrate that DeepThink achieves an average performance improvement of 7.92% compared to a GPT-4-turbo+RAG-based assistant on the real user test set in the advertising domain across dimensions such as relevance, completeness, clarity, accuracy, and actionability.