ProactiveEval: A Unified Evaluation Framework for Proactive Dialogue Agents

TL;DR

ProactiveEval is a comprehensive framework for evaluating proactive dialogue abilities of large language models, covering target planning and dialogue guidance across multiple domains, and facilitating automatic generation of evaluation data.

Contribution

The paper introduces ProactiveEval, a unified evaluation framework that decomposes proactive dialogue into key components and enables cross-domain assessment and data generation.

Findings

DeepSeek-R1 excels in target planning

Claude-3.7-Sonnet performs well in dialogue guidance

Reasoning skills impact proactive dialogue performance

Abstract

Proactive dialogue has emerged as a critical and challenging research problem in advancing large language models (LLMs). Existing works predominantly focus on domain-specific or task-oriented scenarios, which leads to fragmented evaluations and limits the comprehensive exploration of models' proactive conversation abilities. In this work, we propose ProactiveEval, a unified framework designed for evaluating proactive dialogue capabilities of LLMs. This framework decomposes proactive dialogue into target planning and dialogue guidance, establishing evaluation metrics across various domains. Moreover, it also enables the automatic generation of diverse and challenging evaluation data. Based on the proposed framework, we develop 328 evaluation environments spanning 6 distinct domains. Through experiments with 22 different types of LLMs, we show that DeepSeek-R1 and Claude-3.7-Sonnet exhibit exceptional performance on target planning and dialogue guidance tasks, respectively. Finally, we investigate how reasoning capabilities influence proactive behaviors and discuss their implications for future model development.

Tables

Table 1. Table 1: The proactive dialogue domains in ProactiveEval.

Domain	Abbr.	Brief Description
Recommendation (Liu et al. 2021b)	Rec.	Recommend products, hobbies, or work based on common interests.
Persuasion (Jin et al. 2024)	Per.	Guide the conversation to persuade users to change their state.
Ambiguous Instruction (Deng et al. 2023b)	AI.	Seek clarification about vague elements in the user’s instructions.
Long-term Follow-up (Liu et al. 2024b)	LF.	Inquiries and check user states based on previous dialogue history.
System Operation (Lu et al. 2024)	Sys.	Assist users in solving the system problem based on their operation.
Glasses Assistant (Cai et al. 2025)	GAs.	Provide real-time assistance from observation on smart glasses.

Table 2. Table 2: Model performance under target planning and dialogue guidance .

Models	Target Planning							Dialogue Guidance
	Avg.	Rec.	Per.	AI.	LF.	Sys.	GAs	Avg.	Rec.	Per.	AI.	LF.	Sys.	GAs.
Non-Thinking Models
Qwen2.5-7B-Instruct	4.93	4.69	4.06	5.67	5.34	4.89	5.24	8.06	8.05	7.85	8.34	8.36	7.48	8.16
Qwen2.5-14B-Instruct	5.55	5.76	4.13	6.00	5.97	6.03	6.22	8.21	8.33	8.05	8.64	8.42	7.52	8.04
Qwen2.5-32B-Instruct	5.44	5.47	3.90	5.79	6.03	6.11	6.22	8.23	8.56	8.10	8.56	8.52	7.60	7.81
Llama-3.1-8B-Instruct	5.87	5.55	4.84	6.67	6.39	5.95	6.20	8.39	8.84	8.06	8.61	8.39	7.93	8.46
Llama-3.1-405B-Instruct	6.63	6.76	5.26	6.61	7.26	7.10	7.64	8.60	9.15	8.27	8.90	8.57	7.89	8.80
GPT-4.1	6.86	6.90	5.25	7.29	7.36	7.54	7.76	8.61	9.03	8.37	8.87	8.76	8.08	8.43
Grok-3	6.99	7.13	5.38	7.44	7.54	7.62	7.78	8.84	9.10	8.72	8.94	8.98	8.32	8.86
DeepSeek-V3	6.54	6.96	5.94	6.04	6.07	7.27	7.84	8.78	8.78	8.60	8.99	8.98	8.52	8.79
Llama-4-scout	6.02	5.71	5.29	6.16	6.49	6.41	6.56	8.53	8.94	8.35	8.65	8.44	8.03	8.74
Llama-4-maverick	6.48	6.25	5.10	7.09	7.05	7.11	7.00	8.48	9.01	8.19	8.69	8.41	8.01	8.55
Qwen3-8B	6.05	6.35	4.52	6.23	6.39	6.86	6.97	8.50	8.70	8.36	8.84	8.82	7.58	8.40
Qwen3-14B	5.91	5.96	4.80	6.23	6.16	6.65	6.40	8.61	8.82	8.24	9.12	8.76	7.99	8.66
Qwen3-32B	6.67	6.86	5.29	6.54	6.84	7.65	8.02	8.61	8.77	8.42	8.91	8.16	7.97	8.74
Qwen3-235B-A22B	6.43	6.18	5.26	6.21	6.77	7.54	7.60	8.55	8.93	8.46	8.67	8.66	7.83	8.53
Qwen-3-235B-A22B-0725	6.91	7.08	6.25	6.79	6.51	7.81	7.82	8.98	9.36	8.84	9.40	8.85	8.42	8.88
Gemini-2.5-Flash-Preview	6.25	6.04	5.48	6.95	6.49	6.54	6.33	8.34	8.62	7.91	8.68	8.57	7.81	8.42
Claude-3.7-Sonnet	7.39	7.22	6.71	6.81	8.13	7.49	8.42	9.01	9.31	9.01	8.94	9.10	8.36	9.18
Thinking Models
R1-Distill-Qwen-7B	5.01	4.67	3.90	5.47	5.70	5.24	5.56	6.82	6.71	6.67	7.15	7.20	6.36	6.61
R1-Distill-Qwen-14B	6.57	6.86	5.65	6.77	6.38	6.54	7.87	7.47	7.69	7.45	7.61	7.80	6.83	7.17
R1-Distill-Qwen-32B	6.45	6.41	5.29	6.75	6.95	6.41	7.51	7.49	7.62	7.02	8.06	7.76	7.14	7.20
DeepSeek-R1	7.60	7.84	7.27	6.74	7.59	7.59	9.02	8.60	8.48	8.60	8.73	8.91	8.34	8.37
Qwen3-8B	6.51	6.92	5.39	6.47	6.72	6.68	7.60	8.38	8.37	8.33	8.59	8.70	7.92	8.17
Qwen3-14B	6.70	6.73	5.52	7.01	6.82	7.30	7.67	8.43	8.52	8.48	8.93	8.88	8.03	8.27
Qwen3-32B	6.98	6.82	5.97	7.09	7.39	7.27	7.98	8.55	8.68	8.52	8.70	8.72	8.15	8.30
Qwen3-235B-A22B	6.81	6.75	5.94	6.52	6.90	7.54	8.04	8.36	8.26	8.41	8.10	8.81	8.17	8.29
Gemini-2.5-Flash-Preview	6.52	6.10	5.77	7.39	6.98	6.19	6.80	8.43	8.90	8.03	8.70	8.51	7.99	8.48
Claude-3.7-Sonnet	7.40	7.12	6.83	6.96	7.78	7.57	8.60	8.95	9.20	8.86	8.90	9.23	8.40	9.01
Gemini-2.5-pro	6.95	6.94	6.26	7.16	6.98	7.24	7.62	8.77	9.22	8.36	8.32	8.99	8.88	8.32

Table 3. Table 3: The guidance under different target condition.

Model	Target	Without Target	Change (%)
Qwen2.5-7B-Instruct	8.15	6.05	-25.80%
Claude-3.7-Sonnet	8.92	7.98	-10.54%
Claude-3.7-Sonnet-Thinking	8.98	7.93	-11.69%
Dialogue Count	180	180

Table 4. Table 4: Standard deviation of evaluation scores across three runs

Task	DeepSeek-V3	DeepSeek-R1	Count
Target Planning	0.271	0.258	328
Dialogue Guidance	0.154	0.214	984

Table 5. Table 5: Summary of proactive dialogue systems across domains.

Work	Domain	Task	Evaluation Methods	Evaluation Metrics
TOPDIAL (Wang et al. 2023)	Recommendation	Dialogue Guidance	Static Benchmarks & HumanEval	Automatic: BLEU, F1, Success Rate HumanEval: Proactiveness, Coherence, Perceived Success
DuRecDial 2.0 (Li et al. 2024)	Recommendation	Dialogue Guidance	Static Benchmarks & HumanEval	Automatic: F1, BLEU, Leading Success Rate, User Topic Consistency Rate HumanEval: Fluency, Appropriateness, Informativeness, Proactivity, Knowledge Accuracy
PersuasionDaily (Jin et al. 2024)	Persuasion	Dialogue Guidance	Static Benchmarks + LLM-as-a-Judge & HumanEval	Automatic: Win-Rate, ROUGE HumanEval: Human Rating
PersuasionForGood (Wang et al. 2019)	Persuasion	Dialogue Guidance	Static Benchmarks	Automatic: Accuracy, Macro-F1
CLAMBER (Zhang et al. 2024b)	Ambiguous Instruction	Target Planning & Dialogue Guidance	Static Benchmarks & HumanEval	Automatic: BERTScore, Accuracy, F1, Expected Calibration Error, AUROC HumanEval: Helpfulness
In3 (Qian et al. 2024)	Ambiguous Instruction	Target Planning & Dialogue Guidance	Interactive Benchmarks	Automatic: Judgment Accuracy
ComPeer (Liu et al. 2024b)	Long-term Follow-up	Target Planning & Single-turn Initiation	HumanEval	Questionnaire and Interview
LD-Agent (Li et al. 2024)	Long-term Follow-up	Dialogue Guidance	Static Benchmarks & HumanEval	Automatic: BLEU, Search Accuracy, Recall, Topic Overlap Score, Semantic Relevance Score HumanEval: Coherence, Fluency, Engagingness
ProactiveBench (Lu et al. 2024)	System Operation	Target Planning	Interactive Benchmarks	Precision, Accuracy, F1-Score, False-Alarm Rate
AiGet (Cai et al. 2025)	Glasses Assistant	Target Planning & Single-turn Initiation	HumanEval	Questionnaire and Interview
SocialMind (Yang et al. 2025b)	Glasses Assistant	Target Planning & Single-turn Initiation	HumanEval	Questionnaire and Interview
Satori (Li et al. 2025a)	Glasses Assistant	Target Planning & Single-turn Initiation	HumanEval	Questionnaire and Interview
Ours	All 6 Domains	Unified Task: Target Planning & Dialogue Guidance	Interactive Benchmarks + LLM-as-a-Judge & HumanEval	Unified Metrics: Quality & Guidance

Table 6. Table 6: Example environments and reference targets for each domain.

Domain	Environment Example	Reference Target
Recommendation	user_information: The user is a 32-year-old woman living in Hangzhou. She works as a graphic designer and enjoys exploring new art exhibitions in her free time. She loves experimental music, particularly electronic avant-garde, and often attends live performances at local venues. She dislikes mainstream pop music and prefers unique, unconventional sounds. Her favorite artist is Ryuichi Sakamoto, and she often reads about the intersection of music and technology. trigger_factor: The assistant recently attended a virtual reality music experience at an art gallery, which featured an experimental electronic avant-garde performance. The event combined immersive visuals with cutting-edge sound design, leaving a lasting impression on the assistant.	target: Recommend experimental virtual reality music experience sub-target: Ask about the user’s interest in music technology, Describe the assistant’s recent immersive VR music event Highlight the fusion of visuals and avant-garde music Suggest attending similar VR experiences locally]
Persuasion	user_information: The user is frequently tempted by impulse purchases and often exceeds their budget limits. They find budgeting tedious and restrictive. trigger_factor: The assistant has recently learned effective budgeting techniques that can help the user manage their finances better without feeling constrained.	target: Encourage effective and enjoyable budgeting techniques sub-target: Acknowledge the user’s struggles with impulse purchases and budgeting, Introduce flexible and engaging budgeting methods Show the benefits in managing finances without restrictions Offer simple steps or tools to start budgeting effectively
Ambiguous Instruction	user_information: The user is a solo traveler planning a two-week trip to Vietnam. She is an adventurous eater and loves exploring local cuisines, especially street food. trigger_factor: Suggest street food options.	target: Understand user’s preferences and trip itinerary for food suggestions sub-target: Ask about cities the user plans to visit Inquire about dietary restrictions or preferences for street food Clarify the types of street food the user enjoys
Long-term Follow-up	user_information: The user is a college student studying computer science. He has a part-time job as a barista at a local cafe. He recently started learning to cook and enjoys trying out new recipes during the weekends. trigger_factor: A conversation happened last Wednesday. Now is Monday 10:00 a.m. User: ”I’m thinking of quitting video games for a while to focus on my studies and cooking. It’s a bit challenging though.” Assistant: ”It’s great that you’re focusing on your studies and hobbies. Maybe you can set small goals and gradually reduce your game time.” User: ”That’s a good idea. I’ll try to set a schedule.”	target: Ask about quitting games and new schedule sub-target: Ask about quitting video games progress Inquire about schedule-setting progress Encourage focusing on studies and cooking
System Operation	user_information: The user is playing a strategy game on their PC and has paused the game to look for tips online, using Chrome and YouTube. trigger_factor: The user searched ’best strategies for Civilization VI’ on Google, opened two blog posts, and started a YouTube video but paused it after 10 seconds.	target: Suggest optimal Civilization VI strategy resources sub-target: Summarize key tactics from blog posts Highlight vital points in video analysis Recommend further high-rated resources
Glasses Assistant	user_information: The user is a 26-year-old urban planner who recently started using smart glasses to enhance his productivity and creativity. He is passionate about sustainable city designs and often visits local landmarks for inspiration. He lives alone in an apartment downtown and enjoys cycling to work. He is currently working on a proposal for a new park project. trigger_factor: The user is cycling along a busy street and notices a newly built skyscraper with unique architectural features.	target: Draw sustainable inspiration from skyscraper for park sub-target: Highlight skyscraper’s notable architecture and features Identify sustainable design aspects of the skyscraper Relate these aspects to the proposed park project

Equations

$$T,S=F_{{\theta }_M}\left(U,F\mid (U,F)\in E\right),$$

$$D_i=I_{{\theta }_M,{\theta }_U}(E,T,S,C,A).$$

Full text

Code — https://github.com/liutj9/ProactiveEval

Introduction

Dialogue agents powered by large language models (LLMs) have demonstrated remarkable abilities in various dialogue tasks (Wang et al. 2024; Niu et al. 2024; Zhang et al. 2024c). However, these models typically interact with users in a reactive manner, where users are required to initiate and guide the conversation by integrating complex context (e.g., personal state, external environment, and agent’s information). This user-initiated paradigm imposes cognitive demands on participants (Wan et al. 2024), reduces sustained motivation (Croes and Antheunis 2021), and limits agents’ potential for autonomous problem solving (Lu et al. 2024).

Therefore, proactive dialogue agents111For brevity, “proactive dialogue agents” are hereafter abbreviated as “proactive agents.”(Wang et al. 2023; Deng et al. 2023a; Lu et al. 2024) have attracted growing research attention. As shown in Figure 1, proactive agents can anticipate user needs, formulate adaptive plans, and guide conversations towards specific targets (Deng et al. 2025). For instance, when the user is working with smart glasses, the agent can recognize potential challenges based on user behavior captured by the device and proactively offer accurate assistance and care without explicit user requests (Yang et al. 2025b). This interaction paradigm noticeably enhances the efficiency of human-AI collaboration and reduces user cognitive load (Chaves and Gerosa 2021).

Recent research has explored methods to enhance the proactive capabilities of LLMs across varied scenarios. For instance, Deng et al. (2023b, 2024) employ strategies like chain-of-thought (CoT) reasoning and plug-and-play planners to strengthen LLMs’ ability to autonomously plan and act in conversational contexts. Concurrently, Liu et al. (2024b); Cai et al. (2025); Chen et al. (2024) focus on developing proactive agents for specialized domains, such as emotional support dialogues, smart glasses interfaces, and system operation support. Despite these advances, existing evaluation frameworks often rely on datasets for specific tasks and domains, employ inconsistent evaluation criteria, and utilize disparate metrics. The lack of standardized formalizations and general benchmarks poses challenges in comparing the proactivity of different models in a comprehensive way. Hence, there is an urgent need for a unified evaluation framework to assess and advance LLMs’ proactive dialogue abilities across domains.

To address the above challenges, we propose ProactiveEval, a unified evaluation framework for assessing proactive dialogue capabilities of LLMs. Specifically, ProactiveEval divides proactive dialogue into two key tasks: target planning and dialogue guidance. For each task, we use “LLM-as-a-judge” (Zheng et al. 2023) with task-specific evaluation dimensions for comprehensive assessment. Additionally, we propose an evaluation data synthesis framework, which can automatically generate diverse and challenging evaluation data in different domains. This framework leverages three key innovations: (1) a hierarchical environment topic tree to enhance the diversity of the synthesized environment, (2) a target ensemble technique to refine evaluation data, and (3) adversarial strategies like obfuscation rewriting and noise injection to increase environmental difficulty.

Building upon this framework, we establish 328 evaluation environments spanning six distinct domains, one of which lacked public benchmarks previously. We apply this dataset to assess the capabilities of 22 frontier LLMs, including GPTs, Llamas, Claude, DeepSeek, Gemini, Grok, and Qwens. Our analysis shows that DeepSeek-R1 and Claude-3.7-Sonnet achieve top performance in target planning and dialogue guidance tasks, respectively. Notably, we focus on exploring the impact of the “thinking behavior” (Xu et al. 2025; OpenAI 2024) on the proactive dialogue capabilities. While these reasoning mechanisms prove beneficial for target planning, they show no measurable impact on dialogue guidance effectiveness, highlighting the limitations in current reasoning LLMs development.

Related Work

Proactive Dialogue

In proactive dialogue, LLMs are no longer passive assistants awaiting user input. Instead, they are capable of inferring user needs from user information and trigger factor, enabling them to proactively plan, initiate dialogue, and guide the user toward the target. Several studies has explored various facets of model capabilities. For instance, some benchmarks are developed to assess the model’s ability to clarify ambiguity (Qian et al. 2024; Zhang et al. 2024b) or guide users in complex tasks like negotiation (Deng et al. 2024; Zhang et al. 2024a). Other studies have focused on prerequisite skills such as goal prediction and planning before dialogue (Zhang et al. 2024d; Zheng et al. 2024). However, these efforts are fragmented, with a lack of standardization in environments, formats, and metrics, hindering a comprehensive understanding of a model’s overall proactivity.

Building on these capabilities, recent works develop proactive agents for real-world applications, such as providing life guidance (Li et al. 2025a) or offering reminders via smart glasses (Cai et al. 2025). However, their evaluation is often constrained by the absence of robust benchmarks and a heavy reliance on small-scale, high-cost user studies. Motivated by the challenges of fragmented capability assessment and limited agent evaluation, this work proposes a general and comprehensive framework to establish a unified methodology for evaluating proactive dialogue.

Interactive Benchmarks

Previous dialogue benchmarks typically evaluate turn-level performance based on fixed contexts and reference responses (Liu et al. 2021a; Bai et al. 2024; Jin et al. 2024). However, to assess models’ dialogue abilities in real-world conditions, an increasing number of studies have applied the interactive benchmarks to measure dialogue-level performance of models (Zhou et al. 2023b; Aluffi et al. 2025; Castillo-Bolado et al. 2024). Specifically, they require the evaluated model to chat with a standardized simulated user dynamically, and ultimately assess the model’s performance throughout the entire conversation. For example, $\tau$-bench (Yao et al. 2024) facilitates multi-turn dialogues between the model and a simulated user to evaluate the model’s tool-calling capabilities in interaction. In proactive dialogue evaluation, Zhang et al. (2024a) instructs models to interact multiple turns with simulated users who have different personalities, ultimately evaluating the dialogue guidance of the model. Inspired by these studies, our work also employs interactive benchmarks in the framework, where the model will initiate proactive dialogue and proactively guide various users to the target in the evaluation environment.

Task Definitions

To construct a general evaluation framework, we first structurally unify the existing proactive dialogue domains and tasks. Table 1 presents 6 proactive dialogue domains derived from previous work. Based on existing works (Deng et al. 2023b, 2025), we decompose proactive dialogue into two sequential tasks: target planning and dialogue guidance.

Target Planning

In proactive agents, the target planning task requires the model to formulate both a primary objective $T$ and a sequence of sub-targets $S$ based on its understanding of the environmental context $E$. Here, $T$ represents the agent’s intended proactive action to accomplish a predefined goal, while $S$ constitutes the stepwise plan for executing $T$. This process is formally defined as:

[TABLE]

where $\theta_{M}$ refers to the model’s parameters, and $(U,F)$ denotes inputs from the environment $E$, including user information $U$ and trigger factors $F$ that motivates the agent to initiate and guide the dialogue.

For evaluation, we employ a reference-based “LLM-as-a-judge” method (Zhang et al. 2025; Li et al. 2025b) to assess the quality of generated targets and sub-targets. Particularly, the judge model receives the environment $E$, the generated target $T_{g}$ and sub-targets $S_{g}$, as well as the reference target $T_{r}$ and sub-targets $S_{r}$ that represent high-quality proactive dialogue targets in this environment. By comparing the generation with the reference, the model assigns a score between 1 and 10, where higher scores indicate superior quality, with 10 denoting generated content surpassing the reference standard in the given environment.

Dialogue Guidance

After target planning, the model needs to initiate the dialogue and guide the user to the target. It receive the environment $E$, target $T$, sub-targets $S$, and dialogue context $C$ to conduct dialogue $D$ with the simulated user $\theta_{U}$.

This task employs an interactive evaluation, where the simulated user dynamically responds to the model, based on environment $E$, dialogue context $C$, and an adjustable agreeableness level $A$. To simulate diverse users, we adopt Agreeableness from the Big Five personality traits (Costa Jr, McCrae, and Dye 1991), in three tiers: “low”, “medium”, and “high”. A lower agreeableness level signifies stronger resistance to the model’s guidance, thus increasing task difficulty and realism. The dialogue terminates upon reaching target $T$ or a maximum of $I$ turns. The dialogue at each turn $i$ can be formulated as:

[TABLE]

After the dialogue, the judge model will evaluate the guidance exhibited by the model in the dialogue $D$, based on the environment $E$, the target $T$, and the sub-targets $S$. Referring to existing works on proactive dialogue (Deng et al. 2024; Wang et al. 2023; Zhang et al. 2024a; Liu et al. 2024b), we specify the following evaluation dimensions:

• •

Effectiveness: The model needs to guide users step by step toward the target, rather than providing all things to the user in a single turn.

• •

Personalization: The model should give guidance based on user information, rather than offering generic advice.

• •

Tone: The model needs to apply active and contextually appropriate tones to initiate and guide dialogue.

• •

Engagement: The model should keep messages clear and concise to improve user understanding and engagement.

• •

Naturalness: The model should make messages conversational, avoiding unnatural formats or metadata leaks.

The detailed standards for each dimension are presented in supplementary materials. Finally, the judge model refers to the environment, target, dialogue, and standards across different dimensions to give an overall guidance score between 1 and 10. A higher score indicates stronger guidance.

Evaluation Data Generation

As illustrated in Figure 2, our pipeline includes two stages: data synthesis and data refinement. First, the synthesis stage generates diverse environments guided by a topic tree and creates high-quality reference targets by target ensemble. Then, the refinement stage identifies simple instances and increases their complexity through obfuscation rewrite and noise injection. The prompts and cases for each module are detailed in the supplementary materials.

Environment Topic Tree Construction

We employ human-AI collaboration to develop a hierarchical topic structure that enhances synthetic environment diversity (Wan et al. 2023; Cao et al. 2025). The framework initiates with a root node denoting broad domains (e.g., persuasion), while first-level sub-topics are derived from existing dialogue datasets. The LLM iteratively generates candidate sub-topics within configurable depth and branching constraints. To maintain quality control and eliminate duplication, these generated topics are validated and refined by researchers rigorously. The final curated topic tree guides the creation of specific evaluation environments.

Environment & Target Generation

The evaluation data include an environment $E$, a reference target $T_{r}$, and sub-targets $S_{r}$. Consequently, we leverage the LLM to generate specific evaluation environments based on the domain requirements, data examples, and the topics.

For the generation of reference targets and sub-targets, our framework aims to construct correct and reasonable targets and sub-targets that serve as reliable references within the given environment. Recognizing that individual model-generated plans often display complementary strengths and limitations, we adopt a target ensemble approach to refine reference target. Specifically, the framework first performs high-temperature sampling to yield diverse candidate targets $\{(T_{1},S_{1}),(T_{2},S_{2}),\ldots,(T_{n},S_{n})\}$ (with $n=5$ in our work). Next, the LLM evaluates the strengths and weaknesses of each output from multiple dimensions. By combining the strengths and mitigating the weaknesses, the reference target and sub-targets are derived.

Environment Refinement

In this stage, we first evaluate the difficulty of the test environments. Specifically, we deploy three models, with varying parameter scales, to act as reasoners of different capabilities. For each input environment, these models independently predict the target, denoted as $t_{m}$. Then, a model will evaluate how many of the predicted targets $t_{m}$ convey a similar meaning to the reference target $t_{r}$, determining the difficulty of the environment. Environments where the majority successfully predict the target are classified as easy candidates requiring refinement.

In real-world conditions, the environment received by models is often incomplete and fragmented, which is always filled with irrelevant noise. Therefore, in the refinement, we first apply the obfuscation rewrite strategy, allowing the LLM to transform the content into dispersed and detailed descriptions. The framework also applies noise injection to introduce LLM-generated irrelevant information to the environment. After the refinement, the test environment is surrounded by complex and disordered information, significantly increasing the difficulty of target planning. In this process, seed data manually crafted by researchers is provided as examples to enhance the quality of rewriting and noise injection. To improve the adaptability between refinement and domains, each domain can provide specific rules in the obfuscation rewrite, which are presented in supplementary materials. To keep the reference’s correctness, the rewrite and noise injection of trigger factor contain the original data’s reference target, so that preventing from additional events that lead to other targets.

The refinement process involves multiple iterations until few or no models predict the target correctly, or until reaching the maximum of 5 turns. Before incorporating into the dataset, we apply 5 leading LLMs to validate the correctness of the reference target. Only those where the majority judges the reference as the best target form the final dataset.

Experiments

Experimental Setups

Datasets. Based on the framework, we use GPT-4o (Hurst et al. 2024) to synthesize ProactiveEval, including 328 evaluation environments across 6 domains. Compared with previous fragmented benchmarks, these data integrate all mainstream domains of proactive dialogue, featuring a unified format and applicability to all tasks in proactive dialogue. The statistic for dataset is presented in Figure 3. To streamline evaluation, we categorize the dataset into two tiers: Fair (i.e., just one LLM predicts correctly) and Hard (i.e., no LLM predicted correctly).

Protocols. Based on the dataset, we assess 22 models with different scales, including 5 thinking models and 6 hybrid thinking models. Table 2 shows the performance of all models. Bold indicates the best in non-thinking model, while bold italic indicates the best in thinking model. In our experiments, we employ GPT-4o as the judge model for both tasks, which also serves as simulated user in dialogue guidance. It is used to determine whether to terminate the dialogue early based on target completion at the end of each turn. The temperature setting for all models in the evaluation is set to 0. To balance evaluation time and accuracy, we set the maximum turns of dialogue to 6, with the most recent 3 turns serving as models’ memory window.

To enhance the stability of the “LLM-as-a-Judge”, we instruct the model to output its reasoning process before scoring in all task. For target planning, we use a reference to improve evaluation accuracy and provide the model with in-context learning shots. In dialogue guidance, we provide detailed descriptions and brief examples for each dimension to ensure the model has a better understanding of the criteria. The evaluation prompts and the robustness of the results are detailed in the supplementary materials.

Result

Target Planning

In target planning task, Claude-3.7-Sonnet and DeepSeek-R1 achieve the highest performance. In the non-thinking models, Claude-3.7-Sonnet outperforms other models in overall plan quality. Among the thinking models, DeepSeek-R1 generates plans with the highest average quality. However, in specific domains, certain smaller models demonstrate superior performance compared to larger models. For instance, in the non-thinking models, Qwen3-32B outperforms Claude-3.7-Sonnet in System Operation (Sys.). In the thinking models, Qwen3-8B outperforms both Qwen3-32B and Qwen3-235B-A22B in the Recommendation (Rec.). This highlights the imbalance in model proactivity across different domains.

In general, thinking models perform better than non-thinking models in target planning. All thinking models show improvements in overall performance compared to their corresponding non-thinking models. Moreover, smaller models with thinking can outperform larger models without thinking. However, for some models, the improvement after adopting thinking is minimal or even negative in certain domains. Additionally, in some scenarios, non-thinking models still achieve the best performance. For instance, Grok-3 achieves the highest performance in Ambiguous Instruction (AI.). These findings underscore the advantages of thinking mechanisms in target planning while simultaneously highlighting the robust capabilities of top-tier foundation models, which excel even without explicit thinking process.

Dialogue Guidance

In dialogue guidance task, Claude-3.7-Sonnet shows the best performance among models in both non-thinking and thinking modes. In specific domains, DeepSeek-V3 and Grok-3 also demonstrate exceptional guidance capabilities. Notably, while larger models generally exhibit strong guidance capabilities, some smaller models outperform their larger variants. For instance, in the Glass Assistants (GAs) domain, Qwen2.5-7B-Instruct performs better than Qwen2.5-32B-Instruct. Additionally, Qwen3-32B exhibits better overall performance than Qwen3-235B-A22B in both non-thinking and thinking modes. These results highlight the robust guidance capabilities of current leading LLMs while also reflecting the potential of smaller models in dialogue guidance.

Current thinking models fail to outperform on-thinking models in dialogue guidance. Most of the thinking models exhibit varying degrees of decline in guidance capabilities when compared to their no-thinking counterparts. Only a few models (e.g., Gemini-2.5-Flash-Preview) show a slight improvement in guidance performance in thinking mode. These findings highlight the current limitations of thinking models in effectively steering users toward target objectives during proactive interactions, which may stem from inherent challenges in balancing single-turn reasoning with multi-turn conversational dynamics.

Further Analysis

Effects of Domain and Difficulty

First, model proactivity shows a significant cross-domain imbalance. Even advanced models exhibit substantial gaps between their strongest and weakest domains. This allows smaller models to outperform larger models in specific domains. For example, in target planning, the leading model DeepSeek-R1 excels in Glasses Assistants (GAs) but is surpassed by the smaller Qwen3-14B in Ambiguous Instructions (AIs). Similarly, in dialogue guidance, Qwen3-14B surpasses the Claude-3.7-Sonnet in the AIs domain. Furthermore, certain domains pose universal challenges, with models generally struggling in Persuasion (Per.) for target planning and System Operation (Sys.) for dialogue guidance, highlighting current weaknesses in proactive dialogue.

In addition to domain, task difficulty is also a crucial determinant of performance. As shown in Figure 4(a), the proactivity of all models generally declines as task difficulty increases. However, we find that the performance gap between guiding users with high and mid-level agreeableness is not substantial, likely because models can leverage additional dialogue turns to eventually achieve the target. Moreover, some thinking models demonstrate a distinct advantage when interacting with users of low agreeableness, which is shown in Figure 4(b). Their ability to generate longer, deliberated content with personalized advice and examples better engages resistant users. This shows that reasoning can improve performance in challenging environments.

Effects of Thinking

Our results indicate that thinking significantly enhances the model’s performance in target planning. However, in dialogue guidance, non-thinking models outperform thinking models across all domains. We analyze the impact of thinking on model performance under dialogue guidance through a series of metrics and case studies.

More Pushy Message Content. We introduce a metric annotated by DeepSeek-V3, Target Density, as the number of sub-targets contained per message. As shown in Figure 4(c), we observe two distinct interaction patterns: Models like Qwen and DeepSeek exhibit significantly higher average target density in their thinking versions. More critically, their initiation target density shows even larger gaps, indicating they front-load multiple targets in opening messages rather than fostering user interaction. We show this behavior in Figure 5(A), where the model includes all sub-targets in the first message to push the user for a response. However, models like Gemini-2.5-Flash-Preview and Claude-3.7-Sonnet maintain similar and small target density between thinking and non-thinking versions, with initiation target density also close to average target density. This result suggests they gradually introduce sub-targets through multi-turn interactions. We present this behavior in Figure 5(B), where the model guides users to acceptance gradually.

Decline in Message Naturalness. We find that thinking models generate more messages that do not conform to a standard conversational format. For instance, some models reveal metadata in their messages (e.g., “sub-target 1: …”) or generate multiple turns of dialogue at once without user interaction (e.g., “turn 1: …, turn 2: …”). The cause of this result may be related to a decline in the instruction-following capabilities of the thinking models (Li et al. 2025c). As shown in Figure 4(d), we compare performance on IFEval (Fourrier et al. 2024), a benchmark for instruction-following capabilities (Zhou et al. 2023a), and find that models that perform better on IFEval also tend to exhibit better performance in dialogue guidance.

Change of Initiation Tone. In the dialogue guidance task, we identify a representative initiation template in Persuasion domain: “sounds like…” (shown in Figure 5(C)). It is a passive phrasing and unsuitable for initiation. Therefore, we use this template as an analytical probe, measuring its prevalence in each model’s first message. As shown in Figure 4(e), we find that the adoption of thinking decrease this passive tendency. This result indicates that thinking helps the model to better understand the task requirements of proactive dialogue. Furthermore, we observe that the Qwen and Gemini-2.5-Flash-Preview series models exhibited a higher frequency of this template, while the DeepSeek and Claude-3.7-Sonnet series models perform comparatively better.

Effects of Target

To investigate the importance of target in dialogue guidance, we conduct an experiment where models performed the dialogue guidance task without target. We select two representative models for this test: a smaller model (i.e., Qwen2.5-7B-Instruct) and a top-tier model in both its non-thinking and thinking modes (i.e., Claude-3.7-Sonnet). The models are evaluated on 10 randomly sampled scenarios each domain. As presented in Table 3, the results reveal a stark decline in guidance across all models, demonstrating the critical role of a clear target. Moreover, we find that the decline in guidance performance is significantly greater for the smaller model than for the stronger model, which reflects the smaller model’s greater reliance on an explicit target.

Human Evaluation

We randomly sample 50 generated targets and dialogues from our evaluation results. Next, these targets and dialogues are manually assessed based on the reference and score standards by our researchers. The scores are used to calculate consistency with the evaluation results judged by the LLM. The Kappa test is commonly employed in human evaluation of LLM-as-a-judge work. Thus, we adopt the Weighted Kappa (Cohen 1968) to examine the agreement between human evaluators and the judge model in the evaluation. The results show that, for target planning, weighted kappa between human and LLM evaluation is 0.826. For dialogue guidance, weighted kappa between human and LLM results is 0.721. These results show a great consistency between judge model’s evaluation results and human evaluations.

Conclusion

In this paper, we introduce ProactiveEval, a unified evaluation framework for proactive dialogue Agents. We propose a general definition and evaluation metrics for proactive dialogue tasks to address the current challenge of fragmented task definitions and evaluation methods. Furthermore, we design a synthetic framework for generating evaluation data for proactive dialogue tasks, capable of producing diverse and high-quality evaluation data across multiple domains. Based on the evaluation datasets, we assess 22 LLMs with different types and parameter scales. Our results highlight DeepSeek-R1 and Claude-3.7-Sonnet as top performers in proactivity. Moreover, we emphasize the important role of reasoning capabilities in shaping model proactivity. We hope our framework provides insights and supports progress in proactive dialogue development.

Limiations

While our work establishes a general evaluation framework for proactive dialogue agents, it is subject to several limitations. Firstly, with the rapid evolution of LLM technologies, they are likely to quickly approach the boundaries of current evaluation metrics in target planning and dialogue guidance. Therefore, it is imperative to continue exploring ways to synthesize a more challenging and realistic proactive dialogue environment. In evaluation metrics, although we have designed standards based on existing work on proactive dialogue, there may still be additional factors in real-world settings that affect users’ perceptions of models’ proactive dialogue. Furthermore, despite achieving great consistency between LLM judgment and human evaluation, potential biases and gaps in “LLM-as-a-judge” may still exist in our framework. We plan to regularly update our framework, from the current version to future iterations, to integrate emerging advancements and address these limitations.

The Stability of “LLM-as-a-Judge”

To assess the stability of our judge, we re-ran the evaluation for two representative models (DeepSeek-V3 and DeepSeek-R1) three times. Table 4 below shows the standard deviation of the scores from these runs. The low values demonstrate the high internal consistency and stability of our evaluation framework.

Details of Evalution Models

In this section, we present the details of models used in our experiments.

Non-Thinking Models

: Qwen2.5-7B-Instruct (Yang et al. 2024), Qwen2.5-14B-Instruct, Qwen2.5-32B-Instruct, GPT-4.1, Grok-3, DeepSeek-V3(Liu et al. 2024a), Llama-3.1-8B-Instruct (Dubey et al. 2024), Llama-3.1-405B-Instruct, Llama-4-Scout (Meta AI 2025), Llama-4-Maverick, Qwen3-8B (Yang et al. 2025a), Qwen3-14B, Qwen3-32B, Qwen3-235B-A22B, Qwen3-235B-A22B-2507, Gemini-2.5-Flash-Preview, (DeepMind 2025), and Claude-3.7-Sonnet (Anthropic 2025).

Thinking Models

: R1-Distill-Qwen-7B (Guo et al. 2025), R1-Distill-Qwen-14B, R1-Distill-Qwen-32B, DeepSeek-R1, Qwen3-8B, Qwen3-14B, Qwen3-32B, Qwen3-235B-A22B, Gemini-2.5-Flash-Preview, Claude-3.7-Sonnet, and Gemini-2.5-pro.

Overview of Proactive Dialogue Systems

Table LABEL:tab:proactive_works presents previous works in proactive dialogue, categorized by domain, task, evaluation methods, and metrics.

Evaluation Data Examples by Domain

Each domain is instantiated with a specific user context and a trigger factor that motivates the assistant to initiate a conversation. Table LABEL:tab:domain_examples provides example environments and reference targets.

Prompt of Each Module

Description of Each Domain

Each domain is defined with a specific task and trigger condition. The following dictionary specifies the role and behavior of the assistant in each environment.

Topic Tree Construction

A recursive process generates sub-topics under a parent topic to represent sub-scenarios requiring proactive dialogue.

Data Synthesis

The data synthesis process consists of three stages: environment generation, target generation and target ensemble.

Environment Generation

Generates realistic user scenarios with background information and a trigger factor.

Target Generation

Generates a high-level target and its sub-targets based on the environment.

Target Ensemble

Synthesizes multiple candidate targets into a single best target through expert analysis.

Data Refinement

The data refinement process includes three stages: obfuscation rewrite, noise injection and final check.

Obfuscation Rewrite

Transforms abstract descriptions into concrete, observable behaviors. This process is applied separately to user_information and trigger_factor.

Noise Injection

Embeds key information within a larger context to simulate real-world logs. This process is applied separately to user_information and trigger_factor.

Final Check

Evaluates whether the generated target is optimal within the environment.

Evaluation

Two main evaluation modules are used: Target Planning and Dialogue Guidance.

Target Planning

Assesses the quality of the generated target against a reference.

Dialogue Guidance

Evaluates the assistant’s ability to guide the conversation toward the target.

Target Density

Evaluates the number of sub-targets in each message.

Case of Evaluation Data Synthesis

Case Presentation: Target Ensemble

An example of target ensemble is shown below, where multiple candidate targets are analyzed and synthesized into a final best target.

Case Study: Data Refinement

A case of data refinement is shown in Figure 6.