Single- vs. Dual-Prompt Dialogue Generation with LLMs for Job Interviews in Human Resources

TL;DR

This paper compares single- and dual-prompt methods for generating HR job interview dialogues with LLMs, finding that dual-prompt generation produces more human-like interviews despite higher token costs.

Contribution

It introduces and empirically evaluates a dual-prompt dialogue generation method, demonstrating its superiority over single-prompt approaches in creating realistic HR interviews.

Findings

Dual-prompt method yields 2-10 times higher indistinguishability from human interviews.

Dual-prompt interviews require six times more tokens than single-prompt ones.

Results are consistent across different LLMs like GPT-4o and Llama 3.3 70B.

Abstract

Optimizing language models for use in conversational agents requires large quantities of example dialogues. Increasingly, these dialogues are synthetically generated by using powerful large language models (LLMs), especially in domains where obtaining authentic human data is challenging. One such domain is human resources (HR). In this context, we compare two LLM-based dialogue generation methods for producing HR job interviews, and assess which method generates higher-quality dialogues, i.e., those more difficult to distinguish from genuine human discourse. The first method uses a single prompt to generate the complete interview dialogue. The second method uses two agents that converse with each other. To evaluate dialogue quality under each method, we ask a judge LLM to determine whether AI was used for interview generation, using pairwise interview comparisons. We empirically find that, at the expense of a sixfold increase in token count, interviews generated with the dual-prompt method achieve a win rate 2 to 10 times higher than those generated with the single-prompt method. This difference remains consistent regardless of whether GPT-4o or Llama 3.3 70B is used for either interview generation or quality judging.