Role-playing Prompt Framework: Generation and Evaluation

TL;DR

This paper presents a prompt-based framework utilizing GPT for generating role-playing dialogue datasets and evaluating model performance, aiming to reduce resource-intensive manual data collection and assessment.

Contribution

It introduces a novel prompt-based approach for automatic generation and evaluation of role-playing data using GPT, incorporating Rouge-L for performance measurement.

Findings

Effective GPT-based role-playing dataset generation

Automated evaluation correlates well with manual assessments

Reduces resource requirements for data collection and evaluation

Abstract

Large language models (LLMs) exhibit impressive proficiency in natural language generation, understanding user instructions, and emulating human-like language use, which has led to significant interest in their application to role-playing scenarios. However, the manual collection of role-specific script data and the evaluation of model performance are resource-intensive processes. This paper introduces a prompt-based framework designed to leverage GPT's capabilities for the generation of role-playing dialogue datasets and the evaluation of role-playing performance. To validate the effectiveness of the GPT-based generation and evaluation, we further incorporate the recall-oriented Rouge-L metric, providing an additional quantitative measure of performance.