Exploring the Effectiveness of Methods for Persona Extraction
Konstantin Zaitsev

TL;DR
This study evaluates various models for persona extraction in Russian dialogues, introducing a new evaluation metric and analyzing how model size and training techniques affect performance.
Contribution
The paper develops a novel F-score based metric for persona extraction and assesses multiple models, highlighting the impact of model size and NCE Loss on performance.
Findings
All models showed low recall in persona extraction.
NCE Loss improved precision but reduced recall.
Larger models achieved better persona extraction results.
Abstract
The paper presents a study of methods for extracting information about dialogue participants and evaluating their performance in Russian. To train models for this task, the Multi-Session Chat dataset was translated into Russian using multiple translation models, resulting in improved data quality. A metric based on the F-score concept is presented to evaluate the effectiveness of the extraction models. The metric uses a trained classifier to identify the dialogue participant to whom the persona belongs. Experiments were conducted on MBart, FRED-T5, Starling-7B, which is based on the Mistral, and Encoder2Encoder models. The results demonstrated that all models exhibited an insufficient level of recall in the persona extraction task. The incorporation of the NCE Loss improved the model's precision at the expense of its recall. Furthermore, increasing the model's size led to enhanced…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsPersona Design and Applications
