Exploring the Effectiveness of Methods for Persona Extraction

TL;DR

This study evaluates various models for persona extraction in Russian dialogues, introducing a new evaluation metric and analyzing how model size and training techniques affect performance.

Contribution

The paper develops a novel F-score based metric for persona extraction and assesses multiple models, highlighting the impact of model size and NCE Loss on performance.

Findings

All models showed low recall in persona extraction.

NCE Loss improved precision but reduced recall.

Larger models achieved better persona extraction results.

Abstract

The paper presents a study of methods for extracting information about dialogue participants and evaluating their performance in Russian. To train models for this task, the Multi-Session Chat dataset was translated into Russian using multiple translation models, resulting in improved data quality. A metric based on the F-score concept is presented to evaluate the effectiveness of the extraction models. The metric uses a trained classifier to identify the dialogue participant to whom the persona belongs. Experiments were conducted on MBart, FRED-T5, Starling-7B, which is based on the Mistral, and Encoder2Encoder models. The results demonstrated that all models exhibited an insufficient level of recall in the persona extraction task. The incorporation of the NCE Loss improved the model's precision at the expense of its recall. Furthermore, increasing the model's size led to enhanced extraction of personas.