Automated Utterance Labeling of Conversations Using Natural Language Processing

TL;DR

This paper presents an NLP-based automated utterance labeling system for psychological conversations, addressing challenges like multilabel classification and limited data, and demonstrates its effectiveness compared to human labels.

Contribution

It introduces a deep learning approach with domain adaptation and a hierarchical labeling system tailored for psychological conversational data, improving automation and analysis.

Findings

RoBERTa-CON outperforms other machine learning methods

Hierarchical labeling aids strategic data analysis

Automated labels are comparable to human labels in accuracy

Abstract

Conversational data is essential in psychology because it can help researchers understand individuals cognitive processes, emotions, and behaviors. Utterance labelling is a common strategy for analyzing this type of data. The development of NLP algorithms allows researchers to automate this task. However, psychological conversational data present some challenges to NLP researchers, including multilabel classification, a large number of classes, and limited available data. This study explored how automated labels generated by NLP methods are comparable to human labels in the context of conversations on adulthood transition. We proposed strategies to handle three common challenges raised in psychological studies. Our findings showed that the deep learning method with domain adaptation (RoBERTa-CON) outperformed all other machine learning methods; and the hierarchical labelling system that we proposed was shown to help researchers strategically analyze conversational data. Our Python code and NLP model are available at https://github.com/mlaricheva/automated_labeling.