Application of Prompt Learning Models in Identifying the Collaborative Problem Solving Skills in an Online Task

TL;DR

This paper introduces a prompt-based learning pre-trained model that effectively identifies collaborative problem solving skills from behavioral data, outperforming traditional models especially with limited training data.

Contribution

It presents a novel prompt-based learning approach that achieves high accuracy in CPS skill coding with minimal training data, advancing automatic behavioral analysis.

Findings

Highest accuracy, macro F1, and kappa on large datasets

Best performance on small training sets

Potential to replace manual coding in CPS assessment

Abstract

Collaborative problem solving (CPS) competence is considered one of the essential 21st-century skills. To facilitate the assessment and learning of CPS competence, researchers have proposed a series of frameworks to conceptualize CPS and explored ways to make sense of the complex processes involved in collaborative problem solving. However, encoding explicit behaviors into subskills within the frameworks of CPS skills is still a challenging task. Traditional studies have relied on manual coding to decipher behavioral data for CPS, but such coding methods can be very time-consuming and cannot support real-time analyses. Scholars have begun to explore approaches for constructing automatic coding models. Nevertheless, the existing models built using machine learning or deep learning techniques depend on a large amount of training data and have relatively low accuracy. To address these problems, this paper proposes a prompt-based learning pre-trained model. The model can achieve high performance even with limited training data. In this study, three experiments were conducted, and the results showed that our model not only produced the highest accuracy, macro F1 score, and kappa values on large training sets, but also performed the best on small training sets of the CPS behavioral data. The application of the proposed prompt-based learning pre-trained model contributes to the CPS skills coding task and can also be used for other CSCW coding tasks to replace manual coding.