Reasoning Language Models for complex assessments tasks: Evaluating parental cooperation from child protection case reports

TL;DR

This study evaluates reasoning language models' ability to assess parental cooperation in child protection reports, showing high accuracy and highlighting differences in assessment accuracy between mothers and fathers.

Contribution

It introduces a novel workflow for using reasoning language models to evaluate complex case factors in child protection reports, outperforming previous methods.

Findings

Largest RLM achieved 89% accuracy

Higher accuracy for mothers (93%) than fathers (85%)

RLMs effectively assess complex reasoning tasks

Abstract

Purpose: Reasoning language models (RLMs) have demonstrated significant advances in solving complex reasoning tasks. We examined their potential to assess parental cooperation during CPS interventions using case reports, a case factor characterized by ambiguous and conflicting information. Methods: A four stage workflow comprising (1) case reports collection, (2) reasoning-based assessment of parental cooperation, (3) automated category extraction, and (4) case labeling was developed. The performance of RLMs with different parameter sizes (255B, 32B, 4B) was compared against human validated data. Two expert human reviewers (EHRs) independently classified a weighted random sample of reports. Results: The largest RLM achieved the highest accuracy (89%), outperforming the initial approach (80%). Classification accuracy was higher for mothers (93%) than for fathers (85%), and EHRs exhibited similar differences. Conclusions: RLMs' reasoning can effectively assess complex case factors such as parental cooperation. Lower accuracy in assessing fathers' cooperation supports the argument of a stronger professional focus on mothers in CPS interventions.