Counterargument for Critical Thinking as Judged by AI and Humans

TL;DR

This study explores the use of counterarguments in student writing to enhance critical thinking in the context of Generative AI, demonstrating that AI can reliably assess such work with alignment to human judgments.

Contribution

It introduces a mixed-method approach to evaluate student counterarguments using both human and AI assessments, highlighting AI's potential in scalable evaluation.

Findings

Students' counterarguments contain logic, a key aspect of critical thinking.

AI models can assess student work with moderate agreement to human raters.

The study supports using AI for large-scale assessment of critical thinking skills.

Abstract

This intervention study investigates the use of counterarguments in writing for critical thinking by students in the context of Generative AI (GenAI). This is especially as risks of cheating and cognitive offloading exist with the use of GenAI. We presented 36 students in a particular university course with 4 carefully selected thesis statements (from a set of popular debates) to write about anyone of them. We used six established rubrics (focus, logic, content, style, correctness and reference) to conduct three human assessments (two student peer-reviews and one experienced teacher) per writeup on a 5-point Likert scale for all the qualified samples (n) of 35 submissions (after disqualifying one for irregularity). Using the same rubrics and guidelines, we also assessed the submissions using six frontier LLMs as judges. Our mixed-method design included qualitative open-ended feedback per assessment and quantitative methods. The results reveal that (1) the students' self-written counterarguments to AI-generated content contains logic, among other things, which is a key component of critical thinking, and (2) GenAI can be successfully used at scale to assess students' written work, based on clear rubrics, and these assessments generally align with human assessments as shown with Gwets AC2 inter-rater reliability values of 0.33 for all the models except one.