Human-AI Collaboration Increases Skill Tagging Speed but Degrades Accuracy

TL;DR

This study investigates human-AI collaboration in educational skill tagging, showing that AI assistance significantly speeds up the process but reduces accuracy and recall, highlighting trade-offs in collaborative AI applications.

Contribution

It provides empirical data on the effects of AI assistance in educational content tagging, revealing speed gains and accuracy trade-offs in a real-world task.

Findings

AI assistance reduced tagging time by 50%

AI collaboration decreased accuracy by 35%

Humans still exercised discernment with AI recommendations

Abstract

AI approaches are progressing besting humans at game-related tasks (e.g. chess). The next stage is expected to be Human-AI collaboration; however, the research on this subject has been mixed and is in need of additional data points. We add to this nascent literature by studying Human-AI collaboration on a common administrative educational task. Education is a special domain in its relation to AI and has been slow to adopt AI approaches in practice, concerned with the educational enterprise losing its humanistic touch and because standard of quality is demanded because of the impact on a person's career and developmental trajectory. In this study (N = 22), we design an experiment to explore the effect of Human-AI collaboration on the task of tagging educational content with skills from the US common core taxonomy. Our results show that the experiment group (with AI recommendations) saved around 50% time (p < 0.01) in the execution of their tagging task but at the sacrifice of 7.7% recall (p = 0.267) and 35% accuracy (p= 0.1170) compared with the non-AI involved control group, placing the AI+human group in between the AI alone (lowest performance) and the human alone (highest performance). We further analyze log data from this AI collaboration experiment to explore under what circumstances humans still exercised their discernment when receiving recommendations. Finally, we outline how this study can assist in implementing AI tools, like ChatGPT, in education.