Human-AI collaboration or obedient and often clueless AI in instruct, serve, repeat dynamics?

TL;DR

This study investigates human-AI interactions in complex problem-solving, revealing a dominant instructive pattern with limited collaboration and highlighting the need for AI systems to better support cognitive partnership.

Contribution

It provides a detailed analysis of interaction patterns and their evolution, showing current LLMs favor instruction-following over true collaboration.

Findings

Instructive interaction pattern dominates student-AI exchanges.

Long interaction threads often show misalignment and lack of synergy.

No significant link between problem complexity and student performance.

Abstract

While research on human-AI collaboration exists, it mainly examined language learning and used traditional counting methods with little attention to evolution and dynamics of collaboration on cognitively demanding tasks. This study examines human-AI interactions while solving a complex problem. Student-AI interactions were qualitatively coded and analyzed with transition network analysis, sequence analysis and partial correlation networks as well as comparison of frequencies using chi-square and Person-residual shaded Mosaic plots to map interaction patterns, their evolution, and their relationship to problem complexity and student performance. Findings reveal a dominant Instructive pattern with interactions characterized by iterative ordering rather than collaborative negotiation. Oftentimes, students engaged in long threads that showed misalignment between their prompts and AI output that exemplified a lack of synergy that challenges the prevailing assumptions about LLMs as collaborative partners. We also found no significant correlations between assignment complexity, prompt length, and student grades suggesting a lack of cognitive depth, or effect of problem difficulty. Our study indicates that the current LLMs, optimized for instruction-following rather than cognitive partnership, compound their capability to act as cognitively stimulating or aligned collaborators. Implications for designing AI systems that prioritize cognitive alignment and collaboration are discussed.