The Hidden Cost of Contextual Sycophancy: an AI Literacy Intervention in Human-AI Collaboration

TL;DR

This study examines how large language models tend to mirror user errors in collaborative tasks and evaluates whether AI literacy training can reduce this sycophantic behavior.

Contribution

It provides empirical evidence on the emergence of sycophantic alignment in human-AI interactions and assesses the impact of targeted prompting training.

Findings

Lower-quality user inputs lead to poorer AI advice.

Propagation of user errors reduces AI and user performance.

Prompting training improves AI advice but does not eliminate error propagation.

Abstract

Large Language Models (LLMs) are increasingly used in educational settings as interactive tools for collaboration. However, their tendency toward sycophancy, aligning with user beliefs even when incorrect, raises concerns for learning and decision-making, especially for less knowledgeable users. This study investigates how sycophantic alignment emerges in authentic multi-turn human-AI interactions and whether interventions targeting increasing AI literacy and prompting competencies can mitigate its effects. In a controlled mixed-design experiment, 60 participants completed analytical survival ranking tasks by first generating individual rankings and then making final decisions after collaborating with an AI assistant, both before and after receiving either general or sycophancy-focused prompting training. Preliminary results show that LLMs are highly sensitive to user input: lower-quality initial responses lead to poorer AI advice, suggesting that the model mirrors or incorporates user reasoning rather than correcting it or offering better alternatives that are missing or less frequent in the conversation. Critically, the propagation of user errors into AI responses significantly reduced both the quality of AI feedback and final user task performance, revealing a form of contextual sycophantic dependence. While the intervention did not eliminate the propagation of contextual errors, it significantly improved AI advice by reducing the direct mirroring of incorrect user rankings. These findings suggest that prompting and AI literacy alone may be insufficient to ensure epistemically independent AI support, highlighting the need for system-level approaches that better promote critical engagement in human-AI collaboration.