Marked Pedagogies: Examining Linguistic Biases in Personalized Automated Writing Feedback

TL;DR

This study investigates how four popular LLMs generate biased personalized feedback in educational settings, revealing systematic stereotypes and biases based on student attributes like race, gender, and learning needs.

Contribution

It uncovers the presence of stereotype-aligned biases in LLM-generated feedback and introduces the concept of Marked Pedagogies to describe these biased instructional orientations.

Findings

Biases in feedback related to race, gender, and disability.

Models emphasize different content and judgment based on student attributes.

Feedback often exhibits positive bias and limited critique for marginalized groups.

Abstract

Effective personalized feedback is critical to students' literacy development. Though LLM-powered tools now promise to automate such feedback at scale, LLMs are not language-neutral: they privilege standard academic English and reproduce social stereotypes, raising concerns about how "personalization" shapes the feedback students receive. We examine how four widely used LLMs (GPT-4o, GPT-3.5-turbo, Llama-3.3 70B, Llama-3.1 8B) adapt written feedback in response to student attributes. Using 600 eighth-grade persuasive essays from the PERSUADE dataset, we generated feedback under prompt conditions embedding gender, race/ethnicity, learning needs, achievement, and motivation. We analyze lexical shifts across model outputs by adapting the Marked Words framework. Our results reveal systematic, stereotype-aligned shifts in feedback conditioned on presumed student attributes--even when essay content was identical. Feedback for students marked by race, language, or disability often exhibited positive feedback bias and feedback withholding bias--overuse of praise, less substantive critique, and assumptions of limited ability. Across attributes, models tailored not only what content was emphasized but also how writing was judged and how students were addressed. We term these instructional orientations Marked Pedagogies and highlight the need for transparency and accountability in automated feedback tools.