Scaffolding Human-AI Collaboration: A Field Experiment on Behavioral Protocols and Cognitive Reframing

TL;DR

This study tests how behavioral protocols and cognitive reframing as scaffolding interventions influence human-AI collaboration effectiveness in a real-world setting.

Contribution

It provides empirical evidence on how structured use protocols and reframing AI as a thought partner affect productivity and perceptions in workplace AI use.

Findings

Structured protocols reduced document quality and output.

Reframing AI as a thought partner improved document quality at the top of the distribution.

Participants showed positive belief changes, possibly due to recovery from carry-over effects.

Abstract

Organizations have widely deployed generative AI tools, yet productivity gains remain uneven, suggesting that how people use AI matters as much as whether they have access. We conducted a field experiment with 388 employees at a Fortune 500 retailer to test two scaffolding interventions for human-AI collaboration. All participants had access to the same AI tool; we varied only the structure surrounding its use. A behavioral scaffolding intervention (a structured protocol requiring joint AI use within pairs) was associated with lower document quality relative to unstructured use and substantially lower document production. A cognitive scaffolding intervention (partnership training that reframed AI as a thought partner) was associated with higher individual document quality at the top of the distribution. Treatment participants also showed greater positive belief change across the session, though sensitivity analyses suggest this likely reflects recovery from carry-over effects rather than genuine training-induced shifts. Both findings are subject to design limitations including an AM/PM session confound, differential attrition, and LLM grading sensitivity to document length.