Towards Decoding Developer Cognition in the Age of AI Assistants

TL;DR

This study investigates how AI assistants influence developer cognition and productivity by combining physiological measurements and interaction data to understand actual cognitive load versus perceived productivity.

Contribution

It introduces an empirical approach using physiological and interaction data to analyze the impact of AI assistants on developer cognition and productivity.

Findings

Physiological data reveal differences in cognitive load with AI assistance.

Interaction patterns vary based on developer expertise and AI usage.

Perceived productivity does not always align with measured cognitive load.

Abstract

Background: The increasing adoption of AI assistants in programming has led to numerous studies exploring their benefits. While developers consistently report significant productivity gains from these tools, empirical measurements often show more modest improvements. While prior research has documented self-reported experiences with AI-assisted programming tools, little to no work has been done to understand their usage patterns and the actual cognitive load imposed in practice. Objective: In this exploratory study, we aim to investigate the role AI assistants play in developer productivity. Specifically, we are interested in how developers' expertise levels influence their AI usage patterns, and how these patterns impact their actual cognitive load and productivity during development tasks. We also seek to better understand how this relates to their perceived productivity. Method: We propose a controlled observational study combining physiological measurements (EEG and eye tracking) with interaction data to examine developers' use of AI-assisted programming tools. We will recruit professional developers to complete programming tasks both with and without AI assistance while measuring their cognitive load and task completion time. Through pre- and post-task questionnaires, we will collect data on perceived productivity and cognitive load using NASA-TLX.