Understanding the Human-LLM Dynamic: A Literature Survey of LLM Use in Programming Tasks

TL;DR

This survey reviews how large language models are used in programming tasks, analyzing user interactions, benefits, weaknesses, and influencing factors to understand their impact on programming practices.

Contribution

It provides a comprehensive analysis of user interaction behaviors with LLMs in programming, highlighting variability and suggesting directions for future research and practice.

Findings

LLMs show mixed effects on human and task performance.

Interaction patterns vary due to non-determinism of humans and LLMs.

Understanding interaction factors is crucial for improving LLM-assisted programming.

Abstract

Large Language Models (LLMs) are transforming programming practices, offering significant capabilities for code generation activities. While researchers have explored the potential of LLMs in various domains, this paper focuses on their use in programming tasks, drawing insights from user studies that assess the impact of LLMs on programming tasks. We first examined the user interaction behaviors with LLMs observed in these studies, from the types of requests made to task completion strategies. Additionally, our analysis reveals both benefits and weaknesses of LLMs showing mixed effects on the human and task. Lastly, we looked into what factors from the human, LLM or the interaction of both, affect the human's enhancement as well as the task performance. Our findings highlight the variability in human-LLM interactions due to the non-deterministic nature of both parties (humans and LLMs), underscoring the need for a deeper understanding of these interaction patterns. We conclude by providing some practical suggestions for researchers as well as programmers.