An Empirical Study of Bots in Software Development -- Characteristics and Challenges from a Practitioner's Perspective

TL;DR

This empirical study explores the characteristics, usage, and challenges of software engineering bots (DevBots) in industry, highlighting the diversity of user perceptions and the need for clearer classifications and more advanced, transformative bots.

Contribution

It provides a detailed empirical analysis of DevBot usage in industry, identifying user personas and proposing a framework for future research and development.

Findings

Three user personas with different DevBot definitions and uses

Lack of advanced 'smart' bots beyond simple automation and chat interfaces

Potential for transformative impact with more capable DevBots

Abstract

Software engineering bots - automated tools that handle tedious tasks - are increasingly used by industrial and open source projects to improve developer productivity. Current research in this area is held back by a lack of consensus of what software engineering bots (DevBots) actually are, what characteristics distinguish them from other tools, and what benefits and challenges are associated with DevBot usage. In this paper we report on a mixed-method empirical study of DevBot usage in industrial practice. We report on findings from interviewing 21 and surveying a total of 111 developers. We identify three different personas among DevBot users (focusing on autonomy, chat interfaces, and "smartness"), each with different definitions of what a DevBot is, why developers use them, and what they struggle with. We conclude that future DevBot research should situate their work within our framework, to clearly identify what type of bot the work targets, and what advantages practitioners can expect. Further, we find that there currently is a lack of general purpose "smart" bots that go beyond simple automation tools or chat interfaces. This is problematic, as we have seen that such bots, if available, can have a transformative effect on the projects that use them.