Building Your Own Product Copilot: Challenges, Opportunities, and Needs

TL;DR

This paper explores the challenges and opportunities in developing AI-powered product copilots, based on interviews with engineers, highlighting pain points and proposing collaborative solutions for the software engineering community.

Contribution

It provides empirical insights into the difficulties faced by engineers in building AI copilots and suggests new opportunities and tool designs to address these challenges.

Findings

Identified pain points in AI copilot development process

Challenges that strain current engineering practices

Opportunities for improved tools and collaboration

Abstract

A race is underway to embed advanced AI capabilities into products. These product copilots enable users to ask questions in natural language and receive relevant responses that are specific to the user's context. In fact, virtually every large technology company is looking to add these capabilities to their software products. However, for most software engineers, this is often their first encounter with integrating AI-powered technology. Furthermore, software engineering processes and tools have not caught up with the challenges and scale involved with building AI-powered applications. In this work, we present the findings of an interview study with 26 professional software engineers responsible for building product copilots at various companies. From our interviews, we found pain points at every step of the engineering process and the challenges that strained existing development practices. We then conducted group brainstorming sessions to collaborative on opportunities and tool designs for the broader software engineering community.