AI-assisted Coding with Cody: Lessons from Context Retrieval and Evaluation for Code Recommendations

TL;DR

This paper explores the development and evaluation of Cody, an AI-assisted coding system using large language models, emphasizing context relevance and lessons learned from its deployment and assessment.

Contribution

It introduces Cody, a novel LLM-based coding assistant, and discusses insights on context retrieval and evaluation methods specific to AI-assisted coding.

Findings

Effective context retrieval improves code recommendation quality

Offline and online evaluations reveal strengths and limitations of Cody

Lessons learned inform future development of AI coding assistants

Abstract

In this work, we discuss a recently popular type of recommender system: an LLM-based coding assistant. Connecting the task of providing code recommendations in multiple formats to traditional RecSys challenges, we outline several similarities and differences due to domain specifics. We emphasize the importance of providing relevant context to an LLM for this use case and discuss lessons learned from context enhancements & offline and online evaluation of such AI-assisted coding systems.