Codellm-Devkit: A Framework for Contextualizing Code LLMs with Program Analysis Insights

TL;DR

Codellm-Devkit is an open-source Python library that simplifies integrating program analysis insights across multiple languages to enhance code LLMs' performance in various coding tasks.

Contribution

It introduces a user-friendly framework that makes static analysis tools more accessible and easier to integrate with code LLMs, overcoming language-specific and complexity barriers.

Findings

Simplifies program analysis integration for code LLMs

Supports multiple programming languages with a unified interface

Enhances the effectiveness of code LLMs in practical tasks

Abstract

Large Language Models for Code (or code LLMs) are increasingly gaining popularity and capabilities, offering a wide array of functionalities such as code completion, code generation, code summarization, test generation, code translation, and more. To leverage code LLMs to their full potential, developers must provide code-specific contextual information to the models. These are typically derived and distilled using program analysis tools. However, there exists a significant gap--these static analysis tools are often language-specific and come with a steep learning curve, making their effective use challenging. These tools are tailored to specific program languages, requiring developers to learn and manage multiple tools to cover various aspects of the their code base. Moreover, the complexity of configuring and integrating these tools into the existing development environments add an additional layer of difficulty. This challenge limits the potential benefits that could be gained from more widespread and effective use of static analysis in conjunction with LLMs. To address this challenge, we present codellm-devkit (hereafter, `CLDK'), an open-source library that significantly simplifies the process of performing program analysis at various levels of granularity for different programming languages to support code LLM use cases. As a Python library, CLDK offers developers an intuitive and user-friendly interface, making it incredibly easy to provide rich program analysis context to code LLMs. With this library, developers can effortlessly integrate detailed, code-specific insights that enhance the operational efficiency and effectiveness of LLMs in coding tasks. CLDK is available as an open-source library at https://github.com/IBM/codellm-devkit.