APILOT: Navigating Large Language Models to Generate Secure Code by Sidestepping Outdated API Pitfalls

TL;DR

APILOT is a system that improves large language models' ability to generate secure, up-to-date code by maintaining a real-time dataset of outdated APIs and guiding LLMs to avoid deprecated or vulnerable APIs.

Contribution

This work introduces APILOT, a novel approach that leverages a quickly updatable dataset and augmented generation to produce safer, version-aware code from LLMs, addressing outdated API issues.

Findings

Reduces outdated API recommendations by 89.42% on average

Improves code usability by 27.54%

Effective across seven state-of-the-art LLMs

Abstract

With the rapid development of large language models (LLMs), their applications have expanded into diverse fields, such as code assistance. However, the substantial size of LLMs makes their training highly resource- and time-intensive, rendering frequent retraining or updates impractical. Consequently, time-sensitive data can become outdated, potentially misleading LLMs in time-aware tasks. For example, new vulnerabilities are discovered in various programs every day. Without updating their knowledge, LLMs may inadvertently generate code that includes these newly discovered vulnerabilities. Current strategies, such as prompt engineering and fine-tuning, do not effectively address this issue. To address this issue, we propose solution, named APILOT, which maintains a realtime, quickly updatable dataset of outdated APIs. Additionally, APILOT utilizes an augmented generation method that leverages this dataset to navigate LLMs in generating secure, version-aware code. We conducted a comprehensive evaluation to measure the effectiveness of APILOT in reducing the incidence of outdated API recommendations across seven different state-of-the-art LLMs. The evaluation results indicate that APILOT can reduce outdated code recommendations by 89.42% on average with limited performance overhead. Interestingly, while enhancing security, APILOT also improves the usability of the code generated by LLMs, showing an average increase of 27.54% in usability. This underscores APILOT's dual capability to enhance both the safety and practical utility of code suggestions in contemporary software development environments.