On the Effectiveness of Instruction-Tuning Local LLMs for Identifying Software Vulnerabilities

TL;DR

This paper demonstrates that instruction-tuned local LLMs outperform API-based models in identifying specific software vulnerabilities by classifying CWE types, offering a more secure and practical solution.

Contribution

It introduces a reformulation of vulnerability analysis as CWE classification and shows local instruction-tuned LLMs are more effective than API-based models.

Findings

Instruction-tuned local LLMs outperform API-based models in vulnerability classification.

Reformulating the task as CWE identification enhances practical utility.

Local models offer better performance and cost efficiency.

Abstract

Large Language Models (LLMs) show significant promise in automating software vulnerability analysis, a critical task given the impact of security failure of modern software systems. However, current approaches in using LLMs to automate vulnerability analysis mostly rely on using online API-based LLM services, requiring the user to disclose the source code in development. Moreover, they predominantly frame the task as a binary classification(vulnerable or not vulnerable), limiting potential practical utility. This paper addresses these limitations by reformulating the problem as Software Vulnerability Identification (SVI), where LLMs are asked to output the type of weakness in Common Weakness Enumeration (CWE) IDs rather than simply indicating the presence or absence of a vulnerability. We also tackle the reliance on large, API-based LLMs by demonstrating that instruction-tuning smaller, locally deployable LLMs can achieve superior identification performance. In our analysis, instruct-tuning a local LLM showed better overall performance and cost trade-off than online API-based LLMs. Our findings indicate that instruct-tuned local models represent a more effective, secure, and practical approach for leveraging LLMs in real-world vulnerability management workflows.