Multitask-based Evaluation of Open-Source LLM on Software Vulnerability

TL;DR

This paper introduces a comprehensive evaluation pipeline for assessing the performance of large language models on software vulnerability tasks, revealing their strengths and areas for improvement.

Contribution

It presents a multi-task evaluation framework using Big-Vul dataset to compare LLMs with pre-trained models across various vulnerability tasks.

Findings

LLMs outperform pre-trained models in vulnerability assessment and location.

Contextual information improves LLM vulnerability assessment.

LLMs excel at vulnerability description but tend to produce excessive output.

Abstract

This paper proposes a pipeline for quantitatively evaluating interactive Large Language Models (LLMs) using publicly available datasets. We carry out an extensive technical evaluation of LLMs using Big-Vul covering four different common software vulnerability tasks. This evaluation assesses the multi-tasking capabilities of LLMs based on this dataset. We find that the existing state-of-the-art approaches and pre-trained Language Models (LMs) are generally superior to LLMs in software vulnerability detection. However, in software vulnerability assessment and location, certain LLMs (e.g., CodeLlama and WizardCoder) have demonstrated superior performance compared to pre-trained LMs, and providing more contextual information can enhance the vulnerability assessment capabilities of LLMs. Moreover, LLMs exhibit strong vulnerability description capabilities, but their tendency to produce excessive output significantly weakens their performance compared to pre-trained LMs. Overall, though LLMs perform well in some aspects, they still need improvement in understanding the subtle differences in code vulnerabilities and the ability to describe vulnerabilities to fully realize their potential. Our evaluation pipeline provides valuable insights into the capabilities of LLMs in handling software vulnerabilities.