Robustness, Security, Privacy, Explainability, Efficiency, and Usability of Large Language Models for Code

TL;DR

This paper systematically reviews 146 studies on large language models for code, focusing on properties beyond accuracy such as robustness, security, privacy, explainability, efficiency, and usability, and discusses current methods and future directions.

Contribution

It provides the first comprehensive review of non-functional properties of LLM4Code, identifying key research gaps and trends.

Findings

Seven key properties beyond accuracy are identified.

Current methods for evaluating these properties are summarized.

Future research directions are proposed.

Abstract

Large language models for code (LLM4Code), which demonstrate strong performance (e.g., high accuracy) in processing source code, have significantly transformed software engineering. Many studies separately investigate the non-functional properties of LM4Code, but there is no systematic review of how these properties are evaluated and enhanced. This paper fills this gap by thoroughly examining 146 relevant studies, thereby presenting the first systematic literature review to identify seven important properties beyond accuracy, including robustness, security, privacy, explainability, efficiency, and usability. We discuss the current state-of-the-art methods and trends, identify gaps in existing research, and present promising directions for future study.