A systematic literature Review for Transformer-based Software Vulnerability detection

TL;DR

This systematic literature review analyzes 80 studies from 2021 to 2025 on transformer-based models for software vulnerability detection, highlighting trends, challenges, and future research directions.

Contribution

It provides a comprehensive, transformer-centric analysis of recent research, classifying models, datasets, and identifying key technical issues in vulnerability detection.

Findings

Transformer models are classified into encoder, decoder, and combined architectures.

Prevailing research trends include use of source code, logs, and smart contracts.

Key issues include data imbalance, interpretability, scalability, and cross-language generalization.

Abstract

Context: Software vulnerabilities pose significant security threats to software systems, especially as software is increasingly used across many areas of daily life, including health, government, and finance. Recently, transformer-based models have demonstrated promising results in automatic software vulnerability identification due to their robust contextual modelling and representation learning capabilities. Objectives: While numerous systematic literature reviews (SLRs) have examined machine learning and deep learning methods for identifying vulnerabilities, a more transformer-centric analysis remains to be explored. This SLR critically analysed 80 studies published between 2021 and 2025 that utilised transformer models to identify software vulnerabilities. Methods: Using Kitchenhams SLR guidelines, we methodically evaluate current research from various perspectives, encompassing study trends, datasets and sources, programming languages, transformer frameworks, detection detail levels, assessment metrics, reference models, types of vulnerabilities, and experimental configurations. Results: We classify transformer models into encoder, decoder, and combined architectures and analyse both pre-trained and fine-tuned versions utilized on source code, logs, and smart contracts. The results emphasise prevailing research trends, frequently utilised benchmarks, and main baselines. It also uncovers crucial technical issues like data imbalance, interpretability, scalability, and generalization across programming languages. Conclusion: By integrating current evidence and recognising unaddressed research areas, this SLR provides a consolidated resource for researchers and professionals seeking to develop more reliable, precise, and interpretable transformer-based vulnerability identification systems.