VuLASTE: Long Sequence Model with Abstract Syntax Tree Embedding for vulnerability Detection

TL;DR

VuLASTE is a novel long sequence model for vulnerability detection that combines AST path embeddings, byte-level BPE, and advanced attention mechanisms, achieving superior results on a cross-language dataset.

Contribution

The paper introduces VuLASTE, integrating AST path embeddings and Longformer-based attention to improve vulnerability detection in source code.

Findings

VuLASTE outperforms state-of-the-art methods on a cross-language vulnerability dataset.

The model effectively handles long sequences and class imbalance.

It achieves higher top-k hit rates than previous approaches.

Abstract

In this paper, we build a model named VuLASTE, which regards vulnerability detection as a special text classification task. To solve the vocabulary explosion problem, VuLASTE uses a byte level BPE algorithm from natural language processing. In VuLASTE, a new AST path embedding is added to represent source code nesting information. We also use a combination of global and dilated window attention from Longformer to extract long sequence semantic from source code. To solve the data imbalance problem, which is a common problem in vulnerability detection datasets, focal loss is used as loss function to make model focus on poorly classified cases during training. To test our model performance on real-world source code, we build a cross-language and multi-repository vulnerability dataset from Github Security Advisory Database. On this dataset, VuLASTE achieved top 50, top 100, top 200, top 500 hits of 29, 51, 86, 228, which are higher than state-of-art researches.