PLMGH: What Matters in PLM-GNN Hybrids for Code Classification and Vulnerability Detection

TL;DR

This paper empirically evaluates hybrid models combining pretrained language models and graph neural networks for code classification and vulnerability detection, providing insights and guidelines for their design.

Contribution

It systematically compares various PLM-GNN combinations, revealing key factors affecting performance and robustness in code analysis tasks.

Findings

Hybrid models outperform GNN-only baselines.

PLM choice impacts performance more than GNN architecture.

Larger PLMs are not always better feature extractors.

Abstract

Code understanding models increasingly rely on pretrained language models (PLMs) and graph neural networks (GNNs), which capture complementary semantic and structural information. We conduct a controlled empirical study of PLM-GNN hybrids for code classification and vulnerability detection tasks by systematically pairing three code-specialized PLMs with three foundational GNN architectures. We compare these hybrids against PLM-only and GNN-only baselines on Java250 and Devign, including an identifier-obfuscation setting. Across both tasks, hybrids consistently outperform GNN-only baselines and often improve ranking quality over frozen PLMs. On Devign, performance and robustness are more sensitive to the PLM feature source than to the GNN backbone. We also find that larger PLMs are not necessarily better feature extractors in this pipeline, and that the PLM choice has more impact than the GNN choice. Finally, we distill these findings into practical guidelines for PLM-GNN design choices in code classification and vulnerability detection.