Detecting Hard-Coded Credentials in Software Repositories via LLMs

TL;DR

This paper explores the use of Large Language Models, like GPT, to improve detection of hard-coded credentials in software repositories, achieving a 13% higher F1 score than previous methods.

Contribution

It introduces a novel approach using LLM embeddings combined with deep classifiers to better identify hard-coded credentials, surpassing existing state-of-the-art performance.

Findings

Outperforms current methods by 13% in F1 score

Utilizes LLM embeddings for improved context understanding

Provides publicly available code and data for reproducibility

Abstract

Software developers frequently hard-code credentials such as passwords, generic secrets, private keys, and generic tokens in software repositories, even though it is strictly advised against due to the severe threat to the security of the software. These credentials create attack surfaces exploitable by a potential adversary to conduct malicious exploits such as backdoor attacks. Recent detection efforts utilize embedding models to vectorize textual credentials before passing them to classifiers for predictions. However, these models struggle to discriminate between credentials with contextual and complex sequences resulting in high false positive predictions. Context-dependent Pre-trained Language Models (PLMs) or Large Language Models (LLMs) such as Generative Pre-trained Transformers (GPT) tackled this drawback by leveraging the transformer neural architecture capacity for self-attention to capture contextual dependencies between words in input sequences. As a result, GPT has achieved wide success in several natural language understanding endeavors. Hence, we assess LLMs to represent these observations and feed extracted embedding vectors to a deep learning classifier to detect hard-coded credentials. Our model outperforms the current state-of-the-art by 13% in F1 measure on the benchmark dataset. We have made all source code and data publicly available to facilitate the reproduction of all results presented in this paper.