On Trojan Signatures in Large Language Models of Code

TL;DR

This paper investigates whether trojan signatures, previously found in image models, can be detected in large language models of code, and finds that such signatures do not generalize well to code models, making detection challenging.

Contribution

It is the first study to examine weight-based trojan signature detection in large language models of code and demonstrates the difficulty of such detection methods.

Findings

Trojan signatures do not generalize to large language models of code.

Trojaned code models are resistant to detection even under explicit poisoning settings.

Detecting trojans solely from model weights in code models is a challenging problem.

Abstract

Trojan signatures, as described by Fields et al. (2021), are noticeable differences in the distribution of the trojaned class parameters (weights) and the non-trojaned class parameters of the trojaned model, that can be used to detect the trojaned model. Fields et al. (2021) found trojan signatures in computer vision classification tasks with image models, such as, Resnet, WideResnet, Densenet, and VGG. In this paper, we investigate such signatures in the classifier layer parameters of large language models of source code. Our results suggest that trojan signatures could not generalize to LLMs of code. We found that trojaned code models are stubborn, even when the models were poisoned under more explicit settings (finetuned with pre-trained weights frozen). We analyzed nine trojaned models for two binary classification tasks: clone and defect detection. To the best of our knowledge, this is the first work to examine weight-based trojan signature revelation techniques for large-language models of code and furthermore to demonstrate that detecting trojans only from the weights in such models is a hard problem.