Cluster Purge Loss: Structuring Transformer Embeddings for Equivalent Mutants Detection

TL;DR

This paper introduces Cluster Purge Loss, a novel training framework that improves transformer embeddings for equivalent mutant detection by emphasizing intra-class semantic distinctions, leading to better performance and interpretability.

Contribution

The paper proposes Cluster Purge Loss, a new combined loss function that enhances embedding space structuring for code similarity tasks, specifically in detecting equivalent code mutants.

Findings

Achieves state-of-the-art results in mutant detection

Produces more interpretable embedding spaces

Outperforms traditional fine-tuning methods

Abstract

Recent pre-trained transformer models achieve superior performance in various code processing objectives. However, although effective at optimizing decision boundaries, common approaches for fine-tuning them for downstream classification tasks - distance-based methods or training an additional classification head - often fail to thoroughly structure the embedding space to reflect nuanced intra-class semantic relationships. Equivalent code mutant detection is one of these tasks, where the quality of the embedding space is crucial to the performance of the models. We introduce a novel framework that integrates cross-entropy loss with a deep metric learning objective, termed Cluster Purge Loss. This objective, unlike conventional approaches, concentrates on adjusting fine-grained differences within each class, encouraging the separation of instances based on semantical equivalency to the class center using dynamically adjusted borders. Employing UniXCoder as the base model, our approach demonstrates state-of-the-art performance in the domain of equivalent mutant detection and produces a more interpretable embedding space.