Towards Benchmarking Design Pattern Detection Under Obfuscation: Reproducing and Evaluating Attention-Based Detection Method

TL;DR

This study reproduces an attention-based design pattern detection method and evaluates its robustness against code obfuscation, revealing reliance on superficial features and emphasizing the need for semantically robust detection tools.

Contribution

It reproduces the DPDAtt approach, creates an obfuscated benchmark corpus, and evaluates the method's vulnerability to superficial feature reliance under obfuscation.

Findings

DPDAtt classifiers depend heavily on superficial syntactic features.

Obfuscation significantly reduces detection accuracy.

The curated corpus serves as a benchmark for evaluating semantic robustness.

Abstract

This paper investigates the semantic robustness of attention-based classifiers for design pattern detection, particularly focusing on their reliance on structural and behavioral semantics. We reproduce the DPDAtt, an attention-based design pattern detection approach using learning-based classifiers, and evaluate its performance under obfuscation. To this end, we curate an obfuscated version of the DPDAtt Corpus, where the name identifiers in code such as class names, method names, etc., and string literals like print statements and comment blocks are replaced while preserving control flow, inheritance, and logic. Our findings reveal that these trained classifiers in DPDAtt depend significantly on superficial syntactic features, leading to substantial misclassification when such cues are removed through obfuscation. This work highlights the need for more robust detection tools capable of capturing deeper semantic meanings in source code. We propose our curated Obfuscated corpus (containing 34 Java source files) as a reusable proof-of-concept benchmark for evaluating state-of-the-art design pattern detectors on their true semantic generalization capabilities.