Feature Maps: A Comprehensible Software Representation for Design Pattern Detection

TL;DR

This paper introduces Feature Maps, a novel human- and machine-understandable representation of software micro-structures, enabling effective detection of design patterns in source code using machine learning techniques.

Contribution

It proposes Feature Maps and Feature-Role Normalization, a new approach for representing micro-structures that improves pattern detection accuracy and interpretability in software analysis.

Findings

Robust detection of design patterns with imbalanced data

Feature maps outperform traditional methods in accuracy

Both classical and deep learning models benefit from feature maps

Abstract

Design patterns are elegant and well-tested solutions to recurrent software development problems. They are the result of software developers dealing with problems that frequently occur, solving them in the same or a slightly adapted way. A pattern's semantics provide the intent, motivation, and applicability, describing what it does, why it is needed, and where it is useful. Consequently, design patterns encode a well of information. Developers weave this information into their systems whenever they use design patterns to solve problems. This work presents Feature Maps, a flexible human- and machine-comprehensible software representation based on micro-structures. Our algorithm, the Feature-Role Normalization, presses the high-dimensional, inhomogeneous vector space of micro-structures into a feature map. We apply these concepts to the problem of detecting instances of design patterns in source code. We evaluate our methodology on four design patterns, a wide range of balanced and imbalanced labeled training data, and compare classical machine learning (Random Forests) with modern deep learning approaches (Convolutional Neural Networks). Feature maps yield robust classifiers even under challenging settings of strongly imbalanced data distributions without sacrificing human comprehensibility. Results suggest that feature maps are an excellent addition in the software analysis toolbox that can reveal useful information hidden in the source code.

Figures

Tables

Table 1. Table I: The 9 projects contained in the P-MARt dataset. For every project the original and the revised (excluding instances outside of the available source code) number of design pattern instances (i.e. mappings) as well as the number of unique mappings is given.

Project	Original / Revised / Unique
	Singleton	Template Method	Composite	Decorator
075em. JHotDraw	2 / 2 / 2	68 / 34 / 2	3840 / 960 / 1	176 / 44 / 1
JRefactory	2 / 2 / 2	0 / 0 / 0	0 / 0 / 0	0 / 0 / 0
JUnit	2 / 2 / 2	0 / 0 / 0	222 / 74 / 1	396 / 132 / 1
Lexi	2 / 2 / 2	0 / 0 / 0	0 / 0 / 0	0 / 0 / 0
MapperXML	3 / 3 / 3	88 / 44 /4	340 / 85 / 1	0 / 0 / 0
Netbeans	- / - / -	- / - / -	- / - / -	- / - / -
Nutch	1 / 1 / 1	14 / 7 / 2	0 / 0 / 0	0 / 0 / 0
PMD	0 / 0 / 0	1 / 0 / 0	3 / 0 / 0	0 / 0 / 0
QuickUML	1 / 1 / 1	0 / 0 / 0	120 / 30 / 2	0 / 0 / 0
075em. Total	13 / 13 / 13	171 / 85 / 8	4525 / 1149 / 5	572 / 176 / 2

Table 2. Table II: Average test-performance of the Convolutional Neural Network (CNN) and Random Forest (RF) based detections.

Model	Pattern	Accuracy	Precision	Recall	F1	MCC
075em. CNN	Singleton	$0.88\pm 0.09$	$0.82\pm 0.12$	$0.73\pm 0.12$	$0.76\pm 0.10$	$0.72\pm 0.14$
	Template Method	$0.82\pm 0.09$	$0.63\pm 0.18$	$0.62\pm 0.20$	$0.58\pm 0.14$	$0.51\pm 0.12$
	Composite	$0.94\pm 0.02$	$0.86\pm 0.11$	$0.80\pm 0.14$	$0.82\pm 0.10$	$0.79\pm 0.10$
	Decorator	$0.81\pm 0.10$	$0.69\pm 0.15$	$0.80\pm 0.17$	$0.72\pm 0.13$	$0.60\pm 0.16$
075em. RF	Singleton	$0.81\pm 0.07$	$0.63\pm 0.12$	$0.59\pm 0.12$	$0.59\pm 0.06$	$0.47\pm 0.06$
	Template Method	$0.85\pm 0.04$	$0.66\pm 0.21$	$0.58\pm 0.16$	$0.59\pm 0.15$	$0.49\pm 0.12$
	Composite	$0.93\pm 0.04$	$0.89\pm 0.10$	$0.71\pm 0.21$	$0.77\pm 0.16$	$0.79\pm 0.14$
	Decorator	$0.45\pm 0.06$	$0.10\pm 0.15$	$0.09\pm 0.13$	$0.09\pm 0.12$	$-0.35\pm 0.10$

Equations

$$m_{P^k}=\{(\mathbf{a},\mathbf{b})\in {\mathbf{C}}^k\times {\mathbf{R}}^k:\mathbf{a}\text{ complies with }\mathbf{b}\},$$

$$⟨m{⟩}_{{\sim }_{P^k}}=\{x\in {\mathbf{M}}_{P^k}:x{\sim }_{P^k}m\}$$

Full text

I Introduction

Design Patterns (DPs) are elegant and well-tested solutions to recurrent software development problems. Design Patterns – Elements of Reusable Object-Oriented Software, by Gamma et al. [1], is the best-known collection of patterns and inspiration for many follow-ups. They are the result of software developers dealing with problems that frequently occur, solving them in the same or a slightly adapted way. DPs are the generalization of the different adapted implementations, such that they can be reused and applied over and over again in different situations. They solve high-level Object-Oriented (OO) architectural problems dealing with creation, structure or behavior of a small set of classes or objects, but may also circumvent deficiencies and inflexibilities in OO languages.

Pattern descriptions are very detailed and contain their name, intent, motivation, where they are applicable, structures, participants, collaborations, and so forth [1]. A pattern’s semantic is given by the intent, motivation, and applicability, which describe what the pattern does, why the pattern is needed, and where it is useful. Developers weave this information encoded as design patterns into their system as they use them to solve problems. The usage of a pattern is related to specific design decisions during development. However, fast development cycles often prohibit the documentation of these decisions and their rationales. Similarly, the actual usage of the pattern is seldom documented. Hence decision, rationale and their materialization in the form of the pattern’s implementation are lost in the system’s source code. Retrieving this encoded information such that development, redevelopment, and maintenance can profit from it is the primary motivation of Design Pattern Detection (DPD). For the sake of simplicity, we will use the term design pattern detection to describe the process of detecting instances of design patterns. DPD is especially useful for preliminary analysis in maintenance and testing scenarios where the code is unknown or undocumented. The detected patterns hint at structures and dependencies, highlight algorithms and their moving parts, and help to find performance-critical regions.

The biggest challenge in DPD is that patterns are only a guideline for implementing a specific solution; hence each pattern can be implemented in various ways. Each pattern implementation variant resembles the original intent of the pattern but may diverge drastically if compared to other implementation variants. Detecting all variants and mapping the classes to pattern roles is a non-trivial task as enumerating the different variants is not sufficient for real-world setups and their inherent variations.

We use Feature Maps (FMs) as input to Random Forests (RFs) [2] and Convolutional Neural Networks (CNNs) [3] to decide whether a given set of classes maps to the roles of a specific design pattern. These feature maps are human-interpretable, stacked, named subtrees (micro-structures) extracted from a system’s Abstract Semantics Graph (ASG) that can be used as a software representation for a small set of classes. Results indicate that feature maps are an excellent approach to represent software, enabling robust DPD even if DP instances are highly under-represented. More specifically, the contributions of this work are:

• •

a new, flexible and comprehensible software representation called feature maps that are useful for software analysis,

• •

an approach for detecting instances of design patterns in source code by using feature maps in conjunction with supervised machine learning,

• •

an evaluation and methodology proposal of reproducible and comparable design pattern detection.

Section II provides background to design patterns and machine learning. Section III describes our approach for detecting design pattern instances with feature maps. Section IV presents the experiment setup and evaluation results. A general discussion is given in Section V and threats to validity are discussed in Section VI. Section VII summarizes related research and Section VIII concludes this work and offers prospects of design pattern detection with feature maps and machine learning.

II Background

Two ingredients are crucial for the approach within this work. First, Design Patterns (DPs) and their intricacies from a Design Pattern (Instance) Detection (DPD) perspective influence how detectors are built and evaluated. Thus a more formal definition helps to frame the problem and its solution. Secondly, Machine Learning (ML) and its automation capabilities concerning data modeling are essential to understand the approach and its evaluation.

II-A Design Patterns

A Design Pattern (DP) is a set of roles to which participating classes are mapped. These classes communicate (structure and collaboration) in an organized fashion and have specific semantics (intent, motivation, and applicability) attached to them. Design pattern (instance) detection reconstructs the original mappings between classes of a system and roles of a pattern with respect to their communication such that the attached semantics provide information about the system under inspection.

Figure 1 (based on an example by Gamma et al. [1]) illustrates this process in which the input is a set of classes, and the output is a set of mappings between these classes and the roles of a specific pattern. In other words, the task is to annotate a set of classes with the roles of a design pattern. The example is a subsystem of a drawing tool that uses the Composite pattern to interact, in a uniform way, with the drawing primitives and the scene. The Composite pattern organizes objects into a tree structure where internal nodes delegate specific calls while leaf nodes implement the actual behavior of the call. Graphic is the superclass of Picture, Line, and Rectangle and defines a common interface for these. Picture aggregates Graphic objects, and delegates calls to the draw-method. Line and Rectangle provide the actual implementation of draw, hence are called leafs. The goal is now to find a process that reconstructs the original mappings between classes and pattern roles indicated by the question mark in Figure 1. The left side in Figure 1 represents the initial state of the system before the detection process. The right side shows the system with annotated roles after the detection process. Mapping 1, for example, maps Graphic $\mapsto$ Component, Picture $\mapsto$ Composite and Line $\mapsto$ Leaf.

Each role mapping assigns at least one class to one role resulting in multiple possible role mappings for the same subsystem given various implementations of the different roles. Mappings 1 and 2 differ only in one role mapping which is a common scheme in design patterns. Primary roles define the communication scheme within the pattern and drive the communication through a pattern’s class structure, hence are often abstractions. Secondary roles provide the implementation for the abstractions and inherit the protocol from the primary roles, thus are commonly fluctuating in their class assignment. A system usually provides multiple versions of the secondary roles but only a handful of different implementations for the primary roles. All mappings that share the same primary roles belong to the same unique role mapping representing one specific implementation of a pattern within a subsystem. More formally, a pattern mapping is a $k$-fold relation between a set of classes $\bm{C}$ and a set of roles $\bm{R}$ with

[TABLE]

in which $P^{k}$ is a specific pattern with $k$ roles. Each unique role mapping reflects an equivalence class in which the primary roles are compared. Given a set of pattern mappings $\bm{M}_{P^{k}}$ with the equivalence relation $\sim_{P^{k}}:M_{P^{k}}\times M_{P^{k}}$ in which the classes mapped to the primary roles are compared, then

[TABLE]

represents the $P^{k}$ equivalence class from $m$. That is, all role mappings of pattern $P$ with $k$ roles that have the same classes $\bm{C}$ mapped to the primary roles $\bm{R}$. In Figure 1, Graphic maps to the Component role representing the primary role of the Composite pattern. Line, Rectangle, and Picture map to the secondary roles where Mappings 1-2 belong to the Unique Mapping A. Tertiary roles (e.g., Client) that solely function as an entry point for the pattern’s communication sequence are often ignored in detection processes as they do not carry any useful information. The concept of unique mapping carries importance in analysis settings where they explicitly state the boundaries for design pattern instances.

II-B Machine Learning

Machine Learning (ML) describes methods that learn relationships or structure from data in an automated fashion. Essential elements of ML are data (e.g., images or time series), the model (e.g., Random Forest [2] or CNN [3]), the optimization procedure (e.g., Adam [4]) and the evaluation procedure (e.g., Cross-Validation [5] or test-set [6] method).

II-B1 Data

The DPD problem can be framed as a supervised classification problem in which observations are annotated with the ground truth (e.g., is an instance of Composite, is not an instance of Composite). ML algorithms often expect the data to be independent and identically distributed (i.i.d.) meaning that the observations are mutually independent and collected in the same fashion. Especially observation independence is vital as violating the assumption results over-optimistic model performance in classification settings.

Data can be preprocessed by standardizing or rescaling the values into a specific range. In addition to normalizing, data permutation is often employed to increase the amount of available data or to fit models that are robust against simple transformations (i.e., reduce overfitting) [7]. For instance, rotating or mirroring an image does not change the objects it shows (e.g., rotated apple is still an apple).

II-B2 Model

Many different supervised machine learning models of varying complexity and with different strengths and weaknesses exist.

Convolutional Neural Networks (CNNs) [3] are fundamental building blocks in modern machine learning and are prominent for their capabilities for computer vision problems. They learn local correlations within volumes and combine these correlations into high-level features at later stages. For instance, nearby pixels often correlate with each other, e.g., color and texture of an apple on the top right corner of an image. CNNs innately model such local correlations leading to good results in many domains while still having a reasonable amount of model parameters.

A Decision Tree [8] recursively partitions the input space via axis-parallel splits that can be represented as a tree. Each leaf carries a response (e.g., a pattern class) while the nodes represent boolean conditions on an input dimension (e.g., is an interface $\mapsto$ {yes, no}). A Random Forest (RF) [2] contains multiple randomly perturbed decision trees, that reduce the possible high variance of a single tree. The trees are perturbed by fitting each on a subset of the data which creates smoother decision boundaries as multiple splits are averaged in the final prediction.

Hyper-parameter optimization searches the space of possible parameter configurations of an ML algorithm that leads to the best performing model according to some predefined goal function. For instance, tree depth or partition criterion are hyper-parameters of a decision tree.

II-B3 Evaluation

It is crucial that the evaluation procedure provides a measurement of the model that is truthful, i.e., it should not over- or underestimate the performance. Cross-Validation [9] (CV) is a method to evaluate the generalization performance by splitting the dataset into $k$-folds. $k-1$ folds are used to fit a model while the remaining fold is used to estimate its generalization performance. This process is repeated $k$ times leading to $k$ models and estimates that are averaged into a global generalization performance of the ML algorithm regarding the dataset and parameters. CV is a nearly unbiased estimator for the generalization performance except in the case of small datasets in which the evaluation variance may misestimate the generalization. Repeated cross-validation may be used to reduce the variance, trading it for some bias.

Accuracy, Precision, Recall or the Matthews Correlation Coefficient (MCC) [10, 11] are performance metrics for classification models. Accuracy, Recall, and Precision are frequently used metrics and need no further explanation. MCC is a reliable and balanced performance metric for binary classification ranging from -1 to 1 describing the strength of association between model prediction and ground truth. It provides the most accurate measurement of a (binary) model’s performance even if the dataset is imbalanced (skewed distribution of prediction labels). This does not hold for Accuracy, Precision and Recall as they do not consider all types of correctly or incorrectly predicted instances (true/false positive, true/false negatives).

III Design Pattern Instance Detection Pipeline

Figure 2 shows a typical multistage DPD pipeline similar to many DP detectors [12, 13, 14, 15, 16, 17]. Rectangles are artifacts while ellipses are processes. Source Code and Probability are the input and output of the entire DPD pipeline while Micro-structures, Candidate Role Mappings and Feature Maps are intermediate artifacts. Feature Extraction extracts distinctive features from the source code. Candidate Sampling reduces the total number of mappings that need to be evaluated. Feature Normalization transforms the features into a homogeneous space such that the Design Pattern Inference can reason efficiently about it.

III-A Feature Extraction

Feature extraction deals with the inherent complexity of programming languages by extracting high-level concepts that later stages can use to successfully find pattern instances (see Figure 2). DPs describe classes and their loose arrangements and communication paths. Consequently, extracted features ideally capture these arrangements and their relationships to improve reasoning in later stages.

We use Micro-Structures (MSs) [18] as features, which are very small DPs (usually one or two roles). A MS describes a general structural or behavioral property within or between a set of classes (e.g., self-reference, inheritance or method calls). Casting MSs into an easily readable form allows developers and algorithms to effectively comprehend these properties. This includes fast discovery of inheritance and call dependencies but also complex aspects like the implementation of a Composite or Decorator pattern.

The most outstanding difference between MSs and DPs is that the former can be detected using logic or pattern matching, i.e., their size prohibits variance in their actual implementation. MS detectors are sub-graph filters retaining only sub-graphs that match their specified predicate. For example, $Inheritance(sup,sub):\mathbb{T}^{2},(sup,sub)\mapsto Ancestor(sup,sub)$, describes the logic to filter for the Inheritance MS. Given two Type nodes, $sup$ has to be an ancestor of $sub$ in order to fulfill the $Inheritance$ predicate. The result of such an MS detector are sub-graphs from the program’s ASG that are annotated with the MS roles illustrated in Figure 1 for the Inheritance and Aggregation MS. Graphic $\mapsto$ Superclass and Picture $\mapsto$ Subclass for one possible Inheritance MS instance. A complete catalog of Micro-Structures and their detectors is given in our previous work [19], which is a refined and extended catalog based on Maggioni’s work [20]. The catalog is made up of three sub-catalogs: Elemental Design Patterns [21], Design Pattern Clues [22] and Micro-Patterns [23]. Each sub-catalog was defined independently with different motivations and goals, but all of them prove valuable in the process of DPD as Arcelli Fontana et al. [24] concluded in a series of experiments.

III-B Candidate Sampling

Candidate Sampling uses the extracted features (and sometimes the Source Code) to find potential candidates of design pattern instances in a program’s ASG. As discussed, role mappings link concrete classes to specific pattern roles which span a search space of $\binom{n}{k}$ potential mappings where $n$ is the number of available classes and $k$ the number of roles that need to be mapped for a pattern. Hence, finding potential candidates is vital since a full-search is impossible for non-trivial systems. We used Heuristic Search [19] that we proposed in our previous work as candidate sampler. Heuristic Search checks for graph isomorphism [25] between a very general description of the pattern and classes in the program under inspection. It uses primary roles as an entry point to iteratively search for secondary roles that fit the pattern description. Heuristic Search reduces the usual $2^{O(\sqrt{n\log(n)})}$ isomorphism search [26] to a search that is linear with the number of types. For Composite, heuristic search executes the following steps:

1. Collect all types $\bm{sup}$ that are super-types as Component ($\bm{sup}\mapsto$ Component);

2. Collect all types $\bm{sub}$ that are sub-types from $\bm{sup}$ and aggregate $\bm{sub}$ as Composite ($\bm{sub}\mapsto$ Composite);

3. Collect all types $\bm{sib}$ that are sub-types from $\bm{sup}$ and do not aggregate $\bm{sup}$ as Leaf ($\bm{sib}\mapsto$ Leaf).

Each found role mapping is a candidate that is later checked in the design pattern inference stage for their correct consistency with the pattern. A drawback to this heuristic search is that it might miss potential pattern instances. The advantage is that these are simple and efficient algorithms to find potential candidates.

III-C Feature Normalization

Feature-Role Normalization (FRN) is the main contribution of this work. It is used to normalize micro-structures into a fixed sized matrix called Feature Map (FM). FRN allows the usage of a wide variety of existing statistical methods while still retaining the interpretability for manual analysis via software engineers. Figure 2 shows the pipeline in which first the features (micro-structures) are extracted followed by the sampling of candidates from the system. A candidate mapping is a set of classes that map to the design pattern roles which may, may not or may partially map to several micro-structures. Each of the potentially mapped micro-structures may have a different number of roles, resulting in an inhomogeneous feature space. This inhomogeneous feature space is problematic in the inference step as most machine learning methods cannot handle irregularly structured data. Feature-Role Normalization presses this inhomogeneous feature space into a homogeneous two-dimensional feature space that is comprehensible and maintains enough information to be useful for engineers and learning algorithms. More specific, the goal of the normalization, is to take $n=|\bm{f}|$ features each having $m_{fi}=|\bm{r}_{fi}|$ roles and map them to $k=|\bm{r}_{di}|$ design pattern roles such that the result is a fixed-sized matrix. FRN provides one approach to this problem by creating an $n\times k$ matrix where rows represent micro-structures and columns design pattern roles. Values in the feature map, in the context of DPD, are globally unique role identifiers ($id\in\mathbb{N}$), where no two roles of any pattern have the same values associated. That said, roles might be shared across multiple features if they share similar semantic information. Zeros represent missing features meaning that the class mapped to the role does not participate in the micro-structure.

Figure 3 illustrates an example of a feature map describing classes that make up a Composite pattern. Values are role identifiers colored by a simple gradient, and empty gray spots are absent mappings (zeros). Rows depict an illustrative subset of the micro-structures (features) and columns the design pattern roles to which a set of classes are mapped. The class assigned to Component is a Superclass with its children classes assigned to Composite and Leaf. Composite aggregates a Component, delegating and redirecting method calls to it.

This short example mirrors how software engineers would, in a slow fashion, go through the source code of a project to find possible instances of a Composite pattern. Feature maps simplify this process by compacting the relationships between multiple classes into new high-level concepts (e.g., a design pattern) and offer a straightforward way of visualizing them. Naturally, the visualization can be extended by recoloring, grouping or filtering the entries of a feature map, or by visualizing another source code property than identifiers (e.g., number of invocations between roles or complexity measures). This makes FMs a strong visualization technique for software analysis. They provide a compact view of specific aspects of a small set of classes allowing a quick overview of their relationships and class characteristics. Furthermore, the matrix structure has clear advantages regarding automation as many methods, e.g., machine learning algorithms, are designed to work with flat data.

Detection processes that operate on structured data may circumvent the use of FMs for DPD by directly operating on the ASG. Examples are Support Vector Machines [27] (with tree-kernels), Recursive Neural Networks [28], sequence-to-sequence learning models like Long Short-Term Memory Networks [29] or Graph Convolutional Neural Networks [30]. These learning algorithms and their respective models might provide possible performance advantages in the context of DPD by naturally handling the ASG and the MSs. However, some lack computational efficiency while others lack interpretability of the classification decisions.

III-C1 Information Preservation Issues in Feature Maps

FRN normalizes the inhomogeneous feature space caused by micro-structures and their roles into a fixed-sized feature map. The resulting feature map is a sparse representation of attributes and relationships within and between nodes in an ASG. While inspired by adjacency matrices, feature maps are not a proper and full representation of the subgraphs; hence they suffer from two specific issues: noise and collisions.

A feature introduces noise if it has at least two roles and only one of its roles map to a class within the FM. Figure 4 illustrates this problem with the Composite pattern and the features Pure Type, Inheritance and Delegation. On the left side, the class assigned to Leaf is a Superclass. However, neither of the other Composite roles participate in this inheritance hierarchy (both are 0). The Superclass role of the Leaf class exists in the project but does not carry any meaningful information in the context of the current FM mapping. These out of context values inflate the feature map with irrelevant information and introduce noise reducing the usability of FMs. A simple solution to the problem is to force entries to zero in cases where features depict a relationship, but none of the other mapped classes participate in it. Figure 4 depicts this solution on the right side where R3 becomes [math].

A feature suffers from a role collision if it has at least two roles and multiple instances of the feature trace to the same classes. Figure 4 (left) illustrate a collision where two instances of the Delegation MS are present in the classes assigned to Component, Composite, and Leaf. Composite delegates to Component and Leaf delegates to Composite. The multi-assignment for Composite is the result of the class-level abstraction and compactness of FMs. One possible solution is to introduce virtual roles that describe two or more roles like Rx in Figure 4 on the right. Virtual roles incur a certain amount of information loss (direction of relationship) especially if multiple virtual roles have to be used for a feature in a FM. However, virtual roles also allow to express self-loops, for instance if a class aggregates itself (it is Aggregator and Aggregate at the same time). These self-loops are helpful in many cases, for example, $50\%$ of the Singleton pattern instances contain Creation Site / Type To Create indicating that the class is the factory for itself. This information is essential in the context of Singleton detection.

III-D Design Pattern Inference

Design Pattern Inference is the last step of the DPD pipeline (Figure 2) and receives feature maps as input. During training time the models and their parameters are optimized toward a labeled dataset of feature maps. During production time the models only return the probability that an FM belongs to the model’s optimized pattern or not (no labels, no model parameter modifications). The detailed intricacies of the training and production time are subject to the selected model class and may differ quite drastically. For instance, a CNN is optimized via a gradient-based algorithm while decision trees are constructed by splitting the input dimension values via an information theoretical criterion (training time).

IV Design Pattern Instance Detection Study

This study frames the process of DPD (see Figure 1) as a binary classification problem. Each design pattern is detected by a separate model that provides the probability that the input FM belongs to the design pattern. Naturally, multi-label classification, i.e., multiple patterns per observation (FM), is supported by feeding the same FM into different models.

Given this setup, we conducted several experiments with FMs in the context of design pattern instance detection. For better reproducibility and comparability we evaluated only the last two stages of the DPD pipeline highlighted in Figure 2. Experience shows that reimplementing the entire pipeline of a DPD approach including the intermediates (e.g., sampler results) while making the detection results comparable is nearly impossible. That said, the entire pipeline from Figure 2 was implemented, and details about the specific implementation of micro-structures and candidate samplers are given in our previous work [19]. The following evaluation and experimental methodology should be seen as a minimal effort to guarantee comparable research in the future.

IV-A Controlled Variables

Detecting design pattern instances can be a daunting task as there are numerous ways to encode software (e.g., feature maps), select potential candidates (candidate sampling), and build classification models (e.g., RFs or CNNs). In total, the study controls for 7 Experiment Parameters (ExP).

Pattern: Four widely used patterns are considered {Singleton, Template Method, Composite, Decorator}. This selection is based on previous studies [17, 31, 24, 15] but also represents an even selection regarding their categorical distribution (creational, structural and behavioral patterns). 2. 2.

Role Count: A confounding variable is the number of roles of the pattern. The expectation is that patterns with one role are easier to detect than patterns with 2 or 3 roles. However, more roles also imply more conditions through which better precision in the classification can be achieved. The selected patterns have a linear role count distribution with {1, 2, 3, 4} roles for {Singleton, Template Method, Composite, Decorator} excluding tertiary roles. 3. 3.

Classification Model: {Random Forest, CNN} were selected as classification model classes. The rationale for selecting CNNs is their natural ability to handle volumes, i.e., to process FMs while leveraging their structural information. RFs were selected as they are a popular and efficient choice in many ML tasks and are more light-weight regarding hyper-parameters in contrast to CNNs. 4. 4.

Negative-Positive Candidate Ratio: Non-trivial software systems expose a combinatorial number of possible class mappings that can function as pattern candidates. This means a practically infinite number of pattern counterexamples can be found within a system. The candidate samplers (Section III-B) that mitigate this issue favor recall leading to imbalanced distributions of positive and negative examples. Negative-Positive Candidate Ratio (NPCR) captures this skew in the data. The study explores NPCRs of {1, 2, 4, 6, 8, 10}. 5. 5.

Data Augmentation: Shuffling of rows was used as data augmentation method to increase the dataset size and classification difficulty. This inhibits the classifier from learning specific relationships between rows, instead of their content. A permutation count of {0, 1, 5, 10} states how often the dataset was copied, rows augmented and added to the dataset. Permuting rows and columns was not considered as it would result in a nearly random matrix destroying any structural information captured in feature maps. 6. 6.

Optimization Budget: A budget of {200} evaluations was available for each model class (RF or CNN). These evaluations were used to tune their hyper-parameters (such as the number of units within layers, or depth of tree). The optimization selected hyper-parameter configurations that maximized the MCC. 7. 7.

Instance Independence: Describes the confounding variable that instances within a unique mapping correlate thus result in too optimistic classification results if not properly handled. Using standard cross-validation mixes mappings from the same unique mapping into training and test-folds making the folds not independent. This leads to too optimistic evaluation results as the training examples leak information between training and test phase. We used {project-fold cross validation} to avoid information leaks (correlations between samples) during the trials.

IV-B Response Variables

The result of each experiment trial was the generalization performance of the classifier evaluated through cross-validation and measured by {Accuracy, Precision, Recall, F1, Matthews Correlation Coefficient}. MCC was selected as the primary evaluation metric as it provides the most unbiased single number metric for binary classification.

IV-C Data Source

A total of $4$ widely used design patterns were selected for the study, each having peer-reviewed classification examples. These peer-reviewed classifications are given by nine projects that are part of the Pattern-like Micro-Architecture Repository (P-MARt111http://www.ptidej.net/tools/designpatterns/index_html#2 04/10/19) [32]. Applications within the repository range from modeling and drawing tools to static analysis and refactoring frameworks, therefore its diversity is a good representation for the real world. Table IV-C contains the distribution of design pattern instances across projects. Original are the mapping counts in the exploded form provided by the dataset. Revised are the mappings after a manual cleanup of the data. Some design pattern instances trace to classes not contained in the available source code repositories (e.g., java.awt, javax.swing, or third-party libraries) and were removed. Projects that did not include the target pattern were also not considered during the experiments (for the respective pattern) as controlling for imbalance (NPCR) is impossible, e.g., JRefactory for Template Method. At last, the Netbeans project was excluded as we were not able to parse it. Fixing the source code pieces in question was deemed futile as they were too numerous introducing potential bias in the evaluation. Unique are the unique mappings after the revision.