A Pilot Study on Detecting Software Design Patterns with Large Language Models: An Empirical Evaluation

TL;DR

This study evaluates the ability of large language models to detect software design patterns in source code, comparing different models and prompts to assess their effectiveness and potential for aiding software development.

Contribution

It provides an empirical evaluation of four large language models and ensemble methods for detecting five common design patterns in source code.

Findings

NextCoder and Gemma 3 outperform other models in accuracy.

Ensemble approaches improve detection efficiency.

LLMs show promise for automatic design pattern recognition.

Abstract

Design patterns provide reusable solutions to recurring software design problems. Automatically detecting these patterns in source code can help bootstrap new developers' understanding of unfamiliar software system architectures, and can help experienced developers to quickly identify and rectify potential quality issues. While many prior research works have explored graph based and machine-learning based detection techniques, this work evaluates the design pattern recognition capabilities of four Large Language Models and two ensemble approaches consisting three out of the four models. We also compare their performance when prompted with a) Source code, b) PlantUML representation of source code, and c) Text-based descriptions of the source code. We investigate the detection of five design patterns: singleton, adapter, bridge, composite and decorator. Our preliminary results indicate that LLMs show promise for automatically detecting design patterns, with NextCoder and Gemma 3 demonstrating comparatively higher accuracy than other models evaluated, and the ensemble approaches enhancing the overall efficiency of design pattern detection. We identify several directions for future work.