Do Code LLMs Understand Design Patterns?

TL;DR

This paper investigates whether Code Large Language Models understand software design patterns by empirically evaluating their recognition, comprehension, and generation capabilities, revealing significant biases that impact their reliability.

Contribution

It provides the first comprehensive empirical analysis of Code LLMs' understanding of design patterns, highlighting biases affecting downstream software development tasks.

Findings

Code LLMs often fail to recognize design patterns accurately.

Biases in models influence code generation quality.

Understanding of design patterns by LLMs is limited.

Abstract

Code Large Language Models (LLMs) demonstrate great versatility in adapting to various downstream tasks, including code generation and completion, as well as bug detection and fixing. However, Code LLMs often fail to capture existing coding standards, leading to the generation of code that conflicts with the required design patterns for a given project. As a result, developers must post-process to adapt the generated code to the project's design norms. In this work, we empirically investigate the biases of Code LLMs in software development. Through carefully designed experiments, we assess the models' understanding of design patterns across recognition, comprehension, and generation. Our findings reveal that biases in Code LLMs significantly affect the reliability of downstream tasks.