Prevalence of Code Smells in Reinforcement Learning Projects

TL;DR

This study investigates the prevalence of code smells in reinforcement learning projects, revealing significant maintainability issues and common design problems in widely used RL Python codebases.

Contribution

It provides the first empirical analysis of code quality in RL projects, highlighting prevalent code smells and their impact on software maintainability.

Findings

Average of 3.95% code smells in RL projects

Long methods and chains are the most common smells

Code smells indicate responsibility and abstraction issues

Abstract

Reinforcement Learning (RL) is being increasingly used to learn and adapt application behavior in many domains, including large-scale and safety critical systems, as for example, autonomous driving. With the advent of plug-n-play RL libraries, its applicability has further increased, enabling integration of RL algorithms by users. We note, however, that the majority of such code is not developed by RL engineers, which as a consequence, may lead to poor program quality yielding bugs, suboptimal performance, maintainability, and evolution problems for RL-based projects. In this paper we begin the exploration of this hypothesis, specific to code utilizing RL, analyzing different projects found in the wild, to assess their quality from a software engineering perspective. Our study includes 24 popular RL-based Python projects, analyzed with standard software engineering metrics. Our results, aligned with similar analyses for ML code in general, show that popular and widely reused RL repositories contain many code smells (3.95% of the code base on average), significantly affecting the projects' maintainability. The most common code smells detected are long method and long method chain, highlighting problems in the definition and interaction of agents. Detected code smells suggest problems in responsibility separation, and the appropriateness of current abstractions for the definition of RL algorithms.