Measuring design compliance using neural language models -- an automotive case study

TL;DR

This paper presents a neural language model-based system to automatically assess design pattern compliance in automotive control software, significantly aiding architects in ensuring software quality and reducing manual review efforts.

Contribution

It introduces a novel method using pre-trained source code models to automate compliance assessment of automotive design patterns, specifically Controller-Handler.

Findings

Achieves 92% precision in compliance detection

Effectively identifies violations and guides refactoring

Demonstrates practical utility in automotive software design

Abstract

As the modern vehicle becomes more software-defined, it is beginning to take significant effort to avoid serious regression in software design. This is because automotive software architects rely largely upon manual review of code to spot deviations from specified design principles. Such an approach is both inefficient and prone to error. In recent days, neural language models pre-trained on source code are beginning to be used for automating a variety of programming tasks. In this work, we extend the application of such a Programming Language Model (PLM) to automate the assessment of design compliance. Using a PLM, we construct a system that assesses whether a set of query programs comply with Controller-Handler, a design pattern specified to ensure hardware abstraction in automotive control software. The assessment is based upon measuring whether the geometrical arrangement of query program embeddings, extracted from the PLM, aligns with that of a set of known implementations of the pattern. The level of alignment is then transformed into an interpretable measure of compliance. Using a controlled experiment, we demonstrate that our technique determines compliance with a precision of 92%. Also, using expert review to calibrate the automated assessment, we introduce a protocol to determine the nature of the violation, helping eventual refactoring. Results from this work indicate that neural language models can provide valuable assistance to human architects in assessing and fixing violations in automotive software design.