LLM attribution analysis across different fine-tuning strategies and model scales for automated code compliance

TL;DR

This study investigates how different fine-tuning strategies and model sizes influence the interpretive behavior of large language models in automated code compliance tasks, revealing key differences and interpretive strategies.

Contribution

It introduces a perturbation-based attribution analysis to compare interpretive behaviors across fine-tuning methods and model scales, providing insights into model transparency.

Findings

FFT yields more focused attribution patterns than parameter-efficient fine-tuning.

Larger models develop specific interpretive strategies such as prioritizing numerical constraints.

Performance gains plateau for models larger than 7B in semantic similarity.

Abstract

Existing research on large language models (LLMs) for automated code compliance has primarily focused on performance, treating the models as black boxes and overlooking how training decisions affect their interpretive behavior. This paper addresses this gap by employing a perturbation-based attribution analysis to compare the interpretive behaviors of LLMs across different fine-tuning strategies such as full fine-tuning (FFT), low-rank adaptation (LoRA) and quantized LoRA fine-tuning, as well as the impact of model scales which include varying LLM parameter sizes. Our results show that FFT produces attribution patterns that are statistically different and more focused than those from parameter-efficient fine-tuning methods. Furthermore, we found that as model scale increases, LLMs develop specific interpretive strategies such as prioritizing numerical constraints and rule identifiers in the building text, albeit with performance gains in semantic similarity of the generated and reference computer-processable rules plateauing for models larger than 7B. This paper provides crucial insights into the explainability of these models, taking a step toward building more transparent LLMs for critical, regulation-based tasks in the Architecture, Engineering, and Construction industry.