Towards Understanding Fine-Tuning Mechanisms of LLMs via Circuit Analysis

TL;DR

This paper uses circuit analysis to interpret the fine-tuning process of LLMs, revealing circuit dynamics and developing a circuit-aware LoRA method that improves performance and offers insights into task composition.

Contribution

It introduces a circuit analysis approach to understand fine-tuning mechanisms and proposes a circuit-aware LoRA method that outperforms standard LoRA.

Findings

Circuits maintain high node similarity before and after fine-tuning

Edges in circuits undergo significant changes during fine-tuning

Circuit-based LoRA improves performance by 2.46% over standard LoRA

Abstract

Fine-tuning significantly improves the performance of Large Language Models (LLMs), yet its underlying mechanisms remain poorly understood. This paper aims to provide an in-depth interpretation of the fine-tuning process through circuit analysis, a popular tool in Mechanistic Interpretability (MI). Unlike previous studies (Prakash et al. 2024; Chhabra et al. 2024) that focus on tasks where pre-trained models already perform well, we develop a set of mathematical tasks where fine-tuning yields substantial performance gains, which are closer to the practical setting. In our experiments, we identify circuits at various checkpoints during fine-tuning and examine the interplay between circuit analysis, fine-tuning methods, and task complexities. First, we find that while circuits maintain high node similarity before and after fine-tuning, their edges undergo significant changes, in contrast to prior work that shows circuits only add some additional components after fine-tuning. Based on these observations, we develop a circuit-aware Low-Rank Adaptation (LoRA) method, which assigns ranks to layers based on edge changes in the circuits. Experimental results demonstrate that our circuit-based LoRA algorithm achieves an average performance improvement of 2.46% over standard LoRA with similar parameter sizes. Furthermore, we explore how combining circuits from subtasks can enhance fine-tuning in compositional tasks, providing new insights into the design of such tasks and deepening the understanding of circuit dynamics and fine-tuning mechanisms.