Investigating the Impact of Model Complexity in Large Language Models

TL;DR

This paper explores how model complexity affects the performance of large language models, revealing a double descent phenomenon and identifying a sweet spot at minimal complexity through theoretical analysis and HMM-based experiments.

Contribution

It introduces a novel HMM-based modeling approach for autoregressive LLMs and provides a theoretical analysis of the relationship between model complexity and generalization, highlighting a double descent pattern.

Findings

Risk initially increases then decreases with model complexity.

The optimal model complexity occurs at the lowest possible size.

Experimental results align with theoretical predictions.

Abstract

Large Language Models (LLMs) based on the pre-trained fine-tuning paradigm have become pivotal in solving natural language processing tasks, consistently achieving state-of-the-art performance. Nevertheless, the theoretical understanding of how model complexity influences fine-tuning performance remains challenging and has not been well explored yet. In this paper, we focus on autoregressive LLMs and propose to employ Hidden Markov Models (HMMs) to model them. Based on the HMM modeling, we investigate the relationship between model complexity and the generalization capability in downstream tasks. Specifically, we consider a popular tuning paradigm for downstream tasks, head tuning, where all pre-trained parameters are frozen and only individual heads are trained atop pre-trained LLMs. Our theoretical analysis reveals that the risk initially increases and then decreases with rising model complexity, showcasing a "double descent" phenomenon. In this case, the initial "descent" is degenerate, signifying that the "sweet spot" where bias and variance are balanced occurs when the model size is zero. Obtaining the presented in this study conclusion confronts several challenges, primarily revolving around effectively modeling autoregressive LLMs and downstream tasks, as well as conducting a comprehensive risk analysis for multivariate regression. Our research is substantiated by experiments conducted on data generated from HMMs, which provided empirical support and alignment with our theoretical insights.