TuckA: Hierarchical Compact Tensor Experts for Efficient Fine-Tuning

TL;DR

TuckA introduces a hierarchical, tensor-based approach for parameter-efficient fine-tuning that captures diverse data features with fewer parameters, improving performance across multiple tasks.

Contribution

The paper proposes Tucker Adaptation (TuckA), a novel hierarchical tensor expert method that enhances PEFT by using Tucker decomposition and efficient routing for diverse task adaptation.

Findings

Outperforms existing PEFT methods on NLP, vision, and reasoning benchmarks.

Reduces parameter count while maintaining or improving accuracy.

Demonstrates effective data-aware initialization and expert routing strategies.

Abstract

Efficiently fine-tuning pre-trained models for downstream tasks is a key challenge in the era of foundation models. Parameter-efficient fine-tuning (PEFT) presents a promising solution, achieving performance comparable to full fine-tuning by updating only a small number of adaptation weights per layer. Traditional PEFT methods typically rely on a single expert, where the adaptation weight is a low-rank matrix. However, for complex tasks, the data's inherent diversity poses a significant challenge for such models, as a single adaptation weight cannot adequately capture the features of all samples. To address this limitation, we explore how to integrate multiple small adaptation experts into a compact structure to defeat a large adapter. Specifically, we propose Tucker Adaptation (TuckA), a method with four key properties: (i) We use Tucker decomposition to create a compact 3D tensor where each slice naturally serves as an expert. The low-rank nature of this decomposition ensures that the number of parameters scales efficiently as more experts are added. (ii) We introduce a hierarchical strategy that organizes these experts into groups at different granularities, allowing the model to capture both local and global data patterns. (iii) We develop an efficient batch-level routing mechanism, which reduces the router's parameter size by a factor of $L$ compared to routing at every adapted layer (where $L$ is the number of adapted layers) (iv) We propose data-aware initialization to achieve loss-free expert load balancing based on theoretical analysis. Extensive experiments on benchmarks in natural language understanding, image classification, and mathematical reasoning speak to the efficacy of TuckA, offering a new and effective solution to the PEFT problem.