Hybrid-LoRA: Bridging Full Fine-Tuning and Low-Rank Adaptation for Post-Training

TL;DR

Hybrid-LoRA is a novel post-training framework that combines full fine-tuning and Low-Rank Adaptation to efficiently adapt large language models for complex reasoning tasks, achieving near full fine-tuning performance with reduced costs.

Contribution

It introduces a Hybrid-LoRA Score to selectively apply full fine-tuning and LoRA, significantly improving performance over existing PEFT methods in post-training for reasoning tasks.

Findings

Hybrid-LoRA matches full fine-tuning performance with only 10% of modules fully fine-tuned.

It outperforms four state-of-the-art PEFT post-training baselines by up to 5.65%.

Hybrid-LoRA achieves an average improvement of 4.36% over the best baseline.

Abstract

Post-training has become essential for adapting large language models (LLMs) to complex downstream behaviors, including instruction following, preference alignment, and multi-step reasoning. Reinforcement learning with verifiable rewards (RLVR) has recently emerged as a particularly effective post-training paradigm for improving reasoning capabilities, with critic-free algorithms such as GRPO and GSPO enabling scalable optimization. However, RLVR post-training with full fine-tuning (FFT) requires substantial GPU memory and incurs high training costs. Although parameter-efficient fine-tuning (PEFT) methods, such as Low-Rank Adaptation (LoRA), effectively reduce computational costs, they often suffer from a noticeable performance gap compared to full fine-tuning in post-training for complex reasoning tasks. In this paper, we propose Hybrid-LoRA, an efficient hybrid post-training framework that selectively applies full fine-tuning to a small subset of modules less suited to low-rank adaptation, while adapting the remaining components with LoRA. We introduce a novel Hybrid-LoRA Score to rank candidate modules according to their sensitivity to low-rank adaptation under a fixed parameter budget. Experiments show that Hybrid-LoRA closely matches full fine-tuning performance under a 10% full fine-tuning module budget, with the remaining candidate modules adapted by LoRA, consistently outperforming four state-of-the-art PEFT post-training baselines, achieving improvements of up to 5.65% and on average 4.36% over the best baseline.