Structural Priors and Modular Adapters in the Composable Fine-Tuning Algorithm of Large-Scale Models

TL;DR

This paper introduces a novel composable fine-tuning approach for large-scale models that combines graph structural priors with modular adapters, improving multi-task learning efficiency, stability, and accuracy.

Contribution

It presents a new method integrating relation matrices and modular adapters for efficient, stable, and accurate multi-task model fine-tuning under structural constraints.

Findings

Enhanced task prediction accuracy and adapter weight precision.

Improved computational efficiency and training stability.

Effective multi-task adaptation with reduced redundancy.

Abstract

This paper proposes a composable fine-tuning method that integrates graph structural priors with modular adapters to address the high computational cost and structural instability faced by large-scale pre-trained models in multi-task adaptation. The method introduces a relation matrix to model dependencies among tasks, explicitly encoding correlations between nodes and paths into graph structural priors, which provide unified structural constraints for adapter weight allocation and path selection. Modular adapters are embedded into different layers through low-rank mapping and a pluggable mechanism, enabling efficient cross-task composition and reuse under prior guidance. This mechanism not only improves parameter efficiency and training stability but also alleviates path conflicts and redundant computation in multi-task scenarios. Furthermore, experiments on hyperparameter sensitivity, environmental sensitivity, and data sensitivity are conducted to systematically analyze key factors such as routing temperature, gating thresholds, and relation matrix regularization strength, verifying the consistency and superior performance of the method under structural constraints. The results demonstrate that the proposed framework significantly enhances task prediction accuracy, adapter weight allocation precision, and overall computational efficiency while maintaining model lightweight design, highlighting the synergistic advantages of graph priors and modular mechanisms in composable fine-tuning.