Towards Efficient Post-Training via Fourier-Driven Adapter Architectures

TL;DR

This paper introduces Fourier-Activated Adapter (FAA), a novel parameter-efficient fine-tuning method for large language models that leverages Fourier features to modulate semantic information based on frequency components, achieving high performance with low overhead.

Contribution

The paper presents FAA, a new adapter architecture that incorporates Fourier features for frequency-aware modulation, improving post-training efficiency and effectiveness of large language models.

Findings

FAA outperforms existing methods on GLUE, E2E NLG, and instruction-tuning benchmarks.

FAA maintains low computational and memory overhead.

Ablation studies confirm the importance of frequency-aware mechanisms.

Abstract

We propose a novel framework, termed Fourier-Activated Adapter (FAA), for parameter-efficient fine-tuning of large pre-trained language models. By incorporating random Fourier features into lightweight adapter modules, FAA decomposes intermediate representations into complementary low- and high-frequency components, enabling frequency-aware modulation of semantic information. This design allows the model to selectively emphasize informative frequency bands during adaptation while preserving the representational capacity of the frozen backbone. Extensive experiments on GLUE, E2E NLG, and instruction-tuning benchmarks demonstrate that FAA consistently achieves competitive or superior performance compared to existing parameter-efficient fine-tuning methods, while maintaining low computational and memory overhead. Ablation studies further verify the effectiveness of frequency-aware activation and adaptive weighting mechanisms, highlighting FAA as a robust and efficient approach for post-training large language models.