Mixture of Low Rank Adaptation with Partial Parameter Sharing for Time Series Forecasting

TL;DR

This paper introduces a novel two-stage framework with Mixture-of-LoRA modules for time series forecasting, overcoming expressiveness limitations of multi-task models and improving accuracy and efficiency.

Contribution

It proposes a new framework combining foundation models with step-specific LoRA modules and a Mixture-of-LoRA model for adaptive parameter sharing in TSF.

Findings

MoLA outperforms state-of-the-art methods

Enhanced model expressiveness and forecasting accuracy

Efficient parameter sharing across forecast steps

Abstract

Multi-task forecasting has become the standard approach for time-series forecasting (TSF). However, we show that it suffers from an Expressiveness Bottleneck, where predictions at different time steps share the same representation, leading to unavoidable errors even with optimal representations. To address this issue, we propose a two-stage framework: first, pre-train a foundation model for one-step-ahead prediction; then, adapt it using step-specific LoRA modules.This design enables the foundation model to handle any number of forecast steps while avoiding the expressiveness bottleneck. We further introduce the Mixture-of-LoRA (MoLA) model, which employs adaptively weighted LoRA experts to achieve partial parameter sharing across steps. This approach enhances both efficiency and forecasting performance by exploiting interdependencies between forecast steps. Experiments show that MoLA significantly improves model expressiveness and outperforms state-of-the-art time-series forecasting methods. Code is available at https://anonymous.4open.science/r/MoLA-BC92.