Beyond LoRA: Exploring Efficient Fine-Tuning Techniques for Time Series Foundational Models

TL;DR

This paper investigates parameter-efficient fine-tuning methods for Time Series Foundation Models, demonstrating that certain techniques outperform LoRA and achieve state-of-the-art results in ICU vital sign forecasting with minimal parameter updates.

Contribution

It introduces and evaluates novel PEFT techniques for TSFMs, showing improved domain adaptation and efficiency in healthcare applications, especially ICU vital sign prediction.

Findings

FourierFT surpasses SOTA with only 2,400 tuned parameters.

Some PEFT methods outperform LoRA in parameter efficiency.

Achieved state-of-the-art ICU vital forecasting results.

Abstract

Time Series Foundation Models (TSFMs) have recently garnered attention for their ability to model complex, large-scale time series data across domains such as retail, finance, and transportation. However, their application to sensitive, domain-specific fields like healthcare remains challenging, primarily due to the difficulty of fine-tuning these models for specialized, out-of-domain tasks with scarce publicly available datasets. In this work, we explore the use of Parameter-Efficient Fine-Tuning (PEFT) techniques to address these limitations, focusing on healthcare applications, particularly ICU vitals forecasting for sepsis patients. We introduce and evaluate two selective (BitFit and LayerNorm Tuning) and two additive (VeRA and FourierFT) PEFT techniques on multiple configurations of the Chronos TSFM for forecasting vital signs of sepsis patients. Our comparative analysis demonstrates that some of these PEFT methods outperform LoRA in terms of parameter efficiency and domain adaptation, establishing state-of-the-art (SOTA) results in ICU vital forecasting tasks. Interestingly, FourierFT applied to the Chronos (Tiny) variant surpasses the SOTA model while fine-tuning only 2,400 parameters compared to the 700K parameters of the benchmark.