Adapting Large Language Models for Time Series Modeling via a Novel Parameter-efficient Adaptation Method

TL;DR

This paper introduces Time-LlaMA, a novel framework that adapts large language models for time series analysis using a tokenization approach and a dynamic low-rank adaptation technique, achieving state-of-the-art results.

Contribution

The paper presents a new method combining tokenization and D-LoRA to efficiently adapt LLMs for time series modeling, balancing accuracy and inference efficiency.

Findings

Achieves state-of-the-art performance on real-world time series tasks.

Demonstrates effective alignment of time series and language modalities.

Enhances predictive capabilities with dynamic module selection.

Abstract

Time series modeling holds significant importance in many real-world applications and has been extensively studied. While pre-trained foundation models have made impressive strides in the fields of natural language processing (NLP) and computer vision (CV), their development in time series domains has been constrained by data sparsity. A series of recent studies have demonstrated that large language models (LLMs) possess robust pattern recognition and reasoning abilities over complex sequences of tokens. However, the current literature have yet striked a high-quality balance between (a) effectively aligning the time series and natural language modalities, and (b) keeping the inference efficiency. To address the above issues, we now propose the Time-LlaMA framework. Time-LlaMA first converts the time series input into token embeddings through a linear tokenization mechanism. Second, the time series token embeddings are aligned with the text prompts. Third, to further adapt the LLM backbone for time series modeling, we have developed a dynamic low-rank adaptation technique (D-LoRA). D-LoRA dynamically chooses the most suitable LoRA modules at each layer of the Transformer backbone for each time series input, enhancing the model's predictive capabilities. Our experimental results on an extensive collection of challenging real-world time series tasks confirm that our proposed method achieves the state-of-the-art (SOTA) performance.