Period-LLM: Extending the Periodic Capability of Multimodal Large Language Model

TL;DR

Period-LLM is a multimodal large language model designed to improve understanding and reasoning of periodic phenomena across various modalities, addressing limitations in temporal modeling and period conflicts.

Contribution

The paper introduces Period-LLM, a novel multimodal model with a new benchmark and training paradigm to enhance periodic task performance in large language models.

Findings

Period-LLM outperforms existing MLLMs on periodic tasks.

The model demonstrates robust generalization from simple to complex tasks.

Proposed strategies effectively maintain periodic reasoning during semantic alignment.

Abstract

Periodic or quasi-periodic phenomena reveal intrinsic characteristics in various natural processes, such as weather patterns, movement behaviors, traffic flows, and biological signals. Given that these phenomena span multiple modalities, the capabilities of Multimodal Large Language Models (MLLMs) offer promising potential to effectively capture and understand their complex nature. However, current MLLMs struggle with periodic tasks due to limitations in: 1) lack of temporal modelling and 2) conflict between short and long periods. This paper introduces Period-LLM, a multimodal large language model designed to enhance the performance of periodic tasks across various modalities, and constructs a benchmark of various difficulty for evaluating the cross-modal periodic capabilities of large models. Specially, We adopt an "Easy to Hard Generalization" paradigm, starting with relatively simple text-based tasks and progressing to more complex visual and multimodal tasks, ensuring that the model gradually builds robust periodic reasoning capabilities. Additionally, we propose a "Resisting Logical Oblivion" optimization strategy to maintain periodic reasoning abilities during semantic alignment. Extensive experiments demonstrate the superiority of the proposed Period-LLM over existing MLLMs in periodic tasks. The code is available at https://github.com/keke-nice/Period-LLM.