LiFT: Does Instruction Fine-Tuning Improve In-Context Learning for Longitudinal Modelling by Large Language Models?

TL;DR

LiFT is a novel instruction fine-tuning framework that enhances large language models' ability to perform longitudinal reasoning over temporally ordered text, improving in-context learning on temporal tasks.

Contribution

The paper introduces LiFT, a curriculum-based instruction fine-tuning method that unifies diverse longitudinal NLP tasks and improves model generalization and handling of rare change events.

Findings

LiFT outperforms base models on in-distribution and out-of-distribution datasets.

Models trained with LiFT show strong gains on minority change events.

LiFT improves temporal reasoning across models of different sizes.

Abstract

Longitudinal NLP tasks require reasoning over temporally ordered text to detect persistence and change in human behavior and opinions. However, in-context learning with large language models struggles on tasks where models must integrate historical context, track evolving interactions, and handle rare change events. We introduce LiFT, a longitudinal instruction fine-tuning framework that unifies diverse longitudinal modeling tasks under a shared instruction schema. LiFT uses a curriculum that progressively increases temporal difficulty while incorporating few-shot structure and temporal conditioning to encourage effective use of past context. We evaluate LiFT across five datasets. Models trained on longitudinal tasks with different levels of temporal granularity are tested for generalisability on two separate datasets. Across models with different parameter sizes (OLMo (1B/7B), LLaMA-8B, and Qwen-14B), LiFT consistently outperforms base-model ICL, with strong gains on out-of-distribution data and minority change events.