Tuning Language Models for Robust Prediction of Diverse User Behaviors

TL;DR

This paper introduces BehaviorLM, a two-stage fine-tuning method for large language models that improves the prediction of both common and rare user behaviors in intelligent systems.

Contribution

BehaviorLM's progressive fine-tuning approach enhances tail behavior prediction while maintaining performance on frequent behaviors, leveraging LLMs' behavioral knowledge.

Findings

BehaviorLM outperforms existing methods on real-world datasets.

It effectively predicts rare tail behaviors with few-shot learning.

The approach preserves general behavioral knowledge during fine-tuning.

Abstract

Predicting user behavior is essential for intelligent assistant services, yet deep learning models often struggle to capture long-tailed behaviors. Large language models (LLMs), with their pretraining on vast corpora containing rich behavioral knowledge, offer promise. However, existing fine-tuning approaches tend to overfit to frequent ``anchor'' behaviors, reducing their ability to predict less common ``tail'' behaviors. In this paper, we introduce BehaviorLM, a progressive fine-tuning approach that addresses this issue. In the first stage, LLMs are fine-tuned on anchor behaviors while preserving general behavioral knowledge. In the second stage, fine-tuning uses a balanced subset of all behaviors based on sample difficulty to improve tail behavior predictions without sacrificing anchor performance. Experimental results on two real-world datasets demonstrate that BehaviorLM robustly predicts both anchor and tail behaviors and effectively leverages LLM behavioral knowledge to master tail behavior prediction with few-shot examples.