LongLaMP: A Benchmark for Personalized Long-form Text Generation

TL;DR

This paper introduces LongLaMP, a comprehensive benchmark for evaluating personalized long-form text generation, addressing the gap in existing work that mainly focuses on short text personalization.

Contribution

We develop the LongLaMP benchmark to evaluate personalized long-text generation and demonstrate its effectiveness through extensive experiments.

Findings

Personalization significantly improves long-text generation quality.

LongLaMP effectively evaluates diverse long-text generation tasks.

The benchmark supports zero-shot and fine-tuning approaches.

Abstract

Long-text generation is seemingly ubiquitous in real-world applications of large language models such as generating an email or writing a review. Despite the fundamental importance and prevalence of long-text generation in many practical applications, existing work on personalized generation has focused on the generation of very short text. To overcome these limitations, we study the problem of personalized long-text generation, that is, generating long-text that is personalized for a specific user while being practically useful for the vast majority of real-world applications that naturally require the generation of longer text. In this work, we demonstrate the importance of user-specific personalization for long-text generation tasks and develop the Long-text Language Model Personalization (LongLaMP) Benchmark. LongLaMP provides a comprehensive and diverse evaluation framework for personalized long-text generation. Extensive experiments on LongLaMP for zero-shot and fine-tuned language tasks demonstrate the effectiveness of the proposed benchmark and its utility for developing and evaluating techniques for personalized long-text generation across a wide variety of long-text generation tasks. The results highlight the importance of personalization across a wide variety of long-text generation tasks. Finally, we release the benchmark for others to use for this important problem.