Individual Turing Test: A Case Study of LLM-based Simulation Using Longitudinal Personal Data

TL;DR

This study evaluates LLMs' ability to simulate a specific individual using ten years of personal messaging data, revealing current methods' limitations and trade-offs between different approaches.

Contribution

It introduces the 'Individual Turing Test' for assessing personalized LLM simulation and compares various methods using longitudinal personal data.

Findings

Current LLM-based methods do not pass the Individual Turing Test.

Simulation performance is better on strangers than on the actual individual.

Fine-tuning enhances daily chat style simulation; retrieval and memory methods excel in opinions and preferences.

Abstract

Large Language Models (LLMs) have demonstrated remarkable human-like capabilities, yet their ability to replicate a specific individual remains under-explored. This paper presents a case study to investigate LLM-based individual simulation with a volunteer-contributed archive of private messaging history spanning over ten years. Based on the messaging data, we propose the "Individual Turing Test" to evaluate whether acquaintances of the volunteer can correctly identify which response in a multi-candidate pool most plausibly comes from the volunteer. We investigate prevalent LLM-based individual simulation approaches including: fine-tuning, retrieval-augmented generation (RAG), memory-based approach, and hybrid methods that integrate fine-tuning and RAG or memory. Empirical results show that current LLM-based simulation methods do not pass the Individual Turing Test, but they perform substantially better when the same test is conducted on strangers to the target individual. Additionally, while fine-tuning improves the simulation in daily chats representing the language style of the individual, retrieval-augmented and memory-based approaches demonstrate stronger performance on questions involving personal opinions and preferences. These findings reveal a fundamental trade-off between parametric and non-parametric approaches to individual simulation with LLMs when given a longitudinal context.