A Comparative Study of Demonstration Selection for Practical Large Language Models-based Next POI Prediction

TL;DR

This study compares various demonstration selection strategies for large language models in next POI prediction, finding that simple heuristics often outperform complex methods in accuracy and efficiency.

Contribution

It provides a comprehensive evaluation of demonstration selection methods, highlighting the effectiveness of heuristic approaches over embedding-based strategies in real-world POI prediction.

Findings

Heuristic methods outperform embedding-based methods in prediction accuracy.

Simple heuristics are more computationally efficient than complex embedding-based strategies.

Demonstrations selected by heuristics can sometimes surpass fine-tuned models without additional training.

Abstract

This paper investigates demonstration selection strategies for predicting a user's next point-of-interest (POI) using large language models (LLMs), aiming to accurately forecast a user's subsequent location based on historical check-in data. While in-context learning (ICL) with LLMs has recently gained attention as a promising alternative to traditional supervised approaches, the effectiveness of ICL significantly depends on the selected demonstration. Although previous studies have examined methods such as random selection, embedding-based selection, and task-specific selection, there remains a lack of comprehensive comparative analysis among these strategies. To bridge this gap and clarify the best practices for real-world applications, we comprehensively evaluate existing demonstration selection methods alongside simpler heuristic approaches such as geographical proximity, temporal ordering, and sequential patterns. Extensive experiments conducted on three real-world datasets indicate that these heuristic methods consistently outperform more complex and computationally demanding embedding-based methods, both in terms of computational cost and prediction accuracy. Notably, in certain scenarios, LLMs using demonstrations selected by these simpler heuristic methods even outperform existing fine-tuned models, without requiring further training. Our source code is available at: https://github.com/ryonsd/DS-LLM4POI.