Building a Foundation Model for Trajectory from Scratch

TL;DR

This paper provides a step-by-step tutorial on building a trajectory foundation model from scratch using GPT-2, including code, comparisons, and techniques to aid researchers in mobility AI.

Contribution

It offers the first detailed, implementation-level tutorial for creating trajectory foundation models from GPT-2, filling a gap in mobility AI research documentation.

Findings

Demonstrated adaptation of GPT-2 for spatiotemporal trajectory data

Compared architectures of TrajFM, TrajGPT, and TimesFM

Provided educational resources for building mobility foundation models

Abstract

Foundation models are transformative in artificial intelligence, but building them from scratch, especially for mobility trajectories, is not yet clear or documented. This tutorial bridges this gap by demonstrating the steps and code of a minimal implementation of a trajectory-focused foundation model starting from GPT-2. Through a concise, step-by-step, code-driven process, we demonstrate adapting GPT-2 for spatiotemporal data. We then review and compare representative trajectory foundation models, such as TrajFM and TrajGPT, highlighting their architectural innovations and differences. Additionally, we introduce complementary techniques from related domains, like TimesFM's patching approach. Targeted at researchers and practitioners, this tutorial aims to explain the concepts and terminology of foundation models, at the implementation level. We find it timely and indispensable to create this educational material in order to support the SIGSPATIAL community in building and evaluating mobility foundation models, enhancing both research clarity and peer-review effectiveness in mobility AI.