Pre-trained Transformer Uncovers Meaningful Patterns in Human Mobility Data

TL;DR

This paper shows that a transformer model pre-trained on large-scale human mobility data can learn meaningful geographic and mobility concepts, significantly improving performance in related tasks after fine-tuning.

Contribution

It introduces a pre-training framework for transformers on human mobility data that captures diverse geographic and mobility concepts, enhancing downstream task performance.

Findings

Pre-trained embeddings improve task accuracy up to 38%.

The model captures geographic, administrative, and land cover concepts.

Pre-training uncovers meaningful patterns in raw mobility data.

Abstract

We empirically demonstrate that a transformer pre-trained on country-scale unlabeled human mobility data learns embeddings capable, through fine-tuning, of developing a deep understanding of the target geography and its corresponding mobility patterns. Utilizing an adaptation framework, we evaluate the performance of our pre-trained embeddings in encapsulating a broad spectrum of concepts directly and indirectly related to human mobility. This includes basic notions, such as geographic location and distance, and extends to more complex constructs, such as administrative divisions and land cover. Our extensive empirical analysis reveals a substantial performance boost gained from pre-training, reaching up to 38% in tasks such as tree-cover regression. We attribute this result to the ability of the pre-training to uncover meaningful patterns hidden in the raw data, beneficial for modeling relevant high-level concepts. The pre-trained embeddings emerge as robust representations of regions and trajectories, potentially valuable for a wide range of downstream applications.