Situation Graph Prediction: Structured Perspective Inference for User Modeling

TL;DR

This paper introduces Situation Graph Prediction (SGP), a novel task for modeling evolving user perspectives through structured inference from multimodal data, addressing privacy concerns and label scarcity.

Contribution

The paper proposes a structure-first synthetic data generation approach for perspective modeling and demonstrates its feasibility through a diagnostic study with GPT-4o.

Findings

Latent perspective inference is more challenging than surface-level extraction.

Structured synthetic data aligns well with observable traces.

SGP provides a promising direction for privacy-aware user modeling.

Abstract

Perspective-Aware AI requires modeling evolving internal states--goals, emotions, contexts--not merely preferences. Progress is limited by a data bottleneck: digital footprints are privacy-sensitive and perspective states are rarely labeled. We propose Situation Graph Prediction (SGP), a task that frames perspective modeling as an inverse inference problem: reconstructing structured, ontology-aligned representations of perspective from observable multimodal artifacts. To enable grounding without real labels, we use a structure-first synthetic generation strategy that aligns latent labels and observable traces by design. As a pilot, we construct a dataset and run a diagnostic study using retrieval-augmented in-context learning as a proxy for supervision. In our study with GPT-4o, we observe a gap between surface-level extraction and latent perspective inference--indicating latent-state inference is harder than surface extraction under our controlled setting. Results suggest SGP is non-trivial and provide evidence for the structure-first data synthesis strategy.