SituationalLLM: Proactive language models with scene awareness for dynamic, contextual task guidance

TL;DR

SituationalLLM enhances large language models with scene awareness by integrating structured environment data, enabling proactive, context-aware guidance in real-world physical settings, and outperforming baseline models in task-specificity and reliability.

Contribution

The paper introduces SituationalLLM, a novel framework that incorporates scene graphs into LLMs for proactive, environment-aware assistance, trained on a new dataset for improved real-world task guidance.

Findings

Outperforms baseline LLMs in task specificity and reliability.

Successfully identifies environmental gaps and seeks clarifications.

Demonstrates robustness in real-world scenarios.

Abstract

Large language models (LLMs) have achieved remarkable success in text-based tasks but often struggle to provide actionable guidance in real-world physical environments. This is because of their inability to recognize their limited understanding of the user's physical context. We present SituationalLLM, a novel approach that integrates structured scene information into an LLM to deliver proactive, context-aware assistance. By encoding objects, attributes, and relationships in a custom Scene Graph Language, SituationalLLM actively identifies gaps in environmental context and seeks clarifications during user interactions. This behavior emerges from training on the Situational Awareness Database for Instruct-Tuning (SAD-Instruct), which combines diverse, scenario-specific scene graphs with iterative, dialogue-based refinements. Experimental results indicate that SituationalLLM outperforms generic LLM baselines in task specificity, reliability, and adaptability, paving the way for environment-aware AI assistants capable of delivering robust, user-centric guidance under real-world constraints.