SIMS: Simulating Stylized Human-Scene Interactions with Retrieval-Augmented Script Generation

TL;DR

This paper presents SIMS, a hierarchical framework that combines retrieval-augmented script generation with a physics-based control policy to simulate diverse, stylized human-scene interactions with high physical plausibility.

Contribution

The paper introduces a novel hierarchical approach that integrates large language model-based script generation with a multi-condition control policy for realistic and stylized human-scene interaction simulation.

Findings

Outperforms previous methods in task execution and generalization

Generates diverse and stylized human motions

Provides comprehensive datasets for HSI simulation

Abstract

Simulating stylized human-scene interactions (HSI) in physical environments is a challenging yet fascinating task. Prior works emphasize long-term execution but fall short in achieving both diverse style and physical plausibility. To tackle this challenge, we introduce a novel hierarchical framework named SIMS that seamlessly bridges highlevel script-driven intent with a low-level control policy, enabling more expressive and diverse human-scene interactions. Specifically, we employ Large Language Models with Retrieval-Augmented Generation (RAG) to generate coherent and diverse long-form scripts, providing a rich foundation for motion planning. A versatile multicondition physics-based control policy is also developed, which leverages text embeddings from the generated scripts to encode stylistic cues, simultaneously perceiving environmental geometries and accomplishing task goals. By integrating the retrieval-augmented script generation with the multi-condition controller, our approach provides a unified solution for generating stylized HSI motions. We further introduce a comprehensive planning dataset produced by RAG and a stylized motion dataset featuring diverse locomotions and interactions. Extensive experiments demonstrate SIMS's effectiveness in executing various tasks and generalizing across different scenarios, significantly outperforming previous methods.