SOSControl: Enhancing Human Motion Generation through Saliency-Aware Symbolic Orientation and Timing Control

TL;DR

SOSControl introduces a symbolic framework for precise, interpretable control of human motion generation, enabling specification of body part orientations and timing, and outperforms existing methods in quality and controllability.

Contribution

The paper presents the SOS script and an extraction pipeline, along with a control framework that improves motion generation by incorporating saliency-aware symbolic constraints.

Findings

The SOS extraction pipeline produces interpretable, salient keyframe annotations.

SOSControl outperforms baselines in motion quality and controllability.

The framework generalizes well to various motion timing and orientation constraints.

Abstract

Traditional text-to-motion frameworks often lack precise control, and existing approaches based on joint keyframe locations provide only positional guidance, making it challenging and unintuitive to specify body part orientations and motion timing. To address these limitations, we introduce the Salient Orientation Symbolic (SOS) script, a programmable symbolic framework for specifying body part orientations and motion timing at keyframes. We further propose an automatic SOS extraction pipeline that employs temporally-constrained agglomerative clustering for frame saliency detection and a Saliency-based Masking Scheme (SMS) to generate sparse, interpretable SOS scripts directly from motion data. Moreover, we present the SOSControl framework, which treats the available orientation symbols in the sparse SOS script as salient and prioritizes satisfying these constraints during motion generation. By incorporating SMS-based data augmentation and gradient-based iterative optimization, the framework enhances alignment with user-specified constraints. Additionally, it employs a ControlNet-based ACTOR-PAE Decoder to ensure smooth and natural motion outputs. Extensive experiments demonstrate that the SOS extraction pipeline generates human-interpretable scripts with symbolic annotations at salient keyframes, while the SOSControl framework outperforms existing baselines in motion quality, controllability, and generalizability with respect to motion timing and body part orientation control.