From Articulated Kinematics to Routed Visual Control for Action-Conditioned Surgical Video Generation

TL;DR

This paper introduces a hierarchical routing framework for action-conditioned surgical video generation, converting articulated kinematics into control modalities and improving efficiency, stability, and realism.

Contribution

It proposes a novel kinematic-to-visual lifting paradigm and a hierarchically routed control framework with routing loss functions and a budgeted scheme, advancing surgical video synthesis.

Findings

Improves action faithfulness and visual fidelity in generated videos.

Achieves significant latency reduction with maintained control accuracy.

Provides a new benchmark with articulated annotations for surgical videos.

Abstract

Action-conditioned surgical video generation is a critical yet highly challenging problem for robotic surgery. The core difficulty is that low-dimensional control vectors must precisely govern complex image-space evolution. In this work, we propose a kinematic-to-visual lifting paradigm that converts articulated kinematics into a unified set of five image-aligned control modalities. Building on this representation, we introduce a hierarchically routed visual control framework that selectively activates the most relevant control modalities and motion scales. Instead of uniformly applying all control signals, our model performs hierarchical routing to dynamically allocate conditioning capacity. We further design kinematic-prior-guided routing loss functions to ensure physically meaningful, temporally stable, and efficient expert utilization. To improve efficiency, we propose a budgeted training and inference scheme that leverages routing-induced sparsity. By selectively discarding low-significance control pathways during training and execution, our approach enables adaptive computation that is complementary to standard distillation. We additionally construct a new benchmark with curated articulated annotations, obtained through human-in-the-loop semantic labeling and differentiable pose tracking, providing realistic supervision for action-conditioned surgical video generation. Extensive experiments demonstrate that our method consistently improves action faithfulness, visual fidelity, and cross-domain generalization over diverse baselines. Moreover, our efficient variant achieves substantial reductions in latency while maintaining strong control accuracy.