HyperTASR: Hypernetwork-Driven Task-Aware Scene Representations for Robust Manipulation

TL;DR

HyperTASR introduces a hypernetwork-driven framework that dynamically modulates scene representations based on task and phase, improving robotic manipulation by emulating human perceptual adaptation.

Contribution

It presents a novel hypernetwork-based approach that adaptively transforms scene representations conditioned on task objectives and execution phase, enhancing policy learning.

Findings

Significant performance improvements in simulation and real-world tasks.

Enhanced focus on task-relevant scene features through ablation studies.

Better alignment with human-like perceptual adaptation during manipulation.

Abstract

Effective policy learning for robotic manipulation requires scene representations that selectively capture task-relevant environmental features. Current approaches typically employ task-agnostic representation extraction, failing to emulate the dynamic perceptual adaptation observed in human cognition. We present HyperTASR, a hypernetwork-driven framework that modulates scene representations based on both task objectives and the execution phase. Our architecture dynamically generates representation transformation parameters conditioned on task specifications and progression state, enabling representations to evolve contextually throughout task execution. This approach maintains architectural compatibility with existing policy learning frameworks while fundamentally reconfiguring how visual features are processed. Unlike methods that simply concatenate or fuse task embeddings with task-agnostic representations, HyperTASR establishes computational separation between task-contextual and state-dependent processing paths, enhancing learning efficiency and representational quality. Comprehensive evaluations in both simulation and real-world environments demonstrate substantial performance improvements across different representation paradigms. Through ablation studies and attention visualization, we confirm that our approach selectively prioritizes task-relevant scene information, closely mirroring human adaptive perception during manipulation tasks. The project website is at https://lisunphil.github.io/HyperTASR_projectpage/.