Semantic-Contact Fields for Category-Level Generalizable Tactile Tool Manipulation

TL;DR

This paper introduces Semantic-Contact Fields (SCFields), a novel 3D representation combining visual semantics with tactile contact data, enabling category-level generalization in soft tactile tool manipulation through a two-stage sim-to-real learning pipeline.

Contribution

The work presents SCFields, a unified tactile-visual representation learned via simulation and minimal real data, improving generalization across diverse tools in manipulation tasks.

Findings

SCFields enables robust category-level generalization in tactile manipulation.

The two-stage sim-to-real pipeline effectively transfers tactile representations from simulation to real-world.

Experiments show significant performance improvements over vision-only and raw tactile baselines.

Abstract

Generalizing tool manipulation requires both semantic planning and precise physical control. Modern generalist robot policies, such as Vision-Language-Action (VLA) models, often lack the physical grounding required for contact-rich tool manipulation. Conversely, existing contact-aware policies that leverage tactile or haptic sensing are typically instance-specific and fail to generalize across diverse tool geometries. Bridging this gap requires learning representations that are both semantically transferable and physically grounded, yet a fundamental barrier remains: diverse real-world tactile data are prohibitive to collect at scale, while direct zero-shot sim-to-real transfer is challenging due to the complex nonlinear deformation of soft tactile sensors. To address this, we propose Semantic-Contact Fields (SCFields), a unified 3D representation that fuses visual semantics with dense extrinsic contact estimates, including contact probability and force. SCFields is learned through a two-stage Sim-to-Real Contact Learning Pipeline: we first pre-train on large-scale simulation to learn geometry-aware contact priors, then fine-tune on a small set of real data pseudo-labeled via geometric heuristics and force optimization to align real tactile signals. The resulting force-aware representation serves as the dense observation input to a diffusion policy, enabling physical generalization to unseen tool instances. Experiments on scraping, crayon drawing, and peeling demonstrate robust category-level generalization, significantly outperforming vision-only and raw-tactile baselines. Project page: https://kevinskwk.github.io/SCFields/.