NIFT: Neural Interaction Field and Template for Object Manipulation
Zeyu Huang, Juzhan Xu, Sisi Dai, Kai Xu, Hao Zhang, Hui Huang, Ruizhen, Hu

TL;DR
NIFT introduces a neural interaction representation that improves imitation learning for object manipulation by matching interaction templates in neural fields, leading to better generalization across object categories.
Contribution
The paper proposes a novel Neural Interaction Field and Template framework that enhances imitation learning by encoding interactions with spherical distance functions and bisector surface sampling.
Findings
Outperforms state-of-the-art imitation methods.
Generalizes well to new object categories.
Provides a robust interaction representation.
Abstract
We introduce NIFT, Neural Interaction Field and Template, a descriptive and robust interaction representation of object manipulations to facilitate imitation learning. Given a few object manipulation demos, NIFT guides the generation of the interaction imitation for a new object instance by matching the Neural Interaction Template (NIT) extracted from the demos in the target Neural Interaction Field (NIF) defined for the new object. Specifically, the NIF is a neural field that encodes the relationship between each spatial point and a given object, where the relative position is defined by a spherical distance function rather than occupancies or signed distances, which are commonly adopted by conventional neural fields but less informative. For a given demo interaction, the corresponding NIT is defined by a set of spatial points sampled in the demo NIF with associated neural features. To…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsRobot Manipulation and Learning · Human Pose and Action Recognition · Hand Gesture Recognition Systems
