VRKitchen: an Interactive 3D Virtual Environment for Task-oriented Learning
Xiaofeng Gao, Ran Gong, Tianmin Shu, Xu Xie, Shu Wang, Song-Chun Zhu

TL;DR
VRKitchen is a comprehensive virtual reality environment designed for training and testing AI agents in complex, realistic tasks involving detailed object manipulations, with support for human demonstrations and standardized evaluation tools.
Contribution
The paper introduces VRKitchen, a novel VR system enabling complex task learning for AI agents and human demonstrations in a realistic, interactive environment with standardized benchmarks.
Findings
Supports complex object manipulations in a realistic VR setting
Allows human demonstrations for training AI agents
Provides standardized evaluation benchmarks
Abstract
One of the main challenges of advancing task-oriented learning such as visual task planning and reinforcement learning is the lack of realistic and standardized environments for training and testing AI agents. Previously, researchers often relied on ad-hoc lab environments. There have been recent advances in virtual systems built with 3D physics engines and photo-realistic rendering for indoor and outdoor environments, but the embodied agents in those systems can only conduct simple interactions with the world (e.g., walking around, moving objects, etc.). Most of the existing systems also do not allow human participation in their simulated environments. In this work, we design and implement a virtual reality (VR) system, VRKitchen, with integrated functions which i) enable embodied agents powered by modern AI methods (e.g., planning, reinforcement learning, etc.) to perform complex…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsReinforcement Learning in Robotics · Robot Manipulation and Learning · Human Pose and Action Recognition
