PanGEA: The Panoramic Graph Environment Annotation Toolkit

TL;DR

PanGEA is an open-source, web-based toolkit designed for efficient collection of speech and text annotations in photorealistic 3D environments, supporting navigation tasks and adaptable for various grounded language annotations.

Contribution

The paper introduces PanGEA, a lightweight, immersive annotation toolkit with integrated storage and alignment utilities, optimized for large-scale 3D environment annotation tasks.

Findings

Supported 20,000 hours of annotation for Room-Across-Room dataset

Facilitated collection of navigation instructions and following tasks

Demonstrated adaptability for various annotation types

Abstract

PanGEA, the Panoramic Graph Environment Annotation toolkit, is a lightweight toolkit for collecting speech and text annotations in photo-realistic 3D environments. PanGEA immerses annotators in a web-based simulation and allows them to move around easily as they speak and/or listen. It includes database and cloud storage integration, plus utilities for automatically aligning recorded speech with manual transcriptions and the virtual pose of the annotators. Out of the box, PanGEA supports two tasks -- collecting navigation instructions and navigation instruction following -- and it could be easily adapted for annotating walking tours, finding and labeling landmarks or objects, and similar tasks. We share best practices learned from using PanGEA in a 20,000 hour annotation effort to collect the Room-Across-Room dataset. We hope that our open-source annotation toolkit and insights will both expedite future data collection efforts and spur innovation on the kinds of grounded language tasks such environments can support.