Are we ready for beyond-application high-volume data? The Reeds robot perception benchmark dataset

TL;DR

The Reeds dataset offers a comprehensive, high-resolution benchmark for robot perception algorithms using a dynamic boat platform, enabling scalable, fair, and reproducible evaluation of diverse perception methods.

Contribution

This paper introduces the Reeds dataset, a high-volume, multi-sensor robot perception benchmark with a scalable evaluation framework for fair comparison.

Findings

Naive sequential evaluation is impractical for large datasets.

Parallel decoding and evaluation improve scalability.

The dataset provides diverse resolution presets for comprehensive testing.

Abstract

This paper presents a dataset, called Reeds, for research on robot perception algorithms. The dataset aims to provide demanding benchmark opportunities for algorithms, rather than providing an environment for testing application-specific solutions. A boat was selected as a logging platform in order to provide highly dynamic kinematics. The sensor package includes six high-performance vision sensors, two long-range lidars, radar, as well as GNSS and an IMU. The spatiotemporal resolution of sensors were maximized in order to provide large variations and flexibility in the data, offering evaluation at a large number of different resolution presets based on the resolution found in other datasets. Reeds also provides means of a fair and reproducible comparison of algorithms, by running all evaluations on a common server backend. As the dataset contains massive-scale data, the evaluation principle also serves as a way to avoid moving data unnecessarily. It was also found that naive evaluation of algorithms, where each evaluation is computed sequentially, was not practical as the fetch and decode task of each frame would not scale well. Instead, each frame is only decoded once and then fed to all algorithms in parallel, including for GPU-based algorithms.