FISHING Net: Future Inference of Semantic Heatmaps In Grids

TL;DR

This paper introduces FISHING Net, an end-to-end neural network ensemble that fuses multi-modal sensor data into a unified semantic grid for improved perception and short-term prediction in autonomous navigation.

Contribution

It presents a novel framework that transforms diverse sensor inputs into a common top-down semantic grid, simplifying perception and enabling easy sensor data fusion.

Findings

Unified semantic grid improves sensor data integration

Framework effectively combines multiple sensor modalities

Potential extension to various perception tasks

Abstract

For autonomous robots to navigate a complex environment, it is crucial to understand the surrounding scene both geometrically and semantically. Modern autonomous robots employ multiple sets of sensors, including lidars, radars, and cameras. Managing the different reference frames and characteristics of the sensors, and merging their observations into a single representation complicates perception. Choosing a single unified representation for all sensors simplifies the task of perception and fusion. In this work, we present an end-to-end pipeline that performs semantic segmentation and short term prediction using a top-down representation. Our approach consists of an ensemble of neural networks which take in sensor data from different sensor modalities and transform them into a single common top-down semantic grid representation. We find this representation favorable as it is agnostic to sensor-specific reference frames and captures both the semantic and geometric information for the surrounding scene. Because the modalities share a single output representation, they can be easily aggregated to produce a fused output. In this work we predict short-term semantic grids but the framework can be extended to other tasks. This approach offers a simple, extensible, end-to-end approach for multi-modal perception and prediction.