TALoS: Enhancing Semantic Scene Completion via Test-time Adaptation on the Line of Sight

TL;DR

TALoS introduces a test-time adaptation method for Semantic Scene Completion that leverages LiDAR observations and future data to improve scene understanding in driving scenarios.

Contribution

The paper proposes a novel test-time adaptation approach utilizing self-supervision from LiDAR observations and future data for improved SSC performance.

Findings

Significant performance improvements on SemanticKITTI dataset

Effective use of line-of-sight observations for self-supervision

Dual optimization scheme leveraging future observations

Abstract

Semantic Scene Completion (SSC) aims to perform geometric completion and semantic segmentation simultaneously. Despite the promising results achieved by existing studies, the inherently ill-posed nature of the task presents significant challenges in diverse driving scenarios. This paper introduces TALoS, a novel test-time adaptation approach for SSC that excavates the information available in driving environments. Specifically, we focus on that observations made at a certain moment can serve as Ground Truth (GT) for scene completion at another moment. Given the characteristics of the LiDAR sensor, an observation of an object at a certain location confirms both 1) the occupation of that location and 2) the absence of obstacles along the line of sight from the LiDAR to that point. TALoS utilizes these observations to obtain self-supervision about occupancy and emptiness, guiding the model to adapt to the scene in test time. In a similar manner, we aggregate reliable SSC predictions among multiple moments and leverage them as semantic pseudo-GT for adaptation. Further, to leverage future observations that are not accessible at the current time, we present a dual optimization scheme using the model in which the update is delayed until the future observation is available. Evaluations on the SemanticKITTI validation and test sets demonstrate that TALoS significantly improves the performance of the pre-trained SSC model. Our code is available at https://github.com/blue-531/TALoS.