LODE: Locally Conditioned Eikonal Implicit Scene Completion from Sparse LiDAR

TL;DR

This paper introduces a locally conditioned Eikonal implicit scene completion method for large-scale, sparse LiDAR point clouds, significantly improving accuracy in dense scene reconstruction for autonomous driving applications.

Contribution

It proposes a novel Eikonal formulation conditioned on localized shape priors, enabling effective scene completion on large, non-watertight LiDAR data, surpassing previous methods.

Findings

IoU improved from 31.7% to 51.2% on SemanticKITTI

IoU improved from 40.5% to 48.7% on SemanticPOSS

Method is robust to hyper-parameter variations

Abstract

Scene completion refers to obtaining dense scene representation from an incomplete perception of complex 3D scenes. This helps robots detect multi-scale obstacles and analyse object occlusions in scenarios such as autonomous driving. Recent advances show that implicit representation learning can be leveraged for continuous scene completion and achieved through physical constraints like Eikonal equations. However, former Eikonal completion methods only demonstrate results on watertight meshes at a scale of tens of meshes. None of them are successfully done for non-watertight LiDAR point clouds of open large scenes at a scale of thousands of scenes. In this paper, we propose a novel Eikonal formulation that conditions the implicit representation on localized shape priors which function as dense boundary value constraints, and demonstrate it works on SemanticKITTI and SemanticPOSS. It can also be extended to semantic Eikonal scene completion with only small modifications to the network architecture. With extensive quantitative and qualitative results, we demonstrate the benefits and drawbacks of existing Eikonal methods, which naturally leads to the new locally conditioned formulation. Notably, we improve IoU from 31.7% to 51.2% on SemanticKITTI and from 40.5% to 48.7% on SemanticPOSS. We extensively ablate our methods and demonstrate that the proposed formulation is robust to a wide spectrum of implementation hyper-parameters. Codes and models are publicly available at https://github.com/AIR-DISCOVER/LODE.

Tables

Table 1. TABLE I: Scene Completion Results Measured in IoU (%).

Method	Reference	SemanticKITTI	SemanticPOSS
Input	-	10.3	13.0
SIREN [6]	NeurIPS 2020	26.3	36.0
Fourier Features [41]	NeurIPS 2020	28.6	30.9
BACON [42]	CVPR 2022	30.5	\ul40.5
DiGS [43]	CVPR 2022	\ul31.7	37.4
LODE (Ours)	-	51.2 (+19.5)	48.7 (+8.2)

Table 2. TABLE II: Discriminative Model.

Pruning Blocks	Output Block	IoU (%)
Last 1	2 convs	49.5
Last 2	2 convs	49.1
Last 3	2 convs	50.6
Last 4	2 convs	51.0
All	2 convs	51.1
All	4 convs	50.9

Table 3. TABLE VI: Sampling Strategy.

Sample Strategy	IoU (%)
Trilinear	51.0
Nearest	48.1

Table 4. TABLE VII: Semantic Scene Completion Results on SemanticKITTI.

Approach	ext.A	ext.B	JS3C	LODE w/ JS3C
mIoU (%)	20.2	18.0	\ul22.7	23.4

Equations

$$\begin{array}{rl}{\mathcal{C}}_m(\text{a}(\text{x}),& \Phi (\text{x}),{\nabla }_{\text{x}}\Phi (\text{x}))=0,\\ & \forall \text{x}\in {\Omega }_m,m=0,...,M-1.\end{array}$$

$${\mathcal{C}}_0:=|{\nabla }_{\text{x}}\Phi (\text{x})|-1,\text{x}\in {\Omega }_0.$$

$${\mathcal{C}}_1:={\nabla }_{\text{x}}\Phi (\text{x})-\text{n}(\text{x}),\text{x}\in {\Omega }_1.$$

$${\mathcal{C}}_2:=\Phi (\text{x})-\mathrm{SDF}(\text{x}),\text{x}\in {\mathrm{\Omega }}_2.$$

$${\mathcal{C}}_2:=\Phi (\text{x}),\text{x}\in {\Omega }_1.$$

$${\mathcal{C}}_3:=\psi (\Phi (\text{x})),\text{x}\in {\Omega }_3.$$

$${\mathcal{C}}_0^{\prime }:=|{\nabla }_{\text{x}}\Phi (\text{x},\text{e}){|}_{\text{e}=\zeta (\text{x},{\Omega }_1)}|-1,\text{x}\in {\Omega }_0.$$

$${\mathcal{C}}_4:=\rho (\zeta (\text{x},{\Omega }_1)),\text{x}\in {\Omega }_0.$$

$${\mathcal{C}}_1^{\prime }:={\nabla }_{\text{x}}\Phi (\text{x},\text{e}){|}_{\text{e}=\zeta (\text{x},{\Omega }_1)}-\text{n}(\text{x}),\text{x}\in {\Omega }_1^{\prime }.$$

$${\mathcal{C}}_2^{\prime }:=\Phi (\text{x},\text{e}){|}_{\text{e}=\zeta (\text{x},{\Omega }_1)},\text{x}\in {\Omega }_1^{\prime }.$$

$${\mathcal{C}}_3^{\prime }:=\psi (\Phi (\text{x},\text{e}){|}_{\text{e}=\zeta (\text{x},{\Omega }_1)}),\text{x}\in {\Omega }_3^{\prime }.$$

$$\begin{array}{rl}{\mathcal{L}}_{\mathrm{LODE}}& ={\lambda }_1{\int }_{{\Omega }_0}\Vert |{\nabla }_{\text{x}}\Phi (\text{x},\text{e}){|}_{\text{e}=\zeta (\text{x},{\Omega }_1)}|-1\Vert d\text{x}\\ & +{\lambda }_2{\int }_{{\Omega }_1^{\prime }}(1-⟨{\nabla }_{\text{x}}\Phi (\text{x},\text{e}){|}_{\text{e}=\zeta (\text{x},{\Omega }_1)},\text{n}(\text{x})⟩)d\text{x}\\ & +{\lambda }_3{\int }_{{\Omega }_1^{\prime }}\Vert \Phi (\text{x},\text{e}){|}_{\text{e}=\zeta (\text{x},{\Omega }_1)}\Vert d\text{x}\\ & +{\lambda }_4{\int }_{{\Omega }_3^{\prime }}\psi (\Phi (\text{x},\text{e}){|}_{\text{e}=\zeta (\text{x},{\Omega }_1)})d\text{x},\end{array}$$

$$\begin{array}{rl}{\mathcal{L}}_{\mathrm{com}}=-\frac{1}{m}\sum _{i=1}^m& \frac{1}{n_i}\sum _{j=1}^{n_i}[y_{i,j}\log (p_{i,j})\\ & +(1-y_{i,j})\log (1-p_{i,j})],\end{array}$$

$$\begin{array}{rl}& e_i^c=\sum _m^{D_{\mathrm{se}}}\sum _n^{W_{\mathrm{se}}}\sum _k^{H_{\mathrm{se}}}{e}_{mnk}^c\times \max (0,1-|x_i-x_m|)\\ & \times \max (0,1-|y_i-y_n|)\times \max (0,1-|z_i-z_k|),\end{array}$$

$$\begin{array}{rl}& \frac{\partial e_i^c}{\partial e_{mnk}^c}=\sum _m^{D_{\mathrm{se}}}\sum _n^{W_{\mathrm{se}}}\sum _k^{H_{\mathrm{se}}}\max (0,1-|x_i-x_m|)\\ & \times \max (0,1-|y_i-y_n|)\times \max (0,1-|z_i-z_k|).\end{array}$$

$$\begin{array}{rl}{\gamma }_{\mathrm{enc}}(p)=(\sin (2^0\pi p),& \cos (2^0\pi p),\cdots ,\\ & \sin (2^{L-1}\pi p),\cos (2^{L-1}\pi p)).\end{array}$$

$$\begin{array}{rl}\Phi (\text{x})=& {\mathbf{W}}_n({\varphi }_{n-1}\circ {\varphi }_{n-2}\circ \cdots \circ {\varphi }_0)(\text{x})+{\mathbf{b}}_n,\\ & {\text{x}}_j\mapsto {\varphi }_j({\text{x}}_j)=\sin ({\mathbf{W}}_j{\text{x}}_j+{\mathbf{b}}_j),\end{array}$$

$${\mathcal{L}}_{\mathrm{seg}}=-\frac{1}{N_{\mathrm{seg}}}\sum _{i=1}^{N_{\mathrm{seg}}}\sum _{c=1}^C{y}_{i,c}\log (p_{i,c}),$$

$$\begin{array}{r}{\mathcal{L}}_{\mathrm{total}}={\mathcal{L}}_{\mathrm{LODE}}+{\lambda }_5{\mathcal{L}}_{\mathrm{com}}+{\lambda }_6{\mathcal{L}}_{\mathrm{seg}}.\end{array}$$

Full text

I Introduction

Representing 3D data with neural implicit functions is actively explored recently due to its strong modeling capability and memory efficiency [1, 2, 3, 4, 5, 6, 7]. Meanwhile, it can be easily meshed and rendered to facilitate human viewing. While most methods are fully supervised [1, 2, 3], SIREN [6] proposes an Eikonal implicit scene completion method with weak supervision needed. It learns a signed distance function (SDF), which measures the nearest distance to the scene surface, through the process of solving an Eikonal differential equation with only points on the surface and without knowing SDF values in free space. This scheme is promising for large-scale sparse LiDAR data because (1) for non-water-tight scenes, it is hard to define signed distance in free space and (2) it requires less supervision than non-Eikonal formulations and thus is simpler, especially when considering the completion of thousands of scenes.

However, even after an exhaustive parameter search, SIREN fails to fit sparse LiDAR data (Fig. 1-a), which limits its application in many important scenarios such as autonomous driving. This is understandable as SIREN is a pure generative model and LiDAR point clouds are extremely sparse. Specifically, the reasons are three-fold: (1) The sparsity of on-surface points amplifies the negative impact of wrongly sampled off-surface anchors. (2) The normal orientations of sparse points cannot be estimated accurately from their neighbors, which serve as a necessary boundary value constraint for SIREN fitting. (3) Without trustworthy boundary values, enforcing a hard Eikonal constraint leads to even inaccurate SDF values. As shown in Fig. 1-b, the SIREN fitting result is fragmented.

To overcome these limitations, we develop a novel Eikonal implicit formulation by introducing an intermediate embedding domain, where localized shape priors are contained. Instead of directly fitting a function to map 3D Cartesian coordinates to signed distances, we first map the Euclidean space to a corresponding high-dimensional shape embedding space, and then the signed distance space. These shape embeddings function as dense boundary values that entangle both zeroth-order (on-surface points) and first-order (normal directions) constraints, in a data-driven manner. Naturally, the issue of enforcing a hard Eikonal constraint is also alleviated. This proposed formulation is named Locally Conditioned Eikonal Formulation and abbreviated as LODE. The result of LODE is significantly better than SIREN (Fig. 1-c). And the supplementary video demonstrates LODE performs well on in-the-wild sequential LIDAR inputs.

Specifically, to implement LODE, we propose a novel hybrid architecture combining a discriminative model and a generative model. The discriminative part of our method exploits the strong representation learning power of sparse convolution, generating latent shape embeddings from sparse point cloud input. The generative model takes as input the ground truth point cloud coordinates along with pointwise latent shape embeddings retrieved by trilinear sampling and predicts SDF values of these points. During inference, the ground truth points are replaced with the points of interest.

Furthermore, to demonstrate the flexibility of LODE, we extend our method to implicit semantic completion in two ways: (1) by adding a dense discriminative head to predict semantic labels which can be mapped to the implicit function using K-Nearest-Neighbors; (2) by adding a parallel implicit generative head to directly model the implicit semantic field. We evaluate them on SemanticKITTI and achieve results (Fig. 1-d) comparable to state-of-the-art methods.

To summarize, our contributions are as follows:

• $\bullet$

We develop a locally conditioned Eikonal implicit scene completion formulation that incorporates learned shape priors as dense boundary value constraints.

• $\bullet$

We apply the formulation in road scene understanding, leading to the first Eikonal implicit road scene completion method without knowing SDF values in free space.

• $\bullet$

We achieve state-of-the-art completion results on SemanticKITTI and SemanticPOSS, outperforming the best Eikonal completion results by +19.5% and +8.2% IoU. Code, data, and models will be released.

II Related Works

Neural Implicit Representation. The general principle of neural implicit representation is to train a neural network to approximate a continuous function that is hard to parameterize otherwise. [1] proposes to learn deep signed distance functions conditioned on shape embeddings. [2] approximates occupancy functions with conditional batchnorm networks. [8] introduces data-driven shape embeddings into occupancy networks for indoor scene completion. [3] uses hyperplanes as compact implicit representations to reconstruct shapes sharply and compactly. [6] shows that using gradient supervision allows Eikonal SDF learning with only on-surface SDF value supervision and sine activations are critical to its success. [9] combines Gaussian ellipsoids and implicit residuals to represent shapes accurately. Some recent works exploit 3D implicit representations for instance-level understanding from point cloud [10] or RGB [11] inputs. Despite these advances, there is no work yet on implicit scene completion on LiDAR point clouds where the data is extremely sparse with heterogeneous distribution. Our method bridges this gap with the proposed locally conditioned Eikonal formulation.

LiDAR-based scene understanding. While there are many advances in camera-based cognitive scene understanding [12, 13, 14, 15, 16], LiDAR point cloud provides reliably accurate 3D structural information and thus has been mainly leveraged in geometric scene understanding. Numerous LiDAR-based SLAM methods are proposed to improve the quality and real-time performance [17, 18, 19, 20, 21]. [22, 23, 24] further incorporate semantic understanding into LiDAR SLAM systems. These methods explicitly complete a scene with multiple frames of LiDAR data, while our goal is to achieve implicit completion with a single frame. Sparse LiDAR points are also efficiently leveraged in depth completion [25, 26], object detection [27, 28], segmentation [29, 30, 31, 32, 33, 34]. We believe the performance of these methods will be boosted with LODE completing the original sparse representation. Moreover, by providing fine geometric details, LODE may aid in the detection of anomalous obstacles [35] and some other tasks.

III Formulation

III-A Eikonal Implicit Completion Formulation

Eikonal completion methods aim at fitting the signed distance function (SDF) of a scene. The signed distance is the nearest distance from a point of interest to the scene surface, with the sign denoting whether the point is located outside (positive) or inside (negative) of the surface. The iso-surface where the signed distance equals zero implicitly delineates the scene. Formally, the goal of Eikonal implicit completion is to find a function $\Phi(\textbf{x})$, which satisfies a set of $M$ constraints $\mathcal{C}_{m}$, to approximate the underlying SDF. Each constraint relates the function $\Phi(\textbf{x})$ or its gradient to certain input quantities $\textbf{a}(\textbf{x})$ on the corresponding domain $\Omega_{m}$:

[TABLE]

Specifically, these constraints are required:

[TABLE]

[TABLE]

[TABLE]

Here, $\mathcal{C}_{0}$ guarantees $\Phi(\textbf{x})$ satisfies the Eikonal equation in the whole physical space of interest ($\Omega_{0}$), which is a intrinsic property of SDF. $\mathcal{C}_{1}$ forces that the gradients of $\Phi(\textbf{x})$ equal the normal vectors for input on-surface points ($\Omega_{1}$). $\mathcal{C}_{2}$ constrains the values of $\Phi(\textbf{x})$ equal the ground truth SDF for labeled anchor points ($\Omega_{2}$). In this way, the problem can be regarded as an Eikonal boundary value problem, where the differential equation $\mathcal{C}_{0}$ is to be solved under the first-order constraint $\mathcal{C}_{1}$ and the zeroth-order constraint $\mathcal{C}_{2}$.

However, the ground truth SDF values in free space are difficult to obtain. A recent method named SIREN [6] proposes an intriguing variant where the domain of $\mathcal{C}_{2}$ is limited to on-surface points in $\Omega_{1}$. As the ground truth SDF values of points in $\Omega_{1}$ are zero, $\mathcal{C}_{2}$ is reduced to:

[TABLE]

To remedy the lack of constraints on off-surface points, SIREN introduces another constraint:

[TABLE]

Here, $\psi$ pushes $\Phi(\textbf{x})$ values away from 0, for randomly and uniformly sampled off-surface points ($\Omega_{3}\subseteq\Omega_{0}\setminus\Omega_{1}$).

Nevertheless, these constraints fail to cope with the scenario where on-surface points in $\Omega_{1}$ are sampled from sparse LiDAR point cloud data. Reasons are three-fold: (1) The sparsity of on-surface points in $\Omega_{1}$ amplifies the negative impact of $\mathcal{C}_{3}$ on the wrongly sampled off-surface anchors in $\Omega_{3}$ (i.e., located on or near the surface). (2) The normal orientations of sparse points in $\Omega_{1}$ cannot be estimated accurately from their neighbors, leading to an incorrect constraint $\mathcal{C}_{1}$. (3) Without trustworthy boundary value constraints $\mathcal{C}_{3}$ and $\mathcal{C}_{1}$, enforcing the hard Eikonal constraint $\mathcal{C}_{0}$ leads to even inaccurate SDF values in free space.

III-B Locally Conditioned Eikonal Formulation (LODE)

To overcome the aforementioned limitations, we propose a locally conditioned Eikonal formulation $\Phi(\textbf{x},\textbf{e})|_{\textbf{e}=\zeta(\textbf{x},\Omega_{1})}$ to approximate SDF. Here, we use $\zeta(\cdot,\cdot)$ to first map the Euclidean space to a high-dimensional shape embedding space. It functions as a dense boundary value constraint for the differential equation. Then $\Phi(\cdot,\cdot)$ maps the shape embedding space to the signed distance space. As a result, the constraints to be satisfied are formally re-written as:

[TABLE]

[TABLE]

We use $\rho(\zeta(\textbf{x},\Omega_{1}))$ to represent the underlying dense constraint contained in the shape embedding space, which implicitly entangles correct $\mathcal{C}_{1}$, $\mathcal{C}_{2}$, and $\mathcal{C}_{3}$ constraints of the modified formulations:

[TABLE]

[TABLE]

[TABLE]

Here, $\Omega_{1}^{\prime}$ contains the dense ground truth on-surface points and $\Omega_{3}^{\prime}\subseteq\Omega_{0}\setminus\Omega_{1}^{\prime}$. Hence the aforementioned problem of trustworthy boundary values is resolved. Naturally, the issue of enforcing a hard Eikonal constraint is also alleviated.

We implement the proposed LODE formulation in a data-driven manner. The acquisition of functions $\zeta(\cdot,\cdot)$ and $\Phi(\cdot,\cdot)$ can be cast in a loss function that penalizes deviations from the constraints $\mathcal{C}_{0}^{\prime}$, $\mathcal{C}_{1}^{\prime}$, $\mathcal{C}_{2}^{\prime}$, and $\mathcal{C}_{3}^{\prime}$ on their domain:

[TABLE]

where $\lambda_{1}$ - $\lambda_{4}$ are constant weight parameters and $\langle\cdot,\cdot\rangle$ calculates cosine similarity between two vectors.

IV Method

To realize LODE, we propose a hybrid neural network architecture combining a discriminative model with a generative model, as shown in Fig. 2. The discriminative part exploits the strong representation learning power of sparse convolution, generating latent shape embeddings from sparse input $\Omega_{1}$. It together with the differentiable trilinear sampling module works as function $\zeta(\cdot,\cdot)$. The generative model consists of an MLP, functioning as $\Phi(\cdot,\cdot)$. It takes as input the encoded coordinates of ground truth points $\Omega_{1}^{\prime}$ along with pointwise latent shape embeddings and predicts SDF values of these points. Using gradient descent, we can get the optimized $\zeta(\cdot,\cdot)$ and $\Phi(\cdot,\cdot)$ in the parameterized form.

IV-A Discriminative Model

Intuitively, road scenes have the characteristic of repetition. Thus convolutional neural network can be employed as the discriminative model to exploit the translation invariance.

Taking LiDAR points $\Omega_{1}$ as input, we first conduct voxelization to obtain 3D occupancy volume $V_{\rm{occ}}$ with size $1\times D_{\rm{occ}}\times W_{\rm{occ}}\times H_{\rm{occ}}$. Then the discriminative model maps it into a shape embedding volume $V_{\rm{se}}$ with size $d_{\rm{se}}\times D_{\rm{se}}\times W_{\rm{se}}\times H_{\rm{se}}$, where $d_{\rm{se}}$ is the dimension of the shape embedding outputs. To tackle the sparsity of $V_{\rm{occ}}$, we employ the sparse operations of the Minkowski Engine [36] to build the model, which is a multiscale encoder-decoder network. It extracts shape priors via a shape completion process: the encoder consisting of convolutional blocks aggregates localized features, and the decoder involving generative deconvolutional blocks generates dense results. Yet the constant generation of new voxels will destroy the sparsity just as the submanifold dilation problem [37]. To avoid this, we use a pruning block to prune off redundant voxels. It contains a convolutional layer to determine the binary classification result of whether a voxel should be pruned, which is supervised with binary cross-entropy loss:

[TABLE]

where $m$ is the count of supervised blocks, $n_{i}$ denotes the count of voxels in the $i$-th block, $y_{i,j}$ and $p_{i,j}$ are the true and predicted existence probabilities for voxel $i$ respectively.

IV-B Differentiable Trilinear Sampling Module

After generating $V_{\rm{se}}$, pointwise shape embedding $\textbf{e}_{i}\in\mathbb{R}^{d_{\rm{se}}}$ for query point $\textbf{x}_{i}\in\Omega_{0}$ is needed. We use trilinear interpolation to sample $\textbf{e}_{i}$ for $\textbf{x}_{i}$ from its 8 nearest voxel centers to maintain the continuity of the latent shape field at the voxel borders. Formally, with the length of voxel edge normalized, the trilinear sampling for $\textbf{e}_{i}$ can be written as:

[TABLE]

where $e_{i}^{c}$ and $e_{mnk}^{c}$ are shape embeddings on channel $c$ for $\textbf{x}_{i}=(x_{i},y_{i},z_{i})$ and voxel center $\textbf{x}_{mnk}=(x_{m},y_{n},z_{k})$. Then the gradient with respect to $\textbf{e}_{mnk}$ for backpropagation is:

[TABLE]

This differentiable trilinear sampling mechanism allows loss gradients to flow back to $V_{\rm{se}}$ and further back to the discriminative model, making it possible to train discriminative model and the following generative model cooperatively.

IV-C Positional Encoding Module

Positional encoding has proved an effective technique in neural rendering[4] [38] for its capacity to capture high-frequency information. Thus we leverage it to represent more geometric details of the signed distance field. Specifically, the 3D Cartesian coordinate $\textbf{x}_{i}$ is encoded into high-dimensional feature $\textbf{y}_{i}=(\gamma_{\rm{enc}}(x_{i}),\gamma_{\rm{enc}}(y_{i}),\gamma_{\rm{enc}}(z_{i}))\in\mathbb{R}^{d_{\rm{enc}}}$, where

[TABLE]

$L$ is the number of frequency octaves and thus $d_{\rm{enc}}=6L$.

IV-D Generative Model

We use an MLP as the generative model for implicit SDF representation and use sine as a periodic activation function to better model details [6]. Thus $\Phi$ can be formalized as:

[TABLE]

where $\phi_{j}:\mathbb{R}^{M_{j}}\mapsto\mathbb{R}^{N_{j}}$ is the $j^{th}$ layer of the model. Given ${\textbf{x}}_{j}\in\mathbb{R}^{M_{j}}$, the layer applies the affine transform with weights ${\mathbf{W}}_{j}\in\mathbb{R}^{N_{j}\times M_{j}}$ and biases ${\mathbf{b}}_{j}\in\mathbb{R}^{N_{j}}$ on it, and then pass the resulting vector to the sine nonlinearity.

In our implementation, the concatenated vector $[\textbf{y}_{i},\textbf{e}_{i}]$ is taken as input. And model weights are shared for all scenes. We use the proposed loss function (12) to optimize model weights and shape embeddings. Note that during training, $\textbf{x}_{i}$ is sampled from dense ground truth $\Omega_{1}^{\prime}$ instead of sparse input $\Omega_{1}$. Thus, in a data-driven manner, our generative model can effectively map the shape embedding space to the signed distance space with abundant geometric information.

IV-E Semantic Extension

To demonstrate the flexibility of LODE, we extend our method to implicit semantic completion in two ways.

Semantic Extension A. We first semantically segment $V_{\rm{occ}}$ with a sparse CNN. Then a dense CNN is used to predict coarse semantic completion results. Mapping it to our implicit representation using K-Nearest-Neighbor, we get the refined implicit semantic results.

Semantic Extension B. We add a parallel implicit generative head to directly model the implicit semantic label field. Its structure is similar to aforementioned generative model, except that it outputs the probabilities of label classification.

The results are supervised with a cross-entropy loss:

[TABLE]

where $y_{i,c}$ and $p_{i,c}$ are the true and predicted probabilities. $N_{\rm{seg}}$ points and $C$ categories are considered.

IV-F Training and Inference

During training, we randomly sample $N_{\rm{on}}$ on-surface points from $\Omega_{1}^{\prime}$ and $N_{\rm{off}}$ off-surface points from $\Omega_{3}^{\prime}$, optimizing the whole neural network with loss:

[TABLE]

Here, $\lambda_{5}$ and $\lambda_{6}$ are constant weight parameters. Note that $\lambda_{6}=0$ when the semantic extension is not included.

During inference, we uniformly sample $N_{\rm{inf}}^{3}$ points from $\Omega_{0}$ at a specified resolution. And we use a threshold $v_{\rm{th}}$ close to zero to select the points with estimated SDF values smaller than $v_{\rm{th}}$ as explicit surface points for evaluation.

V Experiments

Dataset. We evaluate the proposed LODE on SemanticKITTI [39] and SemanticPOSS [40]. There are $22$ sequences (8550 scans) and 6 sequences (2988 scans) of road scene LiDAR data in the two datasets respectively. Each scan covers a range of 51.2m ahead of the LiDAR, 25.6m to each side, and 6.4m in height. We follow the official split for training and validation. Metric. We use the interactions over union (IoU) metric for evaluation. Implementation Details. For the discriminative model, we set $D_{\rm{occ}}=256$, $W_{\rm{occ}}=256$, $H_{\rm{occ}}=32$, $m=5$. For the generative model, we use $N_{\rm{on}}=N_{\rm{off}}=16000$ and $N_{\rm{inf}}=256$. For training, we set $\lambda_{1}=3000$, $\lambda_{2}=100$, $\lambda_{3}=100$, $\lambda_{4}=50$, $\lambda_{5}=100$ and use the Adam optimizer with an initial learning rate of $10^{-4}$. When the semantic extension is included, $\lambda_{6}=50$.

V-A Scene Completion Effectiveness of LODE

We compare our LODE with other recent strong Eikonal completion methods on the validation sets of SemanticKITTI and SemanticPOSS. The results of these Eikonal methods are obtained by applying them to every LiDAR sweep and taking the average. In Table.I, the first row shows that directly comparing the input sparse point cloud with completion ground truth yields 10.3% and 13.0% IoU. The existing Eikonal methods improve the IoU to 31.7% and 40.5%. Thanks to the new locally conditioned Eikonal formulation, our approach further improves IoU to 51.2% and 48.7%. These results demonstrate the effectiveness of LODE.

This large improvement is better demonstrated with qualitative results in Fig. 3. Though the existing Eikonal methods are successful for clean synthetic point cloud data uniformly sampled on watertight meshes as shown in [6], fitting large-scale outdoor scenes captured by LiDAR (Fig.3-a) is much more difficult. On the one hand, many regions are not sampled thus missing in the point cloud. On the other hand, caused by the mechanism of LiDAR, data sparsity increases with distance and it is extremely sparse at the far end. These result in the lack of effective boundary values for solving the Eikonal differential equation. For this reason, as pure generative models, these methods fail to fit road scenes and produce lots of artifacts (Fig.3-c,d). Our LODE, on the contrary, takes data-driven shape priors generated by a strong sparse convolutional network as the dense boundary values and successfully completes the scenes. As shown in Fig.3-e and highlighted in red boxes, both occluded and incomplete regions are better reconstructed than other Eikonal methods.

In order to show that the significant margins reported in Table. I are robust to Marching Cubes thresholds, we provide an exhaustive evaluation in Fig.4. It is clear that our method outperforms other methods under all inspected thresholds.

V-B Implementation Robustness of LODE

To better understand LODE and demonstrate its robustness to hyper-parameters in implementation, we provide a series of ablation studies on SemanticKITTI as follows.

Discriminative model design. We investigate two factors: (1) Where to add pruning blocks; (2) Conv layer number in the output block that generates shape embeddings. As shown in Table.V, LODE is robust to these design choices.

Does generative model capacity matter? Deeper and wider models usually achieve better results for recognition. To explore whether generative model capacity matters for LODE, we ablate the width, depth, and activations of the MLP. As shown in Table.V, different configurations produce similar results. It demonstrates the capacity of generative model is not a performance bottleneck. Interestingly, using ReLU instead of Sine activation only brings a performance drop of 1.75%. It suggests that in challenging scenarios like ours, using Sine activation is not as critical as in SIREN [6].

Which dimension of shape volume matters? To study which factor is the deciding one for the representation power of the shape volume, we evaluate different shape embedding dimensions and scale sizes. By scale size, we mean the down-sampling ratio $\frac{D_{\rm{occ}}}{D_{\rm{se}}}$. The results are summarized in Table.V, showing that using shape embeddings of dimension 128 is already capable of representing our scenes well. But increasing scale size leads to a sharp drop of IoU, which reflects the importance of the locality of shape priors.

Is trilinear sampling necessary? We justify the necessity of trilinear sampling in our method using Table. VI. A trivial nearest neighbor sampling leads to a performance drop of 2.9%. This is a clear margin that shows the benefit of smoothly interpolating shape embeddings.

How to encode positional information? The goal of positional encoding is to represent fine geometric details of the scene. We investigate positional encoding levels and whether to concatenate original coordinates. As shown in Table.V, when positional encoding is not used or the encoding level is low, the completion IoU decreases dramatically. Through the qualitative results in Fig. 5-a, it is clear that leaving out positional encoding leads to the loss of details.

With these ablations and analyses, we demonstrate the role of each module and the impact of hyper-parameters in detail. Meanwhile, despite the performance fluctuation under different hyper-parameters, our quantitative results are always better than the existing Eikonal methods shown in the second to the fifth row of Table.I, which demonstrates the effectiveness and robustness of LODE.

V-C Flexibility of LODE

Table.VII shows semantic scene completion results on the SemanticKITTI validation set, which is evaluated on 19 categories. With little impact on original completion results, the semantic extension A and B achieve 20.2% and 18.0% mIoU, respectively. Although they under-perform the state-of-the-art method JS3C-Net [44], our models allow implicit completion and models the signed distance field. Qualitative results shown in Fig. 3-f demonstrate faithful semantic implicit completion. Last but not least, we map explicit semantic completion results from JS3C-Net to our implicit completion using K-Nearest-Neighbors, achieving 23.4% mIoU. These results show the flexibility of LODE as it can be easily extended to provide semantic information.

V-D Other Potential Benefits of LODE

As illustrated in Fig.6, with LODE, we can get mesh reconstructions at any resolution, which is due to its continuous representation of the signed distance field. Meanwhile, as shown in Fig.1 and Fig.3, LODE enables the indiscernible LiDAR data to be converted into human-friendly visualizations, which demonstrates its capacity to enhance human understanding of robot-perceived information. Moreover, the proposed LODE has compatibility with implicit planning algorithms [45, 46, 47, 48, 49], and thus can be leveraged to facilitate downstream robotic manipulation tasks.

VI Conclusion

In this study, we propose a novel locally conditioned Eikonal formulation named LODE for implicit scene completion. Learned shape embeddings are treated as dense boundary values that constrain signed distance function learning. We implement the formulation as a hybrid neural network combining discriminant and generative models. The network is trained to implicitly fit road scenes captured by sparse LiDAR point clouds, without accessing exact SDF values in free space. Large-scale evaluations on SemanticKITTI and SemanticPOSS show that our method outperforms existing Eikonal methods by a large margin. We also extend the proposed method for semantic implicit completion in two ways, achieving strong qualitative and quantitative results.

ACKNOWLEDGEMENTS

This work was sponsored by Tsinghua-Toyota Joint Research Fund (20223930097) and Baidu Inc. through Apollo-AIR Joint Research Center.