Not Only Look But Observe: Variational Observation Model of Scene-Level 3D Multi-Object Understanding for Probabilistic SLAM

TL;DR

This paper introduces NOLBO, a variational observation model that estimates scene-level 3D multi-object understanding from 2D images, enhancing probabilistic SLAM by modeling object relations and scene context.

Contribution

It proposes a novel variational auto-encoder based approach to approximate the Bayesian observation model for scene-level multi-object 3D understanding from single shots.

Findings

Enables probabilistic inference with object relations and scene context.

Estimates 3D shape and pose from 2D images using latent variables.

Facilitates object-oriented data association in SLAM.

Abstract

We present NOLBO, a variational observation model estimation for 3D multi-object from 2D single shot. Previous probabilistic instance-level understandings mainly consider the single-object image, not single shot with multi-object; relations between objects and the entire scene are out of their focus. The objectness of each observation also hardly join their model. Therefore, we propose a method to approximate the Bayesian observation model of scene-level 3D multi-object understanding. By exploiting variational auto-encoder (VAE), we estimate latent variables from the entire scene, which follow tractable distributions and concurrently imply 3D full shape and pose. To perform object-oriented data association and probabilistic simultaneous localization and mapping (SLAM), our observation models can easily be adopted to probabilistic inference by replacing object-oriented features with latent variables.

Tables

Table 1. Table 1: Comparison of the Viewpoint Estimations with Ground Truth Bounding Box on Pascal3D+ test dataset

	aero	bike	boat	bottle	bus	car	chair	table	mbike	sofa	train	tv	mean
Acc${}_{\frac{\pi }{6}}$ ([47])	0.81	0.77	0.59	0.93	0.98	0.89	0.80	0.62	0.88	0.82	0.80	0.80	0.81
Acc${}_{\frac{\pi }{6}}$ ([28])	0.78	0.83	0.57	0.93	0.94	0.90	0.80	0.68	0.86	0.82	0.82	0.85	0.81
Acc${}_{\frac{\pi }{6}}$ (Ours)	0.83	0.86	0.70	0.90	0.95	0.96	0.95	0.83	0.83	0.98	0.94	0.91	0.88
MedErr ([47])	13.8	17.7	21.3	12.9	5.8	9.1	14.8	15.2	14.7	13.7	8.7	15.4	13.6
MedErr ([28])	13.6	12.5	22.8	8.3	3.1	5.8	11.9	12.5	12.3	12.8	6.3	11.9	11.1
MedErr (Ours)	14.5	16.5	17.8	10.5	10.1	8.6	11.4	13.7	16.9	10.7	9.2	14.1	11.7

Table 2. Table 2: Average precisions of the trajectory estimation results evaluated on KITTI dataset

	NOLBOMulti16	NOLBOMulti128	NOLBOSingle16	AE16	TLNet16	vVAE16	VO
ATE	25.93	28.66	52.79	37.88	62.56	64.78	30.75
RPE	29.88	31.71	65.03	40.83	73.02	72.07	35.40

Equations

$$\log p\left({\mathcal{S}}^f,\mathcal{L},\mathcal{V},\mathcal{B}\right)$$

$$=\log \sum _{\mathcal{O}}p\left(\mathcal{L},\mathcal{O}\right)p\left(\mathcal{V}\right)p\left({\mathcal{S}}^f|\mathcal{L},\mathcal{V}\right)p\left(\mathcal{B}\right)$$

$$=\log p\left(\mathcal{V}\right)+{\underbrace{\log \sum _{\mathcal{O}}p\left(\mathcal{L},\mathcal{O}\right)}}_{(a)}+{\underbrace{\log p\left({\mathcal{S}}^f|\mathcal{L},\mathcal{V}\right)p\left(\mathcal{B}\right)}}_{(b)}$$

$$\log \sum _{\mathcal{O}}p\left(\mathcal{L},\mathcal{O}\right)$$

$$\ge -KL\left(q_{\varphi }\left(\mathcal{O}|I\right)||p\left(\mathcal{O}|\mathcal{L}\right)\right)+c_1.$$

$$\log p\left({\mathcal{S}}^f|\mathcal{L},\mathcal{V}\right)p\left(\mathcal{B}\right)$$

$$\ge -KL\left(q_{\varphi }\left({\mathcal{Z}}^l,{\mathcal{Z}}^v|I\right)||p_{\psi }\left({\mathcal{Z}}^l,{\mathcal{Z}}^v|\mathcal{L},\mathcal{V}\right)\right)$$

$$+{\mathbb{E}}_q\left[\log p_{\theta }\left({\mathcal{S}}^f|{\mathcal{Z}}^l,{\mathcal{Z}}^v\right)\right]$$

$$-KL\left(q_{\varphi }\left({\mathcal{Z}}^b|I\right)||p\left({\mathcal{Z}}^b\right)\right)+{\mathbb{E}}_q\left[\log p_{\theta }\left(\mathcal{B}|{\mathcal{Z}}^b\right)\right]$$

$$L=\sum _{{\mathbf{l}}_k\in {\mathcal{L}}^o\cup {\mathcal{L}}^{\neg o}}-KL\left(q_{\varphi }\left(o_k|I\right)||p\left(o|{\mathbf{l}}_k\right)\right)+$$

$$\displaystyle\sum_{\boldsymbol{l}_{k}\in\mathcal{L}^{o}}\big{(}-KL(\mathop{q}_{\phi}\left(\boldsymbol{z}^{lv}_{k}|I\right)||\mathop{p}_{\psi}\left(\boldsymbol{z}^{lv}_{k}|\boldsymbol{l}_{k},\boldsymbol{v}_{k}\right))+\mathop{\mathbb{E}}_{q}[\log\mathop{p}_{\theta}(\boldsymbol{s}^{f}_{k}|\boldsymbol{z}^{lv}_{k})]\big{)}$$

$$\displaystyle+\sum_{\boldsymbol{l}_{k}\in\mathcal{L}^{o}}\big{(}-KL\left(q_{\phi}\left(\boldsymbol{z}^{b}_{k}|I\right)||p_{\psi}\left(\boldsymbol{z}^{b}_{k}\right)\right)+\mathop{\mathbb{E}}_{q}\left[\log{p}_{\theta}\left(\boldsymbol{b}_{k}|\boldsymbol{z}^{b}_{k}\right)\right]\big{)}$$

$${\mathbb{E}}_q\left[p_{\theta }\left({\mathbf{s}}^f|{\mathbf{z}}^{lv}\right)\right]$$

$$={\mathbb{E}}_q\left[p_{\theta }\left({\mathbf{s}}^{fo}|{\mathbf{z}}^l\right)\right]{\mathbb{E}}_q\left[p\left({\mathbf{R}}^f|{\mathbf{z}}^v\right)\right]$$

$$={\underbrace{{\mathbb{E}}_q\left[p_{\theta }\left({\mathbf{s}}^{fo}|{\mathbf{z}}^l\right)\right]}}_{(a)}{\underbrace{{\mathbb{E}}_q\left[p\left({\mathbf{R}}^f|{\mathbf{R}}^z\right)\right]}}_{(b)}$$

$$-KL\left(q_{\varphi }\left({\mathbf{z}}^{lv}|I\right)||p_{\psi }\left({\mathbf{z}}^{lv}|\mathbf{l},\mathbf{v}\right)\right)+{\mathbb{E}}_q\left[\log p_{\theta }\left({\mathbf{s}}^f|{\mathbf{z}}^{lv}\right)\right]$$

$$=-KL\left(q_{\varphi }\left({\mathbf{z}}^l|I\right)||p_{\psi }\left({\mathbf{z}}^l|\mathbf{l}\right)\right)+{\mathbb{E}}_q\left[\log p_{\theta }\left({\mathbf{s}}^{fo}|{\mathbf{z}}^l\right)\right]$$

$$-KL\left(q_{\varphi }\left({\mathbf{z}}^v|I\right)||p\left({\mathbf{z}}^v|\mathbf{v}\right)\right)+{\mathbb{E}}_q\left[p\left({\mathbf{R}}^f|{\mathbf{R}}^z\right)\right]$$

$$\mathcal{X},{\mathcal{X}}^{\ell }$$

$$=\underset{\mathcal{X},{\mathcal{X}}^{\ell }}{\mathrm{argmin}}\sum _{t,k,j}-w_{kj}^t\log p\left({\mathbf{x}}_k^{\mathbf{s}}|{\mathbf{x}}_t^x,{\mathbf{x}}_j^{\ell }\right)p\left(\mathbf{z}={\mathbf{v}}_k|{\mathbf{v}}_{t,j}^{l,local}\right),$$

$${\mathcal{L}}^M=\underset{{\mathcal{L}}^M}{\mathrm{argmin}}\sum _{t,k,j}-w_{kj}^t\log p_{\psi }\left(\mathbf{z}={\mathbf{\mu }}_k^{\mathbf{s}l}|{\mathbf{l}}_j\right)$$

$$\log p\left({\mathcal{S}}^f,\mathcal{L},\mathcal{V}\right)$$

$$=\sum _k-KL\left(q_{\varphi }\left(o_k|I\right)||p\left(o|{\mathbf{l}}_k\right)\right)$$

$$+\sum _k\left(-KL\left(q_{\varphi }\left({\mathbf{z}}_k^l|I\right)||p_{\psi }\left({\mathbf{z}}_k^l|{\mathbf{l}}_k\right)\right)+{\mathbb{E}}_q\left[\log p_{\theta }\left({\mathbf{s}}_k^{fo}|{\mathbf{z}}_k^l\right)\right]\right)$$

$$+\sum _k\left(-KL\left(q_{\varphi }\left({\mathbf{z}}_k^v|I\right)||p\left({\mathbf{z}}_k^v|{\mathbf{v}}_k\right)\right)+{\mathbb{E}}_q\left[p\left({\mathbf{R}}_k^f|{\mathbf{R}}_k^z\right)\right]\right).$$

$$p\left(o|\mathbf{l}\right)=\{\begin{array}{ll}ϵ& \text{, l∈L¬o, o=true}\\ 1-ϵ& \text{, l∈L¬o, o=false}\\ 1-{ϵ}^{\prime }& \text{, l∈Lo, o=true}\\ {ϵ}^{\prime }& \text{, l∈Lo, o=false}\end{array}$$

$${\mu }_i^s,{\mu }_i^c$$

$${\sigma }_i^s$$

$${\sigma }_i^c$$

$$w_{ij}^t$$

$$\mathcal{X},\mathcal{L}$$

$$p\left({\mathcal{S}}_t|\mathcal{X},\mathcal{L},{\mathcal{D}}_t\right)=p\left({\mathcal{S}}_t|{\mathcal{X}}_{\alpha },{\mathcal{L}}_{\beta }\right),$$

$$p\left({\mathcal{S}}_t|{\mathcal{X}}_{\alpha },{\mathcal{L}}_{\beta }\right)$$

$$=p\left({\mathbf{X}}_t^{\mathbf{s}}|{\mathbf{X}}_{\alpha }^x,{\mathbf{X}}_{\beta }^{\ell }\right)p\left({\mathcal{S}}_t^f|{\mathcal{L}}_{\beta },{\mathcal{V}}_{\beta }^{\ell ,global},{\mathcal{V}}_{\alpha }^x\right),$$

$${\mathbf{R}}_{{\alpha }_k,{\beta }_k}^{{\mathbf{v}}^{\ell ,local}}={\mathbf{R}}_{{\beta }_k}^{{\mathbf{v}}^{\ell ,global}}{\left({\mathbf{R}}_{{\alpha }_k}^{{\mathbf{v}}^x}\right)}^{-1},$$

$$p\left({\mathcal{S}}_t|{\mathcal{X}}_{\alpha },{\mathcal{L}}_{\beta }\right)=p\left({\mathbf{X}}_t^{\mathbf{s}}|{\mathbf{X}}_{\alpha }^x,{\mathbf{X}}_{\beta }^{\ell }\right)p\left({\mathcal{S}}_t^f|{\mathcal{L}}_{\beta },{\mathcal{V}}_{\alpha ,\beta }^{\ell ,local}\right)$$

$$w_{ij}^t$$

$$\mathcal{X},\mathcal{L}$$

$$\log p\left({\mathcal{S}}^f,\mathcal{L},\mathcal{V}\right)$$

$$\ge c_0+c_1$$

$$+\sum _k-KL\left(q_{\varphi }\left(o_k|I\right)||p\left(o|{\mathbf{l}}_k\right)\right)$$

$$+\sum _k\left(-KL\left(q_{\varphi }\left({\mathbf{z}}_k^l|I\right)||p_{\psi }\left({\mathbf{z}}_k^l|{\mathbf{l}}_k\right)\right)+{\mathbb{E}}_q\left[\log p_{\theta }\left({\mathbf{s}}_k^{fo}|{\mathbf{z}}_k^l\right)\right]\right)$$

$$+\sum _k\left(-KL\left(q_{\varphi }\left({\mathbf{z}}_k^v|I\right)||p\left({\mathbf{z}}_k^v|{\mathbf{v}}_k\right)\right)+{\mathbb{E}}_q\left[p\left({\mathbf{R}}_k^f|{\mathbf{R}}_k^z\right)\right]\right).$$

Full text

1 Introduction

Object-oriented features find various aspects such as semantic scene understanding, task planning and autonomous driving [39]. Real-time object detection and high-level feature estimation are inevitable in various applications such as object recognition, data association and object-oriented simultaneous localization and mapping (SLAM). Recently, a plethora of real-time multi-object detection methods have been developed beyond category classification for a single object image [38, 37, 24]. In some of these existing multi-object detection methods, however, the estimation results are still bound to the object categories and the location on the image (bounding box), and hardly concern other details.

For data association and object-oriented SLAM, it is better to exploit the various object representations such as complete 3D shape as well as categories. However, reconstruction methods such as [29, 50] and [49] are extremely challenging to perform in real-time on multi-object; these methods are conducted after scanning the various viewpoints of each object. Therefore, estimation methods which disentangle representations like 3D shape or viewpoint orientation from a 2D image have been carried out [54, 31]. With the emergence of deep learning, direct inference methods are also developed [41, 33, 52, 57]. Furthermore, by using the multi-object detectors or their network structures, estimation methods for instance-level understanding of multi-object are also studied [60, 21, 31].

These approaches, however, mainly focus on directly obtaining the disentangled features via network modeling, not probabilistic model. Hence data association in Bayesian manner for probabilistic SLAM becomes challenging. Even when outputs of hidden layers of the network are used as object features, it is still challenging to address the Bayesian inference since features follow intractable distributions [53, 52]. As a result, in most cases there is little choice but to look at the outputs of well-designed networks; they just look at the objects so far, but not observe them in the true sense.

In order to approximate the intractable object observation model, [58] and [59] adopt the evidence lower bound (ELBO). However, these methods mainly focus on a single object and hardly consider multi-object observation. For single scene with multi-object, most of the instance-level understandings [11, 52, 54] and object-oriented SLAM [59, 1, 31, 16] should collect cropped region of interest (RoI) by using additional multi-object detector, and then input each image back into their models for single object. Therefore, relations between objects, and between objects and single scene are out of their concern. As object detector and observation model are separated from each other, objectness from multi-object detector is also hardly considered jointly with their work.

To this end, we propose a method for scene-level 3D multi-object understanding from single shot, and SLAM framework. We estimate the joint distribution of multi-object by using variational auto-encoder (VAE), considering the relations between objects and single scene. The complex joint probability of multi-object can be captured in factorized form by leveraging the latent-space. Latent variables are used instead of the object-oriented features for SLAM in Bayesian manner, thus frame-level understanding is easily reflected to SLAM formulations. Since our model possesses latent variables for objectness, object uncertainty for each observation is also considered in probabilistic data association. To achieve fast data association is crucial for real-time SLAM optimization, so we device a generative model that can reduce the dimensionality of latent variables. Overview of our method is shown in Fig. 1.

Our contributions are two-fold: First, we mathematically show that the multi-object observation model considering relations between objects and single scene can be captured, by exploiting the existing multi-object detector structure. Second, we introduce the probabilistic SLAM with our model, so that frame-level understanding and the object uncertainty are seamlessly reflected to the data association.

2 Related work

With the recent advent of neural networks, a number of single object classification and detection methods with high performance have been proposed [19, 14, 40]. Beyond obtaining one feature vector from one image for an object, several multi-object detection techniques from single shot have been developed by introducing new network structures [38, 36, 37, 24, 22]. In particular, some of these methods can be applied to various real-time tasks since the whole detection network is composed of single network pipeline.

Various studies have also been conducted to understand the instance-level representation from 2D images such as object shape, orientation or bounding box. [44, 27, 47] and [21] estimate the orientation of the object by viewpoint classification with discretized bins. In addition, 3D bounding box regression has been carried out to obtain the object location and orientation [42, 46, 32, 28]. In order to estimate the distinct 3D shape of objects, [54] aligns the prior shape to a single object image through key point matching and estimates its 3D shape and orientation together. [33] estimates the 3D mesh with linear combination of parameterized prior shapes. In [52, 11, 57, 51], they have actively utilized non-linear regression and latent variables of neural networks for 3D reconstruction from 2D.

Through multi-object detection and instance-level understanding altogether, learning the disentangled representation of multi-object becomes achievable. [46] exploits the yolov2 structure [37] to estimate the 3D bounding box and center of the multi-object and obtains the orientations. In [21], they estimate the 3D shape rendering and orientation under faster R-CNN structure [38]. They obtain the shape rendering via weighted sum of the parameterized prior shape with PCL. Orientations are estimated by classifying the bins which indicate the discretized object pose. Similarly, in [31], they design the object observation factor to perform data association for pose SLAM. RoIs for multi-object are obtained by [37].

These studies are efficient because they mainly concern direct and accurate estimation of the object characteristics through network modeling; on the other hand, the probabilistic observation models are relatively less considered. Although they exploit the neural network for nonlinear regression, approximating the intractable distribution is rarely concerned. Therefore, Bayesian inference with obtained features are challenging; for example, data association for SLAM is considered only in front-end and additional algorithms are necessary to perform loop closing and place recognition [39, 31].

To handle the intractable target distribution, latent variables can be adopted [8, 45, 17, 12]. In order to understand and utilize the latent space, [20, 34] have studied the relations between latent variables and object visualization by using VAE [17]. However, it is still challenging to apply the proposed method to probabilistic model approximation, as it mainly concentrates on the interpretable graphic codes. To approximate the observation probability, entropy and variational likelihood is exploited in the field of the active vision [6, 30, 3]. Using VAE, [58, 59] have proposed methods to approximate the observation model of 3D objects for Bayesian inference. Based on the ELBO which approximates the observation model, they have shown that how the probabilistic SLAM with data association can be performed with expectation-maximization (EM) algorithm.

However, the methods above only concern the observation model for single object, thus it is inevitable to use multi-object detector for single-object images in the scene. They hardly estimate the object observation model based on the entire scene. The relations between the objects and the scene are also merely considered. It is also challenging to include the object uncertainty in their model, since the objectness is determined by the multi-object detector. Therefore, we introduce the generative story of scene-level multi-object understanding for Bayesian inference, which is, to the best of our knowledge, the first of its kind.

3 Multi-object Observation Model

3.1 Evidence Lower Bound and Encoder

Suppose we select $K$ regions in a single scene and observe the full 3D shape of the arbitrary structure in each region. The $k$’th area of the scene can be defined with RoIs [38], grids of fixed size [36, 37], or grids of various sizes [24, 22]. The typical multi-object detection methods [38, 37, 24, 22] mainly focus on the real-time detection and category inference. For the generative story of object, however, any type of disentangled representations can be involved such as 3D shape or pose. Let $\boldsymbol{s}^{f}_{k}\in\mathcal{S}^{f}$ be the $k$th observed full 3D shape. Similar to [7, 59], we assume that the $\boldsymbol{l}$abel $\boldsymbol{l}_{k}\in\mathcal{L}$ and the $\boldsymbol{v}$iewpoint orientation $\boldsymbol{v}_{k}\in\mathcal{V}$ cast the Bayesian dice to generate the 3D shape as shown in Fig. 2(a): $\boldsymbol{l}$ is the class or instance label of the 3D shape, and $v_{i}\in\boldsymbol{v}$ denotes the orientation $(SO3)$ of the shape related to the observer which can be represented in Euler angles. To address the object location in the scene, $\boldsymbol{b}$ounding box $\boldsymbol{b}_{k}\in\mathcal{B}$ is also included in our story.

When observing single scene $I$mage $I$, objects are jointly related to each other since the objects and their locations determine $I$. To catch the intractable observation model, the joint probability $p\left(\mathcal{S}^{f},\mathcal{L},\mathcal{V},\mathcal{B}\right)$ can be addressed. Since our main concern is the object-oriented features, we solely focus on the 3D shape of the object excluding background in the scene. For object and background discrimination, a latent variable $o_{k}\in\mathcal{O}$ for objectness can be added to the Bayesian graph model as shown in Fig. 2(a). The joint probability then can be fractionated as follows:

[TABLE]

We let $\log p\left(\mathcal{V}\right)=c_{0}$ be a constant; that is, 3D shape can be observed in any arbitrary viewpoints.

To learn the complex probability distribution (1) using VAE, we first need to find the lower bound. The joint probability $(a)$ in (1) can be denoted as $p\left(\mathcal{L},\mathcal{O}\right)=p\left(\mathcal{L}\right)p\left(\mathcal{O}|\mathcal{L}\right)$. Here, we assume that the prior $\log p\left(\mathcal{L}\right)=c_{1}$ is a uniform distribution, that is, any kind of structure can be detected in arbitrary location of a scene. The lower bound of $(a)$ then can be represented as:

[TABLE]

In our work, we assume that the entire scene $I$ is generated by objects and their locations; therefore in (2) and the following, we let variational likelihood $q$ be estimated from $I$ in order to consider the correlations between objects and the entire scene.

Similarly, we can have ELBO of $(b)$ in (1) as the following:

[TABLE]

The graph model including latent variable $\mathcal{Z}=\mathcal{Z}^{l},\mathcal{Z}^{v},\mathcal{Z}^{b}$ in (3) is shown in Fig. 2(b). With (2) and (3), the lower bound of (1) can be achieved; however, the formulation is composed of the joint probability of latent variables for entire objects, which is still intractable.

To relax the problem, we can have the lower bound in factorized form for each object by adopting the mean field inference [17]. We assume that all elements of latent variables $\mathcal{O}$ and $\mathcal{Z}$ are independent to each other. That is, the lower bound can be factorized with $o_{k},\boldsymbol{z}_{k}\in\mathcal{O},\mathcal{Z}$ which are for $k$’th region in $I$. The lower bound $L$ of (1) then can be represented as:

[TABLE]

where $\boldsymbol{z}^{lv}=\boldsymbol{z}^{l},\boldsymbol{z}^{v}$. $\mathcal{L}^{o}$ and $\mathcal{L}^{\neg o}$ are the object and non-object (background) label set respectively. In (4), the first row can be viewed as the lower bound for objectness, second for 3D shape reconstruction and third for bounding box regression. The KL and expectation terms can be learned using encoding and decoding parts of VAE, respectively.

In this manner, joint observation model for multi-object can be captured as factorized form in latent space. Each $\boldsymbol{z}_{k}$ is for each observation but estimated from the entire scene $I$, and relations between objects are naturally taken into account. To implement the network for each observation, $K$ encoders are required. However, since the variational likelihoods of each observation share the parameter $\phi$ and are estimated from $I$ in common, encoders can be combined in single encoder which estimates all likelihoods simultaneously. In our work, we exploit YOLOv2-like structure as an encoder, which enables the real-time performance and end-to-end learning scheme. The graphical model with variational likelihood estimation is depicted in Fig. 2(c).

3.2 Low-Dimensional Latent Variables and Decoder

In order to learn the posterior $p_{\theta}\left(\boldsymbol{s}^{f}|\boldsymbol{z}^{lv}\right)$ for 3D shape reconstruction in (4), we can use 3D decoder which outputs the rotated 3D shape according to the observed viewpoint. When the algorithmic prior operation such as rotation transform exists, however, separating such arithmetically trivial operation from the non-linear regression technique can relieve the whole network, and enable the efficient learning [2, 18, 15, 35]. For the shape reconstruction term of (4), we can let $\boldsymbol{s}^{f}=f_{R}\left(\boldsymbol{s}^{fo},\boldsymbol{R}\right)$, where $f_{R}$ is a function that rotates the basic orientation shape $\boldsymbol{s}^{fo}$ with rotation matrix $\boldsymbol{R}$ arithmetically. Then we have:

[TABLE]

where $\boldsymbol{R}^{f}$ is the rotation matrix according to the shape $\boldsymbol{s}^{f}$. $\boldsymbol{R}^{z}$ is the rotation matrix computed from $\boldsymbol{z}^{v}$, and thus we let $\boldsymbol{z}^{v}$ be the trigonometric value of Euler angles as in [59]. Since we choose the binary voxelized grid to represent the 3D shape, $(a)$ in (5) is assumed to be binary distribution. For orientation, we let $(b)$ follow a Von Mises–Fisher distribution [26, 13].

With (5), the second row of (4) can be expressed as the following:

[TABLE]

For the tractable prior distribution of $\boldsymbol{z}^{l}$, we assume $p_{\psi}\left(\boldsymbol{z}^{l}|\boldsymbol{l}\right)=\mathcal{N}\left(\boldsymbol{z}^{l};\boldsymbol{\mu}\left(\boldsymbol{l}\right),\boldsymbol{I}\right)$. In other words, we let $p_{\psi}\left(\boldsymbol{z}^{l}\right)=\sum_{\boldsymbol{l}}p\left(\boldsymbol{z}|\boldsymbol{l}\right)p\left(\boldsymbol{l}\right)$ be a gaussian mixture model (GMM) as in [48, 59]. $\boldsymbol{\mu}\left(\boldsymbol{l}\right)=f_{\psi}\left(\boldsymbol{l}\right)$ is obtained from prior network with parameter $\psi$, which is trained with VAE simultaneously. The variational likelihoods except for $q_{\phi}\left(o|I\right)$ are assumed to be isotropic Gaussians. For more details of each distribution, see Appendix I.

Our decoder estimates only the basic shape without considering the orientation relative to the observer. To complete observed shape inference, rotation transform on $\boldsymbol{R}^{f}$ should be performed subsequently. In this way, we can relieve the burden of our network and reduce the parameters of it; we empirically found that the dimension of the latent variable $\boldsymbol{z}^{l}$ can be decreased from $128$ to $16$, which is crucial for SLAM performance. As described in the next section, latent variables can replace the object-oriented features, and the metric operation between each feature is inevitable for data association. Reduced dimensional latent variables thus make the SLAM optimization process up to 5 times faster. Additionally, with the low-dimensional Gaussian latent variables, the bubble effect is relaxed [4] and $p_{\psi}\left(\boldsymbol{z}^{l}|\boldsymbol{l}\right)$ becomes distinct according to the label $\boldsymbol{l}$. Hence, robust data association is achieved.

4 Latent Variables and Probabilistic SLAM

Consider the localization and mapping problem with object-oriented features. Let $\mathcal{X}=\{\mathscr{x}_{t}=\left(\boldsymbol{x}^{\mathscr{x}}_{t},\boldsymbol{v}^{\mathscr{x}}_{t}\right)\}^{T}_{t=1}$ be the pose of an observer. Assume we have a collection $\mathscr{L}=\{\ell_{m}=\left(\boldsymbol{l}_{m},\boldsymbol{x}^{\ell}_{m},\boldsymbol{v}^{\ell,global}_{m}\right)\}_{m=1}^{M}$ of $M$ landmarks.

Now suppose that the observer navigates around the area and obtain a set of observation $\mathcal{S}=\{\mathcal{S}^{T}_{t=1}\}$ for $T$ keyframes. Here, $k$’th observation $\boldsymbol{s}_{k}=\left(\boldsymbol{s}^{f}_{k},\boldsymbol{x}^{\boldsymbol{s}}_{k}\right)\in\mathcal{S}_{t}$ can be about either an object or background. In the previous works for object-oriented SLAM [1, 31, 59, 16], they lean to the existing multi-object detection method to get rid of the non-object detection; frame-level joint probability and objectness are hardly considered when formulating SLAM optimization. In our method, since objectness joins our single scene understanding and thus naturally affects the data associations, all observations obtained from the regions of $I$ can be seamlessly used for SLAM.

Adopting the lower bound (4) as approximated observation model, the optimal $\mathcal{X}$ and $\mathscr{L}$ for the probabilistic SLAM is obtained from the maximization step of EM formulation:

[TABLE]

where $\mathcal{X}^{\ell}=\{\boldsymbol{x}^{\ell}_{m},\boldsymbol{v}^{\ell,global}_{m}\}_{m=1}^{M}$ and $\mathcal{L}^{M}=\{\boldsymbol{l}_{m}\}_{m=1}^{M}$.

Before calculating (7), in expectation step, the similarity weight $w^{t}$ is calculated in consideration of the objectness $\mathcal{O}$ of observations. Note that the object-oriented feature is replaced with encoded variables $\boldsymbol{v}_{k}$ and $\boldsymbol{\mu}^{\boldsymbol{s}l}_{k}$ for each $k$’th grid, even though we start with joint probability of $t$’th keyframe. Also, for the likelihood of object-oriented feature, tractable latent priors $p\left(\boldsymbol{z}|\boldsymbol{v}\right)$ and $p_{\psi}$ are used which are isotropic Gaussians. Therefore, with simple derivations, optimal solutions can be achieved even if the inaccurate observations are made, as the objectness of the multiple observation is concurrently considered. Details of the EM formulation can be found in Appendix II.

5 Implementation

To implement the proposed observation network, in this paper we use darknet19 structure [37] for the encoder core (or backborn). We construct the encoder by adding 3 convolutional layers with 1024 filters followed by one convolutional layer on top of the core network. A predictor (encoder) predicts $q_{\phi}$ for each grid. In other words, predictor infers $q_{\phi}\left(o|I\right)$ for objectness, $q_{\phi}\left(\boldsymbol{z}^{l}|I\right)$ for latent variables implying full shape, $q_{\phi}\left(\boldsymbol{z}^{v}|I\right)$ for viewpoint inference, and $q_{\phi}\left(\boldsymbol{z}^{b}|I\right)$ for bounding box. The decoder follows the generator structure of [52], except the input dimension; which in our case is set to 16. A prior network consists of dense layers to represent GMM for prior distribution $p_{\psi}$. As in [59], prior network is trained with VAE simultaneously.

Similar to [37], when training we consider one predictor which predicts the highest IOU with the ground truth bounding box as responsible predictor for an object. After selecting the predictors observing the object in the grids, shape estimation is performed by inputting the latent variables obtained from that predictors to the decoder. Therefore, during both training and testing, the input batch size varies between the encoder and decoder; when single scene enters the encoder, the decoder gets as many latent variables as the number of objects in the scene. The proposed network structure is displayed in Fig. 3.

6 Training details

The proposed network estimates the various representations of multi-object in single scene, with probabilistic distributions. The negative lower bound from (4), which is composed of KL divergence and expectation terms of various distributions, is used as the training loss. The network thus easily diverged without sophisticated training strategy. For the stable optimization procedure, we replace the objectness term in (4) with $KL\left(p\left(o|\boldsymbol{l}\right)||q_{\phi}\left(o|I\right)\right)$ in actual training. Then objectness loss becomes equal to the conventional binary cross-entropy loss.

We also found that the two-stage pretraining make the main training stabilized: pre-pretraining of 2D-3D understanding, and pretraining of NOLBO for single object. We first pre-pretrain the encoder core on the ImageNet dataset [5] for object classification and Render for CNN dataset [44] for viewpoint classification sequentially. In the case of the decoder, it can fall into the local minimum when learning a small number of object instances, since the decoder only infers the basic 3D shape without considering the rotation transform. To alleviate this limitation, the decoder is also pre-pretrained on ModelNet40 dataset [53] which contains 40 classes, and about 300 instances per each class. We construct 3D VAE with decoder, 3D encoder in [52] and prior network for this pre-pretraining.

The NOLBO approximates the observation model by learning 3D reconstruction and pose estimation for multi-objects in 2D scenes. Therefore, we use Pascal3D+ [56] and objectnet3D [55] training datasets comprising 2D-3D aligned annotations considering 3D shape pose. Object orientation $\boldsymbol{v}$ is expressed as azimuth, elevation and in-plane rotation angles. Since these datasets contain 100 classes in total, we manually select 40 classes. Prior to the NOLBO multi-object training, we pretrain NOLBO for single-object [58, 59] on these datasets with pre-pretrained encoder core, decoder and a fresh prior network. As [37], networks are trained on multi-scale images. Since the networks slowly converge when trained on multi-resolution images from the beginning, we first fix the image resolution to $224\times 224$, and change to $448\times 448$ when the accuracy of 3D reconstruction reaches about 70% mAP. Onece the accuracy reaches 70% again, multi-scale training is started.

The networks for NOLBO multi-object are constructed based on encoder core, decoder and prior network of the pretrained NOLBO single-object. Training starts with the multi-resolution images. We use adam optimizer with a starting learning rate of $10^{-5}$ for the first epoch, and increase to $10^{-4}$. For the 20 epochs, we freeze decoder and prior network as they already learn the shape distribution. This allows the network to learn how to infer the 3D shape distribution from 2D scene without diverging. Similar to [37, 59], gaussian blur, HSV saturation, RGB invertion and random brightness are applied to 2D scene data augmentation. Random translation and scaling are also used.

All of our code and pre-trained models are available at https://github.com/bogus2000/NOLBO.

7 Experiments

We evaluate the proposed method in various aspects; disentangled representations and SLAM application. The main purpose of NOLBO is to approximate the object observation model. When the model is applied to the Bayesian inference, latent variables and their prior distributions are used as object-oriented features and their observation models. Therefore, it is important how objects are located or projected on the latent space as latent variables. For comparison of latent spaces, we construct a 2D-3D auto-encoder (AE), vanilla VAE (vVAE) [17] and TL-Network (TLNet) [11] by using our pretrained encoder and decoder structure. We also compare the SLAM results using object-oriented features from above networks.

7.1 Object Pose Estimation

Since NOLBO approximates the observation model, it is possible to perform category classification and viewpoint estimation using MLE (see APPENDIX II). The latent variables for orientation becomes the orientation itself, so viewpoint orientation can directly be inferred from the encoder. To report the qualitative results of our pose estimation method, we train NOLBO with 2D images of fixed size. The comparison of the viewpoint estimation is shown in Table 1. Our method shows competitive results relative to previous works.

7.2 Object Observation and Latent Space

For the data association or MLE, NOLBO have advantages that latent variables follow distinct prior distributions for each class or instance. In order to display how objects are projected into the latent space according to their respective class, we estimate the latent variables from NOLBO and other methods. We use the outputs from 2D encoders of other non-probabilistic methods as latent variables. The dimensions of the latent variables are set to 16, same as NOLBO. We display the latent variables obtained from each network in Fig. 4. We use t-sne method [25] for the latent space dimension reduction in order to plot in 2D. For clarity, we randomly choose 10 classes and show the results. Since NOLBO enforce the latent variables to follow the prior distributions which are learned together, the latent variables are projected to particular distributions according to their categories unlike other methods.

7.3 Object 3D Shape Reconstruction

For comparison of the 3D reconstruction results, we additionally train 3D VAE-GAN. Since this method trains one 3D-GAN for each object category, it is hard to apply the algorithm to 40 categories. Therefore, we manually choose the classes for both training and testing, which have similar shapes to others: bench, chair and sofa. Rest of the methods are evaluated on 40 classes. The precision-recall curve of 3D shape reconstruction is depicted in Fig. 5.

To verify the reconstruction results in various environments, we evaluate NOLBO on MS-COCO [23], TUM [43] and KITTI dataset [10]. Some of the 3D reconstructions of multi-object are displayed in Fig. 6. To clearly show the reconstructed 3D shapes, objects are aligned in arbitrary viewpoint. For more quantitative results of multi-object 3D reconstruction, see our supplementary.

7.4 Multi-object Observation and SLAM

To verify the probabilistic SLAM using our observation model, we choose KITTI dataset for autonomous driving. Among 10 sequences of KITTI, 00, 05, 06 and 07 are used which have various loop closing spots. To evaluate the aspect of data association, we compare our SLAM results with that of using latent variables from other object understanding networks. Results using 128dim-version NOLBO is also compared. Average precisions of the trajectory estimation are represented in Table.2. We also display some of the results in Fig. 7. For SLAM, any visual odometry can be used for initial trajectory estimation. However, to compare data association results more clearly, we use low-performance visual odometry based on simple visual feature matching procedure with known scale as shown in Fig. 7(b). Results using our model with objectness estimated from the entire single scene show better loop closing performance than that of using the single-object model. Even if our method is trained on indoor datasets, it can achieve the robust object observation in the wild and can consider the various loop closing points. As shown in Table.2, with high dimensional (128) latent variables, data association occasionally fails due to the bubble effect. For more results, refer to see our supplementary.

8 Conclusion

We have proposed an observation model approximation for 3D multi-object in single 2D scene. Since 3D objects in a shot are related to each other and follow a complex joint distribution, object-oriented probabilistic SLAM poses a challenge. Therefore, we approximate the joint distribution of the multi-object present in a single 2D scene using VAE. The jointly correlated multi-object observation is represented with factorized tractable distributions in the latent space. Each observation can be estimated in consideration of the entire scene, by exploiting the existing multi-object detector structure. Since our generative story involves the objectness, observation uncertainty naturally affects the probabilistic data association for SLAM. The network is essentially an auto-encoder so that it can be used to estimate the 3D shape and pose of multi-object in a single shot.

Acknowledgement

This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIP) (No. 2017R1A2B2002608), in part by Automation and Systems Research Institute (ASRI), and in part by the Brain Korea 21 Plus Project.

9 Appendix

In this section, we present detailed formulations for probabilistic simultaneous localization and mapping (SLAM). From Appendix I to III, we specify the probability distributions for our observation model, and introduce formulations for several Bayesian inference methods: in Appendix I, we introduce the distributions for latent variables in detail. In Appendix II, we show how our approximated observation model can be adopted for probabilistic semantic SLAM, which can be performed using expectation-maximization (EM) method. Examples of the maximum likelihood estimation (MLE) with our model is presented in Appendix III.

Appendix I : Lower Bound and Probability Distributions for Observation Model

Combining Eqn. (4) and (6) of the paper, the lower bound $L$ of $p\left(\mathcal{S}^{f},\mathcal{L},\mathcal{V}\right)$ can be represented as the following:

[TABLE]

The probability $p\left(o|\boldsymbol{l}\right)$ representing the objectness according to the label is defined as:

[TABLE]

where $\epsilon,\epsilon^{\prime}\ll 1$. For convenience, we let $\epsilon=\epsilon^{\prime}$. Here, $q_{\phi}\left(o|I\right)$ is assumed to be the Bernoulli distribution of objectness.

For the tractable prior distribution of $\boldsymbol{z}^{l}$, we assume $p_{\psi}\left(\boldsymbol{z}^{l}|\boldsymbol{l}\right)=\mathcal{N}\left(\boldsymbol{z}^{l};\boldsymbol{\mu}\left(\boldsymbol{l}\right),\boldsymbol{I}\right)$. In other words, we let $p_{\psi}\left(\boldsymbol{z}^{l}\right)=\sum_{\boldsymbol{l}}p\left(\boldsymbol{z}^{l}|\boldsymbol{l}\right)p\left(\boldsymbol{l}\right)$ be a gaussian mixture model (GMM). $\boldsymbol{\mu}\left(\boldsymbol{l}\right)=f_{\psi}\left(\boldsymbol{l}\right)$ is obtained from prior network with parameter $\psi$, which is trained with VAE simultaneously. To simplify the network structure, the variance is assumed to be $\boldsymbol{I}$. Similarly, the variational likelihood $q_{\phi}$ is assumed to be a multivariate Gaussian; $q_{\phi}\left(\boldsymbol{z}^{l}|I\right)=\mathcal{N}(\boldsymbol{z}^{l};\boldsymbol{\mu}^{\boldsymbol{s}l},\left(\boldsymbol{\sigma}^{\boldsymbol{s}l}\right)^{2})$.

To estimate the orientation of objects, classifying the discretized angles can be trained easily as compared to the continuous regression [27, 21]. For the probabilistic modeling and observation uncertainty, however, it is much useful to assume a noise model rather than a multinoulli distribution. Since $\boldsymbol{v}$ represented as radian is natural to follow a Gaussian distribution, we can let the latent variable be the angle directly by assuming $p\left(\boldsymbol{z}^{v}|\boldsymbol{v}\right)=\mathcal{N}\left(\boldsymbol{z}^{v};\boldsymbol{v},{\sigma}^{2}\boldsymbol{I}\right)$. However, as the direct estimation of the angle is challenging to the network [7], we let the trigonometric function values of $\boldsymbol{v}$ be the latent variables rather than $\boldsymbol{v}$ itself. In other words, we let $p\left(\boldsymbol{z}^{v}|\boldsymbol{v}\right)=\mathcal{N}\left(\boldsymbol{z}^{v};\left(\boldsymbol{\mu}^{s},\boldsymbol{\mu}^{c}\right),\left(\boldsymbol{\sigma}^{s},\boldsymbol{\sigma}^{c}\right)^{2}\boldsymbol{I}\right)$ be the prior of the latent variable related to the orientation [26, 13]. Assuming $\sin v$ and $\cos v$ are i.i.d., $\boldsymbol{\mu}^{s}$, $\boldsymbol{\mu}^{c}$, $\boldsymbol{\sigma}^{s}$ and $\boldsymbol{\sigma}^{c}$ can be represented as follows [26]:

[TABLE]

where $v_{i}\in\boldsymbol{v}$. We set $\sigma$ for $p\left(\boldsymbol{z}|\boldsymbol{v}\right)$ to $0.05\text{}rad\simeq 2.87^{\circ}$ [59], because when the lower precision is used, learning easily diverges. Similar to the prior, a variational likelihood for the orientation is assumed to be a wrapped normal distribution with trigonometric functions: $q_{\phi}\left(\boldsymbol{z}^{v}|\boldsymbol{s}^{f}\right)=\mathcal{N}\left(\boldsymbol{z}^{v};\left(\boldsymbol{\mu}^{ss},\boldsymbol{\mu}^{sc}\right),\left(\boldsymbol{\sigma}^{ss},\boldsymbol{\sigma}^{sc}\right)^{2}\boldsymbol{I}\right)$. Therefore, for $q_{\phi}\left(\boldsymbol{z}^{v}|\boldsymbol{s}^{f}\right)$ we first infer $\sin\boldsymbol{v}^{s},\cos\boldsymbol{v}^{s}$ and $\boldsymbol{\sigma}^{s}$ from the network, then calculate mean and variance similar to (10). In practice, we give an additional constraint $\sin^{2}v+\cos^{2}v=1$ in order to consider that trigonometric functions are not independent. Since the inferred latent variables are the trigonometric values of the pose, rotation matrix $\boldsymbol{R}^{z}$ related to $\boldsymbol{z}^{v}\sim q_{\phi}$ can directly be computed.

Appendix II : Object-oriented Probabilistic Semantic SLAM with Approximated Observation Model

A. EM Formulation of Probabilistic Semantic SLAM

Classical SLAM methods usually divide the problem into two parts: data associations in front-end, and pose optimization in back-end [g2o, 39, orbSLAM, 31]. With these approaches, the huge error is inevitable when the false data association occurs in the front-end, since incorrectly determined data associations are hardly modified and thus bring a highly detrimental effect on pose estimation in the back-end.

To avoid this limitation, the complete SLAM formulation with probabilistic data association can be achieved using Expectation-Maximization (EM) method [1, 58, 59]; both pose and data association can be optimized simultaneously. Consider the localization and mapping problem with object-oriented features. Our observer (camera or drone) collects a set of single shots $\{I_{t}\}^{T}_{t=1}$ as keyframes. Let $\mathcal{X}=\{\mathscr{x}_{t}=\left(\boldsymbol{x}^{\mathscr{x}}_{t},\boldsymbol{v}^{\mathscr{x}}_{t}\right)\}^{T}_{t=1}$ be the pose of an observer that navigates the unknown area; $\boldsymbol{x}^{\mathscr{x}}_{t}$ and $\boldsymbol{v}^{\mathscr{x}}_{t}$ represent 3D position and orientation of the observer, respectively. Assume we have a collection $\mathscr{L}=\{\ell_{m}=\left(\boldsymbol{l}_{m},\boldsymbol{x}^{\ell}_{m},\boldsymbol{v}^{\ell,global}_{m}\right)\}_{m=1}^{M}$ of $M$ landmarks; $\boldsymbol{l}$ stands for the category or instance-level label, and $\boldsymbol{x}^{\ell}_{m}$ and $\boldsymbol{v}^{\ell,global}_{m}$ denote 3D position and orientation of the landmark in the global coordinate, respectively. Now suppose that the observer navigates around the area and obtain a set of 3D object observation $\mathcal{S}=\{\mathcal{S}^{T}_{t=1}\}$. Here, $k$’th observation $\boldsymbol{s}_{k}=\left(\boldsymbol{s}^{f}_{k},\boldsymbol{x}^{\boldsymbol{s}}_{k}\right)\in\mathcal{S}_{t}$ consists of the 3D position $\boldsymbol{x}^{\boldsymbol{s}}$ and the full shape $\boldsymbol{s}^{f}$ considering the object’s orientation related to the observer. The EM formulation for the probabilistic SLAM is expressed as follows [1, 58]:

[TABLE]

$\mathbb{D}_{t}$ is the set of all possible data associations $\mathcal{D}_{t}=\{\left(\alpha_{k},\beta_{k}\right)\}^{K}_{k=1}$ representing that the object detection $\boldsymbol{s}_{k}$ of landmark $\ell_{\beta_{k}}$ was obtained from the observer state $\mathscr{x}_{\alpha_{k}}$. Also, $\mathbb{D}_{t}\left(i,j\right)\subseteq\mathbb{D}_{t}$ is the set of all possible data association $\mathcal{D}_{t}^{\prime}=\{\left(\alpha_{k}^{\prime},\beta_{k}^{\prime}\right)\}$ such that $i$th detection is assigned to $j$th landmark.

Now suppose the data associations for $t$’th keyframe are fixed, then we have:

[TABLE]

where $\mathcal{X}_{\alpha}=\{\mathscr{x}_{\alpha_{k}}\}_{k=1}^{K}$ and $\mathscr{L}_{\beta}=\{\ell_{\beta_{k}}\}_{k=1}^{K}$. Since we assume that $\mathcal{S}^{f}=\{\boldsymbol{s}^{f}_{k}\}_{k=1}^{K}$ is generated by $\mathcal{L}=\{\boldsymbol{l}_{k}\}_{k=1}^{K}$ and $\mathcal{V}=\{\boldsymbol{v}_{k}\}_{k=1}^{K}$, observation probability (13) can be split as follows:

[TABLE]

where $\boldsymbol{X}^{\boldsymbol{s}}_{t}=\{\boldsymbol{x}^{s}_{k}\}_{k=1}^{K}$, $\boldsymbol{X}^{\mathscr{x}}_{\alpha}=\{\boldsymbol{x}^{\mathscr{x}}_{\alpha_{k}}\}_{k=1}^{K}$ and $\boldsymbol{X}^{\ell}_{\beta}=\{\boldsymbol{x}^{\ell}_{\beta_{k}}\}_{k=1}^{K}$. For the first factorized term in (14), we can say that the set of feature’s 3D position $\boldsymbol{X}^{\boldsymbol{s}}_{t}$ is determined by the set of observer’s position $\boldsymbol{X}^{\mathscr{x}}_{\alpha}$ and the observed object’s position $\boldsymbol{X}^{\ell}_{\beta}$. The term $p\left(\mathcal{S}^{f}_{t}|\mathcal{L}_{\beta},\mathcal{V}^{\ell,global}_{\beta},\mathcal{V}^{\mathscr{x}}_{\alpha}\right)$ in (14) denotes that the observer with states $\mathcal{V}^{\mathscr{x}}_{\alpha}$ observes 3D shapes of objects $\boldsymbol{S}^{f}_{t}$ with label $\mathcal{L}_{\beta}$ and pose $\mathcal{V}^{\ell,global}_{\beta}$. It is equivalent that the observer observes the object $\mathcal{L}_{\beta}$ placed according to the local orientation $\mathcal{V}^{\ell,local}_{\alpha,\beta}$, which is related to the observer; relation between $\boldsymbol{v}^{local}_{\alpha_{k},\beta_{k}}\in\mathcal{V}^{\ell,local}_{\alpha,\beta}$ and $\boldsymbol{v}^{global}_{\beta_{k}}\in\mathcal{V}^{\ell,global}_{\beta}$ can be represented as:

[TABLE]

where $\boldsymbol{R}^{\boldsymbol{v}}$ denotes the rotation transform matrix according to the pose $\boldsymbol{v}$. Therefore, without loss of generality, we can rewrite (14) as the following:

[TABLE]

Substituting (16) to (11) and (12) finally yields:

[TABLE]

B. Variational Observation Model Approximation and EM for Probabilistic SLAM

Expectation Step

The evidence lower bound (ELBO) nearly reaches the target distribution when variational auto-encoder (VAE) is correctly converged. Therefore, we can let the lower bound $L$ in (8) approximately represent the joint distribution for the 3D shape observation probability $p\left(\mathcal{S}^{f}|\mathcal{L},\mathcal{V}\right)$ in (16) as follows:

[TABLE]

Take exponential to both sides, we have:

[TABLE]

In (20), $a$ is a constant, and

[TABLE]

Note that although the original joint probability for single scene is intractable and extremely challenging to express in factorized form, we can have such factorized form as (20) that follow tractable Gaussians by leveraging the latent space.

For $\kappa_{KL}^{o}\left(\boldsymbol{l}_{k};I\right)$, we have:

[TABLE]

Since our main concern is object-oriented SLAM, it is necessary to focus on the object observations. To achieve this, we only calculate (21) for the prior $p\left(o|\boldsymbol{l}\right)$ where $\boldsymbol{l}\in\mathcal{L}^{o}$. Then the cases (a) and (b) of (21) remain, which indicate that if the objectness $o$ of an observation is close to true, (21) goes to 1; else 0. In other words, (21) can be seen as the probability of objectness when calculated only on $p\left(o|\boldsymbol{l}\right)$ for $\boldsymbol{l}\in\mathcal{L}^{o}$. With this manner, all observations for $K$ regions can be used naturally by letting the probability of objectness affect to the weight calculation; we can reflect both ‘more object-like’ observations and ‘less-like’ one into the weight. There is no need to just abandon several observations which have the objectness under certain threshold, and give the equal weights to others.

Substituting (20) into (17), we have:

[TABLE]

We assume the 3D positions of $K$ observations are i.i.d. Since $\kappa_{E}\left(\boldsymbol{s}^{fo}\right)$ and $\kappa_{E}\left(\boldsymbol{R}^{f}\right)$ are independent to $\mathcal{D}\in\mathbb{D}$, we can reduce the fraction. Then (22) can be expressed as:

[TABLE]

Meanwhile, the KL-divergence term of $\kappa_{KL}^{z}\left(\boldsymbol{l}_{\beta_{k}};I_{t}\right)$ in (23) is expanded as:

[TABLE]

The prior of $\boldsymbol{z}^{l}$ is assumed to be a multivariate Gaussian: $p_{\psi}\left(\boldsymbol{z}^{l}|\boldsymbol{l}\right)=\mathcal{N}\left(\boldsymbol{z}^{l};\boldsymbol{\mu}\left(\boldsymbol{l}\right),\boldsymbol{I}\right)$. The variational likelihood is also represented as $q_{\phi}\left(\boldsymbol{z}^{l}_{k}|I\right)=\mathcal{N}(\boldsymbol{z};\boldsymbol{\mu}^{\boldsymbol{s}l}_{k},\left(\boldsymbol{\sigma}^{\boldsymbol{s}l}_{k}\right)^{2}\boldsymbol{I})$. Note that $\boldsymbol{\mu}^{\boldsymbol{s}l}_{k}$ and $\boldsymbol{\sigma}^{\boldsymbol{s}l}_{k}$ are the variables estimated from $I$, using the encoder. In other words, these variables are encoded features from the observed $I$. With the prior and the variational likelihood, we have:

[TABLE]

where $Z$ is the normalization constant. With (25), we can express $\kappa_{KL}^{z}\left(\boldsymbol{l}_{\beta_{k}};I_{t}\right)$ as:

[TABLE]

Meanwhile, we have assumed that $\boldsymbol{z}$ for angles follows Gaussian $p\left(\boldsymbol{z};\boldsymbol{v},\sigma^{2}\boldsymbol{I}\right)$ and the network is trained with trigonometric values of $\boldsymbol{z}$. The encoder thus infers the trigonometric values of $\boldsymbol{z}$ and precision: $\sin\boldsymbol{v}^{\boldsymbol{s}}$, $\cos\boldsymbol{v}^{\boldsymbol{s}}$ and $\boldsymbol{\sigma}^{\boldsymbol{s}}$. Using these values, we can calculate $\boldsymbol{v}^{\boldsymbol{s}}$. Therefore, similar to (26), $\kappa_{KL}^{z}\left(\boldsymbol{v}^{\ell,local}_{\beta_{k}};I_{t}\right)$ in (23) can be represented as follows:

[TABLE]

Since the exponential terms in (26) and (27) are independent not to the data association but to $I$, substituting (26) and (27) into (23) and reducing the fraction finally yields:

[TABLE]

Note that (28) can be seen as the result of replacing the original observation model $p\left(\mathcal{S}^{f}|\mathcal{L},\mathcal{V}^{\ell,local}\right)$ in (17) with $\kappa_{KL}^{o}\left(\boldsymbol{l};I\right)$ , $p_{\psi}\left(\boldsymbol{z}^{l}=\boldsymbol{\mu}^{\boldsymbol{s}l}_{k}|\boldsymbol{l}\right)$ and $p\left(\boldsymbol{z}^{v}=\boldsymbol{v}^{\boldsymbol{s}}_{k}|\boldsymbol{v}^{\ell,local}\right)$ , which are objectness and the priors of latent variables respectively. Therefore, when the objectness $o_{k}$, shape feature $\boldsymbol{\mu}^{\boldsymbol{s}l}_{k}$ and orientation feature $\boldsymbol{v}^{\boldsymbol{s}}_{k}$ are obtained from the encoder of VAE, we can directly use these features as observations of 3D shape to have approximated weights for EM.

Maximization Step

In the same manner as the expectation step, we can further exploit the approximated observation model in (20) for the maximization step. Note that $\kappa_{E}$ terms in (20) are independent to $\mathcal{X}$ and $\mathcal{L}$. Substituting (20) into (18), we have:

[TABLE]

With (26) and (27), we get:

[TABLE]

Since $\mathscr{L}=\{\ell_{m}=\left(\boldsymbol{l}_{m},\boldsymbol{x}^{\ell}_{m},\boldsymbol{v}^{\ell,global}_{m}\right)\}_{m=1}^{M}$ and $\boldsymbol{v}^{\ell,global}$ is represented with $\boldsymbol{v}^{\ell,local}$ and $\boldsymbol{v}^{\mathscr{x}}$, we can split (30) as:

[TABLE]

where $\mathcal{X}^{\ell}=\{\boldsymbol{x}^{\ell}_{m},\boldsymbol{v}^{\ell,global}_{m}\}_{m=1}^{M}$ and $\mathcal{L}^{M}=\{\boldsymbol{l}_{m}\}_{m=1}^{M}$. Similar to the expectation step, the maximization step can be calculated using the latent features $\boldsymbol{\mu}^{\boldsymbol{s}l}$ and $\boldsymbol{v}^{\boldsymbol{s}}$ encoded from the encoder. Therefore, instead of using the complex 3D shape $\boldsymbol{s}^{f}$ itself or the intractable single view scene $I$ directly, we can perform EM for probabilistic SLAM only with the encoded latent features and their priors. Since the conditional priors $p_{\psi}\left(\boldsymbol{z}^{l}|\boldsymbol{l}\right)$ and $p\left(\boldsymbol{z}^{v}|\boldsymbol{v}\right)$ are Gaussians, we can numerically calculate the optimal solutions of (31) and (32).

Pose and angle are the continuous values, and various optimization techniques can be applied to obtain solutions of (31). However, for (32), we have no choice but to substitute all the possible labels to obtain the optimal solutions, because we assume $\mathcal{L}$ to be a set of the finite number of labels. Meanwhile, since $p_{\psi}\left(\boldsymbol{z}^{l}|\boldsymbol{l}\right)=\mathcal{N}\left(\boldsymbol{z};\boldsymbol{\mu}^{l},\boldsymbol{I}\right)$ and $\boldsymbol{\mu}\left(\boldsymbol{l}\right)=f_{\psi}\left(\boldsymbol{l}\right)$, we can say that the label $\boldsymbol{l}$ is projected to $\boldsymbol{\mu}\left(\boldsymbol{l}\right)$ with nonlinear regression function $f_{\psi}$. In this point of view, we can redefine landmark as $\ell=\left(\boldsymbol{\mu}\left(\boldsymbol{l}\right),\boldsymbol{x}^{\ell},\boldsymbol{v}^{\ell,global}\right)$. Therefore, substituting $\boldsymbol{l}$ is the same as substituting $\boldsymbol{\mu}\left(\boldsymbol{l}\right)$ in the end, and we can exploit the continuous nonlinear regression of label $\boldsymbol{l}$ similar to [59]; we now concentrate on $\boldsymbol{\mu}\left(\boldsymbol{l}\right)$ rather than $\boldsymbol{l}$. For simplicity, let $\boldsymbol{\mu}\left(\boldsymbol{l}\right)=\boldsymbol{\mu}^{l}$. Then (32) can be expressed as:

[TABLE]

where $\mathcal{U}=\{\boldsymbol{\mu}_{j}^{l}\}$. As (33) is sum of logarithmic Gaussians, we can differentiate (33) with respect to $\boldsymbol{\mu}^{l}$ and obtain optimal solutions. The optimal $\boldsymbol{\mu}_{j}^{l}$ is then represented as the following:

[TABLE]

By (34), we can numerically calculate the optimal feature vector $\boldsymbol{\mu}^{l}$, rather than selecting the optimal label for the shape. It can be seen as updating shape features defined in continuous latent space, and shape estimation from multiple observation is achievable. Since the finite label set is no longer needed, assumption of the static landmarks is also unnecessary. Consequently, each observation can be regarded as an independent landmark, and we can calculate the similarities between them as (28) and then perform optimization with (31) and (34) subsequently.

As the EM formulations are finally modified as (31) and (34), latent variables of other methods can be adopted for SLAM such as vanilla 2D-3D auto-encoder (AE), vanilla VAE (vVAE) [17] and TLNet [11]. We can extract latent variables of these algorithms with region of interest (RoI) image obtained by any other multi-object detector, and perform SLAM optimizations.

Appendix III : MLE with Approximated Observation Model

The approximated observation model can be used for classifying the categories or instances, or estimating the poses of the observed objects by Maximum Likelihood Estimation (MLE). The classification problem for the label set $\mathcal{L}$ of multi-object in $I$ is given as:

[TABLE]

Using (20), we have:

[TABLE]

Note that only the term $\kappa_{KL}^{z}\left(\boldsymbol{l};I\right)$ is related to the label set $\mathcal{L}$. Substituting (36) into (35), we have:

[TABLE]

Putting (26) into (37) finally yields:

[TABLE]

Since $\mathcal{L}=\{\boldsymbol{l}_{k}\}$, we can simply have:

[TABLE]

Similar to the classification, the approximated solution of the object pose estimation problem can be achieved by using (20) and (27) as the following:

[TABLE]