Temporally Consistent Horizon Lines

TL;DR

This paper introduces a novel CNN architecture with an improved residual convolutional LSTM and an adaptive loss function for more accurate and temporally consistent horizon line estimation in videos, supported by an extended KITTI dataset.

Contribution

It presents a new CNN model with residual convolutional LSTM for video horizon line estimation and extends the KITTI dataset with precise horizon labels.

Findings

The proposed method outperforms existing approaches in accuracy.

Temporal consistency improves horizon line estimation stability.

The extended KITTI dataset provides valuable data for future research.

Abstract

The horizon line is an important geometric feature for many image processing and scene understanding tasks in computer vision. For instance, in navigation of autonomous vehicles or driver assistance, it can be used to improve 3D reconstruction as well as for semantic interpretation of dynamic environments. While both algorithms and datasets exist for single images, the problem of horizon line estimation from video sequences has not gained attention. In this paper, we show how convolutional neural networks are able to utilise the temporal consistency imposed by video sequences in order to increase the accuracy and reduce the variance of horizon line estimates. A novel CNN architecture with an improved residual convolutional LSTM is presented for temporally consistent horizon line estimation. We propose an adaptive loss function that ensures stable training as well as accurate results. Furthermore, we introduce an extension of the KITTI dataset which contains precise horizon line labels for 43699 images across 72 video sequences. A comprehensive evaluation shows that the proposed approach consistently achieves superior performance compared with existing methods.

Tables

Table 1. Table 1 : Horizon estimation results on the KITTI Horizon (Sec.  2 ) validation and test sets using the metrics described in Sec.  5.2 . AUC: higher is better; MSE and A T ​ V subscript 𝐴 𝑇 𝑉 A_{TV} : lower is better. Refer to Sec.  5.3 for a detailed discussion.

		AUC (horizon)		MSE $\times {10}^{-3}$		$A_{TV}\times {10}^{-3}$		AUC (pose)
		val	test	val	test	val	test	val	test
Lezama et al. [28]		34.17%	30.45%	>1000	>1000	2397	1537	28.79%	25.28%
Kluger et al. [24]		54.27%	48.21%	>1000	>1000	188.6	206.4	47.29%	41.47%
Simon et al. [37]		57.03%	47.84%	84.26	224.0	65.94	88.71	50.97%	41.98%
Zhai et al. [47]		60.97%	50.98%	>1000	>1000	91.56	1575	53.52%	43.47%
Workman et al. [43]		70.32%	66.48%	9.208	11.19	6.893	8.430	62.36%	58.58%
Average baseline		69.40%	64.18%	8.800	12.20	6.091	5.123	59.45%	54.98%
single frame	trained on HLW	71.10%	63.64%	10.41	14.31	13.90	15.71	64.02%	55.20%
(Sec. 3)	trained on KITTI-H	77.42%	74.08%	6.024	7.025	5.061	5.585	70.51%	66.62%
	w/ exp. smoothing	77.44%	74.11%	5.986	6.987	4.337	4.687	70.50%	66.64%
TCN [2] (3-3-5)		75.42%	71.80%	6.392	8.318	4.945	4.937	67.33%	64.21%
temporally consistent (Sec. 4)		78.09%	74.55%	5.427	6.731	4.619	4.984	71.17%	67.33%
	w/ exp. smoothing	78.11%	74.68%	5.405	6.712	4.159	4.404	71.19%	67.49%

Table 2. Table 2 : Ablation study (Sec.  5.4 ) results on the KITTI Horizon test set. MSE and A T ​ V subscript 𝐴 𝑇 𝑉 A_{TV} scaled by 10 − 3 superscript 10 3 10^{-3} .

	AUC	MSE	$A_{TV}$
Huber loss (Sec. 5.4.1)	71.96%	7.851	7.051
non-temporal (Sec. 5.4.2)	74.36%	7.266	5.699
w/o residual (Sec. 5.4.3)	64.29%	11.60	5.279
naïve residual (Sec. 5.4.3)	74.01%	7.009	4.967
Ours (Sec. 4)	74.55%	6.731	4.984

Table 3. Table 3 : Horizon estimation results on the KITTI Horizon validation and test sets comparing several TCN variants (Sec.  B ) with our single frame CNN and our proposed temporally consistent approach.

	AUC (horizon)		MSE $\times {10}^{-3}$		$A_{TV}\times {10}^{-3}$		AUC (pose)
configuration	val	test	val	test	val	test	val	test
1-3-3-5 ($S_{\mathrm{fov}}=9$)	75.42%	71.80%	6.392	8.318	4.946	4.937	67.33%	64.21%
1-3-5-5 ($S_{\mathrm{fov}}=11$)	75.82%	71.65%	6.498	8.329	4.997	5.119	68.43%	64.14%
1-3-5-7 ($S_{\mathrm{fov}}=13$)	75.83%	72.23%	6.383	7.909	4.932	5.043	68.42%	64.64%
3-3-5-7 ($S_{\mathrm{fov}}=15$)	76.08%	72.25%	6.453	8.185	4.956	5.023	68.66%	64.69%
3-5-5-7 ($S_{\mathrm{fov}}=17$)	75.49%	71.57%	7.084	8.458	5.117	5.263	68.52%	64.15%
5-5-5-7 ($S_{\mathrm{fov}}=19$)	75.76%	72.21%	6.573	8.002	5.075	4.980	68.42%	64.73%
single frame	77.42%	74.08%	6.024	7.025	5.061	5.585	70.51%	66.62%
Ours	78.09%	74.55%	5.427	6.731	4.619	4.984	71.17%	67.33%

Table 4. Table 4 : Horizon estimation results on the KITTI Horizon validation and test sets. We compare our proposed temporally consistent approach with two ablation studies and our single-frame CNN, see Sec.  C.1 . We present the results of the training run with the best validation AUC out of four runs. The numbers in brackets are (mean ± plus-or-minus \pm standard deviation) over all four runs.

	AUC (horizon)		MSE $\times {10}^{-3}$		$A_{TV}\times {10}^{-3}$
	val	test	val	test	val	test
single	77.42%	74.08%	6.024	7.025	5.061	5.585
frame	(77.03 $\pm$ 0.296)	(74.19 $\pm$ 0.219)	(6.125 $\pm$ 0.0635)	(7.084 $\pm$ 0.0886)	(5.276 $\pm$ 0.237)	(5.670 $\pm$ 0.0893)
non-	77.63%	74.36%	5.852	7.266	5.368	5.699
temporal	(77.19 $\pm$ 0.514)	(74.43 $\pm$ 0.332)	(5.926 $\pm$ 0.154)	(7.122 $\pm$ 0.228)	(5.569 $\pm$ 0.234)	(5.982 $\pm$ 0.291)
naïve	78.19%	74.01%	5.534	7.009	4.583	4.967
residual	(77.74 $\pm$ 0.298)	(74.19 $\pm$ 0.262)	(5.723 $\pm$ 0.120)	(6.980 $\pm$ 0.122)	(4.705 $\pm$ 0.212)	(5.056 $\pm$ 0.0988)
Ours	78.09%	74.55%	5.427	6.731	4.619	4.984
	(77.60 $\pm$ 0.296)	(74.42 $\pm$ 0.233)	(5.760 $\pm$ 0.208)	(7.024 $\pm$ 0.206)	(4.716 $\pm$ 0.0912)	(5.071 $\pm$ 0.0687)

Equations

$$\mathbf{h}\propto {\mathbf{K}}^{-\mathsf{T}}\mathbf{Rg},$$

$${\mathbf{h}}_N\propto {\mathbf{K}}_N^{-\mathsf{T}}{\mathbf{R}}_{\mathsf{IMU}\to N}{\mathbf{R}}_{\mathsf{IMU}}{\left[\begin{array}{ccc}0,& 1,& 0\end{array}\right]}^{\mathsf{T}}.$$

$$\mathbf{h}(\omega ,\theta )={\left[\begin{array}{ccc}\sin \theta ,& \cos \theta ,& -\frac{W}{2}\sin \theta -\omega \cos \theta \end{array}\right]}^{\mathsf{T}}.$$

$$s_t=\alpha x_t+(1-\alpha )s_{t-1}$$

$${\mathbf{y}}_t={\mathbf{h}}_t+{\mathbf{x}}_t.$$

$${\mathbf{Y}}_t=\tanh ({\hat{\mathbf{Y}}}_t+{\mathbf{X}}_t).$$

$${\hat{\mathbf{Y}}}_t={\mathbf{W}}_{xy}\ast {\mathbf{X}}_t+{\mathbf{W}}_{hy}\ast {\mathbf{H}}_{t-1}+{\mathbf{W}}_{\hat{h}y}\ast {\hat{\mathbf{H}}}_t.$$

$$L_H(x,\hat{x})=\{\begin{array}{ll}\frac{1}{2}(x-\hat{x}{)}^2& \text{for }\left|x-\hat{x}\right|\le 1,\\ \left|x-\hat{x}\right|-\frac{1}{2}& \text{otherwise.}\end{array}$$

$$L_{\omega ,\theta }=L_H(\omega ,\hat{\omega })+L_H(\theta ,\hat{\theta }).$$

$$y(\omega ,\theta ,x)=\left(x-\frac{W}{2}\right)\tan \theta -\omega .$$

$$L_e=\{\begin{array}{ll}\frac{1}{H}{d}_{y,0}& \text{for }{d}_{y,0}\ge d_{y,W},\\ \frac{1}{H}{d}_{y,W}& \text{otherwise.}\end{array}$$

$$L(t)=\lambda (t)\cdot L_{\omega ,\theta }+(1-\lambda (t))\cdot L_e.$$

$$A_{TV}=\frac{1}{M}\sum _{n=1}^N\sum _{t=1}^{T_n}\left|\frac{\partial L_e^{n,t}}{\partial t}\right|.$$

$${\mathbf{y}}_t=\sum _{m=1}^M{\mathbf{h}}_m{\mathbf{x}}_{t-m+1},{\mathbf{y}}_t\in {\mathbb{R}}^D,$$

$${\mathbf{Y}}_t=\sum _{m=1}^M{\mathbf{H}}_m\ast {\mathbf{X}}_{t-m+1},$$

$$S_{\mathrm{fov}}=1-L+\sum _{l=1}^L{M}_l.$$

Full text

1 Introduction

Horizon lines are important low-level geometric image features that provide essential information about the relation between a 3D scene and the camera observing it. They can be used to infer the camera pose in form of a ground plane normal or a gravity vector, respectively. In autonomous driving, ground planes are often used to infer semantic properties of the dynamic environment [1, 12]. Other applications include estimation of vanishing points [47], which provide information about the 3D structure of a scene, image metrology [7], perspective correction [26] and camera pose estimation [11, 20].

For many applications, utilising temporal consistency has been demonstrated to improve performance. Examples include depth estimation [39], motion segmentation [4], action recognition [22], super resolution [17] and superpixel segmentation [33]. Single image approaches for horizon line estimation may do gross mistakes when the image provides few or misleading clues. As illustrated by Fig. 1, an approach based on multiple images is less susceptible to these problems if it is able to transfer information from previous images of a sequence.

1.1 Contributions

In this work, we present a novel approach for temporally consistent horizon line estimation based on a convolutional neural network combined with an improved convolutional long short-term memory (LSTM). A comprehensive evaluation demonstrates the ability of this approach to generate more accurate horizon line estimates with less variance. Since a naïve loss function does not track the geometric error of horizon lines very well, and a loss based on the geometric error exhibits singularities that may cause instability, we propose an adaptive loss function that combines both losses with a cosine annealing schedule. This loss function yields significantly more accurate horizon estimates, yet ensures that the neural network training remains stable. In an ablation study, we investigate the influence of several hyper-parameters and architecture choices on the performance of the neural network models. Furthermore, the KITTI Horizon dataset is presented, an extension of the well established KITTI benchmark [14]. It contains accurate horizon line annotations for all video sequences of the KITTI dataset [13]. In summary, our main contributions are:

We present a novel CNN architecture for temporally consistent horizon line estimation based on an improved residual convolutional LSTM. 2. 2.

We propose an adaptive loss function that yields accurate horizon line estimates and ensures stable training. 3. 3.

A large-scale video dataset for temporally consistent horizon line estimation, the KITTI Horizon dataset. To the best of our knowledge, this is the first video dataset with accurate horizon line ground truth.

1.2 Types of Horizon Lines

It is possible to distinguish three types of horizon lines: the visible horizon, the true horizon and the geometrical horizon. The visible horizon is the apparent line which separates earth and sky. Its appearance is often shaped by the surroundings of an observer in the presence of entities like mountains, buildings or trees. If the view of an observer is unobstructed – at sea, for example – the visible horizon becomes identical to the true horizon. Assuming a spherical earth surface, the true horizon is the projection of a circle containing all points on the earth which are tangent to light rays passing through the point of view of an observer.

The geometrical horizon $\mathbf{h}$ is defined as the vanishing line, i.e. the projection of the line at infinity, for any plane orthogonal to the local gravity vector $\mathbf{g}$:

[TABLE]

with $\mathbf{R}$ being the orientation and $\mathbf{K}$ being the intrinsic calibration of the camera. Without loss of generality, we assume that $\mathbf{g}\propto\begin{bmatrix}0,&1,&0\end{bmatrix}^{\mkern-1.5mu\mathsf{T}}$ is parallel to the the zenith direction. As illustrated by Fig. 2, the geometrical horizon is generally not identical to the vanishing line of the plane an observer is standing on, as its normal vector may not be parallel to $\mathbf{g}$, e.g. when located on an incline. Being a theoretical construction, the geometrical horizon is imperceptible to an observer. However, given the intrinsic calibration $\mathbf{K}$, knowledge of the geometrical horizon is sufficient to estimate camera tilt and roll w.r.t. a global coordinate system. Fig. 3 illustrates the conceptual differences between the three horizons. Since the remainder of this paper considers the geometrical horizon, it will be simply referred to as the horizon from hereon.

1.3 Related Work

In the past, numerous approaches for horizon line estimation have been proposed, and they can be differentiated into a number of categories. Most methods rely on vanishing points (VPs) [3, 24, 25, 28, 37, 40, 41, 42, 46] which they detect by grouping oriented elements like line segments or edges into clusters which have the same orientation in 3D space. If at least two vanishing points are known, the horizon line can be derived. Some of these methods [25, 34, 40, 42] rely on the Manhattan-world assumption [6], i.e. they restrict their solution space to three VPs of orthogonal directions and are hence applicable to only a limited number of scenes. Others [3, 28, 36, 37, 46] use the more permissible Atlanta-world assumption [36], which expects all horizontal VPs to be of orthogonal direction to a zenith VP. This assumption is still restrictive, as it does not cover scenes which contain planes that are oblique to a defined zenith. Several of the aforementioned methods consider two benchmark datasets in their evaluation: the York Urban Dataset [9] (YUD) and the Eurasian Cities Dataset [3]. Both are relatively small and of limited diversity w.r.t. the types of scenes they depict. In 2016, Workman et al. [43] presented the Horizon Lines in the Wild (HLW) dataset, which contains horizon line ground truth for 100553 images taken at various locations. Availability of such a large-scale dataset has lead to an emergence of deep-learning based algorithms [27, 43, 47] more recently. Workman et al. [43] present a convolutional neural network (CNN) which directly estimates the horizon line from a single image, formulated as either a regression or a classification task. Lee et al. [27] use a different approach: they randomly sample lines within the image borders and feed them, along with the image, into a CNN which incorporates their proposed line pooling layer. This CNN then provides a classification whether the sampled line is the horizon of the image and, in addition, computes refined line coordinates. The method of Zhai et al. [47] is a hybrid approach. It uses a CNN, similar to [43], to predict a horizon line, but then jointly optimises its location together with VPs which are estimated based on line segments that have been detected in a preprocessing step. All these works have in common that they target the problem of single image horizon line estimation. To the best of our knowledge, general datasets and algorithms targeted specifically at horizon line estimation from video sequences do not exist.

2 KITTI Horizon Dataset

We introduce the KITTI Horizon Dataset, a new addition to the KITTI raw dataset [13] with accurate horizon line annotations for all video sequences.

2.1 Limitations of Existing Datasets

Three datasets have been commonly used for horizon line estimation in recent years: the York Urban Dataset [9] (YUD), the Eurasian Cities Dataset [3] (ECD) and Horizon Lines in the Wild [43] (HLW). YUD is a relatively small dataset of 102 images depicting in- and outdoor scenes within a confined area, taken with the same camera under similar conditions. While ECD is somewhat more diverse than YUD, it is still very small with just 103 images. HLW, on the other hand, is significantly larger and contains 100553 images, making it much better suited for data-intensive deep learning approaches. Unlike YUD and ECD, HLW was not labelled manually, but in an automatic process using structure from motion. It appears, however, that this process has limited precision, as some images in HLW have clearly inaccurate horizon line labels. Beyond that, all three datasets have in common that they do not contain video sequences, which means that they can only be used for single image horizon line estimation and are ill-suited for research on temporally consistent horizon line estimation. To our knowledge, the Singapore Maritime Dataset [32] (SMD) is the only video dataset with annotated horizon lines. Although it is relatively large, containing 21981 annotated frames, its diversity is very limited since it exclusively shows maritime scenes of similar appearance. More importantly, however, the horizon labels in SMD describe the true horizon as opposed to the geometrical horizon. Consequently, a new dataset is needed for temporally consistent geometrical horizon line estimation.

2.2 KITTI

KITTI [13] is a computer vision dataset which was captured using a sensor array mounted on top of a vehicle. Sensors used for the recordings include four front-facing video cameras and a high accuracy inertial measurement unit (IMU), among others. Several benchmarks for various applications, such as object detection, depth estimation or semantic segmentation, have been published [14]. For horizon line estimation such a benchmark does not exist. We can, however, compute accurate horizon line ground truth using the IMU data provided by KITTI, at no additional cost.

2.3 Horizon Line Ground Truth

KITTI provides an accurate absolute pose $\mathbf{R}_{\mathsf{IMU}}$ of the IMU in 3D space for every image. Together with the relative pose between IMU and camera $N\in\{1,2,3,4\}$, $\mathbf{R}_{\mathsf{IMU}\rightarrow N}$, we can compute the normalised gravity vector $\mathbf{g}_{N}\propto\mathbf{R}_{\mathsf{IMU}\rightarrow N}\mathbf{R}_{\mathsf{IMU}}\begin{bmatrix}0,&1,&0\end{bmatrix}^{\mkern-1.5mu\mathsf{T}}$ in the coordinate system of the camera.

As explained in Sec. 1.2, the projection of a gravity vector $\mathbf{g}$ into the camera using Eq. 1 yields the horizon line in homogeneous coordinates:

[TABLE]

As this process requires no manual labelling or other human intervention, we can compute the ground truth horizon for all images fully automatically. Fig. 4 shows a few examples. In the left-hand image, the ground plane appears nearly perpendicular to the gravity vector, hence the horizon line is virtually identical to the vanishing line of that ground plane. In the other two images, however, they are clearly distinct due to the fact that the ground plane is sloping downwards (middle image) or upwards (right-hand image).

2.4 Train, validation and test split

The complete published KITTI dataset consists of 47962 frames across 157 sequences. Several sequences show the same stationary scene, and only differ w.r.t. the people walking across the image. As these are of negligible value for our task, we discarded all but one, so that 72 sequences with 43699 frames remain. As no official split exists for the raw dataset, we divided the video sequences into roughly 70% training, and 15% validation and test data each. Care was taken to ensure that sequences showing very similar scenes, e.g. the same intersection, do not end up in different parts of the split. As there is a strong imbalance in sequence length, e.g. some sequences contain less than 100 frames and others have several thousand, we divided one of the longer videos equally and put it into the test and validation sets.

3 Single Image Estimation

We obtained the source code of recent single image algorithms [24, 28, 37, 43, 47]. In addition, we compare our own single image algorithm along with these methods. Thereby, we obtain a detailed and unbiased comparison that clearly highlights the features of our temporally consistent approach. Our single image algorithm is based on a CNN, similar to the regression approach presented in [43]. We parametrise the horizon line $\mathbf{h}$ by offset $\omega$ and slope $\theta$. With $W$ being the image width, its representation in homogeneous coordinates is defined as:

[TABLE]

We replace the GoogleNet [38] of [43] with the more recent and efficient ResNet [18], and use the most shallow 18-layer variant (ResNet18). The classification layer of the ResNet is replaced by two fully connected layers with single real valued outputs for $\omega$ and $\theta$. Apart from downscaling the image, we do not perform any pre- or post-processing.

4 Temporally Consistent Estimation

Possibly the simplest way to utilise the temporal consistency of video sequences is applying a single-frame algorithm first, and then averaging the results. For online applications, a reasonable choice of filter would be the exponential moving average, or exponential smoothing filter [5]. Given a sequence $x_{t}$, the output of the filter is defined as:

[TABLE]

While easy to implement, it always just achieves a compromise between suppressing noise and outliers, and preserving actual trajectory changes. Bai et al. [2] propose temporal convolutional networks (TCN), an extension of regular CNNs by causal convolutions [30] along an additional temporal dimension. Across time, the TCN has a fixed field of view which limits the sequence length along which it is able to infer correlations. We therefore chose to investigate an approach based on long-short term memory (LSTM) [19]. We devised a novel approach combining the ResNet [18] architecture with an improved convolutional LSTM layer.

4.1 Convolutional LSTM

LSTM cells are a particular type of recurrent neural network (RNN) that have been proven effective in modelling both long- and short-term dependencies of sequential data [16, 35, 44]. The convolutional LSTM (ConvLSTM) [39, 45] is a variant that operates on 3D tensors instead of vectors and replaces all matrix multiplications with kernel convolutions. Given a sequence of inputs $\mathbf{X}_{1},\dots,\mathbf{X}_{t}$, the cell state $\mathbf{C}_{t}$ and hidden state $\mathbf{H}_{t}$ of a ConvLSTM can be computed as follows, where ’$*$’ is the convolution operator and ’$\odot$’ denotes the Hadamard product: {IEEEeqnarray}rCl \IEEEyesnumber i_t &= σ(W_xi * X_t + W_hi * H_t-1 + b_i )

f_t = σ(W_xf * X_t + W_hf * H_t-1 + b_f )

o_t = σ(W_xo * X_t + W_ho * H_t-1 + b_o )

C_t = f_t ⊙C_t-1   +

i_t ⊙tanh( W_xc * X_t + W_hc * H_t-1 + b_c )

H_t = o_t ⊙tanh(C_t)

The hidden state is usually treated as the output of the cell, i.e. $\mathbf{Y}_{t}=\mathbf{H}_{t}$. Variants with additional connections [15], or other activation functions [39] exist as well.

4.2 Residual Convolutional LSTM

We propose an improved convolutional LSTM structure that incorporates both residual and dense connections. As previous works [18, 44] have shown, residual connections improve gradient flow in deep neural networks, which makes them easier and faster to train. He et al. [18] integrated residual connections into a CNN. If we consider a shallow stack $l$ of convolutional layers performing an operation $F_{l}(\mathbf{x})$ on an input $\mathbf{x}_{l-1}$, the output $\mathbf{x}_{l}$ of such a stack is: $\mathbf{x}_{l}=g(F_{l}(\mathbf{x}_{l-1})+\mathbf{x}_{l-1})\,$, with $g(\cdot)$ being a nonlinear activation function, e.g. $\mathrm{ReLU}$. In [44], this idea was applied to a network of stacked LSTM cells. Each LSTM cell computes a hidden state $\mathbf{h}_{t}$ and a cell state $\mathbf{c}_{t}$ based on an input $\mathbf{x}_{t}$ and the states at the previous time step: $\mathbf{h}_{t},\mathbf{c}_{t}=\mathrm{LSTM}(\mathbf{h}_{t-1},\mathbf{c}_{t-1},\mathbf{x}_{t})\,$. A residual connection is then applied to generate the final output of the layer:

[TABLE]

In this case, the non-linearity $g(\cdot)$ is part of the LSTM, i.e. it is applied before the residual connection. The notion of improving information flow through a neural network via connections that skip a number of layers was implemented in yet another manner by Huang et al. [21]. In their DenseNet CNN architecture, feature-maps of $M$ preceding layers $\mathbf{x}_{l-M},\dots\mathbf{x}_{l-1}$ are concatenated channel-wise and fed into the current layer $F_{l}(\mathbf{x})$: $\mathbf{x}_{l}=g(F_{l}([\mathbf{x}_{l-M},\dots,\mathbf{x}_{l-1}])\,$. In order to arrive at our improved ConvLSTM, we combine the aforementioned principles and incorporate them as follows. Fig. 5 illustrates our proposed structure on the left side, while the right side shows the standard ConvLSTM with a naïve residual connection as per Eq. 5 for comparison. In keeping with the original ResNet definition, we define a residual connection between input and output:

[TABLE]

As Eq. 4.1 shows, the hidden state $\mathbf{H}_{t}$ amounts to a masked cell state $\mathbf{C}_{t}$. We argue that this inhibits the flow of information from both $\mathbf{X}_{t}$ and $\mathbf{H}_{t-1}$ to the output $\mathbf{Y}_{t}$. Normally, information must pass through Eqs. 4.1-4.1 and thus through $\mathbf{C}_{t}$ before it eventually reaches $\mathbf{Y}_{t}$. We therefore introduce an additional convolutional layer into the ConvLSTM, which directly takes the concatenation of $\mathbf{X}_{t}$, $\mathbf{H}_{t-1}$ and an intermediate hidden state $\mathbf{\hat{H}}_{t}$ as an input, similar to the way convolution layers in DenseNet operate, in order to produce an intermediate output $\mathbf{\hat{Y}}_{t}$:

[TABLE]

Finally, in order to avoid application of the $\tanh$ activation twice onto the information from $\mathbf{C}_{t}$, we switch the order of operation in Eq. 4.1, i.e.: {IEEEeqnarray}rCl \IEEEyesnumber ^H_t &= o_t ⊙C_t   ,

H_t = tanh(^H_t)   .

4.3 Horizon Line Estimation Network

We expand our single image CNN described in Sec. 3 with our modified ConvLSTM presented in Sec. 4.2 in order to create a temporally consistent architecture. As Fig. 6 shows, two ConvLSTM layers are inserted between the last convolutional layer and the global average pooling layer of our ResNet18-based CNN. Intuitively, applying the ConvLSTM at this stage makes most sense, as we would expect it to find temporal correlations between higher-level features which are most pertinent to the task of horizon estimation.

4.4 Loss Function

Our CNN has two real valued outputs: offset $\omega$ and slope $\theta$ of the predicted horizon line. We compute two loss terms; the first one is the Huber loss of $\omega$ and $\theta$ computed w.r.t. the ground truth; the second one is the maximum horizon error within the image. Combining these two losses allows us to benefit from a gain in accuracy elicited by minimising the maximum horizon error, while avoiding the instability it can cause. The Huber loss [23] is defined as:

[TABLE]

We define the first loss term as the Huber loss of $\omega$ and $\theta$ computed w.r.t. the ground truth $\hat{\omega}$ and $\hat{\theta}$:

[TABLE]

As this loss term does not exactly track the maximum horizon error, which is the quantity we actually seek to minimise, we have defined a second loss term. The maximum horizon error is defined as the maximum distance between the estimated horizon $\mathbf{h}(\omega,\theta)$, as defined by Eq. 3, and the ground truth horizon $\mathbf{h}(\hat{\omega},\hat{\theta})$ between the left- and rightmost borders of the image, normalised to image height $H$. The $y$-coordinate of the intersection of $\mathbf{h}$ with a vertical line at $x$ is determined by:

[TABLE]

Let $d_{y,0}$ and $d_{y,W}$ be the left- and right-most distances between the two horizons, $d_{y,x}=\left|y(\omega,\theta,x)-y(\hat{\omega},\hat{\theta},x)\right|\,$. The maximum horizon error $L_{e}$ can then be defined as:

[TABLE]

While $L_{e}$ directly reflects the quantity we aim to minimise, it contains singularities for $\theta=(\nicefrac{{\pi}}{{2}}+n\pi),n\in\mathbb{N}$, due to the $\tan\theta$ term in Eq. 9. This causes $L_{e}$ to become excessively large if $\theta$ is poorly estimated, which may be the case especially at the beginning of neural network training. We therefore use only $L_{\omega,\theta}$ at first, when estimates are still very inaccurate and noisy, and gradually switch over to $L_{e}$ on a cosine schedule similar to [29]. With $t$ being the current epoch and $T$ being the maximum number of epochs, the schedule is defined by: $\lambda(t)=\frac{1}{2}+\frac{1}{2}\cos\left(\pi\cdot\frac{t}{T}\right)\,.$ Using this, the final loss $L$ is defined as:

[TABLE]

5 Experiments

We empirically demonstrate the effectiveness of our temporally consistent horizon line estimation pipeline on the KITTI Horizon validation and test sets and compare it with state-of-the-art single-image algorithms and other temporally consistent baselines. Additional ablation studies show the importance of individual parts of this pipeline for achieving these results.

5.1 Implementation Details

We implemented the proposed neural network architectures using PyTorch [31]. On KITTI Horizon, all networks were trained for 160 epochs with stochastic gradient descent using a cosine annealing learning rate schedule [29] starting at $10^{-1}$ and ending at $10^{-3}$. Training was repeated four times with different random seeds, and the model with the highest validation AUC chosen. We downscale each image by a factor of two and apply cutout [10], colour jitter, random rotations and random shifts for data augmentation. We initialise the weights of the first nine convolutional layers of the networks from a ResNet18 pretrained on ImageNet [8] while other layers are initialised randomly. Training batches always contain $B$ sequences of $S$ consecutive frames from the KITTI Horizon training set, and batch size $B$ and sequence length $S$ were set to fulfil $S\cdot B=128$.

5.2 Evaluation Metrics

As in [24, 28, 37, 43, 47], we compute the maximum horizon error defined in Eq. 10 for every image in the dataset. A cumulative error histogram for errors up to $0.25$ is generated and its area under the curve (AUC) determined for a set of images. This horizon error AUC value gauges the overall accuracy of the estimated horizon lines. We also report the mean squared error (MSE), which is more sensitive to outliers than the AUC. In addition, we compute the estimated camera pose vector $\mathbf{p}\propto\mathbf{R}\mathbf{g}\propto\mathbf{K}^{\mkern-1.5mu\mathsf{T}}\mathbf{h}$ via inversion of Eq. 1. We determine the angular error $\xi$ between $\mathbf{p}$ and the ground truth pose $\mathbf{\hat{p}}$ for every image and report the AUC of the cumulative error histogram for $\xi\leq 5^{\circ}$: $\cos\xi=\frac{\mathbf{p}^{\mkern-1.5mu\mathsf{T}}\mathbf{\hat{p}}}{\left\lVert\mathbf{p}\right\rVert_{2}\left\lVert\mathbf{\hat{p}}\right\rVert_{2}}\,.$ For applications that rely on horizon lines estimated from a video stream, it is desirable for the estimations be accurate as well as stable. We propose another metric to measure undesirable fluctuations that do not reflect actual changes of the horizon over time: the average total variation $A_{TV}$. For a sequence $n$ of length $T_{n}$ of estimated horizons $\mathbf{h}_{n,t}$ and corresponding ground truth $\mathbf{\hat{h}}_{n,t}$, with $t\in[1,T_{n}]$ and $n\in[1,N]$, we compute the derivative $\nicefrac{{\partial L_{e}^{n,t}}}{{\partial t}}$ of the horizon error according to Eq. 10 using second order approximation. With $M=\sum_{n=1}^{N}T_{n}$ being the total number of images, the mean of its absolute calculated over all sequences yields the average total variation:

[TABLE]

This metric is invariant to constant deviations from the ground truth but sensitive to higher frequency fluctuations.

5.3 KITTI Horizon Results

We report all metrics on the KITTI Horizon validation and test set for for following single frame algorithms: the VP based methods of Lezama et al. [28], Kluger et al. [24] and Simon et al. [37], the hybrid approach of Zhai et al. [47] and the CNN based approach of Workman et al. [43]. We also include results for our single frame CNN baseline (cf. Sec. 3), trained on either HLW or KITTI, for an average baseline which simply always predicts the mean of the training set, a TCN [2] based temporally consistent approach with causal convolutions in the last three layers, and of course for our temporally consistent pipeline presented in Sec. 4. The results are listed in Tab. 1 and Fig. 7. As these numbers show, methods based on line segments and vanishing points [24, 28, 37, 47] are unable to deliver consistent and accurate horizon estimates on KITTI. The best performing method among them is Zhai et al. [47] with 60.97%/50.98% AUC (validation/test), which still lags behind the simplest average baseline (69.40%/64.18%). In addition, the very large mean squared error (MSE) and average total variation ($A_{TV}$) values – up to several thousand – indicate that these methods may fail catastrophically in some outlier cases. In comparison, all CNN based methods – including Workman et al. [43] and our own single frame CNN – are significantly more accurate with at least 70.32%/63.64% AUC. More importantly, the comparatively smaller MSE and $A_{TV}$ show that these methods are much less prone to extreme outliers. If we compare the CNN of [43] with our own single-frame CNN trained on HLW, we observe that [43] performs better overall – all metrics but validation AUC are better to a relevant degree. This is unsurprising, as [43] augmented their training with an additional 500000 images sampled from Google Street View, while we just used HLW. Naturally, if trained on the KITTI Horizon dataset, the accuracy of our single frame CNN increases significantly: from 71.10%/63.64% to 77.42%/74.08%, which is a 21.8%/28.7% relative increase. Best results on all metrics are obtained with our temporally consistent approach (Sec. 4), with relative improvements upon the single frame CNN between 1.8% (test AUC) and 12.1% (test $A_{TV}$). While the smoothness $A_{TV}$ of the single frame CNN improves measurably without diminishing overall accuracy if we additionally apply an exponential smoothing filter (Eq. 4, $\alpha=0.5$), similar gains can be achieved when applied to the temporally consistent CNN as well, so it still retains its advantage. We also trained a TCN [2] based on our single frame CNN with causal temporal convolutions of widths 3, 3, and 5 in the last three layers and a receptive field of nine frames. Surprisingly, it performs worse than the single frame CNN on all metrics but $A_{TV}$. We suspect that the TCN is more susceptible to overfitting, as it achieved a lower training loss, but higher validation loss compared to our other CNNs. Compared to our ConvLSTM based network, it is on par w.r.t. $A_{TV}$ on the test set, but measurably worse otherwise.

5.4 Ablation Studies

5.4.1 Loss function

In order to investigate whether our new loss defined in Sec. 4.4 had the desired effect on estimation accuracy, we also trained our main CNN model described in Sec. 4.3 using just the Huber loss defined in Eq. 8 and also used by [43]. As Tab. 2 shows, we report an AUC of 71.96% and an MSE of $7.851\cdot 10^{-3}$ on the test set. Using our newly defined loss, however, we achieve an AUC of 74.55% and an MSE of $6.731\cdot 10^{-3}$, which marks a considerable relative improvement of 9.2% and 14.3% respectively.

5.4.2 Temporal information

As Tab. 1 shows, our temporally consistent approach based on ConvLSTMs is able to achieve more accurate horizon estimates with significantly less variance. In order to ascertain that this is due to the ConvLSTM utilising temporal correlations, and not simply due other architecture changes that arose as a result, we retrained our main CNN model with all temporal connections disabled, i.e. we reset the LSTM states at every time step. On the test set, this yields an AUC of 74.36% and $A_{TV}$ of $5.699\cdot 10^{-3}$. When we enable the temporal connections of the LSTM, overall accuracy increases moderately – 74.55% AUC – and $A_{TV}$ decreases noticeably to $4.984\cdot 10^{-3}$, which is a relative improvement of 12.6%. We conclude that the ConvLSTM is indeed able to retain temporal consistency in a meaningful way.

5.4.3 ConvLSTM Architecture

We compare our ConvLSTM architecture described in Sec 4.2 against a ConvLSTM using a naïve residual path implementation and ConvLSTM without the residual path. As Tab. 2 shows, the naïve residual path already increases accuracy dramatically, from 64.29% to 74.01% AUC, and is evidently crucial for deep LSTM networks. On par w.r.t. $A_{TV}$, our proposed ConvLSTM improves AUC and MSE upon the naïve implementation, yielding a relative improvement of 2.1% and 4.0% respectively. While both approaches are able to generate smooth trajectories, our improved ConvLSTM is measurably more accurate on average.

6 Conclusion

The horizon line is an important geometric feature which can be used in many computer vision tasks, such as camera pose and ground plane estimation. Due to their importance, horizon lines have received considerable attention in recent years. Nonetheless, neither has any other work has focused on temporal consistency, nor are there appropriate datasets available. In this work, an extension of the well-known KITTI database is presented that adds horizon line annotations to 72 sequences. We furthermore propose a neural network for temporally consistent horizon line estimation in video sequences. It utilises an improved convolutional LSTM and an adaptive loss function that yields more accurate horizon line estimates and ensures stable training. The experimental evaluation demonstrates that the proposed architecture achieves superior performance for a diverse set of metrics which measure, for instance, accuracy and smoothness of trajectories.

Acknowledgements

This work was supported by German Research Foundation (DFG) grant Ro 2497 / 12-2.

This appendix is structured as follows: In Sec. A, we state more in-depth details of the implementation of our proposed method. Sec. B describes our implementation of Temporal Convolutional Networks, and provides additional experiments and evaluation. Sec. C.1 provides a more detailed look into the quantitative results of our proposed approach as well as its ablation studies, while Sec. C.2 contains horizon trajectories from KITTI Horizon to highlight a few best-, average- and worst-case examples of our approach. In Sec. C.3, we provide a few examples from the KITTI Horizon dataset which convey an impression of the variety of scenes it contains. Sec. D discusses examples from the Horizon Line in the Wild dataset which have visibly inaccurate horizon line labels.

Appendix A Additional Implementation Details

We implemented all proposed neural network architectures using PyTorch [31] version 0.4.1. We used stochastic gradient descent with momentum ($0.9$) and $L_{2}$ regularisation ($10^{-4}$), and a cosine annealing learning rate schedule [29] starting at $10^{-1}$ and ending at $10^{-3}$. On KITTI Horizon, all networks were trained for 160 epochs, and for 256 epochs on HLW. Where applicable (see Sec. C), training was repeated four times with different random seeds, and the model with the highest validation AUC chosen. We downscale each image by a factor of two and apply the following augmentation techniques:

• •

Random rotations $\beta\sim\mathcal{U}(-2^{\circ},2^{\circ})$ w.r.t. the image centre.

• •

Random shifts $s_{x}\sim\mathcal{U}(-10~{}\mathrm{px},10~{}\mathrm{px})$ and $s_{y}\sim\mathcal{U}(-10~{}\mathrm{px},10~{}\mathrm{px})$.

• •

Horizontal flips with probability $p=0.5$.

• •

Colour jitter with brightness factor $\gamma_{b}\sim\mathcal{U}(0.75,1.25)$, contrast factor $\gamma_{c}\sim\mathcal{U}(0.75,1.25)$, saturation factor $\gamma_{s}\sim\mathcal{U}(0.75,1.25)$ and hue factor $\gamma_{h}\sim\mathcal{U}(-0.25,0.25)$.

• •

Greyscale transformation with probability $p=0.1$.

• •

Cutout [10] with width $w\sim\mathcal{U}(0,512)$ and height $h\sim\mathcal{U}(0,512)$.

We initialise the weights of the first nine convolutional layers of the networks from a ResNet18 pretrained on ImageNet [8] while other layers are initialised randomly. Training batches always contain $B$ sequences of $S$ consecutive frames from the KITTI Horizon training set, and batch size $B$ and sequence length $S$ were set to fulfil $S\cdot B=128$. We set $S=1$ for single frame approaches and $S=32$ for temporally consistent approaches. Sampled sequences were always non-overlapping. At test time, we process the whole sequence as it appears in the dataset.

Appendix B Temporal Convolutional Networks

As a possible alternative to our ConvLSTM based CNN, we briefly discussed Temporal Convolutional Networks (TCN) [2] in our paper. The authors of [2] propose it as a purely feed-forward alternative to recurrent neural network structures – such as LSTM and ConvLSTM – for sequence modelling, and present promising results. We therefore implemented a TCN for the horizon line estimation task and compared it to our proposed ConvLSTM based architecture. The concept of TCNs is based on causal convolutions along the temporal dimension of data. For a sequence of vectors $\mathbf{x}_{t}\in\mathbb{R}^{C}$, the 1D causal convolution across time with a kernel $\mathbf{h}\in\mathbb{R}^{M\times D\times C}$ can be defined as:

[TABLE]

where $M$ denotes the number of elements of the sequence included in the convolution. Unlike a regular convolution, the result $\mathbf{y}_{t}$ of the causal convolution only depends on values of $\mathbf{x}_{\tau}$ for $\tau\leq t$, i.e. no information from the future is considered. This can easily be generalised for sequences of images or feature maps $\mathbf{X}_{t}\in\mathbb{R}^{W\times H\times C}$ and a corresponding kernel $\mathbf{H}\in\mathbb{R}^{M\times A\times B\times D\times C}$:

[TABLE]

where ’$*$’ denotes the 2D convolution operator commonly used in CNNs, $W$ and $H$ are image width and height, and $A\times B$ is the kernel size. Using regular convolutional layers readily available in deep learning frameworks, causal convolutional layers can be realised by simply shifting the output along the temporal axis by $\left\lfloor\nicefrac{{M}}{{2}}\right\rfloor$ steps. If $L$ such layers with temporal convolution lengths $M_{l}$ are stacked to form a deeper network, the temporal field of view of this network becomes:

[TABLE]

We converted our ResNet18 based single-frame CNN into a TCN by replacing the last three respectively four 2D convolutional layers with 3D convolutional layers. We considered various configurations with $1\leq M_{l}\leq 7$ and $9\leq S_{\mathrm{fov}}\leq 19$, which we trained on the KITTI Horizon dataset as described in Sec. A. We named the configurations according to the values set for $M_{l}$ in the last four layers, e.g. 1-3-3-5 means $M_{L-3:L}=[1,3,3,5]$. In order to avoid zero padding in the temporal dimension, we sampled additional $S_{\mathrm{fov}}-1$ previous frames for each sequence in a training batch, if possible. Tab. 3 shows the results of our TCNs on the KITTI Horizon validation and test sets. Compared to our ConvLSTM based approach, the TCNs perform poorly w.r.t. all metrics but $A_{TV}$. They even fall behind the single frame CNN w.r.t. AUC and MSE. We suspect that the TCNs are more susceptible to overfitting. In Fig. 8, we compare the training and validation losses of the 3-3-5-7 TCN with our proposed ConvLSTM network. The TCN achieves a noticeably lower training loss, but converges to a significantly higher validation loss. This indicates a lower ability of the TCN to generalise and may explain the poor validation and test performance.

Appendix C Additional Results

C.1 Quantitative

In order to gauge the uncertainty of our results arising from the randomness involved in neural network training, we repeated training for most of our experiments four times with varying random seeds. These experiments include: (a) our proposed ConvLSTM based CNN (Ours), (b) the ablation study using the naïve residual ConvLSTM, (c) the ablation study using the ConvLSTM with disabled temporal connections (non-temporal) and (d) the single frame CNN. In addition to the results for the training runs which perform best on the validation set, we also report mean and standard deviation over all four runs in Tab. 4. As these results show, the single-frame and non-temporal variants perform worse than the temporally consistent approaches, even when averaged over multiple runs. Comparing our proposed ConvLSTM with the naïve residual variant, we observe similar performance w.r.t. $A_{TV}$ on both validation and test sets. AUC and MSE results on the validation set are similar as well, w.r.t. both best and mean performance. At the same time, the test performance of our proposed model with the best validation performance is measurably better than the naïve residual variant, which indicates an improved generalisation ability of our proposed ConvLSTM.

C.2 Qualitative

In Fig. 9, we show three example horizon line trajectories from the KITTI Horizon dataset. In the first example, Fig. 9(a), the single frame estimation fluctuates heavily, while our proposed temporally consistent approach remains much more stable throughout the sequence. The second example, Fig. 9(b), contains segments where the single frame estimation is moderately better, e.g. between frames 150 and 200, but also segments where the single frame estimation shows severe fluctuations, e.g. frames 200 to 300. Besides that, the results of the two algorithms are mostly very similar. Lastly, in Fig. 9(c), we can observe a failure case of our approach. In the middle section between frames 100 and 300, both algorithms perform similarly, but at the beginning and at the end, our proposed method exhibits a relatively constant but large error.

C.3 KITTI Horizon Dataset

We provide a few examples from the KITTI Horizon dataset with ground truth horizons in Fig. 10, in order to give an impression of the variety of scenes it contains.

Appendix D Horizon Lines in the Wild

As mentioned in the paper, we noticed that some of the horizon line labels provided by the Horizon Lines in the Wild (HLW) [43] dataset are visibly inaccurate. In Fig. 11, we show a few examples which convey the severity of the problem. This is by no means an exhaustive analysis, but we hypothesise that the HLW ground truth contains noticeable errors, which should be kept in mind when using this dataset. However, a more detailed analysis is required to quantify and test our hypothesis.