Stereo Event Lifetime and Disparity Estimation for Dynamic Vision Sensors

TL;DR

This paper introduces a novel, fast, and accurate method for estimating event lifetime and disparity in stereo event-based cameras, enabling improved depth perception in dynamic scenes.

Contribution

It presents a new joint approach for event lifetime and disparity estimation from stereo event data, leveraging stereo matching for single shot processing.

Findings

Approximately twice as fast as separate lifetime and disparity estimation methods.

More accurate disparity estimation compared to traditional separate approaches.

Validated on real-world stereo event-camera data.

Abstract

Event-based cameras are biologically inspired sensors that output asynchronous pixel-wise brightness changes in the scene called events. They have a high dynamic range and temporal resolution of a microsecond, opposed to standard cameras that output frames at fixed frame rates and suffer from motion blur. Forming stereo pairs of such cameras can open novel application possibilities, since for each event depth can be readily estimated; however, to fully exploit asynchronous nature of the sensor and avoid fixed time interval event accumulation, stereo event lifetime estimation should be employed. In this paper, we propose a novel method for event lifetime estimation of stereo event-cameras, allowing generation of sharp gradient images of events that serve as input to disparity estimation methods. Since a single brightness change triggers events in both event-camera sensors, we propose a method for single shot event lifetime and disparity estimation, with association via stereo matching. The proposed method is approximately twice as fast and more accurate than if lifetimes were estimated separately for each sensor and then stereo matched. Results are validated on real-world data through multiple stereo event-camera experiments.

Figures

Tables

Table 1. TABLE I : Number of events with estimated lifetime and pertaining estimation execution times. The proposed method runs nearly twice as fast compared to the decoupled method. The time needed to calculate the disparity is not included.

	Decoupled		Proposed
Scene	Events	Time [ms]	Events	Time [ms]
Figure 1	2692	32	2759	16
Figure 5	2536	30	2498	15
Figure 6	426	11	436	6

Equations

$$S(\mathbf{p}+\Delta \mathbf{p})\approx S(\mathbf{p})+\left(S_x(\mathbf{p}),S_y(\mathbf{p})\right)\Delta \mathbf{p},$$

$$\tau (\mathbf{p})=\max \{\Delta t\},\text{ subject to }\Vert \Delta \mathbf{p}\Vert =1,$$

$$\tau (\mathbf{p})=\max \{\nabla S(\mathbf{p})\Delta \mathbf{p}\},\text{ subject to }\Vert \Delta \mathbf{p}\Vert =1.$$

$$\frac{\partial S}{\partial x}\left(x,y_0\right)=\frac{1}{v_x\left(x,y_0\right)},$$

$$\frac{\partial S}{\partial y}\left(x_0,y\right)=\frac{1}{v_y\left(x_0,y\right)}.$$

$$\nabla S(\mathbf{p})={\left(\frac{1}{v_x(\mathbf{p})},\frac{1}{v_y(\mathbf{p})}\right)}^{\top },$$

$$\tau (\mathbf{p})=\nabla S(\mathbf{p})\frac{\nabla S(\mathbf{p})}{\Vert \nabla S(\mathbf{p})\Vert }=\sqrt{v_x^{-2}+v_y^{-2}}.$$

$$\begin{array}{rl}\mathbf{n}(\mathbf{p})& \propto (1,0,S_x(\mathbf{p}){)}^{\top }\times (0,1,S_y(\mathbf{p}){)}^{\top }\\ & =(-S_x(\mathbf{p}),-S_y(\mathbf{p}),1{)}^{\top }.\end{array}$$

$$\tau (\mathbf{p})=\frac{1}{n_3}\sqrt{n_1^2+n_2^2}.$$

$${\mathbf{n}}_{\mathrm{LS}}=\underset{\mathbf{n}}{\mathrm{argmin}}\Vert \mathbf{An}-\mathbf{b}{\Vert }^2,$$

$$\Delta t_{\mathrm{err}}=|t_i-\widehat{t_i}|.$$

$$C(\mathbf{p},d)=\sum _{(x_i,y_i)\in W_p}\left|{\mathbf{D}}^L(x_i,y_i)-{\mathbf{D}}^R(x_i-d,y_i)\right|.$$

$$C_A^0(\mathbf{p},d)=C(\mathbf{p},d),C_A^i(\mathbf{p},d)=\sum _{\mathbf{q}\in A_p}{C}_A^{i-1}(\mathbf{q},d).$$

$$D(\mathbf{p})=\underset{\mathbf{d}}{\mathrm{argmin}}{C}_A(\mathbf{p},d).$$

$$\tau (\mathbf{p})=\mathrm{median}\{\tau (\mathbf{q}),\mathbf{q}\in M\}.$$

Full text

I Introduction

Event-based vision sensors [1], such as the dynamic vision sensor (DVS) [2] and dynamic and active-pixel vision sensor (DAVIS) [3], are relatively novel biologically inspired cameras that output pixel-wise changes in brightness intensity in the scene, which are referred to as events due to their asynchronous nature. Since event-based vision sensors do not record the brightness intensity of an entire scene at a fixed frame rate, but offer a sparse data stream reporting only changes in the intensity, they have significantly lower power consumption and bandwidth requirements than standard cameras still widely used in computer vision. Furthermore, event-based cameras operate with low latency and with a temporal resolution of a microsecond, thus avoiding the problem of motion blur that makes standard cameras unusable in highly dynamic scenarios. Also, event-based cameras have the advantage of a high dynamic range (HDR) of 140 dB, compared to the 60 dB of standard cameras, making them an interesting sensor for tasks with challenging scene illumination and environmental conditions, often encountered by autonomous mobile robots. A recent survey on event-based vision applications and methods can be found in [4] and an example of a processed DVS output can be seen in Fig. 1.

Standard computer vision algorithms that operate on image intensity frames cannot be directly applied to the fundamentally different event stream of event-based vision sensors. Recently, significant efforts have been made in the area of intensity image reconstruction from events [5, 6]; however, for some applications complex image intensity reconstruction is not needed. If the incoming events are simply accumulated over time, they can be represented as a frame and used as an input to slightly altered standard image processing algorithms. Accumulation can be done directly by choosing a fixed time interval and this approach has been used for tracking and optical flow estimation [7, 8]. Events can also be accumulated by choosing a certain number of events to be shown in each generated frame, as shown in [9] for simultaneous localization and mapping (SLAM). Since the frequency of incoming events depends on the amount of motion of the scene and the camera itself, an accumulation interval that is too long will produce blurred images, while a short interval will cause loss of structure in the scene, thus not allowing correct interpretation. Given that, regardless of the strategy, event accumulation introduces latency and does not exploit fully the asynchronous nature and high temporal resolution of the sensor. To overcome the aforementioned problems, the nature of the sensor implies that each event should be dynamically augmented with its duration and velocity. Authors in [10] introduced the concept of an event lifetime and propose to estimate it by leveraging optical flow [11], thus allowing to identify a set of active events at any given time instance. An event is considered active as long as the change in brightness that triggered it is visible by its respective pixel. Using event lifetimes to identify the set of active events implicitly generates sharp gradient images at any point in time, thus effectively functioning as an edge detector.

Event-based vision sensors have also been proven successful in depth estimation via stereo matching. Some event-driven methods rely on standard stereo matching algorithms by generating artificial event frames. In [12, 13] authors used a fixed event accumulation interval to generate artificial frames, while [14] generated different frames based on event timestamps. A similar approach is to match events by comparing their local context by forming descriptors [15]. However, opposed to choosing a fixed time event accumulation interval, enhancing the event stream with event lifetimes yields more accurate disparity maps [16]. Therein the authors proposed to run stereo lifetime estimation separately for each sensor and subsequently estimate the disparity, which requires either using multiple threads or introduces latency caused by lifetime computation. Since the two sensors are mostly observing the same scene, they will also emit events triggered by the same brightness changes, thus offering the possibility to exploit the fact that the corresponding events should also have the same lifetimes.

In this paper we propose a novel method for event lifetime and disparity estimation in stereo event-cameras. Since events in a stereo setup are caused by the same change in brightness, we couple stereo event disparity and lifetime estimation in order to calculate them efficiently in a single shot. The resulting disparity estimation method is computationally more efficient and accurate in comparison to the method that decouples event lifetime and disparity estimation [16]. The proposed approach is validated on real-world data through multiple stereo event-camera experiments. The algorithm runs in a single thread, making it appropriate for energy-efficient applications in autonomous systems and mobile robotics.

The rest of the paper is organized as follows. In Section II we describe the event lifetime estimation method proposed in [10] based on optical flow and describe the local plane fitting algorithm. Our novel method for stereo event lifetime estimation together with disparity calculation is presented in Section III. Finally, we present the experimental results in Section IV and conclude with final remarks in Section V.

II Optical Flow Based Event Lifetime Estimation

Event lifetime estimation was proposed in [10] as a way to identify the set of active events at any given time instance, resulting in frames of accumulated events with clear structure and filtered noise that can be used as an input to standard computer vision algorithms for further processing. Herein we provide its description for completeness as it is an important part of the proposed stereo event lifetime and disparity estimation method.

II-A Method Overview

The surface of active events (SAE) is a three-dimensional structure defined in the spatio-temporal domain where coordinates of each pixel of the sensor are mapped to the timestamp of the last event that occurred on that position (Figure 2). Formally, SAE can be described using a function $S:\mathbb{R}^{2}\rightarrow\mathbb{R},S(\mathbf{p})=t,$ where $\mathbf{p}=(x,y)^{\top}$ is the pixel position and $t$ is the corresponding timestamp.

The first order Taylor expansion gives the planar approximation of the SAE at a pixel location $\mathbf{p}$:

[TABLE]

where $S_{x}$ and $S_{y}$ denote the first partial derivatives, $\frac{\partial S}{\partial x}$ and $\frac{\partial S}{\partial y}$, respectively. As proposed in [10], lifetime $\tau$ of the event at the point $(\mathbf{p},t)$ is defined as the first order approximation of the maximum temporal increment of $S$ for a displacement of one pixel. Therefore, the lifetime $\tau$ corresponds to the maximum amount of time before the brightness change at the current pixel will trigger a new event in a neighboring pixel. In other words, only one event caused by a single change in brightness should be considered active at some point in time; this can formally be expressed as:

[TABLE]

where $\Delta t=S(\mathbf{p}+\Delta\mathbf{p})-S(\mathbf{p})$. Substitution further yields $\Delta t=\left(S_{x}(\mathbf{p}),S_{y}(\mathbf{p})\right)\Delta\mathbf{p}=\nabla S(\mathbf{p})\Delta\mathbf{p}$. Finally, lifetime $\tau$ can be expressed as:

[TABLE]

Time $t$ being a naturally increasing function, $S(\mathbf{p})$ is a monotonically increasing function of $\mathbf{p}$, thus having nonzero gradient at any point. This allows us to use the inverse function theorem around the location $\mathbf{p}$, as shown in [11]:

[TABLE]

The gradient can thus be expressed as:

[TABLE]

showing its relation to the optical flow velocities, $v_{x}$ and $v_{y}$. Assumption of constant velocities is equivalent to the local planar approximation of the SAE. The maximum temporal increment $\Delta t$ occurs if the displacement $\Delta\mathbf{p}$ matches the unit vector in the direction of gradient $\Delta\mathbf{p}=\nabla S(\mathbf{p})/\|\nabla S(\mathbf{p})\|$. Therefore,

[TABLE]

The normal of the tangent plane to the SAE at the point $\mathbf{p}$ is given by

[TABLE]

Furthermore, combination of (5) and (7) yields $\mathbf{n}(\mathbf{p})~{}\propto~{}(-v_{x}^{-1},-v_{y}^{-1},1)^{\top}$, and naming the normal coordinates as $\mathbf{n}(\mathbf{p})=(n_{1},n_{2},n_{3})^{\top}$ implies $-v_{x}^{-1}=\frac{n_{1}}{n_{3}}$ and $-v_{y}^{-1}=\frac{n_{2}}{n_{3}}$. Components of the SAE tangent plane normal $(n_{1},n_{2},n_{3})^{\top}$ are extracted by fitting a plane to the data using an optimization technique (see Section II-B) that finally enables computation of the lifetime $\tau(\mathbf{p})$ as:

[TABLE]

II-B Local plane-fitting Algorithm

A plane-fitting algorithm is needed to calculate the parameters of the SAE tangent plane to further estimate the lifetime of the given event at the location $\mathbf{p}$. In [11], it was proposed to use an $N\times N\times 2\Delta t$ window centered around the current event to estimate the local plane, including the events from the future. To avoid the introduced $\Delta t$ latency, we use only the past events in the $N\times N$ spatial window based on the local smoothness assumption, as suggested by [10]. Even though therein they completely eliminate the $\Delta t$ parameter and the need to tune it, we use $\Delta t=100\,\mathrm{ms}$ as a time frame in which the past events in the spatial window are considered to ensure that the events that are too old and unrelated to recent events do not affect the estimated tangent plane negatively.

Fitting the plane using the RANSAC algorithm suppresses noise events introduced by the sensor and ensures robustness. First, a plane is fitted to a set of hypothetical inliers, which include the current event and two random past events in the spatio-temporal window. All other past events are then tested against the fitted plane. They are considered as inliers if their distance to the plane is below the threshold $\mu$. If less than $m$ inliers are found, the process is repeated from the beginning with a new hypothetical set of inliers. The event is declared as noise if the condition for the minimum number of inliers is not met in the predefined maximum number of iterations. Otherwise, the plane is reestimated by fitting to all the inliers using the least squares minimization:

[TABLE]

where $\mathbf{A}$ is the $n\times 3$ matrix of the $n$ SAE points $(x_{i},y_{i},t_{i})$ identified as inliers, and $\mathbf{b}=(1\cdots 1)^{\top}$.

Assumption of locally constant velocity allows for prediction of the future events timestamp, as suggested in [10], wherein authors proposed to use the predicted timestamp to compute the regularization factor depending on the absolute difference of the predicted timestamp $\hat{t_{i}}$ and the actual timestamp $t_{i}$ of the incoming events:

[TABLE]

That method performs plane fitting based on the RANSAC algorithm for each event, even if the predicted plane normal is considered very reliable. The more the predicted and actual timestamp are close, the more the plane normal obtained from the previous events is considered reliable. Our method, however, avoids least squares optimization for the events whose timestamps closely match the predicted ones. In that case, we assume that the normal of the current event tangent plane is equal to the predicted normal, saving computational time while retaining accuracy.

III Stereo Event Lifetime and Disparity Estimation

In this section we propose a method for estimating the lifetimes of events from two dynamic vision sensors in stereo settings. Naturally, streams of events emitted by the two sensors are very similar since they mostly observe the same scene. For the same brightness change, both sensors are triggered and emit an event. Therefore, it makes sense to calculate the lifetime of those events only once per brightness change and associate it with the other pertaining event as well.

As Algorithm 1 shows, for the current event, we attempt to find its counterpart that was caused by the same brightness gradient in the SAE of the other sensor, which is equivalent to finding its disparity. We calculate the disparity for each event asynchronously, at the time of its arrival. Even though our approach is event-driven, the method is similar to the standard stereo matching methods in a sense that it operates on frames of accumulated events. We use an event feature descriptor introduced by [16] that supports event-driven stereo matching based on distance transform [17] which describes the context of a pixel in binary images by calculating the distance from the pixel to the nearest active pixel. To apply it to the problem of an event descriptor, coordinates of active events at the time of the current event’s arrival are considered as active pixels, and others as inactive. It is important to note that we make an assumption that the current event is certainly active at its starting point. Also, even if some future events (relative to the current event) from the other sensor have already been processed, we cannot consider them as active since their starting timestamp is later than the timestamp of the current event. As a starting step of descriptor computation, we calculate the vector pointing to the nearest active event for every pixel in a $N\times N$ window centered at the current event’s position $\mathbf{p}$, separately for events of positive and negative polarity. Then, we classify the obtained vectors by orientation. We assign each vector to one of the 12 bins of the histogram, each bin covering a $30^{\circ}$ interval. The final feature descriptor $\mathbf{D}$ of an event is then defined as a vector of $24$ values; $12$ bins of the histogram for the positive, and likewise for the negative events. Although [16] proposed a descriptor invariant to scale and rotation, we do not implement those features since they are not needed for stereo matching on rectified images.

Disparity map is computed in three standard stereo matching steps: i) matching cost calculation, ii) cost aggregation, and iii) disparity estimation. Matching cost of an event in location $\mathbf{p}$ for disparity $d$ is calculated by summing absolute differences between the corresponding event descriptors $\mathbf{D}^{L}$ and $\mathbf{D}^{R}$, from left and right cameras respectively, within a rectangular window $W_{p}$ centered at $\mathbf{p}$:

[TABLE]

Cost aggregation is performed as suggested in [16], in a fixed region $A$ around the location $\mathbf{p}$:

[TABLE]

Finally, disparity estimation is done through the winner–takes–all approach, by finding the disparity $d$ with the minimal aggregated cost:

[TABLE]

In case of successful disparity estimation, we determine the current event’s lifetime as a median of lifetimes of the events in the matched spatial window $M$:

[TABLE]

Other computationally light operations for calculating the lifetime using the matched window can be considered as well, e.g., averaging, maximum, minimum, or similar. We chose median because it is least prone to outliers. In case of failed disparity estimation, we estimate the lifetime for the event using the aforementioned method.

There are two main advantages of this method opposed to calculating the lifetimes of the events from the two sensors separately. First, this method is less computationally complex compared to performing the RANSAC algorithm for plane fitting for each event, as will demonstrated in the Section IV. Furthermore, due to noise and low resolution of the sensor, it is not possible to determine the lifetime of each event using the plane fitting method, so they are falsely declared as noise. In that case, however, it might be possible to determine the lifetime of the corresponding event in the other sensor’s SAE, allowing to retain the event that would otherwise be falsely proclaimed as noise, thus generating a more finely detailed image with sharper gradients. The method is suitable for execution in a single thread, making it convenient to implement in systems with limited computational resources. It is also important to note that our method offers high level of abstraction, i.e., it is possible to use any suitable lifetime estimation and disparity estimation method other than the proposed ones depending on the application.

IV Experimental Results

We evaluate the proposed method on multiple real-world stereo experiments. Our algorithm’s output consists of the disparity map and the image of accumulated events for each of the sensor views. Since there is no metric defined to evaluate the accuracy of lifetime estimation, we compare the outputs qualitatively. We compare the proposed method in terms of both execution time and accuracy with the method proposed in [16] that first executes lifetime estimation separately for both sensors and then follows up with disparity estimation (referred to as the decoupled method). We also compare our output with the method that uses a fixed event accumulation interval, i.e., does not estimate event lifetime. As future work, the proposed method can be evaluated on the complex real-world stereo dynamic vision sensor dataset [18].

IV-A Experimental Setup

Experiments were conducted with two DAVIS240 sensors mounted on a stereo rig with a baseline of 10 cm as shown in Fig. 1(d). Resolution of the sensors is $240\times 180$ pixels. Timestamps of the events were synchronized on the hardware level. Event data was undistorted and stereo rectified using the Kalibr toolbox [19]. We used the DAVIS240 standard camera frames for calibration, but our proposed method operates on events only. Experiments were run on a personal computer with Intel Core i7-7700HQ CPU at 2.80GHz on a single core.

IV-B Implementation Details

The implementation of the proposed method is done in C++ in ROS environment [20]. Stream of events from DAVIS240 is fetched in grouped sequences through a ROS topic. A single thread is subscribed to messages from both left and right sensor. Random scheduling of the incoming ROS messages does not guarantee that the events from one sensor will always be available before the corresponding events from the other sensor, making it impossible to set one sensor as primary, and other as secondary. However, for the current set of events, it is possible to determine whether the corresponding events from the other sensor have already been processed and then decide whether to perform lifetime or disparity estimation first.

IV-C Accuracy Improvement

DVS yield sparse event data in case of a motion occurring within the area of very similar brightness intensity, which the sensor is unable to discern, resulting in poor results of event lifetime and disparity estimation. Figure 4 shows events obtained by recording a moving line from a computer screen. Due to sparse event data caused by screen flickering, the decoupled lifetime estimation method was not able to compute the lifetime of some events from the right sensor, thus proclaiming many events as noise. However, as the two sensors are naturally not completely identical in terms of sensitivity to brightness changes, the method was more successful in computing the lifetimes of events from the left sensor. Since the proposed stereo event lifetime computation method also uses events from the left sensor, it was able to compute lifetimes for more events from the right sensor compared to the decoupled lifetime computation, thus proving better accuracy in terms of noise detection compared to the decoupled method. Experiment shown in Figure 6 confirms that our method also matches the decoupled method in terms of accuracy in case of dense event data. Figures 1 and 5 show the experiments performed on real-world data comparing the proposed method to using a fixed event accumulation time interval of 10 ms, which yields very sparse gradient image lacking structure.

IV-D Execution Time Improvement

Since the plane-fitting algorithm used to estimate the event lifetime is computationally complex, our method aims to reduce the number of times it needs to be executed as much as possible. As a first way of reducing execution time, we propose to use the previously calculated and predicted lifetime, if the prediction was reliable, without fitting a new plane. Furthermore, if the predicted lifetime is not available, our method executes plane-fitting only once for the two corresponding events caused by one brightness change, effectively cutting the execution time in half. The amount of improvement depends on the number of events whose disparity is successfully estimated, in which case we use the efficient method of calculating the current event lifetime by finding the median of lifetimes of the events in the matched window. Experiments show that the execution time is reduced approximately by 50% (see Table I).

V Conclusion

In this paper, we have presented a novel method for stereo lifetime and disparity estimation for event-based cameras. Enhancing the events with their lifetime prior to disparity estimation makes the disparity maps representative of the scene structure invariant to the amount of motion of the sensor or the scene. Since one brightness change triggers an event in both of the sensors, we have proposed to calculate the lifetime for only one of those events, and associate it to the other one via stereo matching, while simultaneously determining its disparity. The proposed method is twice as fast as the method calculating the event lifetimes for each sensor independently, while it also provides disparity maps with more accurate structure, as we have demonstrated through several real-world experiments.