Towards Anytime Optical Flow Estimation with Event Cameras

TL;DR

This paper introduces a new method for high-speed optical flow estimation using event cameras, enabling real-time motion tracking with minimal latency.

Contribution

The paper introduces EVA-Flow, a novel network for event-based optical flow estimation with a low-latency event representation and an unsupervised loss function.

Findings

EVA-Flow achieves super-low-latency (5 ms) and time-dense motion estimation (200 Hz).

The proposed method demonstrates strong generalization across multiple datasets.

The rectified flow warp loss enables unsupervised assessment of intermediate optical flow.

Abstract

Event cameras respond to changes in log-brightness at the millisecond level, making them ideal for optical flow estimation. However, existing datasets from event cameras provide only low-frame-rate ground truth for optical flow, limiting the research potential of event-driven optical flow. To address this challenge, we introduce a low-latency event representation, unified voxel grid (UVG), and propose EVA-Flow, an EVent-based Anytime Flow estimation network to produce high-frame-rate event optical flow with only low-frame-rate optical flow ground truth for supervision. Furthermore, we propose rectified flow warp loss (RFWL) for the unsupervised assessment of intermediate optical flow. A comprehensive variety of experiments on MVSEC, DESC, and our EVA-FlowSet demonstrates that EVA-Flow achieves competitive performance, super-low-latency (5 ms), time-dense motion estimation (200 Hz), and strong generalization.

Figures

Full text

1. Introduction

Optical flow estimation is a fundamental task in computer vision that aims to calculate the motion vector of each pixel in two consecutive frames of images. The event camera [1] is a new type of bio-inspired sensor that only responds to changes in the brightness of the environment. Compared with the traditional frame-based camera, which integrates the brightness at a certain time interval (exposure time) and outputs an image, the event camera has no exposure time and responds once the log value of the pixel intensity changes beyond a certain threshold, and the response is microsecond-level and asynchronous. As optical flow estimation aims to estimate the motion of a scene which aligns with the event camera’s ability to detect changes, it makes the event camera a suitable choice for optical flow estimation [2,3]. Furthermore, event cameras offer distinct advantages for optical flow estimation, such as high temporal resolution and high dynamic range (>120 dB compared to about 60 dB of traditional frame-based cameras [1]). The high dynamic range capability of event cameras can solve the problem of under- or over-exposed images produced by traditional frame-based cameras in poor illumination conditions (such as at night) or high-dynamic scenes (such as cars entering or exiting tunnels) [4,5,6], which can cause inaccurate optical flow estimation in subsequent processing. The high temporal resolution of event cameras eliminates the problem of motion blur even in high-speed motion scenes. Additionally, the high temporal resolution output of event cameras provides hardware support for super high-frame-rate optical flow estimation.

However, most current event-based optical flow estimation methods fail to fully exploit the high temporal resolution and low-latency characteristics of events, resulting in low frame rates for the estimated optical flow. They typically transform the event stream over a time interval into a tensor represented as voxels [2], which are then fed into a frame-based optical flow estimation network similar to image-based approaches [2,3,7]. Moreover, the output frame rate is limited by the underlying dataset. The two most commonly used benchmarks for event-based optical flow estimation, MVSEC [8] and DSEC [9] have low frame rates for their ground-truth optical flow, with DSEC having a frame rate of 10 Hz and MVSEC having a frame rate of 20 Hz. While significant progress has been witnessed in the field, the limited temporal resolution greatly restricts the potential of event-based optical flow estimation in real-world applications.

We present EVA-Flow, a deep architecture designed for EVent-based Anytime optical Flow estimation. EVA-Flow delivers numerous benefits, such as low latency, high frame rate, and high accuracy. Figure 1a–c illustrates the differences between previous event-based optical flow estimation and our proposed approach toward anytime optical flow estimation. While other methods are limited by the dataset’s frame rate, this framework achieves ultra-low latency and offers optical flow estimation at a frequency 20 times higher. Figure 1d illustrates the discrepancy between the predictions of this model and E-RAFT [3] using a sample from the test sequence of the DSEC dataset. The scene depicts a car making a turn inside a tunnel. Our EVA-Flow offers continuous trajectory tracking within a 100 ms time range of the input sample. The transition from yellow to red indicates varying point positions at different times in the trajectory of the car turning inside the tunnel. In contrast, E-RAFT is restricted to generating a single optical flow.

To achieve high-frame-rate event output, we first propose the unified voxel grid (UVG), which generates a bin of UVG representation as soon as the events in a small time interval are ready. Our UVG representation achieves a higher frame rate up to a factor of N and a lower latency of $[eqn]$ compared to the voxel grid representation [7]. N is a hyperparameter that can be adjusted to meet the specific requirements of the task. These bins are then sequentially fed into our EVA-Flow, which consists of a multi-scale encoder and our stacked spatiotemporal motion recurrent (SMR) module. SMR is proposed to predict temporally dense optical flow and enhances the accuracy via spatial-temporal motion refinement. Notably, the architecture is specially designed for extreme low-latency optical flow estimation: as each event bin arrives, it is immediately processed through the SMR module, yielding the optical flow result for the current time instance. This architectural design eliminates the need to wait for all bins of the entire voxel grid to arrive in order to achieve low-latency and high-frame-rate optical flow output.

Furthermore, we introduce rectified flow warp loss (RFWL), a novel method for the precise, unsupervised evaluation of event-based optical flow precision, and apply it to assess the accuracy of our time-dense optical flow estimations.

By comparing our EVA-Flow with other event-based optical flow estimation methods on widely recognized public benchmarks, including the DSEC [9] and MVSEC [8] datasets, we have established that our approach achieves performance on par with leading methods, offering advantages such as reduced latency (5 ms) and higher frame rates (200 Hz). Additionally, the evaluation of time-dense optical flow on DSEC, MVSEC, and EVA-FlowSet underscores the reliability of our framework’s time-dense optical flow, even under the constraint of being solely supervised with low-frame-rate optical flow ground truth during training. Furthermore, the Zero-Shot results derived from both the MVSEC dataset and our EVA-FlowSet clearly illustrate the strong generalization capabilities of our method.

In summary, our key contributions are threefold:

• EVA-Flow Framework: An EVent-based Anytime optical Flow estimation framework that achieves breakthroughs in latency and temporal resolution. The architecture features two novel components: (1) unified voxel grid (UVG) representation enabling ultra-low latency data encoding, and (2) A time-dense feature warping mechanism where shared-weight SMR modules propagate flow predictions across temporal scales. This structural design fundamentally enables single-supervision learning—only final outputs require low-frame-rate supervision while implicitly regularizing intermediate time steps through feature warping recursion.
• Rectified Flow Warp Loss (RFWL): A new unsupervised metric specifically designed for evaluating event-based optical flow precision. This self-consistent measurement provides theoretical guarantees for temporal continuity validation of high-frequency optical flow estimation.
• Systematic Validation: Comprehensive benchmarking demonstrates competitive accuracy, super-low latency, time-dense motion estimation and strong generalization capability. Quantitative analyses using RFWL further confirm the reliability of our continuous-time motion estimation.

2. Related Work

2.1. Optical Flow Estimation

Recently, the field of optical flow estimation has witnessed remarkable advancements owing to the progress of deep learning. FlowNet [10] firstly introduces an end-to-end network with a U-Net architecture to estimate optical flow directly from image frames and many subsequent works [11,12,13] follow this architecture. Later, the works of [14,15] utilize a warp mechanism to conduct a multi-scale refinement, achieving higher accuracy with less computational costs. RAFT [16] proposes a recurrent optical flow network that uses all-pair correlations and GRU iterations to produce high-precision flow results through iterative refinement, which achieves a significant improvement in accuracy. Most of the later optical flow studies are, in essence, improvements upon RAFT. CSFlow [17] introduces a decoupled strip correlation layer to enhance the capacity to encode global context. FlowFormer [18] explores the self-attention mechanism in recurrent flow networks to achieve further flow accuracy boosts, albeit with larger parameter usage. In summary, the use of warp mechanism, correlation volume, and RNN with iterative refinement has brought about significant advancements in the field of optical flow, allowing for improved accuracy and more efficient computation.

2.2. Event-Based Optical Flow

Event optical flow estimation can be categorized into two methodologies: model-based and learning-based approaches. The model-based approach to event optical flow estimation encompasses two distinct strategies. The first strategy involves fitting a local spatiotemporal plane to the event point cloud and subsequently leveraging the derived plane fitting parameters to ascertain the optical flow. This methodology has been extensively explored in the literature, as evidenced by the works of Benosman et al. [19], Mueggler et al. [20], and Low et al. [21]. The second strategy is predicated on the contrast maximization framework, as articulated by Gallego et al. [22]. This approach entails the computation of optical flow by optimizing the contrast of a motion-compensated event frame, a process that has been further elucidated by Shiba et al. [23,24]. However, model-based event optical flow algorithms [19,20,21,25] generally require denoising algorithms like [26] for event preprocessing to improve the accuracy.

Thanks to the availability of large-scale event optical flow datasets [8,9], learning-based event optical flow estimation algorithms [2,3,7] have achieved superior accuracy compared to model-based algorithms, which is also more robust to event noise.

EV-FlowNet [7] firstly utilizes bilinear interpolation on discrete events in both spatial and temporal dimensions. This transformation converts sparse and continuous events into a voxel-based tensor called a voxel grid. Subsequently, the voxel grid is input to FlowNet [10] for estimating event optical flow. This paradigm has been adopted by numerous subsequent works [3,27,28,29,30]. E-RAFT [3], leveraging the sophisticated RAFT [16] optical flow estimation network, has markedly improved the precision of event-based optical flow estimation and has long been regarded as the state-of-the-art method on the DSEC-Flow dataset [9]. Recently, E-Flowformer [31] leveraged transformers [32] to enhance network performance, achieving higher optical flow estimation accuracy with an expanded synthetic dataset, BlinkFlow [31]. TMA [33] and IDNet [34] have leveraged the abundant motion data recorded by event cameras during intermediate time frames to improve optical flow estimation, further enhancing the precision of event-based optical flow. Although these methods achieve high-precision optical flow estimation, they fail to capitalize fully on the low latency and swift response capabilities of event cameras. Constrained by the dataset’s frame rate, they are restricted to generating optical flow estimates at similarly low frame rates, specifically 10 Hz on the DSEC-Flow dataset.

Event cameras offer brightness variation data with super-high temporal resolution; however, there is a paucity of methods for high-frame-rate event optical flow estimation. Due to the lack of time-continuous optical flow ground truth, several studies [35,36] have employed unsupervised contrast-maximization-based losses [23], resulting in lower accuracy for these methods. These self-supervised methods typically perform well on small optical flow datasets (MVSEC) but exhibit poor accuracy on datasets with large displacements (DSEC-Flow). Wan et al. [37] develop an event-image fusion approach that combines events and images for temporal feature learning. While effective for continuous scene understanding, this method requires additional image inputs and recomputing event embeddings for each step during feature fusion. Ponghiran et al. [38] utilized recurrent neural networks to estimate time-dense optical flow from events, but their method required sequential training and was dependent on high-frame-rate ground-truth optical flow for supervision. On the DSEC-Flow dataset, they employed interpolation to achieve high-frame-rate optical flow ground truth; however, due to the large displacements and the nonlinear nature of many motions within this dataset, their accuracy on DSEC-Flow was comparatively poor. Gehrig et al. [39] introduced a continuous-time event optical flow algorithm that leverages dense temporal correlation to estimate pixel motion trajectory parameters (Bézier curves) across an entire time interval, facilitating continuous optical flow estimation. However, this method estimates the pixel motion trajectory for a specified time period only after the accumulation of all events (100 ms on DSEC-Flow). In contrast, our novel approach, EVA-Flow, attains high accuracy, minimal data latency, a very high prediction frequency, and superior model generalization through comprehensive end-to-end training, necessitating only low-frame-rate optical flow ground truth for supervision.

3. Method

In this paper, we propose an EVent-based Anytime optical Flow estimation (EVA-Flow) framework, as shown in Figure 2, which achieves high accuracy, time-dense flow estimation, and low-latency input and output with supervision solely dependent on low-frame-rate optical flow. We first put forward a unified voxel grid (UVG) representation in Section 3.1, which generates event representations with low latency. These UVG bins are sequentially fed into EVA-Flow, where high-frame-rate optical flow estimations are generated with low latency using a Spatiotemporal Motion Recurrent (SMR) module, which is detailed in Section 3.2. Then, the details of the loss function and the supervision regime are described in Section 3.3. Finally, we propose a new criterion to evaluate the reliability of intermediate high-frame-rate optical flow in Section 3.4.

3.1. Event Representation: Unified Voxel Grid

The event representation is crucial for the event-based model [40,41]. A voxel grid [7] is a commonly used event representation that utilizes the temporal dimension of event data, and its effectiveness for optical flow estimation tasks has been verified in many studies [3,7,34,39]. The principle of a voxel grid is to discretize the spatially sparse and temporally continuous information of events by averaging over time and then utilize bilinear interpolation in both spatial and temporal dimensions to obtain a tensor representation of the event data in a voxel form. The tensor has a dimension of B × H × W, where B represents the number of time steps, and its size determines the sampling accuracy of the event data in the temporal dimension. Each channel of the voxel grid can be viewed as an event representation at a specific time. Given a set of events $[eqn]$ and B bins to discretize the time dimension, the definition of voxel grid is depicted as follows:

$$[eqn]$$

where $[eqn]$ denotes the bilinear sampling kernel.

Continuous prediction of high-frame-rate optical flow necessitates corresponding high-frequency event representations. A critical limitation of the standard voxel grid, illustrated in Figure 3, is that generating the complete grid requires waiting for all events within the entire time interval T, making it inherently batch-based and unsuitable for low-latency applications. Furthermore, the effective temporal support for the first ( $[eqn]$ ) and last ( $[eqn]$ ) bins is often halved, leading to representational inconsistency.

To address these limitations, particularly the latency issue, we propose the unified voxel grid (UVG). UVG enables sequential, low-latency event representation. Instead of waiting for the full duration T, UVG allows for the computation of an event representation centered around a specific time $[eqn]$ using only events within a localized temporal window. We achieve this by defining each bin $[eqn]$ consistently with a fixed temporal support $[eqn]$ , centered at its timestamp $[eqn]$ . The representation for a bin is calculated using events $[eqn]$ falling within $[eqn]$ :

$$[eqn]$$

This design makes it possible to obtain the current representation of time for each $[eqn]$ (5 ms for 21 channels with unified voxel grid on DESC), while the voxel grid must wait for the entire period of T (100 ms on DSEC) to generate a complete event representation. This low-latency input can be coupled with our low-latency time-dense optical flow estimation architecture for tracking the optical flow at each $[eqn]$ time step as it arrives which is essential for high-frame-rate optical flow estimation.

3.2. Event Anytime Flow Estimation Framework

In this subsection, we propose the EVA-Flow framework, which predicts time-dense optical flow from serial event bins with low latency. The architecture of EVA-Flow is illustrated in Figure 2. The raw events are converted to UVG and then sequentially fed into the encoder, bin by bin, to generate a four-level feature pyramid ( $[eqn]$ ). The feature pyramid is fed into our stacked spatiotemporal motion recurrent (SMR) modules to update the optical flow in both temporal and spatial dimensions while outputting the refined optical flow results from $[eqn]$ to the current bin.

Serialized low-latency input and output. The UVG inputs are sequentially fed into the network, which contributes to the low latency at the data entry level. During the inference phase, after entering an event bin (excluding the first bin used for initialization), the current optical flow can be directly predicted using the encoder and the stacked SMR module. During the training phase, to facilitate end-to-end training, we follow the frame rate of the dataset and input a full period of UVG at once as a training sample. Following the UVG input of $[eqn]$ , the UVGs of different bins (i.e., with different channels) are converted to the batch dimension and concatenated (resulting in a dimension of $[eqn]$ ) before being passed through the encoder. Once the feature maps are obtained, the corresponding feature maps of different bins are sequentially input into the stacked SMR module, which outputs time-dense optical flow ( $[eqn]$ ).

Spatiotemporal Motion Recurrent (SMR) module. To achieve accurate and time-dense optical flow estimation, we propose the spatiotemporal motion recurrent (SMR) module, illustrated in Figure 2. The core idea of SMR is to employ a hierarchically stacked structure for coarse-to-fine spatial refinement combined with ConvGRU-based temporal recurrence. This design inherently promotes stable convergence and high accuracy. Specifically, the coarse-to-fine spatial processing provides robust initial estimates and guides the refinement process across scales, simplifying the overall estimation task. Concurrently, the iterative residual updates within each level encourage gradual convergence by successively correcting the flow estimate. Moreover, the ConvGRU architecture [42], with its gating mechanisms, ensures stable learning and propagation of temporal motion information, effectively capturing motion dynamics over time. The process within each SMR unit is governed by the following:

$$[eqn]$$

where the superscript i denotes the feature pyramid level (from coarse to fine), and the subscript j indicates the temporal processing step.

Structurally, the SMR module operates along two primary dimensions:

Vertical Dimension (Spatial Coarse-to-Fine Refinement): At any given time step j, the SMR modules are stacked vertically across different spatial resolutions (pyramid levels i). Information propagates from coarser levels ( $[eqn]$ ) to finer levels (i). Specifically, the input features $[eqn]$ at level i are warped using the upsampled flow estimate $[eqn]$ from the coarser level. This aligned feature $[eqn]$ , along with the upsampled flow and the upsampled hidden state $[eqn]$ , are fed into the ConvGRU unit. The resulting hidden state $[eqn]$ is then processed by a flow head (implemented with two convolutional layers) to predict a residual flow, which updates the coarser flow estimate $[eqn]$ to yield the refined flow $[eqn]$ at the current level. This hierarchical refinement progressively enhances spatial accuracy. Importantly, the optical flow $[eqn]$ for the current time step j is predicted using only the features $[eqn]$ and state information available up to that time step. This enables low-latency flow estimation, as the output for time j can be generated as soon as the corresponding event features are processed.

Horizontal Dimension (Temporal Recurrence): Within each spatial level i, the SMR module processes information sequentially across time steps j. The ConvGRU [42] plays a crucial role here, utilizing its internal hidden state ( $[eqn]$ from the previous time step) to update the current hidden state $[eqn]$ . This recurrence mechanism maintains temporal consistency and allows the module to model motion dynamics over time. SMR modules within the same spatial level i (horizontally) share weights across different time steps j, promoting efficient learning of temporal patterns. Conversely, SMR modules at different spatial levels i (vertically) use distinct weights to accommodate varying feature resolutions and channel dimensions.

Time-dense feature warping. Unlike other methods [33,34] that rely on the assumption of linear motion between two ground-truth flows, our approach can handle non-linear motion throughout the entire UVG period. As depicted in Figure 2, during each warping iteration, the dense temporal optical flow from the preceding level is employed to warp the features of the current level at each time step. This structure guarantees consistent prediction patterns of optical flow for each resolution level and time step (SMR modules at the same level are using shared weights), implicitly providing SMR modules with the ability to predict optical flow at the current time step. By implementing this approach, we only need to supervise the final output $[eqn]$ for implicit monitoring of optical flow at intermediate time steps.

3.3. Supervision

Following E-RAFT [3], we also used L1 loss as the loss function. Specifically, we calculate the L1 distance between the ground-truth optical flow and the last optical flow prediction $[eqn]$ of our model as the final loss function. The loss function is defined as follows:

$$[eqn]$$

Due to network structure design (see explanation in Section 3.2 in Time-dense feature warping), we do not need to supervise the optical flow prediction at intermediate time steps.

3.4. Rectified Flow Warp Loss

Motion compensation, as described in [22], entails aligning all events to a reference time by utilizing the estimated event optical flow. Specifically, for each event, the flow value at the event’s position and the time difference from the event’s occurrence to the reference time are used to determine the event’s position at the reference time (see Equation (5)). Henceforth, we refer to the motion-compensated event count image as the Image of Warped Events (IWE).

Given events $[eqn]$ , per-pixel flow estimation $[eqn]$ and a reference time $[eqn]$ , the IWE is defined as follows:

$$[eqn]$$

Accurate flow estimation ensures that events generated by the same edge are aligned to the same pixel location, resulting in higher contrast in the IWE than in the original event frame. Stoffregen et al. [43] utilized this principle to propose flow warp loss (FWL), which can assess the accuracy of event optical flow estimation in an unsupervised way.

$$[eqn]$$

In theory, a more accurate estimation of optical flow results in a clearer, higher contrast, and larger variance for the image after motion compensation. This is because events occurring at the same location in the scene are aligned to the same point. Therefore, a more accurate prediction of optical flow leads to a larger FWL. In general, the image tends to become sharper after motion compensation, indicating an FWL value greater than 1. However, in practice, we find that the FWL can be less than 1, yet the IWE remains visibly sharper than the original, as depicted in Figure 4. This phenomenon, where FWL of IWE is less than 1, has also been observed in the study by Shiba et al. [23]. Upon analysis, we determined that motion compensation has caused some events to be warped outside the image boundaries, leading to a reduced event count in the IWE compared to the original event frame. Thus, we propose rectified flow warp loss (RFWL), which normalizes image brightness based on the total event count (i.e., the sum of pixel values) in the event frames before and after motion compensation. The formulation is given as follows:

$$[eqn]$$

As depicted in Figure 4, the FWL fails to accurately assess the precision of optical flow estimation. The IWEs in columns 1, 2, and 4 display higher sharpness than the original event frames, yet the corresponding FWL values are less than 1. Notably, the IWE in column 3, which exhibits a markedly sharper resolution than the original, has an FWL value of $[eqn]$ . In contrast, our proposed rectified flow warp loss can accurately reflect the relative sharpness of IWEs compared to the original event frames.

4. Experimental Results

Our model underwent evaluation on the DSEC [9], MVSEC [2], and self-collected EVA-FlowSet. Section 4.1 offers an introduction to the aforementioned datasets. Section 4.2 presents the implementation details of our approach. Section 4.3 demonstrates the evaluation of our model using the regular flow evaluation prototype. Section 4.4 reveals the evaluation results of our model for time-dense optical flow. Section 4.5 encompasses ablation studies.

4.1. Datasets

DSEC. The DSEC dataset, introduced by Gehrig et al. [9], consists of 24 sequences captured in real-world outdoor driving scenes. The dataset includes a total of 7800 training samples and 2100 testing samples, captured at a resolution of 640 × 480. It covers both daytime and nighttime scenarios, encompassing small and large optical flows. Moreover, the DSEC dataset offers high-precision sparse ground-truth values for optical flow. As the dataset does not include an official validation set, we divided the training dataset randomly using a fixed seed. To create the training and validation sets, we allocated them in a 4:1 ratio. Consequently, only $[eqn]$ of the training data was utilized for training our model.

MVSEC. MVSEC [2] is a classic real-world event optical flow dataset that comprises sequences from various indoor and outdoor scenes with a resolution of 346 × 260. The event density of the dataset is relatively sparse, and the optical flow ground-truth distribution mostly highlights small flows. As a result, two different time intervals, i.e.,  $[eqn]$ = 1 and $[eqn]$  = 4, are used for accuracy evaluation. $[eqn]$ = 1 means that adjacent grayscale frames are used as a sample, whereas $[eqn]$ = 4 represents a four-fold increase in the grayscale frame interval. We follow the E-RAFT convention [3], conduct training solely on the outdoor day 2 sequence, and evaluate on 800 samples data from outdoor day 1.

EVA-FlowSet. We collect and present EVA-FlowSet, a real-world dataset to assess the generalization and time-dense optical flow of our model. For data collection, we utilized a Davis-346 camera (manufacturer: iniVation AG, Zurich, Switzerland),  which is the same type used in the MVSEC dataset. The dataset consists of four sequences in total, with two sequences depicting fast-moving scenes and the other two showcasing regular motion scenes. During the data-collection process, the camera remains stationary, capturing the movement of a checkerboard calibration board as it travels along a curved trajectory at different predetermined speeds.

4.2. Implementation Details

For the DSEC dataset [9], we use an Adam optimizer with a batch size of 6 and a learning rate of 5 × $[eqn]$ to train for 100k iterations on the training set we divided, which accounts for $[eqn]$ of the total dataset. Then, we reduce the learning rate by a factor of 10 and continue training for another 10k iterations. During the model training on the DSEC dataset, two online data augmentation methods, namely random cropping and horizontal flipping, were employed. Random cropping was performed with a size of 288 × 384, while horizontal flipping was applied with a probability of $[eqn]$ .

For the MVSEC dataset [2], we employ an Adam optimizer with a batch size of 3 and a learning rate of 1 × $[eqn]$ to train for 120k iterations. We utilize random cropping with a size of 256 × 256 and horizontal flipping with a probability of $[eqn]$ during training. For MVSEC scenes with $[eqn]$ = 1, we use the setting with $[eqn]$ = 3; for MVSEC with $[eqn]$ = 4, we use the setting with $[eqn]$ = 9.

4.3. Regular Flow Evaluation Prototype

Evaluation on DSEC. Table 1 presents a comprehensive evaluation on the DSEC-Flow benchmark [9], including performance and computational complexity metrics. The metrics assess accuracy via EPE (L2 endpoint error in pixels), AE (angular error in degrees), and nPE (percentage of pixels with optical flow magnitude error greater than n pixels); model complexity via Params (millions of parameters); and computational cost via GMACs (Giga Multiply-Accumulate operations per inference, in billions).

Analyzing the results, our EVA-Flow establishes itself as a leading solution for time-dense optical flow. Compared to other approaches capable of high-frequency output (Taming_CM, LSTM-FlowNet), EVA-Flow achieves superior temporal characteristics with the lowest input latency (5 ms) and the highest prediction rate (200 Hz). While delivering this enhanced temporal resolution, it maintains a competitive level of accuracy. Although slightly less accurate than the top-performing non-time-dense methods, EVA-Flow demonstrates an excellent trade-off between high-frequency prediction capabilities and flow estimation precision. Crucially, EVA-Flow achieves this performance with remarkable computational efficiency, making it highly suitable for resource-constrained applications.

Importantly, the reported GMACs reflect the cost for generating a single optical flow prediction. This allows a fair comparison of per-prediction efficiency, irrespective of the output rate (e.g., our 200 Hz vs. 10/100 Hz for others). EVA-Flow’s low per-prediction cost is what enables its high-frequency operation.

E-RAFT [3], a representative method for event-based optical flow, incorporates the RAFT’s [16] superior design features, such as correlation computation and iterative optical flow estimation, resulting in enhanced optical flow accuracy. It serves as the foundation for the latest state-of-the-art methods, TMA [33] and BFlow [39]. Consequently, we have chosen E-RAFT as a benchmark for our qualitative experiments.

The qualitative results of our method on the DSEC flow dataset are shown in Figure 5. To make a more intuitive comparison of the accuracy of optical flow estimation, we also visualized the IWEs. It can be seen from the regions of interest (ROIs) in Figure 5 that, compared with E-RAFT, our model can better distinguish the outline of small objects (such as poles) and obtain more accurate optical flow estimates. Our analysis suggests that the distinguishing feature of our approach, in contrast to E-RAFT, lies in the implicit use of warp-aligned event edges to extract motion features, whereas E-RAFT relies on a correlation volume. For small objects with simple, well-defined contours, warping and aligning event features is relatively straightforward. However, due to limited textures, the correlation-based method, which depends on measuring feature similarity, struggles to capture high-quality motion features in areas with insufficient texture, leading to the inferior performance of E-RAFT in such cases.

Evaluation on the MVSEC dataset. The evaluation results of the MVSEC benchmark [2] are presented in Table 2. SSL, USL, and SL stand for self-supervised-, unsupervised-, and supervised learning, respectively. While our model’s accuracy is slightly lower than that of E-RAFT based on the data in the table, it is worth highlighting that our zero-shot results display a significantly better accuracy when directly testing the performance of the DSEC-trained model on the MVSEC dataset. Especially noteworthy is our model’s performance at $[eqn]$ = 4, where our model’s EPE is halved compared to that of E-RAFT ( $[eqn]$ vs. $[eqn]$ ). These results demonstrate the superior generalizability of the proposed EVA-Flow.

The qualitative results of testing MVSEC under the zero-shot setting, i.e., the model is only trained on DSEC, can be found in Figure 6. Owing to the small optical flow of MVSEC, the motion compensation effect is indistinct under the $[eqn]$ = 1 setting. Hence, we apply a longer time range of grayscale frames, i.e.,  $[eqn]$ . There is a marked discrepancy in motion speed between the MVSEC and DSEC datasets, as depicted in Figure 5 and Figure 6. Nonetheless, our model is capable of achieving near-ground-truth optical flow results on the MVSEC dataset, while E-RAFT shows inadequate generalization performances, particularly in non-event ground areas. Additionally, our model demonstrates superior capabilities for small targets such as poles and road markings, accurately depicting their contours with improved motion compensation results.

Evaluation on EVA-FlowSet. Qualitative results of the curve motion scene dataset EVA-FlowSet can be found in Figure 7. By visualizing the optical flow images, it can be observed that, for both fast-moving scenes and normal scenes, our model can estimate the contour of moving objects more accurately than E-RAFT. The magnified ROI area shows that our model obtains sharper motion compensation results in the first three rows of fast-moving scenes, while E-RAFT’s motion compensation performance is comparable to our model in the last three rows of moderate-moving-speed scenes. E-RAFT calculates the correlation between the event voxel grid before and after the start time to estimate the optical flow, which implicitly assumes that the optical flow is constant throughout the entire time. In contrast, our method uses the SMR module iteratively in each time step to obtain continuous optical flow estimation, which is suitable for curve motion scenes. In scenes with moderate movement speeds, even if the motion trajectory is curved, the motion can be approximated as linear due to the relatively slow speed. Therefore, in cases of moderate movement speeds, the motion compensation results of E-RAFT are comparable to those of our model. However, in fast-moving scenes, the linear motion assumption of E-RAFT becomes invalid due to the inherently curved nature of the motion trajectory. In contrast, our model, which employs a dense optical flow estimation framework independent of the linear motion assumption, significantly outperforms E-RAFT in fast-moving scenarios.

4.4. Time-Dense Optical Flow Evaluation

With $[eqn]$ = 15, our model produces 14 consecutive optical flow estimates from events within a 100 ms window on the DSEC test set. The output $[eqn]$ covers 0 ms to 100 × $[eqn]$ ms, while E-RAFT provides only a single estimate for the entire 0 ms to 100 ms range. Due to the absence of time-dense ground truth in the DSEC dataset, a direct comparison with E-RAFT is not feasible. To address this, we generate images of warped events (IWEs) using our model’s time-dense flow estimates and E-RAFT’s interpolated flow and evaluate them with RFWL (refer to Section 3.4). This process is illustrated in Figure 8. All DSEC test sequences are evaluated, and detailed results are presented in Table 3.

Since $[eqn]$ represents forward optical flow from 0 ms to 100 × $[eqn]$ ms, we use this to warp the events within that period to generate the corresponding IWE. Our model’s optical flow estimates at intermediate time steps are generally more accurate than E-RAFT’s interpolated results, leading to better RFWL performance. However, as i approaches the final time step (e.g., i = 14), E-RAFT’s more accurate final optical flow results in a higher RFWL. Nevertheless, across seven sequences, our method surpasses E-RAFT in average RFWL on five sequences, demonstrating that our approach reliably estimates high-frame-rate optical flow even without supervision from high-frame-rate ground truth.

We conducted a comparative analysis using the EVA-FlowSet, which comprises sequences featuring curved movements. The results, depicted in Figure 9, indicate a significant performance improvement of our approach compared to E-RAFT. Since the EVA-FlowSet lacks ground-truth optical flow, both E-RAFT and our model are only trained on the DSEC dataset. Figure 9 presents a visual representation of the evaluation results comparing our method with E-RAFT in real-world scenarios with varying motion speeds. Specifically, in two fast-moving sequences, our model demonstrates a notably superior intermediate RFWL compared to E-RAFT. We observe that the overall accuracy of EVA-Flow is significantly higher during periods of rapid motion, yet it experiences a slight decrease in accuracy towards the end of the event bin cycle. This phenomenon correlates with the aforementioned results of the DSEC dense-time validation experiments. In the other two scenarios characterized by normal-speed sequences, our model’s RFWL performance is comparable to that of E-RAFT. This equivalence arises because, at insufficient movement speeds along a curved path, the motion can be briefly approximated as linear.

We also assessed the efficacy of our dense temporal optical flow methodology from an alternative perspective. In the DSEC-Flow, where the ground-truth flow frame rate is 10 Hz and each sample spans 100 ms, we created a 200 ms ground truth for event optical flow by warping together the preceding and subsequent 100 ms optical flows of our validation set. Thereafter, we applied a model trained on the original dataset, which consists of 100 ms samples, to estimate this extended validation set. Table 4 (right side) displays the outcomes of this experiment. For the EVA-Flow model, which was trained with Bins set to b, we used the adjusted setting of $[eqn]$ for estimating the 200 ms optical flow. This approach maintains the duration of each bin as it was during training and aligns the bin lengths with the extended overall estimation period. Conversely, E-RAFT, when processing a 200 ms sample, adheres to its original configuration of 15 bins. The data in Table 4 show that our model’s error did not escalate when estimating the validation set with an extended time span; rather, its precision improved, with the average end point error (EPE) being less than double that of the test set with 100 ms samples. In comparison, E-RAFT’s EPE surpassed three times the EPE of the 100 ms sample test set.

We also observed similar results on the MVSEC dataset. Table 2 displays the zero-shot results for E-RAFT and our model, which was trained on DSEC and evaluated on MVSEC with $[eqn]$ = 15. When assessing optical flow estimation on the MVSEC dataset with our model, we use the identical checkpoint but adjust the bin count according to varying $[eqn]$ values. Due to the difference in motion speeds between the slow-motion MVSEC and the fast-motion DSEC datasets, we set $[eqn]$ = 4 for $[eqn]$ = 1 and $[eqn]$ = 13 for $[eqn]$ = 4. It is apparent that our model achieves a lower endpoint error (EPE) of $[eqn]$ at $[eqn]$ = 4, which is less than four times the EPE at $[eqn]$ = 1 ( $[eqn]$ ).

Interestingly, the results indicate that as we progressively iterate and generate optical flow for longer time intervals, the average per-unit-time optical flow error does not increase but rather decreases. These findings provide clear evidence for the efficacy of our framework in estimating time-dense optical flow. Furthermore, it underscores a significant advantage of the framework’s adaptable nature: the flexibility to adjust the number of prediction bins. This adjustment minimizes the difference in event rates per bin between the training and prediction phases, thereby enhancing accuracy during deployment.

4.5. Ablation Study

Unified voxel Grid: Table 5 compares the results of our framework on the DSEC test set using a voxel grid [7] and our proposed unified voxel grid representation. The EPE of our proposed unified voxel grid is $[eqn]$ lower than that of the voxel grid. This difference stems from the fact that each event bin in the unified voxel grid adheres to a consistent event representation, while the representation patterns of the first and last bins in the voxel grid do not match those of the middle bins (refer to Figure 3).

Number of Bins (N) Setting: The number of bins N in the UVG representation is a crucial hyperparameter that dictates the temporal granularity ( $[eqn]$ , where T is the total time window, e.g., 100 ms) and the maximum prediction frequency. Setting N involves a trade-off: higher N allows for finer temporal resolution and potentially higher prediction frequency but requires sufficient event density within each shorter interval $[eqn]$ .

Impact During Training (DSEC): We first investigated the effect of N during training on the DSEC dataset, evaluating the final End-Point Error (EPE). As shown in Table 4 (left), increasing N from 6 up to 21 leads to a consistent decrease in the final EPE. This improvement can be attributed to the network estimating smaller displacements in each recurrent step, simplifying the estimation task per iteration. However, when N increases further (e.g., to 31), the accuracy declines as individual bins start to contain insufficient event information, hindering robust flow estimation. Based on these results, we selected $[eqn]$ for our primary model trained on DSEC, as it provides a good balance, achieving a 20-fold increase in potential output frequency compared to standard methods while maintaining high final accuracy.

Impact During Inference (EVA-FlowSet [ $[eqn]$ ] and Guidance): While training with $[eqn]$ yields strong results for the final flow prediction, the optimal N for achieving the best time-dense flow quality during inference can vary depending on scene dynamics. To explore this, we conducted experiments on our EVA-FlowSet, evaluating the average time-dense RFWL using different inference values of N (from 2 to 21) on sequences with varying speeds, using the model trained with $[eqn]$ . The results, presented in Figure 10, reveal that the optimal inference N indeed depends on motion speed. For high-speed sequences, peak RFWL (best time-dense quality) is achieved around N = 11–12 ( $[eqn]$ ms), suggesting that fast motion generates enough events for reliable estimation even with short integration times. Conversely, for normal-speed sequences, performance peaks around N = 9 ( $[eqn]$ ms), indicating that slightly longer integration times are beneficial when motion is slower.

This analysis highlights the flexibility of EVA-Flow. While trained with a specific N (e.g., 21), the inference can be adapted. We provide the following guidance for practical application (for our EVA-Set):

• For high-speed scenarios, using $[eqn]$ during inference is recommended for optimal time-dense performance.
• For moderate/low-speed or low-light (sparse event) scenarios, using a longer $[eqn]$ ms during inference is likely beneficial to ensure sufficient event accumulation per bin.

It is important to note that these ranges serve as initial recommendations. The optimal setting for $[eqn]$ (and consequently N) depends strongly on the specific characteristics of the scene (motion speed, light condition). This adaptability allows users to tune the inference granularity for the specific scenario, maximizing the quality of the time-dense optical flow output.

5. Limitations and Future Work

While EVA-Flow demonstrates significant advantages in low-latency, high-frequency optical flow estimation with high computational efficiency, we acknowledge several limitations that suggest avenues for future research:

• Accuracy Relative to Non-Time-Dense Methods: Although competitive, EVA-Flow’s final prediction accuracy (e.g., EPE) is slightly lower than some state-of-the-art non-time-dense methods that utilize computationally intensive correlation volumes and focus solely on maximizing accuracy between two distant time points. Our current SMR module relies on implicit warp alignment for efficiency and time-density. Future work could explore hybrid approaches, potentially incorporating lightweight correlation features or attention mechanisms to enhance accuracy, especially for complex motions, without drastically increasing latency or computational cost.
• Sparse Temporal Supervision: EVA-Flow achieves time-dense prediction but is still primarily supervised using ground-truth optical flow provided at a much lower frame rate (e.g., 10 Hz on DSEC). While our unsupervised RFWL metric helps validate intermediate flows, the network lacks direct, dense temporal supervision during training. Developing techniques for generating reliable time-continuous ground truth, perhaps through advanced interpolation or simulation, or exploring unsupervised/self-supervised learning objectives specifically designed for time-dense event flow could further enhance intermediate flow accuracy and reliability.
• Event-Only Input: The current EVA-Flow framework operates solely on event data. While this highlights the richness of information within event streams, incorporating asynchronous image frames from the event camera (like Davis sensors) could provide complementary information, particularly in static scenes, low-texture areas, or during periods of low event activity. Designing efficient multi-modal fusion architectures that leverage both event dynamics and frame appearance is a promising direction for improving robustness and overall accuracy.

6. Conclusions

In this paper, we propose EVA-Flow, a learning-based model for anytime flow estimation based on event cameras. Leveraging the unified voxel grid representation, our network is able to estimate dense motion fields bin-by-bin between two temporal slices. By employing a novel spatiotemporal motion refinement module for implicit warp alignment, the proposed EVA-Flow overcomes the low-temporal-frequency issue of event-based optical flow datasets, achieving high-speed and high-frequency flow estimation beyond low-frequency supervised signals. Our approach significantly outperforms other time-dense event flow methods in temporal capabilities and demonstrates exceptional computational efficiency (16.8 GMACs per prediction). It also sustains a competitive level of accuracy compared to non-time-dense techniques. Furthermore, we introduced rectified flow warp loss (RFWL), an unsupervised metric for evaluating time-dense predictions, and showed that EVA-Flow’s prediction granularity can be flexibly adjusted during inference to suit varying scene dynamics. The combination of high temporal resolution, low latency, competitive accuracy, high computational efficiency, and adaptability makes EVA-Flow a highly practical and promising solution for real-time motion perception tasks on resource-aware platforms.