Dual Transfer Learning for Event-based End-task Prediction via Pluggable Event to Image Translation

TL;DR

This paper introduces a dual transfer learning framework that enhances event-based end-task prediction by translating sparse event data into images and transferring learned features, significantly improving tasks like segmentation and depth estimation.

Contribution

It proposes a flexible two-stream dual transfer learning approach combining event-to-image translation with feature and pixel-level transfer, improving end-task performance without extra inference cost.

Findings

Significant performance boost on semantic segmentation

Improved depth estimation accuracy

Effective feature and pixel-level knowledge transfer

Abstract

Event cameras are novel sensors that perceive the per-pixel intensity changes and output asynchronous event streams with high dynamic range and less motion blur. It has been shown that events alone can be used for end-task learning, e.g., semantic segmentation, based on encoder-decoder-like networks. However, as events are sparse and mostly reflect edge information, it is difficult to recover original details merely relying on the decoder. Moreover, most methods resort to pixel-wise loss alone for supervision, which might be insufficient to fully exploit the visual details from sparse events, thus leading to less optimal performance. In this paper, we propose a simple yet flexible two-stream framework named Dual Transfer Learning (DTL) to effectively enhance the performance on the end-tasks without adding extra inference cost. The proposed approach consists of three parts: event to end-task learning (EEL) branch, event to image translation (EIT) branch, and transfer learning (TL) module that simultaneously explores the feature-level affinity information and pixel-level knowledge from the EIT branch to improve the EEL branch. This simple yet novel method leads to strong representation learning from events and is evidenced by the significant performance boost on the end-tasks such as semantic segmentation and depth estimation.