# Future semantic segmentation of time-lapsed videos with large temporal displacement

## TL;DR

This paper introduces a novel recurrent network architecture for future semantic segmentation of time-lapsed videos with large motions, demonstrating improved accuracy and solar irradiance prediction over existing methods.

## Contribution

A new recurrent network with attention mechanism is proposed for predicting future semantic masks in large displacement videos, along with a large-scale sky-video dataset.

## Key findings

- 10.8% improvement in mean IoU over TCNs for 10-minute ahead predictions
- Normalized MAE of 10.5% in solar irradiance measurement over two years
- Effective capture of complex fluid flow in sparse temporal sampling

## Abstract

An important aspect of video understanding is the ability to predict the evolution of its content in the future. This paper presents a future frame semantic segmentation technique for predicting semantic masks of the current and future frames in a time-lapsed video. We specifically focus on time-lapsed videos with large temporal displacement to highlight the model's ability to capture large motions in time. We first introduce a unique semantic segmentation prediction dataset with over 120,000 time-lapsed sky-video frames and all corresponding semantic masks captured over a span of five years in North America region. The dataset has immense practical value for cloud cover analysis, which are treated as non-rigid objects of interest. %Here the model provides both semantic segmentation of cloud region and solar irradiance emitted from a region from the sky-videos. Next, our proposed recurrent network architecture departs from existing trend of using temporal convolutional networks (TCN) (or feed-forward networks), by explicitly learning an internal representations for the evolution of video content with time. Experimental evaluation shows an improvement of mean IoU over TCNs in the segmentation task by 10.8% for 10 mins (21% over 60 mins) ahead of time predictions. Further, our model simultaneously measures both the current and future solar irradiance from the same video frames with a normalized-MAE of 10.5% over two years. These results indicate that recurrent memory networks with attention mechanism are able to capture complex advective and diffused flow characteristic of dense fluids even with sparse temporal sampling and are more suitable for future frame prediction tasks for longer duration videos.