Fast forwarding Egocentric Videos by Listening and Watching

TL;DR

This paper introduces a novel method for fast-forwarding egocentric videos by utilizing psychoacoustic metrics from audio to highlight pleasant sound moments, improving user engagement and viewing experience.

Contribution

It is the first approach to incorporate audio psychoacoustic metrics for video fast-forwarding, integrating sound analysis with visual content to identify engaging segments.

Findings

Effective speed-up with reduced instability.

Enhanced highlighting of pleasant sound segments.

Quantitative improvements in viewing experience.

Abstract

The remarkable technological advance in well-equipped wearable devices is pushing an increasing production of long first-person videos. However, since most of these videos have long and tedious parts, they are forgotten or never seen. Despite a large number of techniques proposed to fast-forward these videos by highlighting relevant moments, most of them are image based only. Most of these techniques disregard other relevant sensors present in the current devices such as high-definition microphones. In this work, we propose a new approach to fast-forward videos using psychoacoustic metrics extracted from the soundtrack. These metrics can be used to estimate the annoyance of a segment allowing our method to emphasize moments of sound pleasantness. The efficiency of our method is demonstrated through qualitative results and quantitative results as far as of speed-up and instability are concerned.