Lagrangian Motion Magnification with Double Sparse Optical Flow Decomposition

TL;DR

This paper introduces a novel Lagrangian motion magnification method for facial micro-expressions, combining deep learning-based optical flow estimation with a double sparse decomposition to enhance subtle facial motions effectively.

Contribution

It presents a new approach that fine-tunes deep learning optical flow for faces, employs double sparse decomposition for local motion analysis, and introduces a novel warping strategy for targeted motion magnification.

Findings

Efficient and accurate optical flow estimation for facial micro-motions.

Effective magnification of localized facial micro-expressions.

Validation through various practical examples.

Abstract

Microexpressions are fast and spatially small facial expressions that are difficult to detect. Therefore motion magnification techniques, which aim at amplifying and hence revealing subtle motion in videos, appear useful for handling such expressions. There are basically two main approaches, namely via Eulerian or Lagrangian techniques. While the first one magnifies motion implicitly by operating directly on image pixels, the Lagrangian approach uses optical flow (OF) techniques to extract and magnify pixel trajectories. In this paper, we propose a novel approach for local Lagrangian motion magnification of facial micro-motions. Our contribution is three-fold: first, we fine tune the recurrent all-pairs field transforms (RAFT) for OFs deep learning approach for faces by adding ground truth obtained from the variational dense inverse search (DIS) for OF algorithm applied to the CASME II video set of facial micro expressions. This enables us to produce OFs of facial videos in an efficient and sufficiently accurate way. Second, since facial micro-motions are both local in space and time, we propose to approximate the OF field by sparse components both in space and time leading to a double sparse decomposition. Third, we use this decomposition to magnify micro-motions in specific areas of the face, where we introduce a new forward warping strategy using a triangular splitting of the image grid and barycentric interpolation of the RGB vectors at the corners of the transformed triangles. We demonstrate the feasibility of our approach by various examples.