AV1 Motion Vector Fidelity and Application for Efficient Optical Flow

TL;DR

This paper analyzes AV1 video codec motion vectors for their fidelity and demonstrates their effectiveness as a fast, high-quality initialization method for optical flow estimation, reducing computation time significantly.

Contribution

It provides a detailed comparison of AV1 motion vectors against ground-truth optical flow and shows their utility in accelerating deep learning-based optical flow methods.

Findings

AV1 motion vectors have high fidelity compared to ground-truth optical flow.

Using AV1 motion vectors as a warm-start speeds up RAFT optical flow convergence fourfold.

Optimal encoder settings improve motion vector fidelity for optical flow applications.

Abstract

This paper presents a comprehensive analysis of motion vectors extracted from AV1-encoded video streams and their application in accelerating optical flow estimation. We demonstrate that motion vectors from AV1 video codec can serve as a high-quality and computationally efficient substitute for traditional optical flow, a critical but often resource-intensive component in many computer vision pipelines. Our primary contributions are twofold. First, we provide a detailed comparison of motion vectors from both AV1 and HEVC against ground-truth optical flow, establishing their fidelity. In particular we show the impact of encoder settings on motion estimation fidelity and make recommendations about the optimal settings. Second, we show that using these extracted AV1 motion vectors as a "warm-start" for a state-of-the-art deep learning-based optical flow method, RAFT, significantly reduces the time to convergence while achieving comparable accuracy. Specifically, we observe a four-fold speedup in computation time with only a minor trade- off in end-point error. These findings underscore the potential of reusing motion vectors from compressed video as a practical and efficient method for a wide range of motion-aware computer vision applications.