Leveraging AV1 motion vectors for Fast and Dense Feature Matching

TL;DR

This paper introduces a method that uses AV1 video compression motion vectors to generate dense feature matches efficiently, comparable to traditional methods like SIFT but with lower computational cost.

Contribution

It repurposes AV1 motion vectors for dense, sub-pixel correspondences, demonstrating efficiency and effectiveness in structure-from-motion tasks.

Findings

Comparable performance to SIFT with less CPU usage

Dense matches enable successful 3D reconstruction on short videos

Resource-efficient approach suitable for scaling in full pipelines

Abstract

We repurpose AV1 motion vectors to produce dense sub-pixel correspondences and short tracks filtered by cosine consistency. On short videos, this compressed-domain front end runs comparably to sequential SIFT while using far less CPU, and yields denser matches with competitive pairwise geometry. As a small SfM demo on a 117-frame clip, MV matches register all images and reconstruct 0.46-0.62M points at 0.51-0.53,px reprojection error; BA time grows with match density. These results show compressed-domain correspondences are a practical, resource-efficient front end with clear paths to scaling in full pipelines.