JIT-Masker: Efficient Online Distillation for Background Matting

TL;DR

JIT-Masker is a real-time, efficient online distillation method for background matting that balances accuracy and speed, enabling practical virtual background applications in video conferencing.

Contribution

It introduces an online distillation pipeline for portrait matting that significantly improves speed while maintaining high quality, suitable for consumer-level virtual background tools.

Findings

5x speedup over saliency detection pipeline

Higher quality results than existing methods

Effective in non-GPU settings

Abstract

We design a real-time portrait matting pipeline for everyday use, particularly for "virtual backgrounds" in video conferences. Existing segmentation and matting methods prioritize accuracy and quality over throughput and efficiency, and our pipeline enables trading off a controllable amount of accuracy for better throughput by leveraging online distillation on the input video stream. We construct our own dataset of simulated video calls in various scenarios, and show that our approach delivers a 5x speedup over a saliency detection based pipeline in a non-GPU accelerated setting while delivering higher quality results. We demonstrate that an online distillation approach can feasibly work as part of a general, consumer level product as a "virtual background" tool. Our public implementation is at https://github.com/josephch405/jit-masker.