MediaPipe: A Framework for Building Perception Pipelines

TL;DR

MediaPipe is a comprehensive framework that simplifies building, testing, and optimizing perception applications across platforms by integrating perception components, measuring performance, and supporting iterative development.

Contribution

It provides an integrated environment for developing perception pipelines, enabling reproducibility and cross-platform deployment, which streamlines the development process.

Findings

Facilitates rapid prototyping and deployment of perception applications.

Supports cross-platform performance measurement and optimization.

Enables iterative improvement of perception pipelines.

Abstract

Building applications that perceive the world around them is challenging. A developer needs to (a) select and develop corresponding machine learning algorithms and models, (b) build a series of prototypes and demos, (c) balance resource consumption against the quality of the solutions, and finally (d) identify and mitigate problematic cases. The MediaPipe framework addresses all of these challenges. A developer can use MediaPipe to build prototypes by combining existing perception components, to advance them to polished cross-platform applications and measure system performance and resource consumption on target platforms. We show that these features enable a developer to focus on the algorithm or model development and use MediaPipe as an environment for iteratively improving their application with results reproducible across different devices and platforms. MediaPipe will be open-sourced at https://github.com/google/mediapipe.