# Efficient and Privacy-Protecting Background Removal for 2D Video Streaming using iPhone 15 Pro Max LiDAR

## TL;DR

This paper presents a real-time background removal method for 2D video streaming on iPhone 15 Pro Max using LiDAR depth data, offering privacy protection and consistent performance across lighting conditions.

## Contribution

It integrates LiDAR with color cameras on a mobile device for efficient, real-time background removal without AI models or chroma keying.

## Key findings

- Achieves 60 fps real-time processing on iPhone 15 Pro Max.
- LiDAR-based removal performs well in various lighting conditions.
- Current depth resolution limits streaming quality.

## Abstract

Light Detection and Ranging (LiDAR) technology in consumer-grade mobile devices can be used as a replacement for traditional background removal and compositing techniques. Unlike approaches such as chroma keying and trained AI models, LiDAR's depth information is independent of subject lighting, and performs equally well in low-light and well-lit environments. We integrate the LiDAR and color cameras on the iPhone 15 Pro Max with GPU-based image processing. We use Apple's SwiftUI and Swift frameworks for user interface and backend development, and Metal Shader Language (MSL) for realtime image enhancement at the standard iPhone streaming frame rate of 60 frames per second. The only meaningful limitations of the technology are the streaming bandwidth of the depth data, which currently reduces the depth map resolution to 320x240, and any pre-existing limitations of the LiDAR IR laser to reflect accurate depth from some materials. If the LiDAR resolution on a mobile device like the iPhone can be improved to match the color image resolution, LiDAR could feasibly become the preeminent method of background removal for video applications and photography.