Invisible-to-Visible: Privacy-Aware Human Instance Segmentation using Airborne Ultrasound via Collaborative Learning Variational Autoencoder

TL;DR

This paper introduces a privacy-preserving human instance segmentation method using airborne ultrasound and collaborative learning variational autoencoders, enabling action recognition without camera images.

Contribution

It proposes a novel task and a CL-VAE model that learns from sound and RGB images during training to perform segmentation solely from sound images at inference.

Findings

CL-VAE outperforms conventional VAEs in segmentation accuracy

The method enables privacy-preserving human action recognition

Sound images can be effectively used for human segmentation

Abstract

In action understanding in indoor, we have to recognize human pose and action considering privacy. Although camera images can be used for highly accurate human action recognition, camera images do not preserve privacy. Therefore, we propose a new task for human instance segmentation from invisible information, especially airborne ultrasound, for action recognition. To perform instance segmentation from invisible information, we first convert sound waves to reflected sound directional images (sound images). Although the sound images can roughly identify the location of a person, the detailed shape is ambiguous. To address this problem, we propose a collaborative learning variational autoencoder (CL-VAE) that simultaneously uses sound and RGB images during training. In inference, it is possible to obtain instance segmentation results only from sound images. As a result of performance verification, CL-VAE could estimate human instance segmentations more accurately than conventional variational autoencoder and some other models. Since this method can obtain human segmentations individually, it could be applied to human action recognition tasks with privacy protection.