Which Are You In A Photo?

TL;DR

This paper introduces an acoustic-based method using Doppler effect on mobile phones to accurately identify and tag people in photos in real-time, addressing key challenges of person recognition and tag association.

Contribution

It presents a novel acoustic scheme leveraging Doppler effect for real-time person identification and tagging during photo capture on mobile phones.

Findings

Tag correlation accuracy exceeds 85% within 3 meters.

The scheme works across various real-life scenarios.

Implemented on 7 Android phones with successful experiments.

Abstract

Automatic image tagging has been a long standing problem, it mainly relies on image recognition techniques of which the accuracy is still not satisfying. This paper attempts to explore out-of-band sensing base on the mobile phone to sense the people in a picture while the picture is being taken and create name tags on-the-fly. The major challenges pertain to two aspects - "Who" and "Which". (1) "Who": discriminating people who are in the picture from those that are not; (2) "Which": correlating each name tag with its corresponding people in the picture. We propose an accurate acoustic scheme applying on the mobile phones, which leverages the Doppler effect of sound wave to address these two challenges. As a proof of concept, we implement the scheme on 7 android phones and take pictures in various real-life scenarios with people positioning in different ways. Extensive experiments show that the accuracy of tag correlation is above 85% within 3m for picturing.