Selfie Taking with Facial Expression Recognition Using Omni-directional Camera

TL;DR

This paper introduces a novel method for visually impaired users to take selfies with multiple people using an omni-directional camera, ensuring everyone is in view and has a good expression.

Contribution

It presents a new approach combining face detection, false detection elimination, and expression recognition on omni-directional video to automatically capture the best selfie frame.

Findings

Effective face detection across all frames

Successful identification of happiest expressions

Generated perspective images with all participants in view

Abstract

Recent studies have shown that visually impaired people have desires to take selfies in the same way as sighted people do to record their photos and share them with others. Although support applications using sound and vibration have been developed to help visually impaired people take selfies using smartphone cameras, it is still difficult to capture everyone in the angle of view, and it is also difficult to confirm that they all have good expressions in the photo. To mitigate these issues, we propose a method to take selfies with multiple people using an omni-directional camera. Specifically, a user takes a few seconds of video with an omni-directional camera, followed by face detection on all frames. The proposed method then eliminates false face detections and complements undetected ones considering the consistency across all frames. After performing facial expression recognition on all the frames, the proposed method finally extracts the frame in which the participants are happiest, and generates a perspective projection image in which all the participants are in the angle of view from the omni-directional frame. In experiments, we use several scenes with different number of people taken to demonstrate the effectiveness of the proposed method.