MicroCam: Leveraging Smartphone Microscope Camera for Context-Aware Contact Surface Sensing

TL;DR

MicroCam is a novel contact sensing system that combines smartphone IMU data and microscopic surface imaging, enabling accurate, robust, and generalizable detection of surface types and textures for enhanced mobile context awareness.

Contribution

This work introduces MicroCam, integrating microscopic surface imaging with deep learning for contact surface detection, a novel approach in mobile context sensing.

Findings

High accuracy in surface and material classification

Robustness across different surface types and conditions

Potential for diverse mobile applications

Abstract

The primary focus of this research is the discreet and subtle everyday contact interactions between mobile phones and their surrounding surfaces. Such interactions are anticipated to facilitate mobile context awareness, encompassing aspects such as dispensing medication updates, intelligently switching modes (e.g., silent mode), or initiating commands (e.g., deactivating an alarm). We introduce MicroCam, a contact-based sensing system that employs smartphone IMU data to detect the routine state of phone placement and utilizes a built-in microscope camera to capture intricate surface details. In particular, a natural dataset is collected to acquire authentic surface textures in situ for training and testing. Moreover, we optimize the deep neural network component of the algorithm, based on continual learning, to accurately discriminate between object categories (e.g., tables) and material constituents (e.g., wood). Experimental results highlight the superior accuracy, robustness and generalization of the proposed method. Lastly, we conducted a comprehensive discussion centered on our prototype, encompassing topics such as system performance and potential applications and scenarios.