ARShoe: Real-Time Augmented Reality Shoe Try-on System on Smartphones

TL;DR

ARShoe is a real-time augmented reality system for virtual shoe try-on on smartphones, combining pose estimation, segmentation, realistic occlusion, and stabilization, supported by a large-scale foot benchmark.

Contribution

This work introduces ARShoe, the first real-time AR shoe try-on system on smartphones, with novel multi-branch network, occlusion handling, stabilization, and a large-scale foot benchmark.

Findings

Achieves real-time performance on smartphones.

Demonstrates high accuracy in pose estimation and segmentation.

Provides realistic virtual shoe try-on with occlusion and stabilization.

Abstract

Virtual try-on technology enables users to try various fashion items using augmented reality and provides a convenient online shopping experience. However, most previous works focus on the virtual try-on for clothes while neglecting that for shoes, which is also a promising task. To this concern, this work proposes a real-time augmented reality virtual shoe try-on system for smartphones, namely ARShoe. Specifically, ARShoe adopts a novel multi-branch network to realize pose estimation and segmentation simultaneously. A solution to generate realistic 3D shoe model occlusion during the try-on process is presented. To achieve a smooth and stable try-on effect, this work further develop a novel stabilization method. Moreover, for training and evaluation, we construct the very first large-scale foot benchmark with multiple virtual shoe try-on task-related labels annotated. Exhaustive experiments on our newly constructed benchmark demonstrate the satisfying performance of ARShoe. Practical tests on common smartphones validate the real-time performance and stabilization of the proposed approach.