Semantic scene synthesis: Application to assistive systems

TL;DR

This paper presents a method for semantic scene synthesis from a single depth image to assist visually impaired individuals, combining deep learning classification with tactile-based scene representation.

Contribution

It introduces a novel approach that classifies scene segments and encodes them with semantic labels inspired by Braille and Kanji, enabling tactile scene synthesis.

Findings

Successfully predicts over 17 object classes

Effective on noisy and incomplete depth data

Provides tactile scene representations for assistive use

Abstract

The aim of this work is to provide a semantic scene synthesis from a single depth image. This is used in assistive aid systems for visually impaired and blind people that allow them to understand their surroundings by the touch sense. The fact that blind people use touch to recognize objects and rely on listening to replace sight, motivated us to propose this work. First, the acquired depth image is segmented and each segment is classified in the context of assistive systems using a deep learning network. Second, inspired by the Braille system and the Japanese writing system Kanji, the obtained classes are coded with semantic labels. The scene is then synthesized using these labels and the extracted geometric features. Our system is able to predict more than 17 classes only by understanding the provided illustrative labels. For the remaining objects, their geometric features are transmitted. The labels and the geometric features are mapped on a synthesis area to be sensed by the touch sense. Experiments are conducted on noisy and incomplete data including acquired depth images of indoor scenes and public datasets. The obtained results are reported and discussed.