TS-RGBD Dataset: a Novel Dataset for Theatre Scenes Description for People with Visual Impairments

TL;DR

This paper introduces TS-RGBD, a new RGB-D dataset with theatre scenes, enabling improved image captioning and human action recognition for aiding visually impaired individuals in complex environments.

Contribution

The paper presents a novel RGB-D dataset for theatre scenes, including ground truth annotations for actions and captions, expanding computer vision applications for visually impaired assistance.

Findings

Image captioning models perform well on theatre scenes

Skeleton-based action recognition models are effective in this context

Dataset enables better scene understanding for visually impaired aid

Abstract

Computer vision was long a tool used for aiding visually impaired people to move around their environment and avoid obstacles and falls. Solutions are limited to either indoor or outdoor scenes, which limits the kind of places and scenes visually disabled people can be in, including entertainment places such as theatres. Furthermore, most of the proposed computer-vision-based methods rely on RGB benchmarks to train their models resulting in a limited performance due to the absence of the depth modality. In this paper, we propose a novel RGB-D dataset containing theatre scenes with ground truth human actions and dense captions annotations for image captioning and human action recognition: TS-RGBD dataset. It includes three types of data: RGB, depth, and skeleton sequences, captured by Microsoft Kinect. We test image captioning models on our dataset as well as some skeleton-based human action recognition models in order to extend the range of environment types where a visually disabled person can be, by detecting human actions and textually describing appearances of regions of interest in theatre scenes.