Knowledge driven Description Synthesis for Floor Plan Interpretation

TL;DR

This paper introduces two deep learning models, DSIC and TBDG, for generating detailed textual descriptions from floor plan images, improving flexibility and robustness over existing methods.

Contribution

It presents novel models that leverage visual features and paragraph-based understanding for more accurate and flexible floor plan image captioning.

Findings

TBDG outperforms state-of-the-art methods in experiments.

Models effectively extract and utilize visual and textual features.

Proposed methods enhance description robustness and detail.

Abstract

Image captioning is a widely known problem in the area of AI. Caption generation from floor plan images has applications in indoor path planning, real estate, and providing architectural solutions. Several methods have been explored in literature for generating captions or semi-structured descriptions from floor plan images. Since only the caption is insufficient to capture fine-grained details, researchers also proposed descriptive paragraphs from images. However, these descriptions have a rigid structure and lack flexibility, making it difficult to use them in real-time scenarios. This paper offers two models, Description Synthesis from Image Cue (DSIC) and Transformer Based Description Generation (TBDG), for the floor plan image to text generation to fill the gaps in existing methods. These two models take advantage of modern deep neural networks for visual feature extraction and text generation. The difference between both models is in the way they take input from the floor plan image. The DSIC model takes only visual features automatically extracted by a deep neural network, while the TBDG model learns textual captions extracted from input floor plan images with paragraphs. The specific keywords generated in TBDG and understanding them with paragraphs make it more robust in a general floor plan image. Experiments were carried out on a large-scale publicly available dataset and compared with state-of-the-art techniques to show the proposed model's superiority.