Bridge Damage Cause Estimation Using Multiple Images Based on Visual Question Answering

TL;DR

This paper introduces a novel framework combining Structure from Motion and Visual Question Answering to estimate bridge damage causes from multiple images, aiding maintenance and automation of infrastructure diagnosis.

Contribution

The paper develops a VQA model tailored for bridge damage assessment and demonstrates its application in real-world damage cause estimation.

Findings

VQA model achieved 67.4% accuracy for damage name questions.

Yes/no questions had 99.1% correct answer rate.

Method successfully applied to actual bridge for damage cause estimation.

Abstract

In this paper, a bridge member damage cause estimation framework is proposed by calculating the image position using Structure from Motion (SfM) and acquiring its information via Visual Question Answering (VQA). For this, a VQA model was developed that uses bridge images for dataset creation and outputs the damage or member name and its existence based on the images and questions. In the developed model, the correct answer rate for questions requiring the member's name and the damage's name were 67.4% and 68.9%, respectively. The correct answer rate for questions requiring a yes/no answer was 99.1%. Based on the developed model, a damage cause estimation method was proposed. In the proposed method, the damage causes are narrowed down by inputting new questions to the VQA model, which are determined based on the surrounding images obtained via SfM and the results of the VQA model. Subsequently, the proposed method was then applied to an actual bridge and shown to be capable of determining damage and estimating its cause. The proposed method could be used to prevent damage causes from being overlooked, and practitioners could determine inspection focus areas, which could contribute to the improvement of maintenance techniques. In the future, it is expected to contribute to infrastructure diagnosis automation.