dacl1k: Real-World Bridge Damage Dataset Putting Open-Source Data to the Test

TL;DR

This paper introduces the dacl1k dataset, a diverse real-world bridge damage dataset, and evaluates the generalization of open-source models for reinforced concrete defect detection in practical scenarios.

Contribution

It provides a new, diverse dataset for bridge defect detection and assesses the real-world applicability of models trained on existing open-source data.

Findings

Best model achieves 32% Exact Match Ratio on dacl1k

Models trained on meta datasets show practical usability

Intrinsic evaluation reveals whether models distinguish datasets or defect classes

Abstract

Recognising reinforced concrete defects (RCDs) is a crucial element for determining the structural integrity, traffic safety and durability of bridges. However, most of the existing datasets in the RCD domain are derived from a small number of bridges acquired in specific camera poses, lighting conditions and with fixed hardware. These limitations question the usability of models trained on such open-source data in real-world scenarios. We address this problem by testing such models on our "dacl1k" dataset, a highly diverse RCD dataset for multi-label classification based on building inspections including 1,474 images. Thereby, we trained the models on different combinations of open-source data (meta datasets) which were subsequently evaluated both extrinsically and intrinsically. During extrinsic evaluation, we report metrics on dacl1k and the meta datasets. The performance analysis on dacl1k shows practical usability of the meta data, where the best model shows an Exact Match Ratio of 32%. Additionally, we conduct an intrinsic evaluation by clustering the bottleneck features of the best model derived from the extrinsic evaluation in order to find out, if the model has learned distinguishing datasets or the classes (RCDs) which is the aspired goal. The dacl1k dataset and our trained models will be made publicly available, enabling researchers and practitioners to put their models to the real-world test.