Fine-tuning vision foundation model for crack segmentation in civil infrastructures

TL;DR

This paper introduces a fine-tuned vision foundation model, CrackSAM, for crack segmentation in civil infrastructure, demonstrating superior zero-shot performance across diverse challenging scenarios and datasets.

Contribution

The work adapts a large-scale vision foundation model for civil engineering crack segmentation using parameter-efficient fine-tuning methods, achieving state-of-the-art zero-shot results.

Findings

CrackSAM outperforms 12 existing models on multiple datasets.

CrackSAM maintains high accuracy under challenging conditions.

Zero-shot capability demonstrates foundation models' potential in civil engineering.

Abstract

Large-scale foundation models have become the mainstream deep learning method, while in civil engineering, the scale of AI models is strictly limited. In this work, a vision foundation model is introduced for crack segmentation. Two parameter-efficient fine-tuning methods, adapter and low-rank adaptation, are adopted to fine-tune the foundation model in semantic segmentation: the Segment Anything Model (SAM). The fine-tuned CrackSAM shows excellent performance on different scenes and materials. To test the zero-shot performance of the proposed method, two unique datasets related to road and exterior wall cracks are collected, annotated and open-sourced, for a total of 810 images. Comparative experiments are conducted with twelve mature semantic segmentation models. On datasets with artificial noise and previously unseen datasets, the performance of CrackSAM far exceeds that of all state-of-the-art models. CrackSAM exhibits remarkable superiority, particularly under challenging conditions such as dim lighting, shadows, road markings, construction joints, and other interference factors. These cross-scenario results demonstrate the outstanding zero-shot capability of foundation models and provide new ideas for developing vision models in civil engineering.