# Multi-Modal Trip Hazard Affordance Detection On Construction Sites

## TL;DR

This paper presents a multi-modal approach combining colour and depth data to improve trip hazard detection on construction sites, demonstrating a 4% F1-score improvement over colour-only methods, aiding automation of safety inspections.

## Contribution

It introduces a comprehensive evaluation of 11 fusion techniques for trip hazard detection using colour and depth images, and provides a new dataset from active construction sites.

## Key findings

- Multi-modal fusion improves trip hazard detection accuracy.
- The approach outperforms colour-only detectors by 4% in F1-score.
- Extensive dataset supports future research in automated safety inspections.

## Abstract

Trip hazards are a significant contributor to accidents on construction and manufacturing sites, where over a third of Australian workplace injuries occur [1]. Current safety inspections are labour intensive and limited by human fallibility,making automation of trip hazard detection appealing from both a safety and economic perspective. Trip hazards present an interesting challenge to modern learning techniques because they are defined as much by affordance as by object type; for example wires on a table are not a trip hazard, but can be if lying on the ground. To address these challenges, we conduct a comprehensive investigation into the performance characteristics of 11 different colour and depth fusion approaches, including 4 fusion and one non fusion approach; using colour and two types of depth images. Trained and tested on over 600 labelled trip hazards over 4 floors and 2000m$\mathrm{^{2}}$ in an active construction site,this approach was able to differentiate between identical objects in different physical configurations (see Figure 1). Outperforming a colour-only detector, our multi-modal trip detector fuses colour and depth information to achieve a 4% absolute improvement in F1-score. These investigative results and the extensive publicly available dataset moves us one step closer to assistive or fully automated safety inspection systems on construction sites.