Autonomous Robotic System using Non-Destructive Evaluation methods for Bridge Deck Inspection

TL;DR

This paper presents an autonomous robotic system equipped with multiple NDE sensors for real-time, cost-effective bridge deck inspection, utilizing machine learning for automated steel rebar detection and condition mapping.

Contribution

The paper introduces a novel robotic platform integrating GPR, ER, and camera sensors with machine learning for automated bridge deck assessment.

Findings

Successful real-time data collection with multiple NDE sensors.

Automated steel rebar detection using machine learning.

Generation of real-time condition maps of bridge decks.

Abstract

Bridge condition assessment is important to maintain the quality of highway roads for public transport. Bridge deterioration with time is inevitable due to aging material, environmental wear and in some cases, inadequate maintenance. Non-destructive evaluation (NDE) methods are preferred for condition assessment for bridges, concrete buildings, and other civil structures. Some examples of NDE methods are ground penetrating radar (GPR), acoustic emission, and electrical resistivity (ER). NDE methods provide the ability to inspect a structure without causing any damage to the structure in the process. In addition, NDE methods typically cost less than other methods, since they do not require inspection sites to be evacuated prior to inspection, which greatly reduces the cost of safety related issues during the inspection process. In this paper, an autonomous robotic system equipped with three different NDE sensors is presented. The system employs GPR, ER, and a camera for data collection. The system is capable of performing real-time, cost-effective bridge deck inspection, and is comprised of a mechanical robot design and machine learning and pattern recognition methods for automated steel rebar picking to provide realtime condition maps of the corrosive deck environments.

Tables

Table 1. TABLE I : Training Data Set

Class	Class Name	Number of Images
1	Hyperbolas	304
2	Not Hyperbolas	1800

Table 2. TABLE II : Automated rebar detection results of the proposed method.

Bridge name	Location	Images	Total rebar	Accuracy	Precision	Run time/image
Kendall Pond Road Bridge	Derry, NH	12	2284	91.46%	97.79%	32.91s
Ramp D Bridge	Lewiston, ME	14	3699	92.89%	93.787%	55.46s

Table 3. TABLE III : Automated rebar detection comparison: proposed method vs. RADAN 7.

Bridge name	Location	Images	Total rebar	Method	True positives	False positives	Accuracy	Precision
Pleasant Valley Bridge	Reno, NV	20	13205	Proposed Method	12768	52	96.69%	99.59%
				RADAN®7	13135	186	99.47%	98.60%
East Helena Bridge	Helena, MT	14	1055	Proposed Method	1046	19	99.15%	98.22%
				RADAN®7	917	151	86.92%	85.86%

Full text

I Introduction

Bridge deck condition assessment is the most important part for bridge health maintenance. The Federal Highway Administration (FHWA) initiated the Long-Term Bridge Performance (LTBP) program to utilize non-destructive evaluation (NDE) technologies for bridge deck condition assessment [1]. In a recent report, the number of concrete highway bridges in the United States with wearing surfaces is over 180,000 [2]. Those bridges are prone to corrosion and without proper inspection, costly maintenances are inevitable.

Even though a wide variety of NDE sensors are actively used in the field, the inspection process is still time-consuming, requiring skilled inspectors [3, 4, 5]. As illustrated in Fig. 1, several inspectors were required to operate each sensor, which brings into question the safety requirements for inspection and the cost-efficiency of the inspection process. Furthermore, since sensor operation was conducted separately, discrepancies in collected data are possible. It will be beneficial if a fully autonomous bridge deck inspection system is developed to address all of these issues, by eliminating the need for costly safety requirements, and reducing the number of paid hours spent on each inspection. There are several research efforts for the development of the automated bridge inspection systems such as Robotic Bridge Inspection Tool (RABIT) [6, 7, 8, 9, 10, 11] associated with NDE data processing [12, 13, 14, 15].

In this paper, a novel robotic system is designed for bridge deck inspection. The system is equipped with a digital single-lens reflex (DSLR) camera for visual crack detection, ground penetrating radar (GPR) for concrete rebar assessment, and two electrical resistivity (ER) sensors for concrete corrosion assessment. This system is designed to perform a comprehensive inspection of bridge decks. The camera system is used for bridge deck visual assessment and surface crack detection, while the GPR provides an in-depth look at the condition of the bridge and the steel rebar objects inside the concrete deck, which is the most vital part of a bridge. The ER sensors provide additional information about concrete deck condition in terms of the resistance of the concrete on the bridge deck, which tells how corroded the bridge deck is. The information collected from multiple sensors is integrated and being automatically processed to produce a comprehensive report of the bridge deck condition map. In addition, we propose a new method using machine learning for rebar detection. In comparison with previous work, this work is distinguished by providing a more cost-effective solution for autonomous bridge deck inspection, as well as showing that the proposed method for rebar picking works as well as the industry standard, and that it can perform in real-time by running it on multiple sets of real bridge data [16].

The structure of the rest of this paper is as follows: in Section II, we discuss about the overall design of the robotic bridge deck inspection. In Section III, a new method for steel rebar detection is proposed. In Section IV, the system’s capabilities are demonstrated through the experimental results.

II Overall Design For Robotic Bridge Deck Inspection

In this section, the development of the robotic platform is described, and short explanations are provided to detail how NDE sensors work.

II-A Seekur Jr mobile robot as a base platform

To have a robot move effectively along a narrow bridge deck, a skid-steering 4-wheel-drive robot model such as Seekur Jr mobile robot (from Omron Adept Technologies, Inc.) is used. To collect data along a bridge deck, the robot needs to move from one end of the bridge to the other end. Then it needs to turn around and continue its movement until the whole bridge deck is covered. The sample movement is illustrated in Fig. 3. The selected mobile platform is able to rotate in place to change its movement direction, so that it can minimize unnecessary movements to conserve power. Besides, the Seekur Jr mobile robot is a waterproof platform, hence it is suitable for this bridge inspection task. The advantage of using this robot is that it is more versatile than its bigger version, Seekur, which had been used in other robotic systems [16]. With a smaller form, the Seekur Jr. robot can manage to move in narrower environments.

II-B Sensor integration

II-B1 GPR

GPR had been used in civil engineering for the last two decades [17, 18]. One of its applications is to evaluate corrosive level of top rebars inside bridge concrete decks [14].

By sending a radar signal into the bridge deck and recording the two way travel time of the signal’s reflection off of objects, it can produce unique signatures of that object; rebar in this case. In Fig. 2(a), as the antenna moves past the buried rebar, it constructs the hyperbola signature of the rebar (red dot) continuously by recording the reflections (yellow lines) from the rebar. To attach the GPR to the robot, a deployment system was designed.

In Fig. 2(b), a 3D model of a GPR deployment system is illustrated. The purpose of this deployment system is that the GPR unit will be touching the ground only when data is being collected. This design would make the robotic system easier to transport between locations.

II-B2 ER

In Fig. 2(c), a diagram of how a Resipod ER works is presented. The selected ER sensor is from Proceq USA, Inc. The ER sensor measures the electrical resistance of the concrete of a bridge deck. Since there is a correlation between concrete deterioration and its resistance (i.e., higher resistance better concrete), the inclusion of ER sensor would provide further information about concrete bridge deck corrosion.

II-C System implementation

A complete 3D design of the robotic system is presented in Fig.2(b). The robotic system was tested in simulation before being implemented.

Then a physical implementation of the robot is shown in Fig. 4. On this system, we also included a DSLR camera system to capture the bridge deck surface, which can be further processed for crack detection. To utilize the robot’s mobility, the GPR and ER deployment systems are coupled with gear shafts so that they are only deployed when necessary. The GPR deployment system is installed on the rear of the robot while the ER deployment system is mounted on the front. The motors provide torques and via motor’s shafts they control movements of GPR’s box and ER sensors.The GPR deployment system with detailed parts is in Fig. 4(b),(c). The robot localization and navigation is based on EKF-based sensor fusion from GPS, innertial measurement unit (IMU) and wheel odometry data [8].

In order to easily visualize the data being collected by the robot, a graphical user interface (GUI) was implemented. The GUI consists of three tabs: the “Scan Selection” tab, the “GPR/ER Data” tab, and the “Camera Data” tab. The “Scan Selection” tab shows the user a small preview of each finished scan and allows the user to dynamically update the data displayed on the other two tabs by selecting a scan. This tab will also update as more scans are performed, meaning it offers the user real-time access to the data being collected by the robotic system. The “GPR/ER Data” tab shows the user a larger representation of the currently selected scan, as well as an image containing red squares where rebar were automatically detected. The “Camera Data” tab lets the user navigate through a gallery of images taken of the ground surface by the camera. This tab automatically updates as the robotic system collects more data. Due to limit of space, the figure of this GUI is not presented here.

III Steel Rebar Detection For Bridge Deck Condition Assessment

III-A Related work

Research in automated object detection using GPR is a recent development. Prior to this research, there were two primary methods for detecting objects in GPR scan images: manual detection using the human eye, and using commercial software [19]. These two methods have respective issues. Manually detecting hundreds of rebar in an image is time consuming and requires training to correctly identify rebar, and using commercial software requires purchasing an expensive software license. The goal of research in automated rebar detection is to offer a third option that is less expensive and requires less time from the user.

Recent research in this field utilizes support vector machines, gradient descent, and various other computationally intense methods for detecting rebar [20, 21, 22, 9, 8]. While the accuracy of some recent methods has been reasonable, there is a recurring problem that methods are tested on data that is ideal or simulated [23, 24]. However, the automated rebar detection method proposed in this paper combines image processing, image classification, and statistical methods to perform an otherwise time consuming task in real-time, while maintaining high accuracy and precision.

III-B Algorithm

The automated rebar detection method proposed in this paper utilizes contrast limited adaptive histogram equalization (CLAHE) to contrast stretch GPR scan images that are low contrast. This makes the rebar signatures more visible to the classifier and the human eye. Pixel intensity information, specifically the location of the black horizontal area that indicates the presence of the ground plane, is used to determine the location of the ground plane within GPR scan images, which functions as the start of the search area for the classifier. Edge detection is used to determine the average vertical location of the rebar in the image, which functions as the end of the search area for the classifier. Laplacian edge detection was used for this paper, but the type of edge detection has little bearing on the results of the method.

A Naive Bayes classifier is used in this paper, since it has been shown to work well for simple classification tasks [25]. The classifier classifies vectors of histogram of oriented gradients (HOG) features that are extracted from 50 by 15 pixel images. HOG features are used because they are invariant to geometric transformations and illumination, and they can quickly be computed. For these reasons, HOG features are preferable to other types of features and have been widely used in computer vision for object detection since they were first described by Dalal and Triggs in 2005 [26, 27, 28]. Information on the method for extracting a HOG feature vector from an image can be found in [26].

The classifier is trained using the HOG feature vectors extracted from manually selected images that are assigned class labels, indicating that they either contain a clearly centered hyperbola (class 1) or don’t (class 2). Information on the training data set can be seen in Table I and Fig. 5. Once training is complete, a sliding window is applied across the GPR scan image, within the search window previously determined. At each sliding window location, a HOG feature vector is extracted from the window and the classifier determines if the window contains a hyperbola. Given a vector of HOG features, $x=(x_{1},...,x_{n})$, where $n$ is the number of features in the vector, and Bayes’ theorem, it holds that $p(C_{k}|x)=p(C_{k})p(x|C_{k})$, where $C_{k}$ is class $k$. This can be rewritten as $p(C_{k}|x)=p(C_{k},x_{1},...,x_{n})$, where $k$ is the number of classes. Now, using the general product rule, this can be rewritten again as $p(C_{k}|x)=p(x_{1}|x_{2},...,x_{n},C_{k})*p(x_{2}|x_{3},...,x_{n},C_{k})...p(x_{n}-1,x_{n},C_{k})*p(x_{n}|C_{k})p(C_{k})$. By assuming that every feature in the feature vector is conditionally independent of the other features, the Bayes model can be written as $p(C_{k}|x)=\frac{1}{Z}p(C_{k})\prod_{i=1}^{n}p(x_{i}|C_{k})$. Using this model, class labels can be assigned to test samples using the equation: $\hat{y}=\operatorname*{arg\,max}_{k\in{\{1,...,K\}}}p(C_{k})\prod_{i=1}^{n}p(x_{i}|C_{k})$, where $\hat{y}$ is the assigned class label given a sample, which is chosen based on the maximum probability of a class given the sample.

The classification process yields a set of points, $P_{OUT}$ indicating the general location of rebar within the GPR scan image. These points effectively function as clusters in image space where rebar are most likely to be. Areas with a large number of points are more likely to contain rebar than areas with few points. Because of the steps that are performed on the points from the classifier, it is not necessary for the classifier to be extremely accurate. This is advantageous because the Naive Bayes classifier can be trained quickly compared to its more complex alternatives like fuzzy classifiers and support vector machines.

Once the classifier has output a set of points indicating general rebar locations within the image, more precise localization must be done. Typical methods for hyperbola localization and fitting include the Hough transform or RANSAC algorithm [20, 22]. However, these methods are time consuming, which makes them less than ideal for use in real-time systems. The method used in this paper is referred to as histogram localization, and uses statistics and pixel information to accurately localize rebar signatures in GPR scan images in real-time.

Histogram localization takes three values as input: the set of points from the classifier, now referred to as $P_{IN}$, the start of the search location, $s$, and the end of the search location, $e$. The first step of this process entails a histogram of $x$ coordinates of each point being accumulated. Then, non-maxima suppresion is performed, leaving only the local maxima remaining in the histogram. Next, the highest intensity pixel along each maxima and within the search area are located. This yields a set of points. Finally, another search is performed in 5 by 5 pixel neighborhood around each point to determine where the highest intensity pixel value is. This is based on the assumption that rebar signatures are largely white, indicating the reflection of the radar off of a metalic object. A final non-maxima suppresion step can be performed to ensure that only the top points in the image are kept, in the cases where multiple points may exist on a single rebar. A detailed description of these steps can be seen in Algorithm 1.

The steps detailed in the algorithm have been tested on real bridge data that was collected for this research. The results show that the proposed method is able to accurately locate rebar within a GPR scan image, in real-time. Run time of the proposed method has been included in the results to show the ability of this system to operate efficiently in real-world situations where time is limited and accuracy is of the utmost importance. More information on results of this algorithm can be seen in Section IV.

IV Experimental Results

To assess the efficiency of the proposed method, data collected from the Pleasant Valley Bridge, on Highway 580, Reno, Nevada is used. The slow lane and shoulder were surveyed on both sides of this bridge on July 28-29, 2016 during the day as marked by yellow lines in Fig. 6.

The rebar picking results from proposed method are compared with the rebar picking results from RADAN®7. RADAN®7 is produced by Geophysical Survey System®, Inc. (GSSI). The results concentrate on rebar locating ability and are presented in terms of accuracy, precision, number of true positives, and number of false positives. A true positive in this case is a rebar being detected. A false positive is a detection being made when there is no rebar present. Accuracy is the percent of rebar that are correctly identified; $true\_positives/total\_reba$r. Precision is the percent of identified objects that are true positives; $true\_positives/(true\_positives+false\_positives)$.

In order to show that the proposed method performs well on multiple sets of real bridge data, it was tested on the Pleasant Valley Bridge and East Helena Bridge, as well as on two additional bridges. Run time has been included in the results for these bridges to show that the proposed method is capable of processing this data in real-time. The results of the proposed method on the additional bridges can be seen in Table 1.

The performance of the propsed method was compared to the performance of RADAN®7. These results can be seen in Table 2. It can be seen that the proposed method performs better than RADAN®7 on the data from the East Helena Bridge, and that the difference in performance between the two is negligable on the data from the Pleasant Valley Bridge.

After obtaining the rebar locations asscociated with their amplitude of reflected signal, the attenuation condition map of the deck is built. For example, the condition map of the Pleasant Valley Bridge is presented in Fig. 7 which provides a visual representation of the corrosive condition of the bridge deck. There are four corrosive levels: no corrosive - blue; low corrosive - green; moderate corrosive - orange; highly corrosive - red. As can be seen, this bridge is in good condition since there are only some very small red areas. Due to limit of space, results of deck image stitching, crack mapping and ER condition map are not presented here.

V Conclusions

In this paper, a novel and autonomous robotic system for bridge deck inspection has been developed. As new bridges are constantly being constructed and the condition of current bridges needs to be monitored effectively, this system can be widely applied to reduce bridge maintenance costs. Also, this system can be easily adapted by equipping it with various NDE sensors. Additionally, the proposed automated rebar detection method allows the robot to process the GPR data in realtime and generate the bridge deck condition map accurately. Future work on this system will include improved GUI integration of NDE sensors, additional invariance of the classifier to edge cases that may typically cause poor performance, and more testing on real bridge data.

Acknowledgement

The authors would like to acknowledge the Nevada Department of Transportation for their support and allowing us to access the bridge for system testing and data collection.