Anti-loosening bolt looseness diagnosis method for flange node based on quick response code
Jun Luo, Kaili Li, BaoYa Shi, Shiwei Chen, Ying Wu, Lei Cheng
TL;DR
This paper introduces a method to detect bolt loosening in flange connections using QR codes and image correction techniques for improved accuracy and reliability.
Contribution
A novel bolt image correction method using QR codes and a loosening diagnosis approach that reduces camera position and rotation effects.
Findings
The proposed method effectively corrects perspective distortion in bolt images.
It reduces the impact of camera position deviation and simultaneous rotation of screws and nuts.
The method successfully detects loosening angles of anti-loosening bolts.
Abstract
Flange connections are widely used in various fields such as civil engineering, aerospace, and mechanical manufacturing due to their convenient installation and reliability. However, the bolts in the flange connections may become loose, which could affect the safety of the connections. Nowadays, the vision-based detection techniques are widely used because of its ability to provide quantitative loosening angles, high efficiency, and low cost. However, the vision-based detection techniques are susceptible to the influence of perspective angle and it is hard to provide the historical information of bolt automatically. Therefore, the quick response code is introduce to record the historical information of bolt and a novel bolt image correction method is proposed based on quick response code. Furthermore, the anti-loosening bolt looseness diagnosis method is established. Firstly, the quick…
Genes, proteins, chemicals, diseases, species, mutations and cell lines named across the full text — each resolved to its canonical identifier and authoritative record.
Click any figure to enlarge with its caption.
Figure 10
Figure 11
Figure 12
Figure 13
Figure 14
Figure 15
Figure 16
Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9- —Natural Science Foundation of Chongqing
- —General Project of Research Plan of Chongqing Administration for Market Regulation
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsImage and Object Detection Techniques · Engineering Structural Analysis Methods · Optical measurement and interference techniques
Introduction
Flange connections are widely used in various fields such as civil engineering, aerospace, and mechanical manufacturing due to their convenient installation and reliability. However, the bolts in the flange connections may become loose, which could affect the safety of the flange connections. Many bolt looseness diagnosis methods are proposed to detect the bolt looseness^1–7^, such as guided wave methods^1]– [2^, acoustic detection methods^4,6^ and vibration signals analysis methods^7^. These methods are significantly better than manual detection in terms of detection accuracy and automation, but there are still certain limitations, such as significant environmental interference and high hardware costs.
Nowadays, the vision-based detection techniques are widely used because of its ability to provide quantitative loosening angles, high efficiency, and low cost^8–16^. However, the vision-based detection techniques are susceptible to the influence of perspective angle and it is hard to provide the historical information of bolt automatically. Fortunately, the quick response code (QR code) has provided possibilities for solving above two problems. With its efficient information storage and fast recognition characteristics, QR code has been widely used in multiple fields, such as in the fields of business, industrial automation, healthcare, etc^17–22^. Therefore, the quick response code is introduce to record the historical information of bolt in this paper. The bolt number, spatial location, and historical inspection information can be stored in QR code for easy retrieval and use at any time. However, Further research is needed on how to use the QR code for distortion correction and identify the loosening angle of anti-loosening bolts.
In this paper, a novel bolt image correction method is proposed based on quick response code, and the anti-loosening bolt looseness diagnosis method is established. Firstly, the QR code is designed with three finder patterns and pasted on the end face of screw. And then, the four intersections of edge lines of the QR code is used to correct the image based on the homography-based perspective rectification method. Additionally, three finder patterns are used to rotate the image to the unified reference state to reduce the effects of camera position deviation and simultaneous rotation of screws and nuts. Finally, the anti-loosening bolt looseness diagnosis method is established by using the change in rotation angles of nut under initial status and loose status. The main contributions of this paper are summarized as follows:
- Based on QR code, an anti-loosening bolt looseness diagnosis method is proposed. The proposed method can easily read bolt numbers and initial bolt status information, which facilitates timely bolt localization and provides the bolt angle under initial state for loosening detection. Additionally, the proposed method can quantitatively detect bolt loosening with high accuracy.
- Based on QR code, a new bolt image correction method is proposed to reduce the influence of external distortion and camera position deviation. The four intersections of edge lines of QR code is used to correct the image based on the homography-based perspective rectification method. And then, the three finder patterns are used to rotate the image to the unified reference state to reduce the effects of camera position deviation. Furthermore, because that the QR code is pasted on the end face of screw, if the QR codes under initial state and loose state are both rotated to a unified reference state, the influence of simultaneous rotation of screws and nuts will be reduced.
- The proposed method was verified using a prototype flange node. The results show that the proposed method in this paper is effective and can be used to detect anti-loosening bolt looseness. Meanwhile, the potential influencing factors were also analyzed.
The remainder of this paper is arranged as follows. Related works on bolt loosening detection and QR code are presented in “Related Work”. The “Proposed Method” section mainly introduces the theory of proposed novel image correction method and anti-loosening bolt looseness diagnosis method. The “Experiments” section mainly verifies the feasibility of proposed method, including method validation and analysis of influencing factors. The “Conclusion” section contains the conclusion.
Related work
Bolt loosening diagnosis based on vision-based technique
Nowadays, the vision-based detection techniques are widely used because of its ability to provide quantitative loosening angles, high efficiency, and low cost. Kong et al. collect and align two bolt images at different times, then detect whether there is rotation during the bolt alignment process to detect the bolt looseness^8^. SY Lee et al. proposed a bolt loosening detection method based on deep learning and image processing techniques. The RCNN algorithm is used to identify bolts and crop image, and the Hough line transformation is used to estimate bolt angles^9^. TC Nguyen et al. innovatively adopted Hough transform technology to accurately identify the loosening state of bolts by analyzing the changes in nut rotation angle^11^. C Xie et al. proposed a flange bolt image correction method based on homography matrix, and the change in nut rotation angle under initial state and loosening state is used to calculate the loosening angle^14^. Jun Luo et al. proposed a novel anti-loosening bolt looseness diagnosis method based on a regular hexagonal hat glued onto the screw, and the prototype flange node experiment of the transmission tower was used for experimental verification. The proposed method can detect the synchronous rotation of the screw and nut in anti-loosening bolt, effectively reducing the occurrence of false alarms^15^. Y Ying et al. proposed an automatic bolt key points extraction algorithm based on stacked hourglass network and the bolt loosening angle can be detected by comparing the rotations of the key points before and after the bolt is damaged^16^.
QR code technology
With its efficient information storage and fast recognition characteristics, QR code has been widely used in multiple fields, such as in the fields of business, industrial automation, healthcare, etc. M Amrutkar et al. developed an inventory management system based on QR codes to encode and store product information, significantly improving the efficiency of information acquisition in the retail process^17^. PR Teja et al. designed a warehouse robot navigation system based on a QR code network laid on the ground, providing a reliable path planning solution for robots^18^. Q Wu et al. developed a new positioning system based on LED visible light and QR code technology^20^. JI Kim et al. proposed a precise indoor positioning method based on smartphone cameras and QR codes^21^.
In summary, there are limited researches on the anti-loosening bolts loosening detection, and there is a lack of rapid extraction and matching mechanisms for historical detection information, which will affect the efficiency of comparing bolt angle data between initial and loose states. Some researches proposed the bolt localization method based on character recognition technology, but there are shortcomings such as complex pasting work and very limited data information. Therefore, the QR code is a better choice. The bolt number, spatial location, and historical inspection information can be stored in QR code for easy retrieval and use at any time. However, Further research is needed on how to use it for distortion correction and identify the loosening angle of anti-loosening bolts.
Proposed method
Flange node with QR codes
In this paper, the QR codes are used to record the bolt numbers, spatial location information, historical inspection information, and so on. Therefore, one bolt corresponds to one QR code, shown in Fig. 1. The QR code is pasted on the end face of screw, and the diagonal length of QR code is slightly smaller than the diameter of screw, which ensures that QR code can closely adhere to the surface of screw. For example, the diameter of screw in Fig. 1 is 18 mm. According to the above principles, the side length of QR code can be taken as 12 mm, and the diagonal dimension is 17 mm, slightly smaller than the screw diameter. The thickness of QR code is 1 mm. The white nut and black washer are used, because that the color combination can maximize the grayscale gradient of the nut edge, improving the accuracy of nut edge line detection^23^.
Fig. 1. Construction diagram of bolts on the flange node.
The proposed bolt image correction method based on QR code
Due to the difficulty in ensuring that the camera lens is strictly aligned with the bolt plane during shooting, the captured bolt images always have perspective distortion, which will affect the identification accuracy of bolt loosening angle. The perspective distortion correction method based on homography matrix is an effective way to reduce the impact of perspective distortion.
For bolt group, the correction points used in the homography-based perspective rectification method can be extracted from the centers of all bolts. For flange node, because of the mutual occlusion of components, it is difficult to find the centers of four bolts as correction points in one image. Xie C et al. proposed a checkerboard perspective correction method^14^. However, because of the limited available space of flange nodes, pasting checkerboard is often restricted, especially there are stiffeners. Therefore, a new bolt image correction method based on QR code is proposed in this paper to reduce the influence of external distortion and camera position. The proposed method can effectively utilize QR code for perspective distortion correction without occupying additional space on flange node, making it more practical. Meanwhile, the three finder patterns on QR code can be used to rotate the image to the unified reference state to reduce the effects of camera position deviation.
While the bolt images are obtained, the captured bolt image can be cropped into several sub-images, and there is only one bolt in each bolt sub-image. And then, the bolt sub-images can be used for image correction, shown in Fig. 2.
Fig. 2. Flow chart of bolt image correction.
The steps of the proposed bolt image correction are as following.
Step 1: The image preprocessing is implemented on bolt sub-image to only extract the QR code area^24^, including image grayscale, image binarization, image erosion, image dilation, image reverse, small area deletion and connected domain processing, etc.
Step 2: The edge points of QR code are extracted using convex hull method, the four edge lines are fitted with these edge points using least squares method, and the four intersection points can be found.
Step 3: The original image can be undergoing perspective distortion correction using the four intersection points and homography-based perspective rectification method. Based on the principle of perspective projection, the image coordinate can be expressed as.
\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left\{ \begin{gathered} u \hfill \\ v \hfill \\ l \hfill \\ \end{gathered} \right\}=\left[ {\begin{array}{*{20}{c}} {{h_{11}}}&{{h_{12}}}&{{h_{13}}} \\ {{h_{21}}}&{{h_{22}}}&{{h_{23}}} \\ {{h_{31}}}&{{h_{32}}}&{{h_{33}}} \end{array}} \right]\left\{ {\begin{array}{*{20}{c}} x \\ y \\ l \end{array}} \right\}$$\end{document}where, w={x,y,l}^T^ is the world coordinate. The quantities hij are the parameters of the homography matrix, which can be calculated using the following Eq.
\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left[ {\begin{array}{*{20}{c}} {{x_1}}&{{y_1}}&1&0&0&0&{ - {u_1}{x_1}}&{ - {u_1}{y_1}} \\ 0&0&0&{{x_1}}&{{y_1}}&1&{ - {v_1}{x_1}}&{ - {v_1}{y_1}} \\ {{x_2}}&{{y_2}}&1&0&0&0&{ - {u_2}{x_2}}&{ - {u_2}{y_2}} \\ 0&0&0&{{x_2}}&{{y_2}}&1&{ - {v_2}{x_2}}&{ - {v_2}{y_2}} \\ {{x_3}}&{{y_3}}&1&0&0&0&{ - {u_3}{x_3}}&{ - {u_3}{y_3}} \\ 0&0&0&{{x_3}}&{{y_3}}&1&{ - {v_3}{x_3}}&{ - {v_3}{y_3}} \\ {{x_4}}&{{y_4}}&1&0&0&0&{ - {u_4}{x_4}}&{ - {u_4}{y_4}} \\ 0&0&0&{{x_4}}&{{y_4}}&1&{ - {v_4}{x_4}}&{ - {v_4}{y_4}} \end{array}} \right]\left[ {\begin{array}{*{20}{c}} {{h_{11}}} \\ {{h_{12}}} \\ {{h_{13}}} \\ {{h_{21}}} \\ {{h_{22}}} \\ {{h_{23}}} \\ {{h_{31}}} \\ {{h_{32}}} \end{array}} \right]=\left[ {\begin{array}{*{20}{c}} {{u_1}} \\ {{v_1}} \\ {{u_2}} \\ {{v_2}} \\ {{u_3}} \\ {{v_3}} \\ {{u_4}} \\ {{v_4}} \end{array}} \right]$$\end{document}where, {x1,y1},{x2,y2},{x3,y3},{x4,y4} are the world coordinates of the four intersection points and {u1,v1},{u2,v2},{u3,v3},{u4,v4} are the corresponding image coordinates of the four intersection points. Based on Eq. (2) and two kinds of coordinate values of the four intersection points, the quantities hij can be estimated. And then, the image can be corrected using Eq. (1).
Step 4: The image preprocessing is used again to extract only the finder patterns by controlling the size parameters of preserved connected domain^24^, three center points of finder patterns can be determined, and the triangle composed of these three center points can be constructed.
Step 5: The rotation reference point is identified by searching for the center point far from the long side of constructed triangle, named as point R. And then, rotate the image with point R as the midpoint so that the point R is located in the upper left corner of the QR code, just like the relative positional relationship of the three center points of finder patterns in the unified reference finder pattern image. Because that the QR code is located in the center of the image, the rotation angle of the image can be determined based on the position of point R in the image. Therefore, the rotation process can be carried out in the following different situations, shown in Fig. 3.
So far, the corrected image is obtained.
Fig. 3. The rotation process under different situations.
Anti-loosening bolt looseness diagnosis
After correcting the bolt image, the bolt looseness can be detected. For anti-loosening bolt, the screw and nut may rotate simultaneously. Therefore, if only the rotation angles of nut in initial and recent states are used for comparative analysis, it may lead to misjudgment. Fortunately, the bolt images in initial and recent states are both corrected using the proposed method in this paper. Because that the QR code is pasted on the end face of screw, the rotation angles of the screw in initial and recent states can be corrected to be the same, and the loose angle of bolt can be characterized by the difference between rotation angles of screw in initial and recent states.
For example, an anti-loosening bolt is shown in Fig. 4. There are two states, i.e. initial state and loose state. The 0° loose state means the screw and nut both rotated 10°. The 30° loose state means there is a relative rotation of 30° between the screw and nut. Based on the proposed image correction method, the images under different states can be corrected. The results show that the proposed image correction method can reduce the influence of simultaneous rotation of screws and nuts. Because that the nuts under initial and 0° loose states have become completely consistent, while the QR codes under initial and 0° loose states are corrected to the same state.
Fig. 4. Anti-loosening Bolt Looseness Example.
Hence, the loose angle of bolt can be calculated by Eq. (3).
\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\theta ^*}={\theta _d} - {\theta _u}$$\end{document}where, θd and θu are the rotation angles of nut in loose and initial states, respectively. The rotation angle of nut can be calculated by Eq. (4).
\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta =\frac{1}{n}\sum\limits_{{j=1}}^{n} {rem\left( {\frac{{{\theta _j}}}{{60}}} \right)}$$\end{document}where, n is the number of nut sidelines, rem() is the remainder after division, and θj is the angle between the jth nut sideline and horizontal line, shown in Fig. 5. It can be seen that there are three sets of parallel sidelines and three angles between the nut sidelines and horizontal line, i.e., θ1, θ2, and θ3. Meanwhile, the relationship between the three angles can be expressed as Eq. (5). While the nut rotate, the angles between the nut sidelines and horizontal line will change, and the bolt loose can be detected.
\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\theta _3}={\theta _2}+{60{{^\circ }}}={\theta _1}+{120{{^\circ }}}$$\end{document}Fig. 5. Schematic diagram of nut sideline angles.
Experiments
Experimental overview
A prototype flange node was used for experimental verification, shown in Fig. 6. The inner diameter and outer diameter of the flange node are 76 mm and 123 mm, respectively. There are 8 bolts and the diameter of the bolt is 20 mm. Since the proposed method in this paper is aimed at image correction and loosening diagnosis of a single bolt, one bolt on the flange node is selected for experimental verification. The side length of QR code is 12 mm and the diagonal dimension is slightly smaller than the screw diameter. The resolution of QR code image used for printing is 96dpi. In this experiment, the iPhone 14 Plus smartphone was used to capture images of bolt. The phone is equipped with a 12 megapixel high-resolution camera, which can produce clear images of 2160 pixels by 3840 pixels. The vertical and horizontal resolutions of the camera are both set as 96dpi. In order to accurately simulate the bolt loosening, several lines are drawn on the gasket and the angle between any two adjacent lines is 10°, shown in Fig. 6.
Fig. 6A prototype flange node and selected bolt.
The validation experiment of the proposed image correction method
In order to discuss the influence of different perspective direction and angle, three perspective directions and several perspective angles were considered. The vertical perspective, horizontal perspective, and bidirectional perspective were studied. The 10°, 20°, 30° and 40° perspective angles were studied for vertical and horizontal perspective. And the 10°-10°, 10°-30° and 30°-10° perspective angles were studied for bidirectional perspective.
Images obtained under a lighting intensity of 192 lx and a shooting height of 40 cm. Based on the proposed image correction method, the bolt images are corrected, shown in Figs. 7, 8 and 9. The results show that the bolt images were all successfully corrected. A square morphological structural element with a width of 3 pixels is adopted in image dilation, and a square morphological structural element with a width of 2 pixels is adopted in image erosion. The threshold value of small area deletion is set as 1000.
Fig. 7. Vertical perspective correction diagram.
Fig. 8. Horizontal perspective correction diagram.
Fig. 9. Bidirectional perspective correction diagram.
The validation experiment of the proposed anti-loosening bolt looseness diagnosis method
In order to verify the effectiveness of the proposed anti-loosening bolt looseness diagnosis method, four bolt loose states are considered, including 0°, 10°, 20° and 30° loose states, shown in Fig. 10. The 0° loose state means screw and nut both rotated 10°. The 10°, 20° and 30° loose states mean there are relative rotation of 10°, 20° and 30° between the screw and nut, respectively. Meanwhile, in order to discuss the influence of different perspective direction and angle, three perspective directions and several perspective angles were considered. The vertical perspective, horizontal perspective, and bidirectional perspective were studied. The 10°, 20°, 30° and 40° perspective angles were studied for vertical and horizontal perspective. And the 10°-10°, 10°-30° and 30°-10° perspective angles were studied for bidirectional perspective.
Fig. 10. Bolt loosening states setting.
20 images are photographed for each perspective angle under different perspective direction. Images obtained under a lighting intensity of 192 lx and a shooting height of 40 cm. Based on the proposed image correction and anti-loosening bolt looseness diagnosis method, the loose angle of bolt can be calculated.
The identified loose angles of bolt under vertical and horizontal perspective cases are shown in Figs. 11a and 12a; Table 1, respectively. The identified loose angles of bolt under bidirectional perspective case are shown in Fig. 13(a) and Table 3. For comparative analysis, the identified loose angles of bolt with uncorrected images under vertical, horizontal and bidirectional perspective cases are shown in Figs. 11b, 12b and 13b; Tables 2 and 3, respectively. The results show that the proposed method can effectively reduce the influence of perspective angles on bolt loosening diagnosis and accurately identify bolt loosening angles. For example, under 0° loose state, using the corrected images, the mean values of identified loose angles are between 0.38° and 2.38°, which generally increase with the increase of perspective angle. The identified results can accurately reflect that the nut and screw are rotating synchronously, and the bolt can be judged as not loose. However, using the uncorrected images, the mean values of identified loose angles are between 7.5° and 13.25°, which indicate that the bolt is loose, which does not match the actual situation.
Fig. 11. The identified loose angles using corrected and uncorrected bolt images under vertical perspective case. (a) Identified loose angles with corrected images, (b) Identified loose angles with uncorrected images.
Fig. 12. The identified loose angles using corrected and uncorrected bolt images under horizontal perspective case. (a) Identified loose angles with corrected images. (b) Identified loose angles with uncorrected images.
Fig. 13. The identified loose angles using corrected and uncorrected bolt images under bidirectional perspective case. (a) Identified loose angles with corrected images. (b) Identified loose angles with uncorrected images.
Table 1. The mean value and standard deviation of identified loose angles with corrected images under unidirectional perspective case.Loosening casesPerspective anglesVertical perspectiveHorizontal perspectiveMean value (°)Standard deviation (°)Maximum deviation (°)95% percentile error(°)Mean value (°)Standard deviation (°)Maximum deviation (°)95% percentile error(°)0° loose state10°0.380.892.52.50.631.082.52.520°1.131.475.02.50.751.605.05.030°1.881.565.05.01.001.222.52.540°1.631.635.05.02.382.167.55.010° loose state10°10.251.092.52.59.501.002.52.520°10.501.002.52.58.881.242.52.530°10.631.562.52.59.631.825.02.540°10.631.755.02.59.381.925.02.520° loose state10°20.380.892.52.520.000.000020°19.880.542.5019.630.892.52.530°20.631.752.52.518.751.252.52.540°19.131.632.52.520.251.925.05.030° loose state10°30.000.000030.130.542.5020°29.630.892.52.529.881.475.02.530°29.131.192.52.528.381.635.02.540°28.881.242.52.527.751.755.05.0
Table 2. The mean value and standard deviation of identified loose angles with uncorrected images under unidirectional perspective case.Loosening casesPerspective anglesVertical perspectiveHorizontal perspectiveMean value (°)Standard deviation (°)Maximum deviation (°)95% percentile error(°)Mean value (°)Standard deviation(°)Maximum deviation (°)95% percentile error (°)0° loose state10°12.630.5415.012.57.500.007.57.520°12.500.0012.512.57.630.5410.07.530°13.251.1515.015.010.001.3712.512.540°7.500.007.57.510.883.4715.015.010° loose state10°7.380.545.02.57.500.002.52.520°7.500.002.52.57.500.002.52.530°6.131.245.05.09.381.345.02.540°2.500.007.57.513.883.117.57.520° loose state10°18.251.152.52.518.381.192.52.520°17.500.002.52.519.880.542.5030°15.000.005.05.018.752.305.02.540°12.003.6710.010.020.133.915.05.030° loose state10°32.500.002.52.531.752.255.05.020°28.383.735.05.031.382.685.05.030°28.753.217.55.027.632.685.05.040°26.133.9110.07.523.632.307.57.5
Table 3. The mean value and standard deviation of identified loose angles under bidirectional perspective case.Loosening casesPerspective anglesCorrected imagesUncorrected imagesMean value (°)Standard deviation (°)Maximum deviation (°)95% percentile error(°)Mean value (°)Standard deviation (°)Maximum deviation (°)95% percentile error (°)0° loose state10°-10°0.380.892.52.57.500.007.57.510°-30°1.631.192.52.511.001.4612.512.530°-10°1.881.755.05.013.251.7917.517.510° loose state10°-10°9.880.962.52.57.500.002.52.510°-30°10.001.582.52.510.252.495.05.030°-10°9.881.852.52.56.631.435.05.020° loose state10°-10°20.631.342.52.517.500.002.52.510°-30°20.251.752.52.519.880.542.5030°-10°20.751.955.05.025.504.5812.512.530° loose state10°-10°30.250.752.52.525.000.005.05.010°-30°30.001.942.52.522.751.0910.07.530°-10°29.881.852.52.527.500.002.52.5
The results in Tables 1, 2 and 3 have proven that a higher identification accuracy can be obtained using the corrected images. In order to further analyze the statistical significance of the accuracy improvement, the Kolmogorov-Smirnov hypothesis-testing is used. The hypothesis-testing assumes that the identification errors using uncorrected images come from the distribution of dataset {Sc}. The dataset {Sc} is composed of the identification errors using corrected images from different perspective angles and different loose states. The identification errors E is defined as the difference between the test value and the true value. If the hypothesis is true, the judgment value H = 0. If the hypothesis is not true, the judgment value H = 1. The hypothesis-testing results can be found in Table 4. The results show that there is a significant difference between the identification accuracy using uncorrected images and corrected images.
Table 4. The hypothesis-testing results for identification errors using uncorrected images.Loosening casesUnidirectional perspective caseBidirectional perspective casePerspective anglesVertical perspectiveHorizontal perspectivePerspective anglesBidirectional perspective0° loose state10°1110°-10°120°1110°-30°130°1130°-10°140°11--10° loose state10°1110°-10°120°1110°-30°030°1030°-10°140°11--20° loose state10°1110°-10°120°1010°-30°030°1130°-10°140°11--30° loose state10°1110°-10°120°1110°-30°130°1130°-10°140°11--
Analysis of the influencing factors
In order to study the influence of different light intensities, camera positions and shooting distances on bolt loosening detection, three different light intensities, three different camera positions and four different shooting distances are considered. The light intensities are 192 lx, 556 lx and 1225 lx. The camera positions are right side, front and left side of the bolt. The shooting distances are 40 cm, 50 cm, 60 cm and 70 cm.
Effect test and result analysis of different lighting intensities
The light intensities 192 lx, 556 lx and 1225 lx are studied. The shooting distance is 40 cm and the camera position is front. 20 images are photographed for each case, and the captured images are shown in Fig. 14(a). Under each light intensity case, the first image is used as initial state. Therefore, the identified loose angle should be 0°. Based on the proposed method in this paper, the loose angles can be detected, shown in Fig. 14(b). In Fig. 14(b), Ave represents the average value, and Std represents the standard deviation.
The results show that the identified loose angles are extremely high stability, which indicates that the impact of light intensity on bolt loosening detection is relatively small. However, extreme lighting conditions will still have an impact on loosening diagnosis. Excessive lighting will cause issues such as overexposure and reflection in images, making it difficult to accurately recognize QR codes or their boundaries. When the light is too dim, the QR code cannot be captured clearly and cannot be effectively recognized. Therefore, the light intensities should be controlled to not be too high or too low.
Fig. 14. The captured images and identified loose angles under different lighting intensities. (a) The captured images. (b) The identified loose angles.
Effect test and result analysis of different camera positions
The camera positions are right side, front and left side of the bolt. The light intensity is 192 lx. The shooting distance is 40 cm. 20 images are photographed for each case, and the captured images are shown in Fig. 15(a). Under each camera position case, the first image is used as initial state. Therefore, the identified loose angle should be 0°. Based on the proposed method in this paper, the loose angles can be detected, shown in Fig. 15(b). In Fig. 15(b), Ave represents the average value, and Std represents the standard deviation. The maximum mean value and standard deviation of the identified loose angles are 0.15° and 0.23°, respectively. The maximum values of the maximum deviation and 95% percentile error under different camera positions are 0.5° and 0.5°, respectively. The results show that the identified loose angles are stability, which indicates that the impact of camera position on bolt loosening detection is small.
Fig. 15. The captured images and identified loose angles under different camera positions. (a) The captured images. (b) The identified loose angles.
Effect test and result analysis of different shooting distances
The shooting distances are 40 cm, 50 cm, 60 cm and 70 cm. The light intensity is 192 lx and the camera position is front. 20 images are photographed for each case, and the captured images are shown in Fig. 16(a). Under each shooting distance case, the first image is used as initial state. Therefore, the identified loose angles should be 0°. Based on the proposed method in this paper, the loose angles can be detected, shown in Fig. 16(b). In Fig. 16(b), Ave represents the average value, and Std represents the standard deviation. The maximum mean value and standard deviation of the identified loose angles are 0.45° and 0.50°, respectively. The maximum values of the maximum deviation and 95% percentile error under different shooting distances are 1.5° and 1.5°, respectively. The results show that the identified loose angles are stability, which indicates that the impact of shooting distance on bolt loosening detection is small, when the shooting distance can be controlled.
Fig. 16. The captured images and identified loose angles under different shooting distances. (a) The captured images. (b) The identified loose angles.
Conclusion
In this paper, using a vision-based technique and QR code, a novel anti-loosening bolt looseness diagnosis method is proposed. The QR code is designed with three finder patterns and pasted on the end face of the screw. And then, a novel image correction method is proposed to correct the perspective distortion for bolt image and reduce the influence of camera position. Furthermore, the anti-loosening bolt looseness diagnosis method is established by using the change in rotation angles of nut under initial status and loose status. Finally, A prototype flange node was used for experimental verification. The following conclusions can be drawn.
- The image correction experiments show that even at larger perspective angles, the proposed method can effectively correct perspective distortion in bolt images.
- The bolt loosening experiments show that the proposed method can effectively reduce the influence of perspective angles on bolt loosening diagnosis and accurately identify bolt loosening angles. For example, under 0° loose state, using the corrected images, the mean values of identified loose angles are between 0.38° and 2.38°, which generally increase with the increase of perspective angle. The identified results can accurately reflect that the nut and screw are rotating synchronously.
- The analysis of influencing factors experiments show the impact of light intensity, camera position and shooting distance on bolt loosening detection are small. However, in order to obtain stable identification results, these influencing factors should also be controlled within a reasonable range. Meanwhile, it should be pointed out that, further research is also needed to apply the proposed method to actual engineering structural sites and more influencing factors should be discussed.
- Compared with piezoelectric sensing or acoustic emission detection technology, the proposed method has slightly lower accuracy, but it only requires pasting a QR code on the screw and using a handheld camera for shooting, which has the outstanding advantages of low hardware cost, low installation requirements and high detection efficiency.
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1Luo, J. et al. Yuanlai.A novel anti-loosening bolt looseness diagnosis of bolt connections using a vision-based technique[J].scientific reports,2024,14(1).10.1038/s 41598-024-62560-8PMC 1110634038769375 · doi ↗ · pubmed ↗
- 2Ying, Y., Shichuan, W. & Wei, Z. Detection of the Bolt Loosening Angle Through Semantic Key Point Extraction Detection by Using an Hourglass Network[J] (Structural Control & Health Monitoring, 2023).
- 3Amalia, P. S., Syahputra, E. R. & Andriani, S. D. Implementation Acceptance Data Management Using Qr-Code[J]. Al’adzkiya Int. Comput. Sci. Inform. Technol. (A Io CSIT) J., 2(1). (2021).
- 4Nursahidin, N. et al. Improving Health Clinic Services Through the Application of QR code-based Digital technology[J] Vol. 8 (Jurnal Ilmu Kesehatan, 2023). 2.