UAV-Based Applications of Deep Learning in Automated Inspection of Civil Infrastructure:
A Systematic Literature Review

Modern technologies such as Unmanned Aerial Vehicle (UAV)-based inspection and deep learning (DL) algorithms introduce new opportunities and challenges in Civil Engineering. To better facilitate the adoption and advancement of UAV-based detection technologies, this paper conducts a systematic literature review on a plethora of articles and performs a comprehensive investigation and comparison across four different topics: (1) investigating the technical specifications of currently utilized UAV platforms and of the employed on-board sensors, (2) summarizing the categories of inspected infrastructure and the corresponding defects, (3) collecting publicly available datasets established on infrastructure defects, (4) illustrating and comparing DL algorithms designed for defect detection. Based on the analysis of collected related work, challenges hindering the development of UAV-based infrastructure inspection, solutions, and potential future opportunities are proposed. This review is aimed at assisting researchers and practitioners to accelerate progress towards more efficient and safe autonomous UAV-based structural inspection in civil engineering.

References

Reviewed papers that have only been included in the statistics:

[1] Z. Wu et al. Applying deep convolutional neural network with 3D reality mesh model for water tank crack detection and evaluation. In: Urban Water Journal 17.8 (2020), pp. 682-695.

[2] F. Nex et al. Towards real-time building damage mapping with low-cost UAV solutions. In: Remote sensing 11.3 (2019), p. 287.

[3] A. Alzarrad et al. Automatic assessment of roofs conditions using artificial intelligence (AI) and unmanned aerial vehicles (UAVs). In: Frontiers in Built Environment 8 (2022), p. 1026225.

[4] J. Ding et al. A precision efficient method for collapsed building detection in postearthquake UAV images based on the improved NMS algorithm and Faster R-CNN. In: Remote Sensing 14.3 (2022), p. 663.

[5] J. Wang et al. Automatic detection of building surface cracks using UAV and deep learning-combined approach. In: Structural Concrete (2024).

[6] K. Lee, S. Lee & H. Y. Kim. Bounding-box object augmentation with random transformations for automated defect detection in residential building façades. In: Automation in Construction 135 (2022), p. 104138.

[7] T. Ghosh Mondal et al. Deep learning-based multi-class damage detection for autonomous post-disaster reconnaissance. In: Structural Control and Health Monitoring 27.4 (2020), e2507.

[8] Y. Wang et al. Real-time damaged building region detection based on improved YOLOv5s and embedded system from UAV images. In: IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing (2023).

[9] S. Tilon et al. Post-disaster building damage detection from earth observation imagery using unsupervised and transferable anomaly detecting generative adversarial networks. In: Remote sensing 12.24 (2020), p. 4193.

[10] M. Vlaminck et al. Region-based CNN for anomaly detection in PV power plants using aerial imagery. In: Sensors 22.3 (2022), p. 1244.

[11] J. Starzyński, P. Zawadzki & D. Harańczyk. Machine learning in solar plants inspection automation. In: Energies 15.16 (2022), p. 5966.

[12] P. Kuznetsov et al. Method for the Automated Inspection of the Surfaces of Photovoltaic Modules. In: Sustainability 14.19 (2022), p. 11930.

[13] A. Barrett et al. Machine Learning Based Damage Detection in Photovoltaic Arrays Using UAV-Acquired Infrared and Visual Imagery. In: 2024 International Conference on Unmanned Aircraft Systems (ICUAS). IEEE. 2024, pp. 264-271.

[14] D.Langenkämper et al. Efficient visual monitoring of offshore windmill installations with online image annotation and deep learning computer vision. In: Global Oceans 2020: Singapore-US Gulf Coast. IEEE. 2020, pp. 1-6.

[15] N. Kerle et al. UAV-based structural damage mapping-Results from 6 years of research in two European projects. In: Gi4DM 2019-Geoinformation for Disaster Management: ISPRS ICWG III/IVa. 2019, pp. 187-194.

[16] Y. Wu et al. Automatic railroad track components inspection using hybrid deep learning framework. In: IEEE Transactions on Instrumentation and Measurement (2023).

[17] Y. Wu et al. Hybrid deep learning architecture for rail surface segmentation and surface defect detection. In: Computer-Aided Civil and Infrastructure Engineering 37.2 (2022), pp. 227-244.

[18] P. Bojarczak & P. Lesiak. UAVs in rail damage image diagnostics supported by deep-learning networks. In: Open Engineering 11.1 (2021), pp. 339-348.

[19] H. S. Munawar et al. Civil infrastructure damage and corrosion detection: An application of machine learning. In: Buildings 12.2 (2022), p. 156.

[20] H.-H. von Benzon & X. Chen. Mapping damages from inspection images to 3D digital twins of large-scale structures. In: Engineering Reports (2024), e12837.

[21] A. Shihavuddin et al. Image based surface damage detection of renewable energy installations using a unified deep learning approach. In: Energy Reports 7 (2021), pp. 4566-4576.

[22] S. Zhou et al. Application Improvement of Deep Learning Algorithm in Small-Sized Fittings, Voltage Balancing Ring and Bare Conductor Detection of Transmission Lines. In: International Journal of Pattern Recognition and Artificial Intelligence 37.11 (2023), p. 2352017.

[23] C. Xu et al. An efficient YOLO v3-based method for the detection of transmission line defects. In: Frontiers in Energy Research 11 (2023), p. 1236915.

[24] B. J. Souza et al. Hybrid-YOLO for classification of insulators defects in transmission lines based on UAV. In: International Journal of Electrical Power & Energy Systems 148 (2023), p. 108982.

[25] G. Singh, S. F. Stefenon & K.-C. Yow. Interpretable visual transmission lines inspections using pseudo-prototypical part network. In: Machine Vision and Applications 34.3 (2023), p. 41.

[26] H. Cheng, Y. Li & Y. Li. Embankment surface crack pixel-wise identification in UAV images based on a lightweight U-Network with transfer learning. In: Structures. Vol. 58. Elsevier. 2023, p. 105640.

[27] H. Cheng et al. Embankment crack detection in UAV images based on efficient channel attention U2Net. In: Structures. Vol. 50. Elsevier. 2023, pp. 430-443.

[28] Z. Zhang et al. A Binocular Vision-Based Crack Detection and Measurement Method Incorporating Semantic Segmentation. In: Sensors 24.1 (2023), p. 3.

[29] A. C. Loerch et al. Comparing the accuracy of sUAS navigation, image co-registration and CNN-based damage detection between traditional and repeat station imaging. In: Geosciences 12.11 (2022), p. 401.

[30] G. Bhattacharya, N. B. Puhan & B. Mandal. Kernelized dynamic convolution routing in spatial and channel interaction for attentive concrete defect recognition. In: Signal Processing: Image Communication 108 (2022), p. 116818.

[31] Q. Zhu et al. Hierarchical convolutional neural network with feature preservation and autotuned thresholding for crack detection. In: IEEE Access 9 (2021), pp. 60201-60214.

[32] V. Hoskere et al. MaDnet: multi-task semantic segmentation of multiple types of structural materials and damage in images of civil infrastructure. In: Journal of Civil Structural Health Monitoring 10.5 (2020), pp. 757-773.

[33] S. Bhowmick, S. Nagarajaiah & A. Veeraraghavan. Vision and deep learning-based algorithms to detect and quantify cracks on concrete surfaces from UAV videos. In: Sensors 20.21 (2020), p. 6299.

[34] S. Egodawela et al. A deep learning approach for surface crack classification and segmentation in unmanned aerial vehicle assisted infrastructure inspections. In: Sensors 24.6 (2024), p. 1936.

[35] I. O. Agyemang et al. Autonomous health assessment of civil infrastructure using deep learning and smart devices. In: Automation in Construction 141 (2022), p. 104396.

[36] S. Jiang & J. Zhang. Real-time crack assessment using deep neural networks with wall-climbing unmanned aerial system. In: Computer-Aided Civil and Infrastructure Engineering 35.6 (2020), pp. 549-564.

[37] S. Jiang, Y. Wu & J. Zhang. Bridge coating inspection based on two-stage automatic method and collision-tolerant unmanned aerial system. In: Automation in Construction 146 (2023), p. 104685.

[38] Z.-f. Wang et al. Convolutional neural-network-based automatic dam-surface seepage defect identification from thermograms collected from UAV-mounted thermal imaging camera. In: Construction and Building Materials 323 (2022), p. 126416.

[39] F. Song, Y. Sun & G. Yuan. Autonomous identification of bridge concrete cracks using unmanned aircraft images and improved lightweight deep convolutional networks. In: Structural Control and Health Monitoring 2024.1 (2024), p. 7857012.

[40] S. Jiang, J. Zhang & C. Gao. Bridge Deformation Measurement Using Unmanned Aerial Dual Camera and Learning-Based Tracking Method. In: Structural Control and Health Monitoring 2023.1 (2023), p. 4752072.

[41] S. Jiang et al. Automatic inspection of bridge bolts using unmanned aerial vision and adaptive scale unification-based deep learning. In: Remote Sensing 15.2 (2023), p. 328.

[42] F. P. García Márquez, P. J. Bernalte Sánchez & I. Segovia Ramírez. Acoustic inspection system with unmanned aerial vehicles for wind turbines structure health monitoring. In: Structural Health Monitoring 21.2 (2022), pp. 485-500.

[43] Z. Yuqing. A Hybrid Convolutional Neural Network and Relief-F Algorithm for Fault Power Line Recognition in Internet of Things-Based Smart Grids. In: Wireless Communications and Mobile Computing 2022.1 (2022), p. 4911553.

[44] M. W. Khan et al. Real-time road damage detection and infrastructure evaluation leveraging unmanned aerial vehicles and tiny machine learning. In: IEEE Internet of Things Journal (2024).

[45] S. Feng et al. Fine-grained damage detection of cement concrete pavement based on UAV remote sensing image segmentation and stitching. In: Measurement 226 (2024), p. 113844.

[46] A. Ji et al. Image-based road crack risk-informed assessment using a convolutional neural network and an unmanned aerial vehicle. In: Structural Control and Health Monitoring 28.7 (2021), e2749.

[47] Z. Chen et al. Data-driven approaches for tornado damage estimation with unpiloted aerial systems. In: Remote Sensing 13.9 (2021), p. 1669.

[48] M. E. Mohammadi, D. P. Watson & R. L. Wood. Deep learning-based damage detection from aerial SfM point clouds. In: Drones 3.3 (2019), p. 68.

[49] M. Rahnemoonfar et al. Floodnet: A high resolution aerial imagery dataset for post flood scene understanding. In: IEEE Access 9 (2021), pp. 89644-89654

[50] C. Luo et al. Autonomous detection of damage to multiple steel surfaces from 360 panoramas using deep neural networks. In: Computer-Aided Civil and Infrastructure Engineering 36.12 (2021), pp. 1585-1599.

[51] L. Zhang et al. ESD-YOLOv8: An Efficient Solar Cell Fault Detection Model Based on YOLOv8. In: IEEE Access (2024).

[52] S. A. Fakhri et al. Pavement crack detection through a deep-learned asymmetric encoder-decoder convolutional neural network. In: International Journal of Pavement Engineering 24.1 (2023), p. 2255359.

[53] Y. Li et al. A novel approach for UAV image crack detection. In: Sensors 22.9 (2022), p. 3305.

[54] Y. Pan et al. Monitoring asphalt pavement aging and damage conditions from low-altitude UAV imagery based on a CNN approach. In: Canadian Journal of Remote Sensing 47.3 (2021), pp. 432-449.

[55] Y. Jiang, S. Han & Y. Bai. Building and infrastructure defect detection and visualization using drone and deep learning technologies. In: Journal of Performance of Constructed Facilities 35.6 (2021), p. 04021092.

[56] M.-M. Naddaf-Sh et al. Real-time road crack mapping using an optimized convolutional neural network. In: Complexity 2019 (2019), pp. 1-17.

[57] C. Feng et al. Efficient real-time defect detection for spillway tunnel using deep learning. In: Journal of Real-Time Image Processing 18.6 (2021), pp. 2377-2387.

[58] V. De Arriba López et al. Automatic non-destructive UAV-based structural health monitoring of steel container cranes. In: Applied Geomatics 16.1 (2024), pp. 125-145.

[59] P. Kuznetsov et al. Method for the Automated Inspection of the Surfaces of Photovoltaic Modules. In: Sustainability 14.19 (2022), p. 11930.

References for deep learning algorithms:

1. CNN variants:

ResNet: K. He et al. Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2016, pp. 770-778.

VGG: K. Simonyan & A. Zisserman. Very deep convolutional networks for large-scale image recognition. In: arXiv preprint arXiv:1409.1556 (2014).

Inception: C. Szegedy et al. Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2015, pp. 1-9.

Xception: F. Chollet. Xception: Deep learning with depthwise separable convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2017, pp. 1251-1258.

ConvNeXt: Z. Liu et al. A convnet for the 2020s. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022, pp. 11976-11986.

HRNet: J. Wang et al. Deep high-resolution representation learning for visual recognition. In: IEEE transactions on pattern analysis and machine intelligence 43.10 (2020), pp. 3349-3364.

EfficientNet: M. Tan & Q. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In: International conference on machine learning. PMLR. 2019, pp. 6105-6114.

ShuffleNet: X. Zhang et al. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2018, pp. 6848-6856.

MobileNet: A. G. Howard et al. Mobilenets: Efficient convolutional neural networks for mobile vision applications. In: arXiv preprint arXiv:1704.04861 (2017).

DenseNet: G. Huang et al. Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2017, pp. 4700-4708.

2. Transformer variants:

Vision Transformer (ViT): A. Dosovitskiy et al. An image is worth 16×16 words: Transformers for image recognition at scale. In: arXiv preprint arXiv: 2010.11929 (2020).

Swin Transformer: Z. Liu et al. Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF international conference on computer vision. 2021, pp. 10012-10022.

Mask2Former: B. Cheng et al. Masked-attention mask transformer for universal image segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022, pp. 1290-1299.

SegFormer: E. Xie et al. SegFormer: Simple and efficient design for semantic segmentation with transformers. In: Advances in neural information processing systems 34 (2021), pp. 12077-12090.

DEtection TRansformer (DETR): N. Carion et al. End-to-end object detection with transformers. In: European conference on computer vision. Springer. 2020, pp. 213-229.1655

3. Object detection models:

SSD: W. Liu et al. Ssd: Single shot multibox detector. In: Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14. Springer. 2016, pp. 21-37.

YOLO: J. Redmon et al. You only look once: Unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2016, pp. 779-788.

EfficientDet: M. Tan, R. Pang & Q. V. Le. Efficientdet: Scalable and efficient object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020, pp. 10781-10790.

R-CNN: R. Girshick et al. Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2014, pp. 580-587.

4. Semantic segmentation models:

FCN: J. Long, E. Shelhamer & T. Darrell. Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2015, pp. 3431-3440.

U-Net: O. Ronneberger, P. Fischer & T. Brox. U-net: Convolutional networks for biomedical image segmentation. In: Medical image computing and computer-assisted intervention-MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18. Springer. 2015, pp. 234-241.

U2-Net: X. Qin et al. U2-Net: Going deeper with nested U-structure for salient object detection. In: Pattern recognition 106 (2020), p. 107404.

SegNet: V. Badrinarayanan, A. Kendall & R. Cipolla. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. In: IEEE transactions on pattern analysis and machine intelligence 39.12 (2017), pp. 2481-2495.

DeepLab: L.-C. Chen et al. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. In: IEEE transactions on pattern analysis and machine intelligence 40.4 (2017), pp. 834-848.

CrackNet: L. Zhang et al. Road crack detection using deep convolutional neural network. In: 2016 IEEE international conference on image processing (ICIP). IEEE. 2016, pp. 3708-3712.

DeepCrack: Y. Liu et al. DeepCrack: A deep hierarchical feature learning architecture for crack segmentation. In: Neurocomputing 338 (2019), pp. 139-153.

PSPNet: H. Zhao et al. Pyramid scene parsing network. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2017, pp. 2881-2890.

Segment Anything Model (SAM): A. Kirillov et al. Segment anything. In: Proceedings of the IEEE/CVF international conference on computer vision. 2023, pp. 4015-4026.

5. Instance segmentation models:

Mask R-CNN: K. He et al. Mask r-cnn. In: Proceedings of the IEEE international conference on computer vision. 2017, pp. 2961-2969

Mask Scoring R-CNN: Z. Huang et al. Mask scoring r-cnn. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019, pp. 6409-6418.