Visual monitoring of civil infrastructure systems via camera-equipped Unmanned Aerial Vehicles (UAVs): a review of related works

Abstract

Over the past few years, the application of camera-equipped Unmanned Aerial Vehicles (UAVs) for visually monitoring construction and operation of buildings, bridges, and other types of civil infrastructure systems has exponentially grown. These platforms can frequently survey construction sites, monitor work-in-progress, create documents for safety, and inspect existing structures, particularly for hard-to-reach areas. The purpose of this paper is to provide a concise review of the most recent methods that streamline collection, analysis, visualization, and communication of the visual data captured from these platforms, with and without using Building Information Models (BIM) as a priori information. Specifically, the most relevant works from Civil Engineering, Computer Vision, and Robotics communities are presented and compared in terms of their potential to lead to automatic construction monitoring and civil infrastructure condition assessment.

Introduction

The application of Unmanned Aerial Vehicles (UAVs), also known as drones, on project sites has exponentially grown in the past few years (ENR 2015). The rapid advances in sensing, battery, and aeronautics technologies, together with autonomous navigation methods and equipped low-cost digital cameras have helped make UAVs more affordable, reliable, and easy to operate (Liu et al. 2014). Today, large numbers of Architecture/Engineering/Construction and Facility Management (AEC/FM) firms and relevant service companies use these platforms to visually monitor construction and operation of buildings, bridges, and other types of civil infrastructure systems. By capturing very large collections of images and videos, along with methods that process the visual data into 3D models, these platforms frequently survey construction sites, monitor work-in-progress, create documents for safety, and inspect existing structures, particularly for hard-to-reach areas (Liu et al. 2014). The purpose of this paper is to provide a concise review of (1) the most recent methods that streamline the collection and analysis of the visual data captured from these platforms, and (2) methods that transform and visualize actionable performance information from the collected data, with and without using Building Information Models (BIM) as a priori information. The relevant works from different research communities are presented and their potential for automating each of the processes mentioned above for both construction monitoring and condition assessment in civil infrastructure systems are discussed.

Relation to other fields

Generating as-is 3D models of the built environment using visual data collected via UAVs, transforming the outcome into useful information about the scene, and guaranteeing the accuracy and completeness during the data collection process, have all received significant attentions from both Computer Vision and Robotics communities. However, direct application of these methods for construction monitoring and civil infrastructure condition assessment purposes is challenging due to (1) the enormous amounts of visual data that need to be frequently captured and rapidly processed (e.g. more than thousands of images are collected per flight on a construction site, which results in 3D point clouds with more than hundred million points); and (2) a large number of civil infrastructure elements that needs to be detected and characterized that all of these elements and their relevant visual data need to be analyzed in less than a few hours to be meaningful for real-time project controls. In construction and civil infrastructure domains, BIM can provide strong a priori information about geometry and appearance of scenes. Therefore, collection of visual data, and detection and analysis of individual elements would be simpler problems compared to the generic problems encountered in Computer Vision and Robotics fields. This makes the problem domain of UAV-driven construction monitoring and civil infrastructure condition assessment interesting, as it can lead to domain-specific findings which can be generalized to solve the generic problems. Such problems being closely related to several research fields encourage interdisciplinary and multi-disciplinary efforts, and it is our hope that this paper contributes to highlighting such open areas of research.

Organization of this paper

This paper starts with problem statements on (1) collection of visual data via UAVs, (2) synthesizing and processing the visual data into actionable performance information, and (3) visualizing and communicating the outcome with practitioners involved in the project (Section 2). The subsequent sections review the most relevant works on each procedure and present the open areas of research. Section 6 concludes the paper.

Review

UAV-driven visual monitoring for construction and civil infrastructure systems

The goal of a UAV-driven visual performance monitoring procedure is to (1) collect images or videos from the most informative views on a project site, (2) analyze them with or without a priori BIM to reason about performance deviations during construction (e.g. progress and quality), (3) monitor ongoing operations for productivity and safety, (4) characterize the as-is conditions of existing civil infrastructure systems, and (5) quickly and frequently visualize and communicate the most updated state of work-in-progress with onsite and offsite project participants. Figure 1 illustrates an example of next generation construction site where camera-equipped UAVs autonomously monitor the construction performance. The following sections describe the most recent research in each procedure, their challenges, and future direction for research.

Fig. 1

A vision for the next generation construction site where camera-equipped UAVs autonomously monitor work-in-progress for improving safety, quality, and productivity

Collecting informative visual data

Providing accurate performance information about the state of construction or existing conditions of civil infrastructures requires UAVs to collect visual data in form of images and videos (e.g., digital: RGB; thermal: T; depth: D; digital + depth: RGB + D) from the most relevant locations and views on a project site. To streamline this process, research in UAV-driven visual data collection need to address the following challenges: (1) autonomous or semi-autonomous path planning, navigation, and take-off and landing procedures; (2) characterization of the criteria necessary for data collection, including the configurations among the images to guarantee complete as-built information; and (3) identification of the most informative views for data collection (e.g., canonical view, top down view, etc. for appearance-based recognition of work-in-progress as in (Han et al. 2015)). Table 1 categorizes prior works based on their level of autonomy and with respect to their specific applications.

Table 1 The level of autonomy in UAV-based visual data collection for construction performance monitoring and civil infrastructure condition assessment

UAV-driven data collection in practice still relies on experienced pilots navigating the UAVs on and around project sites although there have been recent efforts on Simultaneous Localization and Mapping (SLAM) techniques by the research community. Recent studies (Fernandez Galarreta et al. 2015; Kerle 2999; Zollmann et al. 2014) and commercial systems (DJI 2015) provide autonomous navigation and data collection capabilities using GPS waypoints and pre-determined flight path. These methods are beneficial in land surveying or monitoring low-structures with large footprint. Nevertheless, a GPS-driven flight planning method that builds on existing maps has the following limitations (Lin, Han, Fukuchi et al. 2015a): (1) does not account for dynamics on a construction site and their impact on safety (e.g. the location and orientation of temporary resources such as cranes); (2) can be negatively affected by loss/interference in GPS signal at interiors or due to shadowing effects caused by nearby buildings or other structures in densely populated metropolitan areas and high-rise buildings; and (3) can be subject to navigational hazards due to the loss of calibration on magnetometer sensors in proximity to structural and non-structural steel components. In the meantime, research on SLAM techniques for UAVs and ground robots has been mainstream in the Robotics community. Using high resolution laser scanners (Zhang & Singh, S. “LOAM: Lidar Odometry and Mapping in Real-time.” Proc., Robotics: Science and Systems Conference (RSS 2014), monocular cameras (Blo, x, sch, M et al. 2010; Lui et al. 2015), and RGB-D cameras (Loianno et al. 2015), these techniques generate 3D maps of unknown scenes and localize the robot in that environment. The latest efforts such as (Michael et al. 2014) have focused on experimenting a variant of SLAM using RGD-B for scanning post-disaster buildings, yet there is almost little to no work reported on actively testing these algorithms for producing 3D maps on civil infrastructure systems for construction monitoring or condition assessment purposes. Also, methods that account for evolving structures (i.e. large numbers of shoring and frames in room partitions) and dynamic objects (i.e. equipment and human) are not reported in the literature.

As can be seen from the literature, research is mainly focused on challenge 1, and challenge 2 and 3 still lacks investigation in terms of the number of studies conducted. For example, identifying the most informative views for observing different tasks with or without BIM, and collecting visual data to monitor locations, activities of equipment, and craft workers on a jobsite are not well studied. There is an opportunity for leveraging a priori information about geometry and appearance of a site via BIM. When integrated with schedules, BIM can also report on the most likely locations for expected changes on the site to steer data collection. A BIM-driven data collection method has potential to overcome some of these challenges mentioned above such as collision avoidance with structures. Together with 4D (3D + time) point clouds, BIM definitely has potential to minimize many technical challenges facing fully autonomous navigation and data collection for the built environment.

Visual data analytics

The analytics of visual data has been a research subject for more than a decade. Many image processing, computer vision, and geometrical processing techniques are developed that can (1) generate semantically-rich 3D models from collections of overlapping images; (2) manually, semi-automatically, or automatically conduct progress monitoring, surveying, safety inspection, quality monitoring and activity analysis during construction, and (3) streamline condition assessment in existing buildings and infrastructure systems. (Cho et al. 2015; Pătrăucean et al. 2015; Son et al. 2015; Teizer 2015; Yang et al. 2015) provide thorough reviews of these techniques. These all methods can be applied to visual data collected from UAVs, yet only a few studies have validated them in such contexts. Tables 2 and 3 summarize the most recent literature from the last few years that focus on methods that are exclusively developed or applied to images and videos from UAVs.

Table 2 UAV-driven methods for construction and building performance monitoring

Table 3 UAV-driven methods for civil infrastructure condition assessment

Figure 2 categorizes these methods based on their level of automation and with respect to specific applications. As seen, prior works are mainly based on image-based 3D reconstruction procedures to analyze unordered and uncalibrated image collections. While these methods generally work better on ordered and sequential images, yet without a rigid control on the location and viewpoint of the captured images with respect to onsite construction elements, the visual content of the images may not be most suitable for certain applications. For example, for appearance-based construction progress monitoring, leveraging material recognition techniques such as Han et al. (Han et al. 2015) benefits from images that are taken in a canonical view to identify the most updated status of construction for onsite elements. This requires research that identifies the most informative views, and then techniques that can leverage such information as a priori information for data collection. In addition, a detailed assessment of work-in-progress, quality, and existing conditions requires the analysis of geometry, appearance, and interconnectivity among the construction elements. It also requires techniques for as-built and as-damaged information modeling; nevertheless, techniques that address these issues are not well studied. Ongoing efforts such as the Flying Superintendents project at the University of Illinois (Lin, Han & Golparvar-Fard 2015), and the ARIA project at Carnegie Melon University (ARIA (Team 2015)) are geared towards providing specific frameworks together with methods for construction monitoring and civil infrastructure condition assessment to address some of these gaps-in-knowledge.

Fig. 2

The level of automation in UAV-based visual analytics for construction performance monitoring and civil infrastructure condition assessment

Information visualization

Achieving effective flow of information both to and from project sites and conducting actionable analytics for construction monitoring and condition assessment require intuitive visualization of the information produced throughout the process on top of the UAV visual data - images and point clouds. While attention to visual sensing and analytics has been the mainstream of the literature, less work is conducted on interactive visualization. A recent work by (Zollmann et al. 2014) introduces an interactive multi-layer 4D visualization of information captured and analyzed through a UAV-driven procedure in form of mobile augmented reality. Their system, similar to (Karsch 2999) overlays color-coded 3D construction progress information on the physical world and adopts filtering methods to avoid information clutter and issues associated with displaying detailed augmented information. Other recent examples by (Han et al. 2015; Lin, Han, Fukuchi et al. 2015a) introduce web-based tools with scalable system architectures for visualizing and manipulating large scale 4D point cloud data, large collections of images, 4D BIM, and other project information. These methods account for level of details in data representation and dynamically consider the limited computational power and connection bandwidth for visualizing data on commodity smartphones. Since BIM is hosted on the server side of their system architecture, these tools can support pull and push of the geometry and other semantic information from BIM. This provides access to the most updated information and does not require storing BIM locally on the client device (Fig. 3). More research is still needed to map, visualize, and explore modalities of user interaction with operation-level construction performance data (e.g. locations and activities of workers and equipment) and condition assessment information (e.g. the location and characteristics of defects) through these models, both on and off site.

Fig. 3

Web-based visualization and navigation of the state of work-in-progress on construction sites using UAV-collected images, 4D point clouds, and 4D BIM: a image-based 3D point cloud models; b a nested octree structure for mapping and interactions with point cloud data; c the location and orientation of images taken via the UAV are mapped in 3D using pyramids as camera frusta; d texture mapping onto the frontal part of the camera frusta, allowing the users to interact with both images and point cloud data; e conducting linear, area-based, volumetric, and angular measurements; f color-coding changes on the construction site via point cloud data; g 4D BIM superimposed on the point cloud data wherein a location at risk (i.e. a lower reliability in achieving smooth flow of operations w.r.t. the 2-week look ahead schedule) is highlighted in yellow; and h 4D BIM is superimposed on the point cloud model and performance deviations are color-coded. All images are from the ongoing Flying Superintendents project at University of Illinois at Urbana-Champaign

Conclusions

While the application of camera-equipped UAVs provides an unprecedented mechanism for inexpensive, easy, and quick documentation of the as-built data, there are still numerous open problems for further research. Nevertheless, given the editorial limitations on the scope, this paper only reviews the most recent and relevant UAV-driven research on automating construction monitoring and civil infrastructure condition assessment. It presents high level gaps-in-knowledge and opportunities for further research on data collection, analytics, and visualization techniques. Conducting interdisciplinary and multi-disciplinary research to address such technical challenges can make an impact on enhancing the current practices of project controls and condition assessment in the AEC/FM industry. It is our hope that this manuscript can highlight some of the emerging opportunities for multi-disciplinary and interdisciplinary research on autonomous vision-based systems for construction performance monitoring and condition assessment purposes.