Koch Et Al. (2015)

Advanced Engineering Informatics xxx (2015) xxx–xxx
Contents lists available at ScienceDirect
Advanced Engineering Informatics

journal homepage: www.elsevier.com/locate/aei
A review on computer vision based defect detection and condition

assessment of concrete and asphalt civil infrastructure q
Christian Koch a,⇑, Kristina Georgieva b, Varun Kasireddy c, Burcu Akinci c, Paul Fieguth d
a
Dept. of Civil Engineering, University of Nottingham, Nottingham, NG7 2RD, United Kingdom
b
Chair of Computing in Engineering, Ruhr-Universität Bochum, Universitätstraße 150, 44801 Bochum, Germany
c
Dept. of Civil and Environmental Engineering, Carnegie Mellon University, Pittsburgh, PA 15213, United States
d
Dept. of Systems Design Engineering, Faculty of Engineering, University of Waterloo, Waterloo, Ontario N2L 3G1, Canada
a r t i c l e i n f o a b s t r a c t
Article history: To ensure the safety and the serviceability of civil infrastructure it is essential to visually inspect and
Received 22 September 2014 assess its physical and functional condition. This review paper presents the current state of practice of
Received in revised form 17 December 2014 assessing the visual condition of vertical and horizontal civil infrastructure; in particular of reinforced
Accepted 22 January 2015
concrete bridges, precast concrete tunnels, underground concrete pipes, and asphalt pavements. Since
Available online xxxx
the rate of creation and deployment of computer vision methods for civil engineering applications has
been exponentially increasing, the main part of the paper presents a comprehensive synthesis of the state
Keywords:
of the art in computer vision based defect detection and condition assessment related to concrete and
Computer vision
Infrastructure
asphalt civil infrastructure. Finally, the current achievements and limitations of existing methods as well
Condition assessment as open research challenges are outlined to assist both the civil engineering and the computer science
Defect detection research community in setting an agenda for future research.
Infrastructure monitoring Ó 2015 Elsevier Ltd. All rights reserved.
1. Introduction cracks had resulted in five pieces of concrete blocks, as large as sev-
eral tens of centimeters, which had fallen onto the track causing
Manual visual inspection is currently the main form of assess- the train to derail.
ing the physical and functional conditions of civil infrastructure In order to prevent these kinds of accidents it is essential to con-
at regular intervals in order to ensure the infrastructure still meets tinuously inspect and assess the physical and functional condition
its expected service requirements. However, there are still a num- of civil infrastructure to ensure its safety and serviceability.
ber of accidents that are related to insufficient inspection and con- Typically, condition assessment procedures are performed
dition assessment. For example, as a result of the collapse of the I- manually by certified inspectors and/or structural engineers, either
35W Highway Bridge in Minneapolis (Minnesota, USA) in 2007 13 at regular intervals (routine inspection) or after disasters (post-dis-
people died, and 145 people were injured [1]. In the final accident aster inspection). This process includes the detection of the defects
report the National Transportation Safety Board identified major and damage (cracking, spalling, defective joints, corrosion, pot-
safety issues including, besides others, the lack of inspection guid- holes, etc.) existing on civil infrastructure elements, such as build-
ance for conditions of gusset plate distortion; and inadequate use ings, bridges, tunnels, pipes and roads, and the defects’ magnitude
of technologies for accurately assessing the condition of gusset (number, width, length, etc.). The visual inspection and assessment
plates on deck truss bridges. A different, less tragic example is results help agencies to predict future conditions, to support
the accident of a freight train in the Rebunhama Tunnel in Japan investment planning, and to allocate limited maintenance and
in 1999 that resulted in people losing the trust in the safety and repair resources, and thus ensure the civil infrastructure still meets
durability of tunnels. According to [2], the failure to detect shear its service requirements.
This review paper starts with the description of the current
practices of assessing the visual condition of vertical and horizon-
q
Handled by W.O. O’Brien tal civil infrastructure, in particular of reinforced concrete bridges
⇑ Corresponding author at: The University of Nottingham, Faculty of Engineering, (horizontal: decks, girders, vertical: columns), precast concrete
Department of Civil Engineering, Room B27 Coates Building, University Park,
tunnels (horizontal: segmental lining), underground concrete
Nottingham NG7 2RD, UK. Tel.: +44 (0) 115 84 68933.
pipes (horizontal) (wastewater infrastructure), and asphalt pave-
E-mail address: [email protected] (C. Koch).
URL: http://www.nottingham.ac.uk/civeng (C. Koch). ments (horizontal). In order to motivate the potential of computer
http://dx.doi.org/10.1016/j.aei.2015.01.008
1474-0346/Ó 2015 Elsevier Ltd. All rights reserved.
Please cite this article in press as: C. Koch et al., A review on computer vision based defect detection and condition assessment of concrete and asphalt civil
infrastructure, Adv. Eng. Informat. (2015), http://dx.doi.org/10.1016/j.aei.2015.01.008
2 C. Koch et al. / Advanced Engineering Informatics xxx (2015) xxx–xxx
vision, this part focuses on answering the following questions: (1) measure (if necessary) and record information related to damage
what are the common visual defects that cause damage to civil and defects, such as delamination/spall/patched area, exposed
infrastructure; (2) what are the typical manual procedures to rebar, efflorescence/rust staining, cracking, abrasion/wear, distor-
detect those defects; (3) what are the limitations of manual defect tion, settlement and scouring. While this list of defects comprises
detection; (4) how are the defects measured; and (5) what tools the overall list for common RC bridge element categories, such as
and metrics are used to assess the condition of each infrastructure decks and slabs, railings, superstructure, substructure, culverts
element. and approach ways, not all defects are applicable to all components.
Due to the availability of low cost, high quality and easy-to-use Table 1 highlights which defects are applicable to which com-
visual sensing technologies (e.g. digital cameras), the rate of cre- ponents and hence need to be checked for each type of component
ation and deployment of computer vision methods for civil engi- on a bridge. While some of the stated defects are visually detected,
neering applications has been exponentially increasing over the some others of them may require physical measurements for accu-
last decade. Computer vision modules, for example, are becoming rate documentation and assessment. The size of the defect plays an
an integral component of modern Structural Health Monitoring important factor in deciding if it is necessary to go beyond the
(SHM) frameworks [3]. In this regards, the second and largest part visual approach.
of the paper presents a comprehensive synthesis of the state of the In addition to the list of defects stated above, FHWA also man-
art in computer vision based defect detection and condition assess- dates that all bearings should be checked during inspection, irre-
ment of civil infrastructure. In this respect, this part explains and spective of the material type and functional type of the bridge.
tries to categorize several state-of-the-art computer vision Some of the relevant defects for bearings are corrosion, connection
methodologies, which are used to automate the process of defect problems, excessive movement, misalignment, bulging, splitting
and damage detection. Basically, these methods are built upon and tearing, loss of bearing area, and damage. Furthermore, for
common image processing techniques, such as template matching, seals and joints, inspectors focus on a specific set of defects, such
histogram transforms, background subtraction, filtering, edge and as leakage, adhesion loss, seal damage, seal cracking, debris impac-
boundary detection, region growing, texture recognition, and so tion, poor condition of adjacent deck, and metal deterioration or
forth. It is shown, how these techniques have been used, tested damage. While most of these defects can be detected visually,
and evaluated to identify different defect and damage patterns in assessing severity of the defects however needs close-up examina-
remote and close-up images of concrete bridges, precast concrete tion and measurements with suitable tools and equipment.
tunnels, underground concrete pipes and asphalt pavements. All of the existing defects on a bridge are categorized on a scale
The third part summarizes the current achievements and of 1–4 – each corresponding to the condition state of a particular
limitations of computer vision for infrastructure condition assess- element (1-Good, 2-Fair, 3-Poor, and 4-Severe). The condition state
ment. Based on that, open research challenges are outlined to assist is an implicit function of severity and extent of a defect on a com-
both the civil engineering and the computer science research component. Though such categorization of condition states provides
munity in setting an agenda for future research. uniformity for each component and effects, the actual assessment
that results in such categorization can be subjective. Table 2 pro-
2. State of practice in visual condition assessment vides some examples of guidelines provided in [4] for categoriza-
tion of the condition states of different defects. Please refer to
This section presents the state of practice in visual condition Appendix D2.3 in [4] for the complete list of guidelines for all
assessment of reinforced concrete bridges, precast concrete tun- defects.
nels, underground concrete pipes and asphalt pavements. There are typically three ways to perform manual inspection for
concrete bridge elements: visual, physical and advanced. A combi-
2.1. Reinforced concrete bridges nation of these methods is required depending on the condition of
the bridge member under consideration. During visual inspection,
As per US Federal Highway Administration (FHWA)’s recent an inspector focuses on surface deficiencies, such as cracking, spal-
bridge element inspection manual [4], during a routine inspection ling, rusting, distortion, misalignment of bearings and excessive
of a reinforced concrete (RC) bridge, it is mandatory to identify, deflection. Usually, the inspector can visually detect most of the
Table 1
Defectsa related to general bridge elements (Grey: Required; White: Not Required) [4].
a
Del/Spall – Delamination/Spall/Patched area; Exp Rebar – Exposed Rebar; Eff/Rust – Efflorescence/Rust Staining; Crack – Cracking; Abr/Wr – Abrasion/Wear; Distor –
Distortion; Settle – Settlement; Scour – Scouring.
C. Koch et al. / Advanced Engineering Informatics xxx (2015) xxx–xxx 3
Table 2
Examples of defects and guidelines for assessment of condition states [4].
Defects Condition states

1 2 3 4
Good Fair Poor Severe
Delamination/Spall/Patched None Spall: <1 in depth or <6 in Spall: >1 in & >6 in Situation worse than for Condition State 3
Area diameter diameter; unsound patched and if the inspector deems that it might
area or if signs of distress affect the strength or serviceability of the
element
Efflorescence/Rust staining None Surface white without Heavy build-up with rust Situation worse than for Condition State 3
build-up or leaching with- staining and if the inspector deems that it might
out rust staining affect the strength or serviceability of the
element
Cracking Width <0.012 in or Width 0.012–0.05 in or Width >0.05 in or spacing Situation worse than for Condition State 3
spacing >3 ft spacing 1–3 ft <1 ft and if the inspector deems that it might
affect the strength or serviceability of the
element
Abrasion/Wear No abrasion/wear Abrasion or wearing has Coarse aggregate is loose or Situation worse than for Condition State 3
exposed coarse aggregate has popped out of the and if the inspector deems that it might
but the aggregate remains concrete matrix due to affect the strength or serviceability of the
secure in the concrete abrasion or wear element
relevant defects, provided there is suitable access to the bridge ele- and electrical engineers, as both structural elements and functional
ment. However, visual inspections might not be adequate during the systems have to be assessed. However, the focus of this review is
assessment of some specific defects. For example, an inspector can on civil and structural condition assessment of precast concrete
identify visually that there is delamination when looking at a patch tunnels. Accessing the various structural elements for up-close
of concrete, but would not be able to gauge the extent and depth of it visual inspection requires specific equipment and tools. Common-
accurately by just visual inspection. Visual inspections, without uti- ly, dedicated inspection vehicles, such as Aerial bucket trucks and
lization of any other inspection techniques, are also known to be rail-mounted vehicles, equipped with, for example, cameras (used
subjective which might result in unreliable results [5,6]. for documentation), chipping hammers (used to sound concrete),
In contrast to the visual inspection, efforts during physical crack comparator gauges (used to measure crack widths), and
inspections are mainly towards quantifying the defects once they inspection forms (used to document stations, dates, liner types,
are identified visually. For example, to determine delamination defect locations and condition codes), are driven through the tun-
areas in a pier or concrete deck, physical methods, namely, ham- nel and permit the inspectors to gain an up-close, hands-on view of
mer sounding or chain drag may be used [7]. Measurements con- most of the structural elements.
cerning expansion joint openings and bearing positions are also More recently, integrated and vehicle-mounted scanning sys-
essential during the inspection and evaluation of a bridge. In some tems have entered the market. For example, the Pavemetrics Laser
cases, advanced inspection methods like those based on strength, Tunnel Scanning System (LTSS) uses multiple high-speed laser
sonic, ultrasonic, magnetic, electrical, nuclear, thermography, radar scanners to acquire both 2D images and high-resolution 3D pro-
and radiography, are used to detect sub-surface defects or for pre- files of tunnel linings at a speed of 20 km/h [11]. Once digitized
cise measurements of even surface defects [22]. the tunnel data can be viewed and analyzed offline by operators
using multi-resolution 3D viewing and analysis software that
allows for high-precision measurement of virtually any tunnel fea-
2.2. Precast concrete tunnels
ture. A different system is the Dibit tunnel scanner that is manually
moved through the tunnel [12]. It provides an actual comprehen-
Precast concrete tunnels are one example of civil infrastructure
sive visual and geometrical image of the recorded tunnel surface.
components that are becoming increasingly important when
The corresponding tunnel scanner software allows easy, quick
developing modern traffic concepts worldwide. However, it is com-
and versatile data evaluations to visualize the inspected tunnel
monly known that numerous tunnels, for example in the US, are
and manually assess its condition.
more than 50 years old and are beginning to show signs of dete-
According to [9], visual inspection must be made on all exposed
rioration, in particular due to water infiltration [8]. In order to sup-
surfaces of the structural (concrete) elements (e.g. precast segmen-
port owners in operating, maintaining, inspecting and evaluating
tal liners, placed concrete, slurry walls), and all noted defects have
tunnels, the US Federal Highway Administration (FHWA), for
to be documented for location and measured to determine the
example, has provided a Tunnel Operations, Maintenance, Inspec-
scale of severity (Table 3).
tion and Evaluation (TOMIE) Manual [8] and a Highway and Rail
Based on the amount, type, size, and location of defects found
Transit Tunnel Inspection Manual [9] that promote uniform and
on the structural element as well as the extent to which the ele-
consistent guidelines. In addition, Best Practices documents sum-
ment retains its original structural capacity, elements are indi-
marize the similarities and differences of tunnel inspection proce-
vidually rated using a numerical rating system of 0–9, 0 being
dures among different US federal states and transportation
the worst condition (critical, structure is closed and beyond repair)
agencies [10].
and 9 being the best condition (new construction) [9].
There are different types of tunnel inspections: initial, routine,
damage, in-depth and special inspections [8]. Routine inspections
usually follow an initial inspection at a regular interval of five years 2.3. Underground concrete pipes
for new tunnels and two years for older tunnels, depending on con-
dition and age. According to [9], inspections should always be There is a great deal of buried infrastructure in modern cities,
accomplished by a team of inspectors, consisting of registered pro- most of which appears to be out-of-sight and out-of-mind. Thus,
fessional engineers with expertise in civil/structural, mechanical, whereas the number of cracks or depths of potholes in asphalt
Table 3
Common civil/structural defects of concrete tunnels and respective severity scales according to [9].
Defect type/ Minor Moderate Severe

Severity
Scaling <6 mm deep 6–25 mm deep >25 mm deep
Cracking <0.80 mm 0.80–3.20 mm, or <0.10 mm (pre-stressed >3.20 mm, or >0.10 mm (pre-stressed
member) member)
Spalling/Joint Spall <12 mm deep or 75–150 mm in diameter 12–25 mm deep or 150 mm in diameter >25 mm deep or >150 mm in diameter
Pop-Outs (holes) <10 mm in diameter 10–50 mm in diameter 50–75 mm in diameter (>75 mm are spalls)
Leakage Wet surface, no drops Active flow at volume <30 drips per minute Active flow at volume >30 drips per minute
and concrete pavements may very well be the subject of water- camera sees a well-lit pipe at the sides transitioning to com-
cooler conversation, an interest in or an awareness of the state of pletely dark ahead.
underground sewage pipes is quite far removed from the percep-
tion of most citizens.
However there are two key attributes that motivate attention to Sewage pipes are subject to extensive staining and background
underground infrastructure: patterning that can appear as very sudden changes in color or
shade, giving the appearance of a crack.
1. Being buried, the infrastructure is challenging to inspect.
Since the focus of this paper is on the computer vision analysis
2. Being buried, the infrastructure is very expensive to fix or
techniques, this following overview of data acquisition is brief, and
replace.
the reader is referred to substantial review papers [15,17,18,19].
Closed circuit television (CCTV) [15,17,20,18,19,21–24] is the most
Indeed, the costs associated with sewage infrastructure mod-
widespread approach to data collection for sewage pipe inspec-
ernization or replacement are staggering, with dollar figures quot-
tion; nevertheless the sewer infrastructure which has been imaged
ed in the range of one or more trillion dollars [13].
amounts only to a miniscule fraction of perhaps a few percent [19].
There is, however, a strong incentive to undertake research and
Because the most common approach is to have a forward-looking
to develop sophisticated methods for underground concrete pipe
camera looking down the pipe, the CCTV method suffers from
inspection, due to the huge cost gap between trenchless approach-
drawbacks of geometric distortion, a significant drawback in auto-
es and the far more expensive digging up and replacement. The
mated analysis.
North American Society for Trenchless Technology and corre-
Sewer scanner and evaluation technology (SSET)
sponding No-Dig conferences worldwide demonstrate the wide-
[15,16,25,26,19] represents a significant step above CCTV imaging.
spread interest in this strategy, dating back many years [14].
The pipe is scanned in a circular fashion, such that an image of a
Direct human inspection, which is possible, at least in principle,
flattened pipe is produced with very few distortions and is uni-
for above-ground exposed infrastructure such as tunnels and road
formly illuminated. Laser profiling [17,27,28,20] is similar to the
surfaces, is simply not possible for sewage pipes because of their
SSET approach, in that a laser scans the pipe surface circularly, with
relatively small size and buried state. Thus there has long been
an offset camera observing the laser spot and allowing the three-
interest [15] in automated approaches, normally a small remote-
dimensional surface geometry of the pipe to be constructed via
ly-controlled vehicle with a camera.
triangulation.
A sewage pipe would normally be classified [16] into anticipat-
There are a few further strategies, albeit less common, for sewer
ed structures,
pipe inspection. A SONAR approach [15,28,19] has been proposed
for water-filled pipes, where most visual approaches will fail, par-
Undamaged pipe.
ticularly if the water is not clear. Ultrasound methods [29–31,17],
Pipe joints (connections between pipe segments).
widely used to assess cracks in above-ground pipes, have been pro-
Pipe laterals (connections to other pipes).
posed to allow an assessment of crack depth, which is difficult to
infer from visual images. Infrared Thermography [15,17,19] relies
And some number of unanticipated problem classes:
on the fact that holes, cracks, or water intrusion may affect the
thermal behavior of the pipe and therefore be revealed as a ther-
Cracks.
mal signature. Finally, ground penetrating radar [15,17] allows
Mushroom cracks (networks of multiple, intersecting cracks, a
the buried pipe to be studied from the surface, without the clutter
precursor to collapse).
and challenges of driving robots in buried pipes, but at a very sig-
Holes.
nificant reduction in resolution and contrast.
Damaged/eroded laterals or joints.
Because of rather substantial cost associated with data acquisi-
Root intrusion.
tion of sewer pipes, there is significant interest in maximizing the
Pipe collapse.
use of data. Prediction methods [32–35] develop statistical, neural,
or expert system deterioration models to predict pipe state, over
In common with other forms of infrastructure, the primary
time, on the basis of earlier observations.
challenge to sewage pipe inspection is the tedium of manual
examination of many hours of camera data, exacerbated by the
2.4. Asphalt pavements
sheer physical extent of the infrastructure which, in the case of
sewer pipes, exceeds 200,000 km in each of the UK, Japan, Ger-
As reported by the American Society of Civil Engineers (ASCE),
many, and the US [17]. There are, however, a few attributes unique
pavement defects, also known as pavement distress, cost US
to sewage pipe inspection:
motorists $67 billion a year for repairs [36]. Therefore, road surface
should be evaluated and defects should be detected timely to
Lighting is typically poor, since the only light available is that
ensure traffic safety. Condition assessment of asphalt pavements
provided by the inspecting vehicle, and any forward-looking
is essential to road maintenance.
There exist several techniques to detect distress in asphalt defined depending on the PCI value. This description is referred
pavements. These techniques differ in the pavement data which to as pavement condition rating (PCR). PCR classifies the pavement
is being collected and in the way this data is processed. Sensor- condition as failed, serious, very poor, poor, fair, satisfactory or
based techniques utilize devices to measure parameters of the good.
pavement surface. Visual-based techniques make use of observa-
tions of the pavement surface to identify anomalies that indicate 3. Computer vision methods for defect detection and
distress. Depending on the way of processing data, techniques assessment
are classified as purely manual, semi-automated or automated
[37]. Manual processing is entirely performed by experts, while This section presents a comprehensive synthesis of the state of
semi-automated and automated techniques require little or no the art in computer vision based defect detection and assessment
human intervention. of civil infrastructure. In this respect, this part explains and tries
Visual-based techniques consist in manually inspecting the to categorize several state-of-the-art computer vision methodolo-
road surface or employing digital images and computing devices gies, which are used to automate the process of defect and damage
to assess the pavement condition. In case of manual inspection, detection as well as assessment. Fig. 1 illustrates the general com-
trained personnel walks over the road shoulder and rates the pave- puter vision pipeline starting from low-level processing up to high-
ment condition according to distress identification manuals. The level processing (Fig. 1, top). Correspondingly, the bottom part of
disadvantage of this technique is that it is subjective despite the Fig. 1 categorizes specific methods for the detection, classification
use of manuals and it depends on the experience of the personnel. and assessment of defects on civil infrastructure into pre-process-
Also, the personnel are exposed to traffic and weather, which ing methods, feature-based methods, model-based methods, pat-
makes the inspection procedure hazardous. Another issue related tern-based methods, and 3D reconstruction. These methods,
to the manual inspection of the road service is the time required however, cannot be considered fully separately. Rather they build
to perform it. on top of each other. For example, extracted features are learned
To speed up the assessment process, pavement images are ana- to support the classification process in pattern-based methods.
lyzed instead of walking on the roads. Pavement images are Subsequently, it is shown, how these methodologies have been
obtained using downward-looking video cameras mounted on used, tested and evaluated to identify different defect and damage
sophisticated vehicles. When the images and data are analyzed patterns in remote and close-up images of concrete bridges, pre-
by human experts, the process of assessing the pavement condition cast concrete tunnels, underground concrete pipes and asphalt
is semi-automated. However, the rating of the pavement still pavements.
depends on the experience of the analyzer and the subjectivity
issue remains. 3.1. Reinforced concrete bridges
Most distress detection techniques, regardless of whether they
are manual, semi-automated or automated, depend on the pave- Much of the research in defect detection and assessment using
ment distress type. Pavement distress varies in its form and causes. computer vision methods for RC bridges have largely focused on
Commonly, distress is characterized as alligator cracking, bleeding, cracks, and to some extent on spalling/delamination and rusting.
block cracking, depression, longitudinal or transverse cracking, Many of these research studies targeted and contributed success-
patches, potholes, rutting, raveling and more. The U.S. Army Corps fully to the automation of detection and measurement of defects.
of Engineers, for example, distinguishes between 19 types of dis- More studies need to be done to improve the methods used for
tress [38]. automatic assessment as they are currently based on several
Distress types and measurements are defined in visual pave- assumptions.
ment distress identification manuals. Some of these measurements In addition to cracks, there are also other defects that are essen-
and indices vary between different countries, and federal states. tial to be detected and assessed in relation to a RC bridge. Being
Table 4 presents examples of defect assessment measurements able to detect, assess and document all defects as independent
and condition indices defined in such manuals [39–43]. As can be entities is paramount to provide a comprehensive approach for
seen, severity and extent are present in most of the manuals. The bridge inspection.
common procedure to obtain the extent value is to count the Currently, some of the other categories of defects are being
occurrences of the different severity levels for each type of distress inherently detected or assessed as part of other major dominating
for the whole segment and convert the amount of distress into dis- defects present at the using computer vision methods. For exam-
tress percentage. ple, some methods detect abrasion as part of the crack [44]. In
Condition assessment indices are calculated based on the dis- other cases, such as distortion and misalignment of bearings, no
tress measurements. Several pavement condition assessment automated method exists for detecting and assessing them. This
indices have been developed and the procedures of their calcula- clearly indicates that more research needs to be done in the direc-
tion are described in visual distress identification manuals. For tion of automating the detection and assessment of various defects.
instance, the pavement condition index (PCI) is widely used. The To be able to perform automatic assessment and condition rating
pavement condition index is a statistical measure of the pavement assignment, as a first step, it is necessary to identify the relevant
condition developed by the US Army Corps of Engineers [38]. It is a defect parameters to accurately and comprehensively represent
numerical value that ranges from 0 to 100, where 0 indicates the the defect information.
worst possible condition and 100 represents the best possible con- Below we will present the synthesis of the research done so far
dition. A verbal description of the pavement condition can be in the computer vision domain for various types of defects.
Table 4
Examples of pavement defect assessment measurements and condition indices.
Ohio [39] British Columbia [40] Washington [41] South Africa [42] Germany [43]
Measurement Severity, extent Severity, density Severity, extent Degree, extent Extent
Index Pavement condition rating Pavement distress index Pavement condition rating Visual condition index Substance value (surface)
Low-level Intermediate-level High-level

processing processing processing
Image Pre- Segmen- Feature Object Structural

acquisition processing tation extraction recognition analysis
General restoration, thresholding, features classification spatio-

computer contrast edge detection, related to of materials, temporal
vision enhancement, region growing, color, texture, objects, scene
methods noise clustering shape, motion shapes, understan-
reduction locations ding,
Pre- Feature-based Model-based Pattern-based 3D Recon-

processing methods methods methods struction
Specific Haar- Percolation-based Nearest Stereo

Median
methods wavelets models neighbor reconstr.
filtering
used for
defect Morphologic Hough Graph-based Principle Com- Optical
detection, operations transform search ponent Analysis flow
classification, Histogram Histogram of Oriented Generalized Hough Multi-view
and assess- equalization Gradients (HoG) transform reconstruction
ment of civil
Image Laplacian of Gaussian Line-tracing Neural
infrastrucure
stitching (LoG) algorithms networks
Shadow Multi-temporal
Garbor filtering Canny Image stitching
removal methods
Background Wavelet Support vector Image
substraction transform machines registration
Fig. 1. Categorizing general computer vision methods (top) and specific methods to defect detection, classification and assessment of civil infrastructure.
3.1.1. Cracking comprises images from different locations because it is important

Previously, Jahanshahi et al. [45] reviewed automatic defect to offset the environment variability associated with variable light-
detection approaches. Very recently, Rose et al. [46] reviewed ing and shading conditions at different locations on the bridge,
existing crack detection and assessment algorithms for concrete which is often the case with real world bridges. Some researchers
bridges and classified them broadly as edge detection, segmenta- also combined image-based 3D scene constructions with other
tion and percolation, machine learning methods, morphology techniques, in order to obtain depth perception that a 2D image
operations, ground and aerial robot photography, template match- lacks, to support automatic crack detection [54,55].
ing, and other techniques. Building on this categorization, we While the above algorithms demonstrated capabilities to detect
reviewed and discussed some of the existing algorithms below. cracks, it is also important in a bridge inspection to understand the
Abdel-Qader et al. [47] compared various edge detection algo- crack properties such as location, width, length and orientation,
rithms and found the Haar Wavelet method to be the most reliable because condition ratings for bridge elements are assigned based
among them, for the purpose of crack detection. However, the per- on such properties. As outputs of the process of extracting proper-
formance of edge detection algorithms on noisy image data is ties from images are quantities, it is imperative that images are
questionable, and same is the case with morphological operation mapped to the global coordinate system. This requirement stems
based methods [48]. Yamaguchi et al. [49] used scalable local per- from the likelihood that images are collected on field with varying
colation-based image processing techniques and they proved to be configurations, i.e. resolutions, positions, orientations, etc., over
efficient and accurate even for large surface images [50]. Abdel- different inspections, which is primarily due to difficulty in repli-
Qader et al. [51] used a Principle Component Analysis based algo- cating the same image capture configuration as well as a result
rithm to detect cracks on a bridge surface. In this case, the accuracy of rapid advances in camera technologies over relatively shorter
of results varied with camera pose and distance from where time periods. Towards normalizing different images to true world
images are taken. Prasanna et al. [52] developed a histogram-based scale, different researchers used techniques such as 3D pose esti-
classification algorithm and used it along with Support Vector mation, multiple image stitching or by making measurements rela-
Machines to detect cracks on a concrete deck surface. The results tive to the host structural element. In relation to that, some data
of this algorithm on real bridge data highlighted the need for acquisition systems used by researchers also had 3D pose control
improving the accuracy. Nevertheless, training data from various feature. These systems likely comprised surface-based (ground-
locations on the bridge could be used to build the classifier and based, water-based, bridge surface crawler) or aerial robots, which
testing could be done on data from a different location of similar can either, have pre-configured settings or can log accurate image
structural composition. Similarly, Lattanzi and Miller [53] devel- capture configuration dynamically.
oped an automatic clustering method for segmentation based on Targeting to achieve the goal of going beyond mere crack detec-
Canny and K-Means to achieve greater accuracy of crack detection tion, Yu et al. [56] developed a graph-based search method to
under various environmental conditions at a greater speed. Lat- extract crack properties for further assessment and used a
tanzi and Miller’s work is significant, especially if training data ground-based robot for collecting images; however, this method
needed manual input of start and end points of crack [50]. Later, Oh result in inaccurate defect quantification. They could determine if
et al. [57] demonstrated a technique implementing automatic two- spalling had occurred, and if spalling was present, they could
step: crack detection and crack tracing algorithm to be able to retrieve spall properties automatically. Their research also used
detect as well as identify crack properties, such as width and Bridge Condition Index (BCI) after quantifying the defects to map
length, and tested the developed algorithm on a real bridge. They them to condition ratings. While they could achieve reasonably
collected images with a ground-based robotic system that had con- accurate results (85% accuracy) for automatic procedures, their
trolled pan and tilt mechanisms, and used median filter for algorithm could not completely address automatic identification
smoothening in the pre-processing stage, then isolated the candi- and assessment in situations where multiple defects (e.g. spall
date crack points and applied morphological operations such as and crack) interact at the same spatial location.
dilation and thinning to maintain crack segment connectivity. As Though work on spalling detection and assessment started only
part of their study, they compared their results with Fujita, Sobel recently, the progress so far is very promising. The algorithms have
and Canny’s method. The performance of the algorithm proposed been tested with images from decks and columns. Like in the case
by Oh et al. [57] matched the other three methods in terms of of cracks, even spalling needs to be checked for at concrete joints.
eliminating shaded regions and detecting major cracks, while out- Therefore, including images from those locations will be valuable
performing them in the case of thinner cracks. for better detection and assessment performance of the algorithms.
Other researchers targeted developing crack maps. Jahanshahi
et al. [58] proposed a crack detection system to extract a complete
crack map using 3D scene reconstruction, morphological opera- 3.1.3. Other damage scenarios
tions and machine learning classifiers, and followed it up with a Zaurin et al. [3] used video imagery and bridge responses col-
robust photogrammetry-based approach to compensate for cam- lected by strain gauges and fused them together to detect loss of
era perspective errors [59]. In another recent case, Zhu et al. [60] connectivity between different composite sections, and change in
proposed a novel method involving thinning of the crack maps boundary conditions. In the process, unit influence line of the
and subsequent measurement of each crack skeleton point to the bridge is extracted and statistical outlier detection is done to dif-
crack boundary to automatically extract necessary crack para- ferentiate damage state from the baseline state. This method was
meters [50]. More recently, Lim et al. [61] proposed a Laplacian tested using a four span experimental bridge belonging to Univer-
of Gaussian (LoG) based algorithm to perform crack detection sity of Central Florida. Adhikari et al. [64] presented an change
and mapping on an RC bridge deck, and uses a mobile robotic sys- detection approach based on fourier transformation of the images,
tem that can traverse a deck surface to capture images. The robot which could useful for detecting subtle defects such as periodic
stores the spatial locations of image capture and uses robot coordi- and sudden settlement of substructure. The review of the paper
nate system to transform from image coordinate system to global also suggests no proper basis for thresholding, and the results vary
coordinate system. depending on the chosen threshold limit chosen. However, this
The results presented in most of these cases were based on method is a significant improvement over traditional change
application of their methods on bridge deck surface, or in some detection approach using the image difference, and can be used
cases image data of the beams and columns were considered. Gen- to quickly do a temporal comparison of different images. Uhl
erally speaking, most of the images used in these studies were et al. [65] developed a method to detect deflection in structural
images from simple flat and curved surfaces. However, the joints, members by applying homography mapping. Specifically, they
seals, bearings and other connections present more complex geo- implemented an automatic shape filter and a corner detector to
metry, often comprise of many sub-components and generally calculate the deflection using homography mapping between the
have varying material composition. Thus, these conditions render two views. They implemented this algorithm on an experimental
it hard to distinguish cracks from true edges. Also, as bridge inspec- set up in a lab, and also on a real bridge, and verified their results
tors commonly look out for connection related defects, algorithms with the deflection calculated using a laser scanner. The results
should be tested on images from these components. seem to be very accurate with the average difference between both
the measurements being less than 0.5%. Though the deflection is
3.1.2. Delamination/Spalling being calculated accurately, it did not address the problem of dam-
Only recently, there have been developments in the detection age localization and assessment. Kohut et al. [66] extended Uhl
and assessment of spalling on concrete surface and these works et al.’s work [65] to include damage localization and assessment
seem to have drawn inspiration from rusting detection and assess- using a wavelet transforms based analysis method to do irregular-
ment [50]. German et al. [62] considered a combination of segmen- ity detection.
tation, template matching and morphological pre-processing, both Various algorithms, related to detection and assessment of
for spall detection and assessment on concrete columns. They cracking, spalling and some damage scenarios in RC bridges, have
identified length of spalled region along longitudinal direction been discussed above, and our focus was on the progress of the
and distance between exposed reinforcement bars in the trans- computer vision research in terms of automation in detection
verse direction and developed an approach for assessing the cumu- and assessment of these defects.
lative severity of the spalling based on different enumeration levels
– (i) spalling of concrete cover without exposing reinforcement, (ii)
spalling exposing longitudinal reinforcement and that of core con- 3.2. Precast concrete tunnels
crete. The results obtained for the test images indicated spall
detection with a precision of 81.1% and a recall of 80.2% for a set In contrast to concrete bridge inspection, the image and video
of 70 images. However, they indicated that more work is needed data acquired inside a tunnel is much different in terms of artificial
to achieve more detailed categorization of spall property result, lighting and camera distance. From that perspective, it is interest-
with particular focus on spalling that exposes transverse ing to review the current state-of-the-art computer vision algo-
reinforcement. rithms for defect detection in tunnel image data. According to
Adhikhari et al. [63] presented a novel approach based on Chaiyasarno [67], automated tunnel inspection systems that cover
orthogonal transformation, using shape preserving algorithms both defect detection and condition assessment can be grouped
such as affine and projective transformation, to overcome perspec- into the following themes: detection, visualization and
tive and parallax errors of a camera during data collection that can interpretation.
3.2.1. Defect detection appear in an image patch. Potential crack features are pre-defined
In analogy to concrete bridges, the most sought after defects are based on intensity. Abdelqader et al. [51] use a Principal Compo-
cracks as they are the primary indicator of deterioration patterns, nent Principles (PCA) algorithm that reduces the dimensions of fea-
which are due to other severe causes that need to be further ana- ture vectors based on eigenvalues, and extracts cracks from
lyzed [68]. Yu et al. [56] also highlight that cracks are of particular concrete images. The images are first pre-processed by line filters
concern as they most significantly affect the state of the concrete in three directions: vertical, horizontal and oblique; then further
within a tunneling environment. processed by the PCA algorithm and classified based on the nearest
Computer vision methods for crack detection generally involve neighbor algorithm.
a pre-processing step and a crack identification step. First, in the Methods based on pattern recognition considerably rely on
pre-processing step image processing techniques are applied to training data in order to set up robust classifiers. Training and
extract potential crack features, such as edges (threshold-based validation data are usually performed by manual labeling (super-
approaches). Second, the identification step usually applies crack vised learning), which is a labor-intensive and error-prone
modeling (model-based approaches) and/or pattern recognition procedure.
techniques (pattern-based approaches) in order to classify if the
extracted features belong to crack regions. Next to methods 3.2.2. Visualization
described in the previous section, mentionable contributions that The main goal of visualization is to visually organize large
are applicable to crack detection during tunnel inspection are the image and video data sets to enhance inspection. Image stitching
described below. or image mosaicing is a common method to combine and visualize
a collection of images. In the domain of tunnel inspection, Chaiya-
3.2.1.1. Threshold-based approaches. Miyamoto et al. [69] calculate sarn et al. [74] present a system that constructs a mosaic image of
the difference in intensity between each pixel and the average the tunnel surface with little distortion. Their system obtains a
intensity of each row in an image. A pixel that differs considerably sparse 3D model of the tunnel by multi-view reconstruction [75].
from the average is said to be a crack pixel. Fujita et al. [70] use a Then, the Support Vector Machine (SVM) classifier is applied in
line filter based on the Hessian matrix to emphasize line structures order to separate image features lying on the cylindrical surface
associated with cracks before they apply thresholding to separate from those of the non-surface. The reconstructed 3D points are
cracks from background. reprojected into images for accurate cylindrical surface estimation.
The major drawback of threshold-based approaches is the ques- Jahanshahi et al. [76] create stitched images of structural systems
tion on how to choose a suitable threshold for extracting crack fea- from a specialized camera that can tilt and pan. The method
tures. The described algorithms select a threshold based on prior detects missing parts, such as bolts, when comparing images taken
knowledge. However, such methods can hardly be generalized at different times for the purpose of structural health monitoring
and may be inapplicable to the imaging conditions found in real (SHM). The method applies a machine-vision algorithm to perform
tunnel images. Moreover, they are prone to inaccuracy caused by image registration to rectify images so that they are in the same
shadows as the intensities of shadow pixels tend to have a similar coordinate frame.
brightness compared to crack pixels. In general, image stitching provides a feasible way of increasing
the field of view that cannot be achieved by a single image. Conse-
3.2.1.2. Model-based approaches. Ukai [71] developed a crack detec- quently, a wide-angle or stitched image may improve defect detec-
tion system based on the deformation of tunnel walls. Under this tion results, in particular in case of hairline cracks, since the
method, the model of a crack is characterized by eight quantities, stitched image provides a higher resolution of defects, e.g. cracks.
such as area and Feret’s occupancy rate. Subsequently, a filter is
used to remove noise. Yamaguchi et al. [49] modeled cracks based 3.2.3. Change monitoring
on the concept of percolation, which is a physical model describing Apart from detecting cracks, classifying crack patterns and asso-
the phenomenon of liquid permeation. The algorithm starts by ini- ciated sizes, it is essential to observe if cracks in tunnel liners have
tializing a seed region and then the neighboring regions are labeled changed over time and how quickly they do so. This kind of infor-
as crack regions based on the percolation process. Paar et al. [72] mation helps determine the deterioration rate of the structural
present a crack detection algorithm based on the line tracing algo- tunnel components [67].
rithm that assumes a crack is a series of short straight lines con- Lim et al. [77] propose a system for change monitoring of cracks
nected together. Again, the algorithm starts from a seed point from multi-temporal images. Their system is based on a 2D projec-
followed by searches for a line within the neighboring regions. tive transformation that can accurately determine the crack size,
Yu et al. [56] proposed a crack detection method in conjunction which is then monitored in consecutive images as the crack propa-
with a mobile robot system for automated inspection of concrete gates. Although this system that can cope with images taken from
cracks in tunnels. Their method calculates the length, thickness different viewpoints, it requires explicit user input for the control
and orientation of concrete cracks through a graph search; howev- points, which makes the system unscalable for a large number of
er, it requires the crack’s start and end point to be manually pro- images. Chen and Hutchinson [78] propose a framework for con-
vided. Moreover, the robot is required to maintain a constant crete surface crack monitoring and quantification. Their method
distance from the tunnel wall in order to achieve accurate mea- is based on optical flow in order to track the movement of cracks.
surements of the damage properties. This system claims to have However, current solutions related to monitoring cracks or anoma-
an overall detection accuracy rate of 75–85% and a measurement lies rely greatly on some degree of user input [67].
error of recognized cracks of less than 10%.
According to [67], model-based methods for crack detection 3.3. Underground concrete pipes
strongly rely on user input to initialize the seed pixels. Conse-
quently, hairline cracks may not be detection because users may Deplorably, on the basis of a search of sewage pipe inspection
be unable to identify the seed pixels. Due to reliance on the user methods currently offered by North American contractors, most
input, these methods may not be scalable. buried pipe inspection continues to be manual and CCTV based,
implying a slow inspection process subject to operator fatigue
3.2.1.3. Pattern-based approaches. Liu et al. [73] apply a Support and boredom. Although this limitation is frustrating, it strongly
Vector Machine (SVM) classifier to determine if crack features motivates continued research work on machine intelligence and
computer vision in this application, and is the driving motivation segmentation [26] and background subtraction [18] for foreground
for this section. There have been significant with computer vision object extraction, methods of image registration [18] and optical
contributions to pipe inspection, in whole integrated systems such flow [24] for the tracking and association of objects in successive
as PIRAT [28,18], KARO [18], and AIMP [18,79], and the mapping video frames, particularly relevant in CCTV imaging. More
the underworld (MTU) project [19]. advanced methods include texture-based methods, including co-
The computer vision analysis of underground concrete sewer occurrence [21] and histograms of oriented gradients [23], and
pipes has much in common with other forms of infrastructure. In multi-resolution or wavelet-based approaches [29,17]. Not all of
particular, all of the parallel sections in this paper discuss aspects these methods can be described here, and the reader is referred
of crack detection, hole detection, and the classification of cracks to a comprehensive review [81].
into different forms or degrees of severity: multiple cracks, net-
worked cracks etc. The forms of concrete deterioration in different 3.3.2. Parametric models
parts of infrastructure do, after all, share a great deal in common. In principle, any object which we can recognize in an image,
As discussed in Section 2.3 and in review articles [15,17–19], an such as a crack, hole, or joint, can be modeled parametrically, with
unusually wide variety of possible imaging modalities has been parameters explicitly describing properties such as width, length,
developed for buried pipe inspection. In terms of the role of com- radius, and color. The strength of parametric models lies in their
puter vision, we will focus our discussion on the most widespread explicit nature, being relatively easy to understand and diagnose,
methods, which have seen the most attention in the literature, however their limitation lies in their limited generalizability: in
namely the CCTV, SSET, and laser profiling methods. Other practice, any special case for which a given model is unprepared
approaches, such as SONAR, ultrasonics, and ground penetrating leads to a further iteration with a newly revised model addressing
radar do produce image-like data, but of a too specialized nature that case, and after repeated such iterations leading to ugly, clunky
to consider here. models containing a variety of exceptions.
The analysis of buried sewage pipes possesses certain unique Given an explicit model, the most fundamental, albeit slow,
aspects which influence the associated computer vision strategy: approach to detecting such objects in an image is using a general-
ized Hough transform [82,83]. Essentially the Hough transform is a
Lighting: The pipes are buried, dark and, depending on the matched filter, placing the model in all possible parametric permu-
modality of imaging, there may be constraints on the lighting tations at all points in the image and asking regarding degree of fit.
possible, particularly in the case of CCTV imaging. If the number of parameters is sufficiently few, say two parameters
Patterned background and contrast: Sewage pipes suffer from describing the position plus one or two parameters describing size
significant degrees of deposits and staining, which may be dark, and shape, then the Hough approach may be possible, but given
affecting image contrast, or may be highly and irregularly pat- five or more parameters the Hough search space becomes far too
terned, looking very much like any of a number of sewage fail- large to search densely, and optimization approaches are needed.
ure classes – holes, single cracks, networks of cracks, root Significant challenges for parametric approaches arise, by
intrusion, etc. definition, for those objects which cannot be well modeled. So
Limited quality and quantity of data: The slow, expensive whereas a joint (line) or lateral (circle) is relatively simple, a crack
approach to data collection strongly limits the total amount of is more challenging but may be modeled as a set of connected line
data available for machine learning. Furthermore the lack of segments, but a model to describe the wide range of appearances
standardization – varied methodologies of imaging, machine of root intrusions is very difficult. Most parametric computer vision
standards, concrete pipe standards, concrete pipe contents models focus on crack detection, such as modeling a crack as being
and staining – make it challenging to learn broadly applicable darker or having a higher variance than its immediate surround-
approaches. ings [25] or as a set of segments [22].
The methods of image analysis in the literature mostly involve 3.3.3. Morphology
feature extraction or modeling, both of which are widely used in Image morphology represents image shape on the basis of
computer vision and machine learning. Feature extraction [80] is mathematical operations such as shape erosion (shrinking) and
the crucial bridge between a raw image and an information-rich dilation (growing). The morphological approaches are more limit-
feature vector that can be used for classification. The related prob- ed than parametric ones since, in principle, a parametric model
lems of image modeling fall into three categories in the context of can encode any imaginable behavior, however the strength of mor-
pipe inspection, from the most specific to the most abstract: of phological approaches is their elegance and operating in a manner
parametric/explicit models, morphology/shape-based models, similar to humans.
and implicit/black-box models. Any morphological operation is described or controlled through
a structuring element, normally a relatively simple shape, such as a
3.3.1. Feature extraction line, a rectangle, or a disc, which controls the extent to which a
Methods of pattern recognition and classification, such as a sup- given pixel in the image affects its neighbors in dilating or eroding.
port vector machine or nearest neighbor classifier [80], expect to Many textbooks and tutorial papers have been written [84,85] and
be given a vector of values describing the object to be classified. the interested reader is referred to them for greater background.
An image, containing thousands to millions of pixels, represents Much of pipe inspection is on the basis of binary (light/dark)
data in far too dilute a form to be classified, since computation primitive shapes, making image morphology a natural tool. The
time and training data requirements are exponential in the number most basic shapes are elongated (cracks, joints) and round (holes,
of dimensions. Feature extraction is essentially dimensionality laterals), and so analysis can proceed on the basis of one or more
reduction; in the context of analyzing images, computer vision round and one or more rectangular structuring elements. Recent
has developed a vast range of approaches for extracting salient uses of morphological approaches in buried pipes can be found
features. in Sinha et al. [16], Su et al. [86], and Halfawy et al. [22].
Because buried concrete pipes are patterned and poorly lit,
robust feature extraction is an essential step and appears through- 3.3.4. Neural models
out the pipe inspection literature. Methods include edge detection There has been a huge resurgence in computer vision interest in
[25,22] or the Hough transform [22] for edge/line detection, image neural-like models, particularly in the area of deep belief networks
[87]. The key advantage of a neural approach is that all stages of differently well according to varying lighting conditions and shad-
the problem – contrast enhancement, feature extraction, texture/ ows. Fig. 2 illustrates the so-called checker shadow illusion [92].
shape analysis, classification – are machine learned all at once, in Square A looks darker than square B, but their pixel intensities
an integrated fashion. If the machine learning optimization con- are equal. This means, humans might be able to easily identify
verges well, then the integrated approach can offer robust an asphalt crack in an image because it appears darker compared
classification. to the local background. Computers, however, may fail as they
On the other hand the sewage pipe problem, with huge num- sometimes solely rely on global intensity values.
bers of images and a wide range of background patterning and tex- Several solutions to the non-uniform lighting problem have
ture, is a very large nonlinear optimization problem for which been proposed. Varadharajan et al. [93] select only images that
convergence may be poor. Neural-like methods are essentially were taken during daytime and when the weather was overcast
black-box in nature, and therefore the actual effect or role of indi- or mostly cloudy, so that the lighting conditions are good. The dis-
vidual parameters is exceptionally hard to understand, in contrast advantage of this approach is that the selection process is also
to parametric models where the researcher can understand the time-consuming and all captured images must be saved before
operations of different parts of the algorithm and where, although selection and processing, which results in large amounts of data
parameters would ideally be machine learned, in principle the that is stored. Cheng [94] proposed a method to convert all images
parameters could be tuned by hand on the basis of an understand- to a standardized background. For that purpose, a frame is split
ing of their effect. into rectangular windows. The average light intensity of the pixels
Nevertheless, the limitations of the preceding paragraph in the windows is calculated for each window. Notably low average
notwithstanding, neural approaches have seen rather significant values are then replaced by the average value of the neighbor win-
application in buried pipe inspection. In most cases, the neural net- dows. Finally, multipliers are generated based on the average val-
work is preceded by computer vision approaches for feature ues. The multipliers are interpolated for each pixel so that all
extraction, followed by neural learning [29,27,20,21] or neuro- intensities vary around a base intensity. Zou [95] proposed a geo-
fuzzy approaches [26,88]. desic shadow-removal algorithm to remove the pavement shad-
ows while preserving the cracks in images.
3.3.5. 3D reconstruction Another issue related to distress detection in pavement images
A final contribution from computer vision relates to the three is the presence of lane-marking on the images. Nguyen et al. [96]
dimensional reconstruction of a buried pipe, as a direct geometric detect lane-marking regions and do not consider these regions
detection of deep cracks and holes, rather than indirectly through for the distress detection. First, a binary image is obtained by
visual appearance. The computer vision literature has developed applying a threshold. Second, the probabilistic Hough Transform
a vast range of methods for 3D reconstruction, most notably shape is used to detect lines on this binary image. Lane-markings are
from shading and stereo vision, both relatively complex problems. detected based on the orientations and dimensions of these lines.
In contrast, the instruments for pipe inspection employ a laser and A range of techniques are applied to eliminate noise or for
generate 3D shape one dot at a time, a far more constrained prob- image enhancement. Lokeshwor [97] and Radopoulou [98] use
lem and relatively simple compared to 3D scene reconstruction median filtering and morphological operations (erosion, dilation,
from images. opening, closing). Li [99] applies Gaussian smoothing for further
The use of laser reconstruction is widespread in computer denoising. Varadharajan [93] calculates the blur magnitude in
vision, to generate 3D models of heads, limbs for prosthetics, or the images and considers for assessment only images for which
objects for 3D printing. For pipe inspection, methods for 3D recon- the blur-score is below a certain threshold. In some cases it might
struction based on laser illumination are developed in Duran et al. also be beneficial to compress the images to reduce the size and
[27,20] and Kawasue et al. [89]. computation time, as done by Salman [100].
3.4. Asphalt pavements 3.4.2. Defect detection

Several methods have been proposed, which are capable of
3.4.1. Pre-processing detecting different types of distress in pavement images. Zhou
To automatically detect distress on pavement images, it is et al. [101] use wavelet transform to decompose an image into
required to perform some preprocessing of the images. A common approximation and detail coefficients. The detail coefficients repre-
problem is that images are taken under different weather condi- sent distress in the pavement images. Zhou also proposed three
tions or daytime and may contain shadows of trees. As a result statistical criteria and a norm of pavement distress quantification,
non-uniform lighting is present in the images. Many of the meth- which can be used as an index for pavement distress evaluation.
ods for pavement distress detection are based on the assumption Lokeshwor et al. [102] developed an algorithm which applies seg-
that distress pixels are darker than the background. Wang [90] mentation of distress pixels from the background pixels using an
and Tsai et al. [91] have concluded that such methods perform adaptive thresholding technique. User defined decision logic based
Fig. 2. The checker shadow illusion [88]: The squares marked A and B share the same grey intensity (Ó1995, Edward H. Adelson).
on the area covered by the distress pixels categorizes video frames automating the detection of potholes can be divided into 3D recon-
as frames with distress or frames without distress. Most detection struction-based, 2D vision-based methods. Detection methods that
methods are developed for a specific type of distress. Some of the are based on a 3D reconstruction of the pavement surface rely on
methods are presented below. 3D point clouds provided by stereo-vision algorithms using a pair
of video cameras. Also there are hybrid systems available that use
3.4.2.1. Cracks. As cracks are the most common distress type, a digital cameras to capture consecutive images of lines projected by
plenty of crack detection algorithms have been developed and pre- infrared lasers [111]. A stereo-vision based surface model for com-
sented. In particular, methods for real time crack analysis prehensive pavement conditioning has been proposed by Wang
[103,104], crack classification [105] crack depth estimation from [112] and Hou et al. [113]. With the availability of a 3D point cloud,
vision [106], and automating crack sealing have been presented Chang et al. [114] have presented a clustering approach that can
[107,108]. quantitate the severity and coverage of potholes and Jiaqiu et al.
Most of the algorithms for crack detection are based on the [115] have created a method for identifying, locating, classifying
assumption that crack pixels are darker than the surroundings. and measuring sag deformations like potholes and depression.
Based on statistical measures of the pixel intensities, thresholding The drawbacks of stereo-vision-based approaches are that they
methods that classify pixels as crack or non-crack pixels are require a complete 3D reconstruction of the pavement surface
applied. Tsai et al. [91] have made a critical assessment of distress and that the procedure of matching points between the two views
segmentation methods, in particular statistical thresholding, Can- is quite challenging due to the very irregular texture and color of
ny edge detection, multiscale wavelets, crack seed verification, the pavement surface.
iterative clipping methods, and dynamic optimization based meth- Karuppuswamy et al. [116] integrated a vision and motion sys-
ods. Koutsopoulos et al. [109] developed an algorithm for crack tem to detect simulated potholes. Their approach detects potholes
image segmentation based on a model that describes the statistical in the center of a lane. However, it relies on computer generated
properties of pavement images. (simulated) potholes that are larger than 2 ft in diameter and white
Huang et al. [104] also proposed a classification method. An in color. The latter are simplified assumptions that do not reflect
image is divided into cells. Depending on the contrast of each cell realistic pavement conditions. Jahanshahi et al. [117] used a depth
to its neighbor, the cells are classified as crack or non-crack cells. sensor to detect and quantify defects in pavements. Based on the
However, a limitation of the method is that it is hard to find a uni- depth values of the pixels, pixels are classified as deep or flat using
versal contrast threshold [91]. thresholding. Then, the maximum depth of the defective regions is
Salman et al. [100] proposed an algorithm which uses a Gabor computed. However, the limitation of the proposed approach is
filter. The preprocessed pavement image is convolved with the fil- that the data acquisition system, which is the Kinect sensor, is
ter and the real component of the result image is thresholded to designed for indoor use. As a result, all the captured depth values
generate the binary image. Binary images resulting from different- are zero when the Kinect is exposed to direct sunlight.
ly oriented filters are combined and an output image is produced. Koch et al. [118] also presented a computer vision based
The output image contains detected crack segments. approach for pothole detection in asphalt images. Based on sur-
Moussa and Hussain [110] presented an approach for automatic rounding shadows, elliptic shape and grain surface texture, the
crack detection, classification and parameter estimation based on method identifies potholes in images. Image segmentation, shape
machine learning. They apply Graph Cut segmentation to segment approximation, and texture comparison are performed in this
an image into crack and background pixels. A binary vector is created order. The image is divided into defect and non-defect pavement
after segmentation. Seven features are extracted from the vector for regions using histogram shape based thresholding and the triangle
classification purposes. Then, a Support Vector Machine is used to algorithm proposed by Zack et al. [119]. The shape of the pothole is
classify the crack type in transverse cracking, longitudinal cracking, approximated by applying morphological thinning and elliptic
block cracking or alligator cracking. Moussa and Hussain also pre- regression. Finally, the surface texture of the pothole candidate
sented an approach to compute the crack extent and severity based region is compared to the non-defect pavement region using spot
on the length and the width of the crack in the image [110]. filter responses. The region is determined as a pothole if the region
Varadharajan et al. [93] also use machine learning. They assume inside the pothole candidate is coarser and grainer than the one
input images which can contain background, such as cars, traffic outside. Koch et al. extended the method with video processing
signs and buildings. First, the ground plane is segmented out from [120]. Using the described pothole detection method, potholes in
the rest of the image. After that, feature descriptors are computed a sequence of pavement images are counted.
based on the color and texture of the preprocessed pixels. A total of
nine features and data obtained from human annotators are used
3.4.2.3. Patches. Cafiso et al. [121] observed that pixels which
to train a Support Vector Machine which classifies the images.
belong to patches have different gray levels from the pixels which
Li et al. [99] partition the image into crack regions and regions
belong to the background. They use a clustering method to analyze
without cracks using the difference value between the maximum
the image with respect to patches.
and the minimum grayscales of an image region. Then, the fore-
Radopoulou et al. [98] detect patches in pavement images by
ground is separated from the background by segmenting with
applying morphological operations. Patch regions are segmented
Otsu’s method and the images are classified using binary trees
based on the assumption that patch pixels have greater intensities
and back propagation neural networks.
than pixels belonging to the background. Then, texture information
Zou et al. [95] analyze the intensity difference in regions of the
is utilized and four different filters are applied. Subsequently, fea-
image to determine whether the pixels belong to cracks or not.
ture vectors of both intact and patch regions are constructed and
After that, using tensor voting, a crack map is produced. In the
compared after the convolution of the image with the filters.
crack map the probability of the pixels that are likely to be located
along long crack curves is enhanced. The cracks in the image may
sometimes be disconnected, so Zou et al. connect the crack parts 4. Achievements and challenges
with the help of an edge pruning algorithm.
This section summarizes the current achievement and open
3.4.2.2. Potholes. Usually, potholes also differ significantly from the challenges of computer vision for infrastructure condition assess-
background surface. Current computer vision research efforts in ment. A corresponding overview regarding the level of automation
in defect detection and condition assessment is presented in learning and the lack of standardization on defect patterns prevent
Table 5. those methods to perform reasonably well. In addition, detection
models with few parameters have limited generalizability, where-
4.1. Achievements as models with many parameters fail in environments with a wide
range of background pattern and texture due to the poor conver-
When looking at defect detection and condition assessment of gence of inherent non-linear optimization problems.
reinforced concrete bridges – classified as both vertical and With respect to asphalt pavement monitoring, natural weather
horizontal civil infrastructure – it can be concluded that the cur- conditions and the daytime determine the success of available
rent state-of-the-art computer vision based methods contribute computer vision based defect detection and assessment methods.
successfully to the automation of detection and measurements of Shadows from trees, for example are very natural and prevent sev-
defects. The detection, localization and properties retrieval of both eral methods, which usually work well in good lighting conditions,
concrete cracks and concrete spalling is to a very large degree auto- to perform reasonably well in real environments. Moreover, many
mated. Spalling defects can even be quantified and to some extend algorithms endeavor to perform real-time and therefore are based
be mapped to condition ratings. Other important achievements on some kind of thresholding. However, these methods are not
include the ability of computer vision based methods to successful- robust enough for image data with average image quality in prac-
ly support the detection of connectivity losses between composite tice as it is hard to find universal thresholds. Consequently, fully
sections, changes in boundary conditions, changes in substructure automated and comprehensive pavement distress detection and
settlements and deflection of structural members. The accuracy of classification in a real-time environment has remained a challenge.
vision based deflection detection can even compete with methods Also, there is no comprehensive and robust method available to
employing high accurate laser scanners. determine the severity level of distress for defect and condition
With regard to very long horizontal civil infrastructure, such as assessment of asphalt pavements.
precast concrete tunnels, underground concrete pipes and asphalt In general, reliable defect detection and condition assessment of
road networks, it is found that respective data collection technolo- civil infrastructure must be based not only on visual inspection
gies are fully automated. Moreover, available computer vision methods. First, computer vision methods work under the principle
based algorithms successfully support the automation of detecting ‘‘What you see is what you can analyze.’’ This means, that scenes
and localizing defects, such as cracks and joint spalling in concrete under observation have to be sufficiently illuminated to make
tunnels; cracks, holes and joint damage in concrete pipes; and computer vision methods work. Visible shadows, for example,
cracks, potholes and patches in asphalt pavements. In case of might have a significant impact on the capability of CV methods.
bridge and tunnel inspection, computer vision based visualization In case of pothole detection shadows support the process, where
methods (e.g. image stitching) successfully assist in defect detec- in cases of 3D reconstruction they hinder the procedure. Moreover,
tion and assessment as they improve the defect detection results the internal condition of infrastructure components cannot be cap-
due to better resolution. Concerning asphalt pavements, the crack tured, thus neither assessed using visual methods. On top of visual
properties retrieval procedure (type, with, length) is fully automat- assessment techniques (whether manual or CV-supported), other
ed and some computer vision based distress quantification mea- advanced in-depth inspection methods (so-called Non-destructive
sures have the potential to be converted to indexes for distress evaluation (NDE) methods) are required to assess the overall con-
assessment. dition, such as sonic, ultrasonic, magnetic, electrical, nuclear, ther-
mography, radar technologies. However, defects on the surface are
4.2. Challenges good indicators of the overall condition as they are part of many
visual condition assessment manuals. Second, the data quality
Concerning computer vision supported concrete bridge inspec- plays an important role in terms of noise, distance and perspective
tion, it has to be mentioned that the process of image and video to the object of interest and the corresponding image resolution.
data collection is not yet fully automated. In terms of crack detec- For instance, if one wants to detect a crack of 1 mm width, he or
tion and assessment, existing methods need to be improved as she has to make sure that this 1 mm is mapped to a least 1 image
performances on noisy data are questionable and accuracies vary pixel. Third, a number of safety risks are associated with working
with camera pose, camera distance and environmental conditions at certain heights and under heavy traffic. In this case, however,
(lighting and shading at different locations). Moreover, several emerging remote-controlled unmanned aerial vehicles (UAV)
methods still require a significant amount of manual user input. might be a good practical solution for this issue. Forth, the opera-
In general, most of the methods assume images from simple flat tion of cameras always has to face privacy issues when monitoring
and curved concrete surfaces, so that they may fail in cases of more public scenes, such as bridges and roads. Thus, it is recommended
complex geometries and material, such as joints, seals and bear- avoiding people in image and video data.
ings. Accordingly, there are currently no methods available that In summary, the authors conclude that more studies need to be
support the detection and assessment of bearing distortion and conducted to improve the methods and algorithms for integrated
misalignment. condition assessment. It is currently not possible to detect, mea-
When looking at underground civil infrastructure, such as tun- sure assess and document all different defects as independent enti-
nels and pipes, it is concluded that poor lighting conditions, ties to provide an integrated and comprehensive approach for
irregularly patterned background and contrast as well as limited bridge, tunnel, pipe and asphalt inspections. This is mainly due
data quality and quantity impose the most significant problems to the unsolved problem of identifying and assessing multiple
when dealing with computer vision based approaches to defect interacting defects at the same location and the lack of standard-
detection and assessment. With respect to lighting, common meth- ization in identifying relevant defect parameters to comprehen-
ods either use prior knowledge, thus can hardly be generalized or sively represent defect information. Moreover, no publically
they rely on some degree of manual input and therefore do not available large datasets exist to leverage supervised learning meth-
scale well. More recent methods that use machine learning strong- ods for the robust detection and classification of several infrastruc-
ly rely on training data to create robust classifiers. Usually, the ture defect types.
training process is based on supervised learning concepts (manual The following listing highlights the key research questions that
labeling) and is therefore labor-intensive and error prone. With have to be addressed by future research both in the civil engineer-
regard to pipe inspection, the limited amount of data for machine ing and computer science community in order to take the quality
Table 5
Level of automation in computer vision based defect detection and condition assessment: (+) achieved, () partially achieved, ( ) not achieved yet.
of computer vision based defect detection and condition assess- on a comprehensive synthesis of the state of the art in computer
ment of civil infrastructure to the next level: vision based defect detection and condition assessment of civil
infrastructure. Several methodologies have been described and
How can we comprehensively detect, measure and assess inter- categorized, and literature on respective tests and evaluations on
acting defect patterns at the same location to support integrat- the current performances to detect and measure different defect
ed condition assessment of civil infrastructure? and damage pattern in remote and close-up images of buildings,
How can we generalize available detection models to adequate- bridges, roads, pipes and tunnels has been presented. In the third
ly and universally address realistic environmental conditions, part of this paper the current achievements and limitations of com-
such as noisy image and video data, varying lighting conditions, puter vision for infrastructure condition assessment have been
different surface geometries and materials, and different cam- summarized. Finally, open research challenges have been outlined
era poses and distances? to assist both the civil engineering and the computer science
How can we limit the amount of manual user input to improve research community in setting an agenda for future research.
the level of automation from poor defect detection to sophisti-
cated defect and condition assessment?
How can we create sufficiently large, publically available and References
standardized datasets to leverage the power of existing super-
[1] National Transportation Safety Board, Collapse of I-35W Highway Bridge,
vised machine learning methods for detection, classification Minneapolis, Minnesota, August 1, 2007, 2008 <http://www.dot.state.mn.us/
and assessment of defects? i35wbridge/ntsb/finalreport.pdf>.
How can we create unsupervised machine learning methods [2] T. Asakura, Y. Kojima, Tunnel maintenance in Japan 18(2–3) (2003) 161–169.
[3] R. Zaurin, F.N. Catbas, Integration of computer imaging and sensor data for
(online learning) for efficient training and on-demand updating 912 structural health monitoring of bridges, Smart Mater. Struct. 19(1)
of model parameters in defect detection and assessment (2010).
models? [4] AASHTO Publication, Manual for Bridge Element Inspection, 2013.
[5] Z. Zhu, S. German, I. Brilakis, Detection of large-scale concrete columns for
automated bridge inspection, Autom. Construct. 19 (8) (2010) 1047–1055.
[6] FHWA Report, Reliability of Visual Inspection for Highway Bridges, 2001.
5. Summary
[7] NHI-FHWA Online Course, Introduction to Safety Inspection of In-Service
Bridges.
To ensure the safety and serviceability of civil infrastructure it is [8] Federal Highway Admistration (FHWA), Tunnel Operations, Maintenance,
essential to visually inspect and assess its physical and functional Inspection and Evaluation (TOMIE) Manual, 2011.
[9] Federal Highway Administration (FHWA) and Federal Transit Admistration,
condition, either at regular intervals (routine inspection) or after Highway and Rail Transit Tunnel Inspection Manual, U.S. Department of
disasters (post-disaster inspection). Typically, such condition Transportation, 2005.
assessment procedures are performed manually by certified [10] National Cooperative Highway Research Program Project 20-68A, Best
Practices for Roadway Tunnel Design, Construction, Maintenance,
inspectors and/or structural engineers. This process includes the Inspection, and Operation, 2011.
detection of the defects and damage (cracking, spalling, defective [11] Pavemetrics, Laser tunnel scanning system (LTSS), Advanced Vision
joints, corrosion, potholes, etc.) existing on civil infrastructure ele- Technology 3D | Pavemetrics TM, Pavemetrics Systems Inc., <http://
www.pavemetrics.com/en/ltss.html> (accessed 03.11.14).
ments, such as buildings, bridges, roads, pipes and tunnels, and the [12] DIBIT, Tunnel Scanning, Tunnel, Service, Home, DIBIT Messtechnik GmbH,
defects’ magnitude (number, width, length, etc.). The condition DIBIT Messtechnik GmbH <http://www.dibit.
assessment results are used to predict future conditions, to support at/dienstleistungen/tunnelbau/tunnelscanning/> (accessed 03.11.14).
[13] B. Cohen, Fixing America’s crumbling underground water infrastructure,
investment planning, and to allocate limited maintenance and Issue Anal. 4 (2012).
repair resources. [14] T. Iseley, D.M. Abraham S. Gokhale, Intelligent sewer condition evaluation
This paper has presented the current practices of assessing the technologies, in: Proceedings of the International NO-DIG Conference, 1997,
pp. 254–265.
visual condition of vertical and horizontal civil infrastructure, in
[15] R. Wirahadikusumah, D.M. Abraham, T. Iseley, R.K. Prasanth, Assessment
particular of reinforced concrete bridges (horizontal: decks, gird- technologies for sewer system rehabilitation, J. Autom. Construct. 7 (4)
ers, vertical: columns), precast concrete tunnels (horizontal: seg- (1998) 259–270.
mental lining), underground concrete pipes (horizontal) [16] S.K. Sinha, P. Fieguth, Segmentation of buried concrete pipe images, Autom.
Construct. 15 (2006) 47–57.
(wastewater infrastructure), and asphalt pavements (horizontal). [17] O. Duran, K. Althoefer, L. Seneviratne, State of the art in sensor technologies
Following this, the second and largest part of the paper has focused for sewer inspection, IEEE Senors J. 2 (2) (2002) 73–81.
[18] W. Guo, L. Soibelman, J.H. Garrett, Automated defect detection for sewer [51] I. Abdel-Qader, S. Pashaie-Rad, O. Abudayyeh, S. Yehia, PCA-based algorithm
pipeline inspection and condition assessment, Autom. Construct. 18 (5) for unsupervised bridge crack detection, Adv. Eng. Softw. 37 (12) (2006) 71–
(2009) 87–596. 778.
[19] T. Hao, Condition assessment of the buried utility service infrastructure, [52] P. Prasanna, K. Dana, N. Gucunski, B. Basily, Computer-vision based crack
Tunn. Undergr. Space Technol. 28 (2012) 331–344. detection and analysis, in: SPIE Smart Structures and Materials+
[20] O. Duran, K. Althoefer, L.D. Senevatne, Automated pipe defect detection and Nondestructive Evaluation and 945 Health Monitoring, International
categorization using camera/laser-based profiler and artificial neural Society for Optics and Photonics, 2012, p. 834, 542.
network, IEEE 1063 Trans. Autom. Sci. Eng. 4 (1) (2007) 118–126. [53] D. Lattanzi, G.R. Miller, Robust automated concrete damage detection
[21] M.-D. Yang, T.-C. Su, Automated diagnosis of sewer pipe defects based on algorithms for field applications, J. Comput. Civ. Eng. 28 (2) (2014) 253–262.
machine learning approaches, Expert Syst. Appl. 35 (3) (2008) [54] M.R. Jahanshahi, S.F. Masri, Adaptive vision-based crack detection using 3D
1327–1337. scene reconstruction for condition assessment of structures, Autom.
[22] M. Halfawy, J. Hengmeechai, Efficient algorithm for crack detection in sewer Construct. 22 (2012) 567–576.
images from closed-circuit television inspections, J. Infrastruct. Syst. 20(2) [55] M. Torok, M. Golparvar-Fard, K. Kochersberger, Image-based automated 3D
(2014). crack detection for post-disaster building assessment, J. Comput. Civil Eng. 28
[23] M. Halfawy, J. Hengmeechai, Automated defect detection in sewer closed (5) (2014).
circuit television images using histograms of oriented gradients and support [56] S. Yu, J. Jang, C. Han, Auto inspection system using a mobile robot for
vector machine, Autom. Construct. 38 (2014) 1–13. detecting concrete cracks in a tunnel, J. Comput. Civil Eng. 17 (4) (2006) 255–
[24] J. Halfawy, J. Hengmeechai, Optical flow techniques for estimation of camera 263.
motion parameters in sewer closed circuit television inspection videos, [57] J.K. Oh, G. Jang, S. Oh, J.H. Lee, B.J. Yi, Y.S. Moon, J.S. Lee, Y. Choi, Bridge
Autom. Construct. 38 (2014) 39–45. inspection robot system with machine vision, Autom. Construct. 18 (2009)
[25] S.K. Sinha, P. Fieguth, Automated detection of crack defects in buried concrete 929–941.
pipe images, Autom. Construct. 15 (1) (2006) 58–72. [58] M.R. Jahanshahi, S.F. Masri, C.W. Padgett, G.S. Sukhatme, An innovative
[26] M. Chae, D. Abraham, Neuro-fuzzy approaches for sanitary sewer pipeline methodology for detection and quantification of cracks through
condition assessment, J. Comput. Civil Eng. 15 (1) (2001) 4–14. incorporation of depth perception, Mach. Vis. Appl. 24 (2) (2013) 227–241.
[27] O. Duran, K. Althoefer, L.D. Seneviratne, Pipe inspection using a laser-based [59] M.R. Jahanshahi, S.F. Masri, A new methodology for non-contact accurate
transducer and automated analysis techniques, IEEE-ASME Trans. Mechatron. crack width measurement through photogrammetry for automated
8 (3) (2003) 401–409. structural safety evaluation, Smart Mater. Struct. 22 (3) (2013).
[28] R. Kirkham, P.D. Kearney, K.J. Rogers, PIRAT – a system for quantitative sewer [60] Z. Zhu, S. German, I. Brilakis, Visual retrieval of concrete crack properties for
pipe assessment, Int. J. Robot. Res. 19 (11) (2000) 1033–1053. automated post-earthquake structural safety evaluation, Autom. Construct.
[29] S. Iyer, S.K. Sinha, B.R. Tittmann, M.K. Pedrick, Ultrasonic signal processing 20 (7) (2011) 874–883.
methods for detection of defects in concrete pipes, Autom. Construct. 22 [61] R.S. Lim, H.M. La, W. Sheng, A robotic crack inspection and mapping system
(2012) 135–148. for bridge deck maintenance, Autom. Sci. Eng. 11 (2) (2014).
[30] S. Iyer, S.K. Sinha, M.K. Pedrick, B.R. Tittmann, Evaluation of ultrasonic [62] S. German, I. Brilakis, R. DesRoches, Rapid entropy-based detection and
inspection and imaging systems for concrete pipes, Autom. Construct. 22 properties measurement of concrete spalling with machine vision for post-
(2012) 149–164. earthquake safety assessments, Adv. Eng. Inform. 26 (4) (2012) 846–858.
[31] Y. Lei, Z. Zheng, Review of physical based monitoring techniques for condition [63] R.S. Adhikari, O. Moselhi, A. Bagchi, A study of Image-based Element
assessment of corrosion in reinforced concrete, Math. Prob. Eng. (2013). Condition Index for Bridge Inspection, in: ISARC, 2013.
[32] F. Chughtai, T. Zayed, Infrastructure condition prediction models for [64] R.S. Adhikari, O. Moselhi, A. Bagchi, Image-based retrieval of concrete crack
sustainable sewer pipelines, J. Perform. Construct. Facil. 22 (5) (2008) 333– properties for bridge inspection, Autom. Construct. 39 (2014) 180–194.
341. [65] T. Uhl, P. Kohut, K. Holak, K. Krupinski, Vision based condition assessment of
[33] M.A. Hahn, R.N. Palmer, M.S. Merrill, Expert system for prioritizing the structures, J. Phys.: Conf. Ser. 305 (1) (2011).
inspection of sewers: Knowledge base formulation and evaluation, J. Water [66] P. Kohut, K. Holak, T. Uhl, Monitoring of civil engineering structures
Resour. Plann. Manage.-ASCE 128 (2) (2002) 121–129. supported by vision system, in: European Workshop on Structural Health
[34] J. Dirksen, F.H.L.R. Clemens, Probabilistic modeling of sewer deterioration Monitoring, 2012, pp. 1575–1582.
using inspection data, Water Sci. Technol. 57 (10) (2008) 1635–1641. [67] K. Chaiyasarn, Damage Detection and Monitoring for Tunnel Inspection based
[35] E.V. Ana, W. Bauwens, Modeling the structural deterioration of urban on Computer Vision, Christ’s College, University of Cambridge, UK,
drainage pipes: the state-of-the-art in statistical methods, Urban Water J. Dissertation, 2011.
7(Special Issue: SI) (2010) 47–59. [68] K. Soga, Micro-measurement and monitoring system for ageing underground
[36] 2013 Report Card for America’s Infrastructure <http://www. infrastructures (underground m3), in: Proceedings of the World Forum on
infrastructurereportcard.org/a/documents/Roads.pdf>. Smart Materials and 994 Smart Structures Technology (SMSST’07),
[37] NCHRP, Automated Pavement Distress Collection Techniques: A Synthesis of Chongqing & Nanjing, China, 2007.
Highway Practice, 2004. [69] A. Miyamoto, M. Konno, E. Bruhwiler, Automatic crack recognition system for
[38] U.S. Army Corps of Engineers, PAVER Asphalt Distress Manual, TR 97/104, concrete structures using image processing approach, Asian J. Inf. Technol. 6
1997. (5) (2007) 553–561.
[39] The Ohio Department of Transportation, Pavement Condition Rating System, [70] Y. Fujita, Y. Mitani, Y. Hamamoto, A method for crack detection on a concrete
2004. structure, in: ICPR’2006:IEEE 18th International Conference on Pattern
[40] Opus Consultants International (Canada) Limited, Pavement Surface Recognition, vol. 3, 2006, pp. 901–904.
Condition Rating Manual, 2012. [71] M. Ukai, Development of image processing technique for detection of tunnel
[41] Northwest Pavement Management Systems Users Group and R. Keith Kay wall deformation using continuously scanned image, Quart. Rep. – RTRI 41
Washington State Department of Transportation, Pavement Surface (3) (2000) 120–126.
Condition Rating Manual. [72] G. Paar, H. Kontrus, O. Sidla, Optical crack following on tunnel surfaces, in:
[42] The South African National Roads Agency Limited, Visual Pavements Proceedings of the SPIE International Society for Optical Engineering, vol.
Assessment for Concrete Pavements, 1998. 6382, 2006.
[43] German Department of Transportation – Road and Transportation Research [73] Z. Liu, S. Azmin, T. Ohashi, T. Ejima, A tunnel crack detection and classification
Association (FGSV), Additional Technical Contractual Terms and Guidelines systems based on image processing, in: Society of Photo-Optical
for Pavement Condition Survey and Assessment (ZTV ZEB-StB), 2006. Instrumentation Engineers (SPIE) Conference Series, vol. 4664, 2002, pp.
[44] P. Prasanna, Computer Vision for Automated Surface Evaluation of Concrete 145–152.
Bridge Decks, Rutgers University, Master’s thesis, 2013. [74] K. Chaiyasarn, K. Tae-Kyun, F. Viola, R. Cipolla, K. Soga, Image mosaicing via
[45] M.R. Jahanshahi, J.S. Kelly, S.F. Masri, G.S. Sukhatme, A survey and evaluation quadric surface estimation with priors for tunnel inspection, in: 16th 1018
of promising approaches for automatic image-based defect detection of IEEE International Conference on Image Processing (ICIP), 2009, pp. 537–540.
bridge structures, J. Struct. Infrastruct. Eng. 5 (6) (2009) 455–486. [75] N. Snavely, S. Seitz, R. Szeliski, Photo tourism: exploring photo collections in
[46] P. Rose, B. Aaron, D.E. Tamir, L. Lu, J. Hu, H. Shi, Supervised computer-vision- 3d, ACM Trans. Graph. 25 (3) (2006) 835–846.
based sensing of concrete bridges for crack-detection and assessment, [76] M.R. Jahanshahi, S.F. Masri, G.S. Sukhatme, Multi-image stitching and scene
Transport. Res. Board (2014). reconstruction for evaluating defect evolution in structures, Struct. Health
[47] I. Abdel-Qader, O. Abudayyeh, M.E. Kelly, Analysis of egde-detection Monitor. 10 (6) (2011) 643–657.
techniques for crack identification in bridges, J. Comput. Civil Eng. 17 (4) [77] Y. Lim, G. Kim, K. Yun, H. Sohn, Monitoring crack changes in concrete
(2003) 255–263. structures, Comp.-Aided Civil Infrastruct. Eng. 20 (2005) 52–61.
[48] T. Yamaguchi, S. Nakamura, R. Saegusa, S. Hashimoto, Image-based crack [78] Z. Chen, T.C. Hutchinson, Image-based framework for concrete surface crack
detection for real concrete surfaces, IEEJ Trans. Electr. Electron. Eng. 3 (1) monitoring and quantification, Adv. Civil Eng. (2010).
(2008) 128–135. [79] H.-B. Kuntze, H. Haffner, Experiences with the development of a robot for
[49] T. Yamaguchi, S. Hashimoto, Fast crack detection method for large-size smart multisensoric pipe inspection, in: Proceedings of the 1998 IEEE,
concrete surface images using percolation-based image processing, Mach. International Conference on 1093 Robotics & Automation, Leuven, Belgium,
Vision Appl. 21 (5) (2010) 797–809. 1998.
[50] C. Koch, S. Paal, A. Rashidi, Z. Zhu, M. König, I. Brilakis, Achievements and [80] R. Duda, P. Hart, D. Stork, Pattern Classification, Wiley, 2001.
challenges in machine vision-based inspection of large concrete structures, [81] A. Bovik, Handbook of Image and Video Processing, Academic Press, 2004.
Adv. Struct. Eng. 17 (3) (2014) 303–318. [82] K. Castleman, Digital Image Processing, Prentice-Hall, 1996.
[83] L. Shapiro, G. Stockman, Computer Vision, Prentice-Hall, 2001. [105] Y. Sun, E. Salari, E. Chou, Automated pavement distress detection using
[84] P. Soille, Morphological Image Analysis, Springer, 2003. advanced image processing techniques, in: Proceedings of the International
[85] J. Serra, Image Analysis and Mathematical Morphology, Academic Press, 1988. Conference on Electro/Information Technology, 2009, pp. 373–377.
[86] T. Su, M. Yang, Application of morphological segmentation to leaking defect [106] S. Amarasiri, M. Gunaratne, S. Sarkar, Modeling of crack depths in digital
detection in sewer pipelines, Sensors 14 (5) (2014) 8686–8704. images of concrete pavements using optical reflection properties, J.
[87] G. Hinton, S. Osindero, Y. The, A fast learning algorithm for deep belief nets, Transport. Eng. 136 (6) (2010) 489–499.
Neural Comput. (2006). [107] C. Haas, Evolution of an automated crack sealer: a study in construction
[88] S. Sinha, P. Fieguth, Neuro-fuzzy network for the classification of buried pipe technology development, Autom. Construct. 4 (4) (1996) 293–305.
defects, Autom. Construct. 15 (2006) 73–83. [108] Y.S. Kim, H.S. Yoo, J.H. Lee, S.W. Han, Chronological development history of
[89] K. Kawasue, T. Komatsu, Shape measurement of a sewer pipe using a mobile X–Y table based pavement crack sealers and research findings for practical
robot with computer vision, Int. J. Adv. Robot. Syst. 10 (52) (2013). use in the field, Autom. Construct. 18 (5) (2009) 513–524.
[90] K.C. Wang, Positioning and imaging sensors for automated asset management [109] H.N. Koutsopoulos, A.B. Downey, Primitive-based classification of pavement
of transportation facilities, in: Proceedings of the 1st International Conference cracking images, J. Transport. Eng 19 (3) (1993) 136–143.
on Transportation Engineering (ICTE), 2007. [110] G. Moussa, K. Hussain, A new technique for automatic detection and
[91] Y.-C. Tsai, V. Kaul, R.M. Mersereau, Critical assessment of pavement distress parameters estimation of pavement crack, in: 4th International Multi-
segmentation methods, J. Transport. Eng. 136 (1) (2010) 11–19. Conference on Engineering Technology Innovation, IMETI 2011, 2011.
[92] E.H. Adelson, Checkershadow Illusion, 1995 <http://persci.mit.edu/gallery/ [111] Q. Li, M. Yao, X. Yao, B. Xu, A real-time 3D scanning system for pavement
checkershadow>. distortion inspection, Meas. Sci. Technol. 21 (2010) 8.
[93] S. Varadharajan, S. Jose, K. Sharma, L. Wander and C. Mertz, ‘‘Vision for Road [112] K. Wang, Challenges and feasibility for comprehensive automated survey of
1134 Inspection’’, pp. 2–9. pavement conditions, in: Proceedings of the 8th International Conference on
[94] H.D. Cheng, Automatic pavement distress detection system 108(1–4) (1998) Applications of Advanced Technologies in Transportation Engineering, 2004.
219–240. [113] Z. Hou, K. Wang, W. Gong, Experimentations of 3D pavement imaging
[95] Q. Zou, Y. Cao, Q. Li, Q. Mao, S. Wang, CrackTree: automatic crack detection through stereovision, in: Proceedings of the International Conference on
from pavement images, Pattern Recog. Lett. 33 (3) (2012) 227–238. Transportation Engineering (ICTE 2007), 2007.
[96] T.S. Nguyen, M. Avila, B. Stephane, Automatic detection and classification of [114] K. Chang, J.R. Chang, J. Liu, Detection of pavement distresses using 3D laser
defect on road pavement using anisotropy 1141 measure (2009) scanning technology, in: Proceedings of the 2005 ASCE International
617–621. Conference on Computing in Civil Engineering, 2005.
[97] H. Lokeshwor, L.K. Das, S.K. Sud, Method for automated assessment of [115] W. Jiaqiu, M. Songlin, J. Li, Research on automatic identification method of
potholes cracks and patches from road surface video clips, Proc. – Soc. Behav. pavement sag deformation, in: Proceedings of the 9th International
Sci. 104 (2013) 312–321. Conference of Chinese Transportation Professionals (ICCTP), 2009.
[98] S.C. Radopoulou, I. Brilakis, Improving patch distress detection using vision [116] J. Karuppuswamy, V. Selvaraj, M.M. Ganesh, E.L. Hall, Detection and
tracking on video data, in: Proceedings of the 21st International Workshop on avoidance of simulated potholes in autonomous vehicle navigation in an
Intelligent Computing in Engineering, 2014. unstructured environment, in: Intelligent Robots and Computer Vision XIX:
[99] L. Li, G.N. Sun, S. Tan, Automatic pavement crack recognition based on bp Algorithms, Techniques, and Active Vision, Proceedings of SPIE, vol. 4197,
neural network 26(1) (2014) 11–22. 2000, pp. 70–80.
[100] M. Salman, S. Mathavan, K. Kamal, M. Rahman, Pavement crack detection [117] M. Jahanshahi, F. Jazizadeh, S. Masri, B. Becerik-Gerber, Unsupervised
using the Gabor filter, in: 16th International IEEE Conference on Intelligence approach for autonomous pavement-defect detection and quantification
Transportation System (ITSC 2013), 2013, pp. 2039–2044. using an inexpensive depth sensor, J. Comput. Civil Eng. 27 (6) (2013) 743–
[101] J. Zhou, P.S. Huang, F.-P. Chiang, Wavelet-based pavement distress detection 754.
and evaluation, Opt. Eng. 45 (2) (2006). [118] C. Koch, I. Brilakis, Pothole detection in asphalt pavement images, Adv. Eng.
[102] H. Lokeshwor, L.K. Das, S. Goel, Robust method for automated segmentation Inform. 25 (3) (2011) 507–515.
of frames with/without distress from road surface video clips, J. Transport. [119] G.W. Zack, W.E. Rogers, S.A. Latt, Automatic measurement of sister chromatid
Eng. 140 (1) (2014) 31–41. exchange frequency, J. Histochem. Cytochem. 25 (7) (1977) 741–753.
[103] K.P. Wang, W. Gong, Real-time automated survey system of pavement [120] C. Koch, G. Jog, I. Brilakis, Towards automated pothole distress assessment
cracking, J. Infrastruct. Syst. 11 (3) (2005) 154–164. using asphalt pavement video data, J. Comput. Civil Eng. (2012).
[104] Y. Huang, B. Xu, Automatic inspection of pavement cracking distress, J. [121] S. Cafiso, A. Di Graziano, S. Battiato, Evaluation of pavement surface distress
Electron. Imag. 15 (1) (2006). using digital image collection and analysis (2006) 1–10.

Koch Et Al. (2015)

Uploaded by

Copyright:

Available Formats

Koch Et Al. (2015)

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Koch Et Al. (2015)

Uploaded by

Copyright:

Available Formats

Advanced Engineering Informatics xxx (2015) xxx–xxx

Contents lists available at ScienceDirect

Advanced Engineering Informatics

A review on computer vision based defect detection and condition

Defects Condition states

Defect type/ Minor Moderate Severe

Low-level Intermediate-level High-level

Image Pre- Segmen- Feature Object Structural

General restoration, thresholding, features classification spatio-

Pre- Feature-based Model-based Pattern-based 3D Recon-

Specific Haar- Percolation-based Nearest Stereo

3.1.1. Cracking comprises images from different locations because it is important

3.4. Asphalt pavements 3.4.2. Defect detection

You might also like