2019 - Intelligent Thermal Imaging-Based Diagnostics of Turbojet Engines

applied
sciences
Article
Intelligent Thermal Imaging-Based Diagnostics of
Turbojet Engines
Rudolf Andoga * , Ladislav Főző , Martin Schrötter, Marek Češkovič, Stanislav Szabo,
Róbert Bréda and Michal Schreiner
Technical University of Košice, Faculty of Aeronautics, Rampová 7, 041 21 Košice, Slovakia;
[email protected] (L.F.); [email protected] (M.S.); [email protected] (M.Č.);
[email protected] (S.S.); [email protected] (R.B.); [email protected] (M.S.)
* Correspondence: [email protected]; Tel.: +421-55-602-6140

Received: 2 April 2019; Accepted: 27 May 2019; Published: 31 May 2019
Featured Application: The methodology presented in this paper is directly intended for
application in the area of turbojet engines. This paper presents the design of an automated
system which can create diagnostic information using data from an infrared thermal camera,
utilizing methods of computational intelligence and expert systems, which can be considered as
a novel approach. The developed system segments infrared images use a self-organizing feature
map and an expert system to automatically extract diagnostic information about the technical
state of a turbojet engine. The diagnostic system was developed using a small iSTC-21v turbojet
engine with a thrust of 500 N in laboratory conditions and can be simply modified for application
to other engine types. The system is not limited to engines and can be considered for application
in other technical systems which produce heat, such as generators, transmissions, and electronic
circuit boards.
Abstract: There are only a few applications of infrared thermal imaging in aviation. In the area
of turbojet engines, infrared imaging has been used to detect temperature field anomalies in order
to identify structural defects in the materials of engine casings or other engine parts. In aviation
applications, the evaluation of infrared images is usually performed manually by an expert. This
paper deals with the design of an automatic intelligent system which evaluates the technical state
and diagnoses a turbojet engine during its operation based on infrared thermal (IRT) images. A
hybrid system interconnecting a self-organizing feature map and an expert system is designed for
this purpose. A Kohonen neural network (the self-organizing feature map) is successfully applied
to segment IRT images of a turbojet engine with high precision, and the expert system is then
used to create diagnostic information from the segmented images. This paper represents a proof
of concept of this hybrid system using data from a small iSTC-21v turbojet engine operating in
laboratory conditions.
Keywords: expert systems; infrared thermography; intelligent diagnostics; image segmentation;

neural networks; thermal imaging camera; turbojet engines
1. Introduction
Infrared thermography (IRT) is one of the many diagnostic methodologies currently applied
in various types of industry and research [1]. It can be used to detect so-called invisible defects
in a non-destructive way. Compared to traditional areas of IRT applications, such as biomedical
applications [2], energetics, buildings, and other constructions [3–6], the application of this methodology
in aviation is still at an early stage [1].
Appl. Sci. 2019, 9, 2253; doi:10.3390/app9112253 www.mdpi.com/journal/applsci

Appl. Sci. 2019, 9, 2253 2 of 22
The advantages of IRT have been fully demonstrated in pre-flight and post-flight aircraft checks,
where damage (e.g., water in honeycomb structures of composite parts), is monitored by thermal
imaging cameras [7,8]. In aviation, extensive research and development is being conducted on the IRT
diagnostics of corrosion in aluminum materials [9,10], and especially composite structures [11–13].
In aircraft operation, it is possible to use IRT to detect air leaks inside the cabin [14], abnormal strain
in individual parts of aircraft casings, fuselage, and wheel attachments [15], damage to electrical
wiring and insulation [16,17], etc. In aviation, IRT is also used for rescue, firefighting, and surveillance
operations [18,19]. Other applications of IRT diagnostics can be found in the area of unmanned vehicles,
whose thermal imaging cameras are used to inspect power lines [20], pipelines [21], buildings [22],
constructions [23], vegetation [24], etc. Thermography has also recently found an application in the
design of stealth aircraft [25].
For the diagnostics of aviation engines, IRT is mainly used to check their composite covers [26].
Another area where IRT is applied, which is closer to the topic of this paper, is the diagnostics of
turbojet engines; infrared images of the diagnosed engines’ gas turbines are compared to standard IRT
images of engines without defects, which allows temperature field anomalies to be detected and thus
allows defects to be identified [27]. These defects can be damage to the turbine blades, combustion
chamber disturbance, disturbance to the homogeneous field of fuel burning, etc. [28].
Aircraft engine diagnostics and health monitoring [29–31] can be divided into two basic processes:
ground diagnostics and on-board diagnostics with their associated systems. On-board, or online,
diagnostics involve the checking of engine components and thermodynamic processes during the
operation of the engine [32]. These components include individual engine parts, such as compressors,
combustion chambers, and turbines and associated accessories, while thermodynamic process checks
include reviewing parameters such as temperature, pressure, rotational turbo-compressor speed, and
vibrations. Ground, or offline, diagnostics [33,34] involve tribo-technical analysis, non-destructive
checking methods, and checks by thermal cameras as mentioned earlier.
This article aims to develop a progressive method for the online evaluation of the technical state
of a turbojet engine using infrared thermal imaging. Contrary to traditional methods, in which a
comparative analysis of thermal images is performed on some normalized samples, this paper proposes
an intelligent adaptive method which can discriminate anomalous areas of thermal images and create
resulting diagnostic information automatically without any human input or previous training.
Since the output of a thermal imaging camera is a digital image, computer vision techniques and
methodologies should be used to automatically process this output. In order to extract meaningful
information from a thermal image for a diagnostic system, a pattern recognition/classification
approach has to be applied. Many methodologies and an array of methods can be used for this
purpose [35,36]. These include traditional methodologies, which perform pattern recognition using
deterministic image-processing methods such as filtering, feature analysis, and support vector
machines. [37]. However, the most successful methodologies for image classification come from
the area of computational intelligence. These include artificial neural networks, which have been
widely and successfully applied in image processing, pattern recognition, and diagnostic tasks [38,39].
The most complex neural networks, such as convolutional neural networks, are able to classify any
pattern in an image with high precision, as well as detect objects [40]. However, these approaches are
computationally complex and require large amounts of data and training [41]. In order to create a
successful diagnostic method for pattern recognition in IRT images, a simpler methodology should be
used, since such images are more homogeneous than traditional images; they have fewer colors, lower
resolution, and are well structured, with well-defined edges. The motivation for this study was to
develop a methodology which does not require a large amount of computationally intensive training
to successfully segment infrared images using intelligent adaptive algorithms.
Accordingly, in this paper, the field of neural networks with uncontrolled learning is explored.
One of the methods that has been successfully used in pattern recognition and image segmentation is
self-organizing feature maps (SOFMs), which comply with the demand for simplicity, uncontrolled
Appl. Sci. 2019, 9, 2253 3 of 22
learning, and good classification results. SOFMs can be transformed and formalized into the form of a
Kohonen neural network [42,43]. Kohonen neural networks have been successfully applied in image
segmentation tasks, the detection of specific areas in geographic maps [44–46], and the detection of
cracks and corrosion in materials [47]. The results obtained from these applications represent a solid
foundation and an inspiration for this approach to be applied in the segmentation of infrared images
for the detection of anomalies in the temperature fields of different objects.
The results of pattern recognition and image segmentation come in the form of classes present in
the image. In order to extract knowledge from this information, the classes have to be interpreted.
This can be done by simply labeling the classes; however, in the field of technical systems, this may
not be sufficient. In technical objects, some areas are naturally hot and some are naturally cold, and
therefore there needs to be some inference system which interprets the displacement and size of the
classes from thermal images, and generates diagnostic information, which in the simplest case can
be the determination of whether the diagnosed object is in a normal or an abnormal operational
state [48,49]. The traditional approach is to consider a diagnosed system as healthy if it operates within
some specific limits; otherwise, a failure is indicated [48,49]. This is a natural approach, and can be
used when dealing with a technical system where parameters which determine the technical state
(temperature, vibrations, etc.) can be simply measured, such as a jet engine. However, to extract
diagnostic information from a classified image, a more complex approach needs to be used [49,50].
Diagnostic expert systems are a natural candidate for extracting diagnostic information from complex
data, and have been applied in the diagnostics of turbojet engines [49,50]. The aim of the present
study is to create a specialized approach for the creation of a rule-base of an expert system which can
transform class information in an infrared image into useful diagnostic information, thus creating
hybrid architecture.
The functionality of this approach has been evaluated and designed specifically for application to
turbojet engines, and the approach can be considered as novel compared to other intelligent approaches
in turbojet engine diagnostics, which are applied to data measured by the engines’ sensors and do not
deal with visual/infrared image information [51–53]. The aim of this paper is to explore the design
of this new approach and prove its functionality by applying thermal vision-based diagnostics in an
automated adaptive system in the area of turbojet engine diagnostics. The approach can be expanded
and applied to almost any technical system which has been observed by an infrared camera. The
hybrid architecture of our approach, which combines a Kohonen neural network and a specialized
expert system with a designed rule-base, can be considered as a novel and original contribution of our
research. A proof of concept is presented using real data from an iSTC-21v turbojet engine tested and
operated in laboratory conditions.
2. Methodology
2.1. Using Infrared Imaging for Diagnostics

Every object whose temperature is above absolute zero radiates thermal energy. IRT systems
measure this energy to create thermal infrared images. These images can be used to identify heat
radiation anomalies of an object, which are usually characterized by increases or decreases in the
radiated temperature [8,9,13]. The basic principle of IRT diagnostics is shown in Figure 1, with the
following general meaning:
• Image: an object to be captured by a thermal camera.

• Classification: monitoring the object, scanning it, obtaining infrared data, classifying hot and cold
parts as well as possible problematic areas.
• Diagnostic information: evaluating infrared thermal images, for example, identifying defects or
deficiencies, deciding about the technical state of the engine.
Appl. Sci. 2019, 9, x FOR PEER REVIEW
Appl. Sci. 2019, 9, 2253
4 of 23
4 of 22
Figure 1. Procedure for the detection of temperature anomalies in engine casings by analyzing an
Figure 1. Procedure for the detection of temperature anomalies in engine casings by analyzing an
infrared image.
infrared image.
The aforementioned principle is a standard principle when using human expertise to obtain
diagnosticThe information
aforementioned fromprinciple is a standard
infrared images. Examplesprinciple when using
of applications human expertise
in aviation are pre-flight to obtain
and
diagnostic information from infrared images. Examples of applications
post-flight engine checks, where the engine’s casing is checked for defects and cracks caused by freezing in aviation are pre-flight and
post-flight
water engine
or material checks, where
deficiencies. the cases,
In such engine’s the casing is checked
evaluation of thermalfor defects
images and and cracks causedofby
the detection
freezing water or material deficiencies. In
anomalies is strictly dependent on human expert knowledge [49]. such cases, the evaluation of thermal images and the
detection
When aofhuman
anomalies expertis strictly
inspectsdependent on human
infrared images expert knowledge
of a technical object, he [49].
or she basically performs
two operations. First, he/she discriminates different areas of the thermal object,
When a human expert inspects infrared images of a technical image and he or she basically
classifies them
performs two operations. First, he/she discriminates different
into categories according to their temperature. They next use their expert knowledge to areas of the thermal image and
inspect
classifies
hot or coldthem
areasintofor categories
some anomalies according to theirtotemperature.
according their position They
andnext use their expert
temperature. knowledge
The traditional
to inspect hot or cold areas for some anomalies according to their
approach is to superimpose the obtained infrared image over a standardized infrared image of an position and temperature. The
traditional approach is to superimpose the obtained infrared image
object without defects and evaluate the differences between the two images [54]. The whole process over a standardized infrared
image
can of anbe
therefore object without
formalized intodefects and parts—image
two basic evaluate the differences
processing and between the two and
classification, images [54]. The
knowledge
whole process can therefore be formalized into two basic parts—image
acquisition. A variety of methods exist to solve both problems, as mentioned in the introduction, processing and classification,
andare
and knowledge
suitable toacquisition.
identify and A evaluate
variety of methods exist
anomalies. to solve both
As previously problems,
stated, one aimasofmentioned
this study in wasthe
to develop an autonomous diagnostic system which is able to learn and locate anomalies withoutof
introduction, and are suitable to identify and evaluate anomalies. As previously stated, one aim
thisprior
any study was to develop
knowledge an autonomous
(e.g., normal diagnostic
faultless operation IRT system
images).which is able
The other aimtowas learn and locate
to make the
anomalies without
system autonomous. any prior knowledge (e.g., normal faultless operation IRT images). The other aim
wasThe to make the system autonomous.
general concept of the intelligent diagnostic system can be expressed by the scheme shown in
The
Figure 2. The general
dataconcept of the
acquisition intelligent
part diagnostic
of the approach system can
is usually done beusing
expressed by the scheme
the infrared camera,shownand
in image
the Figurepre-processing
2. The data acquisition
can be done partbefore
of thethe approach
data areisfed
usually
into adone using The
classifier. the infrared
effects ofcamera,
the image and
the image pre-processing can be done before the data are fed into a classifier.
pre-processing and filtering are not directly investigated in this study; however, the methodology is The effects of the image
pre-processing
designed and filtering
to be relatively robustare in not
orderdirectly
to useinvestigated
image data of inreasonable
this study;quality
however, withthelow methodology
noise levels.is
designed to be relatively robust in order to use image data
Infrared cameras usually pre-process the image data in order to compensate for air humidity of reasonable quality with low and
noise
levels. Infrared cameras usually pre-process the image data in order
other environmental effects. The robustness is achieved by the application of an intelligent adaptive to compensate for air humidity
and otherand
algorithm, environmental
the design thereforeeffects.consists
The robustness
mainly of is theachieved
classifierby partthe
by application of an intelligent
finding a methodology and
selecting its structure, input/output relations, and the training algorithm in the case of the by
adaptive algorithm, and the design therefore consists mainly of the classifier part finding a
application
ofmethodology
neural networks. and Information
selecting itsabout structure,
classes input/output
in the thermal relations,
image can and bethe training
further usedalgorithm
to evaluateinthe the
case of the
technical stateapplication of neuralobject
of the investigated networks.
(in thisInformation about classes
case, a jet engine), in the thermal
and a diagnostic expertimagesystem can
is abe
natural candidate for the completion of this task. The basis of the proposed methodology is therefore a
further used to evaluate the technical state of the investigated object (in this case, a jet engine), and
diagnostic
centered expert system is a natural candidate for the completion of this task. The basis of the
around:
proposed methodology is therefore centered around:
• Image processing classification system: involving a SOFM.
• Image processing classification system: involving a SOFM.
• Knowledge acquisition system: an expert system for the evaluation of the technical state of
• Knowledge acquisition system: an expert system for the evaluation of the technical state of an
an engine.
engine.
Appl. Sci. 2019, 9, x FOR PEER REVIEW 5 of 23
The other advantage of the designed system is that it does not use the absolute temperature
data obtained from the camera, which are dependent on many environmental and material factors,
butSci.
Appl. rather
2019, only
9, 2253uses image data, which describe relative temperatures and relations as well 5as
of the
22
displacement of the temperature field of the investigated object.
The other advantage of the designed system is that it does not use the absolute temperature
but rather only uses image data, which describe relative temperatures and relations as well as the
displacement of the temperature field of the investigated object.
Figure2.2.The
Figure Theconceptual
conceptualdesign
designofofthe
thediagnostic
diagnosticsystem.
system.
2.2. The other advantage

Self-Organizing FeatureofMaps the designed system is that
in Image Segmentation anditClassification
does not use the absolute temperature
SOFMs
but rather onlyareusescharacterized
image 2.by
data,
Figure Thetheir
which outstanding
describe
conceptual ofproperties
relative
design in clustering
temperatures
the diagnostic (classification),
and relations
system. as well as the data
visualization,ofprobability
displacement the temperature density estimation,
field etc. Kohonen
of the investigated artificial neural networks are one of the
object.
mostSelf-Organizing
2.2. widely used SOFMs Feature [42].Maps in Image Segmentation and Classification
2.2. Self-Organizing
Kohonen artificial Feature Mapsnetworks
neural in Image Segmentation
(KNN) map and Classification
a high-dimensional input space/layer (xi for i =
SOFMs are characterized by their outstanding properties in clustering (classification), data
1, 2,SOFMs
..., n, where n is the number
areprobability
characterized byofestimation,
inputs)
their onto an output
outstanding space/layer
properties (Nj = 1, 2,
in clustering ..., m, where m data
(classification), is the
visualization, density etc. Kohonen artificial neural networks are one of the
number
visualization,of neurons) with
probability fewer or an equal number of dimensions. KNN contains only a single layer
most widely used SOFMsdensity [42]. estimation, etc. Kohonen artificial neural networks are one of the
of neurons—the
most widely used Kohonen
SOFMs [42].layer. The number of neurons n, which is the network’s parameter, is
Kohonen artificial neural networks (KNN) map a high-dimensional input space/layer (xi for i =
optional and usually equals the number of target classes. The inputinputs to the network are
Kohonen artificial neural networks
1, 2, ..., n, where n is the number of inputs) (KNN) map
onto an aoutput
high-dimensional
space/layer (Nj = 1,space/layer (xi for
2, ..., m, where m iis=the
1,
interconnected
..., n, where
2,number
with all
n is thewith
neurons
number ofor
in the
inputs)
Kohonen
onto
layer.
an output
This means
space/layerKNN
that every neuron
(Nj =contains
1, 2, ..., m,
has
where
information
m islayer
the
of neurons) fewer an equal number of dimensions. only a single
from allofthe
number
inputs.with
neurons)
Synaptic
fewer
weights
or an
leading
equal number
to each
of
neuron (designated
dimensions. KNN
as a only
contains
weighta
vector,
single
wij,ofof
layer
of neurons—the Kohonen layer. The number of neurons n, which is the network’s parameter, is
the neuron) can be understood
neurons—the
asnumber
coordinates, whichn,indicate the location of the neuron is in optional
the array
optional andKohonen usuallylayer.equals Thethe numberof neurons
of target which is theThe
classes. network’s
inputsparameter,
to the network are
(Figure 3). Neurons are mostly organized in a regular two-dimensional grid/matrix or
and usually equals
interconnected withtheallnumber
neuronsofintarget classes. layer.
the Kohonen The inputs to the network
This means that everyare interconnected
neuron with
has information
one-dimensional row/vector array.
all neurons
from all theininputs.
the Kohonen
Synaptic layer. This means
weights leadingthat everyneuron
to each neuron(designated
has information from allvector,
as a weight the inputs.
wij, of
Synaptic weights leading to each neuron (designated as
the neuron) can be understood as coordinates, which indicate the location of a weight vector, w , of the neuron)
ij the neuron in the can be
array
understood
(Figure 3).as Neurons
coordinates, arewhich indicate
mostly the location
organized in ofa the neurontwo-dimensional
regular in the array (Figure 3). Neuronsor
grid/matrix
are mostly organized in
one-dimensional row/vector array.a regular two-dimensional grid/matrix or one-dimensional row/vector array.
Figure 3. The structure of a Kohonen network.

Appl. Sci. 2019, 9, 2253 6 of 22
There
There areare
lateral
laterallinks (local
links connections)
(local connections)between
between the neurons in theinKohonen
the neurons the Kohonen layer (dashed line
layer (dashed
inline
Figure 3), which means that any particular neuron affects other neurons in its
in Figure 3), which means that any particular neuron affects other neurons in its neighborhood. neighborhood.
InIn
order
ordertotosegment
segmentananIRT IRTimage,
image,a alayer
layerthat
thathas
has1 1××m𝑚 neurons
neurons (Figure
(Figure4) 4)
is proposed. Using
is proposed. Using
this type of array, it is possible to segment thermal infrared images into m different
this type of array, it is possible to segment thermal infrared images into m different segments, which segments, which
can
canbebe
labeled
labeled asas
different
differentareas
areasofof
the temperature
the temperature field (cold,
field hot,
(cold, hot,etc.). Thus,
etc.). Thus,if if
a higher
a highernumber
number ofof
neurons
neurons is is
used,
used,thetheclassification
classificationwill bebe
will more
moredetailed.
detailed.
Figure 4. 4.AAsingle-row
Figure structure
single-row ofof
structure the Kohonen
the layer.
Kohonen layer.
In the most simplified way, the self-organization process (Kohonen learning) consists of four main
In the most simplified way, the self-organization process (Kohonen learning) consists of four
steps: initialization, competition, cooperation, and adaptation [55]:
main steps: initialization, competition, cooperation, and adaptation [55]:
• • Initialization:
Initialization: Random
Randomvalues
valuesare areset
setfor
forall
allneuron
neuronweights.
weights.The
Theinput
inputvector
vectorfrom
fromthe theinput
input
space
spaceisisselected.
selected.
• • Competition:
Competition:From Fromthetheinputs,
inputs,thethediscriminant
discriminantfunction
function(Equation
(Equation(1),(1),below)
below)ofofa aparticular
particular
neuron
neuronisis computed.
computed. AAKohonen
Kohonenunit unit computes
computes thethe Euclidean
Euclidean distance
distance between
between an input
an input xi and
xits
i and its weight vector w . The winning neuron is the one with the smallest
weight vector wij. The winning neuron is the one with the smallest value of the discriminant
ij value of the
discriminant
function. function.
• • Cooperation:
Cooperation: The Thewinning
winningneuron neuroninfluences
influencesits itsadjacent
adjacentneurons
neurons(i.e.,
(i.e.,the
theweight
weightvectors
vectorsofof
adjacent
adjacent neurons are influenced), however this influence decreases with increasing distancetoto
neurons are influenced), however this influence decreases with increasing distance
other
otherneurons.
neurons.
• • Adaptation:
Adaptation:The Theexcited
excited neurons
neurons decrease
decrease their values of the discriminant
discriminant function
functionin
inrelation
relationto
tothe
theinput
inputpattern
patternthrough
through adjustment
adjustment of the associated
associated weights.
weights. The
Theresponse
responseofofthethewinning
winning
neuron
neurontotothe subsequent
the subsequentapplication
applicationof aofsimilar inputinput
a similar pattern is enhanced.
pattern The process
is enhanced. will stop
The process will
ifstop
the maximum number of iterations is reached.
if the maximum number of iterations is reached.
𝑑 = m x −𝑥 w− 𝑤
2
(1)(1)
X
dj = i ij
i=1
Theself-organization
The self-organizationprocess
processisisgraphically
graphicallydepicted
depictedininFigure
Figure5,5,where
whereblue
bluedots
dotsrepresent
represent
neurons,
neurons, green
green dotsdots represent
represent input vectors,
input vectors, and represent
and red lines red linesconnections
represent between
connections between
neighboring
neighboring
neurons. neurons. The
The temperature temperature
field field which
has five areas, has five
areareas, which
indicated byare
fiveindicated
neurons.byIn five
this neurons.
simplisticIn
this simplistic
example, example,
the labeling the labeling
is performed is performed
by coloring by coloring
the neurons the neurons and
and the corresponding the corresponding
training data, which
trainingthe
activates data, which In
neurons. activates the neurons.
this manner, the blueInarea
thisafter
manner, the blue
adaptation area afterthe
represents adaptation
coldest arearepresents
in the
the coldest field
temperature area in
andthe
thetemperature
red area thefield andarea.
hottest the red area the hottest area.
Appl.
Appl.Sci. 2019,9,9,2253
Sci.2019, x FOR PEER REVIEW 77ofof2223
Figure 5. The self-organization process with a single-row structure of the Kohonen layer.
Figure5.5.The
Figure Theself-organization
self-organizationprocess
processwith
witha asingle-row
single-rowstructure
structureofofthe
theKohonen
Kohonenlayer.
layer.
2.3. Expert Diagnostic Systems
2.3.
2.3.Expert
Expert Diagnostic
Diagnostic Systems
Systems
An expert system consists of a knowledge base, an inference engine (solving or deducing), an
An expert
input/output
An expert system
data consists
interface
system ofofainterface
(user
consists aknowledge base,
or links
knowledge toanan
base, inference
other engine
systems
inference and (solving
engine tools), a or
(solving ordeducing),
working anan
memory,
deducing),
input/output
and
input/output data
an explaining
datainterface
interface(user interface
or communication
(user or links
module
interface to other
in
or links the systems
form
to other of a and
usertools),
systems a working
interface.
and tools), aThesememory,
elements
working and
memory,are
anshown
explaining
and an in or communication
Figure 6. A
explaining or very module
important part
communication in the form
of every
module of a
expert
in the user
form interface.
system These elements
is itsinterface.
of a user inference These are shown
engine,elements in
which uses
are
Figure
data 6.
from A very
the important
knowledge part
base of every expert
(knowledge system
gained is
from its
aninference
expert) engine,
and which
specific
shown in Figure 6. A very important part of every expert system is its inference engine, which uses uses data from
problem-oriented
the knowledge
data
data from thebase
[56]. (knowledge
knowledge basegained from an
(knowledge expert)
gained andan
from specific
expert)problem-oriented data [56].
and specific problem-oriented
data [56].
Figure 6. The structure of expert systems and their integration (reproduced from [57]).
Figure 6. The structure of expert systems and their integration (reproduced from [57]).
The inference engine
Figure 6. and theofknowledge
The structure expert systemsbase
andrepresent the core
their integration of every expert
(reproduced system. The
from [57]).
Theengine
inference inference engine
provides theand the knowledge
mathematical meansbase represent
to evaluate thethe core
rules of every
in the expertbase
knowledge system. The
to reach
a inference
conclusion engine
aboutprovides
The inference engine
the thethe
state and
of mathematical
the knowledge
diagnosed means to evaluate
base[58–60].
object represent thecore
the rules
In applications of inforthe
every knowledge
expert
turbojet base
system.
engines, to
The
the
reach
knowledge a conclusion
inference engine about
provides
base contains the state
thedefining
rules of
mathematical the diagnosed
meansstate
the technical object
to evaluate [58–60].
the rules
of the turbojet In applications
in the
engine for
knowledge
based turbojet
base to
on the outputs
engines,
reach
from the knowledge
thea segmentation
conclusion aboutbase
thecontains
algorithm state
(i.e.,ofrules
thethe defining
diagnosed
Kohonen theobject
neural technical state
[58–60].
network). of design
In
The the turbojet
of theengine
applications based
for turbojet
diagnostic
on
systemtheinterconnected
engines, outputs from the
the knowledge withsegmentation
base contains
a SOFM, algorithm
rules
which (i.e.,
defining
was used the
inthe Kohonen
thetechnical
present neural
state
study, ofisthenetwork).
turbojet
shown The design
engine
in Figure of
7. based
the diagnostic system interconnected with a SOFM, which was used in the
on the outputs from the segmentation algorithm (i.e., the Kohonen neural network). The design of present study, is shown
in
theFigure 7.
diagnostic system interconnected with a SOFM, which was used in the present study, is shown
in Figure 7.
Appl.
Appl.Sci. 2019, 9, 9,
Sci. 2253 8 of 22
Appl.2019, x FOR
Sci. 2019, PEER
9, x FOR REVIEW
PEER REVIEW 8 of 238 of 23
7. The
FigureFigure 7. connection between
The connection thethe
between Kohonen
Kohonenneural
neural network andananexpert
network and expert system.
system.
In thisIndiagnostic
Figure
system,
this diagnostic the knowledge
system,
7. The connection the is represented
knowledge
between
by IF->THEN
is represented
the Kohonen
rules,rules,
by IF->THEN
neural network
which
and an expert
can be
which
system. canwritten
be
in a simple way as IF E THEN H, as follows:
written in a simple way as IF E THEN H, as follows:
In this diagnostic system, the knowledge 𝐸is→represented
𝐻 by IF->THEN rules, which (2)
can be
E→H (2)
written in a simple way as IF E THEN H, as follows:
where E represents an observation (evidence) and H represents a hypothesis.
Rule-based
where E represents ansystems differ (evidence)
observation from classical
and 𝐸
H→
logic 𝐻
systems
representsin their non-monotonic “thinking” and (2)
a hypothesis.
their abilitysystems
Rule-based to process uncertainties.
differ In rule-based
from classical systems, in
logic systems inference is based on the rule
their non-monotonic of modusand
“thinking”
where E represents
ponens an observation (evidence) and H represents a hypothesis.
their ability to(direct thinking)
process [61], which
uncertainties. Inisrule-based
described as:
systems, inference is based on the rule of modus
Rule-based systems differ from classical logic systems in their non-monotonic “thinking” and
ponens (direct thinking) [61], which is described as: 𝐸, 𝐸 → 𝐻
their ability to process uncertainties. In rule-based systems, inference is based on the rule of(3) modus
𝐻
ponens (direct thinking) [61], which is described E, E → as:H
This means that, if the E assumption is true and the rule implies E→H, then the conclusion H is (3)
𝐸,
H𝐸 → 𝐻
true. In the case of an inference network, the conclusions are represented by facts, which correspond (3)
to assumptions of other rules. These inference networks 𝐻 are mostly used in domains where the
This means that, if the E assumption is true and the rule implies E→H, then the conclusion H is
number of possible solutions is limited (e.g., diagnostic problems or classification problems). The
true. InThis
the means
case ofthat, if the E assumption
an inference network, theisconclusions
true and thearerule implies E→
represented byH,facts,
thenwhich
the conclusion
correspond H is
benefit of inference networks is that they can be easily implemented. The following model rules
totrue. In the caseofofother
assumptions
(Equation (4))
an inferenceThese
[61] canrules.
network, the conclusions
be formalized inference networks
in an inference
are
networkarerepresented
mostlyasused
structure
byin
shown
facts, whichwhere
indomains
correspond
Figure 8 [61]: the
to assumptions
number of possible of solutions
other rules. These (e.g.,
is limited inference networks
diagnostic are mostly
problems used in domains
or classification where
problems). Thethe
number A ∧ B → P, B ∧ D → Q, P → X, Q ∧ R → Y, C → P, C ∧ E → R, Q → X (4)
benefit of of possiblenetworks
inference solutionsisisthat
limited
they (e.g.,
can be diagnostic problems or classification
easily implemented. The followingproblems).
model rules The
benefit of(4))
(Equation inference
[61] can networks
be formalizedis that they
in an can be network
inference easily implemented. The following
structure as shown in Figuremodel
8 [61]: rules
(Equation (4)) [61] can be formalized in an inference network structure as shown in Figure 8 [61]:
A ∧ B → P, B ∧ D → Q, P → X, Q ∧ R → Y, C → P, C ∧ E → R, Q → X (4)
A ∧ B → P, B ∧ D → Q, P → X, Q ∧ R → Y, C → P, C ∧ E → R, Q → X (4)
Figure 8. An example of an AND/OR inference network (reproduced from [61]).

Appl. Sci. 2019, 9, 2253 9 of 22
2.4.
2.4.Experimental
ExperimentalSetup
Setupfor
forDesign
Designofofthe
theInfrared
InfraredImaging-Based
Imaging-BasedDiagnostics
DiagnosticsSystem
System
InInorder
orderto to
design an an
design infrared imaging-based
infrared imaging-based diagnostic system
diagnostic for use
system forwith turbojet
use with engines
turbojet and
engines
toand
construct a working knowledge base, a small iSTC-21v turbojet engine, developed
to construct a working knowledge base, a small iSTC-21v turbojet engine, developed in the in the Laboratory
ofLaboratory
Intelligent Control SystemsControl
of Intelligent of Aircraft Enginesofat Aircraft
Systems the Faculty of Aeronautics,
Engines Technical
at the Faculty ofUniversity of
Aeronautics,
Košice (Figure 9), was used [62]. This is a single-stream, single-shaft turbojet engine with a
Technical University of Košice (Figure 9), was used [62]. This is a single-stream, single-shaft turbojet single-stage
one-sided
engine with radial compressor,one-sided
a single-stage an annular combustion
radial chamber,
compressor, and a single-stage
an annular combustion non-cooled
chamber, and gas a
turbine with a non-cooled
single-stage variable exhaust nozzle with
gas turbine with full authority
a variable intelligent
exhaust nozzleelectronic digital
with full control
authority system
intelligent
developed from the TS-21 engine (Ljulka-Saturn, Rybinsk, Russia). A schematic of the
electronic digital control system developed from the TS-21 engine (Ljulka-Saturn, Rybinsk, Russia). measurement
and data acquisition
A schematic of thesystem is shownand
measurement in Figure 10, and the system
data acquisition arrangement of theincamera
is shown Figureis10,shown
and inthe
Figure 11 [62].
arrangement of the camera is shown in Figure 11 [62].
Figure
Appl. Sci. 9,9.xViews
2019, 9.
Figure ofofthe
FOR PEER
Views iSTC-21v
iSTC-21vturbojet
REVIEW
the turbojetengine
engine(left)
(left)and
anditsitsstarting
startingprocess
process(right)
(right)inina alaboratory.
10 of 23
laboratory.
A digital data acquisition system was used to observe the state of the engine [62]. Various
sensors and data acquisition (DAQ) devices serve this purpose and send all the data to the control
personal computer (PC). This PC contains a control system created in the LabVIEW (National
Instruments Corporation, Austin, TX, USA) and Matlab/Simulink (MathWorks, Natick, MA, USA)
software which is able to control and visualize the engine’s performance, save and track all
measured parameters, and update the control algorithms.
For the design of the IRT diagnostics system, the following parameters are utilized:
• n (rpm)—speed of the engine’s shaft.
• T2 (°C)—air temperature at the output of the radial compressor.
• T3 (°C)—gas temperature at the input to the gas turbine.
• T4 (°C)—gas temperature at the output of the gas turbine.
These parameters represent the inner state and temperature of the engine. The interior
temperature was measured by a thermal sensor contained in the engine. The temperature of the
engine’s casing was measured by a thermal camera; this temperature can be correlated with the
inner temperature although it has different dynamics.
Figure 10. Schematic of the measurement and data acquisition system used in this study.
study.
To measure the thermal temperature field around the engine, a FLIR Thermovision A40 thermal
imaging camera was used (FLIR Systems, Wilsonville, OR, USA), and a stationary IR camera was
used for scanning during long measurement processes, where continuous measurement in real time
is necessary. The IR detector of the camera is a 320 × 240 focal plane array uncooled microbolometer.
Its spectral range is 7.5 to 13 µm, its instantaneous field of view (IFOV) is 1.3 mrad, its noise
equivalent temperature difference at 60 Hz is 0.08 °C at 30 °C, its measurement uncertainty is ±2 °C
Appl. Sci. 2019, 9, 2253 10 of 22
Figure 11.
Figure 11.The
Thearrangement
arrangement of FLIR
of the the FLIR Thermovision
Thermovision A40 thermal
A40 thermal imaging imaging
camera incamera in the
the laboratory.
laboratory.
A digital data acquisition system was used to observe the state of the engine [62]. Various sensors
The thermal
and data fields
acquisition weredevices
(DAQ) captured andthis
serve observed
purpose during the whole
and send engine
all the data to operational process,
the control personal
from its start-up
computer to the
(PC). This PCtime of fullathrust,
contains controlassystem
showncreated
by the measured basic parameters
in the LabVIEW (National displayed
Instrumentsin
Figure 12. The infrared camera was positioned 1.5 m from the engine and was
Corporation, Austin, TX, USA) and Matlab/Simulink (MathWorks, Natick, MA, USA) software which turned on 0.5 h before
theable
is measurement
to control and invisualize
order totheallow the performance,
engine’s temperature save of itsand
sensor
tracktoall equalize
measuredwith that of and
parameters, the
environment. Before the
update the control algorithms. measurement, the values of the background air temperature, air humidity,
and For
the the
emissivity
design ofofthetheIRTengine surface
diagnostics werethe
system, determined and inputted
following parameters in the camera. The
are utilized:
environment in the laboratory had a relatively stable humidity (around 50%) and temperature
• n (rpm)—speed
(around 21 °C) duringofthetheexperiments
engine’s shaft.with no direct sunlight. These parameters can have an impact
• T (◦ C)—air temperature at the output of the radial
2
on the precision of the IRT measurements, but cancompressor.
be quite well-controlled in a laboratory
• T (◦ C)—gas temperature at the input to the gas turbine.
environment
3 [39]. The measured state parameters (n (rpm), T2 (°C), T3 (°C), T4 (°C)) for a single run of
• T4 (◦ C)—gas
the iSTC-21v enginetemperature at the
in laboratory output ofare
conditions theshown
gas turbine.
in Figure 12.
These parameters represent the inner state and temperature of the engine. The interior temperature
was measured by a thermal sensor contained in the engine. The temperature of the engine’s casing
was measured by a thermal camera; this temperature can be correlated with the inner temperature
although it has different dynamics.
To measure the thermal temperature field around the engine, a FLIR Thermovision A40 thermal
imaging camera was used (FLIR Systems, Wilsonville, OR, USA), and a stationary IR camera was
used for scanning during long measurement processes, where continuous measurement in real time is
necessary. The IR detector of the camera is a 320 × 240 focal plane array uncooled microbolometer. Its
spectral range is 7.5 to 13 µm, its instantaneous field of view (IFOV) is 1.3 mrad, its noise equivalent
temperature difference at 60 Hz is 0.08 ◦ C at 30 ◦ C, its measurement uncertainty is ±2 ◦ C or ± 2%, its
temperature sampling frequency is 60 Hz, and its observable temperature range is from −40 to 1500 ◦ C.
The camera was controlled via the PC using the ThermaCAM Researcher software (FLIR Systems,
Wilsonville, OR, USA) through the FireWire interface. One of the settings when capturing a video is
the choice of color palette, which is used to convert the levels of the infrared spectrum into the visible
spectrum. This selection is mainly important when there are small temperature differences between
parts of an object or its surroundings. In the case of an object with a wide range of temperatures, such
as a turbojet engine, the standard iron-bow palette is a suitable option [63]. As the engine is a dynamic
object, the displacement of its thermal field needs to be evaluated continuously. The camera allows
uncompressed video to be captured at 25 frames per second in audio video interleave (AVI) format
usingFigure
the selected color palette. The video can then be decomposed into single frames in uncompressed
12. The measured parameters of the iSTC-21v turbojet engine that are usable for infrared
bitmap (BMP) format. The
thermography-based resulting pictures contain IR images of the engine with the assigned RGB
diagnostics.
pixel values from the selected color palette during its operation. Full color resolution of the camera can
be obtained only when taking static images of the engine, which is not an option for the application,
but a video with this resolution would definitely improve the resulting IR images. It can be concluded
Appl. Sci. 2019, 9, 2253 11 of 22
Figure 11. The arrangement of the FLIR Thermovision A40 thermal imaging camera in the
here that parameters of the camera are sufficient for application on an object operating in a wide
laboratory.
temperature range, where fine temperature resolution is not that important.
The thermal fields were captured and observed during the whole engine operational process,
The thermal fields were captured and observed during the whole engine operational process,
from its start-up to the time of full thrust, as shown by the measured basic parameters displayed
from its start-up to the time of full thrust, as shown by the measured basic parameters displayed in
in Figure 12. The infrared camera was positioned 1.5 m from the engine and was turned on 0.5 h
Figure 12. The infrared camera was positioned 1.5 m from the engine and was turned on 0.5 h before
before the measurement in order to allow the temperature of its sensor to equalize with that of the
the measurement in order to allow the temperature of its sensor to equalize with that of the
environment. Before the measurement, the values of the background air temperature, air humidity, and
environment. Before the measurement, the values of the background air temperature, air humidity,
the emissivity of the engine surface were determined and inputted in the camera. The environment in
and the emissivity of the engine surface were determined and inputted in the camera. The
the laboratory had a relatively stable humidity (around 50%) and temperature (around 21 ◦ C) during
environment in the laboratory had a relatively stable humidity (around 50%) and temperature
the experiments with no direct sunlight. These parameters can have an impact on the precision of the
(around 21 °C) during the experiments with no direct sunlight. These parameters can have an impact
IRT measurements, but can be quite well-controlled in a laboratory environment [39]. The measured
on the precision of the IRT◦ measurements, but can be quite well-controlled in a laboratory
state parameters (n (rpm), T2 ( C), T3 (◦ C), T4 (◦ C)) for a single run of the iSTC-21v engine in laboratory
environment [39]. The measured state parameters (n (rpm), T2 (°C), T3 (°C), T4 (°C)) for a single run of
conditions are shown in Figure 12.
the iSTC-21v engine in laboratory conditions are shown in Figure 12.
The measured
Figure 12. The
Figure measured parameters
parameters of
of the
the iSTC-21v
iSTC-21v turbojet
turbojet engine
engine that
that are
are usable
usable for
for infrared
infrared
thermography-based diagnostics.
thermography-based
3. Thermal Image Segmentation and Classification
3.1. Design and Training of the Kohonen Self-Organizing Feature Map

The Kohonen SOFM was used for the segmentation of the IRT images obtained with the thermal
camera at a resolution of 320 × 240 pixels as described in Section 2.4. Each pixel was defined as
a five-dimensional vector P_i (t) = {R,G,B,x,y}, where R, G, and B represent the values of the color
spectrum {R,G,B} = <0,255> and x and y represent the coordinates of each pixel where x = <0,320>;
y = <0,240>. The engine’s temperature as evaluated and processed by the camera was not taken into
account, since it may contain errors created by the camera compensating for environmental conditions
and assigning different temperatures to a single color, thus confusing the classifier. In order to evaluate
the technical state of the engine, the relative temperature-field displacement was used as it is holds
the most important diagnostic information. The dynamic range of the camera was set in the interval
<0,500> ◦ C and was fixed. These parameters define the design of the SOFM. The pilot testing showed
that these multi-dimensional vectors should be classified into five classes. As proposed in Section 2.2,
• N1: the hottest and most critical part of the IRT image, red.
• N2: a hot and critical part of the IRT image, yellow.
• N3: a hot and non-critical part of the IRT image, green.
• N4: a low-temperature and non-critical part, cyan.
Appl. Sci. 2019, 9, 2253 12 of 22
• N5: the temperature of the surrounding objects (not a part of the diagnosed object), blue.
The structure of the network can be visualized as shown in Figure 13. The training parameters
awhich
feature map
were withto
found a single row is
be efficient ableset
were to as
acceptably segment
defined in Table 1,the
andIRT
theimage, withKohonen
standard low demands of
network
computational time and complexity,
learning rule was used [42,43]. which was one of the design goals.
The classes are labeled as Ni and represented by different colors as follows:
Table 1. Training parameters.
• N1 : the hottest and most critical part of the IRT image, red.
• N2 : a hot and critical part of theTopology
IRT image, yellow.
function Grid
• N3 : a hot and non-critical part of the IRT image, green.
Distance computation Direct
• N4 : a low-temperature and non-critical part, cyan.
• N5 : Initial neighborhood
the temperature of the surrounding size
objects (not a part of the3diagnosed object), blue.
Initial
The structure of the network can learning rate
be visualized 0.9 13. The training parameters
as shown in Figure
which were found to be efficient were set as defined in Table 1, and the standard Kohonen network
Training iterations 500
learning rule was used [42,43].
13.The
Figure 13. The structure
structure of theofKohonen
the Kohonen self-organizing
self-organizing feature
feature map map image
for infrared for infrared image
segmentation.
segmentation.
Table 1. Training parameters.
A training dataset was prepared to train

Topology the network as follows.
function Grid The engine was run for 120 s
and thermal images were recorded at two
Distance frames per second, asDirect
computation the pilot experiments showed that
this timing is sufficient sinceInitial neighborhood
the dynamics size temperature of
of the 3 the engine casing have a time
Initial learning rate 0.9
constant of 1.2 s. The training sample therefore contained a set of 240 thermal images. The run of the
Training iterations 500
A training dataset was prepared to train the network as follows. The engine was run for 120 s and
thermal images were recorded at two frames per second, as the pilot experiments showed that this
timing is sufficient since the dynamics of the temperature of the engine casing have a time constant
of 1.2 s. The training sample therefore contained a set of 240 thermal images. The run of the engine,
which was used to obtain the IRT images is shown in Figure 14. To demonstrate the ability of the
trained Kohonen SOFM, five operational points are shown in Figure 14, where the network was tested
if it produced sensible results. The first operational point, Op1 , corresponds to the lowest evaluated
rotational speed of the engine 42,058 rpm, and the last, Op5 , corresponds to the highest rotational speed
(i.e., 50,289 rpm).
The classifications of the five example images are shown in Figure 15a, and the created segment
areas are shown in Figure 15b. Following expert examination of the infrared images and heuristic
evaluation of the temperature field, it can be concluded that the Kohonen network works as expected
and is able to sensibly assign the hottest and coldest parts of the object to each of the five classes/segments.
In order to quantify the results, the number of activations of each neuron are summed, giving the total
segment areas S1 , . . . , S5 corresponding to each of the classes N1 , . . . , N5 . This can be formalized
as follows: Xp
Si = x j ; x = {0; 1} (5)
j=1
where p is the number of activations of the i-th neuron.

engine, which was used to obtain the IRT images is shown in Figure 14. To demonstrate the ability of
the trained Kohonen SOFM, five operational points are shown in Figure 14, where the network was
tested if it produced sensible results. The first operational point, Op1, corresponds to the lowest
evaluated rotational speed of the engine 42,058 rpm, and the last, Op5, corresponds to the highest
Appl. Sci. 2019,
rotational 9, 2253
speed (i.e., 50,289 rpm). 13 of 22
Appl. Sci. 2019, 9,Figure

x FOR 14.
Figure 14. The
PEER measured
measuredparameters
REVIEW
The parametersduring
during aa single
single run
run of
of the
the iSTC-21v
iSTC-21v engine.
engine. 14 of 23
The classifications of the five example images are shown in Figure 15a, and the created segment
areas are shown in Figure 15b. Following expert examination of the infrared images and heuristic
evaluation of the temperature field, it can be concluded that the Kohonen network works as
expected and is able to sensibly assign the hottest and coldest parts of the object to each of the five
classes/segments. In order to quantify the results, the number of activations of each neuron are
summed, giving the total segment areas S1, …, S5 corresponding to each of the classes N1, …, N5. This
can be formalized as follows:
𝑆 =∑ 𝑥 ; x = {0;1} (5)
where p is the number of activations of the i-th neuron.

In this way, the size of each class and the total area of each temperature segment can be
obtained. As can be seen in Figure 16, the total area of the segments in the three hottest classes
increased as the engine speed and temperature increased. As it is not possible to define the exact
areas which correspond to each class (as in traditional image segmentation where, for example, a
river area is clearly defined), a system for the objective evaluation of the trained classifier
performance was designed, and is described in the next section.
Figure15.
Figure 15.The
Thesegmented
segmentedthermal
thermalimages
imagesproduced
producedby
bythe
theKohonen
Kohonenmap.
map.
In this way, the size of each class and the total area of each temperature segment can be obtained.
As can be seen in Figure 16, the total area of the segments in the three hottest classes increased as
the engine speed and temperature increased. As it is not possible to define the exact areas which
correspond to each class (as in traditional image segmentation where, for example, a river area is clearly
defined), a system for the objective evaluation of the trained classifier performance was designed, and
is described in the next section.
Appl. Sci. 2019, 9, 2253 14 of 22
Figure 15. The segmented thermal images produced by the Kohonen map.
Figure 16. The

Figure16. size SSii of
The size of each
each segmented
segmented area
area for
for five
five example
example points.
points.
3.2. Classification System Testing and Results

3.2. Classification System Testing and Results
To validate the ability of the trained Kohonen network to segment the thermal image, a system for
To validate the ability of the trained Kohonen network to segment the thermal image, a system
its evaluation was designed as follows. Five artificial spheres, each with an area of 150 pixels, were
for its evaluation was designed as follows. Five artificial spheres, each with an area of 150 pixels,
created, each of which fell into only one of the classes. As can be seen in Figure 17, the network was
were created, each of which fell into only one of the classes. As can be seen in Figure 17, the network
able to correctly classify these artificial anomalies in this particular case.
was Sci.
Appl. able2019,
to correctly
9, x FOR classify
PEER these artificial anomalies in this particular case.
REVIEW 15 of 23
In order to properly evaluate the functionality of the classifier, five testing infrared images were
divided into 60 segments. The artificially created spherical temperature field anomaly for each class
was moved across all segments, giving a total of 300 testing images for each class and 1500 testing
images in total. The division of the image into segments when the artificial anomaly was included is
shown in Figure 18.
Figure 17.
Figure 17. Thermal
Thermal anomalies
anomalies correctly
correctly classified
classified by
by the
the Kohonen
Kohonen network.
network.
In order to properly evaluate the functionality of the classifier, five testing infrared images were
divided into 60 segments. The artificially created spherical temperature field anomaly for each class
was moved across all segments, giving a total of 300 testing images for each class and 1500 testing
images in total. The division of the image into segments when the artificial anomaly was included is
shown in Figure 18.
The results are summarized in the confusion matrix shown in Table 2; the first row represents
the real class of the anomaly and the first column represents the class output by the network. The
contingency table gives a clear picture of the network’s ability to correctly segment the infrared images.
The network achieved a very good accuracy for the first two classes, and only misclassified
objects for the three hottest classes. The lowest obtained accuracy was for the classification of class N5 ,
which was the coldest class and mostly represented the surroundings of the engine, which are of a
low temperature. This class also contained misclassified objects belonging to class N4 , which only
Figure 18. Division of the infrared image for the insertion of the artificial anomaly.
represented relatively cold parts of the engine or its surroundings. The most important classes for
the evaluation of the technical state of a turbojet engine are those which represent its hottest parts,
the real class of the anomaly and the first column represents the class output by the network. The
contingency table gives a clear picture of the network’s ability to correctly segment the infrared
images.
Appl. Sci. 2019, 9, 2253 15 of 22
that is, N1 , N2 , and N3 . The results could probably be improved by using a larger training dataset or
different Kohonen layer topologies, however they can nevertheless be evaluated as very successful for
the purpose of turbojet engine
Figure 17. diagnostics.
Thermal anomalies correctly classified by the Kohonen network.
Figure 18. Division of the infrared image for the insertion of the artificial anomaly.
anomaly.
Table 2. The confusion matrix of the trained Kohonen self-organizing feature map (SOFM).
the real class of the anomaly and the first column represents theClass
Actual class output by the network. The
contingency table gives a clear picture Nof the network’s
N2 ability
N3 to correctly
N4 segmentN5 the infrared
1
images.
N1 286 5 2 0 0
N2 10 284 15 1 0
Table 2. The
Predicted confusion N
Class matrix of the4trained Kohonen
10 self-organizing
272 feature
4 map (SOFM).
6
3
N4 0 1 10 275 34
Actual class
N5 0 0 1 20 260
N1 N2 N3 N4 N5
Accuracy 95.3% 94.6% 90.6% 91.6% 86%
N1 286 5 2 0 0
Total N2 10 284 (1377/1500)
15 1 0
91.8%
Accuracy
Predicted Class N3 4 10 272 4 6
N4 0 1 10 275 34
4. Expert Diagnostic System Design Based on Infrared Images
N5 0 0 1 20 260
4.1. Design of the Expert System Accuracy 95.3% 94.6% 90.6% 91.6% 86%
Total Accuracy 91.8% (1377/1500)
As presented in Figure 15, areas of the segments are not absolutely representative of the engine’s
technical
The state,
network as their changes
achieved are quite
a very goodslow as thefor
accuracy engine heatstwo
the first up classes,
or cools and
down andmisclassified
only the internal
thermal energy is transferred to its casing, which has its own heating and
objects for the three hottest classes. The lowest obtained accuracy was for the classification cooling dynamics. Theof main
class
task of the expert
N5, which was thesystem
coldest is class
to evaluate the areas
and mostly of the segments
represented and their changes,
the surroundings and to which
of the engine, then present
are of
to the operator
a low temperature. someThisinformation
class alsoabout the technical
contained state of
misclassified the engine.
objects belonging to class N4, which only
To fulfill this task, the designed expert system needs
represented relatively cold parts of the engine or its surroundings. as an input theThetotal areaimportant
most of the segments
classesand
for
some indication of how they evolve over time. Temporal changes in the areas
the evaluation of the technical state of a turbojet engine are those which represent its hottest parts, of the segments can be
predicted by the computation of derivations/differences between two segmented
that is, N1, N2, and N3. The results could probably be improved by using a larger training dataset or images in a set time
different∆t.
interval The standard
Kohonen derivative equation
layer topologies, howevercan they becan
used for this purpose
nevertheless for the i-th
be evaluated as neuron and the
very successful
total area
for the of the i-th
purpose segment
of turbojet Si : diagnostics.
engine
Si (t + ∆t) − Si (t)
dSi (t) = (6)
∆t
From a diagnostic point of view, it makes sense to investigate only the three hottest states (levels
S1 –S3 ) and their derivations S1_d –S3_d , as they represent the most critical temperatures for the engine.
The time derivation of the surface areas of the segments allows the magnitude of the change in the area
of different temperature regions to be determined and early structural failures to be detected. The
knowledge base of the proposed expert diagnostic system consists of a set of rules that describe normal
and atypical engine operation based on the outputs of the Kohonen SOFM. Each rule Ri (i = 1, 2, ..., 8)
Appl. Sci. 2019, 9, 2253 16 of 22
is designed according to precisely defined intervals of the allowed amount of pixels (area sizes) of
individual color levels and intervals of allowed values of their derivations, based on results for the
normal state and atypical states of the object. The outputs of the rules Ri take Boolean logical values Ri
Є{0; 1}. The rules Ri in Table 3 are defined as follows:
IF Si is the area size <from;to> AND Si_d is the derivative <from;to> => Ri , where Si (i = 1, 2, 3) (7)
Table 3. A partial set of rules in the knowledge base of the proposed expert diagnostic system.
IF S1 IS <13,200; 13,300> AND S1_d is <−5; 50 > => R1

IF S2 IS <3600; 3700> AND S2_d is <−10; 30> => R2
IF S3 IS <7300; 7420> AND S3_d is <−5 ; 45> => R3
IF S3_d IS <46; 160 > – – => R4
IF S3_d IS <−85; −6> – – => R5
IF S2_d IS <31; 140> – – => R6
IF S2_d IS <−120; −11> – – => R7
IF S1_d IS <51; 160> – – => R8
The rules Ri are partial conclusions about the technical state of the investigated engine. In order
to obtain some knowledge about the engine’s technical state, the hypotheses which are to be connected
with these rules have to be defined. In order to keep the rule base simple at first, diagnostic signals
(hypotheses) for a turbojet engine are proposed in Table 4.
Table 4. Technical states of the turbojet engine.
D0 Normal condition
D1 Light overheating
D2 Moderate overheating
D3 Critical overheating
D4 Light structural defect
D5 Critical structural defect
These diagnostic signals (Di ; i = 0, 1, 2, ..., 5) are activated by combinations of the proposed rules.
The rules to obtain the diagnostic signals are defined in Table 5.
Table 5. Inference of the diagnostic signals.
IF R1 AND R2 AND R3 => D0

The rules, hypotheses, and inputs can be formalized in the form of an oriented graph. The
inference network is defined in Section 2.3. In the case of a turbojet engine with diagnostics based on
the sizes and derivatives of segmented thermal images, the inference network shown in Figure 19 is
proposed based on rules in Tables 3 and 4, which represent the knowledge base of the expert system.
The inference engine then performs a bottom–up search from data of the segments’ sizes Si and their
derivatives Si_d through intermediate rules Rj up to evaluation of the hypotheses Dk .
Appl. Sci. 2019, 9, 2253 17 of 22
Figure 19.
Figure The inference
19. The inference network
network of
of the
the designed
designed diagnostic
diagnostic expert
expert system.
system.
4.2. Pilot Testing of the Expert System Using Infrared Thermography (IRT) Inputs
4.2. Pilot Testing of the Expert System Using Infrared Thermography (IRT) Inputs
The idea behind the design of the presented expert system is to evaluate its possibilities and
The idea behind the design of the presented expert system is to evaluate its possibilities and its
its ability to correctly present diagnostic signals to the engine’s operator or supervisory control
ability to correctly present diagnostic signals to the engine’s operator or supervisory control system
system [64,65]. The designed rule base was therefore constructed to be very simple in order to evaluate
[64,65]. The designed rule base was therefore constructed to be very simple in order to evaluate the
the five pre-defined temperature states represented by hypotheses D. Pilot tests were conducted to
five pre-defined temperature states represented by hypotheses D. Pilot tests were conducted to
prove that the system works as intended. For real-world engine operation, it was only possible to test
prove that the system works as intended. For real-world engine operation, it was only possible to
the states D0 , D1 , D4 , and D5 . The resulting classifications for these tested situations are shown in
test the states D0, D1, D4, and D5. The resulting classifications for these tested situations are shown in
Table 6. States D and D were artificially induced by quickly adding heating material on the surface
Table 6. States D44 and D55 were artificially induced by quickly adding heating material on the surface
of the engine’s exhaust nozzle, which created anomalies in the temperature field of the engine. The
of the engine’s exhaust nozzle, which created anomalies in the temperature field of the engine. The
condition of light overheating was achieved when the engine was running at a high speed setting, (i.e.,
condition of light overheating was achieved when the engine was running at a high speed setting,
at an operating speed of 50,000 rpm for around 20 s). The rules which were evaluated as true, and the
(i.e., at an operating speed of 50,000 rpm for around 20 s). The rules which were evaluated as true,
way the diagnostic signal/hypothesis was obtained, are shown in Table 6. These results suggest that
and the way the diagnostic signal/hypothesis was obtained, are shown in Table 6. These results
the diagnostic expert system is able to determine basic states represented by hypotheses D0 , D1 , D4 ,
suggest that the diagnostic expert system is able to determine basic states represented by hypotheses
D and that the designed rules work.
D05 , D1, D4, D5 and that the designed rules work.
Additionally, another test was set up using the data from a single engine run, as shown in Figure 14.
Normal and light overheating states were tested during standard operation of the engine, and are
shown in Figure 20. It can be seen that the expert system selected the state D1 (light overheating)
even though the speed of the engine and the corresponding temperatures had decreased. This was
the correct selection, since the temperature of the casing takes more time to cool down, whereas the
inner gas temperature as measured by internal sensors decreases immediately. This set of pilot tests
showed that the inference engine works as intended and that the expert system is able to produce some
sensible results in the evaluation of the technical state of the engine during normal conditions and to
detect artificially induced structural defects. In order to prove the correctness of the setting of the rules
Ri , as well as the definition of intervals in these rules, a more systematic approach has to be taken. This
will be designed in a follow-up study of the system.
Appl. Sci. 2019, 9, 2253 18 of 22
Appl.
Appl.
Appl.
Appl. Sci.
Sci.
Sci.
Sci. 2019,
2019,
2019,
2019, 9,9,xxxx
9,9, FOR
FOR
FOR
FOR PEERREVIEW
PEER
PEER
PEER REVIEW
REVIEW
REVIEW 1919of
1919 of23
ofof 23
2323
Table6.
Table 6. Table
Table
Table 6.Testing
6.6. Testing
Testing
Testing ofthe
of
ofof theexpert
the
the expert
expert
expert system
system
system
system forthe
for
for
for theevaluation
the
the evaluation
evaluation
evaluation of
ofof
ofof hypothesesD
hypotheses
hypotheses
hypotheses D
D
DD ,,DD
00,,00D D ,,DD
11,,11D D ,,and
44,,44andandD
and D
DD55..55..
Testing of the expert system for the evaluation hypotheses 0 , D1 , D4 , and D5 .
DD
D 00—normal
00—normal
—normal condition
condition
condition
R11::11S
RR :S1S1=11==13,270
13,270AND
13,270 ANDSS1_d
AND ===20
S1_d1_d
1_d 20
20
D0 —normal condition
= 13,270
R1 :RS21: S
RR22::2SS22=22==3628
3628 ANDSSS1_d
AND
3628AND
AND S2_d2_d
2_d2_d == 20
==14
1414
R2 : S2 = 3628 AND S2_d = 14
RR33:: 3SS33 = 7374, AND SS 3_d == 24
RR3:3:SS3=33==7374,
7374,AND
7374, AND
AND S3_d3_d
S3_d 3_d ==2424
24
R1 AND R2 AND R3 => D0
R11 AND
RR ANDR
11AND 22 AND
RR ANDR
22AND 33 =>
RR =>D
33=> DD00 00
D
DD11—light overheating
11—light
—light overheating
overheating
RRR S:S11S1—light
11::11D =11==13,227
13,227
13,227 ANDSS1_d
AND
AND
overheating S1_d1_d
1_d===31
3131
R1 : S1 = 13,227 AND S1_d = 31
RR2R
R :22::S22S
:S
22S 2=
=22==3643
3643
3643
3643 AND
AND
AND AND SS2_d
S2_d ===−5
S2_d2_d
2_d =-5
-5-5
R4 : S3_d = 112
R1 ANDR 44::44S
RR 2:S ===112
S3_d3_d
AND3_d
3_d
4 => D1
R112
112
RR
R11 ANDANDR
11AND RR
22 ANDANDR
22AND RR
44 => =>D
44=> DD11 11
D
DD44—light structural
44—light
—light structuraldefect
structural defect
defect
R
DR4R ::22S:S2S2=22==3671
22—light 3671
3671 structuralANDANDSdefect
AND ===333
S2_d2_d
S2_d
2_d
R2 : S2 = 3671 AND S2_d = 3

RR R:55S3_d
5R:55:S
:S3_d
S3_d3_d
====
3_d -67
-67
-67
−67
R8 : S1_d = 92
R2 ANDR RRR885::88S:S1_d ===92
S1_d1_d 928 => D4
92
AND 1_d R
R22 AND
RR ANDR
22AND 55 AND
RR ANDR
55AND 88 =>
RR =>D
88=> DD44 44
D
DD55—critical structural
55—critical
—critical structuraldefect
structural defect
defect
R RR
D5 —critical 55::55S :S3_d
S3_d3_d
structural
3_d ===-72
-72defect
-72
R5 : S3_d = −72
RR7R:R77:S:77S2_d
:S2_d ====
S2_d2_d
2_d -45
-45
-45
−45
R8 : S1_d = 118
R5 ANDR RRR88::788S:SAND
S1_d1_d
1_d1_d ===118
R118
1188 => D5
Appl. Sci. 2019, 9, x FOR PEER REVIEW

RR
R55 ANDANDR
55AND RR
77 ANDANDR
77AND RR
88 => =>D
88=> DD55 55
20 of 23
Additionally,
Additionally,another
Additionally, anothertest
another testwas
test wasset
was setup
set upupusing
usingthe
using thedata
the datafrom
data fromaaasingle
from singleengine
single enginerun,
engine run,as
run, asasshown
shownin
shown inin
Figure
Figure14.
Figure 14.Normal
14. Normaland
Normal andlight
and lightoverheating
light overheatingstates
overheating stateswere
states weretested
were testedduring
tested duringstandard
during standardoperation
standard operationof
operation ofofthe
theengine,
the engine,
engine,
andandare
and areshown
are shownin
shown ininFigure
Figure20.
Figure 20.It
20. ItItcan
canbe
can bebeseen seenthat
seen thatthe
that theexpert
the expertsystem
expert systemselected
system selectedthe
selected thestate
the stateD
state DD (light
11 11(light
(light
overheating)
overheating) even though the speed of the engine
overheating) even though the speed of the engine and the corresponding temperatures had
even though the speed of the engine and
and the
the corresponding
corresponding temperatures
temperatures had
had
decreased.
decreased. This
This was
was thethe correct
correct selection,
selection, since
since the
the temperature
temperature
decreased. This was the correct selection, since the temperature of the casing takes more time to cool of of the
the casing
casing takes
takes more
more time
time to to cool
cool
down,whereas
down,
down, whereasthe
whereas theinner
the innergas
inner gastemperature
gas temperatureas
temperature asasmeasured
measuredby
measured byinternal
by internalsensors
internal sensorsdecreases
sensors decreasesimmediately.
decreases immediately.
immediately.
This
This set of pilot tests showed that the inference engine works as intended and that the expert system
Thisset
setof ofpilot
pilot tests
tests showed
showed that
that the inference
the inference engine
engine works
works asasintended
intended and
and that
that the
theexpert
expert system
system
is
isisable
ableto
able totoproduce
producesome
produce somesensible
some sensibleresults
sensible resultsin
results ininthetheevaluation
the evaluationof
evaluation ofofthe
thetechnical
the technicalstate
technical stateof
state ofofthetheengine
the engineduring
engine during
during
normal
normalconditions
normal conditionsand
conditions andto
and totodetect
detectartificially
detect artificiallyinduced
artificially inducedstructural
induced structuraldefects.
structural defects.In
defects. InInorder
orderto
order totoprove
provethe
prove the
the
correctness
correctnessof
correctness ofofthe
thesetting
the settingof
setting ofofthetherules
the rulesR
rules RRii,,iiasaswell
,as wellas
well asthe
as thedefinition
the definitionof
definition ofofintervals
intervalsin
intervals ininthese
theserules,
these rules,aaamore
rules, more
more
systematic
systematicapproach
systematic approachhas
approach hasto
has totobe
bebetaken.
taken.This
taken. Thiswill
This willbe
will bebedesigned
designedin
designed ininaaafollow-up
follow-upstudy
follow-up studyof
study ofofthethesystem.
the system.
system.
20. The
FigureFigure division
20. The of of
division thetheengine’s
engine’s operation intotechnical
operation into technical states
states D0Dand
D0 and 1.
D1 .
5. Conclusions
This article defines a way to use infrared imaging directly in a new area of aviation, namely, in
the evaluation of the technical state of turbojet engines. A concept of a hybrid system combining a
SOFM, namely a Kohonen neural network, and an expert system, is shown to be a promising
approach to solve the problem of thermal image segmentation and to extract knowledge about the
Appl. Sci. 2019, 9, 2253 19 of 22
5. Conclusions
This article defines a way to use infrared imaging directly in a new area of aviation, namely, in the
evaluation of the technical state of turbojet engines. A concept of a hybrid system combining a SOFM,
namely a Kohonen neural network, and an expert system, is shown to be a promising approach to solve
the problem of thermal image segmentation and to extract knowledge about the technical state of the
engine from the obtained segmented IRT images. The Kohonen neural network defined in this article
is a computationally very simple algorithm, and was able to achieve a precision of higher than 90% in
the segmentation of infrared images obtained by the FLIR A40 camera when introducing an artificial
temperature anomaly. By using the total areas of segmented infrared images corresponding to different
temperatures, and their time derivatives, the technical state of the engine can be extracted and evaluated.
This can be considered as a novel approach, and it has shown promising results as a proof of concept,
which can be further developed and investigated, for example, to take into account environmental or
high noise conditions or the impact of application of different image filtering techniques. Moreover,
accuracy of the classification could also be increased by utilizing a camera with the ability to record a
video with 10 bit color resolution instead of the reduced iron-bow palette for applications, where fine
temperature resolution is important. On the other hand, the developed methodology with an expert
system seems to be robust enough to work well with the reduced color spectrum and the engine is
working in a wide temperature range. The rule base of the expert system proposed in this article can be
further expanded to contain more hypotheses, and can also be developed to use fuzzy logic to evaluate
the rules to improve the precision of the system. It can be considered as a progressive methodology
in the application of computational intelligence and infrared thermography to the area of aviation.
The methodology can also be applied in other fields of technical practice to evaluate the temperature
fields of different technical objects, such as electronic circuit boards, transmissions, rotating machinery,
generators, and electric engines operating in laboratory conditions. The system is designed to be
objective, and does not need any input or examination of the infrared images by humans. It can also be
integrated into a complex situational control algorithm of a turbojet engine in laboratory conditions,
where outputs of the expert system can be used as inputs for the situational classifier and also to tune
adaptive controllers of the engine [62,64,65].
Author Contributions: Conceptualization, R.A., M.S. (Martin Schrötter), and M.Č.; data curation, R.A. and L.F.;
formal analysis, R.A., L.F., M.S. (Martin Schrötter), and S.S.; funding acquisition, R.A. and S.S.; investigation, R.A.,
L.F., M.S. (Martin Schrötter), M.Č., R.B., and M.S. (Michal Schreiner); methodology, R.A., L.F., and R.B.; project
administration, R.A., M.S. (Martin Schrötter), and S.S.; resources, L.F., M.S. (Martin Schrötter), M.Č., R.B., and
M.S. (Michal Schreiner); supervision, R.A., M.Č., and S.S.; validation, R.A., L.F., M.S. (Martin Schrötter), and R.B.;
visualization, L.F., M.S. (Martin Schrötter), M.Č., and M.S. (Michal Schreiner); writing—original draft, R.A., L.F.,
M.S. (Martin Schrötter), and M.Č.; writing—review and editing, R.A., L.F., M.S. (Martin Schrötter), and M.Č.
Funding: This work was supported by the following projects: Efficient Systems and Propulsion for Small Aircraft
(ESPOSA), funded by the European commission in the seventh framework program under grant agreement
ACP1-GA-2011-284859-ESPOSA, the Slovak Research and Development Agency (APVV) under grant agreement
DO7RP-0023-11 and KEGA 044TUKE-4/2019—a small unmanned airplane—the platform for education in the area
of intelligent avionics.
Conflicts of Interest: The authors declare no conflicts of interest.
References
1. Osornio-Rios, R.A.; Antonino-Daviu, J.A.; Jesus Romero-Troncoso, R. Recent Industrial Applications of
Infrared Thermography: A Review. IEEE Trans. Ind. Inform. 2019, 15, 615–625. [CrossRef]
2. Lahiri, B.B.; Bagavathiappan, S.; Jayakumar, T.; Philip, J. Medical applications of infrared thermography:
A review. Infrared Phys. Technol. 2012, 55, 221–235. [CrossRef]
3. Balaras, C.A.; Argiriou, A.A. Infrared thermography for building diagnostics. Energy Build. 2002, 34, 171–183.
[CrossRef]
4. Lisowska, A. Thermographic monitoring of the power transformers. Meas. Autom. Monit. 2017, 63, 154–157.
Appl. Sci. 2019, 9, 2253 20 of 22
5. López-Pérez, D.; Antonino-Daviu, J. Application of Infrared Thermography to Failure Detection in Industrial

Induction Motors: Case Stories. IEEE Trans. Ind. Appl. 2017, 53, 1901–1908. [CrossRef]
6. Yang, R.; He, Y.; Mandelis, A.; Wang, N.; Wu, X.; Huang, S. Induction Infrared Thermography and
Thermal-Wave-Radar Analysis for Imaging Inspection and Diagnosis of Blade Composites. IEEE Trans. Ind.
Inform. 2018, 14, 5637–5647. [CrossRef]
7. Vavilov, V.; Nesteruk, D.A. Detecting water in aviation honeycomb structures: The quantitative approach.
Quant. Infrared Thermogr. J. 2004, 1, 173–184. [CrossRef]
8. Vavilov, V.; Pan, Y.Y.; Nesteruk, D. Infrared thermographic inspection of water ingress in composite
honeycomb panels. Appl. Opt. 2016, 55, 120–125. [CrossRef] [PubMed]
9. Vavilov, V.P. Non-contact one-sided evaluation of hidden corrosion in metallic constructions by using
transient infrared thermography. Rev. Metal. 2003, 39, 235–242. [CrossRef]
10. Eddazi, A.; Belattar, S. Nondestructive Testing Evaluation of Aircraft Fuselage Corrosion by Infrared
Thermography and Finite Element Method. In Proceedings of the 14th International Conference on
Computer Graphics, Imaging and Visualization, Marrakesh, Morocco, 23–25 May 2017.
11. Addepalli, S.; Zhao, Y.; Roy, R.; Galhenege, W.; Colle, M.; Yu, J.; Ucur, A. Non-destructive evaluation
of localised heat damage occurring in carbon composites using thermography and thermal diffusivity
measurement. Measurement 2019, 131, 706–713. [CrossRef]
12. Meola, C.; Boccardi, S.; Carlomagno, G.M. Infrared Thermography in the Evaluation of Aerospace Composite
Materials: Infrared Thermography to Composites; Woodhead Publishing: Cambridge, UK, 2016.
13. Swiderski, W. IR Thermography Nondestructive Testing Methods of Composite Materials Used in Aerospace
Applications. In Proceedings of the 12th International Conference on Quantitative Infrared Thermography,
The e-Journal of Nondestructive Testing, Mahabalipuram, India, 6–10 July 2016.
14. Orton, B.R.; Copaul, M.; Velez-Ospina, C.E. Potential use of infrared thermography to detect aircraft
pressurization leaks. Insight Non-Destr. Test. Cond. Monit. 1999, 41, 164–166.
15. Wang, Y.; Gao, B.; Lok Woo, W.; Tian, G.; Maldague, X.; Zheng, L.; Guo, Z.; Zhu, Y. Thermal Pattern Contrast
Diagnostic of Microcracks With Induction Thermography for Aircraft Braking Components. IEEE Trans. Ind.
Inform. 2018, 14, 5563–5574. [CrossRef]
16. White, J.E.; Tucholski, E.J.; Green, R.E. Nondestructive Testing of Aircraft and Spacecraft Wiring. Mater. Eval.
2003, 61, 1315–1320.
17. Tulloch, J.S.; Jennings, M.C. Thermographic Wiring Inspection. U.S. Patent No. US 6,995,565 B1, 7 February
2006.
18. Short, M.; Torres, J.; Kreckie, J. Thermal Imaging for Aircraft Rescue and Fire Fighting Applications;
DOT/FAA/TC-17/27 Final Report; U.S. Department of Transportation, Federal Aviation Administration:
Cambridge, MA, USA, 2017.
19. Štumper, M.; Kraus, J. Thermal Imaging in Aviation. Mag. Aviat. Dev. 2015, 3, 13–16. [CrossRef]
20. Matikainen, L.; Lehtomäki, M.; Ahokas, E.; Hyyppä, J.; Karjalainen, M.; Jaakkola, A.; Kukko, A.; Heinonen, T.
Remote sensing methods for power line corridor surveys. ISPRS J. Photogramm. Remote Sens. 2016, 119,
10–31. [CrossRef]
21. Gómez, C.; Green, D.R. Small unmanned airborne systems to support oil and gas pipeline monitoring and
mapping. Arab. J. Geosci. 2017, 10, 202. [CrossRef]
22. Kylili, A.; Fokaides, P.A.; Christou, P.; Kalogirou, S.A. Infrared thermography (IRT) applications for building
diagnostics: A review. Appl. Energy 2014, 134, 531–549. [CrossRef]
23. Omar, T.; Nehdi, M.L. Remote sensing of concrete bridge decks using unmanned aerial vehicle infrared
thermography. Autom. Constr. 2017, 83, 360–371. [CrossRef]
24. Harvey, M.C.; Rowland, J.; Luketina, K.M. Drone with Thermal Infrared Camera Provides high resolution
georeferenced imagery of the Waikite Geothermal Area, New Zealand. J. Volcanol. Geotherm. Res. 2016, 325,
61–69. [CrossRef]
25. Baranwal, N.; Mahulikar, S.P. Infrared Signature of Aircraft Engine with Choked Converging Nozzl.
J. Thermophys. Heat Transf. 2016, 30, 1–9. [CrossRef]
26. Ciampa, F.; Mahmoodi, P.; Pinto, F.; Meo, M. Recent Advances in Active Infrared Thermography for
Non-Destructive Testing of Aerospace Components. Sensors 2018, 18, 609. [CrossRef]
Appl. Sci. 2019, 9, 2253 21 of 22
27. MacLeod, J.D.; Steckhan, P.; He, D. Infrared Thermal Imaging System as a Diagnostic Tool for Gas Turbine
Engine Faults. In ASME 1994, International Gas Turbine and Aeroengine Congress and Exposition; American
Society of Mechanical Engineers (ASME): New York, NY, USA, 1994; Volume 2.
28. Bogdan, M.; Błachnio, J.; Kułaszka, A.; Derlatka, M. Assessing the Condition of Gas Turbine Rotor Blades
with the Optoelectronic and Thermographic Methods. Metals 2019, 9, 31. [CrossRef]
29. Yang, X.; Pang, S.; Shen, W.; Lin, X.; Jiang, K.; Wang, Y. Aero Engine Fault Diagnosis Using an Optimized
Extreme Learning Machine. Int. J. Aerosp. Eng. 2016, 2016. [CrossRef]
30. Ntantis, E.; Botsaris, P. Diagnostic Methods for an Aircraft Engine Performance. J. Eng. Sci. Technol. Rev.
2015, 8, 64–72. [CrossRef]
31. Rajani, J. Engine Health Condition Monitoring—A Brief Review. Control Data Fusion E J. 2017, 1, 25–38.
32. Armstrong, J.B.; Simon, D.L. Implementation of an Integrated On-Board Aircraft Engine Diagnostic
Architecture. In Proceedings of the 47th Joint Propulsion Conference and Exhibit cosponsored by the AIAA,
ASME, SAE, and ASEE, San Diego, CA, USA, 31 July 2011–3 August 2011.
33. Rabenoro, T.; Lacaille, J.; Cottrell, M.; Rossi, F. A Methodology for the Diagnostic of Aircraft Engine Based
on Indicators Aggregation. In Proceedings of the CDM 2014: Advances in Data Mining, Applications and
Theoretical Aspects, St. Petersburg, Russia, 16–20 July 2014; Volume 8557, pp. 144–158.
34. Seabra, J.; Campos, A. EHD Analysis of a Roller/Inner Ring Contact in a Jet Engine Roller Bearing.
In Proceedings of the 82nd Meeting of the AGARD Structures and Materials Panel, Sesimbra, Portugal,
6–7 May 1996; Volume 589.
35. Zaitouna, N.M.; Aqel, M.J. Survey on Image Segmentation Techniques. Procedia Comput. Sci. 2015, 65,
797–806. [CrossRef]
36. Karamizadeh, S.; Abdullah, S.M.; Zamani, M.; Kherikhah, A. Pattern Recognition Techniques: Studies on
Appropriate Classifications. ARPN J. Eng. Appl. Sci. 2014, 9, 791–799.
37. Spontón, H.; Cardelino, J. A Review of Classic Edge Detectors. Image Process. Line 2015, 5, 90–123. [CrossRef]
38. Egmont-Petersen, M.; Ridder, D.; Handels, H. Image processing with neural networks—A review. Pattern
Recognit. 2002, 35, 2279–2301. [CrossRef]
39. Vardasca, R.; Vaz, L.; Mendes, J. Classification and Decision Making of Medical Infrared Thermal Images. In
Classification in BioApps; Lecture Notes in Computational Vision and Biomechanics; Dey, N., Ashour, A., Borra, S.,
Eds.; Springer: Cham, Switzerland, 2018; Volume 26.
40. Zhiqiang, W.; Jun, L. A review of object detection based on convolutional neural network. In Proceedings of
the 2017 36th Chinese Control Conference (CCC), Dalian, China, 26–28 July 2017.
41. Rawat, W.; Wang, Z. Deep Convolutional Neural Networks for Image Classification: A Comprehensive
Review. Neural Comput. 2017, 29, 1–98. [CrossRef]
42. Kohonen, T. Self-Organizing Maps, 3rd ed.; Springer: Berlin/Heidelberg, Germany, 2001.
43. Kohonen, T. MATLAB Implementations and Applications of the Self-Organizing Map; Unigrafia Oy: Helsinki,
Finland, 2014.
44. Zimmermann, R.; Brandmeier, M.; Andreani, L.; Mhopjeni, K.; Gloaguen, R. Remote Sensing Exploration of
Nb-Ta-LREE-Enriched Carbonatite (Epembe/Namibia). Remote Sens. 2016, 8, 620. [CrossRef]
45. Kim, S.; Singh, V.P. Spatial Disaggregation of Areal Rainfall Using Two Different Artificial Neural Networks
Models. Water 2015, 7, 2707–2727. [CrossRef]
46. Chi, D. Self-Organizing Map-Based Color Image Segmentation with k-Means Clustering and Saliency Map.
ISRN Signal Process. 2011, 2011. [CrossRef]
47. Xu, Y.; Ran, J.; Chen, H. Kohonen Neural Network Classification for Failure Process of Metallic Organic
Coating in Corrosion Environment. Metals 2017, 7, 147.
48. Ganguli, R. Gas Turbine Diagnostics: Signal Processing and Fault Isolation; CRC Press: Boca Raton, FL, USA,
2013.
49. Li, Y.G. Performance Analysis Based Gas Turbine Diagnostics: A Review. Proc. Inst. Mech. Eng. Part A J.
Power Energy 2002, 216, 363–377. [CrossRef]
50. Mohammadi, E.; Montazeri-Gh, M. Performance Enhancement of Global Optimization-Based Gas Turbine
Fault Diagnosis Systems. J. Propuls. Power 2016, 32, 214–224. [CrossRef]
51. Li, Z.; Zhong, S.-S.; Lin, L. Novel Gas Turbine Fault Diagnosis Method Based on Performance Deviation
Model. J. Propuls. Power 2017, 33, 730–739. [CrossRef]
Appl. Sci. 2019, 9, 2253 22 of 22
52. Meskin, N.; Naderi, E.; Khorasani, K. A Multiple Model-Based Approach for Fault Diagnosis of Jet Engines.
IEEE Trans. Control Syst. Technol. 2013, 21, 254–262. [CrossRef]
53. Vanini, Z.N.S.; Khorasani, K.; Meskin, N. Fault detection and isolation of a dual spool gas turbine engine
using dynamic neural networks and multiple model approach. Inf. Sci. 2014, 259, 234–251. [CrossRef]
54. Szentkuti, A.; Kavanagh, H.S.; Grazio, S. Infrared thermography and image analysis for biomedical use.
Period. Biol. 2011, 113, 385–392.
55. Rojas, R. Neural Networks—A Systematic Introduction; Springer: Berlin, Germany; New York, NY, USA, 1996.
56. Abraham, A. Handbook of Measuring System: Design Rule-Based Expert Systems; John Wiley & Sons: New York,
NY, USA, 2005.
57. Buchanan, B.G.; Smith, R.G. Fundamentals of Expert Systems. Annu. Rev. Comput. Sci. 1988, 3, 23–58.
[CrossRef]
58. Yang, J.; Ye, C.; Zhang, X. An expert system shell for fault diagnosis. Robotica 2001, 19, 669–674. [CrossRef]
59. Chlebus, E.; Krot, K.; Kuliberda, M. Hybrid Artificial Intelligent Systems: Rule-Based Expert System Dedicated for
Technological Applications; Springer: Berlin/Heidelberg, Germany, 2011; pp. 373–380.
60. Dvořák, J. Expertní Systémy; Vysoké Učení Technické v Brne: 2004. Available online: http://www.uai.fme.
vutbr.cz/~{}jdvorak/Opory/ExpertniSystemy.pdf (accessed on 22 February 2019).
61. Sinha, A.P.; Zhao, H. Tuning Expert Systems for Cost-Sensitive Decisions. Adv. Artif. Intell. 2011, 2011.
[CrossRef]
62. Andoga, R.; Főző, L.; Judičák, J.; Bréda, R.; Szabo, S.; Rozenberg, R.; Džunda, M. Intelligent Situational
Control of Small Turbojet Engines. Int. J. Aerosp. Eng. 2018, 2018. [CrossRef]
63. Cruz-Albarran, I.A.; Benitez-Rangel, J.P.; Osornio-Rios, R.; Dominguez, B.T.; Medina, D.A.R.;
Morales-Hernández, L.A. A new approach to obtain a colour palette in thermographic images. Quant.
InfraRed Thermogr. J. 2019, 16, 35–54. [CrossRef]
64. Roman, R.-C.; Radac, M.-B.; Precup, R.-E.; Petriu, E.M. Data-driven Model-Free Adaptive Control Tuned by
Virtual Reference Feedback Tuning. Acta Polytech. Hung. 2016, 13, 83–96. [CrossRef]
65. Andoga, R.; Adamčík, F.; Hrabovský, J.; Vaispacher, T. A hybrid diagnostic system for a small turbojet engine.
NASE More 2016, 63, 86–92. [CrossRef]
© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access
article distributed under the terms and conditions of the Creative Commons Attribution
(CC BY) license (http://creativecommons.org/licenses/by/4.0/).

2019 - Intelligent Thermal Imaging-Based Diagnostics of Turbojet Engines

Uploaded by

Copyright:

Available Formats

2019 - Intelligent Thermal Imaging-Based Diagnostics of Turbojet Engines

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

2019 - Intelligent Thermal Imaging-Based Diagnostics of Turbojet Engines

Uploaded by

Copyright:

Available Formats

applied

Keywords: expert systems; infrared thermography; intelligent diagnostics; image segmentation;

Appl. Sci. 2019, 9, 2253; doi:10.3390/app9112253 www.mdpi.com/journal/applsci

2.1. Using Infrared Imaging for Diagnostics

• Image: an object to be captured by a thermal camera.

2.2. The other advantage

Figure 3. The structure of a Kohonen network.

Figure 3. The structure of a Kohonen network.

Appl. Sci. 2019, 9, x FOR PEER REVIEW 7 of 23

Figure 8. An example of an AND/OR inference network (reproduced from [61]).

Figure 8. An example of an AND/OR inference network (reproduced from [61]).

3. Thermal Image Segmentation and Classification

3.1. Design and Training of the Kohonen Self-Organizing Feature Map

A training dataset was prepared to train

where p is the number of activations of the i-th neuron.

Appl. Sci. 2019, 9,Figure

where p is the number of activations of the i-th neuron.

Figure 16. The

3.2. Classification System Testing and Results

IF S1 IS <13,200; 13,300> AND S1_d is <−5; 50 > => R1

Table 4. Technical states of the turbojet engine.

Table 5. Inference of the diagnostic signals.

IF R1 AND R2 AND R3 => D0

R2 : S2 = 3671 AND S2_d = 3

Appl. Sci. 2019, 9, x FOR PEER REVIEW

5. López-Pérez, D.; Antonino-Daviu, J. Application of Infrared Thermography to Failure Detection in Industrial

You might also like