Heart Disease Prediction and Classification Using Machine Learning Algorithms Optimized by Particle Swarm Optimization and Ant Colony Optimization
Heart Disease Prediction and Classification Using Machine Learning Algorithms Optimized by Particle Swarm Optimization and Ant Colony Optimization
Heart Disease Prediction and Classification Using Machine Learning Algorithms Optimized by Particle Swarm Optimization and Ant Colony Optimization
1
Laboratory of Innovation for New Energy Technologies and Nanomaterials (LITEN),
Faculty of Sciences and Techniques, Hassan 1st University, Settat, Morocco
* Corresponding author’s Email: [email protected]
Abstract: The prediction of heart disease is one of the areas where machine learning can be implemented.
Optimization algorithms have the advantage of dealing with complex non-linear problems with a good flexibility and
adaptability. In this paper, we exploited the Fast Correlation-Based Feature Selection (FCBF) method to filter
redundant features in order to improve the quality of heart disease classification. Then, we perform a classification
based on different classification algorithms such as K-Nearest Neighbour, Support Vector Machine, Naïve Bayes,
Random Forest and a Multilayer Perception | Artificial Neural Network optimized by Particle Swarm Optimization
(PSO) combined with Ant Colony Optimization (ACO) approaches. The proposed mixed approach is applied to
heart disease dataset; the results demonstrate the efficacy and robustness of the proposed hybrid method in
processing various types of data for heart disease classification. Therefore, this study examines the different machine
learning algorithms and compares the results using different performance measures, i.e. accuracy, precision, recall,
f1-score, etc. A maximum classification accuracy of 99.65% using the optimized model proposed by FCBF, PSO
and ACO. The results show that the performance of the proposed system is superior to that of the classification
technique presented above.
Keywords: Heart disease, Artificial neural network, K-nearest neighbour, Support vector machine, Naïve bayes,
Random forest, Classification, Feature selection, Ant colony optimization, Particle swarm optimization, Machine
learning.
fields because of their versatility [4, 5]. The Particle to machine learning, is effective in reducing
Swarm Optimization (PSO) algorithm has been dimensionality, eliminating irrelevant data,
successfully applied to heart disease because of its increasing learning accuracy and improving
simplicity and generality [6]. However, PSO easily understanding of results. In the second step, PSO
fell into the optimal local solution. In addition, the and ACO are applied to select the relevant
ACO algorithm was originally introduced for characteristics of the data set. The best subset of
combinatorial optimization. Recently, ACO characteristics selected by the characteristic
algorithms have been developed to solve continuous selection methods improves the accuracy of the
optimization problems. These problems are classification. Therefore, the third step applies
characterized by the fact that decision variables have classification methods to diagnose heart disease and
continuous domains, unlike discrete problems [7]. measures the classification accuracy to evaluate the
Using a single optimization algorithm has the performance of characteristic selection methods.
disadvantages of low accuracy and generalizability The main objective of this article is the
in solving complex problems. To further explore the prediction heart disease using different classification
application of intelligent optimization in algorithms such as K-Nearest Neighbour, Support
bioinformatics, PSO and ACO are combined in this Vector Machine, Naïve Bayes, Random Forest and a
article, meaning that exploitation and exploration Multilayer Perception | Artificial Neural Network
capacity are combined for binary and multi-class optimized by Particle Swarm Optimization (PSO)
heart disease. In this article, the Fast Correlation- combined with Ant Colony Optimization (ACO)
Based Feature selection (FCBF) method [8] used to approaches. The weka data-mining tool is used to
remove redundant and irrelevant features, the results analyze data from a heart disease. The main
of the PSO optimization are considered the initial contributions of this paper are:
values of the ACO, and then the classification model
for heart disease is constructed after the parameters Extraction of classified accuracy useful for heart
are adjusted. In this study, algorithms such as K- disease prediction
Nearest Neighbour (K-NN), Support Vector Remove redundant and irrelevant features with
Machine (SVM), Naïve Bayes (NB), Random Forest Fast Correlation-Based Feature selection
(RF) and Artificial Neural Network (ANN | MLP) (FCBF) method.
are used. It can be concluded that K-Nearest Optimizations with Particle Swarm
Neighbour and the Random Forest are the best Optimization PSO then we consider the result of
algorithms for the prediction and classification of PSO the initial values of Ant Colony
heart disease dataset. Optimization ACO approaches.
Comparison of different data mining algorithms
1.1 Problem statement on the heart disease dataset.
Identification of the best performance-based
Previous research studies has examined the algorithm for heart disease prediction.
application of machine learning techniques for the
prediction and classification of Heart disease. 1.2 Paper outline
However, these studies focus on the particular
impacts of specific machine learning techniques and The remainder of this paper is organized as
not on the optimization of these techniques using follows. Recent work in this area is discussed in
optimised methods. In addition, few researchers Section 2. Section 3 describes the detailed
attempt to use hybrid optimization methods for an description of the proposed methodology. Section 4
optimized classification of machine learning. The explains in detail the experiments using the
most proposed studies in the literature exploit proposed machine learning models. Finally, Section
optimized techniques such as Particle Swarm 5 presents conclusions and future research directions.
Optimization and Ant Colony Optimization with a
specific ML technique such as SVM, KNN or 2. Related work
Random Forest.
Several experiments are conducted on medical
In this work the Fast Correlation-Based Feature
data sets using multiple classifiers and features
Selection (FCBF) method applied as a first step
selection techniques. There is little research on the
(pre-treatment). When all continuous attributes are
classification of the heart disease dataset. Many of
discretized, the attribute selection attributes relevant
them show good classification accuracy [9].
to mining, from among all the original attributes, are
Tan et al. [10] proposed a hybrid method in
selected. Feature selection, as a pre-processing step
which two machine learning algorithms, Support
International Journal of Intelligent Engineering and Systems, Vol.12, No.1, 2019 DOI: 10.22266/ijies2019.0228.24
Received: October 22, 2018 244
Vector Machine (SVM) and Genetic Algorithm Vembandasamy et al. [14] diagnosed heart
(G.A), are effectively combined with the wrapper disease using the Naive Bayes algorithm. Bayes'
approach. The LIBSVM and the WEKA data mining theorem is used in Naive Bayes. Therefore, Naïve
tool are used to analyze the results of this method. Bayes has a powerful principle of independence.
Five data sets (Iris, diabetes disease, breast cancer The data used are from one of the leading diabetes
disease, heart disease and hepatitis) are collected research institutes in Chennai. The data set consists
from the Irvine UC machine learning repository for of 500 patients. WEKA is used as a tool and
this experiment. After applying the hybrid GA and performs classification using 70% of the Percentage
SVM approach, an accuracy of 84.07% is obtained Split. Naive Bayes offers 86.419% accuracy.
for heart disease. For all diabetes data, 78.26% Some few papers proposed hybrid classification
accuracy is achieved. The accuracy for breast cancer techniques.
is 76.20%. The 86.12% accuracy is the result of X. Liu et al. [15] presented a study to assist in
hepatitis disease. the diagnosis of heart disease using a hybrid
Otoom et al. [11] presented a system for analysis classification system based on the ReliefF and
and follow-up. Coronary artery disease is detected Rough Set (RFRS) method. The proposed system
and monitored by the proposed system. Cleveland consists of two subsystems: the RFRS feature
Heart data are taken from the UCI. This dataset selection system and a classification system with an
consists of 303 cases and 76 attributes/features. 13 overall classifier. A maximum classification
features are used out of 76 features. Two tests with accuracy of 92.59% was achieved according to a
three algorithms: Bayes Naive, Support vector cross-validation scheme of the jackknife.
machine, and Functional Trees FT are performed for A. Malav et al. [16] propose an effective hybrid
detection purposes. The WEKA tool is used for algorithmic approach for predicting heart disease, in
detection. After testing the Holdout test, the 88.3% order to determine and extract unknown knowledge
accuracy is achieved using the SVM technique. In about heart disease using the hybrid method
the cross-validation test, SVM and Bayes net combining the K-means clustering algorithm and the
provide 83.8% accuracy. The accuracy of 81.5 % is artificial neural network. The proposed model
achieved after the use of FT. The 7 best features are achieves an accuracy of 97%.
selected using the Best First selection algorithm. For The common objective of all these techniques is
validation, cross-validation tests are used. By to classify hearth disease using hybrid classification
applying the test to the 7 best features selected, techniques. However, they used only one
Bayes Naive achieved 84.5% accuracy, SVM classification and optimization technique. The
provides 85.1% accuracy and FT classifies 84.5% proposed approach presented a systematic way to
correctly. achieve the desired results by taking into account
Parthiban et al. [12] diagnosed heart disease in different technical optimizations with different
diabetic patients using automatic learning methods. machine learning algorithms.
Naïve Bayes and SVM algorithms are applied using In this article, we present a hybrid approach that
WEKA. A data set of 500 patients collected from involves combining different techniques exploited
the Chennai Research Institute is used. There are the Fast Correlation-Based Feature Selection
142 patients with the disease and 358 patients do not (FCBF) method to filter redundant features in order
have the disease. Using the Naive Bayes algorithm to improve the quality of heart disease classification.
provides 74% accuracy. SVM provides the highest Then, we perform a classification based on different
accuracy of 94.60%. classification algorithms such as K-Nearest
Chaurasia et al. [13] suggested using data Neighbour, Support Vector Machine, Naïve Bayes,
mining approaches to detect heart disease. The Random Forest and a Multilayer Perception |
WEKA data mining tool is used which contains a set Artificial Neural Network optimized by Particle
of machine learning algorithms for mining purposes. Swarm Optimization (PSO) combined with Ant
Naive Bayes, J48 and bagging are used for this Colony Optimization (ACO) approaches
perspective. The UCI machine learning laboratory
provides a data set on heart disease that includes 76 3. Methodology
attributes. Only 11 attributes are used for prediction.
This section includes our PSO/ACO based
Naive berries offer 82.31% accuracy. J48 gives
feature selection and classification system. The main
84.35% accuracy. 85.03% of the accuracy is
structure of the proposed system is shown in Fig. 1.
obtained by bagging. Bagging provides a better
Our system consists of feature selection based Fast
classification rate on this data set.
International Journal of Intelligent Engineering and Systems, Vol.12, No.1, 2019 DOI: 10.22266/ijies2019.0228.24
Received: October 22, 2018 245
Training Heart
Deases dataset
Testing Heart
Deases dataset
Feature Feature
Selection Optimization
Fast Correlation-Based
Feature selection (FCBF)
Preprocessing
Classification
Results
Correlation-Based Feature selection (FCBF), feature PSO/ACO algorithm and then, by using the selected
selection based PSO (Particle Swarm Optimization) best features, new features are classified with
combined with ACO (Ant Colony Optimization), WEKA [17] data mining software implemented in
and classification components based on K-Nearest Java. The components of our proposed system are
Neighbour, Support Vector Machine, Naïve Bayes, explained in detail in the following subsections. Fig.
Random Forest and Artificial Neural Network. The 1 shows the proposed architecture.
training dataset is prepared according to a binary
class classification problem. From the training 3.1 Data set and attributes
dataset, features are selected, after that the best
subset of features is optimized by our combined The data is collected from the UCI machine
learning repository. The data set is named Heart
International Journal of Intelligent Engineering and Systems, Vol.12, No.1, 2019 DOI: 10.22266/ijies2019.0228.24
Received: October 22, 2018 246
Disease DataSet and can be found in the UCI calculation time and costs will be high. Therefore,
machine learning repository. The UCI machine the feature selection, as a pre-treatment step to
learning repository contains a vast and varied machine learning, reduces sizing, eliminates
amount of datasets which include datasets from unresolved data, increases learning accuracy, and
various domains. These data are widely used by improves understanding of results. The recent
machine learning community from novices to increase in the dimensionality of the data poses a
experts to understand data empirically. Various serious problem to the methods of selecting
academic papers and researches have been characteristics with regard to efficiency and
conducted using this repository. This repository was effectiveness. The FCBF's reliable method [8] is
created in 1987 by David Aha and fellow students at adopted to select a subset of discriminatory features
UCI Irvine. Heart disease dataset contains data from prior to classification, by eliminating attributes with
four institutions [18]. little or no effect, FCBF provides good performance
1. Cleveland Clinic Foundation. with full consideration of feature correlation and
2. Hungarian Institute of Cardiology, Budapest. redundancy. In this document, we first standardized
3. V.A. Medical Centre, Long Beach, CA. the data and then selected the features by FCBF in
4. University Hospital, Zurich, Switzerland. WEKA. The number of heart disease attributes
increased from 14 to 7.
For the purpose of this study, the data set
provided by the Cleveland Clinic Foundation is used. 3.4 Feature optimisation
This dataset was provided by Robert Detrano, M.D,
Ph.D. Reason to choose this dataset is, it has less 3.4.1. Particle swarm optimization (PSO)
missing values and is also widely used by the
research community [19]. Swarm intelligence is a distributed solution to
complex problems which intend to solve
3.2 Classification Task complicated problems by interactions between
simple agents and their environment [27 - 29]. In
From the perspective of automatic learning, heart 1995, Russel Eberhart, an electrical engineer and
disease detection can be seen as a classification or James Kennedy, socio-psychologist, were inspired
clustering problem. On the other hand, we formed a by the living world to set up a metaheuristic:
model on the vast set of presence and absence file optimization by particle swarm. This method is
data; we can reduce this problem to classification. based on the collaboration of individuals between
For known families, this problem can be reduced to them: each particle moves and at each iteration, the
one classification only - having a limited set of one closest to the optimum communicates its
classes, including the heart disease sample, it is position to the others so that they can modify their
easier to identify the right class, and the result would trajectory. This idea is that a group of unintelligent
be more accurate than with clustering algorithms. In individuals may have a complex global organization.
this section, the theoretical context is given on all the Due to its recent nature, a lot of research is being
methods used in this research. For the purpose of done on P.S.O., but the most effective so far is the
comparative analysis, five Machine Learning extension to the framework of combinatorial
algorithms are discussed. The different Machine optimization.
Learning (ML) algorithms are K-Nearest Neighbour To apply the PSO it is necessary to define a
(KNN), Random Forest (RF), Support Vector research space made up of particles and an objective
Machine (SVM), Naïve Bayes and Artificial Neural function to be optimized. The principle of the
Network (ANN). The reason to choose these algorithm is to move these particles so that they find
algorithms is based on their popularity [20]. the optimum. Each of these particles is equipped
with:
3.3 Feature selection
From a position, i.e. its coordinates in the
In the heart disease datasets, the number of definition set.
features can reach up to tens of thousands; the heart A speed that allows the particle to move.
disease dataset has 14 attributes. Since a large In this way, during the iterations, each
number of irrelevant and redundant attributes are particle changes position. It evolves
involved in these expression data, the heart disease
according to its best neighbour, its best
classification task is made more complex. If
complete data are used to perform heart disease position, and its previous position. It is
classification, accuracy will not be as accurate, and
International Journal of Intelligent Engineering and Systems, Vol.12, No.1, 2019 DOI: 10.22266/ijies2019.0228.24
Received: October 22, 2018 247
International Journal of Intelligent Engineering and Systems, Vol.12, No.1, 2019 DOI: 10.22266/ijies2019.0228.24
Received: October 22, 2018 248
International Journal of Intelligent Engineering and Systems, Vol.12, No.1, 2019 DOI: 10.22266/ijies2019.0228.24
Received: October 22, 2018 249
International Journal of Intelligent Engineering and Systems, Vol.12, No.1, 2019 DOI: 10.22266/ijies2019.0228.24
Received: October 22, 2018 250
Table 10. Confusion Matrix disease, based on the data of each attribute for each
Absence Presence Class patient. Our goal was to compare different
K-NN 113 37 Absence classification models and define the most efficient
Classifiers without optimization
31 89 Presence
one. From all the tables above, different algorithms
performed better depending upon the situation
SVM 130 20 Absence whether cross-validation, grid search, calibration
24 96 Presence and feature selection is used or not. Every algorithm
RF 127 23 Absence has its intrinsic capacity to outperform other
27 93 Presence algorithm depending upon the situation. For
NB 130 20 Absence
example, Random Forest performs much better with
a large number of datasets than when data is small
24 96 Presence while Support Vector Machine performs better with
MLP 125 25 Absence a smaller number of data sets. Performance of
23 97 Presence algorithms decreased after boosting in the data,
K-NN 125 25 Absence which did not feature, selected while algorithms
were performing better without boosting infeature-
Classifiers optimized by FCBF
33 87 Presence
selected data. This shows the necessity that the data
SVM 129 21 Absence should be feature selected before applying to boost.
24 96 Presence For the comparison of the dataset, performance
RF 127 23 Absence metrics after feature selection, parameter tuning and
30 90 Presence calibration are used because this is a standard
process of evaluating algorithms. The precision
NB 131 19 Absence
average value of the best performance without
24 96 Presence optimization it’s for SVM and NB with 83.6% than
MLP 133 17 Absence RF with 81.4%. These shows SVM and NB are
26 94 Presence performing on average, after optimized by FCBF we
K-NN 150 0 Absence
find the best performance of precision it’s for MLP
with 84.2% than NB with 84% shown In Table 10.
Classifiers optimized by FCBF,
23 97 Presence FCBF, PSO and ACO, we find the best one is K-NN
RF 149 1 Absence with 99.7 % than RF with 99.6 %.
0 120 Presence 4.2.2. Comparison results
NB 136 14 Absence
24 96 Presence We tested the proposed Classifiers optimized by
FCBF, PSO and ACO against other classifications
MLP 144 6 Absence
models used for hearth diseases classification
18 102 Presence described in the related work section. Table 11
compares the proposed our classification technique
4.2 Discussion and comparison with previous research results. Compared to the
existing methods and experiment results, we find
4.2.1. Results discussion that our optimized model performs better than the
In this paper, we applied machine learning other models alone in heart disease prediction and
algorithms on heart diseasedataset to predict heart classification. We take advantage of both the FCBF
International Journal of Intelligent Engineering and Systems, Vol.12, No.1, 2019 DOI: 10.22266/ijies2019.0228.24
Received: October 22, 2018 251
International Journal of Intelligent Engineering and Systems, Vol.12, No.1, 2019 DOI: 10.22266/ijies2019.0228.24
Received: October 22, 2018 252