Three Stream Network Model for Lung Cancer Classification in the CT Images

2.1 Machine learning approaches

The machine learning approaches are mainly based on different handcrafted features and supervised classifiers such as Artificial Neural Network (ANN), Support Vector Machine, and Nearest Neighborhood. These classifiers should be trained with specific features to get proper classification performance. The existing machine learning techniques that are focused on lung cancer classification are discussed. In ANN, the input features play a significant role. Also, it is complex and requires a lot of computations. ANN based ensemble is proposed by Zhou et al. [13] to obtain high accuracy in the classification of carcinoma. Zhu et al. [14] selected 25 features based on the textures in Solitary Pulmonary Nodules (SPN) obtained from the CT images that are classified using the SVM classifier. Similarly, Shao et al. [15] proposed an automated framework to detect the SPN effectively using the SVM classifier. Orozco et al. [16] used features obtained from the wavelet transform to feed it to the SVM for further classification of lung cancer. Features extracted from the gabor filters are given to the K-nearest neighborhood (KNN), a classifier that is optimized using genetic algorithms [17]. The genetic algorithm depends on mutation which is unsupervised. Akram, Javed, and Hussain [18] used thresholding to segment the portion of the lung nodules which is a trial and error method. Later, features are extracted from these regions and fed to the ANN to reach an accuracy of 88.37%. Carvalho Filho et al. [19] utilized taxonomic diversity and distinct criteria to classify lung cancer using SVM classifiers. It shows an accuracy of 98.11%. Sangamithraa and Govindaraju [20] used Gray Level Co-occurrence Matrix (GLCM) to extract the features and it is fed to a Back Propagation Network to get 90.7% accuracy in classification.

2.2 Deep Learning approaches

Deep learning approaches work without any handcrafted features; it automatically extracts the features from the images. This approach is categorized as Traditional CNN, Hybrid CNN, and Transfer learning CNN. The Traditional CNN implies a single deep neural network structure which is a combination of different convolutional layers. Hybrid CNN is the combination of two or more deep neural networks. Transfer Learning CNN is a pretained network that can be fine-tuned by addition/deletion of layers and setting various hyperparameters as per the requirements.

3.1 Preparation of Dataset and Preprocessing

For lung cancer classification, the LIDC/IDRI database is used. It consists of 1018 sample images. For training, 80% of the dataset is used, and the remaining 20% is used for testing. The training images are augmented using rotation, flipping, resizing, and translation. All the images are re-sized into 256×256. A sample image of both normal and malignant lung nodules is shown in Figure 2. Image preprocessing is performing after converting the image to the frequency domain. Un-decimated Discrete Wavelet Transform (UDWT) is used for frequency conversion.

Figure 2

Proposed Tri stream deep network-based model.

For preprocessing the training images, two methods are used here. The first one is the Contrast Limited Adaptive Histogram Equalization (CLAHE) [34, 35] and the other is the Z-score normalization. Preprocessing enhances the image, and then Inverse Un-decimated Discrete Wavelet Transform (IUDWT) is applied. Equalization cum normalization gives better contrast-enhanced images for better training [36, 37] The Z-score normalization is defined by;

where i ranges from 1 to n, x is feature value and std refers to standard deviation.

3.2 Basic CNN Architecture

CNN architecture is generally composed of the convolutional layer, which is the building block, Rectified Linear Unit (ReLU) activation function, and pooling layers. Convolutional layers are used to extract low-level features to high-level features. A depth of many convolutional layers is used to extract these features. This gives an adequate information of the entire image. The ReLU layers help to allow significant information to the next stage. It provides non-linearity to the network. The pooling layer is used to minimize the dimension to reduce computational complexity.

3.3 Residual Architecture for classification

A fine tuned Residual neural network is used as the first stream in this tri-stream deep network model. ResNet is a deep neural network that has a significant connection, i.e., skip or short connection over the network layers, unlike conventional deep neural networks. ResNet is able to train the images with more than 50 layers successfully. It is given in Table 1. Instead of increasing network depth by simply stacking layers together, ResNet introduces an “identity shortcut connection” that skips one or more layers. It is shown in the following Figure 3. This identity connection is very much helpful to avoid the vanishing gradients problems in the network by using the previous layer activations until the present layer learns its weights.

Figure 3

Method of Preprocessing.

During training, the weights adapt to mute the upstream layer and amplify the previously-skipped layer. In the simplest case, only the weights for the adjacent layer's connection are adapted, with no exact weights for the upstream layer. Skipping effectively simplifies the network, using fewer layers in the initial training stages. This speeds learning by reducing the impact of vanishing gradients, as there are fewer layers to propagate through. The network then gradually restores the skipped layers as it learns the feature space. Towards the end of the training, when all layers are expanded, it stays closer to the manifold and thus learns faster. The pre-trained ResNet50 is fine-tuned in the last layers as well as in the fully connected layer to perform the lung cancer classification. It automatically extracts low level to high-level features related to the lung nodules. The skip connection is shown in Figure 4. The Figure 5 represents the different number of layers in the Residual networks.

Figure 4

Identity connections in the ResNet.

Figure 5

Transfer learning model ResNet-18, 34, 50, 101, 152.

3.4 Machine learning architecture

This machine learning architecture [38, 39] is the second stream of the proposed model. The artificial features are calculated from both the spatial [12] and frequency domain. The frequency-domain conversion is carried out by Un-decimated Discrete Wavelet Transform (UDWT). The features that are extracted in both low frequency and high-frequency sub-bands are listed in Table 1. The Energy (EN), Entropy (ENP), and Average (AVG) feature sets are expressed in equation (2), (3), (4).

Where, EN = Energy, ENP = Entropy, AVG = Average, Fv(x, y) = Frequency values of wavelets, p = image width, q = image height

3.5 Custom DNN for lung cancer classification

The custom DNN architecture forms the third stream of the tri stream deep neural network. It is used to extract the essential low level, mid-level, and high-level features automatically using the convolutional layers. The layers used in the custom DNN are given in Table 2. The kernel size of all layers is 3×3, and the number of filters in each convolutional layer is 256, 256, 512, 512, and 512 respectively. The first three convolutional layers have the ReLU activation function, and the last two convolutional layers have the leaky ReLU activation function. Figure 6 shows the block diagram of the custom DNN architecture. The entire training image is normalized by Z-score normalize, and classification is done by softmax classifier. The first and second fully connected layer has 1000 connections. It acts as the neural network classifier. The third fully connected layer represents the number of classes, here it is two. It is found that the augmented training data provide better results after performing training on both the augmented and raw training data.

Figure 6

Architecture of proposed custom deep neural network.

4.1 Performance measures

In this paper, the proposed architecture is evaluated using metrics such as Accuracy (Acc), Specificity (Spec), and Sensitivity (Sen). It is expressed in equation (5), (6), (7). Here, TN, FP, TN, and FN denote True positive, False Positive, True Negative, and False Negative, respectively.

4.2 Training

The augmented dataset is given as input to the three-stream architecture for feature extraction and prediction. Then the obtained prediction scores are combined using weighted fusion to classify the cancerous or non-cancerous samples. The experiment is carried out using MATLAB 2019 in an NVIDIA GEFORCE GPU. Adam optimization is used to minimize the error and train the network efficiently. In this paper, the used batch size is 32, learning a rate of 0.0001, 100 epochs, and momentum 0.9.

4.3 Testing

The trained model is tested on the test dataset to prove its efficiency using the performance metrics. The experiments are carried out by random splitting of training and testing data by 80% and 20% in both the augmented and non-augmented datasets. The comparison of the performance metrics values before performing augmentation is tabulated in Table 3.

From Table 3, it is understood that SVM attains the lowest performance metrics such as the least accuracy sensitivity specificity values. ResNet performs slightly better than SVM but comparatively lower than the custom net. But the combination of the three-stream network perform well compared to separate architecture with an accuracy of 96.3% and a sensitivity value of 94.0% and specificity value of 94.5%. Similarly, Table 4 shows the performance of the individual streams and the combined model after performing image augmentation on the input image. It is seen that the performance achieved is similar to Table 3 but the accuracy, sensitivity, and specificity values are slightly increased. It is because of the huge dataset of images used for training by data augmentation. This lifts the performance values in comparison to those results obtained without performing augmentation.

The performance metrics are also tabulated for different transfer learning architectures such as Alexnet, ResNet, and VGG in Table 5 and Table 6. Table 5 shows the performance without augmented data, and Table 6 shows the performance measures with augmented data. The proposed framework performs better in terms of all metric values compared to other transfer learning networks. The Residual network performance is high in comparison to VGG and Alexnet because of the presence of short connections. Here, even if some of the features are lost in the presentation layer, the previous layer features are added with the next layer to eliminate the vanishing gradients. These connections aids in learning all the essential and important features from every stage. Figure 7 represents the progress of the training process. The validation loss is decreasing for each iteration which reveals that the error minimizes. Also from the first half of the Figure 7, it can be seen that the accuracy increases when the loss keeps on reducing.

Figure 7

Training Progress – Training Accuracy and Loss.

Figure 8 shows the ROC curve for the proposed full/tri stream deep network model. The obtained AUC for this model is 97.93% which is higher than the other one stream and two-stream network.

Figure 8

ROC Comparison of different streams.

Figure 9 shows the ROC curve for the proposed network model together with ResNet and VGG models. The AUC for the proposed is higher than ResNet and VGG deep network. The performance metrics are tabulated for different combination streams without and with augmentation in Tables 7 and 8. The accuracy of the proposed framework with augmentation attains a peak value of 98.2%. From the experimental results, it is concluded that the addition of augmented training data improves the performance of the classification system. Also, the combination of the proposed three-stream architecture leads to improvement in the classification accuracy in comparison to all other networks and existing systems.

Figure 9

ROC Comparison of three different networks

The confusion matrix in Figure 10 is used to visualize the performance of the system in most of the machine learning classification problems. This confusion matrix depends on the true positive, true negative, false positive, and false-negative vales. Since the two-class problem is formulated here, the confusion matrix will have only two class prediction values. It predicts whether it is healthy or cancerous. It is understood from Figure 10 that among the 1244 normal test samples, 1201 normal samples are identified correctly as normal and the remaining 43 normal samples are identified to be cancerous. Also, from among the 1200 cancerous samples, only one sample is identified wrongly. Therefore, the total accuracy of the system is 98.2%. This system performs well for cancerous samples in comparison to that of healthy samples.

Figure 10

Confusion Matrix of the proposed three stream model.

The performance metrics of the existing systems and the proposed model are tabulated in Table 9. Antonio et al. [19] used the taxonomic diversity index and distinction index with the phylogenetic trees to classify lung cancer using SVM. It has the highest specificity and an accuracy of 97.6%. Wenqing et al. [22] could achieve 81.19% using Deep Belief Networks. RuoXi Qin [30] proposed a multidimensional attention system that reduces the feature noise while extracting fine-grained features from the PET and CT scans for classification. The Convolutional Neural Network applied for the CT images provides promising results with 92% accuracy. But it is lower than the proposed system because of the use of single structured CNN. This creates a chance of the addition of unwanted features and the elimination of essential features which leads to decreasing the accuracy. Shudong Wang [31] used a system that uses a transfer learning-based residual network and obtained an accuracy of 95.75%. It is 3% lower than the proposed system because the short connections in the residual network sometime over-learns the important features, and it leads to overfitting. Therefore, in conclusion, the proposed three-stream deep network model performs better in terms of accuracy, specificity, sensitivity in comparison to the existing system.