Next-Gen brain tumor classification: pioneering with deep learning and fine-tuned conditional generative adversarial networks

Generator model

The GAN is a deep learning model comprising two neural networks: the generator and the discriminator. The generator takes a random input vector and produces a real training data sample. In contrast, the discriminator obtains genuine and fake data samples from the generator and attempts to differentiate between them. In a conditional GAN, an additional input variable, such as class labels or attributes, is applied as a conditional vector to generate samples that belong to a specific class or possess specific attributes.

For incorporating the conditioning input, it is concatenated with the random noise vector at the input layer of the generator model. The concatenated input is then passed through multiple layers of neural networks to transform it into an output that mimics the target data distribution. During training, both models are trained adversarial, where the discriminator model tries to distinguish between the realistic and fake data samples, and the generator tries to generate samples that cheat the discriminator into relying on they are genuine. The two models are iteratively trained until the generator creates vague samples from the real data samples according to the discriminator, as shown in Fig. 1.

The generator model is a crucial component of GANs, and its primary objective is to generate synthetic data that closely resembles the training data. The generator model’s behavior depends on the type of model used and the data it is trained on. In the case of GANs, the generator model generates synthetic data, such as images or videos, similar to the real data used for training. The generator model is typically trained alongside a discriminator model, differentiating between real and synthetic data. Therefore, the generator model aims to create synthetic data that appears genuine and can deceive the discriminator model.

The generator model learns patterns and structures in the training data that it can utilize to generate new data. It operates similarly to a typical CNN. In this regard, the design and working of the CNN model are unique. The generator model employs seventeen layers, and the first layer’s input can be identified and its image size determined by adjusting the channel size, height, and weight.

In summary, a Generator Model’s overall behavior is to create new data that closely resembles the training data it was trained on. It accomplishes this by learning the patterns and structures in the training data that it can use to generate new data.

To improve the speed of the network, the framework incorporates an additional layer of batch normalization. Furthermore, the ReLU activation function is utilized. To determine each batch’s highest value, and the max-pooling layer splits the input into multiple pooling sections. The network was trained using a learning rate (Lr) of 0.01 and an epoch value of 30. Various constraints and kernel (K) dimensions ranging from 2 to 8 are also utilized. After the pooling layer, the fully connected layer is introduced, which combines all the outputs from the preceding layers. It connects the learned characteristics of the layers to identify more extensive configurations. The softmax function serves as a normalizing activation function, and the categorization layer uses probabilities to classify the output into predefined categories.

In other words, an extra layer of batch normalization is introduced to improve the network’s speed, and the ReLU activation function is implemented. The max-pooling layer is used to compute the maximum value for each region, and a range of constraints and kernel dimensions are applied. The fully connected layer connects the learned features of the layers to recognize more complex patterns. Finally, the softmax activation function is used to normalize the output, and the classification layer is responsible for categorizing the output into designated categories based on probabilities.

(1) $$FR=0,forx<0and1,forx\ge 0$$

(2) $$F^c=I_c^f(f{i}^{c-1})=(f{i}^{c-1}{)}^t{W}_i^c+T^c$$

(3) $$Con=\frac{(R_d+2P_a-F_d)}{S_{tr}}+1$$

F (R) indicates the ReLU function, and x is the target value, as shown in Eq. (1).

Equation (2), display the fully connected layer output, f filter number, I as input, W indicates the weight, and T is the constant value.

The Con shows the convolutional layer, is the input dimension, indicates the padding, shows the filter’s dimensions, and is the stride in Eq. (3).

Discriminator model

The discriminator model makes up the second half of the CGAN model. In CGAN, the discriminator model is in charge of separating genuine data from fake data and assessing if the fake data produced by the generator model satisfies the requirements set by the input to the network. Real and artificial data and their related condition vectors are all inputs to the discriminator model in a CGAN. Labels, attributes, and any other pertinent information that can be utilized to produce particular sorts of data are included in the condition vectors as extra information about the data.

The discriminator model then processes the input data and condition vectors to yield a single scalar value representing the likelihood that the input data is real or fake, given the associated condition vector. It is trained to output a high probability of accurate data with matching conditions and a low probability of false data with mismatched or wrong conditions as shown in Table 1.

The discriminator model is trained to achieve the highest level of accuracy in identifying authentic and fraudulent data under matching conditions. The discriminator model can be tricked into thinking that bogus data is real with matching criteria by training the generator model to produce false data. It establishes a feedback loop in which the generator model improves at producing realistic data that match the criteria. In contrast, the discriminator model improves at differentiating between actual and fake data.

In a CGAN, the discriminator model is crucial since it advises the generator model on producing more realistic data while guaranteeing that the data matches the requirements. The discriminator’s function is employed to validate the artificially manufactured output of the generator. The discriminator model predicts a true or false binary class label with a domain sample as input. For developing a discriminator, a standard categorization model is used. The discriminator model is dropped since the generator, after training, reveals the exciting space. Since the generator has mastered separating characteristics from instances in the problematic domain, it can occasionally be reused.

It works like a conventional deconvolutional neural network. There are ten layers used in this model with different parameters. The input layer, the top layer, applied some limitations to the channel’s width, height, and weight. The kernel (K) values vary from 2 to 8. Also, with an epoch value of 30, the training procedure’s learning rate (Lr) is 0.001.

Certainly, let’s break down the steps outlined in the pseudocode for the CGAN model:

Input: Variable vectors as R(i) real image: These are real images from the dataset, denoted as R(i), where ‘i’ represents the index of the image.

N(i) noise: These are noise vectors, denoted as N(i), which serve as random input to the generator network for generating fake images.

Output: F(N(i)) indicates fake image: These are the generated fake images produced by the generator network when given the noise vectors N(i).

1. For iteration_num do: This indicates that the following steps will be repeated a certain number of times, specified by the ‘iteration_num’.

2. For step_num(S) do: Within each iteration, there is another loop, which repeats for a certain number of steps, specified by ‘step_num’ denoted as ‘S’.

3. Fake images generation F(N(1)),…, F(N(m)): In this step, the generator network (G) takes a set of noise vectors (N(1), N(2),…, N(m)) as input and produces a corresponding set of fake images (F(N(1)), F(N(2)),…, F(N(m))). Each ’N(i)’ is used to generate a corresponding ‘F(N(i))’. This step generates fake images from noise.

4. Result as real images R(1),…, R(m) : In parallel to generating fake images, this step represents real images from the dataset (R(1), R(2),…, R(m)), where each ‘R(i)’ corresponds to a real image. These real images are used for comparison during the training of the discriminator network.

5. Discriminator (D) model training: In this step, the discriminator network (D) is trained. The training process involves optimizing the discriminator’s parameters denoted as $\theta dm$ to minimize the following loss function:

$\mathrm{\Delta }\theta dm[\log D(Ri)+\log (1-D(F(Ni)))]$

Here, ‘log D(R_i)’ represents the discriminator’s prediction for real images, and ‘log(1-D(F(N_i)))’ represents the prediction for fake images. The discriminator aims to correctly distinguish real images from fake ones.

6. end for: This marks the end of the loop for training the discriminator.

7. Fake images generation F(N(1)),…, F(N(m)): After training the discriminator, the generator network (G) is used again to generate another set of fake images from the same noise vectors.

8. Generator (G) model training: In this step, the generator network (G) is trained. The training process involves optimizing the generator’s parameters denoted as $\theta \mathrm{\_}gm$ to minimize the following loss function: $\mathrm{\Delta }\theta gm\log (1-D(G(Ni)))$

The generator aims to produce fake images that are convincing enough to fool the discriminator. The loss function encourages the generator to generate images that the discriminator is likely to classify as real.

9. end for: This marks the end of the loop for training the generator.

These steps represent the core training process of a CGAN. The generator and discriminator networks are trained iteratively, with the discriminator trying to distinguish real from fake images, and the generator trying to produce fake images that are realistic enough to fool the discriminator. This adversarial training process continues for the specified number of iterations, improving the quality of the generated images over time.

Why choose sequential model

The choice of a sequential model in the CGAN study is motivated by the nature of the input data and the requirements of the conditional generative adversarial network architecture. In the context of image generation and processing, particularly in medical image analysis like brain tumor detection, sequential models offer several advantages:

Handling sequential data: Medical images, including MRI scans, are inherently sequential in nature, as they consist of a series of two-dimensional slices or frames. A sequential model, such as a convolutional recurrent neural network (CRNN) or a similar architecture, is well-suited to handle such sequential data. It can capture spatial dependencies within individual images (through convolutional layers) while also considering the temporal dependencies between images in a sequence (using recurrent layers). This is crucial for understanding the context and structure of the data.

Contextual information: Brain tumor detection in MRI scans often requires understanding the context of each image within a sequence. For example, the appearance of a tumor may change across different slices, and recognizing these changes is essential for accurate diagnosis. A Sequential Model helps in capturing this contextual information, enabling the generator to generate realistic images that maintain consistency across the sequence.

Conditional generation: In the context of a CGAN, the generator produces images conditioned on some input data, typically noise or other images. When generating images, it’s essential to consider both the content of the input data and the sequential context. For example, in medical image synthesis, the Sequential Model can be used to ensure that generated images align with the anatomical structures seen in previous slices.

Model flexibility: Sequential models provide flexibility in designing the architecture to suit the specific needs of the task. They allow for the incorporation of both convolutional layers for spatial feature extraction and recurrent layers for temporal modeling. This flexibility is advantageous for tasks like brain tumor detection, where both spatial and temporal information is crucial.

In summary, the choice of a sequential model in the CGAN study is driven by the need to effectively capture spatial and temporal dependencies in medical image data, ensure conditional generation that respects the sequential context, and ultimately improve the quality and realism of generated images. This choice aligns with the specific requirements of the brain tumor detection task in MRI scans.

Model working

The pre-trained CNN model discriminates in CGAN to distinguish between the true MRIs generated by the generative model and the real ones, allowing the discriminator to extract and learn MRI features. Subsequently, the pre-trained CNN is used to final classify brain tumors. The final fully connected net in the classification module is replaced with a softmax layer to improve classification accuracy. To clarify our methodology, we fine-tuned a pre-trained conditional generative adversarial network (CGAN) to generate realistic brain tumor images. Then, we extracted features from the discriminator network using the pre-trained weights and fed them into a new model for classification. To do this, we created a new sequential model and added the layers of the discriminator up to the last layer, excluding the final classification layer. We froze the layers of the discriminator to ensure that the pre-trained weights were not modified during the training of the new model. Then, we added a new dense layer with two output nodes and a softmax activation function for binary classification. The summary of the new model is as follows:

• 1. new_model = Sequential().

• 2. for i in range(len(disc.layers)−1):

• 3. new_model.add(disc.layers[i]).

• 4. Freeze the layers.

• 5. for layer in new_model.layers:

• 6. layer.trainable = False.

• 7. new_model.add(Dense(classes,activation=‘softmax’)).

In contrast, a few others are generated using a generator model that takes a randomized vector as input from a specific latent space. This model produces some sample images as output. The discriminator takes the real image as input, deconvolutes it and then provides a binary classification for real and fake images during training and testing.

Dataset details and preprocessing

The brain MRI images tumor dataset used in this study is publicly available on Kaggle, Dataset 1: (https://www.kaggle.com/datasets/navoneel/brain-mri-images-for-brain-tumor-detection). Dataset 2: (https://www.kaggle.com/datasets/navoneel/brain-mri-images-for-brain-tumor-detection) Both datasets consist of two classes: “no,” which indicates the absence of a brain tumor, and “yes,” which indicates the presence of a brain tumor. Dataset 1 consists of a total of 239 images, with 84 for “yes” and 155 for “no” classes; on the other hand, dataset 2 consists of a total of 3,000 images, with 1,500 for “yes” and 1,500 for “no”. The dataset contains images of various dimensions, and sample images are shown in Fig. 2. To standardize both datasets, duplications are removed, and all images are resized to 128 × 128 resolution. The datasets are then thoroughly cleaned by removing duplicate values, missing labels, and extensions. Additionally, a histogram equalizer removes any noise from the images. The datasets are divided into 80% training and 20% testing sets for experimentation. Further details about the dataset are provided in Table 2.

Evaluation criterion

The proposed brain tumor classification and detection CGAN network employs several arithmetic calculations outlined in Eqs. (4)–(7). The term accuracy is evaluated by counting the correctly classified positive images (Tp) and negative images (Tn), while positively incorrect classified images are labeled as false positives (Fp). The negatively classified images total images are represented by false negatives (Fn). The statistical (Eqs. (4)–(7)) are necessary to determine these values:

(4) $$Precision=\frac{T_p}{T_p+F_p}$$

(5) $$Recall=\frac{T_p}{T_p+F_n}$$

(6) $$F1-score=\frac{2T_p}{2T_p+F_p+F_n}$$

(7) $$Accuracy=\frac{T_p+T_n}{T_p+T_n+F_p+F_n}$$

Model results

The model generates fake images during training, as shown in Fig. 3. The process works in two stages: CGAN generates the fake images, and then the CNN model takes them as input to classify them as fake or original images based on the exact labels. The confusion matrix for the fake image generation on the test dataset is shown in Fig. 4, where the dataset consists of two classes: Yes and No. The Yes class indicates the occurrence of a brain tumor, while the No class indicates no tumor occurrence detected by the model. Additionally, Fig. 5 illustrates the training and validation accuracy and loss of the applied CGAN model. The blue dotted line represents the training accuracy and loss, while the red one shows the validation accuracy and loss.

Figure 6 compares the generator and discriminator performance on Datasets 1 and 2, measured in terms of loss concerning batch number. Dataset 1 consists of 201 images, and dataset 2 takes 2,400 images of brain tumors and was used to train the GAN model. The training was conducted using a batch size of 8 and 500 epochs with 100 noise dimensions. In total, 12,000 and 140,000 batches were produced throughout 500 epochs in dataset 1 and 2.

The graph shows the maximum loss captured from batches 0 to 6, indicating that both the generator and discriminator improve over time as the number of training epochs increases. The loss of the generator decreases steadily over time, while the loss of the discriminator fluctuates but remains relatively stable overall. These results suggest that the GAN model is effectively learning to generate more realistic images of brain tumors over time. Overall, the graph provides insight into the training process and the performance of the GAN model on both Dataset 1 and Dataset 2.

In Fig. 7, the density graph compares the distribution of real and generated images for both Dataset 1 and Dataset 2. The x-axis represents the pixel intensity values, while the y-axis represents the density of the images. The blue line represents the density of the real images, while the orange line represents the density of the generated images. In both datasets, the density of the generated images is very close to that of the real images, indicating that the generator can produce visually similar images in terms of their pixel intensity distribution. This indicates the effectiveness of the proposed CGAN approach in generating realistic brain tumor images.

Qualitative and error analysis

To provide a more comprehensive understanding of our CGAN model’s performance on Dataset 1 and Dataset 2, we have conducted a qualitative and error analysis in Tables 3 and 4. This analysis aims to shed light on the model’s behavior beyond quantitative metrics. Below, we present key findings from our qualitative and error analysis:

Dataset 1: Accuracy 93%, Class ‘yes’ (Brain Tumor Present): Precision: 0.90, Sensitivity: 1.00, Specificity: 1.00, and F1-score: 0.95. Class ‘no’ (No Brain Tumor): Precision: 1.00, Sensitivity: 0.83, Specificity: 1.00, and F1-score: 0.91.

In our qualitative analysis for Dataset 1, we examined instances where our CGAN model correctly classified brain tumor presence (‘yes’) and where it made errors (‘no’). Visual examples from the dataset were reviewed to gain insights into these cases. We observed that while the model excelled in identifying brain tumor cases with high precision and sensitivity, it encountered challenges in correctly classifying cases without tumors. Further investigation into these errors revealed that some of the false negatives could be attributed to subtle or ambiguous MRI images.

Dataset 2: Accuracy 97%, Class ‘yes’ (Brain Tumor Present): Precision: 0.95, Sensitivity: 0.99, Specificity: 0.96, F1-score: 0.97. Class ‘no’ (No Brain Tumor): Precision: 0.93, Sensitivity: 0.97, Specificity: 0.94, and F1-score: 0.97.

In the qualitative analysis for Dataset 2, we conducted a similar examination of model performance. Our model demonstrated strong precision, sensitivity, and accuracy in identifying brain tumor cases. However, we also noticed instances where the model incorrectly classified images without tumors. This prompted us to investigate the specific characteristics of these false positives.

Our qualitative and error analysis provides valuable insights into the model’s strengths and limitations. We will include additional details, visual examples, and potential strategies for addressing these errors in the revised manuscript, enriching the discussion of our CGAN model’s performance.

Table 5, provides a comparison of accuracy results for various GAN models used in the context of brain tumor classification. These models are evaluated for their effectiveness in accurately classifying brain tumors from medical images. Additionally, the table includes results for the proposed Conditional CGAN model applied to two different datasets, referred to as Dataset 1 and Dataset 2. The table consists of two columns: “Methodology” and “Accuracy.” The “Methodology” column lists the different GAN-based models used for brain tumor classification, along with their respective references, providing readers with the sources for further study. The “Accuracy” column presents the accuracy scores achieved by each methodology, quantifying their performance in brain tumor segmentation.

Optimal DeepMRSeg GAN model proposed by Neelima et al. (2022) achieved an accuracy of 0.91. Deep GAN model proposed by Ghassemi, Shoeibi & Rouhani (2020) achieved an accuracy of 0.93. Ahmad et al. (2022) proposed VAEs-GAN model that was achieved an accuracy of 0.96. (Asiri et al., 2023) introduced MLD-GAN model that also achieved an accuracy of 0.96. Additionally, the Proposed CGAN model is presented in two rows, indicating its performance on two different datasets, labeled as “For Dataset 1” and “For Dataset 2.” The accuracy achieved by the proposed CGAN model is recorded as 0.93 for Dataset 1 and an improved accuracy of 0.97 for Dataset 2.

This table serves as a reference for comparing the accuracy results of different GAN-based models in the context of brain tumor classification, with particular attention to the performance of the proposed CGAN model on two distinct datasets. It provides valuable insights into the effectiveness of various methodologies in this critical medical imaging application.