Disease-image-specific Learning for Diagnosis-oriented Neuroimage Synthesis with Incomplete Multi-Modality Data

2.1. Synthesis of Missing Neuroimages

By providing complementary information, multi-modality neuroimaging data have shown to be effective in achieving holistic understanding of the brain and improving automated identification performance of brain disorders [22]. Since subjects may lack a specific imaging modality the missing data problem has been remaining a common challenge in multi-modality-based diagnosis systems.

Rather than estimating the hand-crafted features of missing images, many machine learning techniques have been developed to impute the missing images directly. For problems with multi-modality imaging data, a popular solution is to perform cross-modality estimation, i.e., synthesizing an image of a specific modality based on the corresponding image of another modality. For instance, Huynh et al. [23] proposed to estimate patches of each computerized tomography (CT) image from its corresponding MR image patches using the structured random forest and auto-context model. Jog et al. [24] attempted to transform T₁-weighted MR patches to T₂-weighted MR patches by training a bagged ensemble of regression trees. Bano et al. [25] designed a cross-modality convolutional neural network (CNN) with a multi-branch architecture operated on various spatial resolution levels for inference between T₁-weighted and T₂-weighted MRI scans. Li et al. [22] proposed a 3-layer CNN for estimating PET images from MRI scans for data completion.

Generative adversarial networks (GANs) have been developed for image transformation from a source domain/modality to a target domain/modality [26], [27], [28], [29], [30], [31], [32], [33]. A typical GAN [26] consists of two neural networks: (a) a generator trained to synthesize an output that approximates the real data distribution, and (2) a discriminator trained to differentiate between the synthetic and real images. Recently, variants of GAN have been used in synthesizing medical images [4], [34], [35], [36], [37]. Ben et al. [35] combined a fully CNN with a conditional GAN to predict PET from CT images. Yi et al. [36] used GAN to generate missing magnetic resonance angiography images from T₁- and T₂-weighted MR images. Sun et al. [11] studied the latent variable representations of different modalities and proposed a flow-based generative model for MRI-to-PET image generation. Pan et al. [4] employed CycGAN to predict PET images from MRI scans and achieved good results in brain disease diagnosis using both real and synthetic multi-modality images. Yan et al. [37] added a structure-consistency loss to the original CycGAN and applied it to estimate MRI data from CT data. In general, these GAN-based image synthesis techniques only pose constraints on data distribution, without considering the discriminative capability of synthetic images in a particular task (e.g., neuroimaging-based brain disease diagnosis). Hence, it is desirable to synthesize missing images via GAN in a task-oriented manner.

2.2. Identification of Disease-relevant Brain Regions

Multi-modality neuroimages have been widely used in automated diagnosis of brain disorders, such as AD and MCI. Previous studies have verified that there exits disease-image specificity. For example, AD and MCI are relevant with brain atrophy, especially in specific regions such as hippocampus and amygdala [17], [38], [39]. Zhang et al. [7] combined the volumetric features extracted from 93 regions of interest (ROIs) in both MRI and PET scans and reported that the most discriminative MRI- and PET-based features are from different brain regions. Zhang et al. [8] further studied the volumetric features of PET and MRI and found that the most discriminative brain regions differ in different tasks. Wachinger et al. [19] studied shape asymmetries of neuroanatomical structures across brain regions and found that the subcortical structures in AD is not symmetric, e.g., shape asymmetry in hippocampus, amygdala, caudate and cortex is predictive of disease onset. Cui et al. [39] focused on hippocampus regions and used a 3D densely connected CNN to combine global shape and local visual features of hippocampus to enhance the performance of AD classification. However, these ROI-based methods ignore relative changes in multiple regions, while relying solely on rigid partition of ROI may ignore small or subtle changes caused by diseases.

To tackle this limitation, patch-based methods have been proposed to capture diseased-relevant pathology in a more flexible manner, without using pre-defined ROIs. Liu et al. [40], [41] partitioned each volume into multiple 3D patches and hierarchically combined patch-based features for AD/MCI identification. Suk et al. [9] developed a deep Boltzmann machine to find latent hierarchical feature representation from paired 3D patches of MRI and PET. Based on hand-crafted morphological features, Zhang et al. [42], [43] first defined multiple disease-relevant anatomical landmarks via group-wise comparison, and then extracted features from image patches around these landmarks for automated disease diagnosis. Similarly, Li et al. [44] detected anatomical landmarks and developed a multi-channel CNN for identifying autism spectrum disorder. Liu et al. [21] and Pan et al. [4] proposed deep learning models with multiple sub-networks to learn image-level representations from multiple local patches located by anatomical landmarks. Note that these methods generally rely on hand-crafted features to select disease-relevant locations (via ROIs or anatomical landmarks), Hence, they have to treat patch selection and classifier training as two standalone steps, thus leading to that those selected patches may not well coordinated subsequent classifiers.

Without pre-defining disease-relevant regions and patches, Lian et al. [17] developed a hierarchical fully convolutional network (FCN) to automatically identify discriminative local patches and regions in whole-brain MR images, upon which task-driven MRI features were then jointly learned and fused to construct hierarchical classification models for disease identification. However, this method cannot explicitly reveal the importance of different brain regions, and also cannot be applied to problems with incomplete multi-modality images.

3.1. Problem Formulation

We aim to construct a computer-aided diagnosis system based on multi-modality data, such as MRI (denoted as $\mathcal{A}$) and PET (denoted as $\mathcal{B}$) data. Denote $M={\left\{\left(A_i,B_i,y_i\right)\right\}}_{i=1}^N$ as a dataset consisting of N subjects, where $A_i\in \mathcal{A}$ and $B_i\in \mathcal{B}$ represent, respectively, the MRI scan, PET scan of the i^th subject. Also, y_i ∈ {0, 1} denotes the class label of the i^th subject, e.g., 1 for AD and 0 for cognitively normal (CN) subjects. An automated diagnosis model with multi-modality data can be formulated as

where ${\widehat{y}}_i$ is the estimated label for the i^th subject.

In practice, however, not all subjects have complete data of both modalities. Accordingly, we assume only the first N_c subjects have complete data (i.e., paired MRI and PET scans) and the remaining N − N_c subjects have only one imaging modality (e.g., MRI). The diagnosis model $\mathbb{P}$ can only be learned based on the first N_c subjects with complete multi-modality data as

where those N − N_c modality-incomplete subjects cannot be used for model learning. Meanwhile, the model defined in Eq. 2 cannot be used to perform prediction for test subjects with only one imaging modality.

To address this issue, one can impute the missing data (i.e., B_i) for the i^th subject, by estimating a virtual ${\widehat{B}}_i$ based on the available modality (i.e., A_i), considering the underlying relevance between two imaging modalities. Denote ${\mathbb{G}}_A:\mathcal{A}\to \mathcal{B}$ as the mapping function from MRI to PET, i.e., ${\widehat{B}}_i={\mathbb{G}}_A\left(A_i\right)$. Then, the diagnosis model can be executed on modality-incomplete subjects as

Based on modality-complete (after imputation) data, $\mathbb{P}$ can be learned by using all subjects as

According to Eqs. 3–4, there are two sequential tasks in the automated diagnosis of incomplete multi-modality data, including (1) learning a reliable mapping function for data imputation (i.e., ${\mathbb{G}}_A:\mathcal{A}\to \mathcal{B}$) to synthesize missing data for modality-incomplete subjects, and (2) learning a classification model (i.e., $\mathbb{P}$) to effectively use multi-modality data for brain disease diagnosis. If these two tasks are performed independently [4], [13], the synthesized data may not be well coordinated with the subsequent diagnosis task. Therefore, we propose the DSDL framework to jointly perform both tasks of image synthesis and disease diagnosis. As illustrated in Fig. 1, this framework include three major components: (1) two single-modality DSNets, i.e., ${\mathbb{P}}_A$ and ${\mathbb{P}}_B$, for disease diagnosis and learning disease-image specificity; (2) a FGAN for missing image synthesis; and (3) a multi-modality DSNet (i.e., ${\mathbb{P}}_{AB}$) for brain disease identification. By jointly training DSNet and FGAN, we can encourage that the disease-image specificity learned by DSNet can be preserved in the image synthesis process, and also those synthetic images are task-oriented for disease diagnosis by focusing on disease-relevant brain regions in each modality.

3.2. Single-modality DSNet

A specific brain disease is often highly relevant with particular regions [17], [18], [19], [21], and disease-relevant regions may differ in MRI and PET scans [7], [8]. To model such disease-image specificity, we propose two single-modality DSNets (see Fig. 2 (a)) for real MRI and real PET scans, respectively. Using both models, we can directly extract features from each input whole-brain image, and identify disease-relevant regions implicitly in each modality. The identified disease-image specificity will be further employed to aid the image synthesis process conducted by FGAN.

3.3. FGAN

Since a pair of MR and PET images scanned from the same subject have underlying relevance but probably different disease-relevant regions, we develop FGAN to synthesize a missing PET image based on its corresponding MRI with a feature-consistency constraint for generating diagnosis-oriented images and an adversarial constraint for generating real-like data. As shown in Fig. 3, our FGAN mainly contains a generative adversarial learning component and two feature-consistency components for MRI and PET modalities, respectively.

3.4. Multi-modality DSNet

Using the proposed FGAN, one can obtain synthetic MRI/PET scans for modality-incomplete subjects, and thus each subject will be represented by complete multi-modality data (i.e., a pair of MRI and PET scans). To handle classification problems with multi-modality data, we extend our single-modality DSNet to a multi-modality version, called mDSNet.

As shown in Fig. 2 (b), our mDSNet ${\mathbb{P}}_{AB}$ consists of sequentially two parts: (1) two parallel feature extraction modules (i.e., ${\mathbb{F}}_A$ and ${\mathbb{F}}_B$) followed by a feature map concatenation operation to fuse futures of MRI and PET and (2) a classifier (i.e., ${ℂ}_{AB}$) with the multi-order feature representation $\tilde{\tilde{U}}$ (see Eq. 7) as its input. The feature extraction modules in mDSNet have the same network architecture as DSNet but different parameters. The proposed mDSNet is trained from sketch using both the real and synthetic images. Since the classifier ${ℂ}_{AB}$ can also input the first-order feature representation $\tilde{U}$ (see Eq. 6), we denote mDSNet with first-order features as mDSNet-1st in this work.

3.5. Implementation Details

In the training stage, the proposed DSDL framework is executed via the following three stages: (1) using only the real MRI (i.e., $\mathbb{A}$) and real PET (i.e., $\mathbb{B}$) data to train two single-modality DSNets (i.e., ${\mathbb{P}}_A$ and ${\mathbb{P}}_B$), respectively, for Disease-image Specificity Identification; (2) using modality-complete subjects with real MRI and PET data to train FGAN for image synthesis; and (3) using the complete (after imputation) paired MRI and PET scans of all training subjects to train mDSNet for disease diagnosis. Note that, to augment the number of samples, The combination (A_i, ${\mathbb{G}}_A(A_i)$) will also be used as a training sample to train mDSNet even B_i exists.

In the test stage, for an unseen test subject with complete multi-modality data, we directly feed its MRI and PET scans to mDSNet for classification. Given an unseen test subject with missing MRI/PET scan, we first impute the missing scan using our trained FGAN model and then feed the complete (after imputation) paired MRI and PET scans into mDSNet for classification.

Our proposed models are implemented by Python with Tensorflow on a platform with GTX 1080 Ti and Intel Core i7-8700. We first train two DSNets ${\mathbb{P}}_A$ and ${\mathbb{P}}_B$ for 40 epochs using complete subjects (i.e., those with paired PET and MR images). We then train ${\mathbb{D}}_A$ and ${\mathbb{D}}_B$ by minimizing $-{\mathfrak{L}}_a(*)$ with fixed ${\mathbb{G}}_A$ and ${\mathbb{G}}_B$, and train ${\mathbb{G}}_A$ and ${\mathbb{G}}_B$ by minimizing $\mathfrak{L}(*)$ with fixed ${\mathbb{D}}_A$ and ${\mathbb{D}}_B$, iteratively, for 100 epochs. After that, we train mDSNet for 40 epochs using both real and synthetic data. We use SGD solver with a learning rate 1 × 10⁻³ to train DSNets and mDSNet, and use the Adam solver [46] with a learning rate of 2 × 10⁻³ to train FGAN. The batch size is set to 1 due to the limitation of GPU memory. More details on hyper-parameter settings could be seen in the Supplementary Materials.

4.1. Materials and Image Pre-processing

We first evaluated the proposed method on two subsets of ADNI [5], including ADNI-1 and ADNI-2. Subjects in these two datasets were divided into four categories: (1) AD, (2) CN, (3) progressive MCI (pMCI) that would progress to AD within 36 months after baseline, and (4) static MCI (sMCI) that would not progress to AD. After removing these subjects appearing in both ADNI-1 and ADNI-2 from ADNI-2, there are 205 AD, 231 CN, 165 pMCI and 147 sMCI subjects in ADNI-1, while there are 162 AD, 209 CN, 89 pMCI and 256 sMCI subjects in ADNI-2. All the above subjects in ADNI-1 and ADNI-2 have baseline MRI data, while only 356 and 581 of these subjects have PET images in ADNI-1 and ADNI-2, respectively. We also used the AIBL [47] dataset for performance evaluation, containing 71 AD and 447 CN subjects. Similar to ADNI-1 and ADNI-2, all subjects in AIBL have MRI scans, and only part of them have PET scans. The demographic and clinical information of three datasets are listed in Table 1. More details of these data are described in Section 2 of the Supplementary Materials.

For data pre-processing, we first performed skull-stripping on MRI scans using FreeSurfer [48], and then linearly aligned each PET scan to its corresponding MRI scan. Next, we affine each MRI scan to the commonly-used MNI template using SPM [49], while its corresponding PET scan (if existed) is also affined using the same affination parameter. In this way, each pair of MRI and PET scans of a same subject have the spatial correspondence.

We performed two groups of experiments on the ADNI-1 and ADNI-2 datasets: synthesizing MR and PET images and diagnosing brain diseases, including AD identification (AD vs. CN classification) and MCI conversion prediction (pMCI vs. sMCI classification). To evaluate the generalization capability of our proposed models for image synthesis and disease classification, we further performed an extra group of experiments on the AIBL dataset.

4.2. Evaluation of Synthetic Neuroimages

4.3. Evaluation of Automated Diseases Diagnosis

5.1. Effect of Feature Maps

In the feature-consistency component (Fig. 3), we add the feature-consistency constraint at each of five Conv layers. To investigate the effect of our feature-consistency constraint at different layers, we perform an experiment by adding such a constraint at only one single Conv layer, and denote the generated FGAN variants as ${\left\{l_i\right\}}_{i=1}^5$. Based on the resulting FGAN models, we can obtain synthetic MRI and PET images. The image quality of these images (regarding SSIM and PSNR) and their performance in AD vs. CN classification (with AUC values marked as *) and pMCI vs. sMCI classification (with AUC values marked as ^†) are shown in Fig. 5. Fig. 5 reveals that adding the feature-consistency constraint on early layers (e.g., l₁) generally results in better synthesis results (in term of SSIM and PSNR) but lower classification results (in term of AUCs). Also, models using such a constraint at the late layers (e.g. l₅) usually generate the reverse results (i.e., better classification performance but low image quality). Therefore, we add the feature-consistency constraint to feature maps of all five Conv layers to balance the synthesis results and classification results.

5.2. Comparison of Different Losses

To evaluate the influence of different losses used in our image synthesis models (i.e., FGAN, FCycGAN and FVoxGAN), we further train the generative model (with only ${\mathbb{G}}_A$ and ${\mathbb{G}}_B$ in Fig. 3) with only one of the following losses: (1) adversarial loss $\left({\mathfrak{L}}_a\right)$, (2) cycle-consistency loss $\left({\mathfrak{L}}_c\right)$, (3) voxel-wise-consistency loss $\left({\mathfrak{L}}_v\right)$, and (4) feature-consistency loss $\left({\mathfrak{L}}_f\right)$. Using each of four different losses, the corresponding generative model can generate synthetic MRI and PET scans. In Fig. 6 (a)–(b), we show the resulting MAE values of these synthetic image along the training epochs. Based on these generated images, we further report the AUC values of our mDSNet in AD vs. CN classification in Fig. 6 (c)–(d), with models trained on ADNI-1 and tested on ADNI-2. Fig. 6 (a)–(b) suggests, when using four different losses, the model that uses only the voxel-wise-consistency loss can produce the most visually realistic image (with respect to MAE values). From Fig. 6 (c)–(d), one can observe that the model using the feature-consistency loss consistently achieves the best classification performance (regarding AUC values), compared with those using the other three losses. Meanwhile, models with the voxel-wise-consistency loss and our feature-consistency loss are more stable compared to those with cycle-consistency loss and adversarial loss. This could be due to the fact that the calculation of the latter two losses relies on updating modules, i.e., another generator and discriminator.

5.3. Enhancing Diagnosis Performance by More Subjects

In the previous experiments, we only use these subjects in ADNI-1 to train our diagnosis model. Since more data may result in better performance, it is possible to use more subjects (e.g., those from ADNI-2) to further improve the performance of the diagnosis performance. Accordingly, we perform another experiment by using both the images in ADNI-1 and ADNI-2 to train the diagnosis model (i.e., mDSNet) but using only complete subjects in ADNI-1 to train the FGAN model. Then we apply the obtained models on the AIBL dataset, with results (denoted as mDSNet-2) reported in Table 5 as well as the original results (denoted by mDSNet). It seems this strategy slightly improves the diagnosis performance (e.g. the AUC score increases from 94.92% to 95.31%). The possible reason could be that more training data make the learned model more general among different sites, thus improving the diagnosis performance.

5.4. More Details of Spatial Cosine Kernel

Following the description of spatial cosine kernel in section 3.2.2, the input of our DSNet is a neuroimage of size 144 × 176 × 144 and the l₂-normalized feature map (spatial representation) is denoted as u = (u₁, u₂, ⋯, u_K) where $u_k=\frac{v_k}{\parallel v_k{\parallel }_2}$. The spatial cosine kernel (defined in Eq. 8) is

where w is the ensemble of hyper-parameters. Due to having the same dimension as u, w can be partitioned into K elements, i.e., w = (w₁, w₂, ⋯, w_K), where w_k has same dimension as u_k. Let ${\beta }_k=\frac{\sqrt{K}\parallel {\mathbf{\text{w}}}_k{\parallel }_2}{\parallel w{\parallel }_2}$, $\left({\text{E}}_{\beta \in \left\{{\beta }_1,{\beta }_2,\cdots ,{\beta }_K\right\}}{\beta }^2=1\right)$, then the spatial cosine kernel can be rewritten as accumulating the similarity between w_k and the k-th element in a feature map as

Herein, $ℂ\left(v_k,w_k\right)\in [-1,1]$ is a cosine kernel that indicates the group difference corresponding to the k-th elements. Meanwhile, β_k, directly proportional to the norm of w_k, indicates the contribution coefficient of the k-th cosine kernel to the classification result. Since β_k is learned automatically and implicitly while training DSNet, it is very convenient to capture the disease-relevant patterns by finding the disease-relevant part of the feature map (U_d), in which each u_k corresponds to a large β_k.

Furthermore, it should be noted that the disease-relevant part (U_d), the residual normal part (U_r), and α are mutually dependent but only one constraint in Eq. 5 should be satisfied. Hence, the decomposition of disease-relevant part and residual normal part is not unique. For example, a possible decomposition of a feature map U could be

where the disease-relevant component of each pattern is placed to the disease-relevant part, and the disease-irrelevant component of each pattern is placed to the residual normal part. In this case, $ℂ(U,w)=ℂ\left(\alpha U_d,w\right)$.

Alternatively, we can use a threshold τ to binarize β_k as follows

and thus decompose U as

It means that we directly place the disease-relevant patterns to disease-relevant parts and place the disease-irrelevant patterns to residual normal part. In this case, $ℂ(U,w)\approx ℂ\left(\alpha U_d,w\right)$. When considering only the prediction labels, we have

Namely, if the disease-relevant parts and residual normal parts can be separated, the coefficient α will not affect the predicted labels.

5.5. Statistical Significance Analysis

The major hypothesis in this work is that a generative model with the feature-consistency constraint can generate the synthetic images which are diagnostically similar to real images. It means that the synthetic images and corresponding real images can deliver similar diagnosis of medical conditions. To verify this, we calculated the inter-class averaged dissimilarity (ICAD) of 80 brain regions in synthetic MRI and PET images and displayed them in Fig. 7. The definition of ICAD is as follows.

Suppose the i-th (i ∈ {1, 2, ⋯, N}) scan has a feature map u_i = (u_1,i, u_2,i, ⋯, u_K,i)(∥u_k,i∥₂ = 1) and a class label y_i ∈ {0, 1}. The similarity of two feature maps u_i and u_j is measured by the spatial cosine kernel,

Then, the ICAD, denoted by S, can be calculated as

A large ICAD value indicates that two classes are easy to be distinguished. Similarly, we can use

to measure the distinguishability regarding the k-th location.

For each modality in Fig. 7, the top six rows (denoted as S₁, S₂, ⋯, S₆ from top to bottom) are the ICAD of the synthetic images generated by GAN, cycGAN, voxGAN, FGAN, FcycGAN, and FvoxGAN, respectively, and the 7^th row (denoted as S₇) is the ICAD of real images. The bottom row for each modality depicts the values of {β_k}, which have been rescaled to [0, 1], indicating the contribution coefficient of each region to the classification task. Where, a larger value means that the region is more relevant to the classification task. Comparing the 7^th row and the bottom row, it suggests that most of the regions with higher inter-class dissimilarities are recognized as more relevant to disease diagnosis. Meanwhile, it reveals that most disease-relevant regions are different across two modalities, which renders the cross-modality image synthesis a challenging task and also makes the feature-consistency constraint a must for any solutions to this task.

We now conduct the statistical significance analysis for our major hypothesis. The null hypotheses (H₀), alternative hypotheses (H₁), and the p-values of H₀ for MRI and for PET are listed in Table 6. It shows that all obtained p-values are smaller than 0.05. Therefore, H₁ is accepted,, which means that a generative model with the feature-consistency constraint can generate images more diagnostically similar to real images.

5.6. “Hallucination” in Generative Unseen Patterns

Currently, it is arguable to first train a generative model on available data for absent data generation and then to use the generated data, together with the available data, to train another model for classification. Especially, the generated data may be affected by the dataset used to train the generative model. For example, if each training image contains a tumor, the data generated by the trained model may also contain a tumor, although it is expected to generate the scan of a normal control. On the other hand, if each training data was acquired from a normal control, it is less impossible for the trained generative model to generate a scan with tumors.

In this study, the abovementioned “hallucination” issue must be addressed when we attempt to use cross-modality image generation to impute missing PET data for the multi-modality based AD diagnosis. The reason lies in the fact that, at an early stage of AD progression, subtle functional changes can be detected by PET long before any structural changes become evident on MRI scans. Thus, when using the MRI scan of a subject with early AD to generate a PET scan, the generative model must be able to produce the disease-related patterns, which do not exist on the input MRI scan. Otherwise, the generated PET scan may not be indeed helpful for disease diagnosis. The major contribution of this work is to address this issue using several tricks, listed below.

• The dataset used to train the generative model contains the cases from all categories (e.g., AD, CN, and MCI). Hence, there will be no “unseen” case in the inference stage.

• We perform image generation in a supervised way. It means that we know the class label of each input MRI scan and accordingly know, statistically, the disease-image-specifics of the PET scans of that class.

• We introduce a feature-consistency constraint to the generative model, encouraging the generated PET scan to preserve the disease-image-specifics for the subsequent diagnosis task. Specifically, the feature-consistency constraint encourages the multi-layer feature maps of a synthetic PET scan (produced by DSNet) and the features of real PET scans of the same class to be consistent. In this way, our FGAN correlates with DSNet, and hence the generated PET scans become consistent with real PET scans from the perspective of classification/diagnosis.

The proposed solution was evaluated in two perspectives: (1) visual similarity and (2) clinical usefulness. The results in Table 2 and Fig. 4 show that the synthetic PET scans generated by our FGAN are similar to real PET scans. Also, the results in Table 3 show that, with our imputed PET scans, the proposed multi-modality based classification model achieves substantially improved performance for AD diagnosis.

Now let us further elaborate the advantage of the proposed solutions over existing generative models like GAN, cycGAN, voxGAN, and condition GAN. The target of this study is to impute missing PET scans and thus to improve the multi-modality diagnosis of AD. However, the distribution matching constraints used in existing generative models may not be able to preserve the discriminative information in the generated images. Although the l₁ (MAE) loss encourages the pixel/voxel-wise consistency, it remains incapable of preserving sufficient discriminative information, as evidenced in our supporting experiments and the experiments in [55]. The potential reason that these generative models are not suitable for this study is that both distribution matching constraints and pixel-wise-consistency are class-independent. Take the pixel-wise-consistency for example, if the input is an MRI scan of an AD subject, the output PET scan is forced to be pixel-wise-consistent with training PET scans, which are from both AD subjects and normal controls. Although the generated PET scan looks like a real PET scan, it may not contain enough disease-image-specifics, i.e., the pathological patterns of AD that can be observed using PET. In contrast, the proposed FGAN generates images in a supervised way, in which the feature-consistency constraint encourages the multi-layer feature maps of a generated PET scan (produced by DSNet) to be consistent with the features of real PET scans of the same class. Hence, the PET scan generated by our FGAN contains rich discriminative information and can help the subsequent diagnosis task.

5.7. Potential Applications in other Scenarios

In practice, applications of image-image translation (besides cross-modality image translation) have been increasingly used in many vision applications, such as security surveillance and autonomous driving. In these applications, sometimes we may encounter a similar issue of requiring a specific modality of data. Although the proposed DSDL framework was developed for neuroimage synthesis and AD diagnosis, the ideas of supervised image generation and feature-consistency constraint are generic and can be extended to these applications of task-specific image-image translation.

Various image-image translation tasks, including “edges to photo”, ”aerial to map”, ”day to night”, and ”BW to color”, have been summarized in [28]. However, using only the distribution-match constraint or pixel-wise-consistency constraint may not handle it well when we want to use the generated images for a classification purpose. For instance, on the task of ”BW to color” for flower classification, the color information, which plays a pivotal role in distinguishing a flower species from others, should be generated during the translation from a grayscale image to its color version. In this scenario, we can apply our proposed feature-consistency constraint to encourage the generated color images to preserve the discriminative information for flower classification (as Section 7 in the Supplemental Materials).

5.8. Limitations and Future Work

The proposed DSDL framework has three major limitations.

First, the spatial cosine kernel used in this model has a fixed stride (i.e., 32 voxels along each axis). Hence, it can only roughly capture disease-specific regions with a fixed size of 32 × 32 × 32 voxels. To capture more precise disease-specific regions, hierarchical structures or multi-scale features will be investigated in our future work.

Second, PET protocols used for building these three databases are very different, particularly those for AIBL. The proposed method has no mechanism to handle this issue. As a result, it is hard to use the PET scans in AIBL, and we have to use synthetic PET scans for AIBL. Hence, data harmonization / adaptation techniques [56], [57] will be studied in our future work to capture unified features regardless of scanning protocols.

Third, the pre-processing pipeline used for this study is purely hand-crafted. It relies heavily on the experience of operators and can hardly be optimized for unseen datasets. In our further work, we plan to embed the pre-processing steps into the target task to avoid the devastating effect caused by inappropriate pre-processing.