Multi-scale Unrolled Deep Learning Framework for Accelerated Magnetic Resonance Imaging

I. Introduction

Magnetic Resonance Imaging (MRI) is one of the most preferred imaging modality in radiology because of its non-ionizing characteristics and superior tissue contrast. However, due to inherent underlying physics and sequential data acquisition fashion in the Fourier space, so called k-space, acquiring MR scans are prohibitively slow [1] , [2]. For this reason, accelerating data acquisition in MR imaging has been of perennial interest. Several image and signal processing techniques such as parallel imaging [3], low-rank and compressed sensing [1], [2], dictionary learning [4] and most recently non-linear low-rank and sparse frameworks [5]-[10] have remarkably improved data acquisition speed, enabling MR imaging in several challenging imaging protocols.

While all these methods are significantly different in terms of exploiting several signal processing tools and understanding different signal properties in MR images, most of these methods rely on exploiting signal and system prior information, making them model-based MR accelerating and reconstruction techniques. With recent advent of fast computing capabilities and access to massive data, a new class of data-centric framework, so-called deep learning (DL) techniques have drawn massive interest in MR acceleration community. Convolutional neural networks (CNN) [11], [12] based DL frameworks [13] have shown promising results in MR acceleration and image reconstruction. CNN based DL frameworks [13] pose MR acceleration and reconstruction problem as a task of finding an appropriate mapping function f (·) parameterized by some neural network parameters θ that maps undersampled corrupted MR images ${\widehat{\mathrm{x}}}_i$ to clean fully sampled images x_i such that $f({\widehat{\mathrm{x}}}_i;\theta )\to {\mathrm{x}}_i$ given a set of undersampled and fully sampled training image data pairs $\{{\widehat{\mathrm{x}}}_i,{\mathrm{x}}_i\}$. While these methods focus on learning a parameterized mapping function f (·) based on given training data pairs for some sort of loss function, typically ℓ₁, ℓ₂ minimization and have shown promising results, they essentially ignore data consistency and system priors [14]-[16]. Another class of DL frameworks [14]-[20] combine data consistency and system priors with image features learned from data-centric DL architectures resulting in a more robust and efficient hybrid data-centric and model-based DL framework for MR acceleration and reconstruction. It is important to note here that at the current stage DL based framework have substantially similar or slightly better reconstruction quality compared to traditional model-based acceleration frameworks. However, DL frameworks are still preferred for two particular reasons: i) They have very short reconstruction (inference) time compared to traditional model-based iterative techniques once the framework is trained on training data, and ii) DL training advocates, ‘One-model-fit-all’ principle, i.e once the DL framework is trained, it eliminates the necessity to tune parameters for each reconstruction unlike in model-based iterative techniques. Both these features typical to DL models are essential for realizing new reconstruction techniques into large scale applications and clinical practices.

In this paper, motivated by the fact that different regions of k-space of MR images capture image features at different scales; for example, lower frequency components of k-space capture image features at coarse scale such as structure, anatomy and smoothness, and higher frequency components capture more detail image features at finer scale, we propose to use multi-scale CNN [11], [21] that captures image features at different scales. Moreover, we combine multi-scale CNN with unrolled DL architecture [14], [15] that incorporates data consistency and parallel imaging to build a robust Multi-scale Unrolled DL architecture for MR image acceleration. We also investigate the consequence of unrolled architecture and multi-scale CNN on reconstructed image quality and compare to architectures without such provisions. The rest of the paper is organized as follows. Section II describes the basic principle behind accelerated MRI with undersampled data, Section III presents the proposed method, and section IV validates the advocated approach using extensive numerical tests on data. Finally, section V concludes the paper.

II. Background

In accelerated MRI, the data acquisition process is characterized by systems of equations such that y = ϕ(x) where ϕ is a forward operator that characterize partial Fourier operation and coil sensitivities [3], $\mathrm{y}\in {\mathcal{C}}^{\mathrm{M}}$ is acquired partial(undersampled) k-space data, and $\mathrm{x}\in {\mathcal{C}}^{\mathrm{N}}$ is the desired image. Clearly, because ϕ is a partial Fourier operator, M << N, hence this system of equations is ill-posed. To solve for x from such ill-posed system, MR image reconstruction problem is formulated as a constrained optimization problem,

Essentially, in traditional model-based methods, the data regualizer is enforced via known signal/image priors such as smoothness, low-rank [1], sparsity [2]. Whereas, in data-centric DL models [13], [15], data regularizers/priors are learned and enforced from given training data. Typically, CNNs are known to be very efficient in learning such data priors in case of images, and have shown very promising results [13], [15], [16].

III. Proposed Framework

CNNs have shown to have an immense ability to capture underlying image features. Different kernel sizes in CNNs capture image features at different scales [11], [21]. Such multi-scale CNNs have shown potential on image recognition, classification and image de-hazing [22]. In MRI, different regions of k-space captures image features at different scales. For example, the lower frequency components of k-space capture coarse features such as anatomical structures, contrast and intensity whereas high frequency components capture finer details. Motivated by this fact we develop a multi-scale CNN for MR image reconstruction and integrate it to unrolled DL architecture to enforce data consistency and coil sensitivity maps, resulting in a robust DL framework for accelerated MRI. Such model captures the best of both learning paradigms: model-based and data-centric deep learning paradigm. Essentially, our framework consists of two important constituents i) Multi-scale CNN block and ii) Unroll Network block as shown in the Fig 1. Two deep learning network (DLN) blocks capture image features at different scale. Each DLN blocks consists of two ResNet [12] blocks. Each ResNet block is a 2 layer convolution neural network with a relu activation as shown in the Fig. 2(a). Different convolution filters with different kernel sizes are used in each DLN block to learn image features at Coarse and Fine scale. These different features are combined through a single layer CNN layer as shown in the Fig. 1. At every DL iteration an Unroll Network block enforces coil-sensitivities and data consistency to ensure robust reconstruction.

IV. Experiments and Results

Our framework is implemented on TensorFlow 1.10.1. Adam optimizer with β₁ = 0.99, β₂ = 0.99, learning rate α = 0.001 is used and the loss function $\mathcal{L}(\cdot )$ is defined to minimize the ℓ₁ norm between the output and the ground truth image during training phase. Coarse scale DLN consists of F = 64 convolution kernel filters of size k₁ = 8 × 8 at each convolution layer whereas, fine scale DLN consists of F = 64 convolution kernel filters of size k₂ = 2 × 2 at each convolution layer. For unroll network, Q = 2 numbers of proximal iterations are carried out. In this paper we present our results on two different data sets from knee and body imaging. Both data sets were 32 coil, 3D volume data. 3D volumetric k-space data were first converted to 2D k-space by taking inverse Fourier transform along z-direction and coil compressed [3] to give 8 coil x − y 2D k-space. As a result, 6400, 8-coil knee MR images and 5200, 8-coil body MR images were obtained. Data were collected at Stanford University and Stanford Hospital with full patient consent and approved by the hospital ethics committee. Both data sets were divided into 60%, 5%, and 35% train, validation and test sets, respectively. For training purpose, a total of 35, 2D variable density undersampling masks with different undersampling rates of 2, 3, 4, 6, and 9 were used. We compare our multi-scale unrolled CNN results with two other frameworks, a naive CNN network without unrolling/data-consistency, and a single-scale unrolled CNN. For both competing networks convolution kernels of 4 × 4 and F = 128 at each layer, i.e double the numbers of filters in multi-scale framework were used, keeping all other common parameters the same.

Fig.3 and Fig.4 show the representative axial knee image reconstructions at undersampling factor of 6 and 9, respectively. It can be clearly seen that the proposed multi-scale framework gives cleaner and sharper reconstructed images typically at higher undersampling rate of 9. We believe, this is particularly because our coarse scale DL network is able to capture smoothness of MR images more efficiently. However, some finer details are lost on both single-scale and multi-scale framework. It is observed that, a naive-CNN framework without data consistency results in noisy reconstruction compared to other methods. However, it is worth to mention here that few reports have been made that deeper naive-CNN frameworks are able to give comparable results [13], [23]. However, for our case where our network has only 2 ResNet blocks the naive-CNN performs substantially poor. Figs 5, 6 show quantitative performance metrics root normalized mean square error (RNMSE), and peak signal to noise ratio (PSNR), respectively on the test data set at different reduction factors(SSIM metric isn’t included here due to space limit). It is clear that, the proposed framework outperforms all other methods in both quantitative metrics. We also reconstructed coronal view knee image slices from the network trained on axial slices to test the generality and transferable potential of the model for multi-view reconstruction. In Fig. 7, we see both single-scale and multi-scale reconstruction are able to infer meaningful coronal view knee image reconstruction from a model trained on axial view, but clearly, multi-scale reconstructions look cleaner and sharper advocating the fact that our proposed framework is able to capture more low-level coarse image features such as smoothness and intensity and enables it into translating multi-view reconstruction from slice training data sets.

Finally, Fig. 8 shows representative body MR image reconstructions from single-scale and multi-scale framework at reduction factor of 4. It is worth to note here that, both reconstructions seem to give comparable results in this case. This is possibly because, the MR images in this case are high-resolution images with many finer details and consisted images of different anatomical regions as well as several noisy images in the training and test case. This possibly affects the behaviour of network, so the ability of multi-scale framework to capture low-level image features such as smoothness could not significantly contribute in the reconstruction. It is desired to further investigate such consequences in future works. Nonetheless, our proposed framework showed substantially improved results in knee imaging both qualitatively and quantitatively as shown in Fig. 3 through Fig. 7.

V. Conclusion

In this paper, we proposed a robust multi-scale unrolled deep learning framework for accelerating magnetic resonance imaging. our proposed method combines the best of two different learning paradigms: model-based learning and data-centric learning by using multi-scale CNN and proximal operator based unrolled deep learning framework. our proposed method have shown to be better in both qualitative and quantitative metrics compared to naive-CNN framework and single-scale unrolled CNN. Superiority of the proposed method is verified through several numerical experiments on different data sets. A detailed further study and experimentation on behaviour of such multi-scale framework is sought for in future endeavors.