Machine-learning approach for quantified resolvability enhancement of low-dose STEM data

The deep-learning method chosen for the denoising process is an artificial neural network known as autoencoder, traditionally used for dimensionality reduction and feature extraction [15]. This consists of two parts: an encoder that maps the input (in this case the noisy images) into a reduced space called the latent space, and a decoder that maps the latent space into the output (in this case the infinite-dose images). The architecture proposed here consists of ten layers in total, five for the encoder and five for the decoder, and the input is made of electron-dose 2D maps resolved over a 128 × 128-pixel grid. The activation function is ReLu for each layer and the optimizer applied during the training is Adam [16]. Each convolutional layer is formed by 32 filters, except for the last layer, in which only one filter is used. This is because the output should have the same size as the input. The kernel size measures three pixels for both spatial directions and the same padding is imposed. Max pooling and up-sampling layers are exploited in order to reduce and increase the data dimension, respectively. For an input with size $(128,128,1)$, the latent space representation size is $(32,32,32)$. The model architecture was optimized by performing hyperparameters tuning over a validation set, additional information can be found in appendix A.

The definition of the loss function is particularly significant in constructing the autoencoder. The commonly used mean squared error (MSE) is not suitable for the training of the proposed dataset. In fact, in the standard MSE equal importance is given to both black (low intensity) and non-black (high intensity) pixels, even if the interest sits mainly with the non-black pixels, which indicate the presence of the atoms. Note that the importance of black and white pixels is reversed when the image is taken in the bright field. It should also be noted that Poisson noise varies according to the pixel intensity. Therefore, when a pixel is black no noise is detected and the corrupted and uncorrupted images are identical. For this reason, we employ a customized loss function, which gives more importance to the pixels that are not black in the original images contained in the training set. This loss function can be described as a weighted MSE (WMSE), with a 1000:1 weight ratio for non-black pixels. It can be expressed as

$$\begin{equation} \mathrm{WMSE}=\frac{1}{n}\sum_{i=1}^{n}w_{i}(\widetilde{y}_{i}-y_{i})^2 \end{equation} \tag{ 2 }$$

$$\begin{equation*} w_{i} = \left\{\begin{array}{ll} 1 & \textrm{if}\ i=\mathrm{min}(y);\\ 1000 & \textrm{otherwise}.\end{array} \right. \end{equation*}$$

where n is the number of pixels in each image, w_i is the weight associated to the ith pixel, $\widetilde{y}_i$ the predicted value and y_i the true value.

The autoencoder is trained on a dataset made of about 27 000 simulated images, all generated by using the Prismatic software [3, 14].

Different materials have been considered in the creation of the training set, namely, graphene, graphite, GaAs, InAs, MoS₂, SrTiO₃, and Si. These include pristine and defective structures incorporating vacancies, generated across various imaging conditions. Details about the training set construction can be found in appendix B. All images have been cropped into 128 × 128-pixel plots, which is the format used to construct the autoencoder. Note that, common real images are usually larger (at least 512 × 512 pixel), but dealing with reduced size allows one to increase the computational efficiency and to identify more details in the reconstructed data.

The pixel intensity of the images generated by Prismatic corresponds to the fractional intensity of the entire electron beam that is scattered to the specified STEM detector at a given pixel. This takes values comprised between 0 and 1, and usually corresponds to around 10%–15% of the entire beam intensity. Measuring the signal in fractional units does not allow us to directly retain information of the actual physical electron dose used. As such, before applying the Poisson noise, it is necessary to convert the pixel values into integers, representing the physical number of electrons at a given pixel. This can be obtained by multiplying the original pixel value by the total electron dose (in units of electrons per Ų) and then by multiplying by the pixel size (in Ų). Such conversion is important to generate a training set with an intensity distribution compatible to that of typical experimental data. As a consequence, any type of pre-processing of the test data can be avoided, which is a crucial condition to use our machine-learning tool during the actual microscope real-time data acquisition. It is also important to remark that in doing so the pixels value, namely the outcome of our autoencoder, will describe a directly measurable quantity with proper physical meaning. This is not common practice in the case of analog-acquired data, where the pixel value cannot be directly associated to an observable and the images are usually scaled between 0 and 1 before training and testing.

The choice of the dose values included in the training set highly affects the performance of the model and the ability to denoise images taken over a wide range of doses. As such one wishes to keep that range as wide as possible. However, both the time required for training and the computational costs also scale with the size training set and diversity of the data, so that a compromise between data variety and computational effort must be found. For this reason we have selected a dose range going from 500 $e^{-}$ Å⁻² to 10 000 $e^{-}$ Å⁻². In the construction of the dataset the dose value for each image is randomly selected within the defined range, so that there is no dose bias across the various materials.

The most straightforward qualitative evaluation consists of a visual comparison between the reconstructed and the corrupted images. The improvement brought by the denoising process is easily recognizable even by researchers, who are not experts in electron microscopy. A visual example is shown in figure 1, which displays the reconstruction of the digital experimental images of a gold nanoparticle deposited on an amorphous carbon substrate, obtained at different dose levels (note that gold is not included in the training set). The data are provided in the form of 20 different frames of the same sample region; by summing up the signals of an increasing number of frames one can obtain multiple images at different doses. In the experimental acquisition the dwell time was 2 µs for each frame and the beam current was approximately 5 pA. This means that the dose of a single frame is 62 $e^{-}$/pixel. The dose of each image is then 62 $e^{-}/$pixel multiplied by the number of frames used for the image. For the sake of brevity, only 3 of the 20 consecutive sums are presented here. In particular, $128\times128$-pixel portions of the images acquired at 62 $e^{-}/$pixel, 372 $e^{-}/$pixel, and 744 $e^{-}/$pixel are shown in figure 1. The top row displays the noisy images, the second one corresponds to the reconstructions obtained after the application of the autoencoder, the third row contains the difference between the noisy and the reconstructed images (referred to as Residual), and the bottom row shows the fast Fourier transform of the Residual. Although gold is not classified as a beam-sensitive material, the example provided is effective, since it tracks the model performance across different dose levels, a task that remains challenging when dealing with experimental data.

Zoom In Zoom Out Reset image size

The reconstructed data always display a significant quality improvement over the original noisy images. In fact, from the reconstructions, one can immediately recognize the five crystallite forming the gold nanoparticle, regardless of the noise level of the original image. The most notable difference between the three reconstructions is the shape of the individual atoms, which appear progressively more round as the dose increases. Note that our algorithm is not trained to necessarily return round atoms, but only to denoise the signal. This is why at very low dose, as in the case of 62 $e^{-}/$pixel, there is significant atomic distortion. It is also worth noting that no further progress is found when adding additional frames to the 744 $e^{-}/$pixel case (namely when increasing the dose). This result allows one to conclude that, upon autoencoder reconstruction, an increase in the dose is not needed to obtain a satisfactory reconstruction. As a consequence, the beam damage to the sample can be reduced.

The two bottom rows of figure 1 show some periodicity in the removed noise. This is an expected feature of the Poisson noise, which scales with the pixel intensity, and does not imply a loss in crystallographic information. Such periodicity in the residual appears to be more evident for high-dose images, a fact that simply demonstrates that the autoencoder can remove the noise at high dose more efficiently than at low dose. It is worth mentioning that crystal structure information in the residual would not be found in images affected by Gaussian noise only. Gaussian noise, in fact, is not intensity-dependent and therefore does not follow the crystal structure. A final important consideration is that the model is able to efficiently denoise the data despite the presence of an amorphous substrate, which generates diffuse scattering and hence additional non-Poisson noise. Substrate scattering was not included in the training set, so that the reconstructed images should be considered as Poisson denoised but still inclusive of substrate scattering.

A quantitative evaluation of the model performance can be achieved through the so-called line profile analysis. This consists in selecting one line of pixels, along the horizontal direction in this case, and by plotting the pixel intensity at each position. One thus obtains an intensity scan profile that can be used to distinguish atoms of different elements, as shown in figure 2. The line profile analysis is here conducted on a synthetic image of a 252 Å thick TePb specimen oriented along the 001 direction, taken at the low dose of 1000 $e^{-}$ Å⁻², with a pixel size of 0.18 Å. This corresponds to a pixel dose of 32 $e^{-}$/pixel. As we can note from the figure, although the intensity profile of the reconstructed image is not identical to that of the infinite-dose one, the reconstruction appears to be accurate enough to localize and distinguish Te and Pb atoms, namely it contains the same content of information of the ground-truth case. In contrast, when the same line profile analysis is conducted over the noisy image the two different species appear indistinguishable so that the chemical information cannot be extracted. The original images used to perform the line profile analysis can be seen in the bottom row of figure 2.

Zoom In Zoom Out Reset image size

Another technique that can be used to quantitatively validate the model, involves the determination of atomic column localization. This essentially defines the position of the various atomic columns, thus allowing one to extract quantitative structural information from the STEM measurements. By performing atomic column localization, it is possible to quantify possible lattice strain and measure its error. This is a technologically useful information, since strain affects many physical properties of a material [18]. Several computational schemes and associated software are available for this purpose, one of them is the Matlab-based package StatSTEM [19]. StatSTEM is based on the principle that, in STEM images, the intensity peaks are located at the atomic column position and these can be approximated by a Gaussian function [20]. Since several localization methods can be used to determine the position of the Gaussians, it is important that our quantitative analysis considers both the localization method and the denoising algorithm, in order to provide a quantitative benchmark of the various possible image-processing workflows.

The precision of the atomic column localization can be estimated by measuring the distances between the various atoms, in both the horizontal and vertical directions. As these are determined solely by the crystalline structure, a statistical distribution of the distances provides a quantitative measure of the accuracy of the combined denoising and localization algorithm. Thus, the standard deviation of the computed distances can be taken as a measure of the accuracy and one can compare results obtained for the infinite-dose, the noisy and the reconstructed images. A reduction in the distance standard deviation corresponds to an enhancement in the image resolvability. The strain error along the horizontal and vertical direction can then be found by dividing the standard deviation in the position by the reference horizontal and vertical distances, respectively.

Figure 3 shows the simulated image used for this investigation. This corresponds to a 252 Å thick Tellurene sample oriented along the 001 direction and imaged with a pixel size of 0.2 Å. The reference horizontal and vertical distances, a and b respectively, are marked by the red arrows. The easiness of identifying these distances makes Tellurene a good candidate for strain error analysis. The plot in figure 3 corresponds to the highest dose considered for this analysis, namely 10 000 $e^{-}$ Å⁻², while denoising is also performed for images taken at 500 $e^{-}$ Å⁻², 1000 $e^{-}$ Å⁻², 2500 $e^{-}$ Å⁻², 5000 $e^{-}$ Å⁻² and 7500 $e^{-}$ Å⁻².

Zoom In Zoom Out Reset image size

Our denoising autoencoder is then tested against a commonly used algorithm for image processing, namely a Gaussian filter [21]. The procedure to measure the column localization by using StatSTEM is as follows. Firstly, one needs to define the starting coordinates for the atomic column positions, namely the local maxima in the image. This can be achieved by using one of the two available peak-finder routines that may include also filters to smooth the image. Such filtering option is not employed in the present study, where the only simplification is the setting of a threshold value to remove nuisance pixel intensities from the background. This step is necessary to avoid the identification of too many fictitious atoms in the case of images characterized by strong noise. In order to set the same value for each image and to make the analysis coherent, the intensity is normalized to 1; then we find that the minimum threshold value compatible with the algorithm memory requirement is 0.12. The difference between the two peak-finder routines lies in the way the smoothing of the image can be achieved, namely either by adding filters or by altering the estimated radius of the atomic columns. The so-called Peak-finder routine 2 is used for this analysis; however, the same results can be achieved with the Peak-finder routine 1 when no filters are imposed.

After this first step, one has the option to manually add atoms positions, an operation that is not considered in this case in order to avoid any human bias in the workflow. Once the starting coordinates are identified, a fitting procedure can be used to model the image as a superposition of Gaussian peaks. In this step, it is possible to specify the width of the atomic columns, by choosing between the Same and Different options. In the first case all Gaussian are taken with the same width. In contrast, the second more computationally demanding option makes the approach more general, since it does not assume that all the Gaussian peaks have the same width. As such it avoids the introduction of any a priori knowledge of the image. The final result of this procedure is a set of coordinates, which correspond to the atomic columns in the image. These values can be used to measure the horizontal and vertical distances a and b between the atomic columns, as shown in figure 3. The values obtained from this operation are displayed as histograms in the appendix. From the histogram corresponding to each image, the strain error can be computed and compared at different dose values. This is shown for the distances a and b of Tellurene (see figure 3) in figure 4.

Zoom In Zoom Out Reset image size

In the figures, the strain error for the infinite-dose case is represented as a blue line and, by definition, it is dose-independent. This, in fact, represents the ultimate theoretical precision achievable by noise-free STEM. In contrast, the noisy images have a strain error that grows with reducing the dose, in an approximate exponential behavior. Our autoencoder is able to drastically improve over the noisy images and returns us a strain error that trails closely that corresponding to the infinite-dose case. In more detail, we find that the denoised images present a strain error, which is approximately dose independent when the electron dose is higher than 2500 $e^{-}$ Å⁻². For lower doses a sharp error increase is reported. Such increase, however, leaves the strain error far below what computed for the noisy images. We then conclude that the autoencoder is less effective at ultra-low dose, but still remains significantly performing across the entire range. Note that one, in principle, can also train different autoencoders for different dose levels. This, however, may not be a good strategy, since often the beam intensity is not accurately measured during the STEM operation. For this reason an autoencoder trained over a broad dose range appears more flexible for real-life situations. Interestingly, simple Gaussian filtering (red lines in the figures) appears unable to improve the column localization of noisy images, with the only exception at very high doses. Therefore, the commonly used Gaussian filter should be avoided when performing accurate quantitive measures of atomic positions, at least when the data-processing workflow remains completely users unbiased. In this situation our results suggest that even the untreated images provide a better estimate of the column positions, unless rather large doses are used.

We believe that the comparison provided here should represent a general benchmarking scheme to compare different denoising workflows in a completely unbiased way.