Unsupervised Radar Target Detection under Complex Clutter Background Based on Mixture Variational Autoencoder

Abstract

The clutter background in modern radar target detection is complex and changeable. The performance of classical detectors based on parametric statistical modeling methods is often degraded due to model mismatch. Existing data-driven deep learning methods require cumbersome and expensive annotations. Furthermore, the performance of the detection network is severely degraded when the detection scene changes, since the trained network with the data from one scene is not suitable for another scene with different data distribution. To this end, it is crucial to develop an unsupervised detection method that can finely model complex and changing clutter scenes. This problem is challenging yet rewarding because it completely eliminates the cost of obtaining cumbersome annotations. In this paper, we introduce GM-CVAE, a novel unsupervised Gaussian Mixture Variational Autoencoder with a one-dimensional Convolutional neural network approach to finely model complex and changing clutter. Furthermore, we develop an unsupervised narrow-band radar target detection strategy based on reconstructed likelihood. Comprehensive experiments are carried out to show that the proposed method realizes the refined modeling of clutter and guarantees superior detection performance in the simulated complex clutter environment. Compared with baselines, the proposed method shows better performance.

1. Introduction

Target detection technology, as the most basic and important function of radar [1,2], has always been paid attention by military and researchers. Adaptive threshold techniques are widely used in radar automatic detection to maintain a constant false alarm rate (CFAR) in an unknown clutter environment. Cell-averaging CFAR (CA-CFAR) [3,4] is the earliest proposed type of CFAR and is very popular in practical radar detection. The background level is estimated by averaging the amplitudes of the reference cells selected from the neighbor cells of the cell under test (CUT), which is optimal in a homogeneous Rayleigh distribution. Under the heterogeneous background environment, data in the reference cells will no longer satisfy the condition of independent and identically distributed (i.i.d.), and the detection performance of the CA-CFAR method will decline inevitably [5]. When the CUT is located in the noise region and the reference window contains clutter samples or other interfering targets, a masking effect will occur, which then will lead to serious degradation of the detection probability. Furthermore, at the clutter edges, the false alarm rate may increase intolerably. Therefore, the greatest of selection CFAR (GO-CFAR) method [6,7] is proposed, which is better than the CA-CFAR method in maintaining the CFAR property in the clutter edges case. After this, the order statistic CFAR (OS-CFAR) [8,9] has been proposed in a heterogeneous background environment. OS-CFAR ranks the amplitudes of the reference cells and selects the k-th sample as the average power level of the CUT, where the choice of the order k depends on the desired false alarm rate.

The CFAR detection algorithms aforementioned are mainly used to solve the target detection problem under the clutter with Gaussian statistical characteristics. Furthermore, aiming at the clutter background with non-Gaussian statistical characteristics, researchers have also proposed a series of models in recent years, such as the Weibull distribution model [10,11], K-distribution model [12,13], spherical invariant stochastic process model [14] and compound Gaussian model [15,16,17,18,19]. For heterogeneous clutter, some methods have been developed especially for the clutter edges case. For example, refs. [20,21] are put forward to solve the clutter edges problem in Gaussian background , while [10,22,23] studied this problem under non-Gaussian distribution. In addition, for the problem of inaccurate estimation of sample covariance matrix due to too little i.i.d. training data available in heterogeneous clutter environment, refs. [24,25] develop matrix information geometry (MIG) detectors and [26,27,28] studies adaptive subspace signal (ASD) detector to improve target detection performance.

The existing radar target detection methods consider different statistical distribution models, corresponding detectors have also been designed by researchers based on different models, and achieved the expected detection performance under the background of their respective assumptions. However, these methods have some common characteristics and limitations. Firstly, they are parametric-based statistical modeling methods. Existing detection methods are all statistical results of a large amount of data, which have been experimentally verified and have good universality. However, due to only utilizing empirical information, the statistical distribution form is fixed, and the pertinence is poor so that the model cannot be corrected with the change of data. In practical applications, there are problems such as model mismatch, which lead to a significant drop in performance. Secondly, the information dimension used is relatively simple, mainly the amplitude one-dimensional feature, and  more high-dimensional features are not further extracted from Doppler and time-domain distributions, etc. The underlying assumption is that the target must have a strong amplitude relative to the clutter background. Therefore, underutilization of feature information often leads to its limited performance. Finally, the accuracy of clutter modeling is relatively coarse, and lacks refined modeling theories and methods. In practice, there may be a variety of clutter in the radar irradiation area, subjecting to different distribution and forming a complex clutter scene, which will also affect the final detection performance.

The ground-based warning radar is a typical application scenario of the CFAR detection method. As for the ground-based warning radar, it has an important responsibility to ensure homeland security, and needs to detect, track and even identify suspicious airborne targets (low altitude penetration, unmanned aerial vehicles and other low, small and slow targets.) and suspicious ground targets in advance. Radar is required to protect the large scope, meanwhile, the ground clutter is more complex (as shown in Figure 1, there may be various terrains in the range covered by the beam, such as mountains with trees, grass, towns and farmland, lakes, and bushes next to shrubs, etc.) and even include artificial active and passive interference. Such a complicated detection scene means that the above-mentioned detection methods for specific clutter backgrounds will no longer be applicable, so it is important to introduce a refined clutter modeling method.

Due to its tremendous capability in learning expressive representations of complex data, a recent trend is to utilize deep learning in a wide variety of tasks [29,30], including radar target detection [31,32,33]. The radar detection task in deep learning can be regarded as a binary classification problem [34,35]. The most typical approach is to learn a classifier on training samples with class labels and then apply it to classify test samples. However, the trained network with the data from one scene is not suitable for another scene with different data distribution. Oftentimes, we need to retrain the classifier model on the new dataset, which requires manually labeling many new training samples. To alleviate this problem and eliminate or reduce the need to label new datasets, domain adaptation methods [36,37,38] are proposed. This method utilizes annotated data from the source domain (there is a large amount of data in the source domain) to train the classifier to operate in the target domain, which is not friendly to unlabeled training datasets. For training datasets lacking effective labels, unsupervised deep learning methods are more suitable. Unfortunately, to the best of our knowledge, there are almost no significant attempts to use unsupervised learning methods for target detection in a ground-based warning radar background. In addition, the military field is usually faced with non-cooperative targets, which are difficult to enumerate and obtain, leading to a serious shortage of target samples. In other words, clutter data and target data are extremely unbalanced, which inevitably leads to the problem of class distribution imbalance. Applying a classification model, in this case, is not a wise move. In the current work, we explore an unsupervised mode for radar target detection that is certainly more challenging than supervised methods. However, it is also more rewarding due to minimal assumptions and hence will encourage the development of novel and more practical algorithms. In approaching unsupervised target detection in ground-based warning radar, we exploit the simple fact that targets are less frequent than the clutter happenings, and attempt to leverage such domain knowledge in a structured manner.

Having said all of the above, in view of the differences in clutter distribution caused by environmental diversity, we aim to use an unsupervised deep generative model to learn robustly and model finely complex clutter distributions, find the specific distribution of interested target according to the characteristics of clutter and target difference, and then extract, amplify and detect the interested target in this paper. As a consequence, motivated by [39,40,41,42], we propose a Gaussian Mixture Variational Autoencoder with a one-dimensional convolutional neural network (GM-CVAE) method considering the attractive properties of the convolutional neural network (CNN). The model takes pure clutter data as input and learns the log-likelihood as output. Specifically, GM-CVAE learns a latent representation of clutter to capture its normal pattern, and then reconstructs the input data from the latent representation. Target and clutter distributions, meanwhile, are different, the reconstruction effect of the model on the target will be worse, so the target detection can be achieved by setting the reconstruction error threshold.

Our contributions are summarized as follows:

• In order to solve the problem of complex and changeable clutter in the radar scanning range in the actual environment, we propose a GM-CVAE framework to realize refined modeling of clutter.
• Considering that it is difficult to obtain the true labels of the data and the imbalance of clutter and target samples, we develop an unsupervised narrow-band radar target detection strategy based on reconstructed likelihood.
• Experiments are used to evaluate our approach on simulated complex clutter datasets. These demonstrate the superiority of our method compared to the baselines.

The remaining parts of this paper are organized as follows. Section 2 presents the relevant preliminary knowledge. Section 3 describes the proposed method and the target detection strategy is presented in Section 4. Section 5 summarizes the numerical results and a general conclusion is presented in Section 6.

2. Preliminaries

In this section, we first introduce the clutter and target characteristics of the ground-based warning radar, respectively. Then, we provide a brief overview of the variational autoencoder, which is the basis of our work.

2.1. Ground Clutter Characteristics

Ground clutter is the main factor affecting the detection and tracking performance of the ground-based warning radar for low-altitude targets and ground targets. Essentially, ground clutter is the vector sum of echoes from many scatterers within the radar resolution cell. The clutter scattering phenomenon can be understood as a random process related to the random shape of the ground, which is usually described by the probability distribution model of clutter amplitude and clutter correlation model [43]. Many experimental facts have verified that for a low-resolution radar with a beam incident angle greater than 5° (antenna beam width greater than 2°, pulse width greater than 1 μs, the clutter amplitude distribution of sea clutter, ground clutter as well as meteorological clutter obeys the Rayleigh distribution. Therefore, this paper assumes that the ground clutter amplitude follows a Rayleigh distribution, the ground clutter power spectrum is described by a Gaussian function in which spectral center shift and spectral broadening are different, and partial relevant parameters in [44] are referred to for ground clutter simulation. In particular, the Rayleigh distribution is the distribution of the envelope of a Gaussian random signal passing through a narrow band linear system and needs to satisfy the following conditions: (1) The scatters within the radar irradiation unit are statistically independent. (2) The distance difference between the two scatters and the radar in the irradiation unit is much smaller than the scale of the irradiation unit, and the gain of the antenna in the unit is constant. (3) The number of scatterers is large and the signal is normally distributed according to the central limit theorem.

2.2. Target Characteristics

The primary task of the ground-based warning radar is to detect unknown targets as early as possible, determine their positions within the detection range as far as possible, and closely monitor the flying targets in the airspace. In order to satisfy the needs of long detection distance, ground-based warning radars generally work in lower frequency bands, mainly S and L frequency bands, and some work in the UHF frequency band, of course, VHF, and even HF frequency bands. The detection distance of ground-based warning radar is more than 350 km, even up to 550 km, but it does not have high requirements on the accuracy and resolution of the detection target. Generally, in low-resolution ground-based warning radar, the target imaged on the range-Doppler spectrum is a point-like target with a small scale, and usually does not cross range cells. Therefore, the radar target model considered in this paper is a point-like target that occupies only one range cell.

2.3. Variational Autoencoder

GM-CVAE is a variant of Variational Autoencoder (VAE) [45,46,47], which uses the Gaussian mixture model (GMM) as the prior of VAE latent space, extracts and expresses features of data through probability modeling. The biggest difference between GM-CVAE and VAE is that the former can handle complex multi-modal data, while the latter may not be able to approximate the original multi-modal data distribution well due to a single Gaussian distribution prior. Hence, a brief description of the variational autoencoder is necessary before introducing our methods.

VAE is a popular reconstruction probabilistic model which combines Bayesian inference with the autoencoder framework. The idea of VAE is that the distribution of complex data can be modeled by lower-dimensional latent variables or representations. In a regular VAE, ref. [47] first presents a computationally tractable method for training this model. Specifically, as shown in Figure 2, the prior distribution of the latent variables $\mathit{z}$, denotes $p_{\theta }\left(\mathit{z}\right)$, are often assumed to obey the unimodal Gaussian distribution, and generally for simplicity, $p_{\theta }\left(\mathit{z}\right)\sim \mathcal{N}(0,1)$, where $p(·)$ is the probability distribution function and $\theta$ is the parameters of decoder. Through the encoder network with parameters $\varphi$, VAE learn posterior distribution $q_{\varphi }\left(\mathit{z}\right|\mathit{x})$ so as to approximate the true posterior distribution $p_{\theta }\left(\mathit{z}\right|\mathit{x})$. Then, it learns likelihood distribution $p_{\theta }\left(\mathit{x}\right|\mathit{z})$ so as to reconstruct the original input $\mathit{x}$ through the decoder network with parameter $\theta$. The better the $\mathit{z}$ learned by the network, the more accurate the reconstruction will be. Given that the input data $\mathit{x}$ is known and the latent variable $\mathit{z}$ is unknown, we hope that two distributions $q_{\varphi }\left(\mathit{z}\right|\mathit{x})$ and $p_{\theta }\left(\mathit{z}\right|\mathit{x})$ are as close as possible. Therefore, the following objective function [47] is obtained

$$\underset{\varphi ,\theta }{min}{D}_{\mathrm{KL}}(q_{\varphi }\left(\mathit{z}\right|\mathit{x})\left|\right|p_{\theta }\left(\mathit{z}\right|\mathit{x})),$$

(1)

where $D_{\mathrm{KL}}$ stands for Kullback–Leibler divergence, which definition equation is $D_{\mathrm{KL}}(q\left(\mathit{x}\right)\left|\right|p\left(\mathit{x}\right))=\int q\left(\mathit{x}\right)\log \frac{q\left(\mathit{x}\right)}{p\left(\mathit{x}\right)}d\mathit{x}$. It is mainly used to measure the similarity between two probability distributions. Next, through statistical derivation, the marginal log-likelihood of the input data $\mathit{x}$ is obtained

$$\log p\left(\mathit{x}\right)=D_{\mathrm{KL}}(q_{\varphi }\left(\mathit{z}\right|\mathit{x})\left|\right|p_{\theta }\left(\mathit{z}\right|\mathit{x}))+{\mathcal{L}}_{\mathrm{VAE}}(\varphi ,\theta ;\mathit{x}),$$

(2)

where

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{VAE}}(\varphi ,\theta ;\mathit{x})={\mathbb{E}}_{q_{\varphi }\left(\mathit{z}\right|\mathit{x})}\left[\log p(\mathit{x},\mathit{z})\right]-{\mathbb{E}}_{q_{\varphi }\left(\mathit{z}\right|\mathit{x})}\left[\log q_{\varphi }\left(\mathit{z}\right|\mathit{x})\right],\end{array}$$

(3)

and ${\mathcal{L}}_{\mathrm{VAE}}(\varphi ,\theta ;\mathit{x})$ is called the variational lower bound.

Since $\mathit{x}$ is deterministic, $\log p\left(\mathit{x}\right)$ is a constant. Consequently, minimizing $D_{\mathrm{KL}}$ term is equivalent to maximizing ${\mathcal{L}}_{\mathrm{VAE}}$ term according to Equation (2). Furthermore, the objective function of VAE can be rewritten as follows

$$\underset{\varphi ,\theta }{max}{\mathcal{L}}_{\mathrm{VAE}}(\varphi ,\theta ;\mathit{x})$$

(4)

Further derivation, Equation (4) can be rewritten as

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{VAE}}(\varphi ,\theta ;\mathit{x})& =-D_{\mathrm{KL}}(q_{\varphi }\left(\mathit{z}\right|\mathit{x})\left|\right|p_{\theta }\left(\mathit{z}\right))+{\mathbb{E}}_{q_{\varphi }\left(\mathit{z}\right|\mathit{x})}\left[\log p_{\theta }\left(\mathit{x}\right|\mathit{z})\right]\hfill \end{array}$$

(5)

The first term on the right side of equal sign in Equation (5) is the regularization term, which is used to minimize the difference between the posterior distribution $q_{\varphi }\left(\mathit{z}\right|\mathit{x})$ and prior distribution $p_{\theta }\left(\mathit{z}\right)$; the second term is the reconstruction term. The solution of ${\mathcal{L}}_{\mathrm{VAE}}(\varphi ,\theta ;\mathit{x})$ is a maximum likelihood estimation process of input data $\mathit{x}$ and can be solved by stochastic gradient descent (SGD) methods [47]. In addition, reconstruction loss is related to the input data form. If the input data is binary, binary cross entropy (BCE) can be used to approximate the loss; if the input data is continuous, the loss can be replaced by the mean square error (MSE).

However, regular VAEs often fail to approximate hidden layer distributions of complex data due to too simple prior assumptions. The idea behind our GM-CVAE is to use a Gaussian mixture distribution as a prior for the stochastic variable $\mathit{z}$, which depends on the categorical variable $\mathit{c}$. The prior of this variant is multi-modal and can learn more complex distributions than the unimodal prior. In this paper, we study a high-dimensional Gaussian mixture model to estimate the sample densities for target detection, which will be clear in next section.

3. Method

This section mainly discusses the Gaussian Mixture Variational Autoencoder with a one-dimensional Convolutional neural network (GM-CVAE), which is proposed to refine clutter modeling in an unsupervised manner to optimize target detection.

3.1. Overview of the Framework

The overall framework shown in Figure 3 comprises data pre-processing, the representation model and detection three modules.

The data pre-processing module firstly performs pulse compression on the radar echo data; then selects Moving Target Indication (MTI) or Adaptive Moving Target Indication (AMTI) for filtering processing according to static or moving clutter; after that, it utilizes Moving Target Detection (MTD) for coherent integration to obtain the range-Doppler spectrum (also called the R-D spectrum) and referring to [33,48], we take the modulus of the R-D spectrum; lastly, it performs a normalization operation to avoid numerical problems and speed up network convergence. we define the normalized R-D spectrum as $\mathbf{X}$, $\mathbf{X}=[{\mathit{x}}_1,{\mathit{x}}_2,\cdots ,{\mathit{x}}_N]\in {\mathbb{R}}^{M\times N}$, where M and N represent the number of Doppler channel and range cell, respectively, and $\mathbb{R}$ denotes the set of real number. The clutter data varies drastically in the spatial domain, and the assumption that data in the CUT and the reference cells are independent and identically distributed cannot be guaranteed. Therefore, ${\mathit{x}}_i\in {\mathbb{R}}^{M\times 1},i=1,2,\cdots ,N$ denotes the Doppler feature vector of the i-th range cell and makes it a sample in this paper. For the sake of description, we will ignore the subscript i for the rest of the discussion.

The difference between the data pre-processing process in this paper and the traditional radar signal processing process is the normalization step. Furthermore, we do not discuss a novel method of clutter suppression, which is beyond the scope of this paper. In the representation module, we propose a GM-CVAE to learn the complex structures and Doppler characteristics of clutter data. Its details will be covered in Section 3.2. Subsequently, target detection is performed based on the reconstruction results inferred from the model. The specific target detection method will be introduced in Section 4.

3.2. Network Architecture

As shown in Figure 4, GM-CVAE is a reconstruction-based model, including an inference network and a generative network. Next, we will introduce the two sub-networks, respectively.

Generative Network. It measures the likelihood of generating data instances given the latent variables. The generative model (as shown in Figure 4a) can factorize as

$$p_{\theta }(\mathit{x},\mathit{z},\mathit{c})=p_{\theta }(\mathit{c},\mathit{z})p_{\theta }\left(\mathit{x}\right|\mathit{z})=p_{\theta }\left(\mathit{c}\right)p_{\theta }\left(\mathit{z}\right|\mathit{c})p_{\theta }\left(\mathit{x}\right|\mathit{z}),$$

(6)

and generate an observed sample $\mathit{x}$ from the latent random variable $\mathit{z}$ and latent discrete variable $\mathit{c}$, where $\theta$ is the network parameter. Specifically, we assign the form of the prior of the latent variable $\mathit{z}$ as

$$\begin{array}{c}\hfill \mathit{z}|\mathit{c}\sim \prod _{k=1}^K\mathcal{N}{\left(\mathit{z}|{\mathit{\mu }}_k,\mathrm{diag}\left({\mathit{\sigma }}_k\right)\right)}^{{\mathit{c}}_k}\end{array}$$

(7)

where $\{\mathit{\mu },\mathit{\sigma }\}={\{{\mathit{\mu }}_k,{\mathit{\sigma }}_k\}}_{k=1}^K$ represent the mean and variance parameters of multiple Gaussian distribution, which are obtained through the fully connected networks, respectively. K denotes the predefined number of components in the Gaussian mixture and $\mathrm{diag}(·)$ function constructs a diagonal matrix. $\mathit{c}={\left({\mathit{c}}_1,\cdots ,{\mathit{c}}_K\right)}^{\mathrm{T}}$ is an explicit latent variable associated with each $\mathit{x}$ and is a one-hot vector subject to categorical prior distribution $\mathit{c}\sim \mathrm{Cat}\left(\mathit{\pi }\right)$ with parameter $\mathit{\pi }\in {\mathbb{R}}_{+}^{K\times 1}$. Thus, the latent variable $\mathit{z}$ conditions on a different Gaussian distributed prior $\mathit{c}$, which indicates the diverse distribution characteristics of the input. Furthermore, by marginalizing $\mathit{z}$, the following equation can achieve

$$\begin{array}{c}\hfill \mathit{z}\sim \sum _{\mathit{c}}{p}_{\theta }\left(\mathit{c}\right)p_{\theta }\left(\mathit{z}\right|\mathit{c})=\sum _{k=1}^K{\pi }_k\mathcal{N}\left({\mathit{\mu }}_k,\mathrm{diag}\left({\mathit{\sigma }}_k\right)\right)\end{array}$$

(8)

Apparently, it is a mixture Gaussian distribution, which has higher representation power than a Gaussian distribution, so it is an ideal choice for characterizing the R-D spectrum characteristics of complex clutter.

Similar to [49], the generated distribution $p_{\theta }\left(\widehat{\mathit{x}}\right|\mathit{z})$ is assigned as a Gaussian distribution and conditioned on both $\mathit{z}$, which can be expressed as

$$\begin{array}{cc}\hfill & \widehat{\mathit{x}}|\mathit{z}\sim \mathcal{N}({\mathit{\mu }}_{\mathit{x}},{\mathit{\sigma }}_{\mathit{x}})\hfill \\ \hfill & {\mathit{\mu }}_{\mathit{x}}=f\left({\mathit{w}}^{{\mathit{\mu }}_{\mathit{x}}}\mathit{z}+{\mathit{b}}^{{\mathit{\mu }}_{\mathit{x}}}\right)\hfill \\ \hfill & {\mathit{\sigma }}_{\mathit{x}}=f\left({\mathit{w}}^{{\mathit{\sigma }}_{\mathit{x}}}\mathit{z}+{\mathit{b}}^{{\mathit{\sigma }}_{\mathit{x}}}\right)\hfill \end{array}$$

(9)

where ${\mathit{\mu }}_{\mathit{x}}$ and ${\mathit{\sigma }}_{\mathit{x}}$ denotes mean and variance parameters of the generated distribution. All ${\mathit{w}}^{*}$-s, ${\mathit{b}}^{*}$-s represent the weight parameters and bias parameters of the corresponding fully connected networks, respectively. $f(·)$ is a nonlinear activation function, here we use LeakyReLU activation function and its expression is

$$\mathrm{LeakyReLU}\left(\mathit{x}\right)=\left\{\begin{array}{cc}\mathit{x},\hfill & if\mathit{x}\ge 0,\hfill \\ \alpha \mathit{x},\hfill & if\mathit{x}<0,\hfill \end{array}\right.$$

(10)

The value of $\alpha$ is 0.1 in our model.

Finally, to better generate the structures of data, the decoder performs the one-dimensional deconvolutional network as ${\mathit{x}}^{\prime }=\mathrm{DCNN}\left(\widehat{\mathit{x}}\right)$, where $\mathrm{DCNN}$ denotes the one-dimensional deconvolutional operation with trainable parameters $\mathit{D}$. As the deconvolution operation proceeds, the spatial resolution gradually increases, where the output of the final deconvolution layer is aimed at reconstructing the input $\mathit{x}$. Note that the constant change of model parameters during deep network training will cause the internal covariate shift [50], resulting in slow and inefficient network training. Hence, a BN layer is connected after the one-dimensional deconvolution layer in the deconvolution network to solve this problem.

Inference Network. It is a difficult problem to directly solve the generative model, i.e., finding maximum a posterior (MAP) of latent variables and maximum likelihood estimation (MLE) of parameters. In response to this issue, a new distribution $q_{\varphi }(\mathit{z},\mathit{c}|\mathit{x})$ is employed to approximate the true posterior distribution $p_{\theta }(\mathit{z},\mathit{c}|\mathit{x})$, which is drawn from specific class and parameterized by trainable parameter $\varphi$. Specifically, according to the mean-field theory approximation [51], the inference model (as shown in Figure 4b) can be factorized as

$$q_{\varphi }(\mathit{z},\mathit{c}|\mathit{x})=q_{\varphi }\left(\mathit{c}\right|\mathit{x})q_{\varphi }\left(\mathit{z}\right|\mathit{x},\mathit{c})$$

(11)

In the inference network, for capturing the structural characteristics of input, a one-dimensional convolution network is conducted, i.e., $\overline{\mathit{x}}=\mathrm{CNN}\left(\mathit{x}\right)$, where $\mathrm{CNN}$ denotes the one-dimensional convolutional operation with parameters $\overline{\mathit{D}}$. Similarly, a BN layer is connected after the one-dimensional convolution layer in Figure 4b. To increase the content covered by a convolutional kernel and the sparsity of the hidden units, a convolutional layer is usually followed by a pooling layer. However, the pooling layer that may reduce the resolution of the feature map is removed from the convolutional encoder, because the radar cross-sectional area of the target itself in the detection scene is small, the reflected energy is weak, and the scale of the target is small on the R-D spectrum.

After that, the parameter of the category distribution, that is, the indicator variable $\mathit{\pi }$, can be expressed as

$$\mathit{\pi }=\mathrm{softmax}({\mathit{w}}^{\mathit{\pi }}\overline{\mathit{x}}+{\mathit{b}}^{\mathit{\pi }})$$

(12)

where ${\mathit{w}}^{\mathit{\pi }}$ and ${\mathit{b}}^{\mathit{\pi }}$ are the weight and bias parameters learned by the network, respectively. The $\mathrm{softmax}$ function guarantees the non-negativity of probabilities. Then, we can obtain $\mathit{c}$ by sampling $\mathit{\pi }$ through Equation (14).

Finally, We parametrize variational factors with networks that output mean and diagonal covariance of variational distributions as well as assign their form as Gaussian posteriors

$$\begin{array}{cc}\hfill & \mathit{z}|\overline{\mathit{x}},\mathit{c}\sim \mathcal{N}({\mathit{\mu }}_{\mathit{z}},\mathrm{diag}\left({\mathit{\sigma }}_{\mathit{z}}\right))\hfill \\ \hfill & {\mathit{\mu }}_{\mathit{z}}=f\left(\prod _{k=1}^K{\left({\mathit{w}}^{{\mathit{\mu }}_{\mathit{z}}}\overline{\mathit{x}}+{\mathit{b}}^{{\mathit{\mu }}_{\mathit{z}}}\right)}^{{\mathit{c}}_k}\right)\hfill \\ \hfill & {\mathit{\sigma }}_{\mathit{z}}=\mathrm{softplus}\left(\prod _{k=1}^K{\left({\mathit{w}}^{{\mathit{\sigma }}_{\mathit{z}}}\overline{\mathit{x}}+{\mathit{b}}^{{\mathit{\sigma }}_{\mathit{z}}}\right)}^{{\mathit{c}}_k}\right)\hfill \end{array}$$

(13)

where $f(·)$ denotes the LeakyReLU activation function, ${\mathit{\mu }}_{\mathit{z}}$ and ${\mathit{\sigma }}_{\mathit{z}}$ are the mean and variance parameters of the latent variable $\mathit{z}$ inferred by networks. Likewise, all ${\mathit{w}}^{*}$-s, ${\mathit{b}}^{*}$-s represent the weight parameters and bias parameters of the corresponding fully connected networks, respectively. $\mathrm{softplus}(·)=\log (1+\exp (·\left)\right)$ is to ensure original data non-negative after nonlinear transformation.

Note that, categorical variable $\mathit{c}$ controls the structural prior of the latent variable $\mathit{z}$. Motivated by [47,52], we employ variational inference to learn $\mathit{c}$ in GM-CVAE. However, $\mathit{c}$ is a discrete variable, and the backpropagation algorithm is only applicable to differentiable layers, so it is not practical to directly optimize $\mathit{c}$. Hence, inspired by [53], we introduce the Gumbel-softmax trick to approximate the original categorical distribution $\mathit{c}$ so that $\mathit{c}$ is differentiable and learnable. Specifically, let the variational distribution $q_{\varphi }\left(\mathit{c}\right)=\mathrm{Gumbel}-\mathrm{softmax}\left(\mathit{\pi }\right)$, $\mathit{\pi }\in {\mathbb{R}}^{K\times 1}$, and 

$$\begin{array}{cc}\hfill & {\mathit{c}}_k=\frac{\exp ((\log {\pi }_k+g_k)/\lambda )}{{\sum }_{l=1}^K\exp ((\log {\pi }_l+g_l)/\lambda )},k=1,\cdots ,K,\hfill \\ \hfill & g_k\sim \mathrm{Gumbel}(0,1)=-\log (-\log \left({\epsilon }_k\right)),\hfill \end{array}$$

(14)

where $\lambda >0$ indicates the softmax temperature and controls the softness of softmax. ${\epsilon }_k$ represents a standard uniform variable. When $\lambda$ tends to 0, the samples from the Gumbel-softmax distribution become one-hot distribution, and the Gumbel-softmax distribution $q_{\varphi }\left(\mathit{c}\right)$ is consistent with the categorical distribution prior $p_{\theta }\left(\mathit{c}\right)$.

Furthermore, the same as regular VAE algorithms, we also use a Gaussian variational distribution $q_{\varphi }\left(\mathit{z}\right)$ to infer the posterior distribution of $\mathit{z}$ in GM-CVAE, i.e.,

$$q_{\varphi }\left(\mathit{z}\right)=\mathcal{N}(\mathit{\mu },{\mathit{\sigma }}^2),$$

(15)

where the two parameters $\mathit{\mu }$ and $\mathit{\sigma }$ are transformed by the inference network, and sampling from the latent variables can be achieved by [47]

$$\mathit{z}=\mathit{\mu }+\mathit{\sigma }\odot \mathit{ϵ},$$

(16)

thus ensuring normal execution of the back-propagation algorithm. Herein, $\mathit{ϵ}$ is a vector with all its entries drawn independently from the standard Gaussian distribution, and “⊙” denotes element-wise product.

The goal of GM-CVAE is to find a better stochastic latent representation $\mathit{z}$ for data $\mathit{x}$. For large and complex terrain scenes, it is difficult to accurately describe the entire ground clutter area with one distribution model, and even the same terrain scene obeys different distributions under the influence of different time and climatic conditions. Different clutter backgrounds obey different probability distributions, resulting in the existence of multiple normal modes in the data center. Therefore, it is an excellent choice for the prior of $\mathit{z}$ to follow the mixture Gaussian distribution.

The generative network and inference network are trained together to approximate the true posterior distribution. Details are described below.

3.3. Offline Training Process

GM-CVAE model can be trained by maximizing the evidence of lower bound (ELBO) [54] on the log-likelihood of input data, ${\mathcal{L}}_{\mathrm{GM}-\mathrm{CVAE}}(\varphi ,\theta ;\mathit{x})$,

$${\mathcal{L}}_{\mathrm{GM}-\mathrm{CVAE}}(\varphi ,\theta ;\mathit{x})={\mathbb{E}}_{q_{\varphi }(\mathit{z},\mathit{c}|\mathit{x})}\left[\log \frac{p_{\theta }(\mathit{x},\mathit{z},\mathit{c})}{q_{\varphi }(\mathit{z},\mathit{c}|\mathit{x})}\right]$$

(17)

Combine Equations (6) and (11), and rewrite Equation (17)

$$\begin{array}{cc}\hfill & {\mathcal{L}}_{\mathrm{GM}-\mathrm{CVAE}}(\varphi ,\theta ;\mathit{x})\hfill \\ \hfill & ={\mathbb{E}}_{q_{\varphi }(\mathit{z},\mathit{c}|\mathit{x})}\left[\log \frac{p_{\theta }(\mathit{x},\mathit{z},\mathit{c})}{q_{\varphi }(\mathit{z},\mathit{c}|\mathit{x})}\right]\hfill \\ \hfill & ={\mathbb{E}}_{q_{\varphi }(\mathit{z},\mathit{c}|\mathit{x})}\left[\log \frac{p_{\theta }\left(\mathit{c}\right)}{q_{\varphi }\left(\mathit{c}\right|\mathit{x})}+\log \frac{p_{\theta }\left(\mathit{z}\right|\mathit{c})}{q_{\varphi }\left(\mathit{z}\right|\mathit{x},\mathit{c})}+\log p_{\theta }\left(\mathit{x}\right|\mathit{z})\right]\hfill \\ \hfill & ={\mathbb{E}}_{q_{\varphi }(\mathit{z},\mathit{c}|\mathit{x})}\left[\log p_{\theta }\left(\mathit{x}\right|\mathit{z})\right]-D_{\mathrm{KL}}\left[q_{\varphi }\left(\mathit{c}\right|\mathit{x})\left|\right|p_{\theta }\left(\mathit{c}\right)\right]-D_{\mathrm{KL}}\left[q_{\varphi }\left(\mathit{z}\right|\mathit{x},\mathit{c})\left|\right|p_{\theta }\left(\mathit{z}\right|\mathit{c})\right]\hfill \end{array}$$

(18)

In Equation (18), the first term is the expected log-likelihood of the data $\mathit{x}$, which aims to guarantee the reconstruction capacity of the generative model. The second term is the regularization between the prior $p_{\theta }\left(\mathit{c}\right)$ and posterior $q_{\varphi }\left(\mathit{c}\right|\mathit{x})$ of the categorical variable $\mathit{c}$ that constrains the two distributions as close as possible. The third term represents the regularization between the actual posterior $p_{\theta }\left(\mathit{z}\right|\mathit{c})$ and approximate posterior $q_{\varphi }\left(\mathit{z}\right|\mathit{x},\mathit{c})$ of the latent variable $\mathit{z}$. The closer the prior and posterior distributions are, the smaller the regularization value. Specifically, since $\mathit{c}$ is a discrete variable, the second term in Equation (18) can be written as

$$\begin{array}{cc}\hfill & D_{\mathrm{KL}}\left[q_{\varphi }\left(\mathit{c}\right|\mathit{x})\left|\right|p_{\theta }\left(\mathit{c}\right)\right]\hfill \\ \hfill & =\sum _{k=1}^K{q}_{\varphi }({\mathit{c}}_k=1|\mathit{x})log\frac{q_{\varphi }({\mathit{c}}_k=1|\mathit{x})}{p_{\theta }({\mathit{c}}_k=1)}\hfill \\ \hfill & =\sum _{k=1}^K{q}_{\varphi }({\mathit{c}}_k=1|\mathit{x})\left[log\left(q_{\varphi }({\mathit{c}}_k=1|\mathit{x})\right)-log\left(\frac{1}{K}\right)\right]\hfill \end{array}$$

(19)

As we assign the prior of latent variables as mixture Gaussian distribution, the KL divergence of the third term in Equation (18) can be calculated as

$$\begin{array}{cc}\hfill & D_{\mathrm{KL}}\left[q_{\varphi }\left(\mathit{z}\right|\mathit{x},\mathit{c})\left|\right|p_{\theta }\left(\mathit{z}\right|\mathit{c})\right]\hfill \\ \hfill & =\int \left(\sum _{k=1}^K{\mathit{c}}_k{q}_{\varphi }\left(\mathit{z}|\mathit{x},\mathit{c}\right)\log \frac{q_{\varphi }\left(\mathit{z}\right|\mathit{x},\mathit{c})}{\mathcal{N}\left({\mathit{\mu }}_k,{\mathit{\sigma }}_k\right)}\right)\hfill \\ \hfill & =\sum _{k=1}^K{\mathit{c}}_k{D}_{\mathrm{KL}}\left[q_{\varphi }\left(\mathit{z}\right|\mathit{x},\mathit{c})\left|\right|\mathcal{N}\left({\mathit{\mu }}_k,{\mathit{\sigma }}_k\right)\right]\hfill \end{array}$$

(20)

Moreover, the analytical expression of KL divergence between two Gaussian distributions is

$$\begin{array}{cc}\hfill & D_{\mathrm{KL}}\left[\mathcal{N}({\mathit{\mu }}_1,\mathrm{diag}\left({\mathit{\sigma }}_1\right))\left|\right|\mathcal{N}({\mathit{\mu }}_2,\mathrm{diag}\left({\mathit{\sigma }}_2\right))\right]=\hfill \\ \hfill & \frac{1}{2}\left[\log \frac{{\mathit{\sigma }}_1}{{\mathit{\sigma }}_2}-d+{\mathit{\sigma }}_2^{-1}{\mathit{\sigma }}_1+{({\mathit{\mu }}_1-{\mathit{\mu }}_2)}^{\mathrm{T}}\mathrm{diag}\left({\mathit{\sigma }}_2^{-1}\right)({\mathit{\mu }}_1-{\mathit{\mu }}_2)\right],\hfill \end{array}$$

(21)

where d is the dimension of variables; $\mathrm{diag}(·)$ function constructs a diagonal matrix and ${(·)}^{\mathrm{T}}$ denotes a transpose operation. The analytical KL divergence expressions in Equations (19)–(21), and the reparameterization tricks of Gumbel-softmax in Equation (14) and Gaussian distribution in Equation (16) allow the gradients of ELBO with respect to parameters in the inference network to be accurately evaluated.

Algorithm 1 summarizes the GM-CVAE training algorithm under complex clutter background in an unsupervised learning fashion.

Algorithm 1: GM-CVAE Training Algorithm

input: The pre-processed R-D spectrum training set ${\mathbf{X}}_{\mathrm{train}}={\left\{{\mathit{x}}_i\right\}}_{i=1}^{N_{\mathrm{train}}}$; number of components K; batch-size M.

output: The encoder parameters $\varphi$ and the decoder parameters $\theta$; reconstruction probability vector for all samples ${\mathit{s}}_{\mathrm{train}}$ (see Equation (22));  $\theta$, $\varphi$ ← Initialize parameters.

repeat

Select a mini-batch training subset ${\left\{{\mathit{x}}_n\right\}}_{n=1}^M$ randomly;

Draw random noise ${\left\{\epsilon \right\}}_{n=1}^M$ from uniform distribution for generating samples ${\left\{\mathit{c}\right\}}_{n=1}^M$ according to Equation (14);

Draw random noise ${\left\{\mathit{ϵ}\right\}}_{n=1}^M$ from normal distribution for generating latent variable ${\left\{\mathit{z}\right\}}_{n=1}^M$ according Equation (16);

Calculate $▽{\mathcal{L}}_{\mathrm{GM}-\mathrm{CVAE}}(\varphi ,\theta ;\mathit{x})$ according to Equations (18)–(21), and update parameters of the two sub-nets jointly;

until convergence

return model parameters $\varphi$ and $\theta$, as well as ${\mathit{s}}_{\mathrm{train}}$.

4. Target Detection Strategy

The GM-CVAE model learns the network parameters to capture the normal patterns of clutter samples after offline training. Now based on the difference between clutter and target echo in Doppler feature space, we can ensure whether the input sample $\mathit{x}$ is a target by utilizing the trained model. The reconstruction probability in the GM-CVAE model indicates its reconstruction difficulty [51]: the lower the reconstruction probability value, the more difficult it is to be accurately reconstructed. For a sample with a target, it will be difficult to be reconstructed accurately, and its reconstruction probability is often lower than that of a sample only with clutter [55]. Inspired by [56], we decided to take the reconstruction probability of $\mathit{x}$ as the final score, denoted as s, and compare it with threshold t to determine whether the observed sample is a target. The calculation formula is as follows

$$s={\mathbb{E}}_{q_{\varphi }(\mathit{z},\mathit{c}|\mathit{x})}\log p_{\theta }\left(\mathit{x}\right|\mathit{z})$$

(22)

If the final score s is less than threshold t, observed sample $\mathit{x}$ is determined as a target; otherwise, it will be judged as a pure clutter sample. The threshold t is learned automatically in the GM-CVAE model. Extreme Value Theory (EVT) takes extreme data as the research object, and infers the distribution of extreme data that may be observed without any distribution based on the original data. The second theorem in EVT, the Peaks-Over-Threshold (POT) [57] model, is based on the generalized Pareto distribution to model the distribution of all observed data that exceeds a sufficiently large threshold to study the tail characteristics of the distribution. POT demonstrates excellent performance in threshold tuning, so we apply the adjusted POT method to help choose the threshold t for GM-CVAE. Algorithm 2 describes the process of threshold detection.

Algorithm 2: GM-CVAE Detection Algorithm

input: The pre-processed R-D spectrum test set ${\mathbf{X}}_{\mathrm{test}}={\left\{{\mathit{x}}_i\right\}}_{i=1}^{N_{\mathrm{test}}}$;

output: reconstruction probability vector for all samples ${\mathit{s}}_{\mathrm{test}}$ (see Equation (22)) and labels $\mathit{y}$.

$\varphi$, $\theta$, ${\mathit{s}}_{\mathrm{train}}$ ← Trained GM-CVAE network.

Similar to the loop body of Algorithm 1, reconstruction probability vector ${\mathit{s}}_{\mathrm{test}}$ is obtained;

Feed ${\mathit{s}}_{\mathrm{train}}$ as the initial threshold to the POT algorithm to generate the final threshold vector $\mathit{t}$;

Element-wise comparison of ${\mathit{s}}_{\mathrm{test}}$ and $\mathit{t}$:

if ${\mathit{s}}_{\mathrm{test}}\left(i\right)<\mathit{t}\left(i\right)$, then then

${\mathit{x}}_i$ is a target, label ${\mathit{y}}_i$ = 1

else

${\mathit{x}}_i$ is a clutter, label ${\mathit{y}}_i$ = 0

end if

5. Numerical Examples

In this section, we first introduce our experiment setup in Section 5.1, including experiment data, evaluation metrics, compared baselines, model hyper-parameters and hardware platform. We then evaluate and compare the performances of GM-CVAE and baselines in different clutter scenes in Section 5.2. Finally, we qualitatively analyze the proposed method in Section 5.3.

5.1. Experiment Setup

Experiment Data. In the simulation, the linear frequency modulated signal is chosen for the waveform of the ground-based warning radar, which operates at a wavelength of 0.23 m, a bandwidth of 2 MHz, an azimuth beam width of $2^{°}$, a pulse width of 0.15 ms, a pulse repetition period of 2.37 ms, a sampling frequency of 4 MHz, and 23 coherent pulses in one CPI. In order to fit different terrains with different data distributions, three types of clutter are designed for simulating complex clutter scenes, and Figure 5 shows their R-D spectrum. The x-axis represents 50 range cells of radar, and the y-axis represents the normalized Doppler frequency, which can be converted to the corresponding radial velocity. The red solid box in the figure represents the primary clutter area, and targets will locate in it if data is not dealt with MTI or AMTI in the data pre-processing stage. The green dotted box represents the secondary clutter area and targets will fall in it if MTI or AMTI is conducted in the data pre-processing stage. Figure 6a demonstrates the Doppler characteristics of three types of clutter. The center frequency and clutter spectrum width of three types of clutter are different, Type-2 is static clutter, while Type-1 and Type-3 are moving clutter with different speeds. Figure 6b exhibits the power intensity of three types of clutter partial range cells. It can be seen that the three types of clutter fluctuate violently in space, and it is difficult for the samples to satisfy the conditions of independent and identical distribution. Therefore, combined with the characteristics of the target data, we use each Doppler vector in the R-D spectrum as sample data, and 1D-CNN is used to extract sample features in our GM-CVAE. In the experiment, we have 2400 clutter instances in total and divide the dataset into two parts: the former 2000 for training and the latter 400 for testing. The ratios of target instances in the testing set is 0.05.

Evaluation Metrics. We mainly employ two performance metrics to evaluate GM-CVAE and other baseline methods: detection probability (denoted as $P_d$) and the false alarm rate (denoted as $P_{fa}$). They can be shown as following [35]

$$\begin{array}{cc}\hfill P_d& =\mathrm{Recall}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}\hfill \\ \hfill P_{fa}& =1-\mathrm{Precision}=\frac{\mathrm{FP}}{\mathrm{TP}+\mathrm{FP}}\hfill \end{array}$$

(23)

Compared Baselines. To demonstrate the effectiveness of GM-CVAE, one-dimensional convolutional VAE (CVAE), classical CA-CFAR [3], GO-CFAR [7] and OS-CFAR [9] detectors as well as adaptive detectors, i.e., adaptive normalized matched filter (ANMF) [19] and ASD [27] are chosen as baselines. For ASD detector, we consider the rank-1 case (that is, the subspace signal model is represented as the product of the target amplitude and the steering vector). In other words, the performance of GM-CVAE is compared with the five classical detection methods and CVAE uses a simple Gaussian prior to learn the latent spatial distribution of the data.

Hyper-parameters. In our experiments, GM-CVAE is implemented based on Pytorch. The hyper-parameters of GM-CVAE are set empirically in our experiments as follows. Both the convolutional and deconvolutional layer L = 2, the kernel sizes and strides of which are $\{3\times 1,3\times 1\}$ and $\{2\times 1,2\times 1\}$, respectively. The dimension of $\mathit{z}$-space is 5 empirically and the components of categorical-space are related to the number of clutter types, and the value set is from $\{1,2,3\}$. For the adjusted POT method, the low quantile $=0.2$ and $q={10}^{-4}$ [57]. GM-CVAE is trained with Adam optimizer [58], with a learning rate of $0.0005$. We run 50 epochs for training, and the batch size is 64. More studies about the choice of hyper-parameters will be discussed later.

Hardware Platform Our experiments are conducted on a host with Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz accelerated by NVIDIA RTX 3080, with 10 GB VRAM of the graphic card.

5.2. Performance Comparison

In the range-Doppler domain, clutter or targets with different velocities occupy different Doppler cells, which is an important feature to effectively distinguish signals of different velocities. The Doppler frequency of the slow target is extremely low, and it often overlaps with the Doppler of low-frequency clutter. At this time, the target is submerged in the strong clutter, and feature extraction is difficult, thereby reducing the detection performance. During data pre-processing, the MTI filter can suppress the static zero-frequency ground clutter well, and has better detection performance for higher-speed moving targets, such as the echo of target 2 in Figure 7. However, when detecting a slow-moving target with a lower Doppler frequency, the MTI filter also greatly suppresses the moving target near the zero frequency, such as the attenuated target 1 echo in Figure 7. Likewise, the AMTI filter can better suppress motion clutter, but it can also suppress targets with similar clutter Doppler frequencies. To this end, two scenarios in target detection are considered, one is to employ the MTI or AMTI filter during data pre-processing, and the position of the target avoids the filter notch in the Doppler dimension (see the green dotted box in Figure 5) and is random in the range dimension; another, is to not utilize MTI or AMTI filter during data pre-processing, and the target is added to primary clutter area (See the red solid box in Figure 5).

5.2.1. Data Pre-Processing with MTI or AMTI

The size of the reference window with the aforementioned traditional detectors is 24, and the length of the guard cell is 2. A total of ${10}^4$ Monte Carlo trials are performed. The selection of order k in OS-CFAR is 14 and the target obeys the Swerling type-0/5 distribution. Note that the comparison between network methods and other detectors: the actual false alarm rate of the test set is used as the constant false alarm rate of CA-CFAR, GO-CFAR, OS-CFAR, ANMF and ASD methods after the network training is completed, and then detection probability of the two methods is compared. If the actual false alarm rate in network methods is zero, the constant false alarm rate in the traditional methods is set to ${10}^{-3}$.

Performances in Single-clutter Scene. Taking Type-2 as an example, the detection probabilities of GM-CVAE, CVAE and the five traditional methods versus Signal to Noise Ratio (SNR) are presented in Figure 8a,b, respectively, where the noise power is about 59.3 dB. The actual false alarm rate of GM-CVAE, CVAE and the five traditional detection methods versus SNR are presented in Figure 8d,e, respectively. Figure 8c,f compare the detection and false alarm performance of GM-CVAE and CVAE, respectively. Specifically, GM-CVAE is far superior to CA-CFAR, GO-CFAR and OS-CFAR methods in terms of detection performance under low SNR conditions. The detection performance of ANMF and ASD in the single clutter scene is similar, and compared with the two detectors, our proposed GM-CVAE has a performance improvement of about 3 dB. Since ANMF is a coherent accumulator, it can improve the accumulation gain by accumulating multiple samples, and thus has better detection performance than the three CFAR detectors. From the perspective of false alarm, the five traditional detection methods are slightly higher than the GM-CVAE method, in which the false alarm value fluctuates reasonably. Similar conclusions occur for CVAE and CA-CFAR, GO-CFAR, OS-CFAR, ANMF and ASD methods. Furthermore, compared with Figure 8c,f, GM-CVAE performs slightly better than CVAE under single clutter background, indicating that GMM prior extracts more information than single Gaussian prior.

Performances in Complex-clutter Scene. To simulate the complexity of multiple clutter in radar irradiation area, different types of clutter regions are spliced according to the range dimension. The complex scene is shown in Figure 9, where the three types of clutter are concatenated in equal proportions. The noise powers of Type-1, Type-2 and Type-3 clutter are 56.0 dB, 59.3 dB, and 64.7 dB, respectively. Figure 10 exhibits the detection performances and actual false alarm rates under complex clutter scene.

GM-CVAE and CVAE in Figure 10a,b provide excellent detection performances while CA-CFAR, GO-CFAR and OS-CFAR detection methods suffer a performance degradation owing to the clutter edge effect. Compared with ANMF (or ASD) and GM-CVAE, the latter is improved by about 9 dB. This is because ANMF and ASD detectors rely on clutter covariance matrix estimation. When the clutter is heterogeneously distributed, it is difficult to have a sufficient number of i.i.d. training samples. Furthermore, $P_{fa}$ of CA-CFAR, GO-CFAR and OS-CFAR increases so significantly that it can no longer maintain the constant false alarm rate, our GM-CVAE continues to work robustly yet. Compared with the detection performances of GM-CVAE and CVAE methods in Figure 8c and Figure 10c, it is evident from the result of comparison that the more complex the clutter distribution is, the more significant the $P_d$ improvement of the proposed method is, i.e., the performance of the two methods in a single-clutter scene is close, while $P_d$ is improved by about 2.5 dB under the complex clutter background. This phenomenon indicates that for complex data distribution, GM-CVAE can learn more abundant representational features than CVAE, realizes refined modeling of clutter, and reflects the superiority of GMM prior.

The SNR improvements of the proposed method under different clutter scenes when compared with the CA-CFAR method are summarized in

Table 1. SNR Improvement of the Referred Methods with Respect to CA-CFAR.

	Method	Type-2 (dB)	Type-1 →${}^1$ 2 (dB)	Type-2 → 3 (dB)	Type-1 → 2 → 3 (dB)
case A ${}^2$	GO-CFAR	+0.8	−1.7	−2.4	+2.4
	OS-CFAR	−0.7	−2.5	−3.0	0.0
	ANMF	+11	+8	+6.4	+5.1
	ASD	+12	+9	+7.5	+6.3
	CVAE	+13.7	+14	+13.4	+14.3
	GO-CFAR	0.0	−1.8	−2.0	+1.8
	OS-CFAR	−0.1	−2.6	−1.4	+1.0
	ANMF	+10.5	+9	+7.7	+8
	ASD	+11.5	+10	+8.8	+9
	GM-CVAE	+14.0	+16.0	+15.3	+18.0

¹ ‘→’ represents a splicing operation along the distance dimension; ² case A represents ‘Data pre-processing with MTI or AMTI’.

. CVAE, OS-CFAR, GO-CFAR, ANMF and ASD methods are also considered for better comparison. Since the constant false alarm parameter in the five traditional methods come from the actual false alarm rate of the network method, CVAE and GM-CVAE are compared with CA-CFAR, respectively. Herein, we set the required SNR of the CA-CFAR method to satisfy the reference. In addition to the two clutter scenarios given above, Table 1 also quantitatively analyzes the SNR enhancement of the other two complex scenarios. The statistic in Table 1 demonstrates that our GM-CVAE provides excellent detection performances compared to traditional CA-CFAR, GO-CFAR, OS-CFAR, ANMF and ASD detection methods, both in single-clutter and complex-clutter scenes. Moreover, with the diversification of clutter distribution, the improvement of GM-CVAE detection performance is more evident.

5.2.2. Data Pre-Processing without MTI or AMTI

Performances in Single-clutter Scene. In order to contrast with the previous experiments, we still choose Type-2 clutter as an example, the size of the reference window with the three CFAR methods is 24, and the length of the guard cell is 2. The selection of order k in OS-CFAR is 15 and the noise power is 52.4 dB. The comparison results of different methods are shown in Figure 11. The five traditional detectors suffer a performance degradation owing to the absence of clutter suppression compared to GM-CVAE (Figure 11a,d) and CVAE (Figure 11b,e). It also can be seen that our GM-CVAE increases by about 2 dB compared with that of CVAE under the same $P_d$ (Figure 11c), the $P_{fa}$ drops significantly at lower SNRs concurrently. This further verifies that the proposed GM-CVAE method can more effectively model the distribution of data in harsh clutter environments. In addition, regardless of the network methods or the traditional CFAR methods, the detection performance drops significantly compared to Figure 8, and the traditional CFAR methods almost completely fail at this time. This indicates that when the target velocity is extremely low, it will be seriously affected by clutter, and it is extremely difficult to extract Doppler information.

Performances in Complex-clutter Scene. The selection of order k in OS-CFAR is 16 and the noise power of the three regions is 53.8 dB, 52.4 dB, and 59.4 dB, respectively. Likewise, Figure 12 exhibits the performance curves under the background of complex clutter. With the dual effects of clutter filtering-free and clutter edge, CA-CFAR, GO-CFAR, OS-CFAR, ANMF and ASD detection methods work poorly compared with CVAE and GM-CVAE methods, and CA-CFAR has the worst effect. Compared to the three CFAR detectors, ANMF and ASD achieve slightly higher detection gains due to the coherent accumulation of samples. Moreover, the five traditional detectors need to sacrifice high SNR to detect the target. $P_{fa}$ of the three CFAR methods multiplies approximately compared with CVAE and GM-CVAE methods, and the order k needs to be chosen carefully. In addition, our proposed GM-CVAE method in this scenario continues to work robustly, with higher detection probability and lower false alarm rate than CVAE. This further explains that the isotropic distribution of simple Gaussian in VAE fails to model the intrinsic multimodality of high dimensional data. In conclusion, our GM-CVAE method does have advantages for modeling complex clutter, and it can improve the performance of radar target detection.

5.3. Qualitative Analysis

The Influence of Model Parameters. In this part, we discuss the sensitivity of model parameters, namely the dimension of hidden stochastic variables (i.e., $\mathit{z}$). The reason we do not talk about the number of components of Gaussian mixture $\mathit{c}$ is because the number of clutter types is assumed to be exactly known in each experiment.

GM-CVAE is a reconstruction-based model. For an input observation, GM-CVAE compresses it to a low dimension $\mathit{z}$-space representation and then uses the representation to reconstruct it. That is to say, in this process, the key features in the original data are retained and finally used for reconstruction; some unimportant features are eliminated. Therefore, if the dimension of the latent space $\mathit{z}$ is too low, it means that there are too few key features, and it is easy to lose mode which will lead to the under-fitting of the model; if the dimension of $\mathit{z}$ is too high, it means that more rare features will contribute to the model, making the search space larger and more saddle points present. This makes the learning rate of the model more slowly, and in addition, it will take up too much video memory. Based on this, when SNR = 11 dB, we set the dimension of $\mathit{z}$ ranges from 2 to 8 in the experiment, observe its impact on the detection performance, and choose the most suitable dimension. It can be seen from Figure 13 that the detection probability $P_d$ increases significantly when the dimension of $\mathit{z}$ increases from 2 to 5. This means that adding key features improves the representation ability of the model. When the dimension of $\mathit{z}$ is 5, the detection probability is the highest, which means that the key features retained at this time are enough to represent the original data. When we continue to increase the dimension of $\mathit{z}$, it will not bring significant improvement, but may reduce the performance of the model due to the increase in the search space and the increase in saddle points. Hence, the dimension of $\mathit{z}$ is set to 5 in this paper.

Visualization of $\mathit{z}$-space Representations. To illustrate that our model can fine model complex clutter distributions, here we visualize the 2D embedding in the original data domain and $\mathit{z}$-latent space on the GM-CVAE model for complex clutter scene using t-distributed stochastic neighborhood embedding (t-SNE) [59] method in Figure 14. Each dot in the figure indicates the latent variable of the observation vector, and each color represents a cluster group. The class boundaries of the training samples are blurred in the original data domain according to Figure 14a,c, while samples are separable in $\mathit{z}$-latent space and work well according to Figure 14b,d. This result indicates that our model can represent different clutter features with the mixture distributed latent variables, thus verifying that GM-CVAE can refine modeling.

The number of components of the Gaussian mixture $\mathit{c}$ in the experiments is consistent with the set number of clutter types. If the number of components of Gaussian mixture $\mathit{c}$ is changed in a multiple, i.e., more than one Gaussian component is used for each cluster to approximate the distribution, and the detection performance in single clutter and complex clutter scenes is considered, the results are shown in Figure 15a. The horizontal axis in Figure 15a represents the multiple expansion of the original Gaussian mixture component number. It can be seen that the detection performance has improved in both scenes, but the improvement effect is minimal. The analysis is as follows: objects in a cluster are similar or related to each other, and objects in different clusters are dissimilar or irrelevant. The more similar within clusters and the greater the gap between clusters, the better the effect. However, as shown in Figure 14b,d, our three kinds of data have a large distance in the $\mathit{z}$ latent space itself. Even if it is divided into more clusters (as shown in Figure 15b, 6 clusters), the distance among the clusters does not widen each other. So the performance gain is quite limited.

Furthermore, to show intuitively the efficiency of our mixture distributed latent variable, four case studies of GM-CVAE are shown in Figure 16. Figure 16a shows the sequence of test samples in the background of single-clutter filtered by MTI or AMTI and Figure 16c displays the corresponding log-likelihood score. It can be seen from Figure 16a,c that even if some targets are similar to the clutter distribution, our model is still able to correctly learn the distribution of the clutter and gives the target points a lower log-likelihood. A similar conclusion can be observed in Figure 16b,d. In addition, the target in the red circle in Figure 16b is submerged in the clutter, but the corresponding score exhibits a clear peak, which further validates the superiority of our model in capturing complex clutter features. In Figure 16e,f, the phenomenon of some targets submerged in clutter is more prominent, but the GM-CVAE score remains stable. For the complex-clutter scenes in Figure 16b,f, GM-CVAE can model the non-stationary input assigned into different cluster groups with different parameters, thus leading to a more stable score, while it exhibits considerable spikes in the regions of targets. Consequently, GM-CVE is robust and effective for modeling complex clutter.

6. Conclusions

Traditional detection methods often face the problem of model mismatch under complex clutter backgrounds. Moreover, it is difficult to obtain target data and labels, which makes supervised deep learning methods difficult to implement. To solve this problem, We propose an unsupervised target detection approach (GM-CVAE). GM-CVAE assigns its latent variables into a mixture Gaussian distribution and thus it can model the normal patterns of clutter samples finely by learning their low-dimensional representations, fully capturing Doppler shape features and various data distributions. Moreover, GM-CVAE provides an intuitive and effective way to detect targets based on reconstruction probability. The extensive experimental results demonstrate that GM-CVAE outperforms the traditional detection approaches and CVAE model with single Gaussian as prior in terms of detection probability and the false alarm rate, and show the superiority of one model that fits a great variety of clutter distributions. In addition, our model also achieves effective detection of slow targets in complex clutter backgrounds without sample labels. Although this work adopts simulated data for training, we believe that it has promising potential to apply unsupervised deep learning to radar target detection on complex clutter data. In addition, we expect to obtain sufficient training data with multiple distributions from the radar system which will definitely prompt the detection model to be more convincing. In the future, we will try to extend and evaluate our model with real-world radar data. Other signal features will be analyzed to improve the model performance to adapt complex clutter environment.