Inter-Comparison of Four Models for Detecting Forest Fire Disturbance from MOD13A2 Time Series

Abstract

Many models for change point detection from time series remote sensing images have been developed to date. For forest ecosystems, fire disturbance detection models have always been an important topic. However, due to a lack of benchmark datasets, it is difficult to determine which model is appropriate. Therefore, we collected and generated a benchmark dataset specifically for forest fire disturbance detection, named CUG-FFireMCD1. The CUG-FFireMCD1 contains a total of 132 pieces of MODIS MOD13A2 time series, and each time series contains at least one fire disturbance. The occurrence time for a forest fire disturbance was determined using the National Cryosphere DesertDataCenter(NCDC) website, and the precise latitude and longitude coordinates were determined using the FireCCI51 dataset. In addition, we selected four commonly used time series change detection models and validate the advantages and limitations of the four models through dataset analysis. Finally, we use the detection results of the models and their applicable scenarios to label the additional change points. The four models we used are breaks for additive season and trend (BFAST), Prophet, continuous change detection and classification (CCDC), and Landsat-based detection of trends in disturbance and recovery (LandTrendR). The experiments show that the BFAST outperformed the other three models in forest fire disturbance detection from MOD13A2 time series, with the successful-detection-proportion rate of 96.2% with the benchmark dataset. The detection effect of the Prophet model is not as good as that of BFAST, but it also performs well, with the successful-detection-proportion rate of 87.9%. The detection results of CCDC and LandTrendR are similar, and the detection success rate is lower than that of BFAST and Prophet, but their detection results can be used as data support for labeling work. However, to apply them perfectly to MOD13A2 time series change detection, it is best to do some model adaptation. In summary, the CUG-FFireMCD1 data were verified using different types of time series change detection models, and the change points we marked are credible. The CUG-FFireMCD1 will surely provide a reliable benchmark for model optimization and the accuracy verification of remote sensing time series change detection.

1. Introduction

Remote sensing data provide continuous, quantitative and spatially observations, which can characterize various land-cover types and conversions between them [1,2]. Remote sensing has long played an important role in representing forest disturbances [3]. Among these, detecting forest disturbance is an important research direction in the field of remote sensing change detection. Forest-disturbance monitoring benefits from retrospective analyses, because they can help scholars to analyze disturbance drivers [4], disturbance patterns [5,6], and post-disturbance restoration [7,8] or restoration dynamics caused by land use [9], forest-policy management [10], and natural disasters [11,12].

Landsat data and MOderate-resolution Imaging Spectroradiometer (MODIS) data provide important data support for time series-level forest disturbance detection. At the same time, various long-term change detection methods for understanding forest disturbance have been proposed. We noticed that, when a change detection model is proposed, scholars are willing to find some true time series [7,13] or simulated time series [9,14] to verify the model’s accuracy and robustness. We believe that this step is essential, but collecting data is time-consuming and labor intensive. Furthermore, sometimes the objectivity of the data cannot be guaranteed. Therefore, a benchmark time series remote sensing change detection dataset is desired, which can provide a reliable benchmark for model optimization and remote sensing time series change detection accuracy verification. The systematic collection of global remote sensing datasets is the key to emerging forest-monitoring projects.

The difficulty of this work is mainly reflected in the collection and labeling of datasets. First, in terms of forest ecosystems, the balance of undisturbed and disturbed forest data is unevenly distributed, with disturbed forests far less abundant than undisturbed forests. Therefore, the historical reference forest disturbance data is limited, which makes it difficult to collect forest disturbance data [15,16]. Second, in the detection of forest disturbance, seasonal changes (differences caused by solar angle differences and changes in vegetation phenology) are not considered, and some obvious changes (forest fires, deforestation) can be manually annotated. However, for small and insignificant changes (infestations of insects, etc.), or due to other noise effects such as cloud cover and cloud shadows, manual annotation becomes unreliable.

In this study, we aimed to establish a benchmark dataset for time series remote sensing change detection, named CUG-FFireMCD1. It contains a total of 132 MOD13A2 time series, and each time series contains at least one fire disturbance. The occurrence time for a forest fire disturbance was determined using the National Cryosphere Desert Data Center(NCDC) website, and the precise latitude and longitude coordinates were determined using the FireCCI51 dataset. In addition, except for the known fire disturbance in each time series, we selected four commonly used time series change detection models for other types of change detection, so as to mark as many of the change points as possible. The four models we used were Breaks for additive season and trend (BFAST), Prophet, Continuous change detection and classification (CCDC), and Landsat-based detection of trends in disturbance and recovery (LandTrendR). We choose two change detection models based on Landsat data (CCDC and LandTrendR) because we found that they also have good adaptability to MODIS in experiments.

Our contributions are mainly reflected in the following aspects:

• We establish a MODIS-based benchmark dataset for forest fire disturbance detection (CUG-FFireMCD1).
• We improve the previous work [9], redefine the position of the change point according to the fitting principle of Prophet, and achieve a good detection ability.
• We compare the detection effects of the four models through the CUG-FFireMCD1 dataset, and verify and analyze their detection advantages and limitations. Additionally, we also demonstrate the adaptability of standard CCDC and LandTrendR models to MOD13A2 data.

2. Method

2.1. Data Collection

For a long time, satellite remote sensing has been a very effective data source for the spatiotemporal detection and monitoring of vegetation dynamics [17]. For example, the development of satellite data-cloud-processing platforms, such as Google Earth Engine (GEE), provides users with a convenient and efficient way to obtain and construct time series data on demand [18,19].

The data source we selected for CUG-FFireMCD1 is the MOD13A2 product, which provides Vegetation Index (VI) values at a per pixel basis at 1 km spatial resolution. There are two primary vegetation layers. The first is the Normalized Difference Vegetation Index (NDVI). The second vegetation layer is the Enhanced Vegetation Index (EVI), which has improved sensitivity over high biomass regions. The algorithm for this product chooses the best available pixel value from all the acquisitions from the 16-day period [20]. Compared with NDVI, EVI improves atmospheric noise, soil background, saturation, seasonal expression and other issues [21,22]. Moreover, EVI has better change detection performance than NDVI in densely vegetated areas (such as forests) [23,24].

Currently, many remote sensing phenology research has focused on extracting phenological indicators from the time series of EVI [25,26], because they represent changes in the photosynthetic capacity of vegetation images. Therefore, based on the consideration of less cloud noise, the MOD13A2 product is a suitable choice. In recent decades, MOD13A2 data have been successfully applied to quantify vegetation activities and detect vegetation dynamics in many biological communities [27,28,29,30,31].

We collected natural disasters caused by fires since 2000 from the NCDC website (http://disaster.casnw.net/#/root/view, accessed on 15 March 2022) and recorded the following three parameters: (1) the city districts and counties in which the fire occurred; (2) the coordinates of the fire; (3) the time of the fire. To determine the location of the fire, we used MODIS FireCCI51 data (https://climate.esa.int/en/projects/fire/, accessed on 15 March 2022, https://developers.google.com/earth-engine/datasets/catalog/ESA_CCI_FireCCI_5_1, accessed on 15 March 2022) which represent a map of the global burned area produced by the European Space Agency (ESA) Fire Interference Climate Change Initiative (CCI) project from satellite observations. FireCCI51 is a monthly global 250 m spatial resolution dataset containing information on burned area as well as ancillary data. It is based on surface reflectance in the Near Infrared (NIR) band from the MODIS instrument onboard the Terra satellite, as well as active fire information from the same sensor of the Terra and Aqua satellites [32,33,34]. We use FireCCI51 data to pinpoint the location and time of the fire disturbance, and then use GEE to randomly select MOD13A2-EVI data within the fire zone. To ensure sufficient time series length to characterize forest fires and recovery, we set the time series length to six years (138 observations). We collected forest fire disturbances at different time points around the world, as shown in Figure 1.

CUG-FFireMCD1 has 132 time series, each of which corresponds to a real fire event, is randomly selected in the fire area, and has at least one stable change point. However, we still faced some problems. We could determine the exact date of the disturbance in the forest based on the event, but the disturbance points of other unknown events still needed to be marked. There might have been multiple hidden change points in a time series. Other label points were determined using four detection models, and this stage ensured the best accuracy of the label. Therefore, we set the labels of the dataset to two types: a reliable label and a label predicted by the models, which was for reference.

Once the fire is out, if there is no change in land-cover, the forest begins to enter the recovery phase, and the plant community begins to re-succeed [35]. Although fires of different sizes have different impacts on the forest ecosystem [36], they all cause drastic changes in the spectral and texture characteristics of the forest canopy [37]. From the perspective of spectral characteristics, after a forest fire occurs, the steady-state canopy spectral trajectory shows abnormal pattern changes, usually a large decline, and then gradually returns to stability over time. Thus, we divided the time series into three categories according to the change patterns of the forest EVI curve caused by the fire, as shown in Figure 2.

• Type1: Once the forest disturbance occurs, the EVI value drops to a low level, and there is no evident recovery trend in the short term. There are 66 such time series.
• Type2: The EVI at the time of the fire does not significantly decrease, but the EVI in the second year is significantly lower than that in the previous year. Such change points are mostly due to winter fires, when vegetation is less and EVI changes are not obvious, but they significantly affect the EVI in the second year. There are 48 such time series.
• Type3: The EVI is significantly reduced at the moment of fire, but in the following period, the EVI recovers quickly, and there is no evident change pattern, so these change points can easily be regarded as noise. There are 18 such time series.

Our dataset is publicly available at Github (https://github.com/CUG-BEODL/CUG-FFireMCD, accessed on 15 March 2022), which includes the data, labels, and location coordinates of each time series for scholars to use. We hope that, through CUG-FFireMCD1, scholars will be able to determine a change detection model suitable for large-scale forest fires and effectively verify the effectiveness of the research model.

2.2. Benchmark Models

2.2.1. BFAST

The BFAST model is a seasonal and trend additive interruption method that can identify long-term trends and abrupt changes in the time series and, at the same time, clearly explain the seasonal components. The BFAST model decomposes time series data into three components—trend, seasonality, and noise—to reduce the influence of noise and seasonal trends on the change detection results [13]. Therefore, BFAST can detect all three changes in the ecosystem (seasonal, gradual, and abrupt) [27]. It can also detect mutations and gradual changes well without time. The change detection of forest fire can be realized using the trend components. BFAST uses the full length of time series to model, can detect changes in different time periods, can accurately capture the forest disturbance caused by the fire, and has high detection accuracy. The BFAST model is applicable to a wide range of scenarios, not limited to changes in forest-disturbance [38,39,40,41], and can achieve good detection results without major parameter changes for different land-cover [42,43,44].

In the past few years, BFAST has been successfully adopted and verified in many studies on various ecosystems. Devries et al. [45] applied this model to Landsat time series change detection, which can detect small-scale forest disturbances, with the overall detection accuracy reaching 78%. Chen et al. [46] applied BFAST to detect the spatiotemporal abrupt changes in land and water in the wetland ecosystem of Poyang Lake (the largest freshwater lake in China), located in the subtropical monsoon region. In their study, the overall accuracy of BFAST for detecting major mutations was 85.8%. However, this model cannot detect forest restoration and repeated disturbance events after disturbance [12]. Moreover, it cannot identify and understand the type of change [9], and experiments have proved that the signal-to-noise ratio affects the detection ability of BFAST [13].

BFAST was proposed to solve the change detection of multi-day composite MODIS data, which is a effective change detection model for the CUG-FFireMCD1. We assign the same BFAST parameters to each data in all datasets: (1) Fourier curve fitting with third-order selection for seasonal model. (2) The Ordinary Least Squares (OLS) residuals-based MOving SUM (MOSUM), a generalized fluctuation test threshold is 0.05, if the result is greater than this value, it means that there are still undetected breakpoints in the trend component [47]. (3) The minimum segment width as fraction of input length is 0.17 based on 23 observations in one year divided by 138 observations in six years. This means that if two changes occur within a year, only the most significant change will be detected [13].

2.2.2. Prophet

The Prophet model is a modular regression model with interpretable parameters [48]. It decomposes each time series into three main components: trend, seasonality, and holiday:

$$y\left(t\right)=g\left(t\right)+s\left(t\right)+h\left(t\right)+{ϵ}_t$$

(1)

where $g\left(t\right)$ is the trend function, which is the value of the nonperiodical change in the model time series; $s\left(t\right)$ is the cyclical change; $h\left(t\right)$ represents the holiday item, which represents the impact of those potential holidays with a non-fixed period in the time series on the model fitting. The error term ${ϵ}_t$ represents any special changes to which the model does not adapt, and it is assumed to be normally distributed. Prophet is based on the principle of time series forecasting for time series change detection. It is highly sensitive to missing data and trend changes and can usually handle outliers well.

Prophet is also good at handling daily periodic data with large outliers and trend changes [49,50]. However, it can only detect abnormal outliers and cannot accurately locate the change points in time series with special change patterns. Yan [9] applied the Prophet model to land-cover change, and used the dynamic time warping (DTW) model to classify the detected sub-time-series, which has application value.

In the experiment, we made some modifications to the previous work [9]. In Yan’s work, when the change rate was greater than two, it was determined as a time series change point. However, we found that the fixed change threshold determination method was unreliable and tended to miss many actual change points. By default, Prophet specifies 25 potential change points, which are uniformly placed in the first 80% of the time series [48].

In this study, we made the following adjustments to previous work, and the parameters mentioned below are based on the python version of Prophet (https://github.com/facebook/prophet, accessed on 15 March 2022):

1.

We allowed detecting of change points on the entire input range (changepoint_range is set to 1) [51].

2.

We assign potential changepoints to each moment (n_changepoints is set to 138). This modification can make the Prophet model sensitive to changes at each moment, which is helpful for accurately detecting forest fire disturbance moments.

3.

The change rate of the trend component conforms to Laplace distribution, and changepoint_prior_scale is the scale parameter, so increasing this parameter can make the curve fit better. We found that changepoint_prior_scale can achieve good detection results between 0.5 and 1. We finally set the parameter to 0.8.

4.

The trend term components of the Prophet model are smooth and slowly changing, so these are not directly applicable to the problem of identifying changes [13]. To solve this problem, we extract the K intervals with the largest local change rate (K = 3 in the experiment) from the change rate of the trend component, and select the most front breakpoint in each interval as the identified change point [48].

Experiments show that this adjustment can solve the problem of too much detection or no change point detection. (b) Accurately locate the moment of occurrence of fire disturbance within the allowable error range. Although there are still two parameters changepoint_prior_scale and K that need to be adjusted, the robustness of these parameters are better than the previous method with a fixed threshold set to two. For the detailed parameters of the original Prophet model, please refer to [9].

2.2.3. CCDC

The CCDC model uses all the bands of Landsat satellite imagery [52]. It initializes the model based on 12 cloud-free observations in the sequence of each pixel, and then detects the changes by comparing the difference between the predicted value of the model and the observed value. If this difference in a certain pixel sequence exceeds the threshold three consecutive times, it is judged as a change. The detection result is more comprehensive than using only the quasi-anniversary image, especially in gradual detection.

The CCDC model can detect various land-cover changes, including gradual changes (such as changes caused by vegetation growth and succession, pests, and abnormal climates) and mutations. Zhu et al. [52] compared and analyzed the potential of a simple linear trend and the CCDC model in detecting the greenness trend of urban suburbs. When detecting the greenness trend of land-cover-change areas, CCDC can provide more detailed and accurate information (evaluate gradients and mutations separately), with mapping accuracy and user accuracy of 67.88∼85.19% and 68∼97.30%, respectively. The advantage of CCDC is that it does not rely on empirical parameters and is not affected by noise in the image. However, the computational cost of CCDC is large, and it is not sensitive to certain subtle forest disturbances; moreover, it cannot detect the changes during model initialization [9].

The performance of CCDC for change detection using Landsat data has been extensively verified, but whether the model is suitable for MOD13A2 data is worth further exploring. Therefore, we used a change detection method based on the CCDC principle to verify the CUG-FFireMCD1.

Since the MOD13A2 product has already done the denoising step, and the data lacks some necessary bands support, we did not apply the FMask model [53], only used the CCDC modeling method on the time series, and we only used one-band EVI data, which was also different from the standard CCDC using full-band data. As for the principle of change detection, it is exactly the same as that for the standard CCDC.

2.2.4. LandTrendR

The LandTrendR model was applied to Landsat Thematic Mapper and Enhanced Thematic Mapper (TM and ETM) images of high-quality observations once a year to detect interference events, and it was originally developed using Landsat data and has been recently expanded to the use of MODIS data [54]. This model aims to detect change trends and disturbance events simultaneously. The model uses an arbitrary time-segmentation technology to segment the spectral trajectory, as well as straight-line segments to simulate the important characteristics of the time trajectory. The time and spectral values of the end points of the segmented-line segment provide the basic information required to generate the change graph.

Kennedy et al. [7] applied the LandTrendR at hundreds of locations in western Oregon and Washington state and used annual land satellite time series to detect forest interference and recovery trends. Zhu et al. applied the LandTrendR to the long-term detection of farmland changes near Dongting Lake, helping researchers and managers to better understand the environmental impacts associated with the ongoing conversion work in the area [6]. The advantage of LandTrendR is that it can more comprehensively detect gradual changes and mutation events [7,55], detect forest disturbance and recovery trends, and capture a wide range of disturbance and recovery phenomena [7]. However, LandTrendR needs to design a series of control parameters and filtering processes to reduce the overfitting phenomenon in the time-segmentation process, and the process of capturing the ideal trajectory feature is very complicated, which can also cause some small-scale false-positive changes. The performance in the first and last years was not good [9].

Considering that forest fires generally include four stages of ”before disturbance–disturbance–disturbance recovery–after disturbance”, we set max-segments (the maximum number of segments allowed in fitting) [7] to 4 in the experiment using the LandTrendR model to detect forest fires.

3. Result

In general, the fire-disturbance time judged by the MOD13A2 time series is later than the actual fire-disturbance time, because the MOD13A2 data are a 16-day composite product. We allow an error in the range of two, that is, the detection error is allowed to be controlled within one month before and after the fire disturbance. It is worth mentioning that we checked the detection results, and each time series was successfully detected by at least one change detection model, which means that each time series has value in use. The results are shown in

Table 1. Detection results for the four change detection models for different types of time series.

(a) The number of fire disturbances that can be successfully detected by the four models in the three types of benchmark datasets.
Models		Type1 (66)		Type2 (48)		Type3 (18)		Total (132)
BFAST		66		44		17		127/132 = 0.962
Prophet		63		38		15		116/132 = 0.879
CCDC		53		29		13		95/132 = 0.72
LandTrendR		60		29		8		97/132 = 0.734
(b) Precision, Recall and F1-score of the four models tested on three benchmark datasets.
Models	Type1			Type2			Type3
	Precision	Recall	F1	Precision	Recall	F1	Precision	Recall	F1
BFAST	0.591	1.0	0.743	0.475	0.936	0.63	0.565	0.944	0.707
Prophet	0.346	0.97	0.51	0.305	0.894	0.455	0.296	0.833	0.437
CCDC	0.616	0.803	0.697	0.461	0.617	0.528	0.556	0.778	0.649
LandTrendR	0.422	0.924	0.579	0.284	0.681	0.401	0.204	0.444	0.28

.

Table 1a shows the number of fire disturbances that can be successfully detected by the four models in the three types of benchmark datasets. It can be seen that the BFAST model exhibits the best experimental results, and the overall successful-detection-proportion rate reached 96.2%. This is largely due to the Bayesian Information Criterion (BIC)-based breakpoint location detection algorithm [13], which enables it to accurately detect breakpoints. The Prophet model has a good performance, with an overall successful-detection-proportion rate of 87.9%, which is significantly higher than CCDC and LandTrendR. The performance of CCDC (72%) and LandTrendR (73.4%) is basically the same, and the cases of detection failure are mainly concentrated in the Type2 and Type3 datasets.

It is not comprehensive to judge the applicability of the models solely from the successful-detection-proportion rate. Table 1b shows the Precision (Equation (2)), Recall (Equation (3)) and F1-score (Equation (4)) of the four models tested on three types datasets. n represents the number of data (132), and the confusion matrix is shown in Figure 3. It can be seen that the accuracies of the four models all exhibit low values, as almost all four models detect more than one change point when applied to each time series. Among them, the Prophet and LandTrendR models need to specify the maximum number of breakpoints in advance, so the indicator is lower. These additional change points will provide key metrics as labeled datasets.

$$precision=\frac{1}{n}\sum _{i=1}^n\frac{A_i}{A_i+B_i}$$

(2)

$$recall=\frac{1}{n}\sum _{i=1}^n\frac{A_i}{A_i+C_i}$$

(3)

$$\mathrm{F}1=\frac{2\times precison\times recall}{precison+recall}$$

(4)

4. Discussion

The detection performance of the four models has been widely recognized by scholars. Therefore, in this section we focus on validating and analyzing the limitations of the models on benchmark datasets. We compare the detection results of the four models for each time series, classify the errors and summarize the reasons.

4.1. BFAST

In order to simply show the detection results for BFAST, we have overlaid the trend components on the original time series. In general, there are not many cases of BFAST model detection failures, and the high robustness of the model is achieved under the harsh conditions of fixed parameters. It can capture subtle mutation points (Figure 4a(i)) and gradient trends with insignificant changes (Figure 4a(ii)).

However, there are also some less common problems with BFAST. They are manifested in the following aspects:

• Trend component overfitting (Figure 4b(i)): Although BFAST captures more subtle trend changes, we often only want to focus on global changes. The main reason is the estimation deviation of the optimal number of breakpoints. We compared all the data in the dataset where this occurs, which may be due to the fact that the growth cycle of vegetation is not a year. The overfitting situation makes the detection precision unsatisfactory.
• Detection failure (Figure 4b(ii)): In this case, due to the small scale of the fire, the EVI change is not obvious. However, we can clearly observe that the amplitude of EVI changes after the disturbance, which may be due to the fact that small-scale fires contribute to the regeneration of vegetation [56]. In addition, if the global change trend is not obvious, the BAFST model is likely to ignore some very subtle changes.

4.2. Prophet

As can be seen from Table 1a, the Prophet model performs well on the CUG-FFireMCD1 benchmark dataset. Its overall detection effect is slightly lower than BFAST. By analyzing each piece of data, Prophet can detect subtle changes (Figure 5a(i)), and can solve cases where BFAST detection fails, as shown in Figure 4b(ii) and Figure 5a(ii). This is consistent with the previous conclusion [9]. Moreover, the detection efficiency of Prophet is better than that of BFAST.

In addition, we utilize the ability of the Prophet model to capture the changing direction to detect the trend decline of EVI, which can achieve detection results that exceed BFAST. However, we didn’t do that because it would be unfair to compare models.

Although the accuracy of Prophet is good, some problems are also obvious:

• Difficulty in accurate detection (Figure 5b(i)): Unlike the BFAST, which precisely locates breakpoints through BIC, the Prophet model fits the overall trend of the time series through a smooth and slow curve, and its trend component is similar to the Seasonal-Trend deAcomposition procedure (STL). If the overall time series trend does not change much, only focus on the mutation point. In general, when the rate of change starts to change significantly from a small value (or zero), the change of the time series begins. However, in the application of time series remote sensing change detection, this detection method is sometimes not effective, and some significant change points will be missed or redundant change points will be detected.
• Conditions for missing subtle change points (Figure 5b(ii)): Although Prophet can detect some subtle changes well, when the trend of the time series changes greatly, then some subtle change points cannot be detected.

4.3. CCDC

The detection effect of the CCDC model is numerically inferior to that of BFAST and Prophet, but its detection effect on MODIS data is worthy of recognition. Different from the other three models, CCDC is fitted by continuous observation data and is an online change detection algorithm. Therefore, CCDC can achieve the most timely and accurate disturbance detection, as shown in the Figure 6a. According to statistics, among the 95 pieces of data successfully detected by CCDC, 82 pieces of data can be accurate to the actual time of disturbance. This result is much higher than other models.

The capabilities of CCDC on MOD13A2 data are mainly limited by three aspects:

• Determination of change points: The limited performance of CCDC on MOD13A2 largely depends on the absence of other band information, which results in no additional reference information when determining the change threshold. In addition, since the MOD13A2 data has fewer outliers than the Landsat series data, the periodic characteristics are obvious, and the Fourier transform is easier to fit the curve. The previous threshold determination method may not be suitable for this [52] or need minor adjustments. These two points work together to make CCDC sensitive to changes in MOD13A2 data. As shown in Figure 6b(i), it can be seen that CCDC has detected many unreliable change points.
• Model initialization: 12 clear observations are not difficult for MOD13A2 data, but the continuous advancement of the initial window affects the parameter state of the model. Perturbations during initialization can cause errors in model detection (Figure 6b(ii)), which is improved in the new version of the CCDC model [57].
• In the Type3 dataset, due to the anti-noise ability of the CCDC model, many change points are regarded as noise, as shown in Figure 6b(iii). In the Landsat series data, continuous anomalies less than three times can be regarded as noise, but there is less noise in the MOD13A2 data, which we can reasonably regard as a small-scale disturbance. Therefore, appropriately reducing the RMSE multiple (The Formula (4) in [52]) may make model detection more accurate.

In general, the experimental results for CUG-FFireMCD1 show that CCDC could still be used for the change detection of MOD13A2 time series, although it was not a model specially developed for MODIS time series change detection. If the model parameters can be optimized according to the MODIS time series during use, it may be able to achieve higher time series change detection accuracy.

4.4. LandTrendR

The experimental results show that the LandTrendR model performed well in detecting the Type1 time series dataset. Perhaps this is because the Type1 dataset represents long-term or even permanent changes in land-cover caused by large-scale fires. The piecewise linear fitting algorithm of LandTrendR can fit such changes well and can detect changes in the different of the disturbance well, as shown in Figure 7a.

But on the change detection of Type2 and Type3 datasets, the LandTrendR model exposes some problems:

• The periodic component cannot be extracted (Figure 7b(i)): The standard LandTrendR model only needs one piece of high-quality Landsat data for time series modeling every year and generally does not consider the cyclical changes within the year, but only pays attention to the interannual trend changes of the time series, while the 16-day synthesized MOD13A2 data contain 23 observations per year; that is, the periodic changes during the year are obvious; thus, the LandTrendR model cannot model the annual periodic component, so it is easy to produce large errors in the fitting of the trend item. In addition, uniformly setting the max-segment parameter of the LandTrendR model to a fixed value is another reason for the low detection precision. That is to say, not all the time series include the four stages of “before disturbance—disturbance- -disturbance recovery-t–after disturbance”, so setting unified parameters may cause more false-alarm results.
• Focus only on global changes (Figure 7b(ii)): The fires occurring in winter or the small scale may be other important reasons. In the Type2 dataset, the fire disturbances mostly occurred in winter; such fires had little effect on vegetation growth in the second year. In the Type3 dataset, if the fire duration was short, it was easy to filter out the fire information in the production process for the 16-day MOD13A2 data product. This will be reflected in a small trend change in the vegetation index time series, which can usually be regarded as noise and ignored by the LandTrendR model.

To sum up, LandTrendR is a time series change detection model developed for interannual time series changes and Landsat data, and does not consider intra-annual periodic changes. In addition, the generalization ability of the LandTrendR is slightly poor, resulting in lower change point detection accuracy for the CUG-FFireMCD1 dataset. Therefore, LandTrendR may be not the best model for MOD13A2 time series change detection.

4.5. Confirmation of Other Change Points

In addition to the known fire disturbances, we cannot determine whether there are still other change points in the CUG-FFireMCD1 dataset. Therefore, in order to detect and determine other unknown change points, we formulate the following labeling rules according to the detection effects of the four models:

1.

For the Type1 dataset, the detection effects of the four models all reach a high standard. So if a change point can be detected by at least three models, it is marked as a reliable change point.

2.

For the Type2 dataset, the changes are not obvious due to subtle disturbances. Only the BFAST model can achieve high-accuracy detection. Therefore, if a change point is detected by the BFAST model and detected by other two or more models at the same time reach, it is marked as a reliable change point. The purpose of this is to eliminate too many spurious change points detected by the BFAST model.

3.

For the Type3 dataset, the detection effect of BFAST and Prophet models is generally better than that of CCDC and LandTrendR. So, if a change point is detected by both the BFAST model and the Prophet model, and at least one of the other two models can detect it, it is marked as a reliable change point.

The change points detected according to the above rules are annotated in the final established CUG-FFireMCD1 benchmark dataset. Figure 8 shows part of the data of CUG-FFireMCD1, including the fire disturbance change points and the change points detected by the models.

5. Conclusions

At present, land-cover time series change detection has become an important research topic in remote sensing big data research. However, the remote sensing data collection process is often affected by factors such as cloud cover, resulting in incomplete time series data, and few remote sensing time series data are available. In addition, due to the complexity of land-cover changes, the annotation of change points based on human interpretation is not completely reliable, and the workload is huge. Therefore, change point labeling and benchmark-dataset production based on the classical time series change detection model may be a solution. However, in the production process for the CUG-FFireMCD1 benchmark dataset, we also faced some problems and challenges:

1.

Interpretability. It has to be admitted that the current popular change detection models rarely detect a complete forest-disturbance process in an interpretable fashion. Although it is feasible to divide sub-intervals and perform time series classification through post-processing [9,52], this is a more tedious process. While implementing change detection, it is crucial for the model to be able to achieve an accurate grasp of changes in land-cover types.

2.

Confidence. As mentioned above, neither human interpretation nor models for change detection can determine 100% of the changes. Therefore, it is necessary to calculate reasonable confidence for each change point.

3.

Prior knowledge. Good prior knowledge is essential for the design of the model. In the field of remote sensing, scholars have already accumulated complete prior knowledge. For example, during the experiment, we found a small number of time series data. During the fluctuation process, the EVI value was less than 0, but no model could accurately detect the anomaly, and this is a relatively easy problem for forest-disturbance detection.

4.

No parameters. At present, it is necessary to set different parameters for different time series to make the change detection model work normally, which undoubtedly increases the research workload. Thus, the development of an adaptive parameter model is particularly important.

This paper presents a time series change detection benchmark, CUG-FFireMCD1, which is composed of 132 pieces of MOD13A2 time series. In addition to at least one real fire disturbance, each time series also includes other change points caused by unknown factors. Thus, we adopted four more commonly used time series change detection models—BFAST, Prophet, CCDC and LandTrendR—to label the change points. We tried our best to avoid a situation in which the hidden change points could not be found through manual interpretation. We divided the dataset into three categories according to the type of MOD13A2 time series changes caused by fire, and compared the application scopes of the four models. Based on the analysis of the results obtained, the following conclusions have been formulated:

1.

The detection results of the BFAST model and the Prophet model are the best, and the successful-detection-proportion rate can reach 96.2% and 87.9%. Their advantage is that they can detect some subtle mutations and insignificant gradual changes, but their common problems are easy overfitting and detection delay. The Prophet model is worse than the BFAST model in terms of precise positioning disturbance time.

2.

Although the detection results of the CCDC and LandTrendR models are not as good as the other two models, they also show good detection results. The advantage of the CCDC model is that it can accurately locate the disturbance time, but it cannot focus on the overall time series trend. The advantage of the LandTrendR model is just the opposite. It can focus on a global trend while ignoring many outliers. Overall, some additional work is still needed to make them better adapted to change detection of MODIS data.

In the forest ecosystem, except for the fire, there are other many factors that cause forest disturbances, such as floods and pests. However, due to the lack of accurate disturbance-event records, we could only use fire disturbances to build the time series change detection benchmark dataset. In addition, we only selected four commonly used time series change detection models during the construction of the CUG-FFireMCD1 benchmark dataset. Although we tried our best to accurately and fully annotate the change points contained in the selected MOD13A2 time series, it may not meet the demand for higher-precision time series change detection. Therefore, using other types of disturbance records to further expand the change point types of the CUG-FFireMCD1 benchmark dataset, and to select or develop a higher-accuracy time series change detection model to further improve the accuracy of the benchmark dataset, will be the focus of future research.