TSGBench: Time Series Generation Benchmark

When selecting baselines for comparison, researchers often choose models from their methodological realm, missing a broad performance comparison across various methodologies. For example, GT-GAN (Jeon et al., 2022) and AEC-GAN (Wang et al., 2023) predominantly consider advanced GAN-based methods, overlooking potential contenders like VAE-based approaches. Furthermore, some methods (Pei et al., 2021; Jeha et al., 2021; Zhou et al., 2023) mainly target specific downstream tasks such as missing value imputation or forecasting, hindering the application of the generated time series to other tasks.

First, the dataset choice greatly influences generation outcomes. Different from other time series tasks, the datasets for TSG are diverse due to their lack of label constraints, broadening data source possibilities. Yet, many studies resort to private and/or synthetic datasets for validation (Yoon et al., 2019; Desai et al., 2021; Seyfi et al., 2022). Private datasets, while varied, hinder reproducibility due to their inaccessibility. Moreover, synthetic datasets, albeit straightforward, may oversimplify real-world complexities, risking biased performance evaluations (Yoon et al., 2019; Xu et al., 2020). Second, preprocessing choices significantly impact generation results. For instance, many methods (Yoon et al., 2019; Jarrett et al., 2021; Jeon et al., 2022) use a fixed 24-unit sliding window to segment time series, which often inadequately represent their full periods and temporal patterns. Practices like normalizing time series to a range of $[0,1]$, either before or after applying the sliding window, can lead to discrepancies in results (Yoon et al., 2019). In addition, datasets such as MIMIC III (Johnson et al., 2016) and PhysioNet 2012 (Goldberger et al., 2000), though public, are raw and require meticulous preprocessing to address issues like missing values or anomalies, adding another layer of variability.

Evaluating the generated time series hinges on three foundational principles: diversity, fidelity, and usefulness (Yoon et al., 2019). Diversity gauges how closely the generated series distribution mirrors the original; fidelity examines the similarity between generated and real series; and usefulness assesses the generated series’ practical utility in predictive tasks. The choice of evaluation measures crucially impacts method efficacy determination. Most studies (Yoon et al., 2019; Jeha et al., 2021; Desai et al., 2021), however, choose just a subset of these measures, introducing biases in performance evaluation. In addition to quantitative evaluation for TSG, many methods (Esteban et al., 2017; Jordon et al., 2018) utilize downstream tasks, like classification and forecasting within the “Train on Synthetic, Test on Real” (TSTR) scheme, to showcase utility. These tasks, though popular, come with inherent challenges. For example, some datasets lack classification labels, and short sequences may be unsuitable for forecasting. To counter this, alternative unsupervised approaches (Esteban et al., 2017; Zhou et al., 2023; Fortuin et al., 2019) like interpolation and clustering have emerged, eliminating the need for external labels. However, they may introduce biases due to their dependency on post-hoc models in time series data (Jeon et al., 2022; Lee et al., 2023).

To address these limitations, a comprehensive benchmark for TSG is crucial. It is worth noting that the time series community has introduced a plethora of benchmarks and surveys in areas like databases (Keogh and Kasetty, 2002; Hao et al., 2021), classification (Ismail Fawaz et al., 2019; Bagnall et al., 2018; Dau et al., 2019), forecasting (Makridakis et al., 2018; Bauer et al., 2021; Makridakis et al., 2021), clustering (Rani and Sikka, 2012; Javed et al., 2020), and anomaly detection (Lai et al., 2021; Jacob et al., 2021; Paparrizos et al., 2022b, a). These benchmarks have been pivotal in driving advancements in their respective fields. Nevertheless, there is a noticeable gap when it comes to the field of TSG despite its growing significance. Some notable contributions include Yan et al. (2022)’s framework for benchmarking electronic health record generation and Nikitin et al. (2023)’s open-source tool aimed at enhancing time series and fostering the use of generative models. A recent study (Hang Lou and Ni, 2023) also provides an overview of prevalent evaluation measures for TSG, complemented with an evaluation pipeline. Yet, to our knowledge, a holistic survey or benchmark dedicated to TSG is still absent.

Regarding L1, we establish a taxonomy for systematic comparison and categorization (§3). Specifically, we categorize TSG methods based on three foundational generative models, offering a new perspective for understanding existing research. On this basis, we compare and analyze the model they used, make connections if different methods use similar techniques, and elaborate upon the inheritance and improvement of relative methods.

In terms of L2, we incorporate ten publicly available, real-world datasets from various application domains into TSGBench to enhance reproducibility and mitigate biases or oversimplification in our evaluations. These datasets vary in terms of training sample size, sequence length, and time series dimensions, thereby contributing to a more comprehensive analysis. Moreover, we introduce a standardized pipeline for preprocessing raw time series datasets for TSG (§4.1).

For L3, we provide a suite of twelve evaluation measures tailored for TSG. This suite encompasses five facets, i.e., model-based, feature-based, distance-based measures, training efficiency, and visualization. This allows for a standardized yet thorough comparison (§4.2). Further, we introduce a novel generalization test based on Domain Adaptation (DA) to evaluate the generalization capabilities of TSG methods. Notably, we define three scenarios (i.e., single DA, cross DA, and reference DA) that align with real-world applications (§4.3).

As part of the benchmark development, we present TSGBench in detail (§5) and have conducted preliminary yet systematic experiments to assess the performance of ten representative TSG methods (§6). The results demonstrate the ability of TSGBench to assess the efficacy of various TSG methods, illuminating their strengths and weaknesses from different perspectives. The generalization test using DA sheds light on domain shift scenarios in TSG. In addition, we provide statistical analysis of method ranking, examining the consistency of different techniques across ten evaluation measures and ten datasets (Figure 1). Our findings underscore the reliability and efficacy of TSGBench in evaluating TSG methods.

We restrict our attention to generation methods designed for general-purpose time series. Although certain methods showcase efficacy in specific domains, they lack the flexibility and adaptability needed for broader applications. Thus, these specialized methods are excluded from our evaluation. Moreover, while fine-tuning hyperparameters could yield superior results for each method, we opted not to engage in such optimization. This decision was made to maintain consistency and fairness in comparisons and to adhere to the resource and time constraints inherent in a comprehensive benchmarking study.

To make a holistic evaluation of different generation methods, we select public time series datasets that are both diverse and representative across application domains. Moreover, as we target the generation methods, we consider purely time series as the input. Thus, edge cases, such as those involving the input of a missing data-position matrix or the causality determined by causal graphs, are not incorporated.

We mainly adhere to the widely accepted “Train on Synthetic, Test on Real” (TSTR) scheme (Esteban et al., 2017; Jordon et al., 2018; Yoon et al., 2019; Pei et al., 2021; Jarrett et al., 2021; Srinivasan and Knottenbelt, 2022; Jeon et al., 2022; Li et al., 2023; Zhou et al., 2023). It assesses the synthetic time series’ relevance for real-world applications. While alternative schemes like “Train on Real, Test on Synthetic” (TRTS) (Smith and Smith, 2020; Seyfi et al., 2022) do exist, their likeness to TSTR and infrequent use render them non-essential for our purposes. Our evaluation suite, illustrated in Figure 4, is meticulously crafted using well-recognized measures from TSG literature. We have omitted measures specific to some TSG studies, like cosine similarity (Li et al., 2022b), due to their limited prevalence for extensive benchmarking. Additionally, TSGBench does not use specific downstream tasks as separate evaluation metrics. This is because the included measures, notably discriminative and predictive scores, cover the main goals of time series downstream tasks, such as classification and forecasting. Furthermore, some tasks, like anomaly detection, require extra data and ground truth, which would complicate the evaluation and surpass the designated scope of TSGBench.

A typical GAN model (Goodfellow et al., 2014), as depicted in Figure 2(a), comprises a generator $G$ and a discriminator $D$. During training, $G$ produces a synthetic time series that $D$ attempts to distinguish from real series. This adversarial cycle continues until $D$ cannot reliably differentiate between the two. For generation, only $G$ is used to create a new time series.

Numerous algorithms employ GAN-based models for TSG. They often incorporate specialized neural networks like RNN, LSTM, and Transformer to capture the sequential and temporal intricacies of time series data (Mogren, 2016; Esteban et al., 2017; Li et al., 2022b; Srinivasan and Knottenbelt, 2022; Yoon et al., 2019; Pei et al., 2021). Some models utilize novel metrics or loss functions for better alignment with specific temporal patterns (Ni et al., 2021; Ni et al., 2020). Others enhance traditional GANs with additional modules, such as extra discriminators, classification layers, error correction, and data augmentation to generate time series with specific temporal attributes (Wang et al., 2023; Seyfi et al., 2022; Jeha et al., 2021). While GAN-based models are effective in generating time series, they can be challenging to train and are often resource and time-intensive (Jeon et al., 2022).

A standard VAE model (Kingma and Welling, 2013; Kingma et al., 2014), as shown in Figure 2(b), contains an encoder and a decoder. The encoder $q_{\theta}(\bm{z}|\bm{s})$ transforms input time series $\bm{s}$ into a latent representation $\bm{z}$, capturing essential features and generating parameters (e.g., mean $\mu$ and variance $\sigma$) that model the inherent uncertainty and variability of the time series. The decoder $p_{\theta}(\hat{\bm{s}}|\bm{z})$ reconstructs time series $\hat{\bm{s}}$ from $\bm{z}$, preserving the model’s capacity to regenerate the temporal patterns. The training phase focuses on minimizing reconstruction loss and the divergence between the learned and a prior standard Gaussian distribution. For generation, the decoder draws from the latent space to create a synthetic time series.

While there are fewer studies on VAE-based methods for TSG compared to GANs, they effectively leverage variational inference to capture the complex temporal aspects of time series data (Desai et al., 2021; Li et al., 2023; Lee et al., 2023). The benefits of VAE models lie in their interpretability and training efficiency. Additionally, the structured and informative latent representations from VAEs can be used for other time series tasks, including representation learning (Fortuin et al., 2019) and imputation (Fortuin et al., 2020).

GANs and VAEs do not directly model the probability density function of time series due to the computational challenge of covering all possible latent representation $\bm{z}$ values. Flow-based models (Dinh et al., 2014, 2016; Kingma and Dhariwal, 2018), as shown in Figure 2(c), address this by using invertible transformations $f$’s to explicitly learn data distributions. They have been increasingly employed in TSG, often utilizing explicit likelihood models or Ordinary Differential Equations (ODEs) (Chen et al., 2018; Rubanova et al., 2019; Kidger et al., 2021; Jeon et al., 2022; Alaa et al., 2021). Their architectures, typically featuring coupling layers, allow for a computable Jacobian determinant and reversibility. This is further enhanced by specific transformation techniques crucial for modeling complex data distributions.

Early efforts (Mogren, 2016; Esteban et al., 2017; Ramponi et al., 2018) blended vanilla GAN architectures from image generation with neural networks like RNN and LSTM tailored for sequential data. Subsequent studies have focused on pioneering techniques to adapt to time series data and boost performance.

• •

  A1: RGAN (Esteban et al., 2017). RGAN is a pioneering work that utilizes the GANs for TSG based on RNNs. It is inspired by the maximum mean discrepancy (Gretton et al., 2006) and tries to measure the statistical difference between generated and practical time series.

• •

  A2: TimeGAN (Yoon et al., 2019). TimeGAN considers temporal dependencies within GANs, by simultaneously learning to encode features, generate representations, and iterate across time. It delivers advanced performance and has become a benchmark model for subsequent methods.

• •

  A3: RTSGAN (Pei et al., 2021). RTSGAN combines an autoencoder into GANs and centers on generating time series with varying lengths and handling missing data.

• •

  A4: COSCI-GAN (Seyfi et al., 2022). COSCI-GAN is devised to explicitly model the complex dynamical patterns across every series, which favors channel/feature relationship preservation.

• •

  A5: AEC-GAN (Wang et al., 2023). AEC-GAN aims to generate long time series with distribution shifts and bias amplification via an error correction module that corrects bias in previously generated data and introduces adversarial samples.

VAE-based methods often exploit variational inference to capture the temporal features effectively. They are generally efficient and have the potential interpretability.

• •

  A6: TimeVAE (Desai et al., 2021). TimeVAE extends VAEs to the general-purpose TSG. It builds on convolution and enhances interpretability through time series decomposition.

• •

  A7: TimeVQVAE (Lee et al., 2023). It employs the STFT to decompose input time series into low-frequency and high-frequency components. Then, it integrates Vector Quantization with VAEs (van den Oord et al., 2017) for enhanced modeling of these components, preserving both the general shape and specific details of the time series.

Recent TSG advancements have explored mixed-type methods, merging flow-based models with techniques like DFT or ODEs, or integrating them with GANs or VAEs.

• •

  A8: Fourier Flows (Alaa et al., 2021). It uses DFT (Oppenheim, 1999) to analyze the time series in the frequency domain and applies a sequence of data-dependent spectral filters to learn their distributions.

• •

  A9: GT-GAN (Jeon et al., 2022). GT-GAN is tailored for dealing with both regular and irregular time series, employing Continuous Time Flow Processes (CTFP) (Deng et al., 2020) for its generator and GRU-ODE for the discriminator;

• •

  A10: LS4 (Zhou et al., 2023). LS4 draws from deep-state space models and incorporates stochastic latent variables to enhance the model’s capacity and leverage the training objectives from VAEs.

While these ten datasets are publicly accessible, few TSG methods have evaluated a majority of them. To circumvent these issues, we introduce a standardized pipeline for preprocessing the raw time series datasets tailored for TSG.

To generate time series in a brief span while preserving meaningful structures, we first follow (Yoon et al., 2019; Jarrett et al., 2021) and segment the long time series $\bm{T}$ into shorter sub-matrices $\{\bm{T}_{1},\bm{T}_{2},\bm{T}_{3},\cdots\}$. With a specified sequence length $l$ and a stride of 1, we convert $\bm{T}$ into $R$ overlapping sub-matrices $\{\bm{T}_{r}\}_{1\leq r\leq R}$, where $R=L-l+1$ and each $\bm{T}_{r}$ has the same $l$. To determine the value of $l$, we employ autocorrelation functions (Parzen, 1963), ensuring that each $\bm{T}_{r}$ encompasses at least one time series period. The time series is then shuffled to approximate an i.i.d. sample distribution (Yoon et al., 2019). To assess the generalization capability of TSG methods, we divide the data into training and testing sets in a 9:1 ratio, allocating a larger portion for training and evaluation as is common in TSG methodology. Additionally, we normalize the dataset to the range of $[0,1]$ to enhance efficiency and numerical stability, resulting in a dataset shape of $(R,l,N)$.

These measures predominantly adhere to the TSTR scheme (Esteban et al., 2017; Jordon et al., 2018). This scheme involves using the synthetically generated series to train a post-hoc neural network, which is subsequently tested on the original time series.

• •

  M1: Discriminative Score (DS) (Yoon et al., 2019). This measure employs a post-hoc time-series classification model with 2-layer GRUs or LSTMs to differentiate between original and generated series (Yoon et al., 2019). Each original series is labeled as real, while the generated series is labeled synthetic. Using these labels, an RNN classifier is trained. The classification error on a test set quantifies the generation model’s fidelity.

• •

  M2: Predictive Score (PS) (Yoon et al., 2019). It involves training a post-hoc time series prediction model on synthetic data (Yoon et al., 2019). Using GRUs or LSTMs, the model predicts either the temporal vectors of each input series for the upcoming steps (Yoon et al., 2019; Jarrett et al., 2021) or the entire vector (Jeon et al., 2022). The model’s performance is then evaluated on the original dataset using the mean absolute error.

• •

  M3: Contextual-FID (C-FID) (Jeha et al., 2021). It extends the concept of Frechet Inception Distance (FID) (Heusel et al., 2017) from image generation to TSG. It quantifies how well the synthetic time series conforms to the local context of the time series. Using the time series embeddings from (Franceschi et al., 2019), it learns embeddings that seamlessly blend with the local context.

These measures are designed to capture inter-series correlations and temporal dependencies, assessing how well the generated time series preserves original characteristics. A key advantage of feature-based measures is their capacity to yield clear and deterministic results, providing an unambiguous assessment of the quality of generated time series.

• •

  M4: Marginal Distribution Difference (MDD) (Ni et al., 2021). This measure computes an empirical histogram for each dimension and time step in the generated series, using the bin centers and widths from the original series. It then calculates the average absolute difference between this histogram and that of the original series across bins, assessing how closely the distributions of the original and generated series align.

• •

  M5: AutoCorrelation Difference (ACD) (Lai et al., 2018). It computes the autocorrelation of both the original and generated time series, then determines their difference (Parzen, 1963; Lai et al., 2018). By contrasting the autocorrelations, we could evaluate how well dependencies are maintained in the generated time series.

• •

  M6: Skewness Difference (SD). Beyond ACF, we also consider the statistical measures (Wang et al., 2023; Hang Lou and Ni, 2023). Skewness is vital for the marginal distribution of a time series, quantifying its distribution asymmetry. Given the mean (standard deviation) of the train time series $\bm{T}_{s}^{tr}$ as $\bm{\mu}_{s}^{tr}$ ($\bm{\sigma}_{s}^{tr}$) and the generated time series $\bm{T}_{s}^{gen}$ as $\bm{\mu}_{s}^{gen}$ ($\bm{\sigma}_{s}^{gen}$), we evaluate the fidelity of $\bm{T}_{s}^{gen}$ by computing the skewness difference between them as:

  (1)

  $$SD=\left|\frac{\mathbb{E}[(\bm{T}_{s}^{gen}-\bm{\mu}_{s}^{gen})^{3}]}{{\bm{\sigma}_{s}^{gen}}^{3}}-\frac{\mathbb{E}[(\bm{T}_{s}^{tr}-\bm{\mu}_{s}^{tr})^{3}]}{{\bm{\sigma}_{s}^{tr}}^{3}}\right|.$$

• •

  M7: Kurtosis Difference (KD). Like skewness, kurtosis assesses the tail behavior of a distribution, revealing extreme deviations from the mean. Using notations from Equation 1, the kurtosis difference between $\bm{T}_{s}^{tr}$ and $\bm{T}_{s}^{gen}$ is calculated as:

  (2)

  $$KD=\left|\frac{\mathbb{E}[(\bm{T}_{s}^{gen}-\bm{\mu}_{s}^{gen})^{4}]}{{\bm{\sigma}_{s}^{gen}}^{4}}-\frac{\mathbb{E}[(\bm{T}_{s}^{tr}-\bm{\mu}_{s}^{tr})^{4}]}{{\bm{\sigma}_{s}^{tr}}^{4}}\right|.$$

Training efficiency is crucial, particularly in cases that demand rapid TSG methods or where computational resources are scarce. However, only a few studies, such as (Desai et al., 2021; Jeon et al., 2022), have been employed for evaluation in this context.

• •

  M8: Training Time. It refers to the wall clock time for training a TSG method. It is a vital measure for evaluating and deploying TSG methods due to economic considerations.

Visualization offers an intuitive and visually interpretive perspective to directly compare and contrast the structures and patterns between the original and generated time series.

• •

  M9: t-SNE (Van der Maaten and Hinton, 2008). It is a prevalent technique for succinctly visualizing the distribution of generated time series compared to the original one within a two-dimensional space.

• •

  M10: Distribution Plot (Jeon et al., 2022). It illuminates the difference between the input and generated time series in terms of density, spread, and central tendency to show how the generated time series closely mirrors the original’s statistics.

As depicted in Figure 4, DS and PS are frequently employed for evaluating TSG methods. However, they can yield unreliable evaluations for several reasons:

1. (1)

   DS and PS rely on the TSTR scheme, necessitating post-hoc model training. The inherent randomness in deep models, such as weight initialization, can cause inconsistent evaluations (Jeon et al., 2022; Lee et al., 2023). While repeated training and averaging of results can mitigate this issue, it is time-intensive.

2. (2)

   These measures often hinge on specific network architectures and configurations, which vary across studies (Yoon et al., 2019; Jeha et al., 2021), complicating the comparisons of different TSG methods.

3. (3)

   They are sensitive to dataset sizes. With smaller datasets, they may not converge effectively due to insufficient training data, compromising their reliability (Desai et al., 2021; Lee et al., 2023).

4. (4)

   The sequence length for TSG is typically short, often just 24-time points, as seen in datasets like Stock and Energy (Yoon et al., 2019). They may not contain enough periodic information to accurately assess the TSG efficacy in real-world scenarios.

In summary, while DS and PS offer specific insights into the quality of generated time series, their lack of robustness, time-intensive nature, and difficulty adapting to varying dataset sizes and sequence lengths make them less suitable for a comprehensive and reliable evaluation of TSG methods.

To mitigate the challenges associated with DS and PS, we propose the incorporation of two distance-based measures to provide an efficient, deterministic evaluation.

• •

  M11: Euclidean Distance (ED). For each original series $\bm{s}^{tr}=(x_{1},\cdots,x_{l})$ and its generated series $\bm{s}^{gen}=(y_{1},\cdots,y_{l})$, $ED=\sqrt{\textstyle\sum_{i=1}^{l}(x_{i}-y_{i})^{2}}$. We take the mean of ED for all series and all samples. Given that the input time series has been preprocessed to fit within the range of $[0,1]$, ED deterministically assesses the similarity between $\bm{s}^{gen}$ and $\bm{s}^{tr}$. It provides a value-wise comparison between the time series.

• •

  M12: Dynamic Time Warping (DTW) (Berndt and Clifford, 1994). Given that ED overlooks alignment, we include DTW to capture the optimal alignment between series regardless of their pace or timing. The alignment facilitated by DTW offers insights into the predictive quality of the generated series. Moreover, as shown by (Shokoohi-Yekta et al., 2017), multi-dimensional DTW (Meert et al., 2020) can enhance downstream classification tasks, serving as a discriminative measure.

Using ED and DTW, we can efficiently and effectively evaluate the quality of generated time series, offering streamlined alternatives to DS and PS with similar evaluation goals.

We introduce a novel generalization test using DA (Alaa et al., 2022) to assess the generalization capabilities of TSG methods on small datasets. While DA has been widely applied in other time series tasks like classification and forecasting (Cai et al., 2021; Wilson et al., 2020; Hu et al., 2020), its application in TSG remains underexplored. This is because typical DA tasks necessitate labels (such as class labels for classification or future values for forecasting) and aim to minimize the distribution shift between source and target domains.

We follow the conventions in DA tasks for time series data (Ragab et al., 2023; Cai et al., 2021; Wilson et al., 2020; Hu et al., 2020), examining examples that range from monitoring patient mortality (Reyes-Ortiz et al., 2012) to recording climate and air quality across cities (Zheng et al., 2015), and assessing different machines in a factory (Cai et al., 2021). The commonality across these examples lies in the time-dependent nature of the data, while the differences come from the unique characteristics inherent to each scenario. To illustrate, we first present a motivating example before offering a formal definition.

In Example 4.1, the time series available are $\bm{T}_{s}^{tr}$, $\bm{T}_{s}^{te}$, and $\bm{T}_{t}^{his}$. To evaluate the generated time series $\bm{T}_{t}^{gen}$, we use a comprehensive time series $\bm{T}_{t}^{gt}$ from the target domain as the ground truth. Since we aim to benchmark TSG methods, we do not explore scenarios like modifying the model’s architectures or hyperparameter tuning. Accordingly, we focus on the following three DA cases.

We extend three datasets HAPT (Reyes-Ortiz et al., 2012), Air (Zheng et al., 2015), and Boiler (Cai et al., 2021) for generalization testing, as they include domain information in line with conventions in time series DA tasks (Ragab et al., 2023; Cai et al., 2021). Specifically, our dataset configurations are as follows:

• •

  HAPT (Reyes-Ortiz et al., 2012). The user is treated as the domain attribute. We randomly select User 14 as the source domain and Users 0, 23, 18, 52, and 20 as the target domains. Our evaluation targets the time series for ‘walking’ as it provides a more concentrated and comparable analysis of each user’s walking pattern.

• •

  Air (Zheng et al., 2015). The city serves as the domain attribute. We randomly chose Tianjin (TJ) as the source domain and selected Beijing (BJ), Guangzhou (GZ), and Shenzhen (SZ) as the target domains. This setup enables us to investigate variations in air quality patterns across different urban settings.

• •

  Boiler (Cai et al., 2021). We employ the boiler machine as the domain attribute. Boiler 1 is randomly chosen as the source domain, with Boilers 2 and 3 serving as the target domains. This configuration allows us to examine operational variations and similarities across different boiler units.

We utilize the evaluation criteria outlined in §4.2 for assessment. DA tasks serve as a valuable lens to scrutinize the generalization capabilities of TSG methods across domains. It is pertinent to mention that for the generalization test, efficient processing of data from the target domain is essential, making the training efficiency of a TSG method a pivotal consideration. Thus, the results from DA can enhance other time series tasks by providing synthetic data for target domains with limited sample availability.

As evident from the first three rows of Figure 5, TimeVQVAE, TimeVAE, RTSGAN, and COSCI-GAN consistently excel across the three model-based measures. In contrast, RGAN and TimeGAN underperform. The superior performance of VAE-based methods can be attributed to their adeptness at capturing temporal dependencies, pivotal for forecasting and representation learning tasks. Some methods like LS4, despite having high DS and PS, secure a relatively low C-FID. This indicates their satisfactory performance in the representation learning tasks. Some outliers appear in Energy and Energy Long, where almost all methods achieve DS around 0.5. This suggests while the generated time series might suffice for predictive tasks, they are easily distinguishable in classification tasks. Additionally, the large standard deviation in DS is worth noting, which will be further analyzed in §6.3.

Upon integrating observations from Table 3, we find that as $l$ increases, PS will slightly drop, implying that shorter sequences present a greater challenge in generating time series. Furthermore, a relatively strong correlation appears between $N$ and these measures, suggesting that the generation of high-dimensional time series presents a steeper challenge, leading to higher DS and PS.

In Figure 5, when examining feature-based measures from rows 4 to 7, Fourier Flow delivers the best performance in ACD, while COSCI-GAN dominates in MDD and SD. This is likely due to their effectiveness in capturing the statistical properties of time series. Interestingly, within each dataset, the performance ranking across all four measures appears to be consistent. This consistency implies a robust correlation among these feature-based measures, indicating that the overall performance of a method in capturing key features is likely consistent as well.

Moreover, feature-based measures, which primarily assess the statistical similarities between original and generated data, tend to improve when $N>10$. This observation suggests that generated time series may better match the statistical properties of the original one when dealing with high-dimensional datasets, potentially due to the rich features available for modeling.

The patterns observed in the distance-based measures, shown in the 8th and 9th rows of Figure 5, differ from those in earlier measures. This divergence arises because distance-based measures direct quantification of dissimilarity between the generated and original time series. Regarding ED and DTW, VAE-based methods stand out. They effectively capture the overall trend of the original time series, preserving both value proximity (ED) and trend similarity (DTW).

An intriguing finding is that the distances between generated and authentic time series amplify as $l$ increases. This is likely because longer sequences introduce more complex temporal dependencies, making them harder to model accurately, leading to larger divergence in values and alignment. Conversely, as $N$ grows, these distances decrease. The reason could be that a larger set of series provides a broader range of temporal patterns and inter-series correlations, benefiting the methods’ evaluation.

In the final row of Figure 5, we explore the training efficiency. For ease of interpretation, we categorize training times into four distinct segments: $<1$ minute, $<1$ hour, $<1$ day, and $\geq 1$ day. Our findings highlight the exceptional efficiency of TimeVAE and LS4, likely due to their VAE-based structure and efficient training strategies that minimize computational demands and iterations needed for convergence. On the other hand, TimeVQVAE, while effective in modeling, presents a more time-intensive training process. The complexity of TimeVQVAE arises primarily from its STFT process, tokenization, and iterative decoding. Also, our fairness-driven decision to maintain consistent hyper-parameters across all datasets potentially escalates training times for those with larger $R$ values. In addition, GAN-based methods generally manifest longer training times. For example, GT-GAN takes training time of more than 1 day on all datasets except Stock, DLG, and Exchange. The inherent intricacy of GANs, requiring concurrent training of generator and discriminator networks to reach equilibrium, leads to a longer convergence period.

At last, we look into the t-SNE visualization and Distribution Plot of the generated time series in Figure 6. VAE-based methods, COSCI-GAN, and RTSGAN excel at generating time series that closely mirror the features and patterns of the original ones. Examining the subtleties among these leading methods reveals variations in their generation capabilities for different distributions. For instance, the DLG dataset, characterized by its bimodal distribution, challenges COSCI-GAN, which struggles to capture both modes accurately. Conversely, TimeVAE and LS4 perform well with the Exchange dataset with the multifaceted peak structure, indicating their innate ability to grasp multifarious distributional patterns.

Some methods, such as RGAN and GT-GAN, which might fare well on a single data distribution, cannot deal with the dramatic shift in the overall distributions (e.g., from Stock to HAPT). This suggests limitations in their ability to adapt to significantly different data distributions. We also find that some methods, e.g., RGAN, while they successfully mimic the distribution of the original time series in their generated output (e.g., Energy), stumble when it comes to the precise matching by t-SNE. This observation aligns with the quantitative results in Figure 5, underscoring the challenges in replicating exact values of the original time series rather than overall distribution characteristics. Moreover, some methods, like TimeGAN, can partially fit the original time series but usually contain extraneous information. They may struggle to handle the inherent noise present in real-world time series.

We initially present the ranking of ten methods under two specific scenarios. First, we evaluate their performance across all datasets for each individual measure, as depicted on the left-hand side of Figure 1. Second, we examine their average ranking across all measures but constricted to each dataset included in TSGBench, as showcased on the right-hand side of Figure 1.

Across all evaluation measures and datasets, no single method consistently dominates, but TimeVQVAE, TimeVAE, COSCI-GAN, RTSGAN, and LS4 often outperform others. Specifically, TimeVAE and LS4 excel in distance-based measures and demonstrate impressive training efficiency, while COSCI-GAN and RTSGAN lead in model-based measures. In contrast, RGAN generally ranks lower in performance. A dataset-centric analysis reveals a similar pattern, with the same group of TimeVQVAE, TimeVAE, COSCI-GAN, RTSGAN, and LS4 achieving high rankings across various datasets. This consistency across different evaluation aspects further validates the robustness and reliability of TSGBench.

To statistically confirm the method ranking, we employ the Friedman test (Friedman, 1937) along with Conover’s test (Conover and Iman, 1979; Terpilowski, 2019) for ranking comparisons. Figure 8 presents the average rankings and the critical difference for each method.

The ten TSG methods can be segregated into four tiers. TimeVQVAE, TimeVAE, COSCI-GAN, LS4, and RTSGAN lead, followed by Fourier Flow, AEC-GAN, and TimeGAN. GT-GAN forms the third cohort, while RGAN occupies the lowest tier. This aligns with the observations from §6.1. The supremacy of the first-tier methods becomes unambiguously clear as they stand statistically distinct from their counterparts spanning the other three tiers. Zooming into this premier group, despite the noticeable lead of TimeVQVAE and TimeVAE, they do not statistically tower over their peers within the group. This statistical overlap is not confined to the elite group but is also exhibited in the second group. Such an overlap indicates that the performance deltas between methods within these groups are not glaringly vast. Conversely, the third and fourth groups’ distinct position underscores a pronounced performance chasm, setting it distinctly apart from others.

TSGBench serves as a toolkit to effectively discern the most appropriate TSG methods for different datasets. When confronted with a new dataset, an insightful strategy would be to juxtapose the statistical properties and distributions of the new time series datasets against those cataloged in TSGBench. This strategy offers a navigational compass, pointing users towards relevant generation techniques.

1. (1)

   As a foundational step, we advocate for users to commence with VAE-based methods (e.g., TimeVAE and LS4). Their consistent leading performance and superior computational efficiency make them go-to choices for initial exploration.

2. (2)

   In applications emphasizing autocorrelation or forecasting, such as predictive maintenance or stock market analysis, the ACD measure becomes crucial. Fourier Flow, which is recognized for maintaining temporal dependencies, is highly suitable for these scenarios. On the other hand, for capturing complex multi-variate relationships in datasets, COSCI-GAN is the recommended choice.

3. (3)

   Subsequent considerations focus on dataset size and domain specificity. For small-sized datasets, RTSGAN and LS4, which excel in single DA, are strong choices. For heterogeneous datasets, or when the goal is to generate time series for a new target domain, TimeVAE and COSCI-GAN stand out for their effectiveness in cross DA.

4. (4)

   Users can further fine-tune their method selection based on specific real-world application needs, which involves identifying the most relevant evaluation measures. In this case, Figure 1 serves as a valuable visual guide.

When evaluating a new TSG method, a comprehensive assessment is essential. Leveraging the features of TSGBench, we offer the following guidelines to facilitate this process. This allows users to tailor their choice of evaluation measures to the specific application requirements.

1. (1)

   For applications where generated series will be used in classification or forecasting, model-based measures are advisable. Nevertheless, considering the robustness issues with DS and PS, we recommend starting with C-FID. Its prowess in gauging fidelity based on representations bestows it with the capability to augment subsequent tasks.

2. (2)

   When the goal is to emphasize the statistical attributes of the dataset, feature-based measures emerge as the preferred option, offering precise insights into the statistical nuances.

3. (3)

   In projects focusing on time series clustering, distance-based metrics assume an elevated importance due to their ability to discern subtle distinctions and similarities within the data.

Thus, users can customize their choice of evaluation measures to align closely with their research goals, ensuring more robust results. Note that not all measures tend to yield uniform outcomes consistently. Users could continually calibrate their evaluations and discern the trade-off between effectiveness and efficiency. This ensures that the chosen measures are not only reflective of the desired outcomes but also optimized for performance.