Epidemics 47 (2024) 100755

Contents lists available at ScienceDirect

Epidemics
journal homepage: www.elsevier.com/locate/epidemics

Nowcasting and forecasting the 2022 U.S. mpox outbreak: Support for
public health decision making and lessons learned
Kelly Charniga a, *, 1, Zachary J. Madewell b, 1, Nina B. Masters c, Jason Asher d,
Yoshinori Nakazawa a, Ian H. Spicknall e
a
Division of High-Consequence Pathogens and Pathology, National Center for Emerging and Zoonotic Infectious Diseases, CDC, USA
b
Center for Global Health, CDC, USA
c
Epidemic Intelligence Service, CDC, USA
d
Center for Forecasting and Outbreak Analytics, CDC, USA
e
Division of Sexually Transmitted Disease Prevention, National Center for HIV, Viral Hepatitis, STD, & TB Prevention, CDC, USA

A R T I C L E I N F O A B S T R A C T

Keywords: In June of 2022, the U.S. Centers for Disease Control and Prevention (CDC) Mpox Response wanted timely
Monkeypox virus answers to important epidemiological questions which can now be answered more effectively through infectious
Modeling disease modeling. Infectious disease models have shown to be valuable tools for decision making during out­
Epidemic
breaks; however, model complexity often makes communicating the results and limitations of models to decision
Real-time analysis
Situational awareness
makers difficult. We performed nowcasting and forecasting for the 2022 mpox outbreak in the United States
using the R package EpiNow2. We generated nowcasts/forecasts at the national level, by Census region, and for
jurisdictions reporting the greatest number of mpox cases. Modeling results were shared for situational aware­
ness within the CDC Mpox Response and publicly on the CDC website. We retrospectively evaluated forecast
predictions at four key phases (early, exponential growth, peak, and decline) during the outbreak using three
metrics, the weighted interval score, mean absolute error, and prediction interval coverage. We compared the
performance of EpiNow2 with a naïve Bayesian generalized linear model (GLM). The EpiNow2 model had less
probabilistic error than the GLM during every outbreak phase except for the early phase. We share our expe­
riences with an existing tool for nowcasting/forecasting and highlight areas of improvement for the development
of future tools. We also reflect on lessons learned regarding data quality issues and adapting modeling results for
different audiences.

1. Background Control and Prevention, 2022b). Human-to-human MPXV transmission

occurs through close contact with infectious material from skin lesions,
The 2022 mpox (formerly known as monkeypox) outbreak is the first respiratory secretions during prolonged face-to-face contact, and fo­
major infectious disease outbreak since the COVID-19 pandemic and mites, such as linens and bedding (WHO/ECDC, 2022). The 2022 mpox
was declared a Public Health Emergency of International Concern by the outbreak began in May and spread rapidly in non-endemic countries.
World Health Organization on July 23, 2022 (World Health Organiza­ This outbreak was characterized by human-to-human transmission of
tion, 2022). As of April 13, 2023, a total of 86,956 confirmed cases have MPXV through close physical contact (often associated with sexual ac­
been reported in 110 countries and territories (U.S. Centers for Disease tivities) and has disproportionately affected gay, bisexual, and other
Control and Prevention, 2022a). Unlike COVID-19, mpox is a disease men who have sex with men (UK Health Security Agency, 2022).
known to be endemic in West and Central Africa for decades; it is caused During a public health crisis, such as the mpox outbreak, difficult and
by monkeypox virus (MPXV), a zoonotic orthopoxvirus (Beer and Rao, rapid decisions with limited available data are often required (Thomp­
2019). Historically, classical symptoms involved fever, headache, mus­ son et al., 2022). Infectious disease models may assist with informing
cle aches, fatigue, lymphadenopathy, and rash (U.S. Centers for Disease policy and practice by predicting the magnitude and duration of an

* Corresponding author.
E-mail address: [email protected] (K. Charniga).
1
These first authors contributed equally to this article.

https://doi.org/10.1016/j.epidem.2024.100755
Received 4 May 2023; Received in revised form 14 January 2024; Accepted 26 February 2024
Available online 2 March 2024
1755-4365/Published by Elsevier B.V. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
K. Charniga et al. Epidemics 47 (2024) 100755

outbreak or epidemic, evaluating characteristics of pathogen trans­ Democratic Republic of Congo, which reflected largely non-sexual
mission, such as transmissibility, and designing vaccination strategies, household spread (Fenner et al., 1988). We considered these historical
among others (Buchwald et al., 2020; Madewell et al., 2021a). However, parameter estimates as a starting point for early outbreak analysis, using
infectious disease models are often complex, integrating data from them (along with sensitivity analyses) until new estimates were gener­
heterogenous sources with many parameter assumptions that are subject ated. Updated estimates characterized by the mean and standard devi­
to uncertainty. These aspects make it challenging to effectively imple­ ation were needed for the global outbreak given the novel mode of
ment such models and communicate the results and potential limitations transmission. In June 2022, we were able to use an estimated mean
to decision makers, other public health partners, and the general public serial interval (i.e., the period of time between symptom onset in the
(Lloyd-Smith et al., 2015; Madewell et al., 2021b). primary case and symptom onset in the secondary case) of 9.8 days (95%
During the COVID-19 pandemic, the state of the art of outbreak credible interval [CrI]: 5.9 – 21.4) from 17 case pairs reported by the
analysis advanced considerably. Methods and tools for estimating key United Kingdom (UK Health Security Agency, 2022). At that time,
epidemiological parameters, such as the effective reproduction number, symptom onset date was available for most reported cases (Figure S2),
Rt, were developed and shared in real-time (Abbott et al., 2020). and imported cases were still contributing to a high proportion of MPXV
Monitoring Rt, the average number of secondary cases caused by a single transmission. EpiEstim results were considered the most appropriate at
infected individual in a large population, during an outbreak is useful for this stage of the outbreak because this method accounts for imported vs.
assessing transmission dynamics and evaluating the effectiveness of locally acquired cases, has a stable codebase, is widely used, and is
public health measures (e.g., vaccination, contact tracing, isolation, and computationally efficient (Nash et al., 2022).
quarantine) (Cori et al., 2013). Nowcasts and forecasts have been pro­ In July 2022, we started exclusively using EpiNow2, which uses a
duced by numerous research groups around the globe (Cramer et al., similar approach as EpiEstim (a branching process model, Supplemen­
2022; Arslan et al., 2021; Birrell et al., 2021; Spreco et al., 2022), the tary methods), but it better accounts for reporting delays and in­
results of which were instrumental for decision makers weighing corporates multiple sources of uncertainty (Abbott et al., 2020); for
possible control measures (Biggerstaff et al., 2021), such as social example, it removes noise associated with weekend effects and uses
distancing measures. Outbreak forecasting predicts specific outcomes (e. random walks for temporal smoothing. Forecasting is supported inter­
g., number of cases, deaths, or hospitalizations) at some specific future nally for Rt, number of infections, cases by date of report, and growth
times (e.g., weeks, months, etc.), whereas nowcasting estimates those rate. Unlike EpiEstim, EpiNow2 does not distinguish between imported
outcomes for the current time, accounting for delays in reporting. vs. locally acquired infections. EpiNow2 is the most computationally
In this manuscript, we share our experience nowcasting and fore­ expensive approach, requiring longer model run times (Supplementary
casting the mpox outbreak, including adapting the modeling output to methods). The model assumes that testing procedures, surveillance
different audiences. We also describe challenges faced vis-a-vis data effort, and reporting delays remain constant over the estimation period.
quality, parameter estimation, and model application and propose ways To use EpiNow2, cases by date of report must be provided as well as the
to improve nowcasting and short-term forecasting efforts for future generation time distribution (the time between infection of a primary
outbreaks. and secondary case), incubation period distribution (the time between
infection and symptom onset in a case), and any other delay distribu­
2. Methods tions (e.g., the delay between symptom onset and report date). The
model estimates the number of new cases by date of report, number of
2.1. Nowcasting/forecasting the mpox outbreak cases by their date of infection, Rt, and time-varying growth rate. Esti­
mates over the last 16 days of the time-series are based on partial data
We used data on probable and confirmed mpox cases in the United due to the presumption of reporting delays. Input parameters and
States (see “Case definition” in Supplementary methods) reported to the methods for adjusting for right-truncation evolved as we learned more
U.S. Centers for Disease Control and Prevention (CDC) by state and local about the outbreak. We performed subnational analyses for jurisdictions
public health jurisdictions from May 17, 2022, through March 16, 2023. with the largest case numbers (e.g., New York City, California, and Il­
Data were submitted in several different formats throughout the linois) as well as the four U.S. Census regions (i.e., Northeast, South,
outbreak period. These formats included: a CDC-operated call center Midwest, West).
through the Emergency Operations Center (EOC), a long and a short case
report form (CRF), and the National Notifiable Diseases Surveillance 2.2. Communication methods
System (NNDSS). Cases could have data submitted via more than one
format and jurisdictions could update data on cases after initial sub­ Rt estimates were shared internally through Situational Reports and
mission (Supplementary methods). All reported data were processed in leadership meetings and publicly through CDC’s Technical Reports (U.S.
CDC’s Data Collation and Integration for Public Health Event Response Centers for Disease Control and Prevention, 2022c) and CDC’s
(DCIPHER) platform, an instance of Palantir Foundry (Palantir Tech­ public-facing mpox website (U.S. Centers for Disease Control and Pre­
nologies Inc, Denver, CO). DCIPHER is a secure, cloud-based data inte­ vention, 2022d). The Technical Reports were co-led by the Center for
gration, analytics, and situational awareness platform used by CDC, Forecasting and Outbreak Analytics (CFA) and the 2022 Multi-National
federal partners, and state, tribal, local, and territorial public health Mpox Outbreak Response. Estimates were generated for distribution at
jurisdictions to collect, collaborate on, and share public health data (U.S. least once per week. The technical reports were intended for scientific
Department of Health and Human Services, 2020). DCIPHER collates audiences. The purpose of sharing these results was to improve under­
data of differing origin, structure, and purpose to provide near real-time standing of the outbreak and inform further scientific inquiry.
insights into public health problems, with the goal of providing a com­
plete picture of situational awareness. 2.3. Performance assessment methods
We considered three approaches for estimating Rt which are imple­
mented in the R packages EpiEstim (version 2.2–4) (Cori et al., 2021), We chose eight dates during four key outbreak phases to retrospec­
earlyR (version 0.0.5) (Jombart et al., 2020), and EpiNow2 (version tively evaluate our short-term (one-week-ahead) forecasts of reported
1.3.2) (Abbott et al., 2021a) (Supplementary methods). Initially, we mpox cases generated from EpiNow2: 1. one month into the outbreak
used all three methods to estimate Rt at the national level as well as for prior to exponential growth (June 13 and June 27); 2. during expo­
jurisdictions reporting the highest incidence of mpox. Although esti­ nential growth (July 5); 3. near the outbreak peak (July 27); and 4.
mates of the historical range of the serial interval of mpox were available during the declining phase (September 6, September 19, October 11, and
at the start of the outbreak, they were based on data from the December 5). We used the dataset available at the time of each key time

2
K. Charniga et al. Epidemics 47 (2024) 100755

point to produce the forecasts and not the final case numbers after the reported case data in near real-time whereas others submitted a large
outbreak ended, which are used as the ground truth for evaluation. number of cases all at once, the latter of which caused large, artificial
Ideally, the same day of the week would be used, but some historical spikes in the time-series. There were instances of duplicate cases being
versions of the dataset were not available for this analysis and some reported from several jurisdictions which may be attributed to the
dates fell on national holidays which may have introduced additional changes in reporting processes. Spurious cases at the end of the time
delays. Like the real-time analyses, we used rash onset date as the first series had to be investigated (and usually removed) because they arti­
reference date to define the reporting delay distribution for all eight time ficially inflated the nowcasts/forecasts. These data issues required us to
points, while the second reference date changed over time (Table S1). monitor the model output closely and modify the code as needed.
We used three metrics to evaluate the forecasts. Our primary metric In early July 2022, reporting of mpox cases to CDC started to lag in
was the weighted interval score (WIS). For each of the eight time points some jurisdictions, especially those most affected by the outbreak. These
considered, WIS was computed for each daily prediction and averaged few jurisdictions were publicly reporting more cases on their websites
across the seven-day forecast. The WIS measures the consistency of a than what CDC had received reports for. This led to a lengthy case
group of prediction intervals with an observed value (probabilistic ac­ reconciliation process during which case data uncertainty prevented it
curacy). The WIS is positive, and lower values correspond to smaller from being used for nowcasting/forecasting at the national level. Also in
error (Supplementary methods) (Cramer et al., 2022). To evaluate the July, an increasing proportion of cases were reported with missing
error in the forecast’s point estimate, we used the mean absolute error symptom onset dates (from 26% to 53% for rash onset date between
∑ ⃒ ⃒
(MAE), which was computed as MAE = N1 Ni=1 ⃒yi − ̂ y i ⃒, where yi is the June 13 and July 5). To ensure each case had a date associated with it for
observed number of mpox cases on day i, ̂ y i is the median forecast on day plotting epidemic curves, a new event date field was created which we
i, and N = 7 (Cramer et al., 2022). We also used prediction interval (PI) started using for nowcasting/forecasting. The new calculated date field
coverage rates, which check the degree to which the model provides selected the best available date among possible date fields based on the
calibrated predictions. Coverage rates are calculated by determining the following priority, ordered from most to least preferable: orthopoxvirus
proportion of times the 50% or 90% PIs included the observed value (for test date, date of call to the call center, date the short CRF was created,
example, a well-calibrated forecast would have a 50% PI coverage close and the long CRF timestamp (Figure S1). The date in which the record
to 0.50. Also see Supplementary methods) (Cramer et al., 2022). was created was least preferred due to artificial spikes in the time series
We compared the performance of EpiNow2 with a naïve Bayesian caused by bulk uploading data. In September 2022, a different date was
generalized linear model (GLM, Supplementary methods). We calcu­ adopted by the response for reporting case data. This date field was
lated the relative WIS and relative MAE for EpiNow2 and the GLM as defined as the earliest among all available dates for a case including
mean score of EpiNow2 symptom onset, which facilitated improved visualization of epidemic
θEpiNow2,GLM = mean score of GLM , where the mean score is the average of
curves. However, this new date presented challenges for its use in the
the models’ performance (WIS or MAE) over all eight dates evaluated. If EpiNow2 framework because the delay from symptom onset to this new
θEpiNow2,GLM was less than 1, that indicated the forecasts generated by date would have more variation than the delay using the original event
EpiNow2 had less error than the GLM, whereas θEpiNow2,GLM > 1 indicated date, including a delay of zero for some cases. Thus, we worked to create
EpiNow2 performed worse. a new event date field specifically for nowcasting/forecasting which was
For both EpiNow2 and the GLM, we removed recent cases (defined as similar to the original event date. The definition was expanded to
cases reported in the last 3 – 5 days) from the time series to adjust for include dates available in NNDSS data.
right truncation of the data for all four outbreak phases (Table S1,
supplementary methods). 3.2. Successes of nowcasting/forecasting
Forecasts were evaluated using mpox data as of March 16, 2023. We
used the most recent version of event date as the basis for the In October, we updated estimates of the serial interval for rash onset
comparison. of 7.0 days (95% CrI 5.8 – 8.4) from 40 case pairs and incubation period
for rash onset of 7.5 days (95% CrI 6.0 – 9.8) from 35 U.S. case-patients
3. Results and used those as model inputs for EpiNow2 (Madewell et al., 2023).
These data were obtained through the collaboration of several U.S. ju­
3.1. Challenges of nowcasting/forecasting risdictions on a special study. The estimated serial interval for the 2022
outbreak was on the lower end of the historical range observed in the
It is voluntary for jurisdictions to report mpox cases to CDC, with Democratic Republic of Congo (7 – 23 days) (Fenner et al., 1988).
only minimal data needed to submit a case report form (e.g., case ID and
reporting jurisdiction) in part, because not all cases may be reached or 3.3. Adapting model output and communicating nowcasts/forecasts
fully investigated. CDC asks jurisdictions to collect and report additional
data variables to achieve situational awareness and surveillance goals. We adapted the presentation of our results for a scientific/technical
The number of variables on the case report form was decreased to reduce audience and the general public. The default plots from EpiNow2
reporting burden. Despite these efforts, jurisdictional case surveillance included three panels: cases by date of report, cases by date of infection,
systems may not have aligned to CDC’s requested data variables, and and Rt. For the Technical Reports, Situational Reports, and response
jurisdictions may choose to limit what data are shared with CDC based updates meetings, we removed the middle panel (cases by date of
on local reporting practices. Received data were subjected to additional infection, Fig. 1) (U.S. Centers for Disease Control and Prevention,
manual data cleaning to standardize formats and correct obvious data 2022b). For the website, we only showed Rt, removed the forecast, and
entry errors. As a result, even key data variables such as demographic removed the 20% credible intervals to minimize confusion (Figure S3)
characteristics (e.g., age, race/ethnicity, HIV status, vaccination) were (U.S. Centers for Disease Control and Prevention, 2022d). We included a
not consistently available across all jurisdictions and time periods, simple description of the plot that could be understood by non-experts.
precluding detailed sub-analyses. For example, out of 29,921 cases in In accordance with CDC’s Data Modernization Initiative, a national
DCIPHER through December 31, 2022, 21,480 (71.8%) were missing effort aimed at modernizing state and national core data and surveil­
HIV status, 16,474 (55.1%) were missing smallpox vaccination, 3913 lance infrastructure (U.S. Centers for Disease Control and Prevention,
(13.1%) were missing gender identity, 3172 (10.6%) were missing race, 2022e), data for the underlying plots were made available for download
2928 (9.8%) were missing ethnicity, and 250 (0.8%) were missing age. as comma-separated values (csv) files with the Technical Reports.
The timing and frequency of data submission varied between jurisdic­ Subnational analyses revealed some differences between regions
tions and changed over the course of the outbreak. Some jurisdictions regarding the start of the outbreak, when it peaked, and how long it

3
K. Charniga et al. Epidemics 47 (2024) 100755

Fig. 1. Effective reproduction number estimates for the U.S. 2022 mpox outbreak intended for a technical/scientific audience. The time period used for estimation
includes June 20, 2022 – November 8, 2022 with a 7-day forecast from November 9 – 15. The top panel shows estimates of cases by date of report with actual cases
shown by gray bars. The bottom panel shows estimates of the effective reproduction number by date. In all panels, shaded regions reflect 90%, 50%, and 20%
credible intervals in order from lightest to darkest. Green shows estimates, red shows estimates based on partial data, and purple shows forecasts. Event date is
determined by a hierarchy across the different data streams where priority is given to diagnosis date, orthopoxvirus test date, orthopoxvirus test confirmation date,
case investigation start date, orthopoxvirus sample collection date, date of call to CDC call center, report date (to public health department, county, or state), date
CDC announced case, and the date the case was entered into DCIPHER, in that order.

lasted (Fig. 4 and Figure S4). For example, Fig. 4 demonstrates a later considered, indicating that EpiNow2 had on average 11% less proba­
introduction date and slightly longer tail for Texas compared to other bilistic error than the GLM. Although EpiNow2 projections were overall
jurisdictions. more accurate than those from the naïve model, both models had the
greatest amount of error around the peak.
EpiNow2 had lower MAE than the GLM for six out of eight time
3.4. Performance assessment points, but performance was similar: the relative MAE was 0.96. In other
words, EpiNow2 had only 4% less point error than the GLM.
The GLM had lower WIS compared to EpiNow2 for early phase of the Overall, predictions were moderately well calibrated. For the 90%
outbreak (Table 1); however, during all other phases, EpiNow2 had a PI, EpiNow2 achieved coverage rates within 10% of the desired
slight advantage. The relative WIS was 0.89 over all eight time points

Table 1
Evaluation of short-term (one-week-ahead) forecasts of reported mpox cases during the 2022 U.S. outbreak. For WIS, bold indicates where one model performed better
than the other.
EpiNow2 Bayesian GLM with negative binomial

Forecast date Outbreak phase 90% PI coverage 50% PI coverage WIS MAE 90% PI coverage 50% PI coverage WIS MAE
Monday, June 13 Early 1 0.71 33.8 4.3 0.86 0.57 6.4 5.1
Monday, June 27 Early 0.86 0 34.0 29.0 0.86 0.14 24.7 26.9
Tuesday, July 5 Exponential growth 0.86 0.71 35.6 32.4 1 0.57 38.5 34.1
Wednesday, July 27 Peak 1 0.57 137.0 107.9 1 0.57 184.3 94.4
Tuesday, September 6 Decline 0.86 0.71 107.6 84.3 0.86 0.43 117.1 86.8
Monday, September 19 Decline 0.86 0.43 61.2 52.4 0.71 0.43 81.8 65.9
Tuesday, October 11 Decline 0.43 0.29 30.6 33.4 0.57 0.14 38.3 41.9
Monday, December 5 Decline 1 0.71 4.3 3.1 1 0.71 6.4 4.6

PI: prediction interval; WIS: weighted interval score; MAE: mean absolute error; GLM: generalized linear model

4
K. Charniga et al. Epidemics 47 (2024) 100755

coverage level for seven out of eight time points compared to six out of between jurisdictions in terms of initial seeding, composition (e.g.,
eight for the GLM. For the 50% PI, EpiNow2 achieved coverage rates population age structure, density, and contact patterns), and public
within 10% of the desired coverage level for only two out of eight time health activities (e.g., vaccination, surveillance methods, frequency of
points compared to five out of eight for the GLM. testing) (De Nicola et al., 2022) as well as the timing and frequency of
Qualitative results of the nowcasts/forecasts are shown in Fig. 2. The case reporting. Most early mpox patients in the United States reported
90% CrIs from EpiNow2 were very wide for the early phase of the recent international travel (Minhaj et al., 2022). Although epidemics are
outbreak, while the GLM had large uncertainty around the outbreak’s stochastic, jurisdictions with early outbreaks likely had more seeding
peak. Both models underestimated reported mpox cases for the seventh events due to higher international travel flows compared to jurisdictions
time point on October 11 which could be due to a discrepancy in the data with outbreaks that occurred later (Bhatia et al., 2021; Johansson et al.,
available at the time versus the ground-truth data (Fig. 3). This time 2014). Differences in the magnitude of the epidemics across jurisdictions
point had the lowest PI coverage rates. may have been related to the population size of gay, bisexual, and other
men who have sex with men, with higher population size leading to
4. Discussion larger outbreaks. While public health officials in certain jurisdictions
read the CDC technical reports, and at least one jurisdiction sought to
We performed nowcasting/forecasting to inform the U.S. response to reproduce Rt estimation for their own jurisdiction, we do not know
the 2022 mpox outbreak in real-time. Validation showed that the whether our subnational Rt estimates and case projections informed the
method implemented in EpiNow2 predicted case counts reasonably response for different jurisdictions.
well, but improvements are needed around key time periods such as the One limitation of nowcasting/forecasting at the subregional or
outbreak’s peak. One reason that the nowcasts/forecasts did not always jurisdictional level is that the effects of bulk uploads are more apparent,
align with reality is because the definition of event date changed over resulting in greater uncertainty (wider credible intervals). Another
time, while the study data were constructed the most recent version of limitation is that movement between jurisdictions could have a greater
the event date field. We found a higher WIS for EpiNow2 in the early impact on subnational estimates, as mobility is not accounted for in our
phase of the outbreak compared to the GLM which could be due to approach. Finally, some jurisdictions stopped reporting rash onset date,
choices of priors for parameters (e.g., wide intervals for Rt). which decreased the sample size available for estimating the reporting
Subnational analyses allowed us to better understand the spatial delay distribution over time.
heterogeneity of the epidemic which may be attributed to differences Interpreting changes in Rt over time can be difficult (Cori et al.,

Fig. 2. Retrospective evaluation of one-week ahead nowcasts/forecasts of mpox cases in the United States. The timeseries starts on May 16, 2022. Black
points correspond to the number of cases in the ground truth data, obtained from DCIPHER on March 16, 2023. The 90% and 50% credible intervals are shown in
blue for the EpiNow2 forecasts and green for the GLM forecasts.

5
K. Charniga et al. Epidemics 47 (2024) 100755

Fig. 3. Time series of reported mpox cases in the United States used for nowcasting/forecasting, May 16, 2022 – March 16, 2023. Retrospective evaluation of
nowcasts/forecasts were performed for eight dates in the outbreak. Eight of the colored lines in the plot correspond to snapshots of the data available at the time the
nowcasts/forecasts were made, while the line labeled “gold” corresponds to the ground truth data, obtained from DCIPHER on March 16, 2023.

2013). We were unable to assess the impact of specific interventions (i. performed only after the data have undergone a rigorous and routine
e., vaccination and behavioral adaptation) on Rt as the timing of control reconciliation and closeout process by data submitters with further
measures likely overlapped. Also, decreases in Rt could have resulted validation by CDC surveillance epidemiologists; however, timely
from the depletion of susceptible individuals among those at increased outbreak response decision support does not allow for such processes.
risk of infection (Murayama et al., 2023). Using a dynamic network Instead, jurisdictions are asked to submit available case data in near
transmission model fitted to mpox cases reported in Washington D.C., real-time and submit additional data or corrections to data entry errors
Clay et al. found that cases decreased initially due to behavioral adap­ as time and resources permit. Snapshots of the surveillance data were
tations (i.e., reductions in sexual partner acquisition by gay, bisexual or saved in an ad hoc manner (by exporting data on a particular day and
other men who have sex with men); however, vaccination decreased saving a csv file locally), and as a consequence, a complete history of the
more cases overall and accelerated the end of the outbreak (Clay et al., data is not available, especially around key points in the outbreak, such
2023). as the peak. A complete history would help to understand key delay
Another limitation of our approach is that we assumed constant Rt distributions and other quirks (e.g., backfilling and revision of reference
during the forecasting period. Although this assumption is likely dates) involved in the data-generating process. Understanding the data
reasonable for short-term (i.e., one week ahead) forecasting, forecasts generating process is crucial for the improvement of methods and tools
over longer periods may suffer from considerable error if Rt does not for nowcasting/forecasting and aligns with one of the five priorities of
remain constant. Additionally, the computing time required for the CDC’s Data Modernization Initiative (Accelerating Data for Action:
analyses was very long. While we were able to increase computational Tapping into more data sources, promoting health equity, and
efficiency by running the model on multiple cores in parallel, the pro­ increasing capacities for scalable outbreak response, forecasting, and
cessing time became particularly cumbersome if analyses needed to be predictive analytics) (U.S. Centers for Disease Control and Prevention,
repeated. Finally, the method we used does not account for under- 2022e). In the future, the process of saving snapshots of the data could
ascertainment, and estimates of the reporting rate for the U.S. mpox be automated.
outbreak are lacking. Ensemble models have been used for a variety of infectious disease
outbreaks, such as COVID-19 (Cramer et al., 2022; Dean et al., 2020),
4.1. Data quality Zika (Oidtman et al., 2021), influenza (Reich et al., 2019), and Ebola
(Viboud et al., 2018). Ensembles combine predictions from several
Nowcasting/forecasting methods perform best when the underlying models that use different methodology and sometimes input data.
surveillance data are accurate, timely, and complete, but they are often Because some models overpredict, while others underpredict, ensemble
sub-optimal and variable as the outbreak evolves; while data may models often outperform individual models over time. In the future, we
improve as an outbreak progresses, they may re-deteriorate once the may consider using at least two simpler models and comparing them.
outbreak slows and intensity of effort is low. Fortunately, the quality and One potentially useful addition to EpiNow2 and other currently
frequency of data improved over the course of the U.S. mpox outbreak. available tools for nowcasting/forecasting outbreaks would be flexi­
Communicating with specific jurisdictions about our priority dates for bility in handling dates. We frequently encountered missing dates for
modeling improved data quality. These prompts to the jurisdictions need cases in the mpox surveillance data. Ideally, a method or tool would be
to be continued regularly throughout the outbreak. Close collaboration able to keep track of multiple dates for a case and estimate missing dates
between epidemiologists/modelers and informaticians, including the based on the full distribution of dates across all cases. Epinowcast is a
use of an issue tracking system in DCIPHER, also facilitated quick new hierarchical nowcasting package that enables more flexibility in
investigation and resolution of data errors. adjusting for truncated data (Abbott et al., 2021b). Novel nowcasting
approaches use hierarchical generalized additive models, which can
provide even more flexibility to modify the model in real-time to the
4.2. Strategies for forecasting the next outbreak evolving data environment (Overton et al., 2023). Another improve­
ment would be to reduce the time required to run the analyses. Rather
For the next outbreak, it is important for CDC to develop strategies than focusing on the efficiency of the MCMC algorithm, computation
for regularly capturing and storing snapshots of surveillance data which time could be reduced if the model only needed to be run on the new
remain easily accessible for systematic analysis. For routine case-based data. Finally, the imputed time series of cases by symptom onset date
surveillance of notifiable diseases, such as rabies, most analyses are

6
K. Charniga et al. Epidemics 47 (2024) 100755

Fig. 4. Effective reproduction number estimates of the 2022 mpox outbreak for the six jurisdictions in the U.S. with the highest case counts. The time period used for
estimation includes June 20, 2022 – November 8, 2022 with a 7-day forecast from November 9 – 15. The left panels show estimates of cases by date of report with
actual cases shown by gray bars. The right panels show estimates of the effective reproduction number by date. In all panels, shaded regions reflect 90%, 50%, and
20% credible intervals in order from lightest to darkest. Green shows estimates, red shows estimates based on partial data, and purple shows forecasts. Event date is
determined by a hierarchy across the different data streams where priority is given to diagnosis date, orthopoxvirus test date, orthopoxvirus test confirmation date,
case investigation start date, orthopoxvirus sample collection date, date of call to CDC call center, report date (to public health department, county, or state), date
CDC announced case, and the date the case was entered into DCIPHER, in that order.

would be a useful data visualization output that is not currently avail­ sharing science and data faster (U.S. Centers for Disease Control and
able in EpiNow2. As described above, defining the date field for the Prevention, 2022f).
presentation of epidemic curves was a challenge in the mpox outbreak
and having an imputed symptom onset date for each case would have 5. Conclusion
been useful for comparison purposes.
CFA played an important advisory role in our nowcasting/fore­ Real-time estimation of Rt as well as nowcasting/forecasting is one
casting efforts. CFA produces models and forecasts to characterize the method for determining the extent to which current public health
state of an outbreak and its course, inform public health decision makers measures are effective and/or need to be modified but is subject to
on potential consequences of deploying control measures, and support limitations. The quality and timeliness of reported data pose challenges
innovation to continuously improve the science of outbreak analytics to these analyses. Ease of use, model computing time, and ability to
and modeling (U.S. Centers for Disease Control and Prevention, 2022f). handle multiple dates are priorities for consideration in the development
In the future, CFA plans to create new tools for outbreak analysis and of future nowcasting/forecasting tools. A naïve model may be superior
modeling. CFA could also serve as a link between CDC modelers and to a complex one, such as EpiNow2, during the early phase of an
jurisdictions with modeling capacity to share experiences and code. outbreak when data scarcity causes Rt to be largely unconstrained,
Technical Reports represent a new way for CDC to share timely infor­ especially once reporting delays are considered. Future outbreak
mation with the federal government, state and local leaders, and sci­ response activities could be enhanced through inclusion of clear and
entists in academia and industry. Technical Reports have been well consistent communication about modeling outputs as well as close
received within and outside CDC (Daskalakis, 2022; Fox, 2022; Rivers, collaboration between modeling and informatics/data teams.
2022) and their publication aligns with CDC’s current restructuring ef­
forts aimed at making the agency more response ready, including

7
K. Charniga et al. Epidemics 47 (2024) 100755

Ethics statement Arslan, S., Ozdemir, M., Ucar, A., 2021. Nowcasting and forecasting the spread of
COVID-19 and healthcare demand in turkey, a modeling study. Front Public Health
8, 575145.
This activity was reviewed by CDC and was conducted consistent Beer, E., Rao, V., 2019. A systematic review of the epidemiology of human monkeypox
with applicable federal law and CDC policy (45 C.F.R. part 46, 21 C.F.R. outbreaks and implications for outbreak strategy. PLoS Negl. Trop. Dis. 13 (10),
part 56; 42 U.S.C. Sect. 241(d); 5 U.S.C. Sect. 552a; 44 U.S.C. Sect. 3501 e0007791.
Bhatia, S., Imai, N., Cuomo-Dannenburg, G., Baguelin, M., Boonyasiri, A., Cori, A., et al.,
et seq). 2021. Estimating the number of undetected COVID-19 cases among travellers from
mainland China [version 3; peer review: 3 approved]. Wellcome Open Res. 5, 143.
Funding Biggerstaff, M., Slayton, R., Johansson, M., Butler, J., 2021. Improving pandemic
response: employing mathematical modeling to confront coronavirus disease 2019.
Clin. Infect. Dis. 74 (5), 913–917.
No specific funding was obtained for this work. Birrell, P., Blake, J., van Leeuwen, E., Gent, N., De Angelis, D., 2021. Real-time
nowcasting and forecasting of COVID-19 dynamics in England: the first wave. Philos.
Trans. R. Soc. Lond. B Biol. Sci. 376 (1829), 20200279.
CRediT authorship contribution statement Buchwald, A., Adams, J., Bortz, D., Carlton, E., 2020. Infectious disease transmission
models to predict, evaluate, and improve understanding of COVID-19 trajectory and
Zachary J Madewell: Formal analysis, Investigation, Methodology, interventions. Ann. Am. Thorac. Soc. 17 (10), 1204–1206.
Clay, P., Asher, J., Carnes, N., Copen, C., Delaney, K., Payne, D., et al., 2023. Modelling
Software, Validation, Visualization, Writing – original draft, Writing – the impact of vaccination and sexual behaviour adaptations on mpox cases in the
review & editing. Kelly Charniga: Conceptualization, Data curation, USA during the 2022 outbreak. Sex. Transm. Infect.
Formal analysis, Investigation, Methodology, Project administration, Cori, A., Ferguson, N., Fraser, C., Cauchemez, S., 2013. A new framework and software to
estimate time-varying reproduction numbers during epidemics. Am. J. Epidemiol.
Software, Validation, Visualization, Writing – original draft, Writing –
178 (9), 1505–1512.
review & editing. Ian H Spicknall: Supervision, Writing – review & Cori A., Cauchemez S., Ferguson N., Fraser C., Dahlqwist E., Demarsh P., et al. EpiEstim:
editing. Yoshinori Nakazawa: Conceptualization, Supervision, Writing Estimate Time Varying Reproduction Numbers from Epidemic Curves. 2.2-4 ed.
CRAN2021.
– review & editing. Jason Asher: Data curation, Methodology, Super­
Cramer, E., Ray, E., Lopez, V., Bracher, J., Brennen, A., Rivadeneira, A., et al., 2022.
vision, Writing – review & editing. Nina B Masters: Software, Writing – Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality
review & editing. in the United States. PNAS 119 (15), e2113561119.
Daskalakis D. You know that you are a #publichealth nerd when you wake up on a
gloomy Saturday to read @CDCgov Twitter; 2022 [updated October 1, 2022.
Declaration of Competing Interest [Tweet]]. Available from: https://twitter.com/dr_demetre/status/
1576196874079735808?s=48&t=9NVB0dQPIU9S4J58AJJlfw.
The authors declare the following financial interests/personal re­ De Nicola, G., Schneble, M., Kauermann, G., Berger, U., 2022. Regional now- and
forecasting for data reported with delay: toward surveillance of COVID-19
lationships which may be considered as potential competing interests. infections. Adv. Stat. Anal. 106 (3), 407–426.
Kelly Charniga reports a relationship with Systems Planning and Anal­ Dean, N., Pastore Y Piontti, A., Madewell, Z., Cummings, D., Hitchings, M., Joshi, K.,
ysis Inc that includes: consulting or advisory. et al., 2020. Ensemble forecast modeling for the design of COVID-19 vaccine efficacy
trials. Vaccine 38 (46), 7213–7216.
Fenner F., Henderson D., Arita I., Jezek Z., Ladnyi I. Smallpox and its Eradication1988.
Data Availability Available from: 〈https://apps.who.int/iris/handle/10665/39485〉.
Fox S. These monkeypox technical reports from the CDC are [three fire emojis]: Twitter;
2022 [updated November 8, 2022. [Tweet]]. Available from: 〈https://twitter.co
Data and code to run the nowcasts/forecasts and perform model m/FoxandtheFlu/status/1589978622026461184〉.
validation are available on GitHub (https://github.com/kcharniga/ Johansson, M., Powers, A., Pesik, N., Cohen, N., Staples, J., 2014. Nowcasting the Spread
mpox_nowcasting). of Chikungunya Virus in the Americas. PLoS One 9 (8), e104915.
Jombart T., Cori A., Nouvellet P., Skarp J., Kamvar Z., Taylor T. earlyR: Estimation of
Transmissibility in the Early Stages of a Disease Outbreak. 0.0.5 ed: CRAN; 2020.
Acknowledgements Lloyd-Smith, J., Funk, S., McLean, A., Riley, S., Wood, J., 2015. Nine challenges in
modelling the emergence of novel pathogens. Epidemics 10, 35–39.
Madewell, Z., Pastore Y Piontti, A., Zhang, Q., Burton, N., Yang, Y., Longini, I., et al.,
We thank all public health professionals involved in reporting mpox
2021a. Using simulated infectious disease outbreaks to inform site selection and
cases to CDC. We also acknowledge the Mpox Response Data Analytics sample size for individually randomized vaccine trials during an ongoing epidemic.
and Visualization Task Force Informatics Team for data management Clin. Trials 18 (5), 630–638.
Madewell, Z., Charniga, K., Masters, N., Asher, J., Fahrenwald, L., Still, W., et al., 2023.
and support, and we thank Dr. Sam Abbott for helpful discussions about
Serial interval and incubation period estimates of monkeypox virus infection in 12 U.
EpiNow2 methods. S. jurisdictions, May – August 2022. Emerg. Infect. Dis. 29 (4), 818–821.
Madewell, Z., Dean, N., Berlin, J., Coplan, P., Davis, K., Struchiner, C., Halloran, M.,
2021b. Challenges of evaluating and modelling vaccination in emerging infectious
Disclaimer
diseases. Epidemics 37, 100506.
Minhaj, F., Ogale, Y., Whitehill, F., Schultz, J., Foote, M., Davidson, W., et al., 2022.
The findings and conclusions in this report are those of the authors Monkeypox outbreak — nine states, May 2022. MMWR Morb. Mortal. Wkly Rep. 71
and do not necessarily represent the official position of the Centers for (23), 764–769.
Murayama, H., Pearson, C.A.B., Abbott, S., Miura, F., Jung S-m, Fearon, E., et al., 2023.
Disease Control and Prevention, U.S. Department of Health and Human Accumulation of immunity in heavy-tailed sexual contact networks shapes mpox
Services. outbreak sizes. J. Infect. Dis.
Nash, R., Nouvellet, P., Cori, A., 2022. Real-time estimation of the epidemic reproduction
number: scoping review of the applications and challenges. PLoS Digit Health 1 (6),
Appendix A. Supporting information e0000052.
Oidtman, R., Omodei, E., Kraemer, M., Castañeda-Orjuela, C., Cruz-Rivera, E., Misnaza-
Supplementary data associated with this article can be found in the Castrillón, S., et al., 2021. Trade-offs between individual and ensemble forecasts of
an emerging infectious disease. Nat. Commun. 12, 5379.
online version at doi:10.1016/j.epidem.2024.100755. Overton, C., Abbott, S., Christie, R., Cumming, F., Day, J., Jones, O., et al., 2023.
Nowcasting the 2022 mpox outbreak in England. PLoS Comput. Biol. 19 (9),
References e1011463.
Reich, N., McGowan, C., Yamana, T., Tushar, A., Ray, E., Osthus, D., et al., 2019.
Accuracy of real-time multi-model ensemble forecasts for seasonal influenza in the
Abbott S., Lison A., Funk S. epinowcast: Flexible hierarchical nowcasting. Zenodo;
U.S. PLOS Comput. Biol. 15 (11), e1007486.
2021b.
Rivers C. Changing trends in monkeypox epidemiology: Substack; 2022 [updated
Abbott, S., Hellewell, J., Thompson, R., Sherratt, K., Gibbs, H., Bosse, N., et al., 2020.
September 2, 2022. Available from: 〈https://caitlinrivers.substack.com/p/changin
Estimating the time-varying reproduction number of SARS-CoV-2 using national and
g-trends-in-monkeypox-epidemiology〉.
subnational case counts [version 2; peer review: 1 approved, 1 approved with
Spreco, A., Jöud, A., Eriksson, O., Soltesz, K., Källström, R., Dahlström, Ö., et al., 2022.
reservations]. Wellcome Open Res. 5, 112.
Nowcasting (Short-Term Forecasting) of COVID-19 hospitalizations using syndromic
Abbott S., Hellewell J., Sherratt K., Gostic K., Hickson J., Badr H., et al. EpiNow2:
healthcare data, Sweden, 2020. Emerg. Infect. Dis. 28, 564–571.
Estimate real-time case counts and time-varying epidemiological parameters. 1.3.2
ed: EpiForecasts; 2021a.

8
K. Charniga et al. Epidemics 47 (2024) 100755

Thompson, J., McClure, R., Scott, N., Hellard, M., Abeysuriya, R., Vidanaarachchi, R., U.S Centers for Disease Control and Prevention. CDC Moving Forward Summary Report
et al., 2022. A framework for considering the utility of models when facing tough 2022g [updated September 1, 2022 December 27, 2022]. Available from: 〈htt
decisions in public health: a guideline for policy-makers. Health Res. Policy Syst. 20, ps://www.cdc.gov/about/organization/cdc-moving-forward-summary-report.ht
107. ml〉.
U.S. Centers for Disease Control and Prevention. 2022a Monkeypox Outbreak Global U.S. Department of Health and Human Services. Privacy Impact Assessment 2020
Map 2022 [updated November 15, 2022. Available from: https://www.cdc.gov/ [updated March 26, 2020. Available from: https://www.hhs.gov/sites/default/files/
poxvirus/monkeypox/response/2022/world-map.html. cdc-dcipher-whr.pdf.
U.S. Centers for Disease Control and Prevention. Signs and Symptoms 2022b [updated UK Health Security Agency. Investigation into monkeypox outbreak in England:
October 18, 2022. Available from: 〈https://www.cdc.gov/poxvirus/monkeypox/sy technical briefing 1 2022 [updated September 23, 2022. Available from: https://
mptoms/index.html〉. www.gov.uk/government/publications/monkeypox-outbreak-technical-briefings/
U.S. Centers for Disease Control and Prevention. Monkeypox Technical Reports 2022c investigation-into-monkeypox-outbreak-in-england-technical-briefing-1#:~:
[updated October 27, 2022. Available from: 〈https://www.cdc.gov/poxvirus/monke text=Without%20correcting%20for%20right%2Dtruncation,8.9)%2C%20see%
ypox/cases-data/technical-report.html〉. 20Figure%202a.
U.S. Centers for Disease Control and Prevention. Outbreak Reproduction Number Viboud, C., Sun, K., Gaffey, R., Ajelli, M., Fumanelli, L., Merler, S., et al., 2018. The
Estimates 2022d [updated November 14, 2022. Available from: 〈https://www.cdc. RAPIDD ebola forecasting challenge: Synthesis and lessons learnt. Epidemics 22,
gov/poxvirus/monkeypox/cases-data/case-nowcasting.html〉. 13–21.
U.S. Centers for Disease Control and Prevention. What is the Data Modernization WHO/ECDC. Interim advice for public health authorities on summer events during the
Initiative? 2022e [updated October 11, 2022. Available from: 〈https://www.cdc. monkeypox outbreak in Europe, 2022 2022 [updated June 14, 2022. Available from:
gov/surveillance/projects/dmi-initiative/index.html#:~:text=CDC%E2%80%99s% https://www.ecdc.europa.eu/sites/default/files/documents/Interim-advice-for-
20Dat%20Modernization%20Initiative%20%28DMI%29%20is%20at%20the,dat% public-health-authorities-on-summer-events-mpx.pdf.
20reporting%2C%20management%2C%20and%20analytics%20across%20public% World Health Organization. WHO Director-General declares the ongoing monkeypox
20health〉. outbreak a Public Health Emergency of International Concern 2022 [updated July
U.S. Centers for Disease Control and Prevention. Center for Forecasting and Outbreak 23, 2022. Available from: https://www.who.int/europe/news/item/23-07-2022-
Analytics 2022f [updated October 28, 2022. Available from: 〈https://www.cdc.gov/ who-director-general-declares-the-ongoing-monkeypox-outbreak-a-public-health-
forecast-outbreak-analysis/index.html〉. event-of-international-concern.