Performance of Nucleic Acid

Amplification Tests for Detection

of Severe Acute Respiratory
Syndrome Coronavirus 2 in
Prospectively Pooled Specimens
Hannah Wang, Catherine A. Hogan, Jacob A. Miller, Malaya K. Sahoo, ChunHong Huang, Kenji O. Mfuh,
Mamdouh Sibai, James Zehnder, Brendan Hickey, Nasa Sinnott-Armstrong, Benjamin A. Pinsky

demonstrated the feasibility of pooled SARS-CoV-2

Pooled nucleic acid amplification tests for severe acute
respiratory syndrome coronavirus 2 could increase avail- nucleic acid amplification testing (NAAT), in pools of
ability of testing at decreased cost. However, the effect <32 individual samples (3–9). However, the potential
of dilution on analytical sensitivity through sample pool- increase in efficiency gained by pooled testing is off-
ing has not been well characterized. We tested 1,648 set by a theoretical dilution-related decrease in ana-
prospectively pooled specimens by using 3 nucleic acid lytical sensitivity (8,10).
amplification tests for severe acute respiratory syndrome Despite this decrease in sensitivity, pooled test-
coronavirus 2: a laboratory-developed real-time reverse ing of blood donors for transfusion-transmitted in-
transcription PCR targeting the envelope gene, and 2 fections, such as those with HIV-1 and hepatitis C
commercially available Panther System assays target- virus, has proven to be safe and effective (11). This
ing open reading frame 1ab. Positive percent agreement
efficacy varies depending on the performance char-
(PPA) of pooled versus individual testing ranged from
acteristics of the assay, the prevalence of infection,
71.7% to 82.6% for pools of 8 and from 82.9% to 100.0%
for pools of 4. We developed and validated an indepen- viral load kinetics, and pooling size, and strategy.
dent stochastic simulation model to estimate effects of For agents with variable seasonal or geographic
dilution on PPA and efficiency of a 2-stage pooled real- prevalence, such as West Nile virus, many blood
time reverse transcription PCR testing algorithm. PPA was banks use adaptive risk-based pooling strategies,
dependent on the proportion of tests with positive results, switching from pooled to individual testing when
cycle threshold distribution, and assay limit of detection. there is an increase in regional prevalence (12).
Adapting a similar risk-based pooling strategy for

T he ability of clinical laboratories to meet the de-

mand for severe acute respiratory syndrome
coronavirus 2 (SARS-CoV-2) testing is critical for
SARS-CoV-2 has the potential to enable more wide-
spread testing of high-risk populations and asymp-
tomatic critical infrastructure workers, guide aggres-
reducing coronavirus disease (COVID-19)–related sive contact-tracing measures, and help direct public
illness, death, and economic impact. Pooled testing health interventions to where they are most needed.
has the potential to decrease resources required for However, there are limited prospective data on as-
population-level screening and can provide valuable say-specific performance characteristics of pooled
data to inform public health policies (1,2). Several testing to guide implementation of such a strategy.
previous experimental and modeling studies have Furthermore, there is little evidence on parallel test
Author affiliations: Stanford University School of Medicine,
performance of different assays on pooled samples
Stanford, California, USA (H. Wang, C.A. Hogan, J.A. Miller,
to direct choice of method.
M.K. Sahoo, C. Huang, J. Zehnder, N. Sinnott-Armstrong,
In this study, we aimed to evaluate the test perfor-
B.A. Pinsky); Stanford Health Care, Stanford (K.O. Mfuh, M. Sibai,
mance characteristics of 1 laboratory-developed and
B.A. Pinsky); Stanford University, Stanford (N. Sinnott-Armstrong)
2 commercially available SARS-CoV-2 NAATs for
1,648 individual respiratory specimens prospectively
DOI: https://doi.org/10.3201/eid2701.203379 grouped in pools of 8 and 4. We used these data to

92 Emerging Infectious Diseases • www.cdc.gov/eid • Vol. 27, No. 1, January 2021

 Nucleic Acid Amplification Tests for SARS-CoV-2

validate a stochastic model to estimate optimal pool the status of the positive sample(s) in that pool. Pools
size, efficiency, and expected positive percent agree- containing positive samples belonging to both groups
ment (PPA) of a 2-stage pooled testing algorithm that were excluded from this analysis. To validate the per-
takes into account prevalence, viral load distribution, formance of the model for additional pool sizes, an
and assay analytical sensitivity. external in silico dataset was obtained on the basis of
pool sizes of 3 and 5. The in silico analysis was per-
Methods formed according to US Food and Drug Administra-
tion recommendations (Appendix, https://wwwnc.
Clinical Specimens cdc.gov/EID/article/27/1/20-3379-App1.pdf) (13).
The Stanford Clinical Virology Laboratory receives
samples from tertiary-care academic hospitals and af- Sample Pooling, Extraction, and NAAT
filiated outpatient facilities in the San Francisco Bay Pools were constructed before nucleic acid extrac-
Area of California. Prospective pooling of consecutive tion by combining 500 μL from each of the individual
nasopharyngeal or oropharyngeal swab specimens samples. For a pool size of 8, this resulted in a total
submitted for SARS-CoV-2 testing during the morn- volume of 4 mL and a dilution factor of 1:8. For a pool
ing shift was conducted during June 10–19, 2020, for size of 4, this resulted in a total volume of 2 mL and a
evaluation of a pool size of 8 and during July 6–July 23, dilution factor of 1:4.
2020, for evaluation of a pool size of 4. Samples sub- Subsequently, total nucleic acids were extracted
mitted for testing were collected from symptomatic from 500 μL taken from each pool and each individual
and asymptomatic inpatients and outpatients, either specimen by using QIAsymphony and the QIAsym-
for clinical care or in the context of COVID-related phony DSP Virus/Pathogen Midi Kit (QIAGEN,
epidemiologic surveillance studies and drug trials at https://www.qiagen.com) and eluted into 60 μL of
our institution. As samples from persons enrolled in AVE buffer according to manufacturer’s instructions.
these studies and trials were received daily in batches, Real-time reverse transcription PCR (rRT-PCR) was
they were randomly evenly distributed among pools performed by using an emergency use authorization
on a daily basis. This distribution was conducted to laboratory-developed test (LDT) targeting the enve-
preserve the independence between samples in the lope gene with the Rotor-Gene Q Instrument (QIA-
same pool; these samples had not been tested before GEN) as described (14–16), with pooled samples test-
receipt in our laboratory and were otherwise treated ed on the same run as component individual samples.
identically to nonresearch samples. Nonresearch A Ct result of 40–45 was considered an indeterminate
samples were otherwise assigned to pools consecu- result, which was adjudicated by repeat testing and
tively. Additional laboratorywide data on proportion resulted as positive if reproducible with an acceptable
of tests positive and cycle threshold (Ct) value distri- amplification curve. Specimens were only reported as
bution were obtained from all specimens (n = 74,162) negative if the internal control human RNase P gene
tested during March 1–June 24, 2020. This study was was detected at a Ct<35.
conducted with Stanford institutional review board On the same day as QIAsymphony extraction,
approval (protocol no. 48973), and individual consent another 500 μL from each pool was transferred to
was waived. a Hologic Panther Specimen Lysis Tube (Hologic,
https://www.hologic.com) and tested by using the
Pool Size Determination Panther Fusion SARS-CoV-2 Assay (Hologic) and
In this study, an initial pool size of 8 was selected on Panther Aptima SARS-CoV-2 assay (Hologic) per the
the basis of pilot experiments with pool sizes ranging manufacturer’s recommendations (17,18). In addition
from 4 to 10 (B.A. Pinsky, unpub. data), and the logisti- to the manufacturer-set cutoff value, receiver operat-
cal consideration that pooling in multiples of 4 would ing characteristic (ROC) curve analysis of pooled rela-
be more efficient for the robotic liquid handlers in our tive light unit (RLU) values, with individual test re-
laboratory. After review of the test performance char- sults as the reference method, was used to determine
acteristics of 8-sample pooling in conjunction with the optimal RLU discrimination threshold. A focused
the results of an independent stochastic simulation electronic medical record review was conducted for
model, additional testing was performed to evaluate all samples.
a pool size of 4 to generate empiric data for further
model validation. Subset analyses of first tests versus Statistical Analysis
follow-up tests were conducted by retrospectively ROC curve analysis was conducted by using R pack-
assigning pools to 1 of the 2 groups on the basis of age pROC (19). PPA and negative percent agreement

Emerging Infectious Diseases • www.cdc.gov/eid • Vol. 27, No. 1, January 2021 93

RESEARCH

(NPA) were calculated by using individual testing Ct value than specimens that underwent follow-up
as the reference method and were reported with tests (Table 2). ROC curve analysis for the Panther
exact (Clopper-Pearson) 95% CIs (20). Passing- Aptima showed optimal cutoff values between 343
Bablok regression was used to compare Ct values and 393 RLUs; a cutoff value of 350 was chosen as the
of the individual LDT, pooled LDTs, and pooled nearest round number (Panther Aptima-350) (Appen-
Panther Fusion assays. The 95% CIs of slope, in- dix Table 1, Figure 1).
tercept, and bias were calculated by using an ordi- Among the tested pools of 8, a total of 41.8%
nary nonparametric bootstrap resampling method (46/110) contained >1 positive sample. The positive
with default parameters in R package mcr. Paired pools comprised 36 pools with 1 positive sample,
t-tests were used to compare the mean differences 9 pools with 2 positive samples, and 1 pool with 4
between paired Ct values among different assays. A positive samples (Table 3). There were 3 false-positive
Student t-test was used to compare the mean differ- pools, 1 on each platform, in which each of the indi-
ence between internal control RNase P Ct values in vidual samples showed negative results. The overall
false-negative and true-negative pools. All compar- PPA of pooled testing ranged from 71.7% to 82.6%,
isons were 2-sided with type I error set at 0.05. We and NPA ranged from 98.4% to 100.0% (Table 4).
used the laboratory-wide Ct value distribution and The 14 pools containing positive first-time diagnostic
a separate limit of detection (LoD) experiment to samples had higher PPAs than the 28 pools contain-
develop a stochastic simulation model to estimate ing positive follow-up test samples in an LDT (Ap-
PPA and efficiency for a 2-stage pooled testing al- pendix Table 3).
gorithm, which was subsequently validated by us- There were 16 total pools for which >1 method
ing the independent empiric pools of 8 and pools of showed false-negative results. Except for the 1 pool
4 data, as well as in silico pools of 5 and pools of 3 containing 4 positive specimens, which was not de-
data. We provide the methods used to develop this tected by Panther Aptima using the manufacturer’s
model (Appendix). cutoff value (Panther Aptima-M), the remaining 15
false-negative pools each contained only 1 positive
Results specimen. For all missed pools, the Ct value of the
individual positive sample was >34 (median 36.6,
Assay Comparisons for Pools of 8 interquartile range 35.5–37.7) (Figure 1). Among in-
To evaluate a pool size of 8, a total of 112 pools from dividual positive specimens in the dataset for pools
896 samples were each tested on 3 different NAAT of 8, a total of 22 (37.9%) had Ct values >34. A total of
platforms (Table 1). Two pools were invalid, 1 by the 13/22 (59.1%) were false negative for the LDT, 11/22
Panther Fusion assay (0.9%), and 1 by the Panther (50.0%) for the LDT Panther Fusion, 15/22 (68%) for
Aptima assay (0.9%), and were excluded from sub- the LDT Panther Aptima-M, and 8/22 (36.4%)for the
sequent analysis. All 16 individual samples in these LDT Panther Aptima-350. Each of these false-negative
2 pools showed negative results. The remaining 110 samples was collected from known symptomatic or
pools contained 880 individual samples. Four sam- convalescent-phase patients being monitored for vi-
ples were tested in duplicate in 2 different pools and ral clearance; none of these samples were initial diag-
showed identical results. Among the 880 individual nostic specimens. The pooled LDT RNase P internal
samples, 58 (6.6%) showed positive results and a me- control Ct values were similar in false-negative (mean
dian Ct value of 31.4 (interquartile range 22.1–35.5). 23.5, 95% CI 22.7–24.3) and true-negative (mean 23.4,
First-time diagnostic specimens had a higher median 95% CI 22.7–24.1; p = 0.7) pools.

Table 1. Performance of nucleic acid amplification tests for detection of severe acute respiratory syndrome coronavirus 2 in
prospectively pooled specimens, by testing platform*
Test name Gene target(s) Internal control Method Strategy Reference
LDT Envelope RNase P rRT-PCR Pools of 8†, pools of 4† (1,14–16)
Panther Fusion ORF1ab Reagent spike-in rRT-PCR Pools of 8†, pools of 5†, pools of 3‡ (17)
Panther Aptima-M ORF1ab Reagent spike-in TMA Pools of 8 with manufacturer-set RLU cutoff† (18)
Panther Aptima-350 ORF1ab Reagent spike-in TMA Pools of 8 with RLU cutoff of >350†§ (18)
*Panther Aptima-M, Panther Aptima with manufacturer-set relative light unit cutoff value; Panther Aptima-350, Panther Aptima with relative light unit cutoff
value >350 considered positive. Both products were from Hologic (https://www.hologic.com). LDT, laboratory-developed test; ORF1ab, open reading
frame 1ab; rRT-PCR, real-time reverse transcription PCR; RLU, relative light unit; TMA, transcription-mediated amplification.
†Pooled testing strategy was assessed empirically at Stanford Clinical Virology Laboratory, with individual samples evaluated by LDT.
‡Pooled testing strategy assessed by in silico sensitivity analysis, with individual samples evaluated by Panther Fusion.
§Panther Aptima RLU cutoff of 350 selected based on receiver operating characteristic curve (Appendix Figure 1,
https://wwwnc.cdc.gov/EID/article/27/1/20-3379-App1.pdf).

94 Emerging Infectious Diseases • www.cdc.gov/eid • Vol. 27, No. 1, January 2021

 Nucleic Acid Amplification Tests for SARS-CoV-2

Table 2. Proportion of tests positive for severe acute respiratory syndrome coronavirus 2 with median C t values in pooled testing and
laboratorywide clinical testing datasets, subset by testing indication*
No. positive samples/no. total samples (%) Median Ct value (IQR)
Dataset All First Follow-up All First Follow-up
Pools of 8† 58/880 (6.6) 24/657 (3.7) 34/223 (15.2) 31.4 (22.1–35.5) 24.4 (18.4–33.1) 34.1 (29.0–36.8)
Pools of 4‡ 38/768 (4.9) 28/491 (5.7) 10/277 (3.6) 29.3 (20.3–33.9) 27.5 (19.4–32.6) 32.2 (24.9–34.5)
Hologic§ 10,000/52,272 (19.1) NA NA 26.2 (20.7–32.6) NA NA
Laboratory- 1,358/74,162 (1.8) 1,109/66,070 (1.7) 249/8,092 (3.1) 28.5 (23.0–34.3) 27.2 (22.2–32.4) 34.2 (29.0–37.4)
wide¶
March 555/8,896 (6.2) 489/8,557 (5.7) 66/339 (19.5) 26.7 (21.9–31.5) 26.4 (21.8–31.2) 28.6 (22.6–35.2)
April 518/22,671 (2.3) 404/21,167 (1.9) 114/1,504 (7.5) 30.6 (24.8–36.0) 28.8 (22.7–34.6) 35.4 (32.9–38.0)
May 172/21,833 (0.8) 136/19,505 (0.7) 36/2,328 (1.5) 27.5 (23.3–34.7) 26.1 (22.5–31.3) 35.4 (30.4–37.3)
June 113/20,762 (0.5) 80/16,841 (0.5) 33/3,921 (0.84) 28.2 (21.2–33.6) 27.4 (21.3–32.7) 30.6 (20.2–34.4)
*Ct, cycle threshold; IQR, interquartile range; NA, not available.
†Pools of 8 specimens were tested in our clinical laboratory during June 10–19, 2020.
‡Pools of 4 specimens were tested in our clinical laboratory during July 6–23, 2020.
§Hologic dataset comprises specimens tested clinically by Panther Fusion (https://www.hologic.com) during March 1–July 31, 2020 at 2 different external
sites. These data were used to perform in silico sensitivity analysis to evaluate pool sizes of 3 and 5.
¶Composed of clinical specimens obtained during March 1–June 24, 2020.

Linearity Studies for Pools of 8 Model Estimates

For pools containing only 1 positive sample, the The modeled PPA estimate is sensitive to the input
pooled rRT-PCRs showed positive systematic bias parameters of proportion of positive tests, assay ana-
when compared with the individual LDT assay, lytical sensitivity, and viral load distribution. The
as shown by the Passing-Bablok regression inter- analytical sensitivity of the assay is approximated
cept value being >0. Mean bias between pooled in this model by the Ct value corresponding to the
and individual Ct values was 3.4 cycles (95% lim- probability of detecting 95% of true-positive samples,
its of agreement 1.2–5.6; p<0.001) by LDT and 4.0 otherwise known as the 95% LoD. Specimens with Ct
cycles (95% limits of agreement 0.0–8.0; p<0.001) by beyond the LoD are assigned a decreasing probability
Panther Fusion (Figure 2). Panther Fusion showed of detection on the basis of a probit regression curve,
negative proportional bias when compared with the shape of which was determined in the initial vali-
individual and pooled LDTs, as shown by Passing- dation of the LDT (Appendix Figure 5). The viral load
Bablok regression slopes with 95% CIs that do not distribution of the tested population is approximated
contain 1. This result is additionally highlighted in in this model by the proportion of samples with Ct
the Bland-Altman plots, which demonstrate that greater than the LoD. This makes the model output
at higher Ct values, Panther Fusion outperforms independent of the actual LoD Ct value itself, enabling
the LDT. the model to be used across different rRT-PCRs.

Table 3. Results of 8-sample pooled testing, by testing platform and number of positive specimens per pool (n = 110) for detection of
severe acute respiratory syndrome coronavirus 2*
Pooled testing Individual testing
Row number and Panther Panther Panther Positive,
Total counts LDT Fusion Aptima-M Aptima-350 (no. 1 PP, no. >1 PP) Negative Total no. pools
1 + + + + 30 (21, 9) 0 30
2 + + – + 2 (1, 1) 0 2
3 + + – – 0 (0, 0) 0 0
4 + – + + 0 (0, 0) 0 0
5 + – – + 0 (0, 0) 0 0
6 + – – – 1 (1, 0) 1† 2
7 – + + + 2 (2, 0) 0 2
8 – + – + 1 (1, 0) 0 1
9 – + – – 0 (0, 0) 1‡ 1
10 – – + + 2 (2, 0) 0 2
11 – – – + 1 (1, 0) 1§ 2
12 – – – – 7 (7, 0) 61 68
No. positive pools 34 36 34 39 46 (36, 10) – –
No. negative pools 76 74 76 71 – 64 –
Total no. pools 110 110 110 110 – – 110
*Panther Aptima with manufacturer-set relative light unit cutoff. Panther Aptima-350, Panther Aptima with relative light unit cutoff value >350 was
considered positive; Ct, cycle threshold; LDT, laboratory-developed test; Pos, positive; RLU, relative light unit; 1 PP, 1 positive specimen in pool; >1 PP,
>2 positive specimens in pool; –, negative; +, positive.
†False-positive LDT Ct value was 37.5.
‡False-positive Panther Fusion Ct value was 38.8.
§False-positive Panther Aptima-350 RLU value was 434.

Emerging Infectious Diseases • www.cdc.gov/eid • Vol. 27, No. 1, January 2021 95

RESEARCH

Table 4. Performance characteristics and efficiency of 8-sample and 4-sample pooled testing, by testing platform (n = 302), for
detection of severe acute respiratory syndrome coronavirus 2 in prospectively pooled specimens*
Average test
Testing platform Pool size PPA, % (95% CI) NPA,% (95% CI) Pools positive, % run/sample
LDT 8 71.7 (56.5–84.0) 98.4 (91.5–100.0) 30.9 0.434
Panther Fusion 8 76.1 (61.2–87.4) 98.4 (91.5–100.0) 32.7 0.452
Panther Aptima-M 8 73.9 (58.9–85.7) 100.0 (94.3–100.0) 30.9 0.434
Panther Aptima-350 8 82.6 (68.6–92.2) 98.4 (91.5–100.0) 34.5 0.470
LDT 4 94.3 (80.8–99.3)† 100 (97.7–100.0) 17.2 0.422
Panther Fusion‡ 4 100.0 (85.8–100.0) 100 (96.7–100.0) 17.6 0.426
Panther Aptima-M 4 82.9 (66.2–93.4)† 100 (97.7–100.0) 15.1 0.401
Panther Aptima-350 4 88.6 (73.3–96.8)† 100 (97.7–100.0) 16.2 0.411
*Panther Aptima-M, Panther Aptima with manufacturer-set RLU cutoff value. Panther Aptima-350, Panther Aptima with RLU cutoff value >350 was
considered positive. Both products from Hologic (https://www.hologic.com). LDT, laboratory-developed test; NPA, negative percent agreement; PPA,
positive percent agreement; RLU, relative light unit.
†Restricting the performance characteristics comparison to only the 136 pools tested by Panther Fusion resulted in a PPA as follows: LDT 100% (95% CI
85.8%–100.0%), Aptima-M 91.7% (95% CI 73.0%–99.0%), and Aptima-350 95.8% (95% CI 78.9%–99.9%).
‡A total of 56 of the 192 pools tested on the other platforms were not tested by Panther Fusion.

If the assay analytical sensitivity is kept constant, prevalence increases, PPA can counterintuitively in-
but the tested population changes such that a greater crease with larger pool sizes because there is a greater
proportion have a Ct value beyond the 95% LoD, PPA likelihood of having more than 1 positive sample in a
decreases (Figure 3, panel A). Conversely, if the pa- given pool, which would be expected to increase PPA.
tient population is kept constant, but assay analytical Similarly, test efficiency can decrease with larger pool
sensitivity increases (i.e., from lower Ct LoD to higher sizes because the likelihood of deconvoluting a posi-
Ct LoD), PPA increases (Figure 4, panel A). However, tive pool increases. Estimated PPA and average tests
if assay analytical sensitivity changes and the tested per sample for inputs of percentage of positive tests
population shifts accordingly such that it retains the 0.1%–15.0% and proportion of samples with Ct value
same proportion Ct >LoD, then the PPA stays con- above the LoD ranging from 5% to 30% are available
stant (Appendix Figure 6). In contrast, the average ex- (Appendix Table 4).
pected tests per sample is almost entirely determined
by pool size and prevalence, whereas analytical sen- Model Sensitivity Analyses and Validation
sitivity (LoD Ct) and the underlying Ct distribution One-way deterministic and probabilistic sensitivity
minimally affect efficiency because of small absolute analyses incorporating uncertainty in the underlying
numbers of false-positive pools (Figure 3, panel B; model assumptions of dilutional effect and probit re-
Figure 4, panel B). To achieve a 5% absolute differ- gression shape demonstrate a moderate (±2% to ±7%)
ence in efficiency with an increase in LoD Ct from 32 effect on PPA, which is more pronounced with larger
to 40, a prevalence of 25% would be required. pool sizes and proportion of Ct values above the
Both PPA and tests per sample are highly de- LoD (Appendix Figure 7). In contrast, these param-
pendent on pool size and prevalence of infection. As eters have a much smaller effect on testing efficiency
Figure 1. Performance of
nucleic acid amplification
tests for detection of severe
acute respiratory syndrome
coronavirus 2 in prospectively
pooled specimens. For a pool
size of 8, paired individual
and pooled Ct values for
each individually positive
sample (n = 58), in order of
increasing individual Ct value.
A) Pools comprising only 1
positive sample/pool. B) Pools
comprising >2 positive samples/
pool. The gray lines span the
range of Ct values associated
with a given pool. Rows without
gray lines indicate individually
positive samples belonging to pools that were negative by both real-time reverse transcription PCR methods. Panther Fusion is from
Hologic (https://www.hologic.com). Ct, cycle threshold; LDT, laboratory-developed test.

96 Emerging Infectious Diseases • www.cdc.gov/eid • Vol. 27, No. 1, January 2021

 Nucleic Acid Amplification Tests for SARS-CoV-2

(Appendix Figure 8). The 95% CIs for the empirically slightly higher for pools of 3 than pools of 5, which
determined and modeled PPAs overlapped for most was probably caused by the high prevalence of 19.1%
of the evaluated empiric datasets, although these val- in this dataset (Appendix Table 3).
ues overestimated PPA for the LDT follow-up tests
only subset (Figure 5). For the in silico validation data, Discussion
the modeled PPA was similar for pool sizes of 5 and 3, In this study, >1,600 samples were tested in pool sizes
despite in silico data analysis predicting a higher PPA of 8 and 4 by using 3 different SARS-CoV-2 platforms,
for pools of 3. Modeled testing efficiency was actually and pooled testing showed decreased PPA relative to
Figure 2. Performance of
nucleic acid amplification tests
for detection of severe acute
respiratory syndrome coronavirus
2 in prospectively pooled
specimens. Passing-Bablok
regression and Bland-Altman
plots for pools of 8 containing only
1 positive sample, tested by A and
B) pooled LDT versus individual
LDT (n = 23) (A, B); pooled
Panther Fusion versus individual
LDT (n = 25) (C, D); and pooled
Panther Fusion versus pooled
LDT (n = 32) (E, F). For the
Passing-Bablok regression
plots (A, C, and E), the solid line
indicates the line of regression.
95% CIs are shaded in gray. The
dashed line indicates the line of
identity. The slope and intercept
of the regression line are reported
with 95% CIs in parentheses. For
the Bland-Altman plots (B, D, and
F), the solid line represents the
mean difference in Ct value. 95%
limits of agreement are shaded
in gray. Panther Fusion is from
Hologic (https://www.hologic.
com). Ct, cycle threshold; LDT,
laboratory-developed test; LOA,
limits of agreement.

Emerging Infectious Diseases • www.cdc.gov/eid • Vol. 27, No. 1, January 2021 97

RESEARCH

Figure 3. Performance of
nucleic acid amplification
tests for detection of severe
acute respiratory syndrome
coronavirus 2 in prospectively
pooled specimens.
Model-estimated PPA and
testing efficiency, by pool size,
proportion of tests positive,
and proportion of samples
with Ct above the 95% LoD.
For these estimates, LoD
has been held constant at
the experimentally-derived
Ct of 35.9, although results
are independent of specific
LoD value. A) Expected
PPA between pooled and
individual testing at pool sizes
of 1–20. PPA decreases with
decreasing proportion of test
results positive (indicated by
colored lines in each plot),
and with increasing proportion
of samples with Ct values
beyond the 95% LoD (each
panel). At >5% test positivity,
expected PPA starts to
increase at larger pool sizes
because there is a greater
likelihood of 2 positive samples
being in the same pool. The
baseline PPA (pool size of
1) reflects the likelihood of
obtaining the same individual
result with repeat (nonpooled)
testing. B) Estimated average
tests per sample that would
be performed at each pool
size, with a lower number
of average tests per sample
corresponding to higher testing
efficiency. Efficiency increases
with decreasing proportion
of test results positive, and slightly increases with increasing samples with Ct above the LoD. Each missed pool results in fewer
deconvolutions, and thus fewer total tests performed. Ct, cycle threshold; LoD, limit of detection; PPA, positive percent agreement.

individual samples. False-negative results occurred method-dependent nature of test performance, a vari-
exclusively in pools containing samples with low es- able that cannot be anticipated, and therefore is not
timated viral load (Ct >34). Overlapping CIS in PPA explicitly accounted for in most statistical models
and NPA at each pool size suggest that the lower test of pooled testing. Thus, method comparison studies
performance is inherent to the pooling process itself, should be performed before large-scale implementa-
rather than the assay. Although Panther Fusion Ct val- tion of any pooled testing strategies, especially those
ues were on average higher than those of the LDT, the that use different platforms for the pooled and indi-
negative proportional bias suggests that at low esti- vidual stages of testing.
mated viral loads (Ct >36), the Panther Fusion outper- The findings of our study contrast with those
formed the LDT. This finding might be caused by the of a recent study=, which concluded that pooling in
different targets of amplification (envelope gene ver- groups of 8 did not compromise test performance (5).
sus open reading frame 1ab) or PCR efficiency. These This finding might be explained by differences in pa-
subtle differences between the 2 assays highlight the tient population, higher proportion of positive pools

98 Emerging Infectious Diseases • www.cdc.gov/eid • Vol. 27, No. 1, January 2021

 Nucleic Acid Amplification Tests for SARS-CoV-2

Figure 4. Performance of nucleic

acid amplification tests for detection
of severe acute respiratory syndrome
coronavirus 2 in prospectively pooled
specimens. Model-estimated PPA
and testing efficiency, by pool size,
proportion of tests positive, and assay
analytical sensitivity as approximated
by the Ct corresponding to the
95% LoD. For these estimates, the
population viral load distribution has
been held constant at 15% of samples
with Ct values >35. A) Expected PPA
between pooled and individual testing
at pool sizes of 1–20. PPA decreases
with decreasing proportion of tests
positive (indicated by colored lines
in each plot) and increases with
increased analytical sensitivity (each
panel). This result occurs because the
proportion of individual samples with
a Ct value above each LoD decreases
as the Ct LoD increases. B) Estimated
average tests per sample that would
be performed at each pool size, with
a lower number of average tests per
sample corresponding to higher testing
efficiency. Efficiency increases with
decreasing proportion of test results
positive, and slightly decreases with
increased analytical sensitivity because
more pools detected results in an
increased number of individual tests
performed at the deconvolution step. Ct,
cycle threshold; LoD, limit of detection;
PPA, positive percent agreement.

Emerging Infectious Diseases • www.cdc.gov/eid • Vol. 27, No. 1, January 2021 99

RESEARCH

and rRT-PCR result interpretation. Another recent in this study; none exceeded a Ct of 30. However, this
study of artificially constructed pools reported no ma- study and other experimental studies have shown
jor decrease in sensitivity in pools of <32 samples (3). empirical increases in pooled Ct values directly pro-
This finding is probably explained by the relatively portional to dilution factor, a relationship that was
low starting Ct values of individual positive samples also observed in our study (3,4,9).

Figure 5. Performance of nucleic acid amplification tests for detection of severe acute respiratory syndrome coronavirus 2 in
prospectively pooled specimens. Empiric and modeled estimates of positive percent agreement (PPA) with 95% CIs for each pool
size, testing platform, and sample type (all versus first initial diagnostic versus follow-up). Black circles indicate empiric PPA point
estimates, and colored horizontal bars indicate 95% CI. The 95% CI for the in silico data are too narrow to be visible in this plot.
Gray boxplots indicate the modeled estimate of PPA, vertical black lines indicate the modeled PPA point estimate, and gray box
indicates the 95% CI of the probabilistic sensitivity analysis. No modeled estimates are available for Panther Aptima because this
is a transcription-mediated amplification assay, and the model is based on dilutional effects inherent to real-time PCR only. The
empiric 95% CIs contain the modeled PPA point estimates for all conditions except for pools of 8 follow-up tests only and the in
silico data. Data used to generate this figure are provided in Appendix Table 3 (https://wwwnc.cdc.gov/EID/article/27/1/20-3379-
App1.pdf). Panther Fusion and Panther Aptima are from Hologic (https://www.hologic.com). LDT, laboratory-developed test; PSA,
probabilistic sensitivity analysis.

100 Emerging Infectious Diseases • www.cdc.gov/eid • Vol. 27, No. 1, January 2021
 Nucleic Acid Amplification Tests for SARS-CoV-2

These differences highlight the effect of viral load value of the assay, assuming 100% detection below
distribution and assay analytical sensitivity on pooled the cutoff value, and 0% detection above it. In con-
test performance, both of which should be taken into trast, our model incorporates the probabilistic na-
account when choosing pool size and diagnostic as- ture of detection at and above the LoD, which better
say. Although samples with Ct values >33 have not approximates reality.
been reported to produce cultivable virus in convales- Our approach is limited by the generalizability
cent phase COVID-19 patients (21), >15% of first-time of the probit regression shape and the equation es-
diagnostic specimens in our laboratory were detected timating dilutional effect, as demonstrated by the
at a Ct >35. A similar proportion of weakly positive variability seen on probabilistic and deterministic
samples that had high Ct values at a public health de- sensitivity analysis. Furthermore, the model as-
partment virology laboratory in New York has been sumes that the PCR is 100% efficient and that it is
described (S.B. Griesemer, unpub. data). Assays with devoid of any proportional bias between individual
lower analytical sensitivity may miss specimens with and pooled tests. In addition, the model might un-
late Ct values, for which the potential associated bur- derestimate PPA and efficiency of pooled testing if
den of onward transmission is currently unclear. samples in each pool are not independent; placing
The stochastic model in this study demonstrated samples with higher pretest probability in the same
that expected PPA between pooled and individual pool would decrease the total number of positive
rRT-PCRs was highly dependent on assay analyti- pools and increase the likelihood of detection. This
cal sensitivity (represented by 95% LoD), viral load feature could be leveraged by pooling specimens
distribution of test-positive patients (represented by from persons in the same household or social dis-
proportion Ct >LoD), pool size, and disease preva- tancing pod, such as coworkers on the same shift or
lence (represented by proportion of tests positive). students sharing a classroom. These factors, among
The model outputs were not always intuitive; larger others, might be the reasons for which the probabi-
pool sizes were not always less sensitive or more ef- listic sensitivity analysis CIs often did not contain
ficient. With increased prevalence, larger pool sizes the empiric point estimate in our validation data.
were more sensitive because they were more likely These unaccounted-for factors might limit the abil-
to contain >1 positive sample/pool. They were also ity of the model to provide a reliable point estimate.
less efficient because a larger proportion were posi- The strengths of our study include its relatively
tive and required deconvolution. large sample size, prospective rather than experimen-
The model output was largely independent of tal construction of pools, and assessment of 2 different
the actual LoD and viral load-to-Ct value relation- pool sizes. It also compared 3 different SARS-CoV-2
ship of a given assay, making it generalizable across assays, 2 of which are commercially available on
different rRT-PCRs. The only input parameters it highly automated platforms suitable for large-scale
requires are the proportion of positive test results testing. Our study was limited by its assessment of
and the proportion of samples with Ct >LoD, both only a 2-stage pooling strategy. An additional limi-
of which should be readily available to any laborato- tation includes selection bias because the proportion
ries conducting clinical testing. Future studies on the of positive test results in the study specimens was
sensitivity of pooled testing strategies should report higher because of the inclusion of follow-up samples
these parameters. from known COVID-19 patients enrolled in clinical
Previous models of pooled testing strategies for research studies. Finally, test performance might vary
SARS-CoV-2 have primarily examined the effect of depending on specimen collection medium, which
pool size and prevalence on testing efficiency but we did not assess in this study (S.B. Griesemer, un-
have not addressed the expected decrement in assay pub. data).
sensitivity that accompanies a putative increase in In conclusion, a 2-stage pooled testing strategy
efficiency (6,22). Those studies that have examined for detection of SARS-CoV-2 by nucleic acid amplifi-
sensitivity did not explicitly model the effect of vari- cation is feasible and has the potential to strongly in-
able viral load distribution of test-positive patients, crease testing capacity. However, increased pool size
a parameter that can vary based on the underlying and efficiency can compromise PPA. More studies
patient population (asymptomatic versus symptom- examining early viral load kinetics and infectiousness
atic and severe versus nonsevere), purpose of testing are needed to fully evaluate the risks versus benefits of
(diagnostic versus follow-up), and specimen type pooled testing. We provide a model to predict optimal
(8,23–27). In addition, previous modeling studies pool size and associated expected PPA based on limit
and in silico analyses have mostly used the Ct cutoff of detection, Ct value distribution, and proportion

Emerging Infectious Diseases • www.cdc.gov/eid • Vol. 27, No. 1, January 2021 101
RESEARCH

of positive test results. If this model can be externally 8. Pilcher CD, Westreich D, Hudgens MG. Group testing for
validated, it might be useful in guiding SARS-CoV-2 SARS-CoV-2 to enable rapid scale-up of testing and
real-time surveillance of incidence. J Infect Dis. 2020;222:
pooled testing in other laboratories and as part of an 903–9. https://doi.org/10.1093/infdis/jiaa378
adaptive risk-based strategy. 9. Eis-Hübinger AM, Hönemann M, Wenzel JJ, Berger A,
Widera M, Schmidt B, et al. Ad hoc laboratory-based
Acknowledgments surveillance of SARS-CoV-2 by real-time RT-PCR using
minipools of RNA prepared from routine respiratory
We thank the Stanford Clinical Virology Laboratory staff samples. J Clin Virol. 2020;127:104381. https://doi.org/
for their dedication and commitment to patient care in the 10.1016/j.jcv.2020.104381
face of unprecedented challenges presented by the 10. Perchetti GA, Sullivan K-W, Pepper G, Huang M-L, Breit N,
COVID-19 pandemic, and Hologic, Inc. for graciously Mathias P, et al. Pooling of SARS-CoV-2 samples to increase
molecular testing throughput. J Clin Virol. 2020;131:104570.
providing in silico data from their pooling evaluations. https://doi.org/10.1016/j.jcv.2020.104570
11. Busch MP, Kleinman SH, Jackson B, Stramer SL, Hewlett I,
N.S.-A. is supported by a Stanford Graduate Fellowship.
Preston S. Committee report: nucleic acid amplification
Work was performed by B.H. during his personal time. testing of blood donors for transfusion-transmitted
infectious diseases: report of the Interorganizational Task
B.H. is an employee of Google LLC. Force on Nucleic Acid Amplification Testing of Blood
Donors. Transfusion. 2000;40:143–59. https://doi.org/
10.1046/j.1537-2995.2000.40020143.x
About the Author 12. Custer B, Tomasulo PA, Murphy EL, Caglioti S, Harpool D,
Dr. Wang is a resident physician in the Department of McEvoy P, et al. Triggers for switching from minipool testing
by nucleic acid technology to individual-donation nucleic
Anatomic and Clinical Pathology at Stanford Hospital, acid testing for West Nile virus: analysis of 2003 data to
Palo Alto, CA. Her primary research interests focus on inform 2004 decision making. Transfusion. 2004;44:1547–54.
infectious disease molecular diagnostics and the https://doi.org/10.1111/j.0041-1132.2004.04227.x
application of machine-learning algorithms and other 13. Food and Drug Administration. Molecular diagnostic
template for laboratories, July 28, 2020 [cited 2020 Sep 18].
computational methods to effectively leverage those https://www.fda.gov/media/135658/download
diagnostics for patient care. 14. Food and Drug Administration. EUA summary:
SARS-CoV-2 RT-PCR assay (Stanford Health Care
Clinical Virology Laboratory). April 8, 2020 [cited 2020 Jul
References 11]. https://www.fda.gov/media/136818/download
1. Hogan CA, Sahoo MK, Pinsky BA. Sample pooling as 15. Bulterys PL, Garamani N, Stevens B, Sahoo MK, Huang C,
a strategy to detect community transmission of Hogan CA, et al. Comparison of a laboratory-developed
SARS-CoV-2. JAMA. 2020;323:1967–9. https://doi. test targeting the envelope gene with three nucleic acid
org/10.1001/jama.2020.5445 amplification tests for detection of SARS-CoV-2. J Clin Virol.
2. Wacharapluesadee S, Kaewpom T, Ampoot W, Ghai S, 2020;129:104427. https://doi.org/10.1016/j.jcv.2020.104427
Khamhang W, Worachotsueptrakun K, et al. Evaluating 16. Corman VM, Landt O, Kaiser M, Molenkamp R, Meijer A,
the efficiency of specimen pooling for PCR-based Chu DKW, et al. Detection of 2019 novel coronavirus (2019-
detection of COVID-19. J Med Virol. 2020;May 13:2193–9. nCoV) by real-time RT-PCR. Euro Surveill. 2020;25:1–8.
https://doi.org/10.1002/jmv.26005 https://doi.org/10.2807/1560-7917.ES.2020.25.3.2000045
3. Yelin I, Aharony N, Shaer Tamar E, Argoetti A, Messer E, 17. Food and Drug Administration. EUA summary: SARS-CoV-2
Berenbaum D, et al. Evaluation of COVID-19 RT-qPCR test assay (Panther Fusion System), April 24, 2020 [cited 2020 Jul
in multi-sample pools. Clin Infect Dis. 2020;May 2:ciaa531. 11]. https://www.fda.gov/media/136156/download
https://doi.org/10.1093/cid/ciaa531 18. Food and Drug Administration. EUA Summary:
4. Abdalhamid B, Bilder CR, McCutchen EL, Hinrichs SH, Aptima SARS-CoV-2, May 14, 2020 [cited 2020 Jul 11].
Koepsell SA, Iwen PC. Assessment of specimen pooling to https://www.fda.gov/media/138096/download
conserve SARS CoV-2 testing resources. Am J Clin Pathol. 19. Robin X, Turck N, Hainard A, Tiberti N, Lisacek F,
2020;153:715–8. https://doi.org/10.1093/ajcp/aqaa064 Sanchez JC, et al. pROC: an open-source package for R and
5. Ben-Ami R, Klochendler A, Seidel M, Sido T, S+ to analyze and compare ROC curves. BMC Bioinformatics.
Gurel-Gurevich O, Yassour M, et al.; Hebrew University- 2011;12:77. https://doi.org/10.1186/1471-2105-12-77
Hadassah COVID-19 Diagnosis Team. Large-scale imple- 20. Altman D, Machin D, Bryant T, Gardner M, editors. Statistics
mentation of pooled RNA extraction and RT-PCR for with confidence. 2nd ed. London: British Medical Journal
SARS-CoV-2 detection. Clin Microbiol Infect. 2020;26:1248– Books; 2000.
53. https://doi.org/10.1016/j.cmi.2020.06.009 21. La Scola B, Le Bideau M, Andreani J, Hoang VT,
6. Eberhardt JN, Breuckmann NP, Eberhardt CS. Multi-stage Grimaldier C, Colson P, et al. Viral RNA load as determined
group testing improves efficiency of large-scale COVID-19 by cell culture as a management tool for discharge of
screening. J Clin Virol. 2020;128:104382. https://doi.org/ SARS-CoV-2 patients from infectious disease wards.
10.1016/j.jcv.2020.104382 Eur J Clin Microbiol Infect Dis. 2020;39:1059–61.
7. Lohse S, Pfuhl T, Berkó-Göttel B, Rissland J, Geißler T, https://doi.org/10.1007/s10096-020-03913-9
Gärtner B, et al. Pooling of samples for testing for 22. Aragón-Caqueo D, Fernández-Salinas J, Laroze D.
SARS-CoV-2 in asymptomatic people. Lancet Infect Dis. Optimization of group size in pool testing strategy for
2020;3099:2019–20. https://doi.org/10.1016/S1473- SARS-CoV-2: a simple mathematical model. J Med Virol.
3099(20)30362-5 2020;Apr 24:1988–94. https://doi.org/10.1002/jmv.25929

102 Emerging Infectious Diseases • www.cdc.gov/eid • Vol. 27, No. 1, January 2021
 Nucleic Acid Amplification Tests for SARS-CoV-2

23. Cherif A, Grobe N, Wang X, Kotanko P. Simulation of pool oropharyngeal saliva samples and serum antibody
testing to identify patients with coronavirus disease 2019 responses during infection by SARS-CoV-2: an
under conditions of limited test availability. JAMA Netw observational cohort study. Lancet Infect Dis. 2020;20:565–74.
Open. 2020;3:e2013075. https://doi.org/10.1001/ https://doi.org/10.1016/S1473-3099(20)30196-1
jamanetworkopen.2020.13075 27. Wölfel R, Corman VM, Guggemos W, Seilmaier M,
24. Pan Y, Zhang D, Yang P, Poon LLM, Wang Q. Viral load Zange S, Müller MA, et al. Virological assessment of
of SARS-CoV-2 in clinical samples. Lancet Infect Dis. hospitalized patients with COVID-2019. Nature.
2020;20:411–2. https://doi.org/10.1016/S1473-3099(20)30113-4 2020;581:465–9. https://doi.org/10.1038/s41586-
25. Xu T, Chen C, Zhu Z, Cui M, Chen C, Dai H, et al. 020-2196-x
Clinical features and dynamics of viral load in imported
and non-imported patients with COVID-19. Int J Infect Dis. Address for correspondence: Benjamin A. Pinsky, Stanford
2020;94:68–71. https://doi.org/10.1016/j.ijid.2020.03.022
University School of Medicine, 3375 Hillview, Rm 2913, Palo Alto,
26. To KK, Tsang OT, Leung WS, Tam AR, Wu TC, Lung DC,
et al. Temporal profiles of viral load in posterior CA 94304, USA; email: [email protected]

May 2020

Respiratory Viruses
•S
 urveillance of Leprosy in Kiribati, •N
 onpharmaceutical Measures for
1935–2017 Pandemic Influenza in Nonhealthcare
Settings—Social Distancing Measures
•B
 iphasic Outbreak of Invasive Group
A Streptococcus Disease in Eldercare • Candidatus Rickettsia xinyangensis
Facility, New Zealand as Cause of Spotted Fever Group
Rickettsiosis, Xinyang, China, 2015
•E
 pidemiology of Tick-Borne Relapsing
Fever in Endemic Area, Spain •P
 retreatment Out-of-Pocket Expenses
•F
 ood Safety and Invasive Cronobacter for Presumptive Multidrug-Resistant
Infections during Early Infancy, Tuberculosis Patients, India, 2016–2017
1961–2018
•C
 apybara and Brush Cutter
•C
 linical Outcomes of Patients Treated Involvement in Q Fever Outbreak in
for Candida auris Infections in a Remote Area of Amazon Rain Forest,
Multisite Health System, Illinois, USA French Guiana, 2014

•M
 osquito Control Activities during •W
 omen’s Awareness and Healthcare
Local Transmission of Zika Virus, Provider Discussions about Zika Virus
Miami-Dade County, Florida, during Pregnancy, United States,
USA, 2016 2016–2017
•E
 ffectiveness of Live Poultry Market •G
 enetic Characterization of Japanese
Interventions on Human Infection with Encephalitis Virus Genotype 5 Isolated
Avian Influenza A(H7N9) Virus, China • Rhizopus microsporus Infections from Patient, South Korea, 2015
•N
 ationwide Monitoring for Plasmodium Associated with Surgical Procedures,
•U
 pdate on Ebola Treatment Center
falciparum Drug-Resistance Alleles Argentina, 2006–2014
Costs and Sustainability, United
to Chloroquine, Sulfadoxine, and States, 2019
•P
 ossible Transmission Mechanisms
Pyrimethamine, Haiti, 2016–2017
of Mixed Mycobacterium tuberculosis
Infection in High HIV Prevalence •A
 Neighbor-Based Approach to
•S
 ystematic Review and Meta-
Country, Botswana Identify Tuberculosis Exposure, the
Analysis of Sex Differences in Social
Kopanyo Study
Contact Patterns and Implications for
•N
 onpharmaceutical Measures for
Tuberculosis Transmission and Control •S
 pecies Distribution and Isolation
Pandemic Influenza in Nonhealthcare
•E
 ffects of Air Pollution and Other Settings—International Travel- Frequency of Nontuberculous
Environmental Exposures on Related Measures Mycobacteria, Uruguay
Estimates of Severe Influenza Illness,
•N
 onpharmaceutical Measures for •Z
 ika Inquiries Made to the
Washington, USA
Pandemic Influenza in Nonhealthcare CDC-INFO System, December
•E
 pidemiologic and Clinical Progression Settings—Personal Protective and 2015–September 2017
of Lobomycosis among Kaiabi Indians, Environmental Measures
Brazil, 1965–2019 •M
 ultidrug-Resistant Salmonella
•B
 lastomycosis in Minnesota, USA, Serotype Anatum in Travelers and
• Zika Virus Circulation in Mali 1999–2018 Seafood from Asia, United States

To revisit the May 2020 issue, go to:

https://wwwnc.cdc.gov/eid/articles/issue/26/5/table-of-contents
®

Emerging Infectious Diseases • www.cdc.gov/eid • Vol. 27, No. 1, January 2021 103