Detecting Network Effects:

Randomizing Over Randomized Experiments∗

Martin Saveski Jean Pouget-Abadie Guillaume Saint-Jacques
MIT Harvard University MIT
[email protected] [email protected] [email protected]

Weitao Duan Souvik Ghosh Ya Xu

LinkedIn LinkedIn LinkedIn
[email protected] [email protected] [email protected]

Edoardo M. Airoldi
Harvard University
[email protected]
ABSTRACT 1 INTRODUCTION
Randomized experiments, or A/B tests, are the standard approach Before deploying new features or products, it is common practice to
for evaluating the causal effects of new product features, i.e., treat- run randomized experiments—or A/B tests—to estimate the effects
ments. The validity of these tests rests on the “stable unit treatment of the suggested change. A part of the population is randomly
value assumption” (SUTVA), which implies that the treatment only selected to receive the new feature/product (treatment), while an-
affects the behavior of treated users, and does not affect the behav- other randomly selected part of the population is given the product
ior of their connections. Violations of SUTVA, common in features status quo (control). The goal of comparing the treatment popula-
that exhibit network effects, result in inaccurate estimates of the tion to the control population is to impute the difference between
causal effect of treatment. In this paper, we leverage a new experi- two universes that we cannot observe simultaneously: one where
mental design for testing whether SUTVA holds, without making everyone receives treatment and another where no one does. This
any assumptions on how treatment effects may spill over between imputation, formalized in the theory of causal inference [16], relies
the treatment and the control group. To achieve this, we simul- on a fundamental independence assumption, known as the “stable
taneously run both a completely randomized and a cluster-based unit treatment value assumption” (SUTVA) [9, 26]. It states that
randomized experiment, and then we compare the difference of the every user’s behavior is affected only by their own treatment and
resulting estimates. We present a statistical test for measuring the not by the treatment of any other user.
significance of this difference and offer theoretical bounds on the However, this independence assumption may not always hold,
Type I error rate. We provide practical guidelines for implementing particularly when testing products with social components that, by
our methodology on large-scale experimentation platforms. Im- design, connect users and allow them to interact with one another.
portantly, the proposed methodology can be applied to settings in Consider the example of testing a feed ranking algorithm: If user A
which a network is not necessarily observed but, if available, can and user B are connected, by changing the ranking of items on user
be used in the analysis. Finally, we deploy this design to LinkedIn’s A’s feed, we impact their engagement with their feed and indirectly
experimentation platform and apply it to two online experiments, change the items that appear on user B’s feed. User B may well have
highlighting the presence of network effects and bias in standard been placed in the control group, but their behavior was impacted
A/B testing approaches in a real-world setting. by the assignment of user A to the treatment group. This spillover
of treatment effects between users violates SUTVA and leads to
KEYWORDS inaccurate treatment effect estimates [7].
A/B testing, Network effects, Network interference, SUTVA. There are three common approaches to minimizing the effects
of spillovers: (i) Cluster-based randomization [2, 10, 34], where
users are clustered based on their connections and treatment is
∗ Martin Saveski and Jean Pouget-Abadie contributed equally to this work. assigned at a cluster level; (ii) Multi-level designs [15, 30], where
treatment is applied with different proportions; (iii) Designs and
analysis that assume specific models of interference [3, 8, 12, 28, 31],
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed e.g. that the interference effect is proportional to the number of
for profit or commercial advantage and that copies bear this notice and the full citation neighbors treated.
on the first page. Copyrights for components of this work owned by others than the While mitigating interference is the end goal of the field of causal
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specific permission inference with network interference, a precursor to that question is
and/or a fee. Request permissions from [email protected]. to detect whether SUTVA holds in the experiments we run. When
KDD’17, August 13-17, 2017, Halifax, NS, Canada we plan an experiment in which we suspect that SUTVA might be
© 2017 Copyright held by the owner/author(s). Publication rights licensed to ACM.
978-1-4503-4887-4/17/08. . . $15.00 violated, a common solution is to use cluster-based randomized
DOI: 10.1145/3097983.3098192
assignments, which minimize the number of edges cut between 2 THEORETICAL FRAMEWORK
treated and control units. Under certain assumptions [10], this In this section, we present our two-stage experimental design for
design can partially mitigate the problem of interference. However, testing whether SUTVA holds and the main theoretical results. We
in practice, cluster-based randomized assignments have higher vari- provide more complete exposition in [24].
ance than the completely randomized assignment, making it more
difficult to accurately estimate the treatment effect. Furthermore, 2.1 Two assignment strategies
cluster-based assignments require an appropriate choice of cluster-
First, we briefly review the notation and main results for the com-
ing of the population, which can be a challenging even if the graph
pletely randomized (CR) design and the cluster-based randomized
of interactions between units is known. If we have a robust way of
(CBR) design. Let G = (V ,E) be a graph of N units (|V | = N ),
testing whether SUTVA holds, we can decide whether the standard
for which we measure outcomes (Yi )i ∈V , and on which we can
completely randomized assignment is valid and whether running a
assign an intervention with two conditions: treatment and control.
cluster-based randomized assignment is necessary.
Each unit’s potential outcome is defined as a function of the entire
Traditionally, testing whether SUTVA holds has been done through
assignment vector Z ∈ {0, 1} N of units to treatment buckets: Yi (Z).
post-experiment analysis: Rosenbaum [25] was the first to state two
If SUTVA holds, Yi (Z) = Yi (Z i ). The causal estimand of interest is
sharp null hypotheses (i.e., there is no treatment effect on anyone)
the Total Treatment Effect (TTE) given by:
which imply that SUTVA does not hold. Under these restricted
null hypotheses, we can obtain exact distribution of network pa- 1 X
µ ..= [Yi (Z = 1) − Yi (Z = 0)]. (1)
rameters. More recent work [1, 4] explicitly tackles testing for N
i ∈V
the non-sharp null that SUTVA holds by considering the distribu-
Finally, for any vector u ∈ Rn , let u = n1 ni=1 ui and σ 2 (u) =
P
tion of chosen network effect parameters for a subpopulation of 1 Pn (u − ū) 2 .
the graph under SUTVA. While the final test is dependent on the n−1 i=1 i
choice of network parameter and subpopulation, the main appeal
Completely Randomized Assignment. In a completely random-
of this analysis-based approach is that it does not require running
ized experiment, we sample the assignment vector Z uniformly at
a modified experiment.
random from the set {z ∈ {0, 1} N : zi = nt }, where nt (resp. nc )
P
Rather than focusing on a post-experiment analysis approach,
we explore new designs tailored to test for network effects. The is the number of units assigned to treatment (resp. control). Let
main idea behind our approach is to simultaneously run a com- Yt ..= {Yi : Z i = 1} and Yc ..= {Yi : Z i = 0}. Recall the definition of
pletely randomized and a cluster-based randomized experiment on the difference-in-means estimator:
different, randomly selected, parts of the population and to compare µ̂cr ..= Yt − Yc .
the resulting treatment effect estimates. If the two estimates are Cluster-based Randomized Assignment. In a cluster-based ran-
very different, that is an indication of network effects. However, if domized assignment, we randomize over clusters of units in the
the two estimates are very similar, we expect SUTVA to hold. This graph, rather than individual units. We suppose that each unit in V
allows us to test whether SUTVA holds without making any mod- is assigned to one of m clusters. We sample the cluster assignment
eling assumptions of how units interact with each other. Moreover, vector z uniformly at random from {v ∈ {0, 1}m : vi = mt }. Units
P
if SUTVA holds, we are still able to estimate the treatment effect in cluster j are assigned to the same treatment bucket as cluster
using the part of the population that received a completely random- j: Z i = 1 ⇔ z j = 1 if i ∈ j, where mt (resp. mc ) is the number
ized assignment. If SUTVA does not hold, we can still rely on the of clusters assigned to treatment (resp. control). We let Y 0 be the
estimate obtained from the cluster-based randomized experiment. vector of aggregated potential outcomes, defined as Yj0 ..= i ∈j Yi ,
P
This work, together with [24], is part of a two-paper series. In
the sum of all outcomes within that cluster. Let Yt0 ..= {Yj0 : z j = 1},
[24], we introduce the methodology and main theoretical results,
while in this paper we present the practical implementation details, Yc0 ..= {Yj0 : z j = 0}. The aggregated difference-in-means estimator
and provide an in-depth analysis of the experimental results. In is given by:
m  0 
doing so, we make the following main contributions: µ̂cbr ..= Yt − Yc0 .
N
• We present the randomized experimental design for testing 2.2 A two-stage randomized design
whether SUTVA holds, detailed more thoroughly in [24], When SUTVA holds, µ̂cr and µ̂cbr are unbiased estimators of the to-
and present the essential results on variance estimation tal treatment effect under a completely randomized assignment and
and bounding the Type I error rate (Section 2). a cluster-based randomized assignment respectively [22]. When
• We provide detailed implementation guidelines to help SUTVA does not hold, their unbiasedness is no longer guaranteed.
practitioners deploy the proposed design on large-scale In fact, when interference is present, we expect the estimate of the
experimentation platforms. We discuss how to compute total treatment effect to be different under a completely randomized
balanced clustering at scale (Section 3.2), as well as how to design than under a cluster-based randomized design.
stratify to reduce variance (Section 3.3). Consider for example the case where only the unit’s immediate
• We deploy our framework on LinkedIn’s experimental plat- neighborhood affects their outcome:
form and we report the results of two large-scale experi-
ments (Section 3), achieving significance in one of them, ∀i, Z, Z 0 , [ ∀j ∈ N (i), Z i = Z j0 ] =⇒ Yi (Z) = Yi (Z 0 ), (2)
highlighting the presence of network effects. where N (i) ..= {j ∈ V : (i, j) ∈ E} be the neighborhood of i.
 CR VS CBR TREATMENT CONTROL

CR (W = cr) (Z = 1) (Z = 0)

µ̂cr
CR
2
ˆcr

CBR (W = cbr) (Z = 1) (Z = 0)
ˆ2

CBR
µ̂cbr

2
ˆcbr

(A) (B) (C) (D) (E)

Figure 1: Illustration of the proposed experimental design for detecting network effects. (A) Graph of all units and the connec-
tions between them; the dashed circles represent (equally-sized) clusters. (B) Assigning clusters to treatment arms: completely
randomized (CR) and cluster-based randomized assignment (CBR). (C) Assigning units to treatment buckets—treatment and
control—using the corresponding strategy. (D) Computing the treatment effect within each treatment arm: µ̂cr and µ̂cbr , and
2 and σ̂ 2 . (E) Computing the difference of the estimates from each treatment arm: ∆ = µ̂ − µ̂
variance: σ̂cr cbr cr cbr , and the total
2 2 + σ̂ 2 .
variance: σ̂ = σ̂cr cbr

In other words, if the assignment Z is such that a unit i’s neigh- assignment) respectively, and let ncr and ncbr = N − ncr
borhood is assigned entirely to treatment (resp. control), then we be the resulting number of units assigned to each arm.
observe Yi (Z) = Yi (1) (resp. Yi (Z) = Yi (0)), in which case we can (iii) Conditioned on W, we sample Zcr ∈ {0, 1}nc r using a com-
estimate the treatment effect for unit i. The probability of assign- pletely randomized assignment to assign units in treatment
ing the whole neighborhood of a unit to treatment or to control is arm CR to treatment and control. Let ncr,t and ncr,c be
very small under a completely randomized assignment. One way the number of units that we wish to assign to treatment
to increase this probability is to assign entire clusters of units to and control respectively.
treatment or to control (Section 2.1). Cluster-based randomization (iv) Still conditioned on W, we sample Zcbr using a cluster-
designs are used to reduce the bias under a completely random- based randomized assignment to assign units in treatment
ized assignment when interference is believed to happen primarily arm CBR to treatment and control. Let mcbr,t and mcbr,c
through each unit’s first-degree neighborhood. be the number of clusters assigned to treatment and control
Thus, the main idea behind this work is to set up a two-level ex- respectively.
perimental design to test for interference, such that different parts The resulting assignment vector Z of units to treatment and control
of the graph G will receive treatment through different random- is such that Zcr ⊥ ⊥ Zcbr |W.
ized strategies. By comparing the estimates from each randomized
strategy, we test the effect of the randomized strategy, which is null 2.3 Testing for the SUTVA null
under SUTVA. This is done by first assigning units to treatment
Next, we present a statistical test for accepting or rejecting the
arms and then, within each treatment arm, applying a specific as-
SUTVA null. We provide a more detailed, step by step, derivation
signment strategy to assign units to treatment buckets (Figure 1).
in [24]. We define the two estimates of the causal effect for this
The two-stage design works as follows:
experiment as follows:

(i) We initially cluster the graph G into m clusters C. Note µ̂cr (W, Z) ..= Y cr,t − Y cr,c , (3)
that we do not necessarily need to fully observe the graph, m
µ̂cbr (W, Z) ..= cbr Y 0cbr,t − Y 0cbr,c ,
 
(4)
we just need to have a meaningful clustering of the users. ncbr
(ii) We sample the units to treatment arms assignment vector where we have introduced the following notation:
W ∈ {cr ,cbr } N using a cluster-based randomized assign-
ment. We denote by ω ∈ {cr ,cbr }m the corresponding clus- Ycr,t ..= {Yi : Wi = cr ∧ Z i = 1},
ter assignment vector to treatment arms CR (ω j = cr ) and Ycr,c ..= {Yi : Wi = cr ∧ Z i = 0},
CBR (ω j = cbr ). Let mcr and mcbr be the number of clus- 0
Ycbr,t ..
= {Yj0 : ω j = cbr ∧ z j = 1},
ters assigned to treatment arms CR (completely random- 0 ..
ized assignment) and CBR (cluster-based randomization Ycbr,c = {Yj0 : ω j = cbr ∧ z j = 0}.
 In order to test whether the estimates of each arm are signifi- decreases as the number of edges cut by the initial clustering also
cantly different, we must divide the difference of the estimates by decreases, with all else being equal:
its variance under the null. It is uncommon in randomized experi-
Yi = β 0 + β 1Z i + β 2 ρ i + ϵi ,
ments to know the variance of the chosen estimators exactly, but
where ρ i = 1
we can usually settle for an empirical upper-bound. i ∈V |N (i) ∩ C (i)| / |N (i)|, with N (i) being the
P
N
We consider the following variance estimators, computable from neighborhood of unit i and C (i) being the units in i’s cluster.
the observed data: Finally, note that the results presented in this section hold re-
gardless of whether the true network of interactions between users
Sˆcr,t
2 .. Sˆcr,c
σ̂cr = + , (5) is observed. Knowing this network, at least partially, allows us to
ncr,t ncr,c find more meaningful clusters of users and increases our ability to
m2 Sˆcbr,t
0 Sˆcbr,c
0 detect network effects when they are present i.e. reduce the Type
2 ..
σ̂cbr = 2cbr *. + +/ , (6) II error. One can achieve similar results by using any network or
ncbr mcbr,t mcbr,c
, - domain knowledge deemed relevant for the experiment at hand,
where we introduced the following empirical variance quantities not necessarily the true network.
in each treatment arm and treatment bucket:
2.4 Incorporating Stratification
Sˆcr,t ..= σ 2 (Yi : Wi = cr ∧ Z i = 1), So far, we have used completely randomized assignment to assign
Sˆcr,c ..= σ 2 (Yi : Wi = cr ∧ Z i = 0), clusters to treatment arms and, in the CBR treatment arm, to assign
clusters to treatment buckets. Under this randomization strategy,
Sˆcbr,t
0 ..
= σ 2 (Yj0 : ω j = cbr ∧ z j = 1),
it is possible that—by chance—we may end up with very different
Sˆcbr,c
0 ..
= σ 2 (Yj0 : ω j = cbr ∧ z j = 0). populations in the two treatment arms, or in the CBR arm, differ-
ent treatment and control groups. For example, we may assign all
Finally, we consider the sum of these two empirical variance clusters with highly active users in the same treatment arm. Strati-
quantities: fication prevents such scenarios by design. Instead of randomizing
σ̂ 2 ..= σ̂cr
2 2
+ σ̂cbr . (7) all clusters at once, we first divide them into more homogeneous
groups (strata) and we randomize within each stratum. Stratifica-
As shown in [24], the following results holds when the clustering
tion has two key advantages: (i) it ensures that all covariates used
is balanced (i.e., all clusters have the same number of nodes):
to create the strata will be balanced, (ii) it improves the precision
E W,Z [µ̂cbr − µ̂cr ] = 0, (8) of the treatment effect estimator. In this section, we extend our test
2 for detecting network effects to incorporate stratification.
varW,Z [µ̂cr − µ̂cbr ] ≤ E W,Z [σ̂ ]. (9) Suppose that each graph cluster c ∈ C is assigned to one of S
Note that Eq. 9 is the only statement requiring the clustering to strata. In this section, we assume that the strata are given, but in
be balanced. When SUTVA holds, both estimators have the same Section 3.3 we show how to construct them. We denote by V (s)
expectation under their respective assignment strategy regardless of the nodes in the graph which belong to strata s. Within each strata
whether the clustering is balanced. In order to control the variance s ∈ [1,S], we assign mcr (s) clusters completely at random to the
however, as is done in Eq. 9, we restrict ourselves to balanced CR treatment arm and mcbr (s) clusters to the CBR treatment arm,
clusterings. There are now multiple ways in which we can reject denoted by vector W(s), sampled uniformly at random from vectors
{v ∈ {cr ,cbr } |V (s ) | : j I[v j = cr ] = mcr (s)}.
P
the null. If we reject the null that SUTVA holds when:
Let Zcr (s) be the assignment of units the treatment arm CR
| µ̂cr − µ̂cbr | 1 to treatment buckets within strata s. If Vcr (s) is the subset of V
√ ≥ √ , (10)
σ̂ 2 α in strata s assigned to treatment arm CR, Zcr (s) is chosen uni-
formly at random from the vectors {u ∈ {0, 1} |Vc r (s ) | : i ui =
P
then the type I error of our test is no greater than α if σ̂ 2 ≥
ncr,t (s)}. Similarly, let zcbr (s) be the assignment of clusters in
var[µ̂cr − µ̂cbr ], which we can only guarantee in expectation (Eq. 9).
treatment arm CBR to the treatment buckets within strata s. If
A less conservative approach is to suppose that the quotient follows
Ccbr (s) is the subset of clusters in strata s assigned to treatment
a normal distribution N (0, 1), for which we obtain (1 − α ) × 100%
arm CBR, zcbr (s) is chosen uniformly at random from the vectors
confidence intervals:
{v ∈ {0, 1} | Ccbr (s ) | : j v j = mcbr,t (s)}.
P
CI 1−α (T ) = T − z α ,T + z 1− α ,
 
(11) Let ncbr (s) be the total number of units assigned to treatment
2 2
arm CBR and mcbr (s) be the total number of clusters assigned to
α
where z α and z 1− α are the quantile of the standard normal
2 2 2 treatment arm CBR within strata s. We can extend the previous
distribution. estimators of the average treatment effect under stratification. Let
While bounding the Type I error of our test allows us to assume
SUTVA under which the moments of our test statistic become Ycr,t (s) ..= {Yi : i ∈ Vcr (s) ∧ Z i = 1},
tractable, the same cannot be said of the Type II error, where we Ycr,c (s) ..= {Yi : i ∈ Vcr (s) ∧ Z i = 0},
must assume a specific interference model. We refer the reader to 0
Ycbr,t (s) ..= {Yj0 : j ∈ Ccbr (s) ∧ z j = 1},
[24], where we show that under the following popular model of 0
interference, the Type II error rate of our suggested design and test Ycbr,c (s) ..= {Yj0 : j ∈ Ccbr (s) ∧ z j = 0}.
 STRATIFICATION CR VS CBR TREATMENT CONTROL

STRATA 1 CR µ̂cr (s1 )

2
ˆcr (s1 ) (s1 )
CBR µ̂cbr (s1 ) ˆ 2 (s1 )
2
ˆcbr (s1 )
STRATA 2 CR µ̂cr (s2 )
2
ˆcr (s2 ) (s2 )
CBR µ̂cbr (s2 ) ˆ 2 (s2 ) ˆ2
2
ˆcbr (s2 )
STRATA 3 CR µ̂cr (s3 )
2
ˆcr (s3 ) (s3 )
CBR µ̂cbr (s3 ) ˆ 2 (s3 )
2
ˆcbr (s3 )

(A) (B) (C) (D) (E) (F) (G)

Figure 2: Illustration of the proposed experimental design for detecting network effects, using stratification to reduce variance
and improve covariate balance. (A) Graph of all units and their connections; the dashed circles represent clusters. (B) Assigning
clusters to strata (Section 2.4), (C) Assigning clusters within each strata to treatment arms: completely randomized (CR) and
cluster-based randomized assignment (CBR). (D) Assigning units within strata to treatment buckets: treatment and control,
using corresponding assignment strategy. (E) Computing the treatment effects of each treatment arm within each strata: µ̂cr (s)
2 (s) and σ̂ 2 (s). (F) Computing the difference between the estimated effects
and µ̂cbr (s), and variance within each strata: σ̂cr cbr
using CR and CBR within each strata: ∆(s), and sum variances in each strata: σ̂ 2 (s). (G) Aggregating the differences across
strata to compute the overall difference in differences (∆) and the total variance (σ̂ 2 ).

The stratified estimators are given by: changes to the majority of their user base. As a result, these compa-
nies have each built mature experimentation platforms. However,
..
µ̂cr (s) Ycr,t (s) − Ycr,c (s)
= how many of the experiments run on these platforms violate SUTVA
m (s)  0 is an open question. Together with the team running LinkedIn’s
µ̂cbr (s) ..= cbr

Ycbr,t (s) − Ycbr,c
0 (s)
ncbr (s) experimentation platform, we applied the proposed theoretical
∆(s) ..= µ̂cr (s) − µ̂cbr (s). framework to test for interference in two randomized experiments
on LinkedIn.
Note that for every strata s, the following quantity σ̂ 2 (s) upper- LinkedIn users can interact with content posted by their connec-
bounds varW(s ),Z(s ) [∆(s)] in expectation: tions through an algorithmically sorted feed. To improve the user
experience, teams at LinkedIn continually modify the feed ranking
2 ˆ0 Sˆcbr,c
Sˆcr,t (s) Sˆcr,c (s) mcbr (s) * Scbr,t (s) (s)
0
algorithm and seek to determine the impact of these changes on
σ̂ 2 (s) ..= + + 2 + +/ .
ncr,t (s) ncr,c (s) n (s) mcbr,t (s) mcbr,c (s) key user metrics by running randomized experiments. Experiments
.
cbr , - of this kind are a typical case where the treatment effects may spill
Since the assignment of units to treatment is independent across over between the treatment and control units: if a user is assigned
strata, we can extend the previous test across strata by considering to an effective treatment then they are more likely to interact with
the following numerator and denominator: the feed, which in turn will impact the feeds of their connections.
X m(s) The goal of our experiments is to determine whether these spillover
∆= ∆(s) effects significantly bias the treatment effect estimators or SUTVA
M
s ∈[1,S ] can be safely assumed.
X m(s) ! 2 In remainder of this section, we provide a detailed, step by step,
σ̂ 2 = σ̂ 2 (s) description of the deployment of our design on LinkedIn’s exper-
M
s ∈[1,S ] imentation platform. We show how to find balanced clusters at
We refer the reader to Figure 2 for an illustration. scale, evaluating four different algorithms (Section 3.2); how to con-
struct strata based on cluster covariates in order to ensure covariate
balance and reduce the variance of the treatment effect estimates
3 EXPERIMENTS ON LINKEDIN’S PLATFORM
(Section 3.3); how to use lagged outcomes to reduce variance even
3.1 Experimental Scenario further (Section 3.4); and finally, we report the results of two exper-
Major Internet companies like Google [29], Microsoft [18], Face- iments testing different feed ranking algorithms (Section 3.5). The
book [5], or LinkedIn [35] rely heavily on experimentation to un- guidelines provided in this section are not specific to LinkedIn and
derstand the effects of each product decision, before deploying any can be applied to any large-scale experimentation platform.
3.2 Clustering the graph Table 1: Evaluation of the different balanced clustering algo-
rithms. We report the percentage of within-cluster edges per
The main challenge of implementing the proposed test for inter-
clustering of the Netherlands LinkedIn graph. The values in
ference is finding a clustering algorithm that (i) finds clusters of
bold represent the best performance. For BLP, we report re-
highly inter-connected nodes, (ii) generates balanced clusters, and
sults only for k = 100 and k = 300, since the running times of
(iii) scales to the size of the LinkedIn graph. The LinkedIn graph
one iteration for larger values of k were too long.
contains hundreds of millions of nodes and billions of edges. As
a result, we restrict our search to parallelizable streaming algo-
rithms. Furthermore, because of the way social networks tend to be Number of
BLP reFENNEL reLDG METIS
structured, most clustering (community detection) techniques find clusters (k)
clusters with highly skewed size distribution [13, 19]. We therefore 100 26.7% 31.7% 35.6% 35.0%
further restrict our search to clustering algorithms which explicitly 300 22.7% 27.7% 29.9% 29.4%
enforce balance. We report the results of our evaluation of the 500 - 26.1% 27.7% 27.0%
relevant state-of-the-art clustering algorithms. Before presenting 1000 - 23.9% 24.7% 23.8%
the empirical results, we provide a brief justification of why we
need balanced clustering.
a node to another cluster, (ii) solving a Linear Program to find an
Why balanced clustering is necessary. As stated in Section 2.3, optimal relocation of nodes while maintaining balance, and (iii)
having clusters of equal size simplifies the theoretical analysis of moving the nodes to the desired clusters according to the relocation
the variance of our estimators under the null hypothesis. There found in step ii. Note that step i and iii can be easily parallelized
are also two main practical reasons for partitioning the graph into and ran in streaming fashion. Step ii requires solving an LP with
clusters of equal size: (i) variance reduction, and (ii) balance on linear number of variables and quadratic number of constraints
pre-treatment covariates. w.r.t. the number of clusters.
First, clusters of equal size will tend to have more similar aggre-
gated outcomes per cluster (Y 0 ), which leads to smaller variance of Restreaming Linear Deterministic Greedy (reLDG) [23] is a restream-
2 . In particular, the values of Sˆ0
the estimator, σ̂cbr and Sˆcbr,c
0 in
cbr,t ing version of the Linear Deterministic Greedy algorithm [27].
Equation 6 will tend to be smaller. Nishimura and Ugander [23] show that restreaming significantly
Second, due to homophily [21], users who are connected will increases the quality of the clusters compared to a single pass. LDG
tend to be more similar to each other, leading to homogeneous assigns each node u to a cluster i according to:
clusters. Thus, if the clusters are not balanced, then large clusters of |Ci |
!
similar users will tend to dominate the treatment/control population arg max |Ci ∩ N (u)| 1 − , (12)
i ∈1...k C
(depending on where the clusters were randomly assigned), making
it harder to achieve balance on pre-treatment covariates. where C (i) is the set of all nodes in cluster i in the most recent
Thus, even if unbalanced clustering results in clusters of higher assignment, N (u) is the set of neighbors of u, and C is the capacity
quality, in terms of number of edges cut, that will not necessarily (i.e., maximum number of nodes) allocated for each cluster (usually
help us to achieve our ultimate goal of detecting network effects. set to nk to achieve perfect balance). The first term maximizes the
Finally, note that these observations are not specific to our design, number of edges within clusters, while the second term enforces
but hold for any cluster-based randomized design. balance on the cluster sizes.

Evaluation of balanced clustering algorithms. To test different Restreaming FUNNEL (reFUNNEL) [23] is a restreaming version
clustering algorithms, we extracted a subgraph of the full LinkedIn of the FUNNEL algorithm [32], which is itself a streaming gen-
graph containing only active members from the Netherlands and eralization of the modularity maximization. It assigns nodes to
the connections between them. The subgraph contains >2.5M nodes clusters as:
and >300M edges. We picked the Netherlands because (i) it is a arg max |Ci ∩ N (u)| − α |Ci |,
i ∈1...k
tightly-connected graph, and (ii) it fits in memory on a single ma-
chine, which allowed us to compare the streaming algorithms to where α is a hyper-parameter. Note that, unlike LDG, FUNNEL
state-of-the-art batch algorithms. We tested four algorithms. ensures only approximate balance, unless α ≥ d nk e. Nishimura
and Ugander [23] suggest increasing α in each restreaming pass
METIS [17] is a widely-used batch graph clustering algorithm, and to achieve best results. We run with linearly and logarithmically
thus serves as our baseline to compare the quality of the clustering increasing schedules.
achieved by the streaming algorithms. It consists of three phases: (i) We set the number of clusters to k = {100, 300, 500, 1000}. For
coarsening of the original graph, (ii) partitioning of the coarsened each algorithm and value of k, we measured the percentage of edges
graph, (iii) uncoarsening of the partitioned graph. found within the clusters (Table 1). We ran BLP for 10 iterations,
and reLDG and reFUNNEL for 20 iterations. In both cases this was
Balanced Label Propagation (BLP) [33] is an iterative algorithm that enough for the algorithms to converge. We found that for larger
greedily maximizes edge locality by (i) given the current cluster numbers of clusters (k ≥ 300) running one iteration of the BLP
assignment, determining the reduction in edges cut from moving algorithm using the GLPK solver (GNU Linear Programming Kit)
takes more than a day. It is worth noting that the bottleneck of the
Table 2: Results of clustering the full LinkedIn graph. We Table 3: The effect of stratification on pre-treatment vari-
ran the parallel version of reLDG on 350 Hadoop nodes for ance. We report the empirical variance of the difference-in-
35 iterations. difference-in-means estimator (∆) using the pre-treatment
outcomes. To avoid disclosing raw numbers all values are
Number of Percantage of edges Cluster sizes multiplied by a constant.
clusters (k) within clusters (mean ± std)
1000 35.59% 43893.2 ± 634.5 Number of Balanced k-means
No stratification
3000 28.54% 14631.1 ± 109.3 clusters (k) stratification
5000 26.16% 8778.6 ± 199.3 1000 0.890 1.000
7000 22.77% 6270.5 ± 40.5 3000 0.592 0.650
10000 21.09% 4389.3 ± 67.2 5000 0.590 0.545
7000 0.445 0.451
10000 0.400 0.372
algorithm is solving the LP, which depends on the number of clus-
ters, rather than the size of the graph. reLDG and reFUNNEL do not we do have historical data about the past behavior of the users,
have this limitation as their running time, for a single pass, is O (nk ) including the pre-treatment outcomes (the key metric of interest
and, in practice, larger values of k do not significantly increase the just before launching the experiment). Although, we hope that the
running time. In terms of clustering quality, reLDG consistently treatment will significantly increase the outcome, we do expect
outperforms all other algorithms (Table 1), including METIS which that the pre-treatment outcomes will be highly correlated with the
requires the full graph to be loaded in memory. reFUNNEL per- post-treatment outcomes. Note that even in the worst-case scenario
forms worse than reLDG and METIS, except for k = 1000 when when the selected covariates fail to predict the post-treatment out-
it achieves similar clustering quality as METIS. Finally, BLP lags comes, the treatment effect estimates still remain unbiased and
behind, performing significantly worse than all other methods. have no more sampling variance than the estimates we would have
obtained using a completely randomized assignment [14].
Clustering the full graph. Next, we apply the reLDG—the best We describe each cluster using four covariates: number of edges
performing balanced clustering algorithm—on the full LinkedIn within the cluster, number of edges to other clusters, and two
graph containing hundreds of millions of nodes and billions of metrics that characterize users’ engagement with the LinkedIn
edges. As mentioned above, reLDG’s running time is O (nk ) and feed averaged over all users in the cluster (one of which is the pre-
can be easily parallelized [23]. We ran the parallel version of reLDG treatment outcome). To group clusters into strata, we used balanced
on 350 Hadoop nodes for 35 iterations. We set k = {1000, 3000, k-means clustering. We experimented with two algorithms: [20]
5000, 7000, 10000} and a leniency of 1% for the balance constraint, to led to more balanced groups of clusters (strata) but does not scale to
slightly sacrifice balance for better clustering quality. We find that more than 5000 data points, whereas [6] is faster, more scalable but
even when clustering the graph in 1000 clusters more than one third outputs clusters (strata) that are slightly less balanced. We report
(35.59%) of all edges are between nodes within the same clusters results only for the latter.
(Table 2). Expectedly, as we increase k, the number of edges within We cannot measure the effects of stratification on the post-
clusters decreases, with k = 10000 having 21.09% of the edges treatment variance since we can run the experiment only once,
within clusters. We observe that most clusters are of similar size, either with or without stratification. However, we can measure
except for very few clusters that are smaller due to the algorithm’s the effects on the pre-treatment variance. Table 3 shows the pre-
leniency. We also looked at the distributions of the number of edges treatment variance of the difference-in-difference-in-means esti-
within each cluster (proxy for possible network effects) and the mator under our design with and without stratification. For small
number of edges to other clusters (proxy for possible spillovers). values of k = {1000, 3000} stratifying increases the variance. How-
We find that although there is some heterogeneity between the ever, for larger values of k = {5000, 7000, 10000} stratification leads
clusters, there are very few outliers (Table 2). Finally, we analyzed to smaller or similar variance.
the sensitivity of clustering quality to different initializations. For
k = {3000, 5000}, we run reLDG with four random initializations 3.4 Variance reduction using lagged outcomes
and, in both cases, we found very small differences—with standard
deviation of 0.03%—between different runs. In order to further reduce the variance of the estimator of the
network effect, we define a new variable as the difference between
post-treatment and pre-treatment outcomes, as suggested in [11]:
3.3 Stratifying the clusters
To ensure balance on cluster-level covariates and to reduce the Yi∗ = Yi,t − Yi,t −1 ,
variance of the treatment effect estimates—as discussed in Sec- where Yi,t is the outcome of unit i at period t and Yi,t −1 is the
tion 2.4—we use stratification to assign clusters to treatment arms, outcome of unit i one period unit prior to t. Since this is only a
and in the CBR treatment arm, clusters to treatment buckets. Strati- question of choosing the appropriate outcome variable, this does
fication produces the greatest gains in variance reduction when the not change the validity of our procedure. In practice, we chose 2
covariates used to stratify are predictive of the outcomes. While and 4 weeks as the difference between t and t − 1 in the first and
we cannot observe the outcomes before we run the experiment, the second experiment, respectively.
Table 4: Results of two online experiments testing for network effects ran on 20% and 36% of all LinkedIn users. The outcomes
are related to the level of users’ engagement with the feed. We report results pre-treatment, post-treatment, and post-treatment
using the variance reduction technique explained in Section 3.4. We refer to the first treatment arm as BR, instead of CR, since
we used a Bernoulli randomized assignment instead of a completely randomized assignment (Section 3.5). To avoid disclosing
raw numbers all values are multiplied by a constant, except for the final row which displays the two-tailed p-value of the test
statistic T under assumption of normality T ∼ N (0, 1).

Experiment 1 Experiment 2
post-treatment post-treatment
Statistic pre-treatment post-treatment pre-treatment post-treatment
(Y = Yt − Yt−1 ) (Y = Yt − Yt−1 )
BR Treatment Effect (µ̂br ) -0.0261 0.0432 0.0559 0.0230 0.2338 0.2108
CBR Treatment Effect (µ̂cbr ) 0.0638 0.1653 0.0771 0.2733 0.8123 0.5390
Delta (∆ = µ̂br − µ̂cbr ) -0.0899 -0.1221 -0.0211 -0.2504 -0.5785 -0.3281
BR standard deviation (σ̂br ) 0.0096 0.0098 0.0050 0.3269 0.3414 0.2911
CBR standard deviation (σ̂cbr ) 0.0805 0.0848 0.0260 0.9332 0.9966 0.5613
Delta standard deviation (σ̂ ) 0.0811 0.0856 0.0265 0.9367 1.0000 0.5712
p-value (2-tailed) 0.2670 0.1530 0.4246 0.5753 0.2560 0.0483

3.5 Experimental results Prior to launching the experiments, we ran a series of balance
We ran two separate experiments on the LinkedIn graph, in August checks on the background characteristics of the users to ensure
and November 2016, respectively. The experiments tested two sepa- that there are no systematic differences between the populations
rate changes in the feed ranking algorithm. Note that the treatment in the two treatment arms and between the treatment and control
did not have any trivial network effect, e.g, it did not specifically groups within each arm. We compared the distributions of number
encourage users to interact with each other. The outcome of interest of connections per user, levels of user activity, and the number of
is the number of sessions in which the user actively engages with premium member accounts.
their feed. We report the results of the two experiments in Table 4. To
avoid disclosing raw numbers, we multiply all values by a constant,
Practical Considerations. In order to address some of the chal- expect for the last row, which displays the two-tailed p-value.
lenges of running experiments in real-time on a non-fixed popula- Experiment 1. We ran the experiment for two weeks on 20% of
tion, the LinkedIn experimentation platform [35] is implemented to the LinkedIn users: 10% assigned to Bernoulli randomization (BR)
run Bernoulli randomized assignments (BR) and not completely ran- and 10% assigned to cluster-based randomization (CBR). We first
domized assignments. A Bernoulli randomized assignment assigns test whether there is any systematic bias of the outcomes in the
each unit to treatment or control with probability p independently assignment prior to experiment. We apply the test for network
at random, whereas a completely randomized assignment (CR) as- effects on pre-treatment outcomes (A/A test) and, as expected, we
signs exactly nt = bp · N c units to treatment, chosen uniformly at do not find any significant effects. Next, we test for network effects
random. For large sample sizes, as it is the case in our experiments, post-treatment: we fail to reject the null hypothesis that SUTVA
the difference between a CR and BR assignment is negligible for the holds. The treatment effects within each arm were not as significant
purpose of our test. The edge cases where no units are assigned to as expected, which potentially led to smaller networks effects and
treatment or control are very unlikely when N is large. In [24], we not enough evidence for our test to reject the null. We observe that
provide a formal explanation for why running a Bernoulli random- most of the variance comes from the CBR treatment arm. Using the
ized assignment does not affect the validity of our test in practice. lagged outcomes (Y = Yt − Yt−1 ) reduces the variance (σ̂ ) 3.2 times,
With these constrains in mind, we run our experiments as follows: but also reduces the difference in treatment effects (µ̂br − µ̂cbr ) 5.8
times. We still fail to reject the null.
(i) We cluster the LinkedIn graph into balanced clusters (Sec-
tion 3.2). We set the number of clusters to 3000 in the first Experiment 2. In this experiment, we used a larger test population,
experiment and to 10000 in the second experiment. 36% of all LinkedIn users: 20% assigned to Bernoulli randomization
(ii) We stratify the clusters based on their covariates using (BR) and 16% assigned to cluster-based randomization (CBR). We
balanced k-means stratification (Section 3.3). also ran the experiment for a longer period of time: 4 weeks. As in
(iii) In each stratum, we randomly assign clusters to the CBR experiment 1, we first test for network effects using pre-treatment
treatment arm, and subsequently to the treatment or con- outcomes (A/A test) and we do not find significant effects. Testing
trol bucket. post-treatment, we also fail to reject the null. However, by applying
(iv) Units that were not assigned to the CBR treatment arm are the variance reduction technique described in Section 3.4, we reduce
passed to the main experimentation pipeline, where a sub- the standard deviation (σ̂ ) 1.8 times, while also reducing the differ-
population is first sampled at random, and then assigned ence in treatment effects (µ̂br − µ̂cbr ) 1.8 times. We find significant
to treatment or control using Bernoulli randomization. network effects: we reject the null that SUTVA holds (p = 0.048).
 Although we did not reject the null hypothesis twice, in both [8] Guillaume W Basse and Edoardo M Airoldi. 2015. Optimal design of experiments
experiments we found positive network effects: the difference-in- in the presence of network-correlated outcomes. preprint arXiv:1507.00803 (2015).
[9] David Roxbee Cox. 1958. Planning of experiments. (1958).
means estimate was higher in the cluster-based randomization [10] Eckles Dean, Karrer Brian, and Ugander Johan. 2017. Design and Analysis of
treatment arm (µ̂cbr ) than the Bernoulli randomization one (µ̂br ). Experiments in Networks: Reducing Bias from Interference. Journal of Causal
Inference (2017).
Given the nature of the outcome variable, this behavior is expected. [11] Alex Deng, Ya Xu, Ron Kohavi, and Toby Walker. 2013. Improving the sensitivity
In fact, the outcome of interest, which measures the number of of online controlled experiments by utilizing pre-experiment data. In Proceedings
sessions in which a user engages with an item on their feed, yields of the ACM international conference on Web search and data mining.
[12] Laura Forastiere, Edoardo M Airoldi, and Fabrizia Mealli. 2016. Identification
itself to positive interference effects: the more of user’s connections and estimation of treatment and interference effects in observational studies on
engage with items on their feed, the more content to engage on networks. preprint arXiv:1609.06245 (2016).
the user’s feed there will be. The fact that we rejected the null [13] Santo Fortunato. 2010. Community detection in graphs. Physics reports (2010).
[14] Alan S Gerber and Donald P Green. 2012. Field experiments: Design, analysis,
in the second experiment, but not in the first, suggests that small and interpretation. WW Norton.
treatment effects in the first experiment were likely the reason why [15] Michael G Hudgens and M Elizabeth Halloran. 2008. Toward Causal Inference
with Interference. J. Amer. Statist. Assoc. (2008).
no strong interference effects were observed. [16] Guido W Imbens and Donald B Rubin. 2015. Causal Inference in Statistics, Social,
and Biomedical Sciences. Cambridge University Press.
4 FUTURE WORK [17] George Karypis and Vipin Kumar. 1998. Multilevel k-way partitioning scheme
for irregular graphs. Journal of Parallel and Distributed computing (1998).
This work opens many avenues for future research. In this section, [18] Ron Kohavi, Alex Deng, Brian Frasca, Toby Walker, Ya Xu, and Nils Pohlmann.
we highlight a few. 2013. Online controlled experiments at large scale. In Proceedings of ACM SIGKDD
international conference on Knowledge discovery and data mining.
First, the proposed design compares a cluster-based randomized [19] Jure Leskovec, Kevin J Lang, Anirban Dasgupta, and Michael W Mahoney. 2009.
assignments with a completely randomized assignment. A very Community structure in large networks: Natural cluster sizes and the absence
of large well-defined clusters. Internet Mathematics (2009).
similar test could be investigated where instead of comparing these [20] Mikko I Malinen and Pasi Fränti. 2014. Balanced k-means for clustering. In Joint
two designs, each treatment arm assigns different proportions of IAPR International Workshops on Statistical Techniques in Pattern Recognition
treated units. An interesting result would be to understand how the (SPR) and Structural and Syntactic Pattern Recognition (SSPR).
[21] Miller McPherson, Lynn Smith-Lovin, and James M Cook. 2001. Birds of a feather:
Type I and Type II error of these two hierarchical designs compare. Homophily in social networks. Annual review of sociology (2001).
Second, there is a growing literature on identifying heteroge- [22] Joel A Middleton and Peter M Aronow. 2011. Unbiased estimation of the average
neous treatment effect, which we believe can be adapted to this treatment effect in cluster-randomized experiments. (2011).
[23] Joel Nishimura and Johan Ugander. 2013. Restreaming graph partitioning: simple
framework. Cursory analysis of the experiments run showed that versatile algorithms for advanced balancing. In Proceedings of the ACM SIGKDD
interference effects seemed strongest for moderate users of the international conference on Knowledge discovery and data mining.
[24] Jean Pouget-Abadie, Martin Saveski, Guillaume Saint-Jacques, Weitao Duan, Ya
LinkedIn platforms, but were weaker for very recurrent users of Xu, Souvik Ghosh, and Edoardo Airoldi. 2017. Testing for arbitrary interference
LinkedIn and less-recurrent users. on experimentation platforms. preprint arXiv:1704.01190 (2017).
Finally, as mentioned at the end of Section 2, the type II error of [25] Paul R Rosenbaum. 2007. Interference between units in randomized experiments.
J. Amer. Statist. Assoc. (2007).
our test is strongly dependent on how “good” our initial clustering [26] Donald B Rubin. 1990. Formal mode of statistical inference for causal effects.
of the graph is. With all else being equal, this means cutting fewer Journal of statistical planning and inference (1990).
(possibly weighted) edges of the graph. A follow-up to our work [27] Isabelle Stanton and Gabriel Kliot. 2012. Streaming graph partitioning for large
distributed graphs. In Proceedings of the ACM SIGKDD international conference
would be to explore clustering algorithms which manage both on Knowledge discovery and data mining.
objectives of minimizing edges cut but also managing the final [28] Daniel L Sussman and Edoardo M Airoldi. 2017. Elements of estimation theory for
causal effects in the presence of network interference. preprint arXiv:1702.03578
empirical variance. (2017).
[29] Diane Tang, Ashish Agarwal, Deirdre O’Brien, and Mike Meyer. 2010. Overlap-
ACKNOWLEDGMENTS ping experiment infrastructure: More, better, faster experimentation. In Proceed-
ings of the ACM SIGKDD international conference on Knowledge discovery and
We would like to thank Guillaume Basse, Igor Perisic, Edmond data mining.
Awad, and Dean Eckles for useful comments and discussions. This [30] Eric J Tchetgen Tchetgen and Tyler J VanderWeele. 2012. On causal inference in
the presence of interference. Statistical Methods in Medical Research (2012).
work was partially done while Martin Saveski and Guillaume Saint- [31] Panos Toulis and Edward Kao. 2013. Estimation of Causal Peer Influence Effects.
Jacques were interns at LinkedIn. In International Conference on Machine Learning.
[32] Charalampos Tsourakakis, Christos Gkantsidis, Bozidar Radunovic, and Milan
Vojnovic. 2014. Fennel: Streaming graph partitioning for massive scale graphs. In
REFERENCES Proceedings of the ACM international conference on Web search and data mining.
[1] Peter M Aronow. 2012. A general method for detecting interference between [33] Johan Ugander and Lars Backstrom. 2013. Balanced label propagation for parti-
units in randomized experiments. Sociological Methods & Research (2012). tioning massive graphs. In Proceedings of the ACM international conference on
[2] Peter M Aronow, Joel A Middleton, and others. 2013. A class of unbiased Web search and data mining.
estimators of the average treatment effect in randomized experiments. Journal [34] Johan Ugander, Brian Karrer, Lars Backstrom, and Jon Kleinberg. 2013. Graph
of Causal Inference (2013). cluster randomization: Network exposure to multiple universes. In Proceedings
[3] Peter M Aronow and Cyrus Samii. 2013. Estimating average causal effects under of the ACM SIGKDD international conference on Knowledge discovery and data
interference between units. preprint arXiv:1305.6156 (2013). mining.
[4] Susan Athey, Dean Eckles, and Guido W Imbens. 2016. Exact P-values for [35] Ya Xu, Nanyu Chen, Addrian Fernandez, Omar Sinno, and Anmol Bhasin. 2015.
Network Interference. J. Amer. Statist. Assoc. (2016). From infrastructure to culture: A/B testing challenges in large scale social net-
[5] Eytan Bakshy, Dean Eckles, and Michael S Bernstein. 2014. Designing and works. In Proceedings of the ACM SIGKDD International Conference on Knowledge
deploying online field experiments. In Proceedings of the international conference Discovery and Data Mining.
on World wide web.
[6] Arindam Banerjee and Joydeep Ghosh. 2006. Scalable clustering algorithms with
balancing constraints. Data Mining and Knowledge Discovery (2006).
[7] Guillaume Basse and Edoardo Airoldi. 2017. Limitations of design-based causal
inference and A/B testing under arbitrary and network interference. preprint
arXiv:1705.05752 (2017).