Algorithms, games, and evolution

Erick Chastaina, Adi Livnatb, Christos Papadimitriouc,1, and Umesh Vaziranic

a
Computer Science Department, Rutgers University, Piscataway, NJ 08854; bDepartment of Biological Sciences, Virginia Tech, Blacksburg, VA 24061; and
c
Computer Science Division, University of California, Berkeley, CA, 94720

Contributed by Christos Papadimitriou, April 12, 2014 (sent for review November 16, 2013)

Even the most seasoned students of evolution, starting with Darwin theory, namely strategic individual behavior and conflict between
himself, have occasionally expressed amazement that the mecha- individuals, are extraneous to our analysis.
nism of natural selection has produced the whole of Life as we see We now state our assumptions and results. We consider an
it around us. There is a computational way to articulate the same infinite panmictic population of haplotypes involving several un-
amazement: “What algorithm could possibly achieve all this in linked (i.e., fully recombining) loci, where each locus has several
a mere three and a half billion years?” In this paper we propose alleles. These assumptions are rather standard in the literature.
an answer: We demonstrate that in the regime of weak selection, They are made here to simplify exposition and algebra, and there is
the standard equations of population genetics describing natural no a priori reason to believe that they are essential for the results,
selection in the presence of sex become identical to those of a re- beyond making them easily accessible. For example, Nagylaki’s
peated game between genes played according to multiplicative theorem (4), which is the main analytical ingredient of our results,
weight updates (MWUA), an algorithm known in computer science holds even in the presence of diploidy and partial recombination.
to be surprisingly powerful and versatile. MWUA maximizes a Nagylaki’s theorem states that weak selection in the presence
tradeoff between cumulative performance and entropy, which of sex proceeds near the Wright manifold, where the population
suggests a new view on the maintenance of diversity in evolution. genetic dynamics becomes (SI Text)

coordination games | learning algorithms 1 t
t 

xit+1 ð jÞ = x ð jÞ Fi ð jÞ ;
Xt i

P recisely how does selection change the composition of the

gene pool from generation to generation? The field of pop-
ulation genetics has developed a comprehensive mathematical
where xit ð jÞ is the frequency of allele j of locus i in the population at
generation t, X is a normalizing constant to keep the frequencies
summing to 1, and Fit ð jÞ is the mean fitness at time t among gen-
framework for answering this and related questions (1). Our
otypes that contain allele j at locus i (see ref. 4 and SI Text). Under
analysis in this paper focuses particularly on the regime of weak
the assumption of weak selection, the fitnesses of all genotypes are
selection, now a widely used assumption (2, 3). Weak selection close to one another, say within the interval [1 − «, 1 + «], and so
assumes that the differences in fitness between genotypes are small the fitness of genotype g can be written as Fg = 1 + «Δg, where « is
relative to the recombination rate, and consequently, through a the selection strength, assumed here to be small, and Δg ∈ [−1, 1]
result due to Nagylaki et al. (4) (see also ref. 1, section II.6.2), can be called the differential fitness of the genotype. With this in
evolution proceeds near linkage equilibrium, a regime where the mind, the equation above can be written
probability of occurrence of a certain genotype involving various
alleles is simply the product of the probabilities of each of its 1 t

xit+1 ð jÞ = x ð jÞ 1 + eΔti ð jÞ ; [1]
alleles. Based on this result, we show that evolution in the regime Xt i
of weak selection can be formulated as a repeated game, where
the recombining loci are the players, the alleles in those loci are where Δti ð jÞ is the expected differential fitness among genotypes
the possible actions or strategies available to each player, and the that contain allele j at locus i (see Fig. 1 for an illustration of
expected payoff at each generation is the expected fitness of an population genetics at linkage equilibrium).
organism across the genotypes that are present in the population.
Moreover, and perhaps most importantly, we show that the equa- Significance
tions of population genetic dynamics are mathematically equivalent
to positing that each locus selects a probability distribution on Theoretical biology was founded on the mathematical tools of
alleles according to a particular rule which, in the context of the statistics and physics. We believe there are productive con-
theory of algorithms, game theory, and machine learning, is nections to be made with the younger field of theoretical com-
known as the multiplicative weight updates algorithm (MWUA). puter science, which shares with it an interest in complexity and
MWUA is known in computer science as a simple but surpris- functionality. In this paper, we find that the mathematical de-
ingly powerful algorithm (see ref. 5 for a survey). Moreover, scription of evolution in the presence of sexual recombination
there is a dual view of this algorithm: each locus may be seen as and weak selection is equivalent to a repeated game between
genes played according to the multiplicative weight updates
selecting its new allele distribution at each generation so as to
algorithm, an algorithm that has surprised computer scientists
maximize a certain convex combination of (i) cumulative expected
time and again in its usefulness. This equivalence is informative
fitness and (ii) the entropy of its distribution on alleles. This con-
for the pursuit of two key problems in evolution: the role of sex
nection between evolution, game theory, and algorithms seems to us and the maintenance of variation.
rife with productive insights; for example, the dual view just men-
tioned sheds new light on the maintenance of diversity in evolution. Author contributions: E.C., A.L., C.P., and U.V. designed research, performed research, con-
Game theory has been applied to evolutionary theory before, tributed new reagents/analytic tools, analyzed data, and wrote the paper.
to study the evolution of strategic individual behavior (see, e.g., The authors declare no conflict of interest.
refs. 6, 7). The connection between game theory and evolution Freely available online through the PNAS open access option.
that we point out here is at a different level, and arises not in the See Commentary on page 10398.
analysis of strategic individual behavior, but rather in the analysis 1
To whom correspondence should be addressed. E-mail: [email protected].
of the basic population genetic dynamics in the presence of This article contains supporting information online at www.pnas.org/lookup/suppl/doi:10.
sexual reproduction. The main ingredients of evolutionary game 1073/pnas.1406556111/-/DCSupplemental.

10620–10623 | PNAS | July 22, 2014 | vol. 111 | no. 29 www.pnas.org/cgi/doi/10.1073/pnas.1406556111

 SEE COMMENTARY
We now introduce the framework of game theory (see Fig. 2 1 t

for an illustration) and the MWUA (SI Text), studied in com- xit+1 ð jÞ = x ð jÞ 1 + euti ð jÞ ; [2]
Zt i
puter science and machine learning, and rediscovered many
t
P t Z is a normalizing
times over the past half-century; as a result of these multiple where constant designed to ensure that
rediscoveries, the algorithm is known with various names across x
j i ð jÞ = 1, so xit+1 is a probability distribution; « is a crucial small
subfields: “the experts algorithm” in the theory of algorithms, positive parameter, and uti ð jÞ denotes the expected utility gained
“Hannan consistency” in economics, “regret minimization” in by player i choosing action j in the regime of the mixed actions by
game theory, “boosting” and “winnow” in artificial intelligence, the other players effective at time t. This algorithm (i) is known
etc. Here we state it in connection to games, which is only a small to converge to the min–max actions if the game is two-player
part of its applicability (see SI Text for an introduction to the zero-sum; (ii) is also shown here to converge to equilibrium for
MWUA in connection to the so-called “experts problem” in the coordination games of interest in the present paper (SI Text,
Corollary 5); (iii) is a general “learning algorithm” that has been
computer science).
shown to be very successful in both theory and practice; and (iv)
A game has several players, and each player i has a set Ai of
if, instead of games, it is applied to a large variety of optimization
possible actions. Each player also has a utility, capturing the way problems, including linear programming, convex programming,
whereby her actions and the actions of the other players affect and network congestion, it provably converges to the optimum
this player’s well-being. Formally the utility of a player is a quite fast.
function that maps each combination of actions by the players to It can be now checked that the two processes expressed in Eqs.
a real number (intuitively denoting the player’s gain, in some 1 and 2, evolution under natural selection in the presence of sex
monetary unit, if all players choose these particular actions). In and multiplicative weight updates in a coordination game, are
general, rather than choosing a single action, a player may in- mathematically identical (SI Text, Theorem 3). That is, the in-
stead choose a mixed or randomized action, that is, a probabi- teraction of weak selection and sex is equivalent to the MWUA in
listic distribution over her action set. Here we only need to a coordination game between loci in which the common utility is
consider coordination games, in which all players have the same the differential fitness of the organism. The parameter « in the
utility function—that is, the interests of the players are perfectly algorithm, which, when small signifies that the algorithm is taking
aligned, and their only challenge is to coordinate their choices a “longer-term view” of the process to be solved (SI Text), cor-
responds to the selection strength in evolution, i.e., the magni-
effectively. Coordination games are among the simplest games;
tude of the differences between the fitness of various genotypes.
the only challenge in such a game is for the players to “agree” on The MWUA is known in computer science as an extremely
a mutually beneficial action. simple and yet unexpectedly successful algorithm, which has sur-
How do the players choose and adjust their choice of ran- prised us time and again by its prowess in solving sophisticated
domized (mixed) actions over repeated play? Assume that at computational problems such as congestion minimization in
time t, player i has mixed action xit , assigning to each action j ∈ Ai networks and convex programming in optimization. The obser-
the probability xti ð jÞ. The MWUA algorithm (5) adjusts the mixed vation that multiplicative weight updates in a coordination game
strategy for player i in the next round of the game according to are equivalent to evolution under sex and weak selection makes
the following rule: an informative triple connection between three theoretical fields:

A B

C D
EVOLUTION

Fig. 1. Equations of population genetics formulated in the 1930s constitute the standard mathematical way of understanding evolution of a species by
tracking the frequencies of various genotypes in a large population. In the simple example shown here, a haploid organism with two genetic loci A and B has
three alleles in each of its two loci named A1, A2, A3 and B1, B2, B3 for a total of nine genotypes. In A we show the fitness of each genotype, that is, its
expected number of offspring. The fitness numbers shown in A are all close to each other, reflecting weak selection, where the individual alleles’ con-
COMPUTER SCIENCES

tributions to fitness are typically minuscule. Initially, each genotype occurs in the population with some frequency; in this particular example these numbers
are initially equal (B); naturally, their sum over all nine genotypes is 1 (frequencies are truncated to the fourth decimal digit). C shows how the genotype
frequencies evolve in the next generation: each individual of a given genotype produces a number of offspring that is proportional to its fitness shown in A,
and the resulting offspring inherits the alleles of its parents with equal probability (because we are assuming, crucially, sexual reproduction). The genotype
frequencies in the next generation are shown in C, calculated through the standard recurrence equations of population genetics. We also show in the margins
of the table the allele frequencies, obtained by adding the genotype frequencies along the corresponding row or column. Ten generations later, the fre-
quencies are as shown in D.

Chastain et al. PNAS | July 22, 2014 | vol. 111 | no. 29 | 10621
 A B

C D

Fig. 2. A simple coordination game is played by two players: the row player, who chooses a row, and the column player, who chooses a column. After the
two players make a choice, they both receive (or both pay, in case of a negative entry) the same amount of money, equal to the number at the chosen row
and column (A). Coordination games are the simplest possible kind of a game, one in which the strategic interests of all players are completely aligned—that
is to say, there is no conflict at all. They are of interest when it is difficult for the players to know these numbers, or to communicate and agree on a mutually
beneficial combination (in this example, third row and second column). Notice that this particular coordination game is closely related to the fitness landscape
shown in Fig. 1A: If P is a payoff in this game, the corresponding entry of Fig. 1A is equal to 1 + « · P, where « is a small parameter here taken to be 0.01.
Suppose that each of the two players chooses each of the three options with some probability, initially 1=3 for all (B); in game theory such probabilistic play is
called a mixed action. How do we expect these probabilities to change over repeated play? One famous recipe is the MWUA, in which a player “boosts” the
probability of each option by multiplying it by 1 + «G, where G is the expected amount of money this option is going to win the player in the current round of
play, and « is the same small parameter as above. For example, the second action of the row player has G equal to 2 (the average of 3, −1, and 4), and so the
probability of playing the second row will be multiplied by 1.02. Then these weights are “renormalized” so they add up to 1, yielding the marginal prob-
abilities shown in C. The probabilities after 10 such rounds of play are shown in D. Comparing now the numbers in the margins of Figs. 1D and 2D, we notice
that they are essentially the same. This is what we establish mathematically in this paper: the two processes—repeated coordination games played through
multiplicative updates, and evolution under weak selection—are essentially identical. This conclusion is of interest because the MWUA is known in computer
science to be surprisingly powerful.

evolutionary theory, game theory, and the theory of algorithms– distribution; see SI Text.] Subtracting this equation from its ho-
machine learning. molog with t replaced by t + 1, and applying the approxima-
So far we have presented the MWUA by “how it works” (in- tion expðeuti ð jÞÞ ≈ ð1 + euti ð jÞÞ, we obtain the precise Eq. 2 (the
formally, it boosts alleles proportionally to how well they do in normalization Zt is obtained from μt and μt+1; see SI Text for the
the current mix). There is an alternative way of understanding more detailed derivation).
the MWUA in terms of “what it is optimizing.” That is, we Thus, because Eqs. 1 and 2 are identical, we conclude that, in
imagine that the allele frequencies of each locus in each gener- the weak selection regime, natural selection is tantamount to
each locus choosing at each generation its allele frequencies in
ation are the result of a deliberate optimization by the locus of
the population so as to maximize the sum of the expected cumu-
some quantity, and we wish to determine that quantity. P lative differential fitness over the alleles, plus the distribution’s
Returning to the game formulation, define Uit ð jÞ = tτ=0 uτi ð jÞ
entropy. Note that quantity 3 is maximized by genes, not by
to be the cumulative utility obtained by player i by playing strategy j individuals, and that, interestingly, it is maximized with respect
over all t first repetitions of the game, and consider the quantity to current frequencies while being dependent (through Ut) on
X all past frequencies, and although there is some precedent to
1X t the use of “historical fitness” (9), its importance in this context
xti ð jÞUit ð jÞ − x ð jÞln xti ð jÞ: [3]
j
e j i is unexpected.
This alternative view of selection provides a new insight into
The first term is the current (at time t) expected cumulative an important question in evolutionary biology, namely: How is
utility. The second term of 3 is the entropy (expected negative genetic diversity maintained in the presence of natural selection
logarithm) of the probability distribution {xi( j), j = 1, . . . jAi j}, (10)? That the MWUA process enhances the entropy of the
multiplied by a large constant 1=e. Suppose now that player i alleles’ distribution (while at the same time optimizes expected
wished to choose the probabilities of actions xti ðjÞs with the sole cumulative utility) hints at such a mechanism. In fact, entropy is
goal of maximizing the quantity 3. This is a relatively easy opti- enhanced inversely proportional to s (the quantity corresponding
mization problem, because the quantity 3 to be maximized is strictly in the population genetics domain to the parameter «), the se-
concave, and therefore it has a unique maximum, obtained through lection strength: The weaker the selection, the more it favors
the Karush–Kuhn–Tucker conditions of optimality (8) (SI Text, high entropy. Naturally, entropy will eventually vanish when the
section 4): process quiesces at equilibrium: One allele per locus will even-
tually be fixed, and in fact this equilibrium may be a local, as
1
 opposed to global, fitness maximum. However, we believe that it
Uit ð jÞ − 1 + ln xti ð jÞ + μt = 0: is interesting and significant that the entropy of the allele dis-
e
tribution is favored by selection in the transient; in any event,
t
[Here μP is the Lagrange multiplier associated with the con- mutations, environmental changes, and finite population effects
t t
straint j xi ð jÞ = 1 seeking to keep the xi ð jÞs a probability are likely to change the process before equilibrium is reached.

10622 | www.pnas.org/cgi/doi/10.1073/pnas.1406556111 Chastain et al.

 SEE COMMENTARY
This new way of understanding the maintenance of variation in role of sex is the maintenance of diversity has been critiqued
evolution (selection as a tradeoff between fitness and entropy (15), because sex does not always increase diversity, and diversity
maximization) is quite different from previous hypotheses for is not always favorable. The MWUA connection sheds new light
the maintenance of variation (e.g., refs. 11, 12). Another rather on the debate, because sex is shown to lead to a tradeoff between
surprising consequence of this characterization is that, under weak increasing entropy and increasing (cumulative) fitness.
selection, all past generations, no matter how distant, have equal The connection between the three fields, evolution, game
influence on the change in the allele mix of the current generation. theory, and learning algorithms, described here was not acces-
Our discussion has focused on the evolution of a fixed set of sible to the founders of the modern synthesis, and we hope that it
alleles; that is, we have not discussed mutations. Mutations are, expands the mathematical tracks that can be traveled in evolu-
of course, paramount in evolution, as they are the source of tion theory.
genetic diversity, and we believe that introducing mutations to
the present analysis is an important research direction. Here we ACKNOWLEDGMENTS. We thank Satish Rao, Noam Nisan, and Monty Slatkin
focus on the selection process, which is rigorously shown to be for helpful discussions and comments, and the visitors of the Spring 2014
program on Evolutionary Biology and Computation at the Simons Institute
tantamount to a tradeoff, for each locus, between maximizing for the Theory of Computing for engaging discussions. Special thanks are
diversity and maximizing expected cumulative fitness. due to Nick Barton for his constructive criticism of an earlier version, which
We can now note a simple yet important point. Because led to several important improvements of our presentation, including the
multiplicative weight updates by the loci operate in the presence correction of a potentially confusing oversight. The research of E.C. was
supported by National Science Foundation Grant CCF-1064785. The research
of sex, the triple connection uncovered in this paper is in- of C.P. was supported by National Science Foundation Grant CCF-0964033,
formative for the “queen of problems in evolutionary biology,” and by Templeton Foundation Grant 39966. The research of U.V. was
namely the role of sex in evolution (13, 14). The notion that the supported by National Science Foundation Grant CCR-0905626.

1. Bürger R (2000) The Mathematical Theory of Selection, Recombination, and Mutation 9. Leibler S, Kussell E (2010) Individual histories and selection in heterogeneous pop-
(Wiley, Chichester). ulations. Proc Natl Acad Sci USA 107(29):13183–13188.
2. Nagylaki T (1993) The evolution of multilocus systems under weak selection. Genetics 10. Lewontin RC, Hubby JL (1966) A molecular approach to the study of genic hetero-
134(2):627–647. zygosity in natural populations. II. Amount of variation and degree of heterozygosity
3. Nei M (2005) Selectionism and neutralism in molecular evolution. Mol Biol Evol 22(12): in natural populations of Drosophila pseudoobscura. Genetics 54(2):595–609.
2318–2342. 11. Dobzhansky T (1955) A Review of Some Fundamental Concepts and Problems of
4. Nagylaki T, Hofbauer J, Brunovský P (1999) Convergence of multilocus systems under Population Genetics (Cold Spring Harbor Lab Press, Cold Spring Harbor, NY), Vol 20,
weak epistasis or weak selection. J Math Biol 38(2):103–133. pp 1–15.
5. Arora S, Hazan E, Kale S (2012) The multiplicative weights update method: A meta- 12. Kimura M (1968) Evolutionary rate at the molecular level. Nature 217(5129):
algorithm and applications. Theory Comput 8:121–164. 624–626.
6. Maynard-Smith J (1982) Evolution and the Theory of Games (Cambridge Univ Press, 13. Bell G (1982) The Masterpiece of Nature: The Evolution and Genetics of Sexuality
Cambridge, UK). (Univ of California Press, Berkeley, CA).
7. Weibull JW (1995) Evolutionary Game Theory (MIT Press, Cambridge, MA). 14. Barton NH, Charlesworth B (1998) Why sex and recombination? Science 281(5385):
8. Boyd SP, Vandenberghe L (2004) Convex Optimization (Cambridge Univ Press, 1986–1990.
Cambridge, UK). 15. Otto SP (2009) The evolutionary enigma of sex. Am Nat 174(Suppl 1):S1–S14.

EVOLUTION
COMPUTER SCIENCES

Chastain et al. PNAS | July 22, 2014 | vol. 111 | no. 29 | 10623