A Definition of Continual Reinforcement Learning

David Abel André Barreto Benjamin Van Roy

[email protected] [email protected] [email protected]
DeepMind DeepMind DeepMind

Doina Precup Hado van Hasselt Satinder Singh

arXiv:2307.11046v1 [cs.LG] 20 Jul 2023

[email protected] [email protected] [email protected]

DeepMind DeepMind DeepMind

Abstract

In this paper we develop a foundation for continual reinforcement learning.

1 Introduction
In The Challenge of Reinforcement Learning, Sutton states: “Part of the appeal of reinforcement
learning is that it is in a sense the whole AI problem in a microcosm” [35]. Indeed, the problem
facing an agent that learns to make better decisions from experience is at the heart of AI. Yet, when
we study the reinforcement learning (RL) problem, it is typical to restrict our focus in a number of
ways. For instance, we often suppose that a complete description of the state of the environment is
available to the agent, or that the interaction stream is subdivided into episodes. Beyond these standard
restrictions, however, there is another significant assumption that constrains the usual framing of RL:
We tend to concentrate on agents that learn to solve problems, rather than agents that learn forever.
For example, consider an agent learning to play Go: Once the agent has discovered how to master the
game, the task is complete, and the agent’s learning can stop. This view of learning is often embedded
in the standard formulation of RL, in which an agent interacts with a Markovian environment with
the goal of efficiently identifying an optimal behavior, at which point learning can cease.
But what if this is not the best way to model the RL problem? That is, instead of viewing learning
as finding a solution, we can instead think of it as endless adaptation. This suggests the study
of the continual reinforcement learning (CRL) problem [30, 31, 33, 15], as first explored in the
thesis by Ring [29], with close ties to supervised never-ending [24, 26] and continual learning
[30, 31, 34, 25, 32, 12, 18, 28, 3].
Despite the prominence of CRL, the community lacks a clean, general definition of this problem. It is
critical to develop such a definition to promote research on CRL from a clear conceptual foundation,
and to guide us in understanding and designing continual learning agents. To these ends, this paper is
dedicated to carefully defining the CRL problem. Our definition is summarized as follows:

The CRL Problem (Informal)

An RL problem is an instance of CRL if the best agents never stop learning.

The core of our definition is framed around two new insights that formalize the notion of “agents
that never stop learning”: (i) we can understand every agent as implicitly searching over a set of
behaviors (Theorem 3.1), and (ii) every agent will either continue this search forever, or eventually
stop (Remark 3.2). We make these two insights rigorous through a pair of operators on agents that
we call the generates and reaches operators. Using these tools, we then define CRL as any instance
of the RL problem in which all of the best agents never stop their implicit search. We provide two
motivating examples of CRL, illustrating that traditional multi-task RL and continual supervised

Preprint.
learning are special cases of our definition. We further identify necessary properties of both CRL
(Theorem 5.1), and the new operators (Theorem C.2, Theorem C.25). Collectively, these definitions,
insights, and results formalize many intuitive concepts at the heart of continual learning, and open
new research pathways surrounding continual learning agents.

2 Preliminaries
We first introduce key concepts and notation. Our conventions are inspired by Ring [29], the recent
work by Dong et al. [9] and Lu et al. [19], as well as general RL [13, 14, 16, 17, 23].

Notation. We let capital calligraphic letters denote sets (𝒳), lower case letters denote constants and
functions (𝑥), italic capital letters denote random variables (𝑋), and blackboard capitals denote the
natural and real numbers (N, R, N0 = N ∪ {0}). Additionally, we let Δ(𝒳) denote the probability
simplex over the set 𝒳. That is, the function 𝑝 : 𝒳 × 𝒴 → Δ(𝒵) expresses a probability mass
function 𝑝(· | 𝑥, 𝑦), over 𝒵, for each 𝑥 ∈ 𝒳 and 𝑦 ∈ 𝒴. Lastly, we use ¬ to denote logical negation,
and we use ∀𝑥∈𝒳 and ∃𝑥∈𝒳 to express the universal and existential quantifiers over a set 𝒳.

2.1 Agents and Environments

We begin by defining environments, agents, and related artifacts.

Definition 2.1. An agent-environment interface is a pair (𝒜, 𝒪) of countable sets 𝒜 and 𝒪.

We refer to elements of 𝒜 as actions, denoted 𝑎, and elements of 𝒪 as observations, denoted 𝑜. We

assume |𝒜| ≥ 2 and |𝒪| ≥ 1, but otherwise the sets may be arbitrary.
Definition 2.2. The histories with respect to interface (𝒜, 𝒪) are the set of sequences of action-
observation pairs,
∞
(𝒜 × 𝒪)𝑡 .
Ø
ℋ= (2.1)
𝑡=0

Histories define the possible interactions between an agent and an environment that share an interface.
We refer to an individual element of ℋ as a history, denoted ℎ, and we let ℎ ℎ ′ express the history
resulting from the concatenation of any two histories ℎ, ℎ ′ ∈ ℋ . Furthermore, the set of histories
of length 𝑡 ∈ N0 is defined as ℋ𝑡 = (𝒜 × 𝒪)𝑡 , and we use ℎ 𝑡 ∈ ℋ𝑡 to refer to a history containing
𝑡 action-observation pairs, ℎ 𝑡 = 𝑎0 𝑜1 . . . 𝑎 𝑡−1 𝑜 𝑡 , with ℎ0 = ∅ the empty history. An environment is
then a function that produces observations given a history.
Definition 2.3. An environment with respect to interface (𝒜, 𝒪) is a function 𝑒 : ℋ × 𝒜 → Δ(𝒪)

This model of environments is general in that it can capture Markovian environments such as Markov
decision processes (MDPs, Puterman, 2014) and partially observable MDPs (Cassandra et al., 1994),
as well as both episodic and non-episodic settings. We next define an agent as follows.
Definition 2.4. An agent with respect to interface (𝒜, 𝒪) is a function, 𝜆 : ℋ → Δ(𝒜). We let Λ
denote the set of all agents, and Λ ⊆ Λ denote any non-empty subset of Λ.

This treatment of an agent captures the mathematical way experience gives rise to behavior. This is in
contrast to a mechanistic account of an agent as proposed by Dong et al. [9] and Sutton [37].

2.2 Realizable Histories

We will be especially interested in the histories that occur with non-zero probability as a result of the
interaction between a particular agent and environment.
Definition 2.5. The realizable histories of a given (𝜆, 𝑒) pair define the set of histories of any length
that can occur with non-zero probability from the interaction of 𝜆 and 𝑒,
∞ 𝑡−1
( )
Ø Ö
𝜆,𝑒
ℋ = ℋ̄ = ℎ 𝑡 ∈ ℋ𝑡 : 𝑒(𝑜 𝑘+1 | ℎ 𝑘 , 𝑎 𝑘 )𝜆(𝑎 𝑘 | ℎ 𝑘 ) > 0 . (2.2)
𝑡=0 𝑘=0

2
Lastly, given a realizable history ℎ, we will refer to the realizable history suffixes, ℎ ′, which, when
concatenated with ℎ, produce a realizable history ℎ ℎ ′ ∈ ℋ̄ .
Definition 2.6. The realizable history suffixes of a given (𝜆, 𝑒) pair, relative to a history prefix
ℎ ∈ ℋ 𝜆,𝑒 , define the set of histories that, when concatenated with prefix ℎ, remain realizable,

ℋℎ𝜆,𝑒 = ℋ́ = {ℎ ′ ∈ ℋ : ℎ ℎ ′ ∈ ℋ 𝜆,𝑒 }. (2.3)

When clear from context, we abbreviate ℋ 𝜆,𝑒 to ℋ̄ , and ℋℎ𝜆,𝑒 to ℋ́ , where ℎ, 𝜆, and 𝑒 are obscured
for brevity.

2.3 Reward, Performance, and the RL Problem

Supported by the arguments of Bowling et al. [4], we assume that all of the relevant goals or purposes
of an agent are captured by a deterministic reward function (in line with the reward hypothesis [36]).
Definition 2.7. We call 𝑟 : 𝒜 × 𝒪 → R a reward function.

We remain agnostic to how the reward function is implemented; it could be a function inside of
the agent, or the reward function’s output could be a special scalar in each observation. Such
commitments do not impact our framing. When we refer to an environment we will implicitly mean
that a reward function has been selected as well. We remain agnostic to how reward is aggregated to
determine performance, and instead adopt the function 𝑣 defined as follows.
Definition 2.8. The performance, 𝑣 : ℋ × Λ × ℰ → [vmin , vmax ] is a bounded function for fixed
constants vmin , vmax ∈ R.

The function 𝑣(𝜆, 𝑒 | ℎ) expresses some statistic of the received future random rewards produced
by the interaction between 𝜆 and 𝑒 following history ℎ, where we use 𝑣(𝜆, 𝑒) as shorthand for
𝑣(𝜆, 𝑒 | ℎ0 ). While we accommodate any 𝑣 that satisfies the above definition, it may be useful to
think of specific choices of 𝑣(𝜆, 𝑒 | ℎ 𝑡 ), such as the average reward,

lim inf 1𝑘 E𝜆,𝑒 [𝑅 𝑡 + . . . + 𝑅 𝑡+𝑘 | 𝐻𝑡 = ℎ 𝑡 ], (2.4)

𝑘→∞

where E𝜆,𝑒 [ · · · | 𝐻𝑡 = ℎ 𝑡 ] denotes expectation over the stochastic process induced by 𝜆 and 𝑒
following history ℎ 𝑡 . Or, we might consider performance based on the expected discounted reward,
𝑣(𝜆, 𝑒 | ℎ 𝑡 ) = E𝜆,𝑒 [𝑅 0 + 𝛾𝑅 1 + . . . | 𝐻𝑡 = ℎ 𝑡 ], where 𝛾 ∈ [0, 1) is a discount factor.
The above components give rise to a simple definition of the RL problem.
Definition 2.9. A tuple (𝑒 , 𝑣, Λ) defines an instance of the RL problem as follows:
Λ∗ = arg max 𝑣(𝜆, 𝑒). (2.5)
𝜆∈Λ

This captures the RL problem facing an agent designer that would like to identify an optimal agent
(𝜆∗ ∈ Λ∗ ) with respect to the performance (𝑣), among the available agents (Λ), in a particular
environment (𝑒).

3 Agent Operators: Generates and Reaches

We next introduce two new insights about agents, and the operators that formalize them:

1. Theorem 3.1: Every agent can be understood as searching over a set of behaviors.
2. Remark 3.2: Every agent will either continue their search forever, or eventually settle on a
choice of behavior

We make these insights precise by introducing a pair of operators on agents: (1) any set of agents
generates (Definition 3.4) another set of agents, and (2) a given agent reaches (Definition 3.5) an agent
set. Together, these operators enable us to define learning as the implicit search process captured by
the first insight, and continual learning as the process of continuing this search indefinitely.

3
3.1 Operator 1: An Agent Basis Generates an Agent Set.

The first operator is based on two complementary intuitions.

From the first perspective, an agent can be understood as implicitly searching over a space of
behaviors. For instance, in an MDP, agents can be interpreted as searching over the space of policies.
It turns out this insight can be extended to any agent and any environment.
The second complementary intuition notes that, as agent designers, we often first identify the
behaviors we would like an agent to search over. Then, it is natural to design agents that search
through this space of behaviors. For instance, we might be interested in all behaviors representable
by a neural network of a certain size and architecture. When we design agents, we then consider
all agents (choices of loss function, optimizer, memory, and so on) that search through the space
of assignments of weights to this particular neural network using standard methods like stochastic
gradient descent. We codify these intuitions in the following definitions.
Definition 3.1. An agent basis, ΛB ⊂ Λ, is any non-empty subset of Λ.
Notice that an agent basis is a choice of agent set, Λ. We explicitly call out a basis with distinct
notation (ΛB ) as it serves an important role in the discussion that follows. For example, we next
introduce learning rules as functions that switch between elements of an agent basis for each history.
Definition 3.2. A learning rule over an agent basis ΛB is a function, 𝜎 : ℋ → ΛB , that selects a
base agent for each history. We let Σ denote the set of all learning rules over ΛB , and let Σ ⊆ Σ
denote any non-empty subset of Σ.
A learning rule is a mechanism for switching between base behaviors following each experience. We
use 𝜎(ℎ)(ℎ) to refer to the action distribution selected by the agent 𝜆 = 𝜎(ℎ) at any history ℎ.
Definition 3.3. Let Σ be a set of learning rules over some basis ΛB , and 𝑒 be an environment. We
say that a set Λ is 𝚺-generated by ΛB in 𝑒, denoted ΛB ⊢𝑒 Σ Λ, if and only if
∀𝜆∈Λ ∃𝜎∈Σ ∀ ℎ∈ℋ̄ 𝜆(ℎ) = 𝜎(ℎ)(ℎ). (3.1)

Thus, any choice of Σ together with a basis ΛB induces a family of agent sets whose elements can be
understood as switching between base behaviors according to the rules prescribed by Σ. We then
say that a basis generates an agent set in an environment if there exists a set of learning rules that
switches between the basis elements to produce the agent set.
Definition 3.4. We say a basis ΛB generates Λ in 𝑒, denoted ΛB ⊢𝑒 Λ, if
ΛB ⊢𝑒 Σ Λ. (3.2)

Intuitively, an agent basis ΛB generates another agent set Λ just when the agents in Λ can be
understood as switching between the base agents. It is in this sense that we can understand agents as
searching through a set of behaviors—an agent is just a particular sequence of history-conditioned
switches over the basis. For instance, let us return to the example of a neural network: The agent
basis might represent a specific neural network architecture, where each element of this basis is an
assignment to the network’s weights. The learning rules are different mechanisms that choose the
next set of weights in response to experience (such as stochastic gradient descent). Together, the
agent basis and the learning rules generate the set of agents that search over choices of weights in
reaction to experience. We present a cartoon visual of the generates operator in Figure 1(a).
Now, using the generates operator, we revisit and formalize the central insight of this section: Every
agent can be understood as searching over a space of behaviors. We take this implicit search process
to be the behavioral signature of learning.
Theorem 3.1. For any agent-environment pair (𝜆, 𝑒), there exists infinitely many choices of a basis,
ΛB , such that both (1) 𝜆 ∉ ΛB , and (2) ΛB ⊢𝑒 {𝜆}.
Due to space constraints, all proofs are deferred to Appendix B.
We require that 𝜆 ∉ ΛB to ensure that the relevant bases are non-trivial generators of 𝜆. This theorem
tells us that no matter the choice of agent or environment, we can view the agent as a series of
history-conditioned switches between basis elements. In this sense, we can understand the agent as
if 1 it were carrying out a search over the elements of some ΛB .
1We use as if in the sense of the positive economists, such as Friedman [11].

4
3.2 Operator 2: An Agent Reaches a Basis.

Our second operator reflects properties of an agent’s limiting behavior in relation to a basis. Given an
agent and a basis that the agent searches through, what happens to the agent’s search process in the
limit: does the agent keep switching between elements of the basis, or does it eventually stop? For
example, in an MDP, many agents of interest eventually stop their search on a choice of a fixed policy.
We formally define this notion in terms of an agent reaching a basis according to two modalities: an
agent (i) sometimes, or (ii) never reaches a basis.
𝑒
Definition 3.5. We say agent 𝜆 ∈ Λ sometimes reaches ΛB in 𝑒, denoted 𝜆 ⇝ ΛB , if
∃ ℎ∈ℋ̄ ∃𝜆B ∈ΛB ∀ ℎ′ ∈ℋ́ 𝜆(ℎ ℎ ′) = 𝜆B (ℎ ℎ ′). (3.3)

That is, for at least one history that is realizable by the pair (𝜆, 𝑒), there is some basis behavior (𝜆B )
that produces the same action distribution as 𝜆 forever after. By contrast, we say an agent never
reaches a basis just when it never becomes equivalent to a base agent.
/𝑒 ΛB , iff ¬(𝜆 ⇝
Definition 3.6. We say agent 𝜆 ∈ Λ never reaches ΛB in 𝑒, denoted 𝜆 ⇝ 𝑒
ΛB ).

In Appendix B, we further define the always reaches operator (a special case of sometimes reaches),
but as it is not critical to the discussion that follows, we defer its introduction until later.
The reaches operators formalize the insight that, since every agent can be interpreted as if it were
searching over a basis, every agent will either (1) sometimes, or (2) never stop this search. We can
now plainly state this fact as follows.
Remark 3.2. For any agent-environment pair (𝜆, 𝑒) and any choice of basis ΛB such that ΛB ⊢𝑒 {𝜆},
exactly one of the following two properties must be satisfied:
𝑒
(1) 𝜆 ⇝ ΛB , /𝑒 ΛB .
(2) 𝜆 ⇝ (3.4)

Thus, by Theorem 3.1, every agent can be thought of as implicitly searching over a set of behaviors,
and by Remark 3.2, every agent will either (1) sometimes, or (2) never stop this search. We take
this implicit search process to be the signature of learning, and will later exploit this perspective to
define a continual learning agent (Definition 4.1) as one that continues its search forever. Our analysis
in Section 5 further elucidates basic properties of both the generates and reaches operators, and
Figure 1(b) visualizes the set relations that emerge between a basis ΛB and an agent set Λ it generates
through the reaches operator. We summarize all definitions and notation in a table in Appendix A.

{⌫ 2 ⇤ ⌫ ⇤B } {⌫ 2 ⇤ ⌫ ⇤B }
⇤B ` ⇤
⌫2⇤ ⌫⇤ ⇤ ⇤ ⌫2⇤ ⌫⇤ ⇤ ⇤
=

= <latexit sha1_base64="nvn57sEquRQbVU/gaZl+jnAPX4Y=">AAAlOXicnVrdbxy3Eb/0M71+Je1jX4hYByTGSdEpTuqmEGDHkZMCdixLchJEqwjcXd4dq/0yyZNOWRDoe1/bv6Z/SR/7VvS1/0BnSO4uubfnGlFiaTnzG+5wODOc4V1cZVyq/f1/vvGDH/7oxz/56Zs/G//8F7/81a/fevs3X8pyJRL2IimzUnwdU8kyXrAXiquMfV0JRvM4Y1/FV4+Q/9U1E5KXxZm6rdhFThcFn/OEKiA9P7x8687+3r75IZsPM/dwZ+R+ji/f3nknSstklbNCJRmV8nw2q9RFTYXiScb0eDyetD/jCTmmyRVdMLk3jlaSVXZ0vlLz+xc1L6qVYkWiyYTQLCtvCNB37xNDD/Bns4t6XhYWDD8TAkxyfzfmipzNCLKkL1DTXOZULXWfaJAbVHmbxxtEtcxD2hLsJwSbGw2MEoYCpr8KX74SWYNxQMnzKmPkxckTgrsgmVK8WARCcVleKRrLbvZKlHMmcetotvtyRTOubglAMraxVrusVjLOgBWXVKQErUBwfWUWShU8YXNBk04qKXNgqgYNVhVk9v7BlDCVBNtX5zwRJS5EN7INpVwIWi1vg82LS6XKHDewVDmXSWjV+SrLKngIqOfFKo/BbS/qgqqY9zanokJe8SokFuymWiu2VlPzZHcfnJHs7u6So7UStPVFJIUTcjTglMhVPOeLKUnky1WpmJySlMqlWGVsShS/+m5K4jiHX/DPrJMn6ynJV5nioryZEgi0GOIKuPh2IecwAc0WpeDgS+axkmwF0VOmMGGegvlzloJ9YbVcsRzi4IbKW7C/nYHGa/sQw36ABmvnvRPyGB/CPUHNFK9oaJVGGqifMohXwU7N7uIED7MKXsFUHVmUrh8+PY1fB5npGidNrYEfZooJ2Cd+zYgq0UTGdSCemSAJJCfPHz9u1zOOTPRAKKyqeoxeZLKZiaZDJVZsaogxZLOrPhECzIAP46whIaRPS8CoHW0celhCK64gpr5j0wK2gRc4w0UNOQy2EYIclhalbA551UxRp4Kluj757BNdz/b3p+Qe/PtgX/dAOb7Jwfanf/hgejC718fcLEEvhzm4d29Kml/aWDOGJV9zk59xi8GbITBzWqRg/USfQyY0u5DQrL4z07qHkB1CJmIIEXcI2KohhOgQYgjAKqnr6BpcpJI8K4semxcpT3T3hg35+Ai4Ehh7By2sPtJ9GMwMUabdA18sFRUQZwOorEFlbD4IMtNsn8KIbxH9AlWM6y/62n1j6d/06c8t/XmffmLpJxvGLJQoq1vL/VxbZ4muc17gTtTX+tL8hbHumHQdMum6ZaKrgXUVuLlgmWEDSev6UBv/Os3hqDVR6OsB23F+AHsuwa1YXK5x11EjeAIN5UowlKijlMsqo7dS3cJxdudA42snR2uKx9vHBDd1325qUn+tbXqgYgHKT81fut5r08tTmPAZZACqSnG3jixMQyGxiKb49CogrNcB4WkbcN0g19G0AzdnwsCPYZAvjuCU/hQDlpsYJI4xKAGcR2UBGWMBpQnbs1tQxcbsx7g7kDVpszVVHIOpuQq5ER5a9a42VAvQuhPJ0+oVeODaPUgqJwAx2eGbANUds6+Cg7xaCwAFegwJNaqgTc6YNOcTCjPwjjpSdKW7sTQecubQR8U1HKAFlpStUEeqmW5ptc/QG1BZm3mPWsWBJwMZaXNsYnZ2Sp7Fkolr6kaniiqsrVCSWoiZ72EzXenBDedZw5FG1NBOWxpGWsMwkfGhbhBWjYVd8cS+EUd1VF6bQ9FJfGQSZCVVqTFHQWGftu7kBBxxMr/sABMPAS9/4iD48sf6shWxqLhcFSlL3XSKZymrrajGioSnLBY8XTAag26bIjKUka16WbYFlmUt0jdDp/Ic9vGaJa0W/oLnxjhugsFFN9O31YqnmXsb+YRKHrwydiCbUD+BdxLfhg1bdnxfGGdziLgn6LFkHCyEr4dFOgYKTMgJo8mSNdoWpRKWUOOzPdWgXuWL5vRCVAsZZie0MIgBvnPG7Aay/DaMf3D4yAmUXbZswJOnLKG8WsAxtpRyJwJ5SNE7OIiuoF7chYPigK2jJZ4x9U6j8Y62+Yx8xgrM5O26F80YXAOrchNBTxgVBXRSBGv0BpkhEQkQQxxOAN0jwwynHh38xWe1PmMxDoQNdTtr+6gn7vyN9QBQekg5AIUXh+hAkwGB62ElNpjhi9uXXb/iZb7/wQFf1J0vNooguVsmtDBgoyVVYUr22XKTH4Q8Kc1Zjcm0KV2wBYUiA7b3Gn0Al5FC7pv2XXBKopOunIPRE6jeAgKMoCSR2IxFS+hNoY8uoLNzmsFDUpoxaMhewl/jU51qJm3vDeV2MgQxlpzXK9/VbKICoRdett81UqQqM57cNvnHjuqo4sOzO4CZ73gA45eTDdb1AW54aav4sckmN9CHSQJo8iWFbgUPINOfYBxBz8ArRUTQREJNCIEsaFhphU1hi6nPoHLsRrmuiyapCPNqTO5B8d+Aa9FVHA1U6Ankn2XZ0rH0FeCQpvKwdW8fAZVzhzBlcw9h526256QNL7TGHIpqH4YaYGUu7CZ+zsEdBW+qhGUzNDO11bvw6DEVdYdrEXI1n3somqwUG8IxbFHxzgdZtTfzIIBC82hLCLwQHJivJVDjPB7i2yjhIsFFuiPClbXz2hDJcQYiHFrqW1cTtuM5LL7x/I4KnYISxixPsWJ0J/kgCq+A+sh3bSH3LXQcNC/BV6Ocp2R52ezee0apRxR8tFUpcaNGZbz5eaaWTJCnXCZ7UKmbILH3P8CHIp+wl67+qvEXe6n7FRjWbMaVTBdl2ygv4u52cl3P5ZVupoX2RNGSK0UomdMiuSVw+EFjhGUSUUsIwH7HnwLARPIYav8FeHZza6Qn50YsKQWci4f7lboYR9cSDMDqWZ7rMUb8GHJv2omMzevPlqyEds69SdlR7f5C+LoHn3u3Yd8d5tdzKJd1/Rh/B3TQroRyTEB7+6h9DBDM9o66dk1kyMW2U1zpcze+qE8sIQBlDIzlYZ6YcQCpoMUupennPOCxRw3gadv96brrBHsm6UB3t6IAJKF+hlOOIqZ97hmp+DNLFPTYaKX22dsr4q5DccvM0w1Hb6kZ1FRWS8mS5uU+oN3VLXx/oS1kEmLcHmybwtd+2xS++beCPF/ZBrE73bHNgfanAgKAeEWGuX2EvS3n0tg5T4F5DqWpKitMi4dzmklzRWhvpkOaqSKQRA5JS8TLIbACBODh/Uq1MEfatyS8kzY90uHsI6DAMQ0vJBYDbI56AiUUMlSrR8cY42cjQK9wETZn4FOTwMyq/A45tUDIE+cGeFG7bOHoJoHFc5vIoPm3s8H/d2Z7Wl8Qm+D89KGJm8HTQ9cmnXgUM3aPvvD/11h+L41Pr5hKllsVP/g+ioOyfjtj/Q28DQ/lNFVlAllWMeHF2kxHii5AtSVridreFkcu4OgKDlU2x4tlL2G6Y9WFi4/xw94d6xh0PqTNfMjGfOszXeY1Z2sXaz4izHVWExdvoSJdljYXKG6BPuaoWXTTWbhPOHzMw4bo3UoWG0oFWdPotKQVGDvQyJKa1VERLPwYxo5lPhkLDI+ExmB80TP5Y0NprmpW8TwrqXol4NVTmNzkc5tjyIjbjOWzT9t0aowo5Sqv+piHLbWxT0Z5YOdHSDCJEEIup1fMfOihSAJV/xWuPyjkmSrLDCuKcYRYqjKmwLZjMCvEVZLjdffDswfQ7SiMGUh+xBCKPBiaD24e1A1L1/BfM2OJ9Zbt8aoKooyviQstGRx6JhiTFRSfOXpJWFb4maLDQLKAhAk1MPPiFeYLp8K6J+rNbsRQmR4ZuyGMqm2amaPTLzk2FUPIayiGE/cUM6vdVMyQ8VbZBeI23SB8N4qqTfUQ9RrqAayvHkoOqGfIJrs36WSbgpiI9GaNtamjAb6Gkojra2lkB9S09AnBxLhNwX7RuqmZQbyGZojra2ZkBzSzdLy6gpm2qZah43l17KZq2ev5Xbbpd9mw3zkyBq7NL8TSXe3JUmLPJPm+6hoIzElhUG0/+Xo22HJ8hRpty6dhvGw9JkO/fdU513OdrSenSWxpSmhBzPcCCFamRJY5I6YGCbs3QzJ3dE15kzAsJnS0bL5YUB8ksNWzStlfpjBxIPu2R9CVZ2D9rgXwEjp+KSFsR00XD3jsGfHjPLzOkfw7yNyIPW++K3T4Lhy1Yg+H713Yj9+bn6goU3Yu4dhlh3ayaSrozdSYiMC7Dw+g2SRmgvcIetcf8drGn+LyrTuz/neINh++PNibfbi3//zenQcv/mK/X/Tm6Hejd0bvjmaj348ejD4fHY9ejJIRG/119LfR33f+sfOvnX/v/MdCf/CG+07Sb0fBz85//wehHBVX</latexit>
{

..
<latexit sha1_base64="x1XUCM/GIpCAGiRVVXkaD0GQmA0=">AAAlOnicnVpfbxy3Eb+k/9Lrv6R97AsR64DEOCk6xUndFALsOHJSwI4VSU6CaBWBu8u7Y7X/TPKkUxYE+gH62n6afpG+9q3oaz9AZ0juLrm35xhRYmk58xvucDgznOFdXGVcqv39f732+o9+/JOf/uyNn49/8ctf/fo3b7712y9luRIJe56UWSm+jqlkGS/Yc8VVxr6uBKN5nLGv4qtHyP/qmgnJy+JM3VbsIqeLgs95QhWQTqL68s07+3v75odsPszcw52R+zm+fGvn7Sgtk1XOCpVkVMrz2axSFzUViicZ0+PxeNL+jCfkmCZXdMHk3jhaSVbZ0flKze9f1LyoVooViSYTQrOsvCFA371PDD3An80u6nlZWDD8TAgwyf3dmCtyNiPIkr5ATXOZU7XUfaJBblDlbR5vENUyD2lLMKAQbG40MEoYCtj+Knz5SmQNxgElz6uMkecnTwhug2RK8WIRCMVleaVoLLvZK1HOmcS9o9nuixXNuLolAMnYxlrtslrJOANWXFKRErQCwfWVWShV8ITNBU06qaTMgakaNFhVkNl7B1PCVBJsX53zRJS4EN3INpRyIWi1vA02Ly6VKnPcwFLlXCahVeerLKvgIaCeF6s8Br+9qAuqYt7bnIoKecWrkFiwm2qt2FpNzZPdfXBGsru7S47WStDWF5EUTsjRgFMiV/GcL6YkkS9WpWJySlIql2KVsSlR/Oq7KYnjHH7BP7NOnqynJF9liovyZkog0mIILODi24WcwwQ0W5SCgy+Zx0qyFURPmcKEeQrmz1kK9oXVcsVyiIMbKm/B/nYGGq/tQwz7ARqsnfdOyGN8CPcENVO8oqFVGmmgfsIgXgU7NbuLEzzMKngFU3VkUbp++PQ0fhVkpmucNLUGfpgpJmCf+DUjqkQTGdeBeGaCJJCdPH/8qF3PODLRA6GwquoxepFJZyaaDpVYsakhxpDOrvpECDADPoyzhoSQPi0Bo3a0cehhCa24gpj6jk0L2AZe4AwXNeQw2EYIclhalLI5JFYzRZ0Klur65NOPdT3b35+Se/Dv/X3dA+X4Jgfbn/7x/enB7F4fc7MEvRzm4N69KWl+aWPNGJZ8zU2Cxi0Gb4bAzGmRgvUTfQ6Z0OxCQrP6zkzrHkJ2CJmIIUTcIWCrhhCiQ4ghAKukrqNrcJFK8qwsemxepDzR3Rs25OMj4Epg7B20sPpI92EwM0SZdg98sVRUQJwNoLIGlbH5IMhMs30KI75F9HNUMa4/72v3jaV/06d/Yelf9Oknln6yYcxCibK6tdzPtHWW6DrnBe5Efa0vzV8Y645J1yGTrlsmuhpYV4GbC5YZNpC0rg+18a/THI5aE4W+HrAd5wew5xLcisXlGncdNYIn0FCuBEOJOkq5rDJ6K9UtHGd3DjS+dnK0pni8fURwU/ftpib119qmByoWoPzU/KXrvTa9PIUJn0EGoKoUd+vIwjQUEotoik8vA8J6HRCetgHXDXIdTTtwcyYM/BgG+fwITulPMGC5iUHiGIMSwHlUFpAxFlCasD27BVVszH6MuwNZkzZbU8UxmJqrkBvhoVXvakO1AK07kTytXoIHrt2DpHICEJMdvglQ3TH7KjjIy7UAUKDHkFCjCtrkjElzPqEwA++oI0VXuhtL4yFnDn1UXMMBWmBJ2Qp1pJrpllb7DL0BlbWZ96hVHHgykJE2xyZmZ6fkWSyZuKZudKqowtoKJamFmPkeNtOVHtxwnjUcaUQN7bSlYaQ1DBMZH+gGYdVY2BVP7BtxVEfltTkUncSHJkFWUpUacxRU9mnrTk7AESfzyw4w8RDw8icOgi9/rC9bEYuKy1WRstRNp3iWstqKaqxIeMpiwdMFozHotikiQxnZqpdlW2BZ1iJ9M3Qqz2Efr1nSauEveG6M4yYYXHQzfVuteJq5t5GPqeTBK2MHsgn1Y3gn8W3YsGXH94VxNoeIe4IeS8bBQvh6WKRjoMCEnDCaLFmjbVEqYQk1PttTDepVvmhOL0S1kGF2QguDGOA7Z8xuIMtvw/gHh4+cQNllywY8ecoSyqsFHGNLKXcikIcUvYOD6ArqxV04KA7YOlriGVPvNBrvaJvPyKeswEzernvRjME1sCo3EfSEUVFAJ0WwRm+QGRKRADHE4QTQPTLMcOrRwV98VuszFuNA2FG3s7aPeuLO31gPAKWHlANQeHGIDjQZELgeVmKDGb64fdn1S17m+x8c8EXd+WKjCJK7ZUILAzZaUhWmZJ8tN/lByJPSnNWYTJvSBVtQKDJge6/RB3AZKeS+ad8FpyQ66co5GD2B6i0gwAhKEonNWLSE3hT66AI6O6cZPCSlGYOG7AX8NT7VqWbS9t5QbidDEGPJeb3yXc0mKhB67mX7XSNFqjLjyW2Tf+yojio+PLsDmPmOBzB+OdlgXR/ghpe2ih+bbHIDfZgkgCZfUuhW8AAy/QnGEfQMvFJEBE0k1IQQyIKGlVbYFLaY+gwqx26U67pokoowr8bkHhT/DbgWXcXRQIWeQP5Zli0dS18BDmkqD1v39hFQOXcIUzb3EHbuZntO2vBCa8yhqPZhqAFW5sJu4mcc3FHwpkpYNkMzU1u9C48eU1F3uBYhV/O5h6LJSrEhHMMWFe98kFV7Mw8CKDSPtoTAG8GB+VoCNc7jIb6NEi4SXKQ7IlxZO68NkRxnIMKhpb51NWE7nsPiG8/vqNApKGHM8hQrRneSD6LwCqiPfMcWct9Cx0HzEnw1ynlKlpfN7r1rlHpEwUdblRI3alTGm59naskEecplsgeVugkSe/8DfCjyCXvh6q8af7EXul+BYc1mXMl0UbaN8iLubifX9Vxe6WZaaE8ULblShJI5LZJbAocfNEZYJhG1hADsd/wpAEwkj6H2X4BnN7dGenJuxJJSwLl4uF+pi3F0LcEArJ7luR5jxI8h96adyNi8/mzJSmjn3JuUHdXuL4Sve/C5dxv23WF+PYdyWdeP8XdAB+1KKMcEtLeP2scAwWzvqGvXRIZcbDvFlT5344v6xBICUMbAWB7miRkHkApa7FKafs4DHnvUAJ623Z+uu06wZ5IOdHcrCkAS6mc45Shi2ueekYq/sERBj41Wap+9vSLuOhS3zDzdcPSWmkFNZbWULGle7gPaXd3C9xfaQiYhxu3Btil87bdN4Zt/K8jzlW0Qu9Md2xxofy4gAIhXZJjbR9jbci6NnfMUmOdQmqqywrR4OKeZNFeE9mY6pJkqAknkkLREvBwCK0AAHt6vVAtzpH1Lwjtp0yMdzj4EChzT8EJiMcDmqCdQQiFDtXp0jDF+NgL0ChdhcwY+NQnMrMrvkFMLhDxxboAXtcsWjm4SWDy3iQyafzsb/H9ntqf1BbEJzk8fmrgZPD10bdKJRzFj9+gLf7/G8gdpfHrFVLLcqvjBD1EclPXbGetv4G14KKepKhPIsooJL9ZmOlJ0AaotWUvU9rY4cgFHV3CosjleLHsJ0x2rLlx8jB/27ljHoPMhbeZDNuZbn+kyrzlbu1jzEWGus5q4eAsV6bK0uUBxC/QxR82im87CfcLhYx42RO9WsthQKsiaRqclrcDYgUaW1KyOimDhxzB2LPPJWGB4JDQG44ueyR8bSnNVs4rnWUnVSwEvn8LkJp/bHENG3GYsn33aplNjRClXedXHPGypjX0yygM7P0KCSYQQcjm9YuZDD0USqPqvcP1BIc9UWWZYUYwjxFKVMQW2HYNZIa6SHK+7H549gG5HYcxA8iOGUOTB0Hxw86BuWLqG/5oZS6y3bI9XVRBlfE1caMng0DPBmKyg+MzRS8Kyws8UHQaSBSRMqIGZF68wXzgV1j1Rb3Yjhsr0yNgNYVRt08wcnX7JsakYQl5BMZy4p5hZ7aZihoy3yi4Qt+kG4btRVG2qh6hXUA9gffVQckA9QzbZvUkn2xTERKQ3a6xNHQ3wFZREXF9LIzugpqVPCCbGbQr2i9ZNzQziFTRDXF8zIzugmaXj1RXMtE21DB3Pq2M3Vcteze+yTb/Lhv3OkTFwbX4hlu5qT5YSeybJ91TXQGBOCoNq+8nXs8GW4yvUaFs+DeNl6zEZ+u3Lzrme62w9OU1iS1NCC2K+F0CwMiWyzBkxNUjYvRmSuaNrypuEYTGho2XzxYL6IIGtnlXK/jKFiQPZtz2CrjwD63ctgJfQ8UsJYTtqunjAY8+IH+fhdY7k30HmRux582Whw3fgqBV7OHz3wn783vxERZmycwnHLju0k01TQW+mxkQE3n14AM0mMRO8S9C7/oTXNv4Ul2/emfW/Q7T58OXB3uyDvf0v7t158Pyv9vtFb4x+P3p79M5oNvrD6MHos9Hx6PkoGc1Hfxv9ffSPnX/u/HvnPzv/tdDXX3PfSfrdKPjZ+d//ARMoFfs=</latexit>
⌫1B
. ⇤ ⇤
<latexit sha1_base64="wc9+SOjDL7IlDDoLRC8mBmD+/lo=">AAAlPnicnVpfcxu3EWfSfyn7L2kf+4KJxZnEQymi4qRuOpqx48hJZ+xYkWTHE52iwd2BJKL7ZwCkqNxgph+hr+2n6dfoF+hbp6997C6AuwOOpOqJEkuH3d/iFovdxS7IuMq4VPv7/3zjzR/9+Cc//dlbPx/+4pe/+vVv3n7nty9kuRAJe56UWSlexlSyjBfsueIqYy8rwWgeZ+zr+OoR8r9eMiF5WZypm4pd5HRW8ClPqALSi2iZlkpevn1nf2/f/JD1h4l7uDNwP8eX7+y8G6VlsshZoZKMSnk+mVTqoqZC8SRjejgcjtqf4Ygc0+SKzpjcG0YLySo7Ol+o6f2LmhfVQrEi0WREaJaV1wTou/eJoQf4s8lFPS0LC4afEQEmub8bc0XOJgRZ0heoaS5zqua6TzTINaq8yeM1oprnIW0ORhSCTY0GRglDAftfhS9fiKzBOKDkeZUx8vzkCcGtkEwpXswCobgsrxSNZTd7Jcopk7h/NNt9taAZVzcEIBlbW6tdVisZZ8CKSypSglYguL4yC6UKnrCpoEknlZQ5MFWDBqsKMvngYEyYSoLtq3OeiBIXohvZhlLOBK3mN8HmxaVSZY4bWKqcyyS06nSRZRU8BNTzYpHH4LsXdUFVzHubU1Ehr3gVEgt2Xa0UW6mxebK7D85Idnd3ydFKCdr6IpLCCTkacEzkIp7y2Zgk8tWiVEyOSUrlXCwyNiaKX30/JnGcwy/4Z9bJk9WY5ItMcVFejwlEWwzBBVx8u5BTmIBms1Jw8CXzWEm2gOgpU5gwT8H8OUvBvrBarlgOcXBN5Q3Y385A45V9iGE/QIOV894ReYwP4Z6gZopXNLRKIw3UzxjEq2CnZndxgodZBa9gqo4sStcPn57Gr4PMdI2TptbADzPFBOwTXzKiSjSRcR2IZyZIAhnK88dP2vUMIxM9EAqLqh6iF5mUZqLpUIkFGxtiDCntqk+EADPgwzhrSAjp0xIwakcbhh6W0IoriKnv2biAbeAFznBRQw6DbYQgh6VFKZtCcjVT1Klgqa5PPv9U15P9/TG5B/8+3Nc9UI5vcrD98R8/HB9M7vUx13PQy2EO7t0bk+aXNtaMYclLbpI0bjF4MwRmTosUrJ/oc8iEZhcSmtV3Jlr3ELJDyERsQsQdArZqE0J0CLEJwCqp62gJLlJJnpVFj82LlCe6e8OafHwEXAmMvYMWVh/pPgxmhijT7oHP5ooKiLMNqKxBZWy6EWSm2T6FEd8i+iWqGNdf9rX7xtK/6dO/svSv+vQTSz9ZM2ahRFndWO4X2jpLtMx5gTtRL/Wl+Qtj3THpKmTSVctEVwPrKnBzwTLDBpLW9aE2/nWaw1FrotDXA7bj/AD2XIJbsbhc4a6jRvAEGsqFYChRRymXVUZvpLqB4+zOgcbXjo5WFI+3Twhu6r7d1KR+qW16oGIGyo/NX7raa9PLU5jwGWQAqkpxt44sTEMhMYvG+HQbENbrgPC0DbhqkKto3IGbM2HDj2GQL4/glP4MA5abGCSOsVECOI/KAjLGDEoTtme3oIqN2Y9xdyBr0mZrqjgGU3MVciM8tOpdbagWoHUnkqfVLXjg2j1IKicAMdnhmwDVHbOvgoPcrgWAAj02CTWqoE3OmDTnEwoz8I46UnShu7E0HnLm0EfFEg7QAkvKVqgj1Uy3tNpn6DWorM28R63iwJOBjLQ5NjE7OybPYsnEkrrRqaIKayuUpBZi5nvYTFd6cMN51nCkETW005aGkdYwTGR8pBuEVWNmVzyyb8RRHZVLcyg6iY9NgqykKjXmKKju09adnIAjjqaXHWDkIeDlTxwEX/5YX7YiFhWXiyJlqZtO8SxltRXVWJHwlMWCpzNGY9BtXUSGMrJVL8u2wLKsRfpm6FSewj4uWdJq4S94aozjJti46Gb6tlrxNHNvI59SyYNXxg5kE+qn8E7i27Bhy47vC+NsDhH3BD2WjIOF8NVmkY6BAiNywmgyZ422RamEJdT4bE81qFf5rDm9ENVCNrMTWhjEBr5zxuwasvw2jH9w+MgRlF22bMCTpyyhvJrBMTaXcicCeUjROziIrqBe3IWD4oCtojmeMfVOo/GOtvmMfM4KzOTtumfNGFwDq3ITQU8YFQV0UgRr9AaZIREJEEMcTgDdI8MMpx4d/MVntT5jMQ6EXXU7a/uoR+78jfUGoPSQcgMUXhyiA002CCw3K7HGDF/cvmx5y8t8/4MDvqg7X2wUQXK3TGhhwEZzqsKU7LPlOj8IeVKasxqTaVO6YAsKRQZs7xJ9AJeRQu4b911wTKKTrpyD0ROo3gICjKAkkdiMRXPoTaGPLqCzc5rBQ1KaMWjIXsFf41OdaiZt723K7WQTxFhyWi98V7OJCoSee9l+10iRqsx4ctPkHzuqo4pvnt0BzHzHGzB+OdlgXR/ghpe2ih+abHINfZgkgCYvKHQreACZ/gTjCHoGXikigiYSakIIZEHDSitsCltMfQaVYzfKdV00SUWYV2NyD4r/BlyLruJooEKPIP/My5aOpa8AhzSVh617+wionDuEKZt7CDt3sz0nbXihNaZQVPsw1AArc2E38QsO7ih4UyXMm6GZqa3ehUePqag7XIuQi+nUQ9FkodgmHMMWFe98kFV7M28EUGgebQmBt4Ib5msJ1DiPh/g2SrhIcJHuiHBl7bQ2RHKcgQiHlvrG1YTteAqLbzy/o0KnoIQxy1OsGN1JvhGFV0B95Hu2kPsWOg6al+CrUc5TMr9sdu99o9QjCj7aqpS4UaMy3vw8U3MmyFMukz2o1E2Q2Psf4EORT9grV3/V+Iu90v0KDGs240qmi7JtlBdxdzu5rufySjfTQnuiaMmFIpRMaZHcEDj8oDHCMomoOQRgv+NPAWAieQi1/ww8u7k10qNzI5aUAs7Fw/1KXQyjpQQDsHqS53qIET+E3Jt2IkPz+rM5K6Gdc29SdlS7vxC+7sHn3m3Ydzfz6ymUy7p+jL8DOmhXQjkmoL191D4GCGZ7R127JjLkYtsprvS5G1/UJ5YQgDIGxvIwT8w4gFTQYpfS9HMe8NijBvC07f503XWCPZN0oLtbUQCSUD/DKUcR0z73jFR8xxIFPTZaqX329oq461DcMvN0zdFbagY1ldVSsqR5uQ9od3UL319oCxmFGLcH26bwtd82hW/+rSDPV7ZB7E53bHOg/bmAACBekWFuH2Fvy6k0ds5TYJ5DaarKCtPi4ZRm0lwR2pvpkGaqCCSRQ9IS8XIIrAABeHi/Ui3MkfYtCe+kTY90OPkYKHBMwwuJxQCbo55ACYUM1erRMYb42QjQK1yEzRn41CQwsyq/Q04tEPLEuQFe1C5bOLpJYPHUJjJo/u1s8P+dyZ7WF8QmOD99aOJm8PTQtUknHsWM3aMv/P81lj9I49MrppL5VsUPfojioKzfzlh/A2/DQzlNVZlAllVMeLE20ZGiM1BtzlqitrfFkQs4uoBDlU3xYtlLmO5YdeHiY/ywd8c6Bp0PaTMfsjHf+kyXec3Z2sWajwhzndXExVuoSJelzQWKW6CPOWoW3XQW7hMOH/OwIXq3ksWaUkHWNDrNaQXGDjSypGZ1VAQLP4axY5lPxgLDI6ExGJ/1TP7YUJqrmkU8zUqqbgXcPoXJTT63OYaMuM1YPvu0TafGiFIu8qqPedhSG/tklAd2foQEkwgh5HJ6xcyHHookUPVf4fqDQp6pssywohhGiKUqYwpsOwSzQlwlOV53Pzx7AN2OwpiB5EcMociDofng5kHdsHQN/zUzllhv2R6vqiDK+Iq40JLBoWeCMVlA8Zmjl4RlhZ8pOgwkC0iYUAMzL15hvnAqrHui3uxGDJXpkbEbwqjappk5Ov2SY10xhLyGYjhxTzGz2nXFDBlvlV0gbtMNwnetqFpXD1GvoR7A+uqh5Ab1DNlk9yadbFMQE5Fer7HWdTTA11AScX0tjewGNS19RDAxblOwX7Sua2YQr6EZ4vqaGdkNmlk6Xl3BTNtUy9DxvDp2XbXs9fwuW/e7bLPfOTIGrs0vxNJd7clSYs8k+YHqGgjMSWFQbT/5ejbYcnyFGm3Lp2G8bD0mQ7+97Zzruc7Wk9MktjQltCDmewEEK1Miy5wRU4OE3ZshmTu6prxJGBYTOpo3XyyoDxLY6kml7C9TmDiQfdsj6MozsH7XAngJHb+UELajposHPPaM+HEeXudI/j1kbsSeN18YOnwPjlqxh8P3L+zH781PVJQpO5dw7LJDO9k4FfR6bExE4N2HB9BsEjPB+wS96094beNPcfn2nUn/O0TrDy8O9iYf7e1/de/Og+d/sd8vemvw+8G7g/cGk8EfBg8GXwyOB88HyeC7wV8Hfxv8fecfO//a+ffOfyz0zTfcd5J+Nwh+dv77Px0vF9g=</latexit>

⌘
<latexit sha1_base64="rgfaSOwiaoN6fKRls5UCaqbnwdA=">AAAl2XicnVpfbxy3EZfTpk2v/5z2sS9E7AMS96ToFCd1Uwiw48hJATtxJDkJolUE7i7vjtX+M8mTTlkQaN+KvvYD9HP123RmyN3l7t25RpRYImd+Qw7JmeEM7+Iqk9rs7//31hs/+embP/v5W78Y/fJXv/7Nb2+//buvdblUiXiRlFmpvo25FpksxAsjTSa+rZTgeZyJb+LLx8j/5kooLcvi1NxU4jzn80LOZMINkC5u/2cc8bkoDIwh9fcfMBblMmXRQlc8EfVBnlsWAqYDVgieDsEHAxaBw9lGfnJNfRZBWyhuhPYwPVpc3L6zv7dPP2y9MfWNOzv+5/nF23ffidIyWeYgn2Rc67PptDLnNVdGJpmwo9Fo3P6Mxuw5Ty5hMr03ipZaVK53tjSzB+e1LKqlEUVi2ZjxLCuvGdB3HzCi9/Cn0/N6VhYODD9jBkz2YDeWhp1OGbJ0KFDzXOfcLOyQSMg1qr7J4zWiWeR92gLOVykxIw1ICaKAaVz2J1+qrMF4oJZ5lQn24vgpQyvRwhhZzHtCcVleGh7rbvRKlTOh0bR4tvtyyTNpbhhAMrG2VresVjLOgBWXXKUMd4Hh+sqsL1XIRMwUTzqppMyBaRo07Kpi0/cPJkyYpHd8dS4TVeJCbCPbUMq54tXipnd4cWlMmeMBliaXOunv6myZZRU0etSzYpnH4FbndcFNLAeHU3GlL2XVJxbiuloZsTITarnTB2Nku7u77GhlFG9tEUn9ASVu4ITpZTyT8wlL9MtlCX4yYSnXC7XMxIQZefnDhMVxDr/gH61TJqsJy5eZkaq8njAIBDH4PXBxdqVnMADP5qWSYEvUrLRYgveUKQyYp7D9uUhhf2G10gjw4fE11zew/24EHq9cI4bzAA1W3nrH7Ak2+meCmhlZ8f6uNNJA/VSAvypxQqeLAzzKKphCmDpyKFs/enYSvw4yszUOmroNfpQZoeCc5JVgpsQtItMBfxaKJRA8A3v8uF3PKCLvAVdYVvUIrYiiLXnToVFLMSFiDNH2ckgEByPwYZw1JIQMaQlsakcb9S0s4ZU04FM/iEkBxyALHOG8hhgGxwhODkuLUjGDuE9D1KkSqa2PP/vE1tP9/Qm7D/8+2LcDUI4zedj+5M8fTA6m94eY6wXo5TEH9+9PWPPL0m7GsOQrSfcHHjFYMzhmzosUdj+xZxAJ6RQSntV3ptYOELpD6ERtQsQdAo5qE0J1CLUJICpt6+gKTKTSMiuLAVsWqUxsN8OafHwEXA2MvYMWVh/ZIQxGBi+zviHnC8MV+NkGVNagMjHbCKJhtg9B4ltEv0AV4/qLoXbfOfp3Q/pXjv7VkH7s6Mdrm1kYVVY3jvu5dcYSXeWywJOor+wF/YW+7Zh81WfyVctEU4PdNWDmSmTEBpK19aEl+zrJ4aolLwz1gOM4O4Az12BWIi5XeOqoEbRAQ71UAiXqKJW6yviNNjdwnd05sDjt+GjF8Xr7mOGh7rtDTepvrQsPXM1B+Qn95au9Nrw8gwG/rDAhKdW9OnIwC4nEPJpg61VAWK8HQmsbcNUgV9GkAzd3woYfYrAvjuCW/hQdVpIPMs/YKAGcx2UBEQPyqUTsuSOoYtr253g6EDV5czRVHMNWS9PnRnhp1buWqA5gbSeSp9Ur8MB1Z5BUXgB8ssM3Dmo75lAFD3m1FgDq6bFJqFEF9+RUaLqfUFiAddSR4Uvb9TVZyKlHHxVXcIEWmFK2Qh2pFral1SHDrkF1TeMetYoDT/dktIuxCZ3shH0Za6GuuO+dGEyOnSR3EBrvUTNcGcCJ82XD0SRKtJOWhp7WMMgzPrQNwqkxdyseuxmxV0flFV2KXuIjCpCVNqXFGAWFR9qakxfwxPHsogOMAwRM/tRDcPIn9qIVcai4XBapSP1wRmapqJ2oxYxEpiJWMp0LHoNu6yK6L6Nb9bJsCyzLWmS4DZ3KMzjHK5G0WoQLntHm+AE2LroZvs1WAs38bOwTrIjCKWMPcgH1E5iThXvYsHXHD4VxNI+IB4IBS8e9hcjVZpGOgQJjdix4shCNtkVplCPU2Ha3GuSrct7cXohqIZvZCS8IsYHvjTG7hii/DRNeHCFyPMICF9MGvHnKEtKrOVxjC63vRiAPIfoudqJLyBd34aI4EKtogXdMfbfR+K518Yx91hSrft1t8QqmgVk5edBTwVUBlRTDHL1BZkhEAviQhBvADsgwwklAB3sJWa3NOIwHYcHfjto27djfv7HdANQBUm+AwsR9dE+TDQJXm5VYY/Ynbie7esVkof3BBV/UnS02iiC5WyaUMLBHC276ITlk63V+z+VZSXc1BtMmdcESFJIMON4rtAFcRgqxbzI0wQmLjrt0DnpPIXvrEaAHKYnGYixaQG0KdXQBlZ3XDBpJSX3QULyEv2RTnWoUtvc2xXa2CUI7OauXoam5QAVCL4Jov0tSrCozmdw08cf16qiSm0f3ABrv+QZMmE42WF8H+O6Fy+JHFE2uoQ7TDNDsaw7VCl5AVJ+gH0HNICvDVK+IhJwQHFnxfqbVLwpbTH0KmWPXy21dNEFF0dQY3HvJfwOuVZdxNFBlxxB/FmVLx9RXgUFS5uHy3iECMucOQWnzAOHGbo7nuHUv3I0ZJNUhDDXAzFy5Q/xcgjkq2WQJi6ZLI7XZuwroMVd1h2sRejmbBSieLI3YhBNYouKbD7LqYOSNAA7Fo0sh8MFyw3gtgZPxBIjvo0SqBBfprwif1s5qIrLnGYhIKKlvfE7Y9mew+MbyOypUCkbRtjzDjNHf5BtR+AQ0RL7rErnvoeLgeQm2Ss+bi4vm9N4jpR5zsNFWpcT3GpXx5edLsxCKPZM62YNMnZzEvf8AH5J8Jl76/KvGX+KlHWZgmLORKVEV5cqowOPudXJdzRWkblRCB6K4k0vDOJvxIrlhcPlBYYRpEjMLcMBhxZ8CgDx5BLn/HCy7eTWy4zMSS0oF9+LhfmXOR9FV9xw8Qo8fQexNO5ERTX+6ECWUc34m43q1/wvu6xsh917DvreZX88gXbb1E/zdo4N2JaRjCsrbx22zhxCudrS1LyL7XCw71aU98/3z+tgReqBMwGYFmKfU70EqKLFLTfVcAHweUHvwtK3+bN1VgoMt6UD3tqIApCF/hluOI6ZtDzap+JtIDNTYuEttOzgr5p9D8ciodS3RWmoBOZXTUoukmTwEtKe6hR8utIWM+xh/BtuGCLXfNkS4/VtBga1sg7iT7th0of21AAdgQZJBr49wtuVM0z7nKTDPIDU1ZYVh8XDGM01PhO5luk+jLAJJ7JC1RHwcgl0ABzx8UJkW5kn7joRv0lQjHU4/Agpc0zAhcxhgS9QTKH0hojo9OsYIPxsBeoWLcDEDW00Ao1WFFXLqgBAnzgh4Xvto4ekUwOKZC2RQ/LvR4P870z1rz5kLcGH4sMyPEOhhawonAYX6vhkK/3+N9Y/S+ORSmGSxVfGDH6M4KBuWM87ewNrwUk5TUyYQZY1Qga9NbWT4HFRbiJZo3Wtx5B2OL+FSFTN8WA4Cpr9WvbuEmNDt/bWOThdC2siHbIy3IdNHXrpbO18LEf1Y5zTx/tZXpIvS9IDiFxhijppFN5WF/4QjxDxqiMGrZLGmVC9qkk4LXsFm9zRypGZ1XPUW/hz6nkWfjPU2HgnNhsn5YMufEKV5qlnGs6zk5pWAVw9BsSnkNtcQibuIFbJP2nBKm6j1Mq+GmEcttdmfjMvePj9GAgVCcLmcXwr60MOwBLL+S1x/L5EXpiwzzChGEWK5yYSBvR3BtoJfJTk+dz86fQjVjkGfgeDHiFDkvS59cPOwbli2hv+aEUvMt1yNV1XgZXLFvGvp3qVHzpgsIfnM0Ur6aUUYKToMBAsImJADi8BfYbz+UJj3RIPRSQyVGZCxGkKv2qYZXZ1hyrGuGEJeQzEceKAYrXZdMSLjq7J3xG26gfuuJVXr6iHqNdQD2FA9lNygHpEpujfhZJuCGIjseo61riMBX0NJxA21JNkNajr6mGFg3KbgMGld14wQr6EZ4oaakewGzRwdn65gpG2qZWh4QR67rlr2enaXrdtdttnuPBkd18UX5ug+9xQpc3eSft90BQTGpL5Tbb/5Bnuw5frqa7Qtnvb9Zes12bfbV91zA9PZenNSYEtTxgtG3wtgmJkyXeaCUQ7Sr96IRG90TXqTCEwmbLRovlhQHyRw1NPKuF+UmHiQm+0xVOUZ7H5XAgQBHb+U0C9HqYoHPNaM+HEePudo+QNEbsSeNd9lOnwXrlq1h933zt3H781PVJSpONNw7YpDN9gkVfx6QlvEYO7DAyg2GQ3wHkPr+gs+24RDXNy+Mx1+h2i98fXB3vTDvf2v7t95+OLv7vtFb+38YeednXd3pjt/2nm48/nO850XO8mtN2/98db9Wx+Oz8b/GP9z/C8HfeOW/07S73d6P+N//w9GdE0T</latexit>

⌫2B ⌫ 8
<latexit sha1_base64="SO7Y9mgGpBgo+2tPNNq+7PKM59w=">AAAlPnicnVpfbxy3Eb+k/9Lrv6R97AsR64DEOCk6xUndFALsOHJSwE4USXaCaBWBu8u7Y7T/TPKkUxYE+hH62n6afo1+gb4Vfe1jZ0juLrm35xpRYmk58xtyOJwZzuxdXGVcqv39f772+o9+/JOf/uyNn49/8ctf/fo3b7712+eyXImEPUvKrBRfx1SyjBfsmeIqY19XgtE8zthX8dUj5H91zYTkZXGmbit2kdNFwec8oQpIzyO6YIW6fPPO/t6++SGbDzP3cGfkfo4v39p5O0rLZJWDcJJRKc9ns0pd1FQonmRMj8fjSfsznpBjmlzBSnJvHK0kq+zofKXm9y9qXlQrxYpEkwmhWVbeEKDv3ieGHuDPZhf1vCwsGH4mBJjk/m7MFTmbEWRJX6CmucypWuo+0SA3qPI2jzeIapmHtCUYUQg2NxoYJQwF7H8VLr4SWYNxQMnzKmPk2ckTgkchmVK8WARCcVleKRrLbvZKlHMm8fxotvtiRTOubglAMraxV7utVjLOgBWXVKQErUBwf2UWShU8YXNBk04qKXNgqgYNVhVk9t7BlDCVBMdX5zwRJW5EN7INpVwIWi1vg8OLS6XKHA+wVDmXSWjV+SrLKngIqOfFKo/Bdy/qgqqY9w6nokJe8SokFuymWiu2VlPzZE8fnJHs7u6So7UStPVFJIUTcjTglMhVPOeLKUnki1WpmJySlMqlWGVsShS/+n5K4jiHX/DP7JMn6ynJV5nioryZEoi2GIILuLi6kHOYgGaLUnDwJfNYSbaC6ClTmDBPwfw5S8G+sFuuWA5xcEPlLdjfzkDjtX2I4TxAg7Xz3gl5jA/hmaBmilc0tEojDdRPGMSrYKfmdHGCh1kFSzBVRxal64dPT+NXQWa6xklTa+CHmWICzolfM6JKNJFxHYhnJkgCGcrzx4/a/YwjEz0QCquqHqMXmZRmoulQiRWbGmIMKe2qT4QAM+DDOGtICOnTEjBqRxuHHpbQiiuIqe/ZtIBj4AXOcFFDDoNjhCCHrUUpm0NyNVPUqWCprk8+/VjXs/39KbkH/97f1z1Qjis52P70j+9PD2b3+pibJejlMAf37k1J80sba8aw5WtukjQeMXgzBGZOixSsn+hzyITmFBKa1XdmWvcQskPIRAwh4g4BRzWEEB1CDAFYJXUdXYOLVJJnZdFj8yLlie5W2JCPj4ArgbF30MLqI92HwcwQZdo98MVSUQFxNoDKGlTG5oMgM832KYz4FtHPUcW4/ryv3TeW/k2f/qWlf9mnn1j6yYYxCyXK6tZyP9PWWaLrnBd4EvW1vjR/Yaw7Jl2HTLpumehqYF0Fbi5YZthA0ro+1Ma/TnO4ak0U+nrAcZwfwJlLcCsWl2s8ddQInkBDuRIMJeoo5bLK6K1Ut3Cd3TnQuOzkaE3xevuI4KHu20NN6q+1TQ9ULED5qflL13ttenkKE34BGYCqUtytIwvTUEgsoik+vQwI+3VAeNoGXDfIdTTtwM2dMPBjGOTzI7ilP8GA5SYGiWMMSgDnUVlAxoBiKmF79giq2Jj9GE8HsiZtjqaKYzA1VyE3wkur3tWGagFadyJ5Wr0ED1x7BknlBCAmO3wToLpj9lVwkJdrAaBAjyGhRhW0yRmT5n5CYQbeUUeKrnQ3lsZDzhz6qLiGC7TAkrIV6kg10y2t9hl6AyprM+9RqzjwZCAjbY5NzMlOyRexZOKautGpogprK5SkFmLme9hMV3pww/mi4UgjaminLQ0jrWGYyPhANwirxsLueGJXxFEdldfmUnQSH5oEWUlVasxRUN2nrTs5AUeczC87wMRDwOJPHAQXf6wvWxGListVkbLUTad4lrLaimqsSHjKYsHTBaMx6LYpIkMZ2aqXZVtgWdYifTN0Ks/hHK9Z0mrhb3hujOMmGNx0M31brXiaudXIx1TyYMnYgWxC/RjWJL4NG7bs+L4wzuYQcU/QY8k42AhfD4t0DBSYkBNGkyVrtC1KJSyhxmd7q0G9yhfN7YWoFjLMTmhhEAN854zZDWT5bRj/4vCREyi7bNmAN09ZQnm1gGtsKeVOBPKQondwEF1BvbgLF8UBW0dLvGPqnUbjHW3zGfmUFZjJ230vmjG4BlblJoKeMCoK6KQI1ugNMkMiEiCGONwAukeGGU49OviLz2p9xmIcCLvqdtb2UU/c/RvrAaD0kHIACguH6ECTAYHrYSU2mOHC7WLXL1nM9z+44Iu688VGESR324QWBmy0pCpMyT5bbvKDkCeluasxmTalC7agUGTA8V6jD+A2Ush9074LTkl00pVzMHoC1VtAgBGUJBKbsWgJvSn00QV0dk4zeEhKMwYN2Qv4a3yqU82k7b2h3E6GIMaS83rlu5pNVCD0zMv2u0aKVGXGk9sm/9hRHVV8eHYHMPMdD2D8crLBuj7ADS9tFT822eQG+jBJAE2eU+hW8AIy/QnGEfQMvFJEBE0k1IQQyIKGlVbYFLaY+gwqx26U67pokoowS2NyD4r/BlyLruJooEJPIP8sy5aOpa8AhzSVh617+wionDuEKZt7CDt3czwnbXihNeZQVPsw1AArc2EP8TMO7ih4UyUsm6GZqa3ehUePqag7XIuQq/ncQ9FkpdgQjmGLiu98kFV7Mw8CKDSPtoTAt4ID87UEapzHQ3wbJVwkuEl3Rbiydl4bIjnOQIRDS33rasJ2PIfNN57fUaFTUMKY5SlWjO4mH0ThK6A+8h1byH0LHQfNS/DVKOcpWV42p/euUeoRBR9tVUrcqFEZ3/x8oZZMkKdcJntQqZsgse9/gA9FPmEvXP1V4y/2QvcrMKzZjCuZLsq2UV7E3e3kup7LK91MC+2JoiVXilAyp0VyS+Dyg8YIyySilhCA/Y4/BYCJ5DHU/gvw7OatkZ6cG7GkFHAvHu5X6mIcXUswAKtnea7HGPFjyL1pJzI2y58tWQntnFtJ2VHt/kL4ugefe7dh3x3m13Mol3X9GH8HdNCuhHJMQHv7qH0MEMz2jrp2TWTIxbZTXOlzN76oTywhAGUMjOVhnphxAKmgxS6l6ec84LFHDeBp2/3puusEeybpQHe3ogAkoX6GW44ipn3uGan4jiUKemy0UvvsnRVxr0PxyMzTDUdvqRnUVFZLyZJmcR/QnuoWvr/RFjIJMe4Mtk3ha79tCt/8W0Ger2yD2JPu2OZC+3MBAUC8IsO8fYSzLefS2DlPgXkOpakqK0yLh3OaSfOK0L6ZDmmmikASOSQtEV8OgRUgAA/vV6qFOdK+JeE7adMjHc4+BApc07AgsRhgc9QTKKGQoVo9OsYYPxsBeoWbsDkDn5oEZnbld8ipBUKeODfAi9plC0c3CSye20QGzb+dDf6/M9vT+oLYBOenD03cDJ4eujbpxKOYsXv0hf+/xvIHaXx6xVSy3Kr4wQ9RHJT12xnrb+BteCmnqSoTyLKKCS/WZjpSdAGqLVlL1PZtceQCjq7gUmVzfLHsJUx3rbpw8TF+2LtrHYPOh7SZD9mYb32my7zmbu1izUeEuc5q4uItVKTL0uYFitugjzlqNt10Fu4TDh/zsCF6byWLDaWCrGl0WtIKjB1oZEnN7qgINn4MY8cyn4wFhkdCYzC+6Jn8saE0r2pW8TwrqXop4OVTmNzkc5tryIjbjOWzT9t0aowo5Sqv+piHLbWxT0Z5YOdHSDCJEEIup1fMfOihSAJV/xXuPyjkmSrLDCuKcYRYqjKmwLZjMCvEVZLj6+6HZw+g21EYM5D8iCEUeTA0H9w8qBuWruG/ZsYS6y3b41UVRBlfExdaMrj0TDAmKyg+c/SSsKzwM0WHgWQBCRNqYObFK8wXToV1T9Sb3YihMj0ydkMYVds0M1enX3JsKoaQV1AMJ+4pZna7qZgh41tlF4jbdIPw3SiqNtVD1CuoB7C+eig5oJ4hm+zepJNtCmIi0ps11qaOBvgKSiKur6WRHVDT0icEE+M2BftF66ZmBvEKmiGur5mRHdDM0vHVFcy0TbUMHc+rYzdVy17N77JNv8uG/c6RMXBtfiGW7mpPlhJ7J8n3VNdAYE4Kg2r7zdezwZbrK9RoWz4N42XrNRn67cvuuZ7rbL05TWJLU0ILYr4XQLAyJbLMGTE1SNi9GZJ5R9eUNwnDYkJHy+aLBfVBAkc9q5T9ZQoTB7KrPYKuPAPrdy2Al9DxSwlhO2q6eMBjz4gf5+HrHMm/h8yN2PPmC0OH78BVK/Zw+O6F/fi9+YmKMmXnEq5ddmgnm6aC3kyNiQisfXgAzSYxE7xL0Lv+hK9t/Cku37wz63+HaPPh+cHe7IO9/S/v3Xnw7C/2+0VvjH4/env0zmg2+sPoweiz0fHo2SgZfTf66+hvo7/v/GPnXzv/3vmPhb7+mvtO0u9Gwc/Of/8HXQoXtw==</latexit>

H
.. 3 2 ⇤ ⌫1
⌫{⌫ ⇤B } 2
⌫1B {⌫ 2 ⇤ ⌫ / ⇤B }
<latexit sha1_base64="EGh0EySAkPfByUtB+9KdXFKYrPQ=">AAAlRXicnVpfbxy3Eb+k/9Lrv6R97AsR64DEOCk6xUndFALsOHJSwI4VSU6CaBWBu8u7Y7X/TPKkUxYE+i362n6afoZ+iL4VfW1nSO4uubfnGFFiaTnzG3I4nBnO7F1cZVyq/f1/vfb6j378k5/+7I2fj3/xy1/9+jdvvvXbL2W5Egl7npRZKb6OqWQZL9hzxVXGvq4Eo3mcsa/iq0fI/+qaCcnL4kzdVuwip4uCz3lCFZAuIrpghYIJuPz2/cs37+zv7Zsfsvkwcw93Ru7n+PKtnbejtExWOcyRZFTK89msUhc1FYonGdPj8XjS/own5JgmV7Cg3BtHK8kqOzpfqfn9i5oX1UqxItFkQmiWlTcE6Lv3iaEH+LPZRT0vCwuGnwkBJrm/G3NFzmYEWdIXqGkuc6qWuk80yA2qvM3jDaJa5iFtCbYUgs2NBkYJQ4FjuAoXX4mswTig5HmVMfL85AnBE5FMKV4sAqG4LK8UjWU3eyXKOZN4jDTbfbGiGVe3BCAZ29ir3VYrGWfAiksqUoJWILi/MgulCp6wuaBJJ5WUOTBVgwarCjJ772BKmEqC46tznogSN6Ib2YZSLgStlrfB4cWlUmWOB1iqnMsktOp8lWUVPATU82KVx+DCF3VBVcx7h1NRIa94FRILdlOtFVurqXmypw/OSHZ3d8nRWgna+iKSwgk5GnBK5Cqe88WUJPLFqlRMTklK5VKsMjYlil99NyVxnMMv+Gf2yZP1lOSrTHFR3kwJBF0MMQZcXF3IOUxAs0UpOPiSeawkW0H0lClMmKdg/pylYF/YLVcshzi4ofIW7G9noPHaPsRwHqDB2nnvhDzGh/BMUDPFKxpapZEG6icM4lWwU3O6OMHDrIIlmKoji9L1w6en8asgM13jpKk18MNMMQHnxK8ZUSWayLgOxDMTJIFE5fnjR+1+xpGJHgiFVVWP0YtMZjPRdKjEik0NMYbMdtUnQoAZ8GGcNSSE9GkJGLWjjUMPS2jFFcTUd2xawDHwAme4qCGHwTFCkMPWopTNIceaKepUsFTXJ59+rOvZ/v6U3IN/7+/rHijHlRxsf/rH96cHs3t9zM0S9HKYg3v3pqT5pY01Y9jyNTe5Go8YvBkCM6dFCtZP9DlkQnMKCc3qOzOtewjZIWQihhBxh4CjGkKIDiGGAKySuo6uwUUqybOy6LF5kfJEdytsyMdHwJXA2DtoYfWR7sNgZogy7R74YqmogDgbQGUNKmPzQZCZZvsURnyL6OeoYlx/3tfuG0v/pk//wtK/6NNPLP1kw5iFEmV1a7mfaess0XXOCzyJ+lpfmr8w1h2TrkMmXbdMdDWwrgI3FywzbCBpXR9q41+nOVy1Jgp9PeA4zg/gzCW4FYvLNZ46agRPoKFcCYYSdZRyWWX0VqpbuM7uHGhcdnK0pni9fUTwUPftoSb119qmByoWoPzU/KXrvTa9PIUJn0EGoKoUd+vIwjQUEotoik8vA8J+HRCetgHXDXIdTTtwcycM/BgG+fwIbulPMGC5iUHiGIMSwHlUFpAxoKZK2J49gio2Zj/G04GsSZujqeIYTM1VyI3w0qp3taFagNadSJ5WL8ED155BUjkBiMkO3wSo7ph9FRzk5VoAKNBjSKhRBW1yxqS5n1CYgXfUkaIr3Y2l8ZAzhz4qruECLbCkbIU6Us10S6t9ht6AytrMe9QqDjwZyEibYxNzslPyLJZMXFM3OlVUYW2FktRCzHwPm+lKD244zxqONKKGdtrSMNIahomMD3SDsGos7I4ndkUc1VF5bS5FJ/GhSZCVVKXGHAVFftq6kxNwxMn8sgNMPAQs/sRBcPHH+rIVsai4XBUpS910imcpq62oxoqEpywWPF0wGoNumyIylJGtelm2BZZlLdI3Q6fyHM7xmiWtFv6G58Y4boLBTTfTt9WKp5lbjXyM7Y+/ZOxANqF+DGsS34YNW3Z8Xxhnc4i4J+ixZBxshK+HRToGCkzICaPJkjXaFqUSllDjs73VoF7li+b2QlQLGWYntDCIAb5zxuwGsvw2jH9x+MgJlF22bMCbpyyhvFrANbaUcicCeUjROziIrqBe3IWL4oCtoyXeMfVOo/GOtvmMfMoKzOTtvhfNGFwDq3ITQU8YFQV0UgRr9AaZIREJEEMcbgDdI8MMpx4d/MVntT5jMQ6EzXU7a/uoJ+7+jfUAUHpIOQCFhUN0oMmAwPWwEhvMcOF2seuXLOb7H1zwRd35YqMIkrttQgsDNlpSFaZkny03+UHIk9Lc1ZhMm9IFW1AoMuB4r9EHcBsp5L5p3wWnJDrpyjkYPYHqLSDACEoSic1YtITeFProAjo7pxk8JKUZg4bsBfw1PtWpZtL23lBuJ0MQY8l5vfJdzSYqEHruZftdI0WqMuPJbZN/7KiOKj48uwOY+Y4HMH452WBdH+CGl7aKH5tscgN9mCSAJl9S6FbwAjL9CcYR9Ay8UkQETSTUhBDIgoaVVtgUtpj6DCrHbpTrumiSijBLY3IPiv8GXIuu4migQk8g/yzLlo6lrwCHNJWHrXv7CKicO4Qpm3sIO3dzPCdteKE15lBU+zDUACtzYQ/xMw7uKHhTJSyboZmprd6FR4+pqDtci5Cr+dxD0WSl2BCOYYuK73yQVXszDwIoNI+2hMCXgwPztQRqnMdDfBslXCS4SXdFuLJ2XhsiOc5AhENLfetqwnY8h803nt9RoVNQwpjlKVaM7iYfROEroD7yHVvIfQsdB81L8NUo5ylZXjan965R6hEFH21VStyoURnf/DxTSybIUy6TPajUTZDY9z/AhyKfsBeu/qrxF3uh+xUY1mzGlUwXZdsoL+LudnJdz+WVbqaF9kTRkitFKJnTIrklcPlBY4RlElFLCMB+x58CwETyGGr/BXh289ZIT86NWFIKuBcP9yt1MY6uJRiA1bM812OM+DHk3rQTGZvlz5ashHbOraTsqHZ/IXzdg8+927DvDvPrOZTLun6MvwM6aFdCOSagvX3UPgYIZntHXbsmMuRi2ymu9LkbX9QnlhCAMgbG8jBPzDiAVNBil9L0cx7w2KMG8LTt/nTddYI9k3Sgu1tRAJJQP8MtRxHTPveMVPyFJQp6bLRS++ydFXGvQ/HIzNMNR2+pGdRUVkvJkmZxH9Ce6ha+v9EWMgkx7gy2TeFrv20K3/xbQZ6vbIPYk+7Y5kL7cwEBQLwiw7x9hLMt59LYOU+BeQ6lqSorTIuHc5pJ84rQvpkOaaaKQBI5JC0RXw6BFSAAD+9XqoU50r4l4Ttp0yMdzj4EClzTsCCxGGBz1BMooZChWj06xhg/GwF6hZuwOQOfmgRmduV3yKkFQp44N8CL2mULRzcJLJ7bRAbNv50N/r8z29P6gtgE56cPTdwMnh66NunEo5ixe/SFv19j+YM0Pr1iKlluVfzghygOyvrtjPU38Da8lNNUlQlkWcWEF2szHSm6ANWWrCVq+7Y4cgFHV3Cpsjm+WPYSprtWXbj4GD/s3bWOQedD2syHbMy3PtNlXnO3drHmI8JcZzVx8RYq0mVp8wLFbdDHHDWbbjoL9wmHj3nYEL23ksWGUkHWNDotaQXGDjSypGZ3VAQbP4axY5lPxgLDI6ExGF/0TP7YUJpXNat4npVUvRTw8ilMbvK5zTVkxG3G8tmnbTo1RpRylVd9zMOW2tgnozyw8yMkmEQIIZfTK2Y+9FAkgar/CvcfFPJMlWWGFcU4QixVGVNg2zGYFeIqyfF198OzB9DtKIwZSH7EEIo8GJoPbh7UDUvX8F8zY4n1lu3xqgqijK+JCy0ZXHomGJMVFJ85eklYVviZosNAsoCECTUw8+IV5gunwron6s1uxFCZHhm7IYyqbZqZq9MvOTYVQ8grKIYT9xQzu91UzJDxrbILxG26QfhuFFWb6iHqFdQDWF89lBxQz5BNdm/SyTYFMRHpzRprU0cDfAUlEdfX0sgOqGnpE4KJcZuC/aJ1UzODeAXNENfXzMgOaGbp+OoKZtqmWoaO59Wxm6plr+Z32abfZcN+58gYuDa/EEt3tSdLib2T5HuqayAwJ4VBtf3m69lgy/UVarQtn4bxsvWaDP32Zfdcz3W23pwmsaUpoQUx3wsgWJkSWeaMmBok7N4Mybyja8qbhGExoaNl88WC+iCBo55Vyv4yhYkD2dUeQVeegfW7FsBL6PilhLAdNV084LFnxI/z8HWO5N9B5kbsefO9ocN34KoVezh898J+/N78REWZsnMJ1y47tJNNU0FvpsZEBNY+PIBmk5gJ3iXoXX/C1zb+FJdv3pn1v0O0+fDlwd7sg739L+7defD8r/b7RW+Mfj96e/TOaDb6w+jB6LPR8ej5KBm9GP1t9PfRP3b+ufPvnf/s/NdCX3/NfSfpd6PgZ+d//weTwhqg</latexit>

<latexit sha1_base64="PfCOhlItpDKnd01kvxYuNjd/IX8=">AAAlRXicnVpfbxy3Eb+k/9Lrv6R97AsR64DEOCk6xUndFALsOHJSwI4VSU6CaBWBu8u7Y7X/TPKkUxYE+i362n6afoZ+iL4VfW1nSO4uubfnGFFiaTnzG3I4nBnO7F1cZVyq/f1/vfb6j378k5/+7I2fj3/xy1/9+jdvvvXbL2W5Egl7npRZKb6OqWQZL9hzxVXGvq4Eo3mcsa/iq0fI/+qaCcnL4kzdVuwip4uCz3lCFZAuIrpghYIJuPx2dvnmnf29ffNDNh9m7uHOyP0cX76183aUlskqhzmSjEp5PptV6qKmQvEkY3o8Hk/an/GEHNPkChaUe+NoJVllR+crNb9/UfOiWilWJJpMCM2y8oYAffc+MfQAfza7qOdlYcHwMyHAJPd3Y67I2YwgS/oCNc1lTtVS94kGuUGVt3m8QVTLPKQtwZZCsLnRwChhKHAMV+HiK5E1GAeUPK8yRp6fPCF4IpIpxYtFIBSX5ZWisexmr0Q5ZxKPkWa7L1Y04+qWACRjG3u122ol4wxYcUlFStAKBPdXZqFUwRM2FzTppJIyB6Zq0GBVQWbvHUwJU0lwfHXOE1HiRnQj21DKhaDV8jY4vLhUqszxAEuVc5mEVp2vsqyCh4B6XqzyGFz4oi6oinnvcCoq5BWvQmLBbqq1Yms1NU/29MEZye7uLjlaK0FbX0RSOCFHA06JXMVzvpiSRL5YlYrJKUmpXIpVxqZE8avvpiSOc/gF/8w+ebKeknyVKS7KmymBoIshxoCLqws5hwlotigFB18yj5VkK4ieMoUJ8xTMn7MU7Au75YrlEAc3VN6C/e0MNF7bhxjOAzRYO++dkMf4EJ4JaqZ4RUOrNNJA/YRBvAp2ak4XJ3iYVbAEU3VkUbp++PQ0fhVkpmucNLUGfpgpJuCc+DUjqkQTGdeBeGaCJJCoPH/8qN3PODLRA6GwquoxepHJbCaaDpVYsakhxpDZrvpECDADPoyzhoSQPi0Bo3a0cehhCa24gpj6jk0LOAZe4AwXNeQwOEYIcthalLI55FgzRZ0Klur65NOPdT3b35+Se/Dv/X3dA+W4koPtT//4/vRgdq+PuVmCXg5zcO/elDS/tLFmDFu+5iZX4xGDN0Ng5rRIwfqJPodMaE4hoVl9Z6Z1DyE7hEzEECLuEHBUQwjRIcQQgFVS19E1uEgleVYWPTYvUp7oboUN+fgIuBIYewctrD7SfRjMDFGm3QNfLBUVEGcDqKxBZWw+CDLTbJ/CiG8R/RxVjOvP+9p9Y+nf9OlfWPoXffqJpZ9sGLNQoqxuLfczbZ0lus55gSdRX+tL8xfGumPSdcik65aJrgbWVeDmgmWGDSSt60Nt/Os0h6vWRKGvBxzH+QGcuQS3YnG5xlNHjeAJNJQrwVCijlIuq4zeSnUL19mdA43LTo7WFK+3jwge6r491KT+Wtv0QMUClJ+av3S916aXpzDhM8gAVJXibh1ZmIZCYhFN8ellQNivA8LTNuC6Qa6jaQdu7oSBH8Mgnx/BLf0JBiw3MUgcY1ACOI/KAjIG1FQJ27NHUMXG7Md4OpA1aXM0VRyDqbkKuRFeWvWuNlQL0LoTydPqJXjg2jNIKicAMdnhmwDVHbOvgoO8XAsABXoMCTWqoE3OmDT3Ewoz8I46UnSlu7E0HnLm0EfFNVygBZaUrVBHqpluabXP0BtQWZt5j1rFgScDGWlzbGJOdkqexZKJa+pGp4oqrK1QklqIme9hM13pwQ3nWcORRtTQTlsaRlrDMJHxgW4QVo2F3fHEroijOiqvzaXoJD40CbKSqtSYo6DIT1t3cgKOOJlfdoCJh4DFnzgILv5YX7YiFhWXqyJlqZtO8SxltRXVWJHwlMWCpwtGY9BtU0SGMrJVL8u2wLKsRfpm6FSewzles6TVwt/w3BjHTTC46Wb6tlrxNHOrkY+x/fGXjB3IJtSPYU3i27Bhy47vC+NsDhH3BD2WjION8PWwSMdAgQk5YTRZskbbolTCEmp8trca1Kt80dxeiGohw+yEFgYxwHfOmN1Alt+G8S8OHzmBssuWDXjzlCWUVwu4xpZS7kQgDyl6BwfRFdSLu3BRHLB1tMQ7pt5pNN7RNp+RT1mBmbzd96IZg2tgVW4i6AmjooBOimCN3iAzJCIBYojDDaB7ZJjh1KODv/is1mcsxoGwuW5nbR/1xN2/sR4ASg8pB6CwcIgONBkQuB5WYoMZLtwudv2SxXz/gwu+qDtfbBRBcrdNaGHARkuqwpTss+UmPwh5Upq7GpNpU7pgCwpFBhzvNfoAbiOF3Dftu+CURCddOQejJ1C9BQQYQUkisRmLltCbQh9dQGfnNIOHpDRj0JC9gL/GpzrVTNreG8rtZAhiLDmvV76r2UQFQs+9bL9rpEhVZjy5bfKPHdVRxYdndwAz3/EAxi8nG6zrA9zw0lbxY5NNbqAPkwTQ5EsK3QpeQKY/wTiCnoFXioigiYSaEAJZ0LDSCpvCFlOfQeXYjXJdF01SEWZpTO5B8d+Aa9FVHA1U6Ankn2XZ0rH0FeCQpvKwdW8fAZVzhzBlcw9h526O56QNL7TGHIpqH4YaYGUu7CF+xsEdBW+qhGUzNDO11bvw6DEVdYdrEXI1n3somqwUG8IxbFHxnQ+yam/mQQCF5tGWEPhycGC+lkCN83iIb6OEiwQ36a4IV9bOa0MkxxmIcGipb11N2I7nsPnG8zsqdApKGLM8xYrR3eSDKHwF1Ee+Ywu5b6HjoHkJvhrlPCXLy+b03jVKPaLgo61KiRs1KuObn2dqyQR5ymWyB5W6CRL7/gf4UOQT9sLVXzX+Yi90vwLDms24kumibBvlRdzdTq7rubzSzbTQnihacqUIJXNaJLcELj9ojLBMImoJAdjv+FMAmEgeQ+2/AM9u3hrpybkRS0oB9+LhfqUuxtG1BAOwepbneowRP4bcm3YiY7P82ZKV0M65lZQd1e4vhK978Ll3G/bdYX49h3JZ14/xd0AH7UooxwS0t4/axwDBbO+oa9dEhlxsO8WVPnfji/rEEgJQxsBYHuaJGQeQClrsUpp+zgMee9QAnrbdn667TrBnkg50dysKQBLqZ7jlKGLa556Rir+wREGPjVZqn72zIu51KB6Zebrh6C01g5rKailZ0izuA9pT3cL3N9pCJiHGncG2KXztt03hm38ryPOVbRB70h3bXGh/LiAAiFdkmLePcLblXBo75ykwz6E0VWWFafFwTjNpXhHaN9MhzVQRSCKHpCXiyyGwAgTg4f1KtTBH2rckfCdteqTD2YdAgWsaFiQWA2yOegIlFDJUq0fHGONnI0CvcBM2Z+BTk8DMrvwOObVAyBPnBnhRu2zh6CaBxXObyKD5t7PB/3dme1pfEJvg/PShiZvB00PXJp14FDN2j77w92ssf5DGp1dMJcutih/8EMVBWb+dsf4G3oaXcpqqMoEsq5jwYm2mI0UXoNqStURt3xZHLuDoCi5VNscXy17CdNeqCxcf44e9u9Yx6HxIm/mQjfnWZ7rMa+7WLtZ8RJjrrCYu3kJFuixtXqC4DfqYo2bTTWfhPuHwMQ8bovdWsthQKsiaRqclrcDYgUaW1OyOimDjxzB2LPPJWGB4JDQG44ueyR8bSvOqZhXPs5KqlwJePoXJTT63uYaMuM1YPvu0TafGiFKu8qqPedhSG/tklAd2foQEkwgh5HJ6xcyHHookUPVf4f6DQp6pssywohhHiKUqYwpsOwazQlwlOb7ufnj2ALodhTEDyY8YQpEHQ/PBzYO6Yeka/mtmLLHesj1eVUGU8TVxoSWDS88EY7KC4jNHLwnLCj9TdBhIFpAwoQZmXrzCfOFUWPdEvdmNGCrTI2M3hFG1TTNzdfolx6ZiCHkFxXDinmJmt5uKGTK+VXaBuE03CN+NompTPUS9gnoA66uHkgPqGbLJ7k062aYgJiK9WWNt6miAr6Ak4vpaGtkBNS19QjAxblOwX7RuamYQr6AZ4vqaGdkBzSwdX13BTNtUy9DxvDp2U7Xs1fwu2/S7bNjvHBkD1+YXYumu9mQpsXeSfE91DQTmpDCott98PRtsub5Cjbbl0zBetl6Tod++7J7ruc7Wm9MktjQltCDmewEEK1Miy5wRU4OE3ZshmXd0TXmTMCwmdLRsvlhQHyRw1LNK2V+mMHEgu9oj6MozsH7XAngJHb+UELajposHPPaM+HEevs6R/DvI3Ig9b743dPgOXLViD4fvXtiP35ufqChTdi7h2mWHdrJpKujN1JiIwNqHB9BsEjPBuwS960/42saf4vLNO7P+d4g2H7482Jt9sLf/xb07D57/1X6/6I3R70dvj94ZzUZ/GD0YfTY6Hj0fJaMXo7+N/j76x84/d/6985+d/1ro66+57yT9bhT87Pzv/0oYGp4=</latexit>

<latexit sha1_base64="ZHrf5whPsJp+0ZcdYQF8zcO7IrY=">AAAlRXicnVpfbxy3Eb+k/9Lrv6R97AsR64DEOCk6xUndFALsOHJSwI4VSU6CaBWBu8u7Y7X/TPKkUxYE+i362n6afoZ+iL4VfW1nSO4uubfnGFFiaTnzG3I4nBnO7F1cZVyq/f1/vfb6j378k5/+7I2fj3/xy1/9+jdvvvXbL2W5Egl7npRZKb6OqWQZL9hzxVXGvq4Eo3mcsa/iq0fI/+qaCcnL4kzdVuwip4uCz3lCFZAuIrpghYIJuPz24PLNO/t7++aHbD7M3MOdkfs5vnxr5+0oLZNVDnMkGZXyfDar1EVNheJJxvR4PJ60P+MJOabJFSwo98bRSrLKjs5Xan7/ouZFtVKsSDSZEJpl5Q0B+u59YugB/mx2Uc/LwoLhZ0KASe7vxlyRsxlBlvQFaprLnKql7hMNcoMqb/N4g6iWeUhbgi2FYHOjgVHCUOAYrsLFVyJrMA4oeV5ljDw/eULwRCRTiheLQCguyytFY9nNXolyziQeI812X6xoxtUtAUjGNvZqt9VKxhmw4pKKlKAVCO6vzEKpgidsLmjSSSVlDkzVoMGqgszeO5gSppLg+OqcJ6LEjehGtqGUC0Gr5W1weHGpVJnjAZYq5zIJrTpfZVkFDwH1vFjlMbjwRV1QFfPe4VRUyCtehcSC3VRrxdZqap7s6YMzkt3dXXK0VoK2voikcEKOBpwSuYrnfDEliXyxKhWTU5JSuRSrjE2J4lffTUkc5/AL/pl98mQ9JfkqU1yUN1MCQRdDjAEXVxdyDhPQbFEKDr5kHivJVhA9ZQoT5imYP2cp2Bd2yxXLIQ5uqLwF+9sZaLy2DzGcB2iwdt47IY/xITwT1EzxioZWaaSB+gmDeBXs1JwuTvAwq2AJpurIonT98Olp/CrITNc4aWoN/DBTTMA58WtGVIkmMq4D8cwESSBRef74UbufcWSiB0JhVdVj9CKT2Uw0HSqxYlNDjCGzXfWJEGAGfBhnDQkhfVoCRu1o49DDElpxBTH1HZsWcAy8wBkuashhcIwQ5LC1KGVzyLFmijoVLNX1yacf63q2vz8l9+Df+/u6B8pxJQfbn/7x/enB7F4fc7MEvRzm4N69KWl+aWPNGLZ8zU2uxiMGb4bAzGmRgvUTfQ6Z0JxCQrP6zkzrHkJ2CJmIIUTcIeCohhCiQ4ghAKukrqNrcJFK8qwsemxepDzR3Qob8vERcCUw9g5aWH2k+zCYGaJMuwe+WCoqIM4GUFmDyth8EGSm2T6FEd8i+jmqGNef97X7xtK/6dO/sPQv+vQTSz/ZMGahRFndWu5n2jpLdJ3zAk+ivtaX5i+Mdcek65BJ1y0TXQ2sq8DNBcsMG0ha14fa+NdpDletiUJfDziO8wM4cwluxeJyjaeOGsETaChXgqFEHaVcVhm9leoWrrM7BxqXnRytKV5vHxE81H17qEn9tbbpgYoFKD81f+l6r00vT2HCZ5ABqCrF3TqyMA2FxCKa4tPLgLBfB4SnbcB1g1xH0w7c3AkDP4ZBPj+CW/oTDFhuYpA4xqAEcB6VBWQMqKkStmePoIqN2Y/xdCBr0uZoqjgGU3MVciO8tOpdbagWoHUnkqfVS/DAtWeQVE4AYrLDNwGqO2ZfBQd5uRYACvQYEmpUQZucMWnuJxRm4B11pOhKd2NpPOTMoY+Ka7hACywpW6GOVDPd0mqfoTegsjbzHrWKA08GMtLm2MSc7JQ8iyUT19SNThVVWFuhJLUQM9/DZrrSgxvOs4YjjaihnbY0jLSGYSLjA90grBoLu+OJXRFHdVRem0vRSXxoEmQlVakxR0GRn7bu5AQccTK/7AATDwGLP3EQXPyxvmxFLCouV0XKUjed4lnKaiuqsSLhKYsFTxeMxqDbpogMZWSrXpZtgWVZi/TN0Kk8h3O8Zkmrhb/huTGOm2Bw0830bbXiaeZWIx9j++MvGTuQTagfw5rEt2HDlh3fF8bZHCLuCXosGQcb4ethkY6BAhNywmiyZI22RamEJdT4bG81qFf5orm9ENVChtkJLQxigO+cMbuBLL8N418cPnICZZctG/DmKUsorxZwjS2l3IlAHlL0Dg6iK6gXd+GiOGDraIl3TL3TaLyjbT4jn7ICM3m770UzBtfAqtxE0BNGRQGdFMEavUFmSEQCxBCHG0D3yDDDqUcHf/FZrc9YjANhc93O2j7qibt/Yz0AlB5SDkBh4RAdaDIgcD2sxAYzXLhd7Poli/n+Bxd8UXe+2CiC5G6b0MKAjZZUhSnZZ8tNfhDypDR3NSbTpnTBFhSKDDjea/QB3EYKuW/ad8EpiU66cg5GT6B6CwgwgpJEYjMWLaE3hT66gM7OaQYPSWnGoCF7AX+NT3WqmbS9N5TbyRDEWHJer3xXs4kKhJ572X7XSJGqzHhy2+QfO6qjig/P7gBmvuMBjF9ONljXB7jhpa3ixyab3EAfJgmgyZcUuhW8gEx/gnEEPQOvFBFBEwk1IQSyoGGlFTaFLaY+g8qxG+W6LpqkIszSmNyD4r8B16KrOBqo0BPIP8uypWPpK8AhTeVh694+AirnDmHK5h7Czt0cz0kbXmiNORTVPgw1wMpc2EP8jIM7Ct5UCctmaGZqq3fh0WMq6g7XIuRqPvdQNFkpNoRj2KLiOx9k1d7MgwAKzaMtIfDl4MB8LYEa5/EQ30YJFwlu0l0Rrqyd14ZIjjMQ4dBS37qasB3PYfON53dU6BSUMGZ5ihWju8kHUfgKqI98xxZy30LHQfMSfDXKeUqWl83pvWuUekTBR1uVEjdqVMY3P8/UkgnylMtkDyp1EyT2/Q/wocgn7IWrv2r8xV7ofgWGNZtxJdNF2TbKi7i7nVzXc3mlm2mhPVG05EoRSua0SG4JXH7QGGGZRNQSArDf8acAMJE8htp/AZ7dvDXSk3MjlpQC7sXD/UpdjKNrCQZg9SzP9Rgjfgy5N+1Exmb5syUroZ1zKyk7qt1fCF/34HPvNuy7w/x6DuWyrh/j74AO2pVQjglobx+1jwGC2d5R166JDLnYdoorfe7GF/WJJQSgjIGxPMwTMw4gFbTYpTT9nAc89qgBPG27P113nWDPJB3o7lYUgCTUz3DLUcS0zz0jFX9hiYIeG63UPntnRdzrUDwy83TD0VtqBjWV1VKypFncB7SnuoXvb7SFTEKMO4NtU/jab5vCN/9WkOcr2yD2pDu2udD+XEAAEK/IMG8f4WzLuTR2zlNgnkNpqsoK0+LhnGbSvCK0b6ZDmqkikEQOSUvEl0NgBQjAw/uVamGOtG9J+E7a9EiHsw+BAtc0LEgsBtgc9QRKKGSoVo+OMcbPRoBe4SZszsCnJoGZXfkdcmqBkCfODfCidtnC0U0Ci+c2kUHzb2eD/+/M9rS+IDbB+elDEzeDp4euTTrxKGbsHn3h79dY/iCNT6+YSpZbFT/4IYqDsn47Y/0NvA0v5TRVZQJZVjHhxdpMR4ouQLUla4navi2OXMDRFVyqbI4vlr2E6a5VFy4+xg97d61j0PmQNvMhG/Otz3SZ19ytXaz5iDDXWU1cvIWKdFnavEBxG/QxR82mm87CfcLhYx42RO+tZLGhVJA1jU5LWoGxA40sqdkdFcHGj2HsWOaTscDwSGgMxhc9kz82lOZVzSqeZyVVLwW8fAqTm3xucw0ZcZuxfPZpm06NEaVc5VUf87ClNvbJKA/s/AgJJhFCyOX0ipkPPRRJoOq/wv0HhTxTZZlhRTGOEEtVxhTYdgxmhbhKcnzd/fDsAXQ7CmMGkh8xhCIPhuaDmwd1w9I1/NfMWGK9ZXu8qoIo42viQksGl54JxmQFxWeOXhKWFX6m6DCQLCBhQg3MvHiF+cKpsO6JerMbMVSmR8ZuCKNqm2bm6vRLjk3FEPIKiuHEPcXMbjcVM2R8q+wCcZtuEL4bRdWmeoh6BfUA1lcPJQfUM2ST3Zt0sk1BTER6s8ba1NEAX0FJxPW1NLIDalr6hGBi3KZgv2jd1MwgXkEzxPU1M7IDmlk6vrqCmbaplqHjeXXspmrZq/ldtul32bDfOTIGrs0vxNJd7clSYu8k+Z7qGgjMSWFQbb/5ejbYcn2FGm3Lp2G8bL0mQ7992T3Xc52tN6dJbGlKaEHM9wIIVqZEljkjpgYJuzdDMu/omvImYVhM6GjZfLGgPkjgqGeVsr9MYeJAdrVH0JVnYP2uBfASOn4pIWxHTRcPeOwZ8eM8fJ0j+XeQuRF73nxv6PAduGrFHg7fvbAfvzc/UVGm7FzCtcsO7WTTVNCbqTERgbUPD6DZJGaCdwl615/wtY0/xeWbd2b97xBtPnx5sDf7YG//i3t3Hjz/q/1+0Ruj34/eHr0zmo3+MHow+mx0PHo+SkYvRn8b/X30j51/7vx75z87/7XQ119z30n63Sj42fnf/wFu7Rqf</latexit> <latexit sha1_base64="PfCOhlItpDKnd01kvxYuNjd/IX8=">AAAlRXicnVpfbxy3Eb+k/9Lrv6R97AsR64DEOCk6xUndFALsOHJSwI4VSU6CaBWBu8u7Y7X/TPKkUxYE+i362n6afoZ+iL4VfW1nSO4uubfnGFFiaTnzG3I4nBnO7F1cZVyq/f1/vfb6j378k5/+7I2fj3/xy1/9+jdvvvXbL2W5Egl7npRZKb6OqWQZL9hzxVXGvq4Eo3mcsa/iq0fI/+qaCcnL4kzdVuwip4uCz3lCFZAuIrpghYIJuPx2dvnmnf29ffNDNh9m7uHOyP0cX76183aUlskqhzmSjEp5PptV6qKmQvEkY3o8Hk/an/GEHNPkChaUe+NoJVllR+crNb9/UfOiWilWJJpMCM2y8oYAffc+MfQAfza7qOdlYcHwMyHAJPd3Y67I2YwgS/oCNc1lTtVS94kGuUGVt3m8QVTLPKQtwZZCsLnRwChhKHAMV+HiK5E1GAeUPK8yRp6fPCF4IpIpxYtFIBSX5ZWisexmr0Q5ZxKPkWa7L1Y04+qWACRjG3u122ol4wxYcUlFStAKBPdXZqFUwRM2FzTppJIyB6Zq0GBVQWbvHUwJU0lwfHXOE1HiRnQj21

⌫(⌘) ⌫B ⌫B
<latexit sha1_base64="A64zJS85yBmXavd70Jvqwwv8Gr8=">AAAlQXicnVpfcxu3EWfSfyn7L2kf+4KJxBnbQymi4qRuOpqx4shJZ+xYkeQkjU7R4O5AEtX9MwBKVG4w0+/Q1/bT9FP0I/St09e+dBfA3QFHUvVEiaXD7m+BxWJ3sXtkXGVcqr29f77x5g9++KMf/+Stnw5/9vNf/PJXb7/z6y9luRAJe5mUWSm+jqlkGS/YS8VVxr6uBKN5nLGv4qsnyP/qmgnJy+JM3VbsIqezgk95QhWQ/hTRGSvUvfn9y7e39nb3zA9ZfZi4h62B+zm+fGf73Sgtk0UO8klGpTyfTCp1UVOheJIxPRwOR+3PcESOaXIFi8ndYbSQrLKj84WaPrqoeVEtFCsSTUaEZll5Q4C+84gYeoA/m1zU07KwYPgZEWCSRzsxV+RsQpAlfYGa5jKnaq77RINcocrbPF4hqnke0uZgRyHY1GhglDAUOIKrcPGFyBqMA0qeVxkjL0+eETwNyZTixSwQisvyStFYdrNXopwyiUdIs51XC5pxdUsAkrGVvdpttZJxBqy4pCIlaAWC+yuzUKrgCZsKmnRSSZkDUzVosKogk/f2x4SpJDi+OueJKHEjupFtKOVM0Gp+GxxeXCpV5niApcq5TEKrThdZVsFDQD0vFnkM7ntRF1TFvHc4FRXyilchsWA31VKxpRqbJ3v64IxkZ2eHHC2VoK0vIimckKMBx0Qu4imfjUkiXy1KxeSYpFTOxSJjY6L41XdjEsc5/IJ/Zp88WY5JvsgUF+XNmEDAxRBfwMXVhZzCBDSblYKDL5nHSrIFRE+ZwoR5CubPWQr2hd1yxXKIgxsqb8H+dgYaL+1DDOcBGiyd947IU3wIzwQ1U7yioVUaaaB+wiBeBTs1p4sTHGYVLMFUHVmUrg+fn8avg8x0jZOm1sCHmWICzolfM6JKNJFxHYhnJkgCScrzx4/a/QwjEz0QCouqHqIXmaxmoulAiQUbG2IMWe2qT4QAM+CDOGtICOnTEjBqRxuGHpbQiiuIqe/YuIBj4AXOcFFDDoNjhCCHrUUpm0J+NVPUqWCprk8+/VjXk729MXkI/97f0z1Qjis52N749++P9ycP+5ibOejlMPsPH45J80sba8aw5Wtu8jQeMXgzBGZOixSsn+hzyITmFBKa1VsTrXsI2SFkItYh4g4BR7UOITqEWAdgldR1dA0uUkmelUWPzYuUJ7pbYUU+PgKuBMbufgurj3QfBjNDlGn3wGdzRQXE2RpU1qAyNl0LMtNsnsKIbxD9HFWM68/72n1j6d/06V9Y+hd9+omln6wYs1CirG4t9zNtnSW6znmBJ1Ff60vzF8a6Y9JlyKTLlomuBtZV4OaCZYYNJK3rA2386zSHq9ZEoa8HHMf5Ppy5BLdicbnEU0eN4Ak0lAvBUKKOUi6rjN5KdQvX2da+xmVHR0uK19tHBA91zx5qUn+tbXqgYgbKj81futxt08tzmPAFZACqSvGgjixMQyExi8b4dBcQ9uuA8LQJuGyQy2jcgZs7Yc2PYZDPj+CW/gQDlpsYJI6xVgI4T8oCMgbUUwnbtUdQxcbsx3g6kDVpczRVHIOpuQq5EV5a9Y42VAvQuhPJ0+oOPHDtGSSVE4CY7PBNgOqO2VfBQe7WAkCBHuuEGlXQJmdMmvsJhRl4Rx0putDdWBoPOXPoo+IaLtACS8pWqCPVTLe02mfoFaiszbxHreLAk4GMtDk2MSc7Ji9iycQ1daNTRRXWVihJLcTMd9hMV3pww3nRcKQRNbTTloaR1jBMZHygG4RVY2Z3PLIr4qiOymtzKTqJD02CrKQqNeYoKPDT1p2cgCOOppcdYOQhYPFnDoKLP9WXrYhFxeWiSFnqplM8S1ltRTVWJDxlseDpjNEYdFsVkaGMbNXLsg2wLGuRvhk6ladwjtcsabXwNzw1xnETrN10M31brXiaudXIx1TyYMnYgWxC/RjWJL4NG7bs+L4wzuYQcU/QY8k42AhfrhfpGCgwIieMJnPWaFuUSlhCjc/2VoN6lc+a2wtRLWQ9O6GFQazhO2fMbiDLb8L4F4ePHEHZZcsGvHnKEsqrGVxjcym3I5CHFL2Ng+gK6sUduCj22TKa4x1Tbzcab2ubz8inrMBM3u571ozBNbAqNxH0jFFRQCdFsEZvkBkSkQAxxOEG0D0yzHDq0cFffFbrMxbjQNhYt7O2j3rk7t9YrwFKDynXQGHhEB1oskbger0SK8xw4Xax6zsW8/0PLvii7nyxUQTJ3TahhQEbzakKU7LPlqv8IORJae5qTKZN6YItKBQZcLzX6AO4jRRy37jvgmMSnXTlHIyeQfUWEGAEJYnEZiyaQ28KfXQBnZ3TDB6S0oxBQ/YK/hqf6lQzaXt3XW4n6yDGktN64buaTVQg9NLL9jtGilRlxpPbJv/YUR1VfP3sDmDmO16D8cvJBuv6ADe8tFX80GSTG+jDJAE0+ZJCt4IXkOlPMI6gZ+CVIiJoIqEmhEAWNKy0wqawxdRnUDl2o1zXRZNUhFkak3tQ/DfgWnQVRwMVegT5Z162dCx9BTikqTxs3dtHQOXcIUzZ3EPYuZvjOWnDC60xhaLah6EGWJkLe4ifcXBHwZsqYd4MzUxt9S48ekxF3eFahFxMpx6KJgvF1uEYtqj4zgdZtTfzWgCF5tGWEPhicM18LYEa5/EQ30YJFwlu0l0Rrqyd1oZIjjMQ4dBS37qasB1PYfON53dU6BSUMGZ5jhWju8nXovAVUB95zxZy30LHQfMSfDXKeUrml83p3TdKPaHgo61KiRs1KuObnxdqzgR5zmWyC5W6CRL7/gf4UOQT9srVXzX+Yq90vwLDms24kumibBvlRdyDTq7rubzSzbTQnihacqEIJVNaJLcELj9ojLBMImoOAdjv+FMAmEgeQu0/A89u3hrp0bkRS0oB9+LBXqUuhtG1BAOwepLneogRP4Tcm3YiQ7P82ZyV0M65lZQd1e4vhK978LkPGvaD9fx6CuWyrp/i74AO2pVQjglob5+0jwGC2d5R166JDLnYdoorfe7GF/WJJQSgjIGxPMwzMw4gFbTYpTT9nAc89qgBPG27P113nWDPJB3owUYUgCTUz3DLUcS0zz0jFX9miYIeG63UPntnRdzrUDwy83TD0VtqBjWV1VKypFncB7SnuoHvb7SFjEKMO4NNU/jab5rCN/9GkOcrmyD2pDu2udD+WEAAEK/IMG8f4WzLqTR2zlNgnkNpqsoK0+LBlGbSvCK0b6ZDmqkikEQOSEvEl0NgBQjAg0eVamGOtGdJ+E7a9EgHkw+BAtc0LEgsBtgc9QRKKGSoVo+OMcTPRoBe4SZszsCnJoGZXfkdcmqBkCfODfCidtnC0U0Ci6c2kUHzb2eD/7cmu1pfEJvg/PShiZvB00PXJp14FDN2j77w/9dYfi+NT6+YSuYbFd//PoqDsn47Y/0NvA0v5TRVZQJZVjHhxdpER4rOQLU5a4navi2OXMDRBVyqbIovlr2E6a5VFy4+xg97d61j0PmQNvMhG/Otz3SZ19ytXaz5iDDXWU1cvIWKdFnavEBxG/QxR82mm87CfcLhYw4bovdWslhRKsiaRqc5rcDYgUaW1OyOimDjxzB2LPPJWGB4JDQG47OeyZ8aSvOqZhFPs5KqOwF3T2Fyk89triEjbjOWzz5t06kxopSLvOpjDltqY5+M8sDOT5BgEiGEXE6vmPnQQ5EEqv4r3H9QyDNVlhlWFMMIsVRlTIFth2BWiKskx9fdh2ePodtRGDOQ/IghFHkwNB/cPK4blq7hv2bGEust2+NVFUQZXxIXWjK49EwwJgsoPnP0krCs8DNFh4FkAQkTamDmxSvMF06FdU/Um92IoTI9MnZDGFWbNDNXp19yrCqGkNdQDCfuKWZ2u6qYIeNbZReIm3SD8F0pqlbVQ9RrqAewvnoouUY9QzbZvUknmxTERKRXa6xVHQ3wNZREXF9LI7tGTUsfEUyMmxTsF62rmhnEa2iGuL5mRnaNZpaOr65gpk2qZeh4Xh27qlr2en6Xrfpdtt7vHBkD1+YXYumu9mQpsXeSfE91DQTmpDCoNt98PRtsuL5CjTbl0zBeNl6Tod/edc/1XGfjzWkSW5oSWhDzvQCClSmRZc6IqUHC7s2QzDu6prxJGBYTOpo3Xyyo9xM46kml7C9TmDiQXe0JdOUZWL9rAbyEjl9KCNtR08UDHntG/DgPX+dI/h1kbsSeN98ZOrgHV63YxeH9C/vxe/MTFWXKziVcu+zATjZOBb0ZGxMRWPtgH5pNYia4T9C7/oCvbfwpLt/emvS/Q7T68OX+7uSD3b0vHm49fvkX+/2itwa/Hbw7uDeYDH43eDz4bHA8eDlIBvngr4O/Df6+/Y/tf23/e/s/FvrmG+47Sb8ZBD/b//0f3UgYjg==</latexit>

⌫3B . B
<latexit sha1_base64="wc9+SOjDL7IlDDoLRC8mBmD+/lo=">AAAlPnicnVpfcxu3EWfSfyn7L2kf+4KJxZnEQymi4qRuOpqx48hJZ+xYkWTHE52iwd2BJKL7ZwCkqNxgph+hr+2n6dfoF+hbp6997C6AuwOOpOqJEkuH3d/iFovdxS7IuMq4VPv7/3zjzR/9+Cc//dlbPx/+4pe/+vVv3n7nty9kuRAJe56UWSlexlSyjBfsueIqYy8rwWgeZ+zr+OoR8r9eMiF5WZypm4pd5HRW8ClPqALSi2iZlkpevn1nf2/f/JD1h4l7uDNwP8eX7+y8G6VlsshZoZKMSnk+mVTqoqZC8SRjejgcjtqf4Ygc0+SKzpjcG0YLySo7Ol+o6f2LmhfVQrEi0WREaJaV1wTou/eJoQf4s8lFPS0LC4afEQEmub8bc0XOJgRZ0heoaS5zqua6TzTINaq8yeM1oprnIW0ORhSCTY0GRglDAftfhS9fiKzBOKDkeZUx8vzkCcGtkEwpXswCobgsrxSNZTd7Jcopk7h/NNt9taAZVzcEIBlbW6tdVisZZ8CKSypSglYguL4yC6UKnrCpoEknlZQ5MFWDBqsKMvngYEyYSoLtq3OeiBIXohvZhlLOBK3mN8HmxaVSZY4bWKqcyyS06nSRZRU8BNTzYpHH4LsXdUFVzHubU1Ehr3gVEgt2Xa0UW6mxebK7D85Idnd3ydFKCdr6IpLCCTkacEzkIp7y2Zgk8tWiVEyOSUrlXCwyNiaKX30/JnGcwy/4Z9bJk9WY5ItMcVFejwlEWwzBBVx8u5BTmIBms1Jw8CXzWEm2gOgpU5gwT8H8OUvBvrBarlgOcXBN5Q3Y385A45V9iGE/QIOV894ReYwP4Z6gZopXNLRKIw3UzxjEq2CnZndxgodZBa9gqo4sStcPn57Gr4PMdI2TptbADzPFBOwTXzKiSjSRcR2IZyZIAhnK88dP2vUMIxM9EAqLqh6iF5mUZqLpUIkFGxtiDCntqk+EADPgwzhrSAjp0xIwakcbhh6W0IoriKnv2biAbeAFznBRQw6DbYQgh6VFKZtCcjVT1Klgqa5PPv9U15P9/TG5B/8+3Nc9UI5vcrD98R8/HB9M7vUx13PQy2EO7t0bk+aXNtaMYclLbpI0bjF4MwRmTosUrJ/oc8iEZhcSmtV3Jlr3ELJDyERsQsQdArZqE0J0CLEJwCqp62gJLlJJnpVFj82LlCe6e8OafHwEXAmMvYMWVh/pPgxmhijT7oHP5ooKiLMNqKxBZWy6EWSm2T6FEd8i+iWqGNdf9rX7xtK/6dO/svSv+vQTSz9ZM2ahRFndWO4X2jpLtMx5gTtRL/Wl+Qtj3THpKmTSVctEVwPrKnBzwTLDBpLW9aE2/nWaw1FrotDXA7bj/AD2XIJbsbhc4a6jRvAEGsqFYChRRymXVUZvpLqB4+zOgcbXjo5WFI+3Twhu6r7d1KR+qW16oGIGyo/NX7raa9PLU5jwGWQAqkpxt44sTEMhMYvG+HQbENbrgPC0DbhqkKto3IGbM2HDj2GQL4/glP4MA5abGCSOsVECOI/KAjLGDEoTtme3oIqN2Y9xdyBr0mZrqjgGU3MVciM8tOpdbagWoHUnkqfVLXjg2j1IKicAMdnhmwDVHbOvgoPcrgWAAj02CTWqoE3OmDTnEwoz8I46UnShu7E0HnLm0EfFEg7QAkvKVqgj1Uy3tNpn6DWorM28R63iwJOBjLQ5NjE7OybPYsnEkrrRqaIKayuUpBZi5nvYTFd6cMN51nCkETW005aGkdYwTGR8pBuEVWNmVzyyb8RRHZVLcyg6iY9NgqykKjXmKKju09adnIAjjqaXHWDkIeDlTxwEX/5YX7YiFhWXiyJlqZtO8SxltRXVWJHwlMWCpzNGY9BtXUSGMrJVL8u2wLKsRfpm6FSewj4uWdJq4S94aozjJti46Gb6tlrxNHNvI59SyYNXxg5kE+qn8E7i27Bhy47vC+NsDhH3BD2WjIOF8NVmkY6BAiNywmgyZ422RamEJdT4bE81qFf5rDm9ENVCNrMTWhjEBr5zxuwasvw2jH9w+MgRlF22bMCTpyyhvJrBMTaXcicCeUjROziIrqBe3IWD4oCtojmeMfVOo/GOtvmMfM4KzOTtumfNGFwDq3ITQU8YFQV0UgRr9AaZIREJEEMcTgDdI8MMpx4d/MVntT5jMQ6EXXU7a/uoR+78jfUGoPSQcgMUXhyiA002CCw3K7HGDF/cvmx5y8t8/4MDvqg7X2wUQXK3TGhhwEZzqsKU7LPlOj8IeVKasxqTaVO6YAsKRQZs7xJ9AJeRQu4b911wTKKTrpyD0ROo3gICjKAkkdiMRXPoTaGPLqCzc5rBQ1KaMWjIXsFf41OdaiZt723K7WQTxFhyWi98V7OJCoSee9l+10iRqsx4ctPkHzuqo4pvnt0BzHzHGzB+OdlgXR/ghpe2ih+abHINfZgkgCYvKHQreACZ/gTjCHoGXikigiYSakIIZEHDSitsCltMfQaVYzfKdV00SUWYV2NyD4r/BlyLruJooEKPIP/My5aOpa8AhzSVh617+wionDuEKZt7CDt3sz0nbXihNaZQVPsw1AArc2E38QsO7ih4UyXMm6GZqa3ehUePqag7XIuQi+nUQ9FkodgmHMMWFe98kFV7M28EUGgebQmBt4Ib5msJ1DiPh/g2SrhIcJHuiHBl7bQ2RHKcgQiHlvrG1YTteAqLbzy/o0KnoIQxy1OsGN1JvhGFV0B95Hu2kPsWOg6al+CrUc5TMr9sdu99o9QjCj7aqpS4UaMy3vw8U3MmyFMukz2o1E2Q2Psf4EORT9grV3/V+Iu90v0KDGs240qmi7JtlBdxdzu5rufySjfTQnuiaMmFIpRMaZHcEDj8oDHCMomoOQRgv+NPAWAieQi1/ww8u7k10qNzI5aUAs7Fw/1KXQyjpQQDsHqS53qIET+E3Jt2IkPz+rM5K6Gdc29SdlS7vxC+7sHn3m3Ydzfz6ymUy7p+jL8DOmhXQjkmoL191D4GCGZ7R127JjLkYtsprvS5G1/UJ5YQgDIGxvIwT8w4gFTQYpfS9HMe8NijBvC07f503XWCPZN0oLtbUQCSUD/DKUcR0z73jFR8xxIFPTZaqX329oq461DcMvN0zdFbagY1ldVSsqR5uQ9od3UL319oCxmFGLcH26bwtd82hW/+rSDPV7ZB7E53bHOg/bmAACBekWFuH2Fvy6k0ds5TYJ5DaarKCtPi4ZRm0lwR2pvpkGaqCCSRQ9IS8XIIrAABeHi/Ui3MkfYtCe+kTY90OPkYKHBMwwuJxQCbo55ACYUM1erRMYb42QjQK1yEzRn41CQwsyq/Q04tEPLEuQFe1C5bOLpJYPHUJjJo/u1s8P+dyZ7WF8QmOD99aOJm8PTQtUknHsWM3aMv/P81lj9I49MrppL5VsUPfojioKzfzlh/A2/DQzlNVZlAllVMeLE20ZGiM1BtzlqitrfFkQs4uoBDlU3xYtlLmO5YdeHiY/ywd8c6Bp0PaTMfsjHf+kyXec3Z2sWajwhzndXExVuoSJelzQWKW6CPOWoW3XQW7hMOH/OwIXq3ksWaUkHWNDrNaQXGDjSypGZ1VAQLP4axY5lPxgLDI6ExGJ/1TP7YUJqrmkU8zUqqbgXcPoXJTT63OYaMuM1YPvu0TafGiFIu8qqPedhSG/tklAd2foQEkwgh5HJ6xcyHHookUPVf4fqDQp6pssywohhGiKUqYwpsOwSzQlwlOV53Pzx7AN2OwpiB5EcMociDofng5kHdsHQN/zUzllhv2R6vqiDK+Iq40JLBoWeCMVlA8Zmjl4RlhZ8pOgwkC0iYUAMzL15hvnAqrHui3uxGDJXpkbEbwqjappk5Ov2SY10xhLyGYjhxTzGz2nXFDBlvlV0gbtMNwnetqFpXD1GvoR7A+uqh5Ab1DNlk9yadbFMQE5Fer7HWdTTA11AScX0tjewGNS19RDAxblOwX7Sua2YQr6EZ4vqaGdkNmlk6Xl3BTNtUy9DxvDp2XbXs9fwuW/e7bLPfOTIGrs0vxNJd7clSYs8k+YHqGgjMSWFQbT/5ejbYcnyFGm3Lp2G8bD0mQ7+97Zzruc7Wk9MktjQltCDmewEEK1Miy5wRU4OE3ZshmTu6prxJGBYTOpo3XyyoDxLY6kml7C9TmDiQfdsj6MozsH7XAngJHb+UELajposHPPaM+HEeXudI/j1kbsSeN18YOnwPjlqxh8P3L+zH781PVJQpO5dw7LJDO9k4FfR6bExE4N2HB9BsEjPB+wS96094beNPcfn2nUn/O0TrDy8O9iYf7e1/de/Og+d/sd8vemvw+8G7g/cGk8EfBg8GXwyOB88HyeC7wV8Hfxv8fecfO//a+ffOfyz0zTfcd5J+Nwh+dv77Px0vF9g=</latexit>
{
{

<latexit sha1_base64="x1XUCM/GIpCAGiRVVXkaD0GQmA0=">AAAlOnicnVpfbxy3Eb+k/9Lrv6R97AsR64DEOCk6xUndFALsOHJSwI4VSU6CaBWBu8u7Y7X/TPKkUxYE+gH62n6afpG+9q3oaz9AZ0juLrm35xhRYmk58xvucDgznOFdXGVcqv39f732+o9+/JOf/uyNn49/8ctf/fo3b7712y9luRIJe56UWSm+jqlkGS/Yc8VVxr6uBKN5nLGv4qtHyP/qmgnJy+JM3VbsIqeLgs95QhWQTqL68s07+3v75odsPszcw52R+zm+fGvn7Sgtk1XOCpVkVMrz2axSFzUViicZ0+PxeNL+jCfkmCZXdMHk3jhaSVbZ0flKze9f1LyoVooViSYTQrOsvCFA371PDD3An80u6nlZWDD8TAgwyf3dmCtyNiPIkr5ATXOZU7XUfaJBblDlbR5vENUyD2lLMKAQbG40MEoYCtj+Knz5SmQNxgElz6uMkecnTwhug2RK8WIRCMVleaVoLLvZK1HOmcS9o9nuixXNuLolAMnYxlrtslrJOANWXFKRErQCwfWVWShV8ITNBU06qaTMgakaNFhVkNl7B1PCVBJsX53zRJS4EN3INpRyIWi1vA02Ly6VKnPcwFLlXCahVeerLKvgIaCeF6s8Br+9qAuqYt7bnIoKecWrkFiwm2qt2FpNzZPdfXBGsru7S47WStDWF5EUTsjRgFMiV/GcL6YkkS9WpWJySlIql2KVsSlR/Oq7KYnjHH7BP7NOnqynJF9liovyZkog0mIILODi24WcwwQ0W5SCgy+Zx0qyFURPmcKEeQrmz1kK9oXVcsVyiIMbKm/B/nYGGq/tQwz7ARqsnfdOyGN8CPcENVO8oqFVGmmgfsIgXgU7NbuLEzzMKngFU3VkUbp++PQ0fhVkpmucNLUGfpgpJmCf+DUjqkQTGdeBeGaCJJCdPH/8qF3PODLRA6GwquoxepFJZyaaDpVYsakhxpDOrvpECDADPoyzhoSQPi0Bo3a0cehhCa24gpj6jk0L2AZe4AwXNeQw2EYIclhalLI5JFYzRZ0Klur65NOPdT3b35+Se/Dv/X3dA+X4Jgfbn/7x/enB7F4fc7MEvRzm4N69KWl+aWPNGJZ8zU2Cxi0Gb4bAzGmRgvUTfQ6Z0OxCQrP6zkzrHkJ2CJmIIUTcIWCrhhCiQ4ghAKukrqNrcJFK8qwsemxepDzR3Rs25OMj4Epg7B20sPpI92EwM0SZdg98sVRUQJwNoLIGlbH5IMhMs30KI75F9HNUMa4/72v3jaV/06d/Yelf9Oknln6yYcxCibK6tdzPtHWW6DrnBe5Efa0vzV8Y645J1yGTrlsmuhpYV4GbC5YZNpC0rg+18a/THI5aE4W+HrAd5wew5xLcisXlGncdNYIn0FCuBEOJOkq5rDJ6K9UtHGd3DjS+dnK0pni8fURwU/ftpib119qmByoWoPzU/KXrvTa9PIUJn0EGoKoUd+vIwjQUEotoik8vA8J6HRCetgHXDXIdTTtwcyYM/BgG+fwITulPMGC5iUHiGIMSwHlUFpAxFlCasD27BVVszH6MuwNZkzZbU8UxmJqrkBvhoVXvakO1AK07kTytXoIHrt2DpHICEJMdvglQ3TH7KjjIy7UAUKDHkFCjCtrkjElzPqEwA++oI0VXuhtL4yFnDn1UXMMBWmBJ2Qp1pJrpllb7DL0BlbWZ96hVHHgykJE2xyZmZ6fkWSyZuKZudKqowtoKJamFmPkeNtOVHtxwnjUcaUQN7bSlYaQ1DBMZH+gGYdVY2BVP7BtxVEfltTkUncSHJkFWUpUacxRU9mnrTk7AESfzyw4w8RDw8icOgi9/rC9bEYuKy1WRstRNp3iWstqKaqxIeMpiwdMFozHotikiQxnZqpdlW2BZ1iJ9M3Qqz2Efr1nSauEveG6M4yYYXHQzfVuteJq5t5GPqeTBK2MHsgn1Y3gn8W3YsGXH94VxNoeIe4IeS8bBQvh6WKRjoMCEnDCaLFmjbVEqYQk1PttTDepVvmhOL0S1kGF2QguDGOA7Z8xuIMtvw/gHh4+cQNllywY8ecoSyqsFHGNLKXcikIcUvYOD6ArqxV04KA7YOlriGVPvNBrvaJvPyKeswEzernvRjME1sCo3EfSEUVFAJ0WwRm+QGRKRADHE4QTQPTLMcOrRwV98VuszFuNA2FG3s7aPeuLO31gPAKWHlANQeHGIDjQZELgeVmKDGb64fdn1S17m+x8c8EXd+WKjCJK7ZUILAzZaUhWmZJ8tN/lByJPSnNWYTJvSBVtQKDJge6/RB3AZKeS+ad8FpyQ66co5GD2B6i0gwAhKEonNWLSE3hT66AI6O6cZPCSlGYOG7AX8NT7VqWbS9t5QbidDEGPJeb3yXc0mKhB67mX7XSNFqjLjyW2Tf+yojio+PLsDmPmOBzB+OdlgXR/ghpe2ih+bbHIDfZgkgCZfUuhW8AAy/QnGEfQMvFJEBE0k1IQQyIKGlVbYFLaY+gwqx26U67pokoowr8bkHhT/DbgWXcXRQIWeQP5Zli0dS18BDmkqD1v39hFQOXcIUzb3EHbuZntO2vBCa8yhqPZhqAFW5sJu4mcc3FHwpkpYNkMzU1u9C48eU1F3uBYhV/O5h6LJSrEhHMMWFe98kFV7Mw8CKDSPtoTAG8GB+VoCNc7jIb6NEi4SXKQ7IlxZO68NkRxnIMKhpb51NWE7nsPiG8/vqNApKGHM8hQrRneSD6LwCqiPfMcWct9Cx0HzEnw1ynlKlpfN7r1rlHpEwUdblRI3alTGm59naskEecplsgeVugkSe/8DfCjyCXvh6q8af7EXul+BYc1mXMl0UbaN8iLubifX9Vxe6WZaaE8ULblShJI5LZJbAocfNEZYJhG1hADsd/wpAEwkj6H2X4BnN7dGenJuxJJSwLl4uF+pi3F0LcEArJ7luR5jxI8h96adyNi8/mzJSmjn3JuUHdXuL4Sve/C5dxv23WF+PYdyWdeP8XdAB+1KKMcEtLeP2scAwWzvqGvXRIZcbDvFlT5344v6xBICUMbAWB7miRkHkApa7FKafs4DHnvUAJ623Z+uu06wZ5IOdHcrCkAS6mc45Shi2ueekYq/sERBj41Wap+9vSLuOhS3zDzdcPSWmkFNZbWULGle7gPaXd3C9xfaQiYhxu3Btil87bdN4Zt/K8jzlW0Qu9Md2xxofy4gAIhXZJjbR9jbci6NnfMUmOdQmqqywrR4OKeZNFeE9mY6pJkqAknkkLREvBwCK0AAHt6vVAtzpH1Lwjtp0yMdzj4EChzT8EJiMcDmqCdQQiFDtXp0jDF+NgL0ChdhcwY+NQnMrMrvkFMLhDxxboAXtcsWjm4SWDy3iQyafzsb/H9ntqf1BbEJzk8fmrgZPD10bdKJRzFj9+gLf7/G8gdpfHrFVLLcqvjBD1EclPXbGetv4G14KKepKhPIsooJL9ZmOlJ0AaotWUvU9rY4cgFHV3CosjleLHsJ0x2rLlx8jB/27ljHoPMhbeZDNuZbn+kyrzlbu1jzEWGus5q4eAsV6bK0uUBxC/QxR82im87CfcLhYx42RO9WsthQKsiaRqclrcDYgUaW1KyOimDhxzB2LPPJWGB4JDQG44ueyR8bSnNVs4rnWUnVSwEvn8LkJp/bHENG3GYsn33aplNjRClXedXHPGypjX0yygM7P0KCSYQQcjm9YuZDD0USqPqvcP1BIc9UWWZYUYwjxFKVMQW2HYNZIa6SHK+7H549gG5HYcxA8iOGUOTB0Hxw86BuWLqG/5oZS6y3bI9XVRBlfE1caMng0DPBmKyg+MzRS8Kyws8UHQaSBSRMqIGZF68wXzgV1j1Rb3Yjhsr0yNgNYVRt08wcnX7JsakYQl5BMZy4p5hZ7aZihoy3yi4Qt+kG4btRVG2qh6hXUA9gffVQckA9QzbZvUkn2xTERKQ3a6xNHQ3wFZREXF9LIzugpqVPCCbGbQr2i9ZNzQziFTRDXF8zIzugmaXj1RXMtE21DB3Pq2M3Vcteze+yTb/Lhv3OkTFwbX4hlu5qT5YSeybJ91TXQGBOCoNq+8nXs8GW4yvUaFs+DeNl6zEZ+u3Lzrme62w9OU1iS1NCC2K+F0CwMiWyzBkxNUjYvRmSuaNrypuEYTGho2XzxYL6IIGtnlXK/jKFiQPZtz2CrjwD63ctgJfQ8UsJYTtqunjAY8+IH+fhdY7k30HmRux582Whw3fgqBV7OHz3wn783vxERZmycwnHLju0k01TQW+mxkQE3n14AM0mMRO8S9C7/oTXNv4Ul2/emfW/Q7T58OXB3uyDvf0v7t158Pyv9vtFb4x+P3p79M5oNvrD6MHos9Hx6PkoGc1Hfxv9ffSPnX/u/HvnPzv/tdDXX3PfSfrdKPjZ+d//ARMoFfs=</latexit>
a Gene a es b Reaches

F gure 1 A v sua of he genera es ( ef ) and reaches (r gh ) opera ors Generates: An agen bas s
ΛB compr sed of hree base agen s ΛB = {𝜆1B 𝜆2B 𝜆3B } genera es a se of agen s Λ con a n ng
agen s ha can be unders ood as sw ch ng be ween he base agen s n env ronmen 𝑒 For ns ance
on some fin e rea zab e h s ory ℎ ∈ ℋ̄ we dep c an agen 𝜆 ∈ Λ sw ch ng be ween he hree
base e emen s In he firs chunk of h s ory he agen ’s behav or s equ va en o ha of 𝜆3B hen
sw ches o 𝜆1B hen o 𝜆2B and fina y back o 𝜆1B Reaches: On he r gh we v sua ze he d v s on
of a se of agen s Λ re a ve o a bas s ΛB n some env ronmen hrough he reaches opera or The
bas s ΛB s a subse of he agen s ha a ways reach he bas s (defined n Append x B) Add ona y
he agen s ha a ways reach ΛB are a subse of hose ha some mes reach ΛB and he se of agen s
ha never reach he bas s (bo om) s he comp emen of hose ha some mes reach ( op)

5
 𝑒
Considerations on the Operators. Naturally, we can design many variations of both ⊢𝑒 and ⇝ . For
instance, we might be interested in a variant of reaches in which an agent becomes 𝜖-close under
an appropriate metric to any of the basis elements, rather than require exact behavioral equivalence.
Concretely, we highlight four axes of variation that modify the definitions of the operators. We state
these varieties for reaches, but similar modifications can be made to the generates operator, too:
1. Realizability. An agent reaches a basis (i) in all histories (and thus, all environments), or (ii)
in the histories realizable by a given (𝜆, 𝑒) pair.
2. History Length. An agent reaches a basis over (i) infinite or, (ii) finite length histories.
3. In Probability. An agent reaches a basis (i) with probability one, or (ii) with high probability.
4. Similarity or Equality. An agent reaches a basis by becoming (i) sufficiently similar to a
base agent, or (ii) equivalent to a base agent.
Rather than define all of these variations precisely for both operators, we acknowledge their existence,
and simply note that the formal definitions of these variants follow naturally.

4 Continual Reinforcement Learning

We now provide a precise definition of CRL. The definition formalizes the intuition that CRL captures
settings in which the best agents do not converge—they continue their implicit search over a basis
indefinitely.

4.1 Definition: Continual RL

To introduce our definition of CRL, we first define continual learning agents using the never reaches
operator.
Definition 4.1. An agent 𝜆 is a continual learning agent in 𝑒 relative to ΛB if and only if 𝜆 ⇝/𝑒 ΛB .
This means that an agent is a continual learning agent in an environment relative to a basis ΛB when
its search over the base behaviors continues forever. Notice that such an agent might be considered
a continual learner with respect to one basis but not another; it depends on the choice of basis. We
explore this fact more in Section 5. By similar reasoning, we might understand an agent as convergent
relative to a basis when it always reaches that basis.
Then, using these tools, we formally define the CRL problem as follows.

Definition 4.2. Consider an RL problem (𝑒 , 𝑣, Λ). Let ΛB ⊂ Λ be a basis such that ΛB ⊢𝑒 Λ, and
let Λ∗ = arg max𝜆∈Λ 𝑣(𝜆, 𝑒). We say (𝑒 , 𝑣, Λ, ΛB ) defines a CRL problem if ∀𝜆∗ ∈Λ∗ 𝜆∗ ⇝/𝑒 ΛB .

Said differently, an RL problem is an instance of CRL if all of the best agents are continual learning
agents relative to basis ΛB . This problem encourages a significant departure from how we tend to
think about designing agents: Given a basis, rather than try to build agents that can solve problems by
identifying a fixed high-quality element of the basis, we would like to design agents that continue to
update their behavior indefinitely in light of their experience.

4.2 CRL Examples

We next detail two examples of CRL to provide further intuition.

Q-Learning in Switching MDPs. First we consider a simple instance of CRL based on the standard
multi-task view of MDPs. In this setting, the agent repeatedly samples an MDP to interact with
from a fixed but unknown distribution [38, 6, 2]. In particular, we make use of the switching MDP
environment from Luketina et al. [20]. The switching MDP environment 𝑒 consists of a collection of
𝑛 underlying MDPs, 𝑚1 , 𝑚2 , . . . , 𝑚𝑛 , with a shared action space and state space. We refer to this
state space using observations, 𝑜 ∈ 𝒪. The environment has a fixed constant positive probability
of 0.001 to switch the underlying MDP, which yields different transition and reward functions until
the next switch. The agent can only observe each environment state 𝑜 ∈ 𝒪, which does not reveal

6
 <1
<latexit sha1_base64="sOjbLo6PuaLYJ2W8lA/TY8Pcu2A=">AAAQ1nicjVhbb9s2FFa7W+ddmm6Pe+EWFOgCJZAUO44LBOiaXvbQS5olbdEoCyiZttlIokZRSTxOw16GPQ3Y0/Zr9kP2b3ZIybZEOUUMRKLO+Xj48fDwHDJBGtFMOM5/166/9/4HH3504+POJ59+9vnNlVtfvMxYzkNyGLKI8dcBzkhEE3IoqIjI65QTHAcReRWc7ir9qzPCM8qSAzFNyXGMxwkd0RALEP0Qn7gnK6vOhqN/qN1wq8aqVf32Tm7d/NcfsjCPSSLCCGfZkeuk4lhiLmgYkaJzG62vr6M9HJ7iMcnUR8fPM5KWgqMY8zFNdpwNjybHckxYTASfFnWMJAmY51gQG8VYTHg2ymwE8wiAdmwjHGdKrBtiUgqyaRxAIxozTktZNE4zkgNVNgQ7KR2xRNgoy4MRHdtolEdRCmPZKMx+ypkgMMIQZxOeRyAT9PRnGwUB2AkYOxU4ALX6WpDIeWSjC70ETfLQqykYZmrohuxoxHFMYOYTNtw5oKdvjmU81LJh0QEPYpowpHptNCwl5Dy9EORC2LqlnKDh68rjerUzhJMheqbmo6WgfEBI+pSCVHPNOv6QjCBc9JccQqjkpJD7j+8X0rEHm7bndgsTk+Y8jWYo1+nZPcf2Np0WLmoCtzx70Le9Xq8FHNZG9cBWz+4D6LaBqnNzt+1e13a7/TYsqsP6ju0O3KWDjjkhSYXrbdkumPQG4D/w0dPpcu/ULA+2beR2HXgMWhOv41zHA4yngP1NE1in6rlbNvI2e/Bok+UQCJW9ASBcVxl1W/bOJ1TM7YF/0OxhAhnHyXiO3FTstgbw2PRMZN1L25tqKn0bbb/bm8qVYLCv0C1k1EAqfuWUen0TKfAM5nkuTMNVvnFa3k5pcjqPHTXmlnKks23ipiSK2PncO8pYV/EcDNpzxtMK1wVb3YH60zvrCR1PhBkVjYDTboR9A3Rbdq8+9SZXRcJTseZ5mypAfdjuIYtj2NvSJ2lWSP8Mc2jQiCUq4x5MU8jpEYpoTMXdJj4BIXTIkyEUAiJkgnyuZoU5Z+fIp8lIwOR9hSqMnlmuesLzRFK0g9zix8SEpJxBpPrqVQc1Uff3HgBGp/JYqg/DyuO6/rHWq00J32pSKqOpdhA0e8VhceQel/0AJ1dd024cLBBBsAQAaZGGxQwQyxYgeDjXyoemkmNQLnxpaCOljchoqfKZMhvIZ6bJN6X8jSl/UcpfmPL9Ur5fmC4fQngVR96x7PgBgYK7qC9HnEEshIwnhO+4KdTEAMrLuJJCNO7orHLc8c8yKDxEujEUNHDFCEoJhCxVR4cC3Vl1v72LwIUCraokQmDURQ1TdB4QOB9wopbuearKOeNr0ofyH1PYE/D2bdV6FxBfzIDQMlaGgp3hLGxIBFFGx7+0/KCnoIo6IEtHhHBwIbzwJ7NyL70Qwhw8UT70TCpQZU1MCOMkXpOzRiEPylZR10uagE/DPBMshjOIASLJGeUsUecmucBAfHYQuJGTBWsf7DVNqdD1Deu6G3A1xR10GSdYc17IXcZZBP6eXkZLw67AS+FMYrrvEmal/DJiykdyn8DR8PQyUgpyBU7KsEFJL0CbkRZDltnjLGWZDurL6EFySwtZA15GUgOvwFLhTJq67xKepVyd4/AZQbg1MlRNOO3rUWcRDgfcnINoAnu4DGSajGd5AOA5rH6hxxcs1XtDdWVCeQW+yr08w5VflcnSY7lAGI1wEk7hdHyBsE4dSExglGzDSEMA0Dn43Wmoq9KQnjQwqm4ISqZFJTNTqvOuKVTptiZbfEJhUoJap0pi5j51FPi6ZyY/8NWSDAd1twyWrBE5cCfK41THifxu0W5gQpa8JaFQPoUNOW8bmNlGNffsHBFhVdt39auhIRc41ifxh1WjoY0IuLOQT/SroUlrQd6O+BmKl1u1sWVnuhEOIZE+Us9m7lSKtaWa0tzacnvV+5KMO1xUJFmrTk37C9BaEzUvPfssgM22O4vaPeia0fLaJm/Dpix/vhKQBC6UEOny+YFbGLoMdhrJZNyST3AKtzdTPMIxjaYyTaO3dRVc5yEcqjtjRfAALn2PQDKnWGfY4NsxSpZe67X5Ynfgwu+a1/t246W34fY2nBfd1XuHv5VX/xvWV9Y31h3LtfrWPet7a886tEJrbP1p/W39s/J65deV31f+KKHXr1X/LvjSavxW/vofHHkZmA==</latexit>

<2
<latexit sha1_base64="yqDt4hg3Zw5elrwBpwmphvge++M=">AAAQ1nicjVhbb9s2FFa7W+ddmm6Pe+EWFOgCJZAUO44LBOiaXvbQS5olbdEoCyiZttlIokZRSTxOw16GPQ3Y0/Zr9kP2b3ZIybZEOUUMRKLO+Xj48fDwHDJBGtFMOM5/166/9/4HH3504+POJ59+9vnNlVtfvMxYzkNyGLKI8dcBzkhEE3IoqIjI65QTHAcReRWc7ir9qzPCM8qSAzFNyXGMxwkd0RALEP0Qn3gnK6vOhqN/qN1wq8aqVf32Tm7d/NcfsjCPSSLCCGfZkeuk4lhiLmgYkaJzG62vr6M9HJ7iMcnUR8fPM5KWgqMY8zFNdpwNjybHckxYTASfFnWMJAmY51gQG8VYTHg2ymwE8wiAdmwjHGdKrBtiUgqyaRxAIxozTktZNE4zkgNVNgQ7KR2xRNgoy4MRHdtolEdRCmPZKMx+ypkgMMIQZxOeRyAT9PRnGwUB2AkYOxU4ALX6WpDIeWSjC70ETfLQqykYZmrohuxoxHFMYOYTNtw5oKdvjmU81LJh0QEPYpowpHptNCwl5Dy9EORC2LqlnKDh68rjerUzhJMheqbmo6WgfEBI+pSCVHPNOv6QjCBc9JccQqjkpJD7j+8X0rEHm7bndgsTk+Y8jWYo1+nZPcf2Np0WLmoCtzx70Le9Xq8FHNZG9cBWz+4D6LaBqnNzt+1e13a7/TYsqsP6ju0O3KWDjjkhSYXrbdkumPQG4D/w0dPpcu/ULA+2beR2HXgMWhOv41zHA4yngP1NE1in6rlbNvI2e/Bok+UQCJW9ASBcVxl1W/bOJ1TM7YF/0OxhAhnHyXiO3FTstgbw2PRMZN1L25tqKn0bbb/bm8qVYLCv0C1k1EAqfuWUen0TKfAM5nkuTMNVvnFa3k5pcjqPHTXmlnKks23ipiSK2PncO8pYV/EcDNpzxtMK1wVb3YH60zvrCR1PhBkVjYDTboR9A3Rbdq8+9SZXRcJTseZ5mypAfdjuIYtj2NvSJ2lWSP8Mc2jQiCUq4x5MU8jpEYpoTMXdJj4BIXTIkyEUAiJkgnyuZoU5Z+fIp8lIwOR9hSqMnlmuesLzRFK0g9zix8SEpJxBpPrqVQc1Uff3HgBGp/JYqg/DyuO6/rHWq00J32pSKqOpdhA0e8VhceQel/0AJ1dd024cLBBBsAQAaZGGxQwQyxYgeDjXyoemkmNQLnxpaCOljchoqfKZMhvIZ6bJN6X8jSl/UcpfmPL9Ur5fmC4fQngVR96x7PgBgYK7qC9HnEEshIwnhO+4KdTEAMrLuJJCNO7orHLc8c8yKDxEujEUNHDFCEoJhCxVR4cC3Vl1v72LwIUCraokQmDURQ1TdB4QOB9wopbuearKOeNr0ofyH1PYE/D2bdV6FxBfzIDQMlaGgp3hLGxIBFFGx7+0/KCnoIo6IEtHhHBwIbzwJ7NyL70Qwhw8UT70TCpQZU1MCOMkXpOzRiEPylZR10uagE/DPBMshjOIASLJGeUsUecmucBAfHYQuJGTBWsf7DVNqdD1Deu6G3A1xR10GSdYc17IXcZZBP6eXkZLw67AS+FMYrrvEmal/DJiykdyn8DR8PQyUgpyBU7KsEFJL0CbkRZDltnjLGWZDurL6EFySwtZA15GUgOvwFLhTJq67xKepVyd4/AZQbg1MlRNOO3rUWcRDgfcnINoAnu4DGSajGd5AOA5rH6hxxcs1XtDdWVCeQW+yr08w5VflcnSY7lAGI1wEk7hdHyBsE4dSExglGzDSEMA0Dn43Wmoq9KQnjQwqm4ISqZFJTNTqvOuKVTptiZbfEJhUoJap0pi5j51FPi6ZyY/8NWSDAd1twyWrBE5cCfK41THifxu0W5gQpa8JaFQPoUNOW8bmNlGNffsHBFhVdt39auhIRc41ifxh1WjoY0IuLOQT/SroUlrQd6O+BmKl1u1sWVnuhEOIZE+Us9m7lSKtaWa0tzacnvV+5KMO1xUJFmrTk37C9BaEzUvPfssgM22O4vaPeia0fLaJm/Dpix/vhKQBC6UEOny+YFbGLoMdhrJZNyST3AKtzdTPMIxjaYyTaO3dRVc5yEcqjtjRfAALn2PQDKnWGfY4NsxSpZe67X5Ynfgwu+a1/t246W34fY2nBfd1XuHv5VX/xvWV9Y31h3LtfrWPet7a886tEJrbP1p/W39s/J65deV31f+KKHXr1X/LvjSavxW/vofLOkZmQ==</latexit>

<=
<latexit sha1_base64="JawjiQnBxWx7K4i8YjzFvVd0rfE=">AAAQ1nicjVhbb9s2FFa7W+ddmm6Pe+EWFOgCJZAUO44LBOiaXvbQS5olbdEoCyiZttlIokZRSTxOw16GPQ3Y0/Zr9kP2b3ZIybZEOUUMRKLO+Xj48fDwHDJBGtFMOM5/166/9/4HH3504+POJ59+9vnNlVtfvMxYzkNyGLKI8dcBzkhEE3IoqIjI65QTHAcReRWc7ir9qzPCM8qSAzFNyXGMxwkd0RALEP0QnyQnK6vOhqN/qN1wq8aqVf32Tm7d/NcfsjCPSSLCCGfZkeuk4lhiLmgYkaJzG62vr6M9HJ7iMcnUR8fPM5KWgqMY8zFNdpwNjybHckxYTASfFnWMJAmY51gQG8VYTHg2ymwE8wiAdmwjHGdKrBtiUgqyaRxAIxozTktZNE4zkgNVNgQ7KR2xRNgoy4MRHdtolEdRCmPZKMx+ypkgMMIQZxOeRyAT9PRnGwUB2AkYOxU4ALX6WpDIeWSjC70ETfLQqykYZmrohuxoxHFMYOYTNtw5oKdvjmU81LJh0QEPYpowpHptNCwl5Dy9EORC2LqlnKDh68rjerUzhJMheqbmo6WgfEBI+pSCVHPNOv6QjCBc9JccQqjkpJD7j+8X0rEHm7bndgsTk+Y8jWYo1+nZPcf2Np0WLmoCtzx70Le9Xq8FHNZG9cBWz+4D6LaBqnNzt+1e13a7/TYsqsP6ju0O3KWDjjkhSYXrbdkumPQG4D/w0dPpcu/ULA+2beR2HXgMWhOv41zHA4yngP1NE1in6rlbNvI2e/Bok+UQCJW9ASBcVxl1W/bOJ1TM7YF/0OxhAhnHyXiO3FTstgbw2PRMZN1L25tqKn0bbb/bm8qVYLCv0C1k1EAqfuWUen0TKfAM5nkuTMNVvnFa3k5pcjqPHTXmlnKks23ipiSK2PncO8pYV/EcDNpzxtMK1wVb3YH60zvrCR1PhBkVjYDTboR9A3Rbdq8+9SZXRcJTseZ5mypAfdjuIYtj2NvSJ2lWSP8Mc2jQiCUq4x5MU8jpEYpoTMXdJj4BIXTIkyEUAiJkgnyuZoU5Z+fIp8lIwOR9hSqMnlmuesLzRFK0g9zix8SEpJxBpPrqVQc1Uff3HgBGp/JYqg/DyuO6/rHWq00J32pSKqOpdhA0e8VhceQel/0AJ1dd024cLBBBsAQAaZGGxQwQyxYgeDjXyoemkmNQLnxpaCOljchoqfKZMhvIZ6bJN6X8jSl/UcpfmPL9Ur5fmC4fQngVR96x7PgBgYK7qC9HnEEshIwnhO+4KdTEAMrLuJJCNO7orHLc8c8yKDxEujEUNHDFCEoJhCxVR4cC3Vl1v72LwIUCraokQmDURQ1TdB4QOB9wopbuearKOeNr0ofyH1PYE/D2bdV6FxBfzIDQMlaGgp3hLGxIBFFGx7+0/KCnoIo6IEtHhHBwIbzwJ7NyL70Qwhw8UT70TCpQZU1MCOMkXpOzRiEPylZR10uagE/DPBMshjOIASLJGeUsUecmucBAfHYQuJGTBWsf7DVNqdD1Deu6G3A1xR10GSdYc17IXcZZBP6eXkZLw67AS+FMYrrvEmal/DJiykdyn8DR8PQyUgpyBU7KsEFJL0CbkRZDltnjLGWZDurL6EFySwtZA15GUgOvwFLhTJq67xKepVyd4/AZQbg1MlRNOO3rUWcRDgfcnINoAnu4DGSajGd5AOA5rH6hxxcs1XtDdWVCeQW+yr08w5VflcnSY7lAGI1wEk7hdHyBsE4dSExglGzDSEMA0Dn43Wmoq9KQnjQwqm4ISqZFJTNTqvOuKVTptiZbfEJhUoJap0pi5j51FPi6ZyY/8NWSDAd1twyWrBE5cCfK41THifxu0W5gQpa8JaFQPoUNOW8bmNlGNffsHBFhVdt39auhIRc41ifxh1WjoY0IuLOQT/SroUlrQd6O+BmKl1u1sWVnuhEOIZE+Us9m7lSKtaWa0tzacnvV+5KMO1xUJFmrTk37C9BaEzUvPfssgM22O4vaPeia0fLaJm/Dpix/vhKQBC6UEOny+YFbGLoMdhrJZNyST3AKtzdTPMIxjaYyTaO3dRVc5yEcqjtjRfAALn2PQDKnWGfY4NsxSpZe67X5Ynfgwu+a1/t246W34fY2nBfd1XuHv5VX/xvWV9Y31h3LtfrWPet7a886tEJrbP1p/W39s/J65deV31f+KKHXr1X/LvjSavxW/vofB2UZ1Q==</latexit>

<3
<latexit sha1_base64="Z0arzvXie0v6qa241DDUQBmHOfA=">AAAQ1nicjVhbb9s2FFa7W+ddmm6Pe+EWFOgCNZBkO44LBOiaXvbQS5qlFzTKAkqmbTaSqFFUEo/TsJdhTwP2tP2a/ZD9mx1Ssi1RThEDkahzPh5+PDw8h0yQRjQTjvPflavvvf/Bhx9d+7jzyaeffX597cYXLzOW85C8CFnE+OsAZySiCXkhqIjI65QTHAcReRWc7Cr9q1PCM8qSAzFLyVGMJwkd0xALEP0QH3eP19adTUf/ULvhVo11q/rtHd+4/q8/YmEek0SEEc6yQ9dJxZHEXNAwIkXnJrp9+zbaw+EJnpBMfXT8PCNpKTiMMZ/QZMfZ9GhyJCeExUTwWVHHSJKAeY4FsVGMxZRn48xGMI8AaMc2wnGmxLohpqUgm8UBNKIJ47SURZM0IzlQZSOwk9IxS4SNsjwY04mNxnkUpTCWjcLsp5wJAiOMcDbleQQyQU9+tlEQgJ2AsROBA1CrryWJnEc2OtdL0CQPvZqCUaaGbsgOxxzHBGY+ZaOdA3ry5kjGIy0bFR3wIKYJQ6rXZsNSQs7Sc0HOha1bygkaflt5XK92hnAyQk/VfLQUlPcJSZ9QkGquWccfkTGEi/6SIwiVnBRy/9G9Qjr2sGt7bq8wMWnO02iOcp2+3Xdsr+u0cFETuOXZw4Ht9fst4Kg2qge2+vYAQDcNVJ2bu233e7bbG7RhUR02cGx36K4cdMIJSSpcf8t2waQ3BP+Bj57MVnunZnm4bSO358Bj2Jp4Hec6HmA8BRx0TWCdqudu2cjr9uHRJsshECp7Q0C4rjLqtuydTalY2AP/oPnDBDKOk8kC2VXstobw6Homsu6l7a6aysBG2+/2pnIlGBwodAsZNZCKXzml/sBECjyHeZ4L03CVb5yWt1OanCxiR425pRzpbJu4GYkidrbwjjLWUzyHw/ac8azC9cBWb6j+9M56TCdTYUZFI+C0G2HfAN2W3ctPvclVkfBUrHleVwWoD9s9ZHEMe1v6JM0K6Z9iDg0asURl3INZCjk9QhGNqbjTxCcghA55MoJCQIRMkM/VrDDn7Az5NBkLmLyvUIXRM8tVT3geS4p2kFv8mJiQlDOIVF+96qAm6t7efcDoVB5L9WFYeVTXP9J6tSnhW01KZTTVDoJmrzgsDt2jsh/g5Lpr2o2DJSIIVgAgLdKwmANi2QIEDxZa+cBUcgzKpS8NbaS0ERmvVD5VZgP51DT5ppS/MeXPS/lzU75fyvcL0+UjCK/i0DuSHT8gUHCX9eWQM4iFkPGE8B03hZoYQHmZVFKIxh2dVY46/mkGhYdIN4aCBq4YQymBkKXq6FCgW+vut3cQuFCgdZVECIy6rGGKzn0C5wNO1NI9S1U5Z3xD+lD+Ywp7At6+rVrvAuLzORBaxspQsDOahw2JIMro5JeWH/QUVFEHZOmIEA4uhBf+dF7upRdCmIMnyoeeSQWqrIkpYZzEG3LeKORB2SrqekkT8GmYZ4LFcAYxQCQ5pZwl6twklxiIzw4CN3KyZO2DvaYpFbq+YV13A66muIMu4gRrzgu5yziLwN+zi2hp2CV4KZxJTPddwayUX0RM+UjuEzganlxESkEuwUkZNijpBWgz0mLIMnucpSzTQX0RPUhuaSFrwItIauAlWCqcSVP3XcGzlKtzHD4lCLdGhqoJp3096jzC4YCbcxBNYQ+XgUyTyTwPADyH1S/0+IKlem+orkwor8BXuZfnuPKrMll6LBcIozFOwhmcjs8R1qkDiSmMkm0aaQgAOge/Ow31VBrSkwZG1Q1BybSoZGZKdd41hSrd1mTLTyhMSlDrVEnM3KeOAl/3zeQHvlqR4aDulsGSNSIH7kR5nOo4kd8t2w1MyJK3JBTKp7AhF20DM9+o5p5dICKsavuufjU05BzH+iT+oGo0tBEBdxbysX41NGktyNsRP0fxcqs2tuxcN8YhJNKH6tnMnUqxsVJTmttYba96X5BxR8uKJGvVqWl/CdpoohalZ58FsNl251G7B10zWl7b5E3YlOXPVwKSwIUSIl0+O3ALQ5fBTiOZjFvyKU7h9maKxzim0UymafS2roLrPIRDdWesCB7Ape8hSBYU6wwbfDtGydJrvbFY7A5c+F3zet9uvPQ23f6m87y3fvfFb+XV/5r1lfWNdctyrYF11/re2rNeWKE1sf60/rb+WXu99uva72t/lNCrV6p/F3xpNX5rf/0PPVkZmg==</latexit>

(a) Switching MDP Visual (b) Switching MDP Experiment

Figure 2: A visual of a grid world instance of the switching MDPs problem (left), and results from a
simple experiment contrasting continual learning and convergent Q-learning (right). The environment
pictured contains 𝑛 distinct MDPs. Each underlying MDP shares the same state space and action
space, but varies in transition and reward functions, as indicated by the changing walls and rewarding
locations (stars, circles, and fire). The results pictured on the right contrast continual Q-learning (with
𝛼 = 0.1) with traditional Q-learning that anneals its step-size to zero over time.

the identity of the currently active MDP. The rewards of each underlying MDP are structured so
that each MDP has a unique optimal policy. We assume 𝑣 is defined as the average reward, and
the agent basis is the set of 𝜖-greedy policies, for fixed 𝜖 = 0.15. That is, the basis contains every
policy that, for each state, chooses a fixed action with probability 0.85 and with the remaining 0.15
chooses uniformly at random among all actions. Consequently, the set of agents we generate, ΛB ⊢𝑒 Λ,
consists of all agents that switch between these 𝜖-greedy policies.
Now that the components (𝑒 , 𝑣, Λ, ΛB ) have been defined, we can see that this is indeed an instance
of CRL: None of the base agents can be optimal, as the moment that the environment switches
between its underlying MDP, we know that any previously optimal memoryless policy will no longer
be optimal in the next MDP following the switch. Therefore, any agent that converges to the basis ΛB
cannot be optimal either for the same reason. We conclude that all optimal agents in Λ are continual
learning agents relative to the basis ΛB —they each continuously change their base behavior.
We present a visual of this domain in Figure 2(a), and conduct a simple experiment contrasting the
performance of 𝜖-greedy continual Q-learning (orange) that uses a constant step-size of 𝛼 = 0.1, with
a convergent Q-learning (pink) that anneals its step size over time to zero. Both use 𝜖 = 0.15, and
we set the number of underlying MDPs to 𝑛 = 10. We present the mean episodic reward with 95%
confidence intervals, averaged over 250 runs, in Figure 2(b). Since both variants of Q-learning can be
viewed as searching over ΛB , the annealing variant (pink) that stops its search will under-perform
compared to the continual approach (orange). These results support the unsurprising conclusion that
it is better to track than converge in this setting.

Continual Supervised Learning. Second, we illustrate the breadth of our CRL definition by
showing how it can capture continual supervised learning. We adopt the problem setting studied
by Mai et al. [22]. Let 𝒳 denote a set of objects to be labeled, each belonging to one of 𝑘 ∈ N
classes. The observation space 𝒪 consists of pairs, 𝑜 𝑡 = (𝑥 𝑡 , 𝑦𝑡 ), where 𝑥 𝑡 ∈ 𝒳 and 𝑦𝑡 ∈ 𝒴,
where each 𝑥 𝑡 is an input and 𝑦𝑡 is the label for the previous input 𝑥 𝑡−1 . That is, 𝒪 = 𝒳 × 𝒴.
We assume by convention that the initial label 𝑦0 is irrelevant and can be ignored. The agent will
observe a sequence of object-label pairs, (𝑥0 , 𝑦0 ), (𝑥1 , 𝑦1 ), . . ., and the action space is a choice of
label, 𝒜 = {𝑎 1 , . . . , 𝑎 𝑘 } where |𝒴| = 𝑘. The reward for each history ℎ 𝑡 is +1 if the agent’s most
recently predicted label is correct for the previous input, and −1 otherwise:
+1 𝑎 𝑡−1 = 𝑦𝑡 ,

𝑟(ℎ 𝑡 ) = 𝑟(ℎ 𝑡−1 𝑎 𝑡−1 𝑜 𝑡 ) = 𝑟(𝑎 𝑡−1 𝑦𝑡 ) = (4.1)
−1 𝑎 𝑡−1 = otherwise.

Concretely, the continual learning setting studied by Mai et al. [22] supposes the learner will receive
samples from a sequence of probability distributions, 𝑑0 , 𝑑1 , . . ., each supported over 𝒳 × 𝒴. The
(𝑥, 𝑦) ∈ 𝒳 × 𝒴 pairs experienced by the learner are determined by the sequence of distributions. This

7
can be easily captured by our formalism by appropriately defining an environment (Definition 2.3)
whose conditional distributions 𝑒(𝑂 𝑡+1 | ℎ 𝑡 ) match the sequence of distributions 𝑑0 , 𝑑1 , . . .. To
see this, note that the choice of distribution 𝑑 𝑖 can itself be formalised as a function of history
𝑓 : ℋ → Δ(𝒳 × 𝒴).
Now, is this an instance of CRL? To answer this question precisely, we need to select a (Λ, ΛB ) pair.
We adopt the basis ΛB = {𝜆B : 𝑥 𝑦 ↦→ 𝑦 𝑖 , ∀ 𝑦𝑖 ∈𝒴 } where each basis element maps all observations
onto a fixed label 𝑦 𝑖 . By the universal set of learning rules Σ, this basis generates all possible
classifiers over the histories (𝑥0 𝑦0 )(𝑥1 𝑦1 ), . . .. Now, our definition says the above is an instance of
CRL if every optimal agent endlessly switches between classifiers, rather than adopt a fixed classifier.
Consequently, if there is an optimal classifier in ΛB , then this will not be an instance of CRL. If,
however, the environment imposes enough distributional shift (changing labels, adding mass to new
elements, and so on), then the only optimal agents will be those that always switch among the base
classifiers, in which case the setting is an instance of CRL.

5 Properties of CRL
Our formalism is intended to be a jumping off point for new lines of thinking around agents and
continual learning. We defer much of our analysis and proofs to the appendix, and here focus on
highlighting necessary properties of CRL.
Theorem 5.1. Every instance of CRL (𝑒 , 𝑣, Λ, ΛB ) necessarily satisfies the following properties:

1. There exists a Λ′B such that (1) Λ′B ⊢𝑒 Λ, and (2) (𝑒 , 𝑣, Λ, Λ′B ) is not an instance of CRL.
2. ΛB is not a superset of Λ,
3. Λ ⊂ Λ.
4. If |Λ| is finite, there exists an agent set, Λ◦ , such that |Λ◦ | < |Λ| and Λ◦ ⊢𝑒 Λ.
5. If |Λ| is infinite, there exists an agent set, Λ◦ , such that Λ◦ ⊂ Λ and Λ◦ ⊢𝑒 Λ.

This theorem tells us several things. The first point of the theorem has peculiar implications. We see
that as we change a single element (the basis ΛB ) of the tuple (𝑒 , 𝑣, ΛB , Λ), the resulting problem can
change from CRL to not CRL. By similar reasoning, an agent that is said to be a continual learning
agent according to Definition 4.1 may not be a continual learner with respect to some other basis.
We discuss this point further in the next paragraph. Next, by point (2.) we know that ΛB is not
a superset of Λ. Motivated by this, we tend to think of ΛB as a proper subset of Λ, and thus our
definition and visual in Figure 1(b) suppose ΛB ⊂ Λ. However, it is worth noting that there might be
stylized settings where it is useful to consider the case that ΛB intersects with, but is a not a subset of,
Λ. Point (3.) shows that, in CRL, the space of agents we are interested in designing cannot be the
set of all agents. These agents might be limited as a result of a restricted computational or memory
budget, or by making use of a constrained set of learning rules. This gives an initial connection
between bounded agents and the nature of continual learning—we explore this connection further
in Appendix C. Points (4.) and (5.) show that Λ cannot be minimal. That is, there are necessarily
some redundancies in the design space of the agents in CRL—this is expected, since we are always
focusing on agents that search over the same set of basis behaviors. While these five points give an
initial character of the CRL problem, we note that further exploration of the properties of CRL is an
important direction for future work.

Canonical Agent Bases. It is worth pausing and reflecting on the concept of an agent basis. As
presented, the basis is an arbitrary choice of a set of agents—consequently, point (1.) of Theorem 5.1
may stand out as peculiar. From this point, it is reasonable to ask if the fact that our definition of
CRL is basis-dependant renders it vacuous. We argue that this is not the case for two reasons. First,
we conjecture that any definition of continual learning that involves concepts like “learning” and
“convergence” will have to sit on top of some reference object whose choice is arbitrary. Second, and
more important, even though the mathematical construction allows for an easy change of basis, in
practice the choice of basis is constrained by considerations like the availability of computational
resources. It is often the case that the domain or problem of interest provides obvious choices of
bases, or imposes constraints that force us as designers to restrict attention to a space of plausible

8
bases. For example, as discussed earlier, a choice of neural network architecture might comprise a
basis—any assignment of weights is an element of the basis, and the learning rule 𝜎 is a mechanism
for updating the active element of the basis (the parameters) in light of experience. In this case, the
number of parameters of the network is constrained by what we can actually build. Further, we can
think of the learning rule 𝜎 as something like stochastic gradient descent, rather than a rule that can
search through the basis in an unconstrained way. In this sense, the basis is not arbitrary. We as
designers choose a class of functions to act as the relevant representations of behavior, often limited
by resource constraints on memory or compute. Then, we use specific learning rules that have been
carefully designed to react to experience in a desirable way—for instance, stochastic gradient descent
updates the current choice of basis in the direction that would most improve performance. For these
reasons, the choice of basis is not arbitrary, but instead reflects the ingredients involved in the design
of agents as well as the constraints necessarily imposed by the environment.

6 Discussion

In this paper, we carefully develop a simple mathematical definition of the continual RL problem.
We take this problem to be of central importance to AI as a field, and hope that the conceptual
basis we provide can serve as an opportunity to think about CRL and its related artifacts more
carefully. Our proposal is framed around two fundamental insights about agents: (i) every agent
can be understood as though it were searching over a space of behaviors (Theorem 3.1), and (ii)
every agent, in the limit, will either sometimes or never stop this search (Remark 3.2). These two
insights are formalized through the generators and reaches operators, which provide a rich toolkit for
understanding agents in a new way—for example, we find straightforward definitions of a continual
learning agent (Definition 4.1) and learning rules (Definition 3.2). We anticipate that further study of
families of learning of rules can directly inform the design of new learning algorithms; for instance,
we might characterize the family of continual learning rules that are guaranteed to yield continual
learning agents, and use this to guide the design of principled continual learning agents (in the spirit
of continual backprop by Dohare et al. [8]). In future work, we intend to further explore connections
between our formalism of continual learning and some of the phenomena at the heart of recent
empirical continual learning studies, such as plasticity loss [21, 1], in-context learning [5], and
catastrophic forgetting [10]. More generally, we hope that our definitions, analysis, and perspectives
can help the community to think about continual reinforcement learning in a new light.

Acknowledgements

The authors are grateful to Michael Bowling, Clare Lyle, Georgios Piliouras, and Razvan Pascanu for
comments on a draft of the paper, as well as Elliot Catt, Will Dabney, Sebastian Flennerhag, Steven
Hansen, Anna Harutyunyan, Joe Marino, Joseph Modayil, Remi Munos, Brendan O’Donoghue, Matt
Overlan, Mark Rowland, Tom Schaul, Yannick Shroecker, Yunhao Tang, Shantanu Thakoor, Zheng
Wen, and all of the 2023 Barbados RL Workshop participants for inspirational conversations.

References
[1] Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C Machado. Loss of
plasticity in continual deep reinforcement learning. arXiv preprint arXiv:2303.07507, 2023.

[2] David Abel, Yuu Jinnai, Yue Guo, George Konidaris, and Michael L. Littman. Policy and value
transfer in lifelong reinforcement learning. In Proceedings of the International Conference on
Machine Learning, 2018.

[3] Megan M Baker, Alexander New, Mario Aguilar-Simon, Ziad Al-Halah, Sébastien MR Arnold,
Ese Ben-Iwhiwhu, Andrew P Brna, Ethan Brooks, Ryan C Brown, Zachary Daniels, et al. A
domain-agnostic approach for characterization of lifelong learning systems. Neural Networks,
160:274–296, 2023.

[4] Michael Bowling, John D. Martin, David Abel, and Will Dabney. Settling the reward hypothesis.
In Proceedings of the International Conference on Machine Learning, 2023.

9
 [5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are
few-shot learners. Advances in Neural Information Processing Systems, 2020.
[6] Emma Brunskill and Lihong Li. PAC-inspired option discovery in lifelong reinforcement
learning. In Proceedings of the International Conference on Machine Learning, 2014.
[7] Anthony R. Cassandra, Leslie Pack Kaelbling, and Michael L. Littman. Acting optimally in
partially observable stochastic domains. In Proceedings of the AAAI Conference on Artificiall
Intelligence, 1994.
[8] Shibhansh Dohare, Richard S Sutton, and A Rupam Mahmood. Continual backprop: Stochastic
gradient descent with persistent randomness. arXiv preprint arXiv:2108.06325, 2021.
[9] Shi Dong, Benjamin Van Roy, and Zhengyuan Zhou. Simple agent, complex environment:
Efficient reinforcement learning with agent states. Journal of Machine Learning Research, 23
(255):1–54, 2022.
[10] Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive
sciences, 3(4):128–135, 1999.
[11] Milton Friedman. Essays in positive economics. University of Chicago press, 1953.
[12] Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu. Embracing change: Con-
tinual learning in deep neural networks. Trends in cognitive sciences, 24(12):1028–1040,
2020.
[13] Marcus Hutter. A theory of universal artificial intelligence based on algorithmic complexity.
arXiv preprint cs/0004001, 2000.
[14] Marcus Hutter. Universal artificial intelligence: Sequential decisions based on algorithmic
probability. Springer Science & Business Media, 2004.
[15] Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual rein-
forcement learning: A review and perspectives. arXiv preprint arXiv:2012.13490, 2020.
[16] Tor Lattimore. Theory of general reinforcement learning. PhD thesis, The Australian National
University, 2014.
[17] Jan Leike. Nonparametric general reinforcement learning. PhD thesis, The Australian National
University, 2016.
[18] Timothée Lesort, Vincenzo Lomonaco, Andrei Stoian, Davide Maltoni, David Filliat, and
Natalia Dı́az-Rodrı́guez. Continual learning for robotics: Definition, framework, learning
strategies, opportunities and challenges. Information fusion, 58:52–68, 2020.
[19] Xiuyuan Lu, Benjamin Van Roy, Vikranth Dwaracherla, Morteza Ibrahimi, Ian Osband, and
Zheng Wen. Reinforcement learning, bit by bit. arXiv preprint arXiv:2103.04047, 2021.
[20] Jelena Luketina, Sebastian Flennerhag, Yannick Schroecker, David Abel, Tom Zahavy, and
Satinder Singh. Meta-gradients in non-stationary environments. In Proceedings of the Confer-
ence on Lifelong Learning Agents, 2022.
[21] Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will
Dabney. Understanding plasticity in neural networks. arXiv preprint arXiv:2303.01486, 2023.
[22] Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online
continual learning in image classification: An empirical survey. Neurocomputing, 469:28–51,
2022.
[23] Sultan J Majeed. Abstractions of general reinforcement Learning. PhD thesis, The Australian
National University, 2021.
[24] Tom Mitchell. Machine Learning. McGraw-Hill, 1997.

10
[25] German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual
lifelong learning with neural networks: A review. Neural networks, 113:54–71, 2019.
[26] Emmanouil Antonios Platanios, Abulhair Saparov, and Tom Mitchell. Jelly bean world: A
testbed for never-ending learning. arXiv preprint arXiv:2002.06306, 2020.
[27] Martin L Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming.
John Wiley & Sons, 2014.
[28] Matthew Riemer, Sharath Chandra Raparthy, Ignacio Cases, Gopeshh Subbaraj, Maximilian
Puelma Touzel, and Irina Rish. Continual learning in environments with polynomial mixing
times. Advances in Neural Information Processing Systems, 2022.
[29] Mark B Ring. Continual learning in reinforcement environments. PhD thesis, The University of
Texas at Austin, 1994.
[30] Mark B Ring. Child: A first step towards continual learning. Machine Learning, 28(1):77–104,
1997.
[31] Mark B Ring. Toward a formal framework for continual learning. In NeurIPS Workshop on
Inductive Transfer, 2005.
[32] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Ex-
perience replay for continual learning. Advances in Neural Information Processing Systems,
2019.
[33] Tom Schaul, Hado van Hasselt, Joseph Modayil, Martha White, Adam White, Pierre-Luc Bacon,
Jean Harb, Shibl Mourad, Marc Bellemare, and Doina Precup. The Barbados 2018 list of open
issues in continual learning. arXiv preprint arXiv:1811.07004, 2018.
[34] Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwinska,
Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework
for continual learning. In Proceedings of the International Conference on Machine Learning,
2018.
[35] Richard S Sutton. Introduction: The challenge of reinforcement learning. In Reinforcement
Learning, pages 1–3. Springer, 1992.
[36] Richard S Sutton. The reward hypothesis, 2004. URL http://incompleteideas.net/rlai.
cs.ualberta.ca/RLAI/rewardhypothesis.html.
[37] Richard S Sutton. The quest for a common model of the intelligent decision maker. arXiv
preprint arXiv:2202.13252, 2022.
[38] Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement learning:
a hierarchical Bayesian approach. In Proceedings of the International Conference on Machine
learning, 2007.

11
A Notation
We first provide a table summarizing all relevant notation.

Notation Meaning Definition

𝒜 Actions
𝒪 Observations

ℋ𝑡 Length 𝑡 histories ℋ𝑡 = (𝒜 × 𝒪)𝑡

ℋ ℋ= ∞ 𝑡=0 ℋ𝑡
Ð
All histories
ℎ A history ℎ∈ℋ
ℎ ℎ′ History concatenation
ℎ𝑡 Length 𝑡 history ℎ 𝑡 ∈ ℋ𝑡
ℋ̄ = ∞
Î𝑡−1
ℋ̄ = ℋ 𝜆,𝑒 𝑡=0 ℎ 𝑡 ∈ ℋ𝑡 : 𝑘=0 𝑒(𝑜 𝑘 | ℎ 𝑘 , 𝑎 𝑘 )𝜆(𝑎 𝑘 | ℎ 𝑘 ) > 0
Ð 
Realizable histories
ℋ́ = ℋℎ𝜆,𝑒 Realizable history suffixes ℋ́ = {ℎ ′ ∈ ℋ : ℎ ℎ ′ ∈ ℋ 𝜆,𝑒 }

𝑒 Environment 𝑒 : ℋ × 𝒜 → Δ(𝒜)
ℰ Set of environments

𝜆 Agent 𝜆 : ℋ → Δ(𝒜)
Λ Set of all agents
Λ Set of agents Λ⊆Λ
ΛB Agent basis ΛB ⊂ Λ

𝑟 Reward function 𝑟:ℋ →R

𝑣 Performance 𝑣 : ℋ × Λ × ℰ → [vmin , vmax ]

𝜎 Learning rule 𝜎 : ℋ → ΛB
Σ Set of all learning rules
Σ Set of learning rules Σ⊆Σ

ΛB ⊢𝑒 Σ Λ Σ-generates ∀Λ ∃𝜎∈Σ∀ ℎ∈ℋ̄ 𝜆(ℎ) = 𝜎(ℎ)(ℎ)

ΛB ⊢𝑒 Λ Generates ∃Σ⊆Σ ΛB ⊢𝑒 Σ Λ
ΛB |=Σ Λ Universally Σ-generates ∀Λ ∃𝜎∈Σ∀ ℎ∈ℋ 𝜆(ℎ) = 𝜎(ℎ)(ℎ)
ΛB |= Λ Universally generates operator ∃Σ⊆Σ ΛB |=Σ Λ

𝜆⇝ 𝑒
ΛB Sometimes reaches ∃ ℎ∈ℋ̄ ∃𝜆B ∈ΛB ∀ ℎ′ ∈ℋ́ 𝜆(ℎ ℎ ′) = 𝜆B (ℎ ℎ ′)
𝑒 𝑒
𝜆⇝ / ΛB Never reaches ¬(𝜆 ⇝ ΛB )
𝜆 □⇝ 𝑒
ΛB Always reaches ∀ ℎ∈ℋ̄ ∃𝑡∈N0 ∀ ℎ ◦ ∈ℋ́𝑡:∞ ∃𝜆B ∈ΛB ∀ ℎ′ ∈ℋ́ 𝜆(ℎ ℎ ◦ ℎ ′) = 𝜆B (ℎ ℎ ◦ ℎ ′)

Table 1: A summary of notation.

B Proofs of Presented Results

We next provide proofs of each result from the paper. Our proofs make use of some extra notation:
we use ⇒ as logical implication, and we use 𝒫(𝒳) to denote the power set of any set 𝒳. Lastly, we
use ∀𝒜⊆𝒳 and ∃𝒜⊆𝒳 as shorthand for ∀𝒜∈𝒫(𝒳) and ∃𝒜∈𝒫(𝒳) respectively.

12
B.1 Section 3 Proofs

Our first result is from Section 3 of the paper.

Theorem 3.1. For any pair (𝜆, 𝑒), there exists infinitely many choices of a basis, ΛB , such that both
(1) 𝜆 ∉ ΛB , and (2) ΛB ⊢𝑒 {𝜆}.

Proof of Theorem 3.1.

Choose a fixed but arbitrary pair (𝜆, 𝑒). Then, enumerate the realizable histories, ℋ 𝜆,𝑒 , and
let ℎ 1 denote the first element of this enumeration, ℎ 2 the second, and so on.
Then, we design a constructive procedure for a basis that, when repeatedly applied, induces
an infinite enumeration of bases that satisfy the desired two properties. This constructive
procedure for the 𝑘-th basis will contain 𝑘 + 1 agents, where each agent is distinct from 𝜆,
but will produces the same action as the agent every 𝑘 + 1 elements of the history sequence,
ℎ 1 , ℎ 2 , . . ..
For the first (𝑘 = 1) basis, we construct two agents. The first, 𝜆1B , chooses the same action
distribution as 𝜆 on each even numbered history: 𝜆1B (ℎ 𝑖 ) = 𝜆(ℎ 𝑖 ). Then, this agent will choose
a different action distribution on the odd length histories: 𝜆1B (ℎ 𝑖+1 ) ≠ 𝜆(ℎ 𝑖+1 ), for 𝑖 any even
natural number. The second agent, 𝜆2B will do the opposite to 𝜆1B : on each odd numbered
history ℎ 𝑖+1 , 𝜆2B (ℎ 𝑖+1 ) ≠ 𝜆(ℎ 𝑖+1 ), but on every even numbered history, 𝜆2B (ℎ 𝑖 ) = 𝜆(ℎ 𝑖 ).
Observe first that by construction, 𝜆 ≠ 𝜆1B , and 𝜆 ≠ 𝜆2B , since there exist histories where they
choose different action distributions. Next, observe that the basis, ΛB = {𝜆1B , 𝜆2B }, generates
{𝜆} in 𝑒 through the following set of learning rules, Σ: given any realizable history, ℎ ∈ ℋ 𝜆,𝑒 ,
check whether the history has an even or odd numbered index in the enumeration. If odd,
choose 𝜆1B , and if even, choose 𝜆2B .
More generally, this procedure can be applied for any 𝑘:
[ℎ] == 𝑖,

𝜆(ℎ)
ΛB𝑘 = {𝜆1B , . . . , 𝜆B𝑘+1 }, 𝜆B𝑖 (ℎ) = (B.1)
≠ 𝜆(ℎ) otherwise,

where we use the notation [ℎ] == 𝑖 to express the logical predicate asserting that the modulos
of the index of ℎ in the enumeration ℎ 1 , ℎ 2 , . . . is equal to 𝑖.
Further, ≠ 𝜆(ℎ) simply refers to any choice of action distribution that is unequal to 𝜆(ℎ).
Thus, for all natural numbers 𝑘 ≥ 2, we can construct a new basis consisting of 𝑘 base agents
that generates 𝜆 in 𝑒, but does not contain the agent itself. This completes the argument. □

B.2 Section 5 Proofs

Our result is from Section 5 of the paper establishing basic properties of CRL.
Theorem 5.1. Every instance of CRL (𝑒 , 𝑣, Λ, ΛB ) satisfies the following properties:

1. There exists a Λ′B such that (1) Λ′B ⊢𝑒 Λ, and (2) (𝑒 , 𝑣, Λ, Λ′B ) is not an instance of CRL.
2. ΛB is not a superset of Λ.
3. Λ ⊂ Λ.
4. If |Λ| is finite, there exists an agent set, Λ◦ such that |Λ◦ | < |Λ| and Λ◦ ⊢𝑒 Λ.
5. If |Λ| is infinite, there exists an agent set, Λ◦ such that Λ◦ ⊂ Λ and Λ◦ ⊢𝑒 Λ.

We prove this result in the form of four lemmas, corresponding to each of the five points of the
theorem (with the fourth lemma, Lemma B.4, covering points 4. and 5.). Some of the lemmas make
use of properties of generates and reaches that we establish later in Appendix C.

13
Lemma B.1. For all instances of CRL (𝑒 , 𝑣, Λ, ΛB ), there exists a choice Λ′B such that (1) Λ′B ⊢𝑒 Λ,
and (2) (𝑒 , 𝑣, Λ, Λ′B ) is not an instance of CRL.

Proof of Lemma B.1.

Recall that a tuple (𝑒 , 𝑣, Λ, ΛB ) is CRL just when all of the optimal agents Λ∗ do not reach
the basis. Then, the result holds as a straightforward consequence of two facts. First, we
can always construct a new basis containing all of the optimal agents, Λ◦B = ΛB ∪ Λ∗ . By
property three of Theorem C.2, it follows that Λ◦B ⊢𝑒 Λ. Second, by Proposition C.22, we
know that every element 𝜆◦B ∈ Λ◦B will always reach the basis, 𝜆◦B □⇝ 𝑒
Λ◦B . Therefore, in the
◦
tuple (𝑒 , 𝑣, Λ, ΛB ), each of the optimal agents will reach the basis, and therefore this is not an
instance of CRL. □

Lemma B.2. Every instance of CRL must involve an agent basis ΛB that is not a superset of Λ. That
is, any tuple (𝑒 , 𝑣, Λ, ΛB ) such that Λ ⊆ ΛB is not an instance of CRL.

Proof of Lemma B.2.

Recall that by Proposition C.22, we know that any agent 𝜆 in a set Λ also always reaches that
𝑒
set, 𝜆 □⇝ Λ. Thus, if Λ ⊆ ΛB , we know that every agent 𝜆 ∈ Λ always reaches the basis,
𝑒
𝜆 □⇝ ΛB .
Since the set of optimal agents,
Λ∗ = arg max 𝑣(𝜆, 𝑒),
𝜆∈Λ

is a subset of Λ, it follows that every optimal agent 𝜆∗ ∈ Λ∗ also satisfies 𝜆∗ □⇝

𝑒
ΛB . But this
directly violates the condition of CRL: recall that the definition of CRL requires that every
optimal agent never reaches the given basis. We conclude that every tuple (𝑒 , 𝑣, Λ, ΛB ) where
Λ ⊆ ΛB is not an instance of CRL. □

Lemma B.3. For any instance of CRL, Λ ⊂ Λ.

Proof of Lemma B.3.

We proceed toward contradiction and suppose that we have an instance of CRL (𝑒 , 𝑣, Λ, ΛB )

where Λ = Λ. But, by Proposition C.24, we know that all agents with always reach Λ, and
therefore, every optimal agent 𝜆∗ ∈ Λ∗ satisfies 𝜆 □⇝
𝑒
Λ. This directly violates the definition
of CRL, and we conclude. □

Before stating the next lemma, we note that points (4.) and (5.) of Theorem 5.1 are simply expansions
of the definition of a minimal agent set, which we define precisely in Definition C.4 and Definition C.5.
Lemma B.4. For any instance of CRL, Λ is not minimal.

Proof of Lemma B.4.

14
 We first show that Λ cannot be minimal. To do so, we consider the cases where the rank
(Definition C.3) of Λ is finite and infinite separately.

(Finite Rank Λ.)

If rank(Λ) is finite and minimal, then it follows immediately that there is no agent set of
smaller rank that generates Λ. By consequence, since ΛB ⊂ Λ and ΛB ⊢𝑒 Λ, we conclude that
Λ cannot be minimal. ✓

(Infinite Rank Λ.)

If rank(Λ) is infinite and minimal, then there is no proper subset of Λ that uniformly generates
Λ by definition. By consequence, since ΛB ⊂ Λ and ΛB ⊢𝑒 Λ, we conclude that Λ cannot be
minimal. ✓

This completes the argument of both cases, and we conclude that for any instance of CRL, Λ
is not minimal. □

C Additional Analysis
Finally, we present a variety of additional results about agents and the generates and reaches operators.

C.1 Analysis: Generates

We first highlight simple properties of the generates operator. Many of our results build around the
notion of uniform generation, a variant of the generates operator in which a basis generates an agent
set in every environment. We define this operator precisely as follows.
Definition C.1. Let Σ be a set of learning rules over some basis ΛB . We say that a set Λ is uniformly
𝚺-generated by ΛB , denoted ΛB |=Σ Λ, just when
∀𝜆∈Λ ∃𝜎∈Σ ∀ ℎ∈ℋ 𝜆(ℎ) = 𝜎(ℎ)(ℎ). (C.1)
Definition C.2. We say a basis ΛB uniformly generates Λ, denoted ΛB |= Λ, if
∃Σ⊆Σ ΛB |=Σ Λ. (C.2)

We will first show that uniform generation entails generation in a particular environment. As a
consequence, when we prove that certain properties hold of uniform generation, we can typically also
conclude that the properties hold for generation as well, though there is some subtlety as to when
exactly this implication will allow results about |= to apply directly to ⊢𝑒 .
Proposition C.1. For any (ΛB , Λ) pair, if ΛB |= Λ, then for all 𝑒 ∈ ℰ, ΛB ⊢𝑒 Λ.

Proof of Proposition C.1.

Recall that in the definition of uniform generation, ΛB |= Λ, we require,

∃Σ⊆Σ∀𝜆∈Λ ∃𝜎∈Σ ∀ ℎ∈ℋ 𝜆(ℎ) = 𝜎(ℎ)(ℎ). (C.3)
Now, contrast this with generates with respect to a specific environment 𝑒,
∃Σ⊆Σ∀𝜆∈Λ ∃𝜎∈Σ ∀ ℎ∈ℋ̄ 𝜆(ℎ) = 𝜎(ℎ)(ℎ). (C.4)
The only difference in the definitions is that the set of histories quantified over is ℋ in the
former, and ℋ̄ = ℋ 𝜆,𝑒 in the latter.
Since ℋ̄ ⊆ ℋ for any choice of environment 𝑒, we can conclude that when Equation C.3, it
is also the case that Equation C.4 holds, too. Therefore, ΛB |= Λ ⇒ ΛB ⊢𝑒 Λ for any 𝑒. □

15
We next summarize five properties of the generates operator.
Theorem C.2. The following properties hold of the generates operator:

1. Generates is transitive: For any triple (Λ1 , Λ2 , Λ3 ) and 𝑒 ∈ ℰ, if Λ1 ⊢𝑒 Λ2 and Λ2 ⊢𝑒 Λ3 ,

then Λ1 ⊢𝑒 Λ3 .
2. Generates is not commutative: there exists a pair (Λ1 , Λ2 ) and 𝑒 ∈ ℰ such that Λ1 ⊢𝑒 Λ2 ,
but ¬(Λ2 ⊢𝑒 Λ1 ).
3. For all Λ and pair of agent bases (Λ1B , Λ2B ) such that Λ1B ⊆ Λ2B , if Λ1B ⊢𝑒 Λ, then Λ2B ⊢𝑒 Λ.
4. For all Λ and 𝑒 ∈ ℰ, Λ ⊢𝑒 Λ.
5. The decision problem, Given (𝑒 , ΛB , Λ), output True iff ΛB ⊢𝑒 Λ, is undecidable.

The proof of this theorem is spread across the next five lemmas below.
The fact that generates is transitive suggests that the basic tools of an agent set—paired with a set
of learning rules—might be likened to an algebraic structure. We can draw a symmetry between an
agent basis and the basis of a vector space: A vector space is comprised of all linear combinations
of the basis, whereas Λ is comprised of all valid switches (according to the learning rules) between
the base agents. However, the fact that generates is not commutative (by point 2.) raises a natural
question: are there choices of learning rules under which generates is commutative? An interesting
direction for future work is to explore this style of algebraic analysis on agents.
Lemma C.3. Generates is transitive: For any triple (Λ1 , Λ2 , Λ3 ) and 𝑒 ∈ ℰ, if Λ1 ⊢𝑒 Λ2 and
Λ2 ⊢𝑒 Λ3 , then Λ1 ⊢𝑒 Λ3 .

Proof of Lemma C.3.

Assume Λ1 ⊢𝑒 Λ2 and Λ2 ⊢𝑒 Λ3 . Then, by Proposition C.10 and the definition of the generates
operator, we know that
∀𝜆2 ∈Λ2 ∃𝜎1 ∈Σ1 ∀ ℎ∈ℋ̄ 𝜆2 (ℎ) = 𝜎1 (ℎ)(ℎ), (C.5)
∀𝜆3 ∈Λ3 ∃𝜎2 ∈Σ2 ∀ ℎ∈ℋ̄ 𝜆 (ℎ) = 𝜎 (ℎ)(ℎ),
3 2
(C.6)
where Σ1 and Σ2 express the set of all learning rules over Λ1 and Λ2 respectively. By definition
of a learning rule, 𝜎, we rewrite the above as follows,
∀𝜆2 ∈Λ2 ∀ ℎ∈ℋ̄ ∃𝜆1 ∈Λ1 𝜆2 (ℎ) = 𝜆1 (ℎ), (C.7)
∀𝜆3 ∈Λ3 ∀ ℎ∈ℋ̄ ∃𝜆2 ∈Λ2 𝜆3 (ℎ) = 𝜆2 (ℎ). (C.8)

Then, consider a fixed but arbitrary 𝜆3 ∈ Λ3 . We construct a learning rule defined over Λ1
as 𝜎 1 : ℋ → Λ1 that induces an equivalent agent as follows. For each realizable history,
ℎ ∈ ℋ̄ , by Equation C.8 we know that there is an 𝜆2 such that 𝜆3 (ℎ) = 𝜆2 (ℎ), and by
Equation C.7, there is an 𝜆1 such that 𝜆2 (ℎ) = 𝜆1 (ℎ). Then, set 𝜎1 : ℎ ↦→ 𝜆1 such that
𝜆1 (ℎ) = 𝜆2 (ℎ) = 𝜆3 (ℎ)
Since ℎ and 𝜆3 were chosen arbitrarily, we conclude that
∀𝜆3 ∈Λ3 ∀ ℎ∈ℋ̄ ∃𝜆1 ∈Λ1 𝜆3 (ℎ) = 𝜆1 (ℎ).
But, by the definition of Σ, this means there exists a learning rule such that
∀𝜆3 ∈Λ3 ∃𝜎1 ∈Σ1 ∀ ℎ∈ℋ̄ 𝜆3 (ℎ) = 𝜎1 (ℎ)(ℎ).
This is exactly the definition of Σ-generation, and by Proposition C.10, we conclude Λ1 ⊢𝑒
Λ3 . □

Lemma C.4. Generates is not commutative: there exists a pair (Λ1 , Λ2 ) and 𝑒 ∈ ℰ such that
Λ1 ⊢𝑒 Λ2 , but ¬(Λ2 ⊢𝑒 Λ1 ).

16
Proof of Lemma C.4.

The result follows from a simple counterexample: consider the pair

Λ1 = {𝜆 𝑖 : ℎ ↦→ 𝑎1 }, Λ2 = {𝜆 𝑖 : ℎ ↦→ 𝑎1 , 𝜆 𝑗 : ℎ ↦→ 𝑎2 }.

Note that since 𝜆 𝑖 is in both sets, and Λ1 is a singleton, we know that Λ2 ⊢𝑒 Λ1 in any
environment. But, by Proposition C.12, we know that Λ1 cannot generate Λ2 . □

Lemma C.5. For all Λ and pair of agent bases (Λ1B , Λ2B ) such that Λ1B ⊆ Λ2B , if Λ1B ⊢𝑒 Λ, then
Λ2B ⊢𝑒 Λ.

Proof of Lemma C.5.

The result follows as a natural consequence of the definition of generates. Recall that Λ1B ⊢𝑒 Λ
just when,
∃Σ1 ⊆Σ Λ1B ⊢𝑒 Σ1 Λ (C.9)
𝜎 1 (ℎ)
≡ ∃Σ1 ⊆Σ ∃𝜎1 ∈Σ1 ∀ ℎ∈ℋ̄ 𝜆(ℎ) = 𝜆B (ℎ), (C.10)
𝜎 1 (ℎ)
where again 𝜆B ∈ Λ1B is the base agent chosen by 𝜎1 (ℎ). We use superscripts Σ1 and 𝜎 1 to
signify that 𝜎1 is defined relative to Λ1B , that is, 𝜎1 : ℋ → Λ1B ∈ Σ1 .
But, since Λ1B ⊆ Λ2B , we can define Σ2 = Σ1 and ensure that Λ2B ⊢𝑒 Σ2 Λ, since the agent basis
Λ1B was already sufficient to generate Λ. Therefore, we conclude that Λ2B ⊢𝑒 Λ. □

Lemma C.6. For all Λ and 𝑒 ∈ ℰ, Λ ⊢𝑒 Λ.

Proof of Lemma C.6.

This is a direct consequence of Proposition C.24. □

Lemma C.7. The decision problem, AGENTS G ENERATE, Given (𝑒 , ΛB , Λ), output True iff ΛB ⊢𝑒 Λ,
is undecidable.

Proof of Lemma C.7.

We proceed as is typical of such results by reducing AGENTS G ENERATE from the Halting
Problem.
In particular, let 𝑚 be a fixed but arbitrary Turing Machine, and 𝑤 be a fixed but arbitrary
input to be given to machine 𝑚. Then, H ALT defines the decision problem that outputs True
iff 𝑚 halts on input 𝑤.
We construct an oracle for AGENTS G ENERATE that can decide H ALT as follows. Let (𝒜, 𝒪)
be an interface where the observation space is comprised of all configurations of machine 𝑚.
Then, we consider a deterministic environment 𝑒 that simply produces the next configuration
of 𝑚 when run on input 𝑤, based on the current tape contents, the state of 𝑚, and the location
of the tape head. Note that all three of these elements are contained in a Turing Machine’s
configuration, and that a single configuration indicates whether the Turing Machine is in a
halting state or not. Now, let the action space 𝒜 consist of two actions, {𝑎no-op , 𝑎halt }. On
execution of 𝑎 no-op no-op, the environment moves to the next configuration. On execution of
𝑎halt , the machine halts. That is, we restrict ourselves to the singleton agent set, Λ, containing
the agent 𝜆◦ that outputs 𝑎halt directly following the machine entering a halting configuration,

17
 and 𝑎 no-op otherwise:

𝑎halt 𝑜 is a halting configiration,


◦
𝜆 : ℎ𝑎𝑜 ↦→ , Λ = {𝜆◦ }.
𝑎no-op otherwise.

Using these ingredients, we take any instance of H ALT, (𝑚, 𝑤), and consider the singleton
agent basis: Λ1B = {𝑎 no-op }.
We make one query to our AGENTS G ENERATE oracle, and ask: Λ1B ⊢𝑒 Λ. If it is True, then
the histories realizable by (𝜆◦ , 𝑒) pair ensure that the single agent in Λ never emits the 𝑎halt
action, and thus, 𝑚 does not halt on 𝑤. If it is False, then there are realizable histories in 𝑒 in
which 𝑚 halts on 𝑤. We thus use the oracle’s response directly to decide the given instance of
H ALT. □

We find many similar properties hold for reaches, which we present in the next subsection in
Theorem C.25.
We next show that the subset relation implies generation.
Proposition C.8. Any pair of agent sets (Λsmall , Λbig ) such that Λsmall ⊆ Λbig satisfies
Λbig |= Λsmall . (C.11)

Proof of Proposition C.8.

The result follows from the combination of two facts. First, that all agent sets generate
themselves. That is, for arbitrary Λ, we know that Λ ⊢𝑒 Λ, since the trivial set of learning rules,
Σtr = {𝜎𝑖 : ℎ ↦→ 𝜆 𝑖 , ∀𝜆𝑖 ∈Λ }, (C.12)
that never switches between agents is sufficient to generate the agent set.
Second, observe that removing an agent from the generated set has no effect on the generates
operator. That is, let Λ′ = Λ \ 𝜆, for fixed but arbitrary 𝜆 ∈ Λ. We see that Λ ⊢𝑒 Λ′, since Σtr
is sufficient to generate Λ′, too. By inducting over all removals of agents from Λ, we reach
our conclusion. □

Next, we establish properties about the sets of learning rules that correspond to the generates operator.
Proposition C.9. For any (ΛB , Σ, Λ) such that ΛB |=Σ Λ, it holds that
|Λ| ≤ |Σ|. (C.13)

Proof of Proposition C.9.

We proceed toward contradiction, and assume |Λ| > |Σ|. Then, there is at least one learning
rule 𝜎 ∈ Σ that corresponds to two or more distinct agents in Λ. Call this element 𝜎◦ , and
without loss of generality let 𝜆1 and 𝜆2 be two distinct agents that are each generated by 𝜎◦ in
the sense that,
𝜆1 (ℎ) = 𝜎◦ (ℎ)(ℎ), 𝜆2 (ℎ) = 𝜎◦ (ℎ)(ℎ), (C.14)
for every ℎ ∈ ℋ . But, by the distinctness of 𝜆 and 𝜆 , there must exist a history ℎ in
1 2
which 𝜆1 (ℎ) ≠ 𝜆2 (ℎ). We now arrive at a contradiction as such a history cannot exist: By
Equation C.14, we know that 𝜆1 (ℎ) = 𝜎 ◦ (ℎ)(ℎ) = 𝜆2 (ℎ) for all ℎ. □

We see that the universal learning rules, Σ, is the strongest in the following sense.
Proposition C.10. For any basis ΛB and agent set Λ, exactly one of the two following properties
hold:

1. The agent basis ΛB uniformly generates Λ under the set of all learning rules: ΛB |=Σ Λ.

18
 2. There is no set of learning rules for which the basis Σ-uniformly generates the agent set:
¬∃Σ⊆Σ ΛB |=Σ Λ.

Proof of Proposition C.10.

The proof follows from the law of excluded middle. That is, for any set of learning rules
Σ, either it generates Λ or it does not. If it does generate Λ, by Lemma C.5 so does Σ. By
consequence, if Σ does not generate Λ, neither do any of its subsets. □

Furthermore, uniform generation is also transitive.

Theorem C.11. Uniform generates is transitive: For any triple (Λ1 , Λ2 , Λ3 ), if Λ1 |= Λ2 and
Λ2 |= Λ3 , then Λ1 |= Λ3 .

Proof of Theorem C.11.

Assume Λ1 |= Λ2 and Λ2 |= Λ3 . Then, by Proposition C.10 and the definition of the uniform
generates operator, we know that
∀𝜆2 ∈Λ2 ∃𝜎1 ∈Σ1 ∀ ℎ∈ℋ 𝜆2 (ℎ) = 𝜎1 (ℎ)(ℎ), (C.15)
∀𝜆3 ∈Λ3 ∃𝜎2 ∈Σ2 ∀ ℎ∈ℋ 𝜆 (ℎ) = 𝜎 (ℎ)(ℎ),
3 2
(C.16)
where Σ1 and Σ2 express the set of all learning rules over Λ1 and Λ2 respectively. By definition
of a learning rule, 𝜎, we rewrite the above as follows,
∀𝜆2 ∈Λ2 ∀ ℎ∈ℋ ∃𝜆1 ∈Λ1 𝜆2 (ℎ) = 𝜆1 (ℎ), (C.17)
∀𝜆3 ∈Λ3 ∀ ℎ∈ℋ ∃𝜆2 ∈Λ2 𝜆 (ℎ) = 𝜆 (ℎ).
3 2
(C.18)

Then, consider a fixed but arbitrary 𝜆3 ∈ Λ3 . We construct a learning rule defined over Λ1
as 𝜎1 : ℋ → Λ1 that induces an equivalent agent as follows. For each history, ℎ ∈ ℋ , by
Equation C.18 we know that there is an 𝜆2 such that 𝜆3 (ℎ) = 𝜆2 (ℎ), and by Equation C.17,
there is an 𝜆1 such that 𝜆2 (ℎ) = 𝜆1 (ℎ). Then, set 𝜎1 : ℎ ↦→ 𝜆1 such that 𝜆1 (ℎ) = 𝜆2 (ℎ) =
𝜆3 (ℎ). Since ℎ and 𝜆3 were chosen arbitrarily, we conclude that
∀𝜆3 ∈Λ3 ∀ ℎ∈ℋ ∃𝜆1 ∈Λ1 𝜆3 (ℎ) = 𝜆1 (ℎ).
But, by the definition of Σ, this means there exists a learning rule such that
∀𝜆3 ∈Λ3 ∃𝜎1 ∈Σ1 ∀ ℎ∈ℋ 𝜆3 (ℎ) = 𝜎1 (ℎ)(ℎ).
This is exactly the definition of Σ-uniform generation, and by Proposition C.10, we conclude
Λ1 |= Λ3 . □

Next, we show that a singleton basis only generates itself.

Proposition C.12. Any singleton basis, ΛB = {𝜆}, only uniformly generates itself.

Proof of Proposition C.12.

Note that generates requires switching between base agents. With only a single agent, there
cannot be any switching, and thus, the only agent that can be described as switching amongst
the elements of the singleton set ΛB = {𝜆} is the set itself. □

C.1.1 Rank and Minimal Bases

As discussed in the paper, one natural reaction to the concept of an agent basis is to ask how we can
justify different choices of a basis. And, if we cannot, then perhaps the concept of an agent basis

19
is disruptive, rather than illuminating. We suggest that in many situations the choice is quite clear.
However, there are some objective properties of different bases that can help us to evaluate possible
choices of a suitable basis. For instance, some bases are minimal in the sense that they cannot be
made smaller while still retaining the same expressive power (that is, while generating the same agent
sets). Identifying such minimal sets may be useful, as there is good reason to consider only the most
compressed agent bases.
To make these intuitions concrete, we introduce the rank of an agent set.
Definition C.3. The rank of an agent set, rank(Λ), is the size of the smallest agent basis that
uniformly generates it:
rank(Λ) = min |ΛB | s.t. ΛB |= Λ. (C.19)
ΛB ⊂ Λ

For example, the agent set,

Λ = {𝜆0 : ℎ ↦→ 𝑎0 , (C.20)
𝜆 : ℎ ↦→ 𝑎1 ,
1

𝑎0 | ℎ| mod 2 = 0,

𝜆 : ℎ ↦→
2
𝑎1 | ℎ| mod 2 = 1,
},
has rank(Λ) = 2, since the basis,
ΛB = {𝜆0B : ℎ ↦→ 𝑎0 , 𝜆1B : ℎ ↦→ 𝑎1 },
uniformly generates Λ, and there is no size-one basis that uniformly generates Λ by Proposition C.12.
Using the notion of an agent set’s rank, we now introduce the concept of a minimal basis. We suggest
that minimal bases are particular important, as they contain no redundancy with respect to their
expressive power. Concretely, we define a minimal basis in two slightly different ways depending on
whether the basis has finite or infinite rank. In the finite case, we say a basis is minimal if there is no
basis of lower rank that generates it.
Definition C.4. An agent basis ΛB with finite rank is said to be minimal just when there is no smaller
basis that generates it,
∀Λ′B ⊂Λ Λ′B |= ΛB ⇒ rank(Λ′B ) ≥ rank(ΛB ). (C.21)

In the infinite case, as all infinite rank bases will have the same effective size, we instead consider a
notion of minimiality based on whether any elements can be removed from the basis without changing
its expressive power.
Definition C.5. An agent basis ΛB with infinite rank is said to be minimal just when no proper
subset of ΛB uniformly generates ΛB .
∀Λ′B ⊆ΛB Λ′B |= ΛB ⇒ Λ′B = ΛB . (C.22)

Notably, this way of looking at minimal bases will also apply to finite rank agent bases as a direct
consequence of the definition of a minimal finite rank basis. However, we still provide both definitions,
as a finite rank basis may not contain a subset that generates it, but there may exist a lower rank basis
that generates it.
Corollary C.13. As a Corollary of Proposition C.8 and Definition C.4, for any minimal agent basis
ΛB , there is no proper subset of ΛB that generates ΛB .
Regardless of whether an agent basis has finite or infinite rank, we say the basis is a minimal basis of
an agent set Λ just when the basis uniformly generates Λ and the basis is minimal.
Definition C.6. For any Λ, a minimal basis of 𝚲 is any basis ΛB that is both (1) minimal, and (2)
ΛB |= Λ.
A natural question arises as to whether the minimal basis of any agent set Λ is unique. We answer
this question in the negative.
Proposition C.14. The minimal basis of a set of agents is not necessarily unique.

Proof of Proposition C.14.

20
 To prove the claim, we construct an instance of an agent set with two distinct minimal bases.
Let 𝒜 = {𝑎 0 , 𝑎1 }, and 𝒪 = {𝑜0 }. We consider the agent set containing four agents. The first
two map every history to 𝑎0 and 𝑎1 , respectively, while the second two alternate between 𝑎0
and 𝑎 1 depending on whether the history is of odd or even length:
Λ = {𝜆0 : ℎ ↦→ 𝑎0 , (C.23)
𝜆 : ℎ ↦→ 𝑎1 ,
1

𝑎0 | ℎ| mod 2 = 0,

𝜆2 : ℎ ↦→
𝑎1 | ℎ| mod 2 = 1,
𝑎0 |ℎ| mod 2 = 1,

𝜆 : ℎ ↦→
3
𝑎1 |ℎ| mod 2 = 0,
}.

Note that there are two distinct subsets that each universally generate Λ:
Λ0,1
B = {𝜆B , 𝜆B },
0 1
Λ2,3
B = {𝜆B , 𝜆B }.
2 3
(C.24)

Next notice that there cannot be a singleton basis by Proposition C.12, and thus, both Λ0,1
B and
Λ2,3
B satisfy (1) |Λ 0,1
B | = |Λ2,3
B | = rank(Λ), and (2) both Λ 0,1
B |= Λ, Λ 2,3
B |= Λ. □

Beyond the lack of redundancy of a basis, we may also be interested in their expressive power. For
instance, if we compare two minimal bases, Λ1B and Λ2B , how might we justify which to use? To
address this question, we consider another desirable property of a basis: universality.
Definition C.7. An agent basis ΛB is universal if ΛB |= Λ.
Clearly, it might be desirable to work with a universal basis, as doing so ensures that the set of agents
we consider in our design space is as rich as possible. We next show that there is at least one natural
basis that is both minimal and universal.
Proposition C.15. The basis,
Λ◦B = {𝜆 : 𝒪 → Δ(𝒜)}, (C.25)
is a minimal universal basis:
1. Λ◦B |= Λ: The basis uniformly generates the set of all agents.
2. Λ◦B is minimal.

Proof of Proposition C.15.

We prove each property separately.

1. Λ◦B |= Λ

First, we show that the basis is universal: Λ◦B |= Λ. Recall that this amounts to showing that,
∀𝜆∈Λ∀ ℎ∈ℋ ∃𝜆′ ∈Λ◦B 𝜆(ℎ) = 𝜆′(ℎ). (C.26)
Let 𝜆 ∈ Λ and ℎ ∈ ℋ be fixed but arbitrary. Now, let us label the action distribution produced
by 𝜆(ℎ) as 𝑝𝜆(ℎ) . Let 𝑜 refer to the last observation contained in ℎ (or ∅ if ℎ = ℎ0 = ∅). Now,
construct the agent 𝜆◦B : 𝑜 ↦→ 𝑝𝜆(ℎ) . By construction of Λ◦B , this agent is guaranteed to be a
member of Λ◦B , and furthermore, we know that 𝜆◦B produces the same output as 𝜆 on ℎ. Since
both 𝜆 and ℎ were chosen arbitrarily, the construction will work for any choice of 𝜆 and ℎ,
and we conclude that at every history, there exists a basis agent Λ◦B ∈ Λ◦B that produces the
same probability distribution over actions as any given agent. Thus, the first property holds. ✓

21
 2. Λ◦B is minimal.

Second, we show that Λ◦B is a minimal basis of Λ. Recall that since rank(Λ◦B ) = ∞, the
definition of a minimal basis means that:
∀ΛB ⊆Λ◦B ΛB |= Λ ⇒ ΛB = Λ◦B . (C.27)

To do so, fix an arbitrary proper subset of ΛB ∈ 𝒫(Λ◦B ). Notice that since ΛB is a proper
subset, there exists a non-empty set ΛB such that,

ΛB ∪ ΛB = Λ◦B .

Now, we show that ΛB cannot uniformly generate Λ by constructing an agent from ΛB . In

particular, consider the first element of ΛB , which, by construction of Λ◦B , is some mapping
from 𝒪 to a choice of probability distribution over 𝒜. Let us refer to this agent’s output
probability distribution over actions as 𝑝. Notice that there cannot exist an agent in Λ◦B that
chooses 𝑝, otherwise ΛB would not be a proper subset of Λ◦B . Notice further that in the set of
all agents, there are infinitely many agents that output 𝑝 in at least one history. We conclude
that ΛB cannot uniformly generate Λ, as it does not contain any base element that produces 𝑝.
The set ΛB was chosen arbitrarily, and thus the claim holds for any proper subset of Λ◦B , and
we conclude. ✓

This completes the proof of both statements. □

We next highlight two corollaries related to properties of a basis.

Corollary C.16. As a direct consequence of Proposition C.15, every universal basis has infinite rank.
Corollary C.17. As a consequence of point 1 and point 3 of Theorem 5.1, for every instance of CRL
(𝑒 , 𝑣, Λ, ΛB ), at least one of the two properties must hold:

1. ΛB is not universal,
2. The set of learning rules, Σ, that ensures ΛB ⊢𝑒 Σ Λ, satisfies Σ ⊂ Σ.

The second corollary highlights a further connection between CRL and bounded agents—the agents
of interest in any CRL problem must either implicitly (1) search over a constrained set of behaviors,
or (2) make use of a constrained set of learning rules.

C.1.2 Orthogonal and Parallel Agent Sets

Drawing inspiration from vector spaces, we introduce notions of orthogonal and parallel agent bases
according to the agent sets they generate.
Definition C.8. A pair of agent bases (Λ1B , Λ2B ) are orthogonal if any pair (Λ1 , Λ2 ) they each
uniformly generate
Λ1B |= Λ1 , Λ2B |= Λ2 , (C.28)
satisfy
Λ1 ∩ Λ2 = ∅. (C.29)

Naturally this definition can be modified to account for environment-relative generation (⊢𝑒 ), or to be
defined with respect to a particular set of learning rules in which case two bases are orthogonal with
respect to the learning rule set just when they generate different agent sets under the given learning
rules. As with the variants of the two operators, we believe the details of such formalisms are easy to
produce.
A few properties hold of any pair of orthogonal bases.
Proposition C.18. If two bases Λ1B , Λ2B are orthogonal, then the following properties hold:

22
 1. Λ1B ∩ Λ2B = ∅.

2. Neither Λ1B nor Λ2B are universal.

Proof of Proposition C.18.

We prove each property independently.

1. Λ1B ∩ Λ2B = ∅

We proceed toward contradiction. That is, suppose that both Λ1B is orthogonal to Λ2B , and that
Λ1B ∩ Λ2B ≠ ∅. Then, by the latter property, there is at least one agent that is an element of
both bases. Call this agent 𝜆◦B . It follows that the set Λ◦B = {𝜆◦B } is a subset of both Λ1B and
Λ2B . By Proposition C.8, it follows that Λ1B |= Λ◦B and Λ2B |= Λ◦B . But this contradicts the fact
that Λ1B is orthogonal to Λ2B , and so we conclude. ✓

2. Neither Λ1B nor Λ2B are universal.

We again proceed toward contradiction. Suppose without loss of generality that Λ1B is
universal. Then, we know Λ1B |= Λ. Now, we consider two cases: either Λ2B generates some
non-empty set, Λ2 , or it does not generate any sets. If it generates a set Λ2 , then we arrive at a
contradiction as Λ2 ∩ Λ ≠ ∅, which violates the definition of orthogonal bases. If if does not
generate a set, this violates the definition of a basis, as any basis is by construction non-empty,
and we know that containing even a single element is sufficient to generate at least one agent
set by Proposition C.12. Therefore, in either of the two cases, we arrive at a contradiction, and
thus conclude the argument. ✓

This concludes the proof of each statement. □

Corollary C.19. For any non-universal agent basis ΛB , there exists an orthogonal agent basis, Λ†B .

Conversely, two agent bases Λ1B , Λ2B are parallel just when they generate the same agent sets.
Definition C.9. A pair of agent bases (Λ1B , Λ2B ) are parallel if for every agent set Λ, Λ1B |= Λ if and
only if Λ2B |= Λ.
Proposition C.20. If two bases Λ1B , Λ2B are parallel, then the following properties hold:

1. Both Λ1B |= Λ2B and Λ2B |= Λ1B .

2. rank(Λ1B ) = rank(Λ2B ).

3. Λ1B is universal if and only if Λ2B is universal.

Proof of Proposition C.20.

We prove each property separately.

1. Both Λ1B |= Λ2B and Λ2B |= Λ1B .

The claim follows directly from the definition of parallel bases. An agent set Λ is uniformly
generated by Λ1B if and only if it is uniformly generated by Λ2B . Since by Proposition C.8 we
know both Λ1B |= Λ1B and Λ2B |= Λ2B , we conclude that both Λ1B |= Λ2B and Λ2B |= Λ1B . ✓

23
 2. rank(Λ1B ) = rank(Λ2B ).

Recall that the definition of rank refers to the size of the smallest basis that uniformly generates
it,
rank(Λ) = min |ΛB | s.t. ΛB |= Λ.
ΛB ⊂ Λ
Now, note that by property (1.) of the proposition, both sets uniformly generate each other.
Therefore, we know that
rank(Λ1B ) ≤ min |Λ1B |, |Λ2B | , rank(Λ2B ) ≤ min |Λ1B |, |Λ2B | ,
 

since the smallest set that generates each basis is no larger than the basis itself, or the other
basis.
3. Λ1B is universal if and only if Λ2B is universal.

The claim again follows by combining the definitions of universality and parallel: If Λ1B
is universal, then by definition of parallel bases, Λ2B must uniformly generate all the same
agent sets including Λ, and therefore Λ2B is universal, too. Now, if Λ1B is not universal, then it
does not uniformly generate Λ. By the definition of parallel bases, we conclude Λ2B does not
generate Λ as well. Both directions hold for each labeling of the two bases without loss of
generality, and we conclude. ✓

This completes the argument for each property, and we conclude. □

C.2 Analysis: Reaches

We now establish other properties of the reaches operator.

C.2.1 Basic Properties of Reaches

First, we show that the always reaches operator implies sometimes reaches.
𝑒 𝑒
Proposition C.21. If 𝜆 □⇝ Λ, then 𝜆 ⇝ Λ.

Proof of Proposition C.21.

𝑒
Assume 𝜆 □⇝ Λ. That is, expanding the definition of always reaches, we assume
∀ ℎ∈ℋ̄ ∃𝑡∈N0 ∀ ℎ ◦ ∈ℋ́𝑡:∞ ∃𝜆B ∈ΛB ∀ ℎ′ ∈ℋ́ 𝜆(ℎ ℎ ◦ ℎ ′) = 𝜆B (ℎ ℎ ◦ ℎ ′). (C.30)

Further recall the definition of can reach 𝜆 ⇝ ΛB is as follows

∃ ℎ∈ℋ̄ ∃𝜆B ∈ΛB ∀ ℎ′ ∈ℋ́ 𝜆(ℎ ℎ ′) = 𝜆B (ℎ ℎ ′). (C.31)

Then, the claim follows quite naturally: pick any realizable history ℎ ∈ ℋ̄ . By our initial
𝑒
assumption that 𝜆 □⇝ Λ, it follows (by Equation C.30) that there is a time 𝑡 and a realizable
◦
history suffix ℎ for which
∃𝜆B ∈ΛB ∀ ℎ′ ∈ℋ́ 𝜆(ℎ ℎ ◦ ℎ ′) = 𝜆B (ℎ ℎ ◦ ℎ ′).

By construction of ℎ ℎ ◦ ∈ ℋ́ , we know ℎ ℎ ◦ is a realizable history. Therefore, there exists a

realizable history, ℎ ∗ = ℎ ℎ ◦ , for which ∃𝜆B ∈ΛB ∀ ℎ′ ∈ℋ́ 𝜆(ℎ ∗ ℎ ′) = 𝜆B (ℎ ∗ ℎ ′) holds. But this is
exactly the definition of can reach, and therefore, we conclude the argument. □

Next, we highlight the basic fact that every agent in a set also reaches that set.
𝑒
Proposition C.22. For any agent set Λ, it holds that 𝜆 □⇝ Λ for every 𝜆 ∈ Λ.

24
Proof of Proposition C.22.

The proposition is straightforward, as any 𝜆 ∈ Λ will be equivalent to itself in behavior for all
histories. □

Corollary C.23. As a corollary of Proposition C.22, any pair of agent sets (Λsmall , Λbig ) where
Λsmall ⊆ Λbig , satisfies
𝑒
∀𝜆∈Λsmall 𝜆 □⇝ Λbig . (C.32)

Next, we highlight another straightforward property that sometimes reaches and never reaches are
simply logical negations of one another.
Proposition C.24. For any 𝑒, the set of all agents Λ (i) uniformly generates all other agent sets, and
(ii) is always reached by all agents:
(𝑖) ∀Λ⊆Λ Λ |= Λ, (C.33)
𝑒
(𝑖𝑖) ∀𝜆∈Λ 𝜆 □⇝ Λ. (C.34)

Proof of Proposition C.24.

We prove each of the two properties separately.

(i). ∀Λ⊆Λ Λ |= Λ

The property holds as a straightforward consequence of Proposition C.8: Since any set Λ is a
subset of Λ, it follows that Λ |= Λ. ✓

𝑒
(ii). ∀𝜆∈Λ 𝜆 □⇝ Λ

The property holds as a straightforward consequence of Proposition C.22: Since every agent
𝑒
satisfies 𝜆 ∈ Λ, it follows that 𝜆 ⇝ Λ. ✓

This concludes the argument of both statements. □

We next present the definition of always reaches alongside the proof of Theorem C.25.
𝑒
Definition C.10. We say agent 𝜆 ∈ Λ always reaches ΛB , denoted 𝜆 □⇝ ΛB , if
∀ ℎ∈ℋ̄ ∃𝑡∈N0 ∀ ℎ ◦ ∈ℋ́𝑡:∞ ∃𝜆B ∈ΛB ∀ ℎ′ ∈ℋ́ 𝜆(ℎ ℎ ◦ ℎ ′) = 𝜆B (ℎ ℎ ◦ ℎ ′). (C.35)

The nested quantifiers allows the agent to become equivalent to different base behaviors depending on
the evolution of the interaction stream. For example, in an environment that flips a coin to determine
whether 𝑎 heads or 𝑎tails is optimal, the 𝜆 might output 𝑎heads indefinitely if the coin is heads, but 𝑎 tails
otherwise. In this case, such an agent will still always reach the basis ΛB = {𝜆1B : ℎ ↦→ 𝑎heads , 𝜆2B :
ℎ ↦→ 𝑎tails }.
Our next theorem highlights properties of reaches.
Theorem C.25. The following properties hold of the reaches operator:
𝑒
1. ⇝ /𝑒 are not transitive, while always reaches is transitive.
and ⇝
2. Sometimes reaches is not commutative: there exists a pair (Λ1 , Λ2 ) and 𝑒 ∈ ℰ such that
𝑒
∀𝜆1 ∈Λ1 𝜆1 ⇝ /𝑒 Λ1 .
Λ2 , but ∃𝜆2 ∈Λ2 𝜆2 ⇝
𝑒
3. For all pairs (Λ, 𝑒), if 𝜆 ∈ Λ, then 𝜆 ⇝ Λ.
𝑒
4. Every agent satisfies 𝜆 ⇝ Λ in every environment.
𝑒
5. The decision problem, Given (𝑒 , 𝜆, Λ), output True iff 𝜆 ⇝ Λ, is undecidable.

25
Again, we prove this result through five lemmas that correspond to each of the above properties.
Many of these properties resemble those in Theorem C.2. For instance, point (5.) shows that deciding
whether a given agent sometimes reaches a basis in an environment is undecidable. We anticipate
that the majority of decision problems related to determining properties of arbitrary agent sets will
be undecidable, though it is still worth making these arguments carefully. Moreover, there may be
interesting special cases in which these decision problems are decidable (and perhaps, efficiently
so). Identifying these special cases and their corresponding efficient algorithms is another interesting
direction for future work.
𝑒
Lemma C.26. ⇝ and ⇝ /𝑒 are not transitive, while always reaches is transitive.

Proof of Lemma C.26.

We first show that always reaches is transitive, then point out counterexamples illustrating that
both sometimes reaches and never reaches are not transitive.

(i). Always reaches is transitive.

𝑒 𝑒
We proceed by assuming that both ∀𝜆1 ∈Λ1 𝜆1 □⇝ Λ2 and ∀𝜆2 ∈Λ2 𝜆2 □⇝ Λ3 and show that it
𝑒
must follow that ∀𝜆1 ∈Λ1 𝜆 □⇝ Λ . To do so, pick a fixed but arbitrary 𝜆1 ∈ Λ1 , and expand
1 3
𝑒
𝜆1 □⇝ Λ2 as
∀ ℎ∈ℋ̄ ∃𝑡∈N0 ∀ ℎ ◦ ∈ℋ́𝑡:∞ ∃𝜆2 ∈Λ2 ∀ ℎ′ ∈ℋ́ 𝜆1 (ℎ ℎ ◦ ℎ ′) = 𝜆2 (ℎ ℎ ◦ ℎ ′).

Now, observe that for any realizable history ℎ ℎ ◦ ℎ ′, we know that the corresponding 𝜆2 that
𝑒
produces the same action distribution as 𝜆1 also satisfies 𝜆2 □⇝ Λ3 . Thus, there must exist
some time 𝑡¯ at which, any realizable history ℎ̄ ℎ̄ , will satisfy ∃𝜆3 ∈Λ3 ∀ ℎ̄′ ∈ℋ̄¯ 𝜆2 ( ℎ̄ ℎ̄ ◦ ℎ̄ ′) =
◦

𝜆3 ( ℎ̄ ℎ̄ ◦ ℎ̄ ′. But then there exists a time, 𝑡¯, that ensures every 𝜆2 ∈ Λ2 will have a corresponding
𝜆3 ∈ Λ3 with the same action distribution at all subsequent realizable histories. Therefore,
∀ ℎ∈ℋ̄ ∃𝑡 ′ ∈N0 ∀ ℎ ◦ ∈ℋ́ ′ ∃𝜆2 ∈Λ2 ∀ ℎ′ ∈ℋ́ 𝜆1 (ℎ ℎ ◦ ℎ ′) = 𝜆2 (ℎ ℎ ◦ ℎ ′) .
𝑡 :∞
| {z }
∃𝜆3 ∈Λ3 =𝜆3 (ℎ ℎ ◦ ℎ ′ )

Rewriting,
∀ ℎ∈ℋ̄ ∃𝑡 ′ ∈N0 ∀ ℎ ◦ ∈ℋ́ ′ ∃𝜆3 ∈Λ3 ∀ ℎ′ ∈ℋ́ 𝜆1 (ℎ ℎ ◦ ℎ ′) = 𝜆3 (ℎ ℎ ◦ ℎ ′).
𝑡 :∞

But this is precisely the definition of always reaches, and thus we conclude. ✓ (ii). Sometimes
reaches and never reaches are not transitive.

𝑒
We construct two counterexamples, one for each of “sometimes reaches” (⇝ ) and “never
𝑒
/ ).
reaches” (⇝

Counterexample: Sometimes Reaches. To do so, we begin with a tuple (𝑒 , Λ1 , Λ2 , Λ3 )

such that both
𝑒 𝑒
∀Λ1 ∈Λ1 𝜆1 ⇝ Λ1 , ∀Λ2 ∈Λ2 𝜆2 ⇝ Λ2 .
1 1
We will show that there is an agent, 𝜆 ∈ Λ1 , such that 𝜆 ⇝ /𝑒 Λ3 , thus illustrating that
sometimes reaches is not transitive. The basic idea is that sometimes reaches only requires an
𝑒
agent stop its search on one realizable history. So, 𝜆1 ⇝ Λ2 might happen on some history ℎ,
but each 𝜆2 ∈ Λ2 might only reach Λ3 on an entirely different history. As a result, reaching
Λ2 is not enough to ensure the agent also reaches Λ3 .
In more detail, the agent sets of the counterexample are as follows. Let 𝒜 = {𝑎1 , 𝑎2 } and
1
𝒪 = {𝑜1 , 𝑜2 }. Let Λ2 be all agents that, after ten timesteps, always take 𝑎2 . 𝜆 is simple: it
always takes 𝑎 1 , except on one realizable history, ℎ ◦ , (and all of the realizable successors

26
 of ℎ ◦ , ℋℎ𝜆,𝑒 1 𝑒
◦ ), where it switches to taking 𝑎 2 after ten timesteps. Clearly 𝜆 ⇝ Λ , since
2
1
after ten timesteps, we know there will be some 𝜆2 such that 𝜆 (ℎ ◦ ℎ ′) = 𝜆2 (ℎ ◦ ℎ ′) for all
realizable history suffixes ℎ ′. Now, by assumption, we know that 𝜆2 ⇝ 𝑒
Λ3 . This ensures
there is a single realizable history ℎ such that there is an 𝜆 where 𝜆 (ℎ ℎ ) = 𝜆3 (ℎ ℎ ′) for any
3 2 ′
realizable suffix ℎ ′. To finish the counterexample, we simply note that this realizable ℎ can be
different from ℎ ◦ and all of its successors. For example, ℎ ◦ might be the history containing
only 𝑜1 for the first ten timesteps, while ℎ could be the history containing only 𝑜 2 for
the first ten timesteps. Thus, this 𝜆1 never reaches Λ3 , and we conclude the counterexample. ✓

Counterexample: Never Reaches. The instance for never reaches is simple: Let 𝒜 =
{𝑎1 , 𝑎2 , 𝑎3 }, and Λ1 = Λ3 . Suppose all agents in Λ1 (and thus Λ3 ) only choose actions 𝑎1
and 𝑎3 . Let Λ2 be a singleton, Λ2 = {𝜆2 } such that 𝜆2 : ℎ ↦→ 𝑎 2 . Clearly, every 𝜆1 ∈ Λ1 will
never reach Λ2 , since none of them ever choose 𝑎2 . Similarly, 𝜆2 will never reach Λ3 , since
no agents in Λ3 choose 𝑎2 . However, by Proposition C.22 and the assumption that Λ1 = Λ3 ,
𝑒
we know ∀𝜆1 ∈Λ1 𝜆1 □⇝ Λ3 . This directly violates transitivity. ✓
This completes the argument for all three cases, and we conclude. □

Lemma C.27. Sometimes reaches is not commutative: there exists a pair (Λ1 , Λ2 ) and 𝑒 ∈ ℰ such
𝑒
that ∀𝜆1 ∈Λ1 𝜆1 ⇝ /𝑒 Λ1 .
Λ2 , but ∃𝜆2 ∈Λ2 𝜆2 ⇝

Proof of Lemma C.27.

The result holds as a straightforward consequence of the following counterexample. Consider

the pair of agent sets
Λ1 = {𝜆 𝑖 : ℎ ↦→ 𝑎1 }, Λ2 = {𝜆 𝑖 : ℎ ↦→ 𝑎1 , 𝜆 𝑗 : ℎ ↦→ 𝑎2 }.
𝑒
Note that since 𝜆 𝑖 is in both sets, and Λ1 is a singleton, we know that 𝜆 ⇝ Λ1 in any
environment by Lemma C.28. But, clearly 𝜆 𝑗 never reaches Λ1 , since no agent in Λ1 ever
chooses 𝑎1 . □

𝑒
Lemma C.28. For all pairs (Λ, 𝑒), if 𝜆 ∈ Λ, then 𝜆 ⇝ Λ.

Proof of Lemma C.28.

The proposition is straightforward, as any 𝜆 ∈ Λ will be equivalent to itself in behavior for all
histories. □
𝑒
Lemma C.29. Every agent satisfies 𝜆 ⇝ Λ in every environment.

Proof of Lemma C.29.

This is again a direct consequence of Proposition C.24. □

𝑒
Lemma C.30. The decision problem, AGENT R EACHES, Given (𝑒 , 𝜆, Λ), output True iff 𝜆 ⇝ Λ, is
undecidable.

Proof of Lemma C.30.

We again proceed by reducing AGENT R EACHES from the Halting Problem.

In particular, let 𝑚 be a fixed but arbitrary Turing Machine, and 𝑤 be a fixed but arbitrary
input to be given to machine 𝑚. Then, H ALT defines the decision problem that outputs True
iff 𝑚 halts on input 𝑤.

27
 We construct an oracle for AGENT R EACHES that can decide H ALT as follows. Consider
the same observation space used in the proof of Lemma C.7: Let 𝒪 be comprised of all
configurations of machine 𝑚. Then, sequences of observations are simply evolution of
different Turing Machines processing possible inputs. We consider an action space, 𝒜 =
{𝑎halted , 𝑎not−yet }, where agents simply report whether the history so far contains a halting
configuration.
Then, we consider a deterministic environment 𝑒 that simply produces the next configuration of
𝑚 when run on input 𝑤, based on the current tape contents, the state of 𝑚, and the location of
the tape head. Note again that all three of these elements are contained in a Turing Machine’s
configuration.
Using these ingredients, we take any instance of H ALT, (𝑚, 𝑤), and build the singleton agent
set ΛB containing only the agent 𝜆halted : ℎ ↦→ 𝑎halted that always reports the machine as
having halted. We then consider whether the agent that outputs 𝑎not−yet indefinitely until 𝑚
reports halting, at which point the agent switches to 𝑎halted .
𝑒
We make one query to our AGENT R EACHES oracle, and ask: 𝜆 ⇝ ΛB . If it is True, then the
branching agent eventually becomes equivalent to 𝜆halted in that they both indefinitely output
𝑎halted on at least one realizable history. Since 𝑒 is deterministic, we know this equivalence
holds across all histories. If the query reports False, then there is no future in 𝑒 in which 𝑚
halts on 𝑤, otherwise the agent would become equivalent to 𝜆halted . We thus use the oracle’s
response directly to decide the given instance of H ALT. □

C.3 An Example of a Family of Learning Rules

Next, we highlight a valuable new perspective that is afforded by our definitions: We can use learning
rules to characterize families of agents. Recall that a learning rule is a mechanism for modeling an
agent’s plausible history-conditioned switches among elements of an agent basis, ΛB . As a motivating
example, we consider a family of model-based learning agents that make us of a simple form of a
model-based learning rule, as follows.
Definition C.11. We say 𝜎 : ℋ → ΛB defines a simple model-based learning rule relative to basis
ΛB and environment model 𝑒ˆ when, for every history ℎ, 𝜎(ℎ) = arg max 𝑣(𝜆, 𝑒ˆ | ℎ).
𝜆∈ΛB

Then, a model-based agent is an agent that uses one of the above learning rules to (i) maintain a model
of the environment, 𝑒ˆ : ℋ × 𝒜 → Δ(𝒪), and (ii) carry out an idealized form of planning wherein
the learning rule produces the basis element that maximizes long term performance in that model
of the environment at the current history. We might add further restrictions such as computational
constraints that prevent the learning rule from computing 𝑣 exactly, or constrain the agent’s memory
so that it can only maintain a finite summary of each ℎ. We leave such considerations for future work.
We first prove that these model-based learning rules are universal in the sense that every agent can be
understood as applying a simple model-based learning rule over some basis.
Proposition C.31. For any 𝑒 ∈ ℰ and agent 𝜆 ∈ Λ, there exists a choice of basis ΛB and simple
model-based learning rule 𝜎 such that, for every realizable ℎ ∈ ℋ̄ ,
𝜆(ℎ) = 𝜎(ℎ)(ℎ). (C.36)

Proof of Proposition C.31.

The argument is actually quite straightforward, as for any agent 𝜆⋄ ∈ Λ, we can always
consider the model-based learning rule defined over the singleton agent basis,
𝜎(ℎ) = arg max 𝑣(𝜆, 𝑒 | ℎ). (C.37)
𝜆∈{𝜆⋄ }

Clearly, by necessity, the above learning rule will pick 𝜆⋄. Since 𝜆⋄ was chosen arbitrarily, we
conclude that this construction holds for all agents. □

28
We next strengthen this result to show the property holds even under learning rules defined over the
universal basis.
Proposition C.32. For each agent 𝜆 ∈ Λ, there is a model-based learning rule defined over the
universal basis, 𝜎 : ℋ → Λ, that produces that agent.

Proof of Proposition C.32.

The argument follows from the fact that every agent is optimal in at least one environment. To
see this, note that the constant-zero reward function, 𝑟(ℎ) = 0, ensures all agents are contained
in the set of optimal agents. Then, pick an arbitrary agent 𝜆⋄ ∈ Λ. Note that by the first
claim, each agent is optimal (for any choice of 𝑣) in at least one environment. Call one of the
environment’s that 𝜆⋄ is optimal in 𝑒 ⋄. Then, the learning rule,
𝜎(ℎ) = arg max 𝑣(𝜆, 𝑒 ⋄ | ℎ), (C.38)
𝜆∈Λ

can be chosen to break arg max ties by always choosing 𝜆⋄. □

Finally, we highlight one interesting consequence of this specific family of learning rules in the
context of CRL.
Theorem C.33. In any CRL problem (𝑒 , 𝑣, Λ, ΛB ), where ΛB ⊢𝑒 Σ Λ according to a set of model-based
learning rules Σ, every optimal agent 𝜆∗ ∈ Λ∗ replans infinitely often. That is, for any optimal agent
𝜆∗ ∈ Λ∗ , its corresponding learning rule 𝜎 ∈ Σ satisfies the following property: ∀ ℎ∈ℋ̄ ∃ ℎ′ ∈ℋ́
arg max 𝑣(𝜆B , 𝑒ˆ | ℎ) ≠ arg max 𝑣(𝜆B , 𝑒ˆ | ℎ ℎ ′) . (C.39)
𝜆B ∈ΛB 𝜆B ∈ΛB
| {z } | {z }
𝜎(ℎ) 𝜎(ℎ ℎ ′ )

Proof of Theorem C.33.

We proceed toward contradiction and assume that there is a CRL problem in which an optimal
model-based agent 𝜆∗ only replans finitely many times.
That is, we assume that there is an optimal model-based agent, 𝜆∗ ∈ Λ∗ whose corresponding
learning rule 𝜎 satisfies the following: ∃ ℎ∈ℋ̄ ∀ ℎ′ ∈ℋ́ such that,
arg max 𝑣(𝜆, 𝑒ˆ | ℎ) = arg max 𝑣(𝜆, 𝑒ˆ | ℎ ℎ ′) .
𝜆∈ΛB 𝜆∈ΛB
| {z } | {z }
𝜎(ℎ) 𝜎(ℎ ℎ ′ )

Then, consider the behavior of 𝜆∗ at some realizable history ℎ. By the above equivalence, we
know that for every realizable suffix ℎ ′ that might follow ℎ, the agent will produce the same
behavior, 𝜆 ∈ ΛB , output from
arg max 𝑣(𝜆, 𝑒ˆ | ℎ). (C.40)
𝜆∈ΛB
Therefore, it follows that 𝜆∗ ⇝
𝑒
ΛB : there exists a realizable history under which 𝜆∗ will
produce the same behavior as an element from ΛB on all futures. However, recall that by
assumption we are in a CRL problem, meaning all optimal agents must never reach ΛB .
We have reached a contradiction, and thus conclude the proof. □

Thus, in CRL, every optimal model-based agent necessarily replans over its environment model 𝑒ˆ
infinitely often. This result is suggestive of the kinds of insights that can be unlocked through a
mathematical abstraction of learning rules and the agent-families they induce. We anticipate that
further study of kinds of learning rules can directly inform the design of new learning algorithms;
for instance, we might specify the family of continual learning rules that are guaranteed to yield a
continual learning agent, and use this to guide the design of effective continual learning agents.

29