Meta-Learning in Neural Networks A Survey

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 44, NO.
9, SEPTEMBER 2022 5149
Meta-Learning in Neural Networks: A Survey

Timothy Hospedales , Antreas Antoniou, Paul Micaelli, and Amos Storkey
Abstract—The field of meta-learning, or learning-to-learn, has seen a dramatic rise in interest in recent years. Contrary to conventional
approaches to AI where tasks are solved from scratch using a fixed learning algorithm, meta-learning aims to improve the learning
algorithm itself, given the experience of multiple learning episodes. This paradigm provides an opportunity to tackle many conventional
challenges of deep learning, including data and computation bottlenecks, as well as generalization. This survey describes the
contemporary meta-learning landscape. We first discuss definitions of meta-learning and position it with respect to related fields, such
as transfer learning and hyperparameter optimization. We then propose a new taxonomy that provides a more comprehensive
breakdown of the space of meta-learning methods today. We survey promising applications and successes of meta-learning such as
few-shot learning and reinforcement learning. Finally, we discuss outstanding challenges and promising areas for future research.
Index Terms—Meta-learning, learning-to-learn, few-shot learning, transfer learning, neural architecture search
1 INTRODUCTION Neural network meta-learning has a long history [7], [17],

[18]. However, its potential as a driver to advance the fron-
machine learning models are typically
C ONTEMPORARY
trained from scratch for a specific task using a fixed
learning algorithm designed by hand. Deep learning-based
tier of the contemporary deep learning industry has led to
an explosion of recent research. In particular meta-learning
has the potential to alleviate many of the main criticisms of
approaches specifically have seen great successes in a vari-
contemporary deep learning [4], for instance by improving
ety of fields [1], [2], [3]. However there are clear limitations
data efficiency, knowledge transfer and unsupervised learn-
[4]. For example, successes have largely been in areas where
ing. Meta-learning has proven useful both in multi-task sce-
vast quantities of data can be collected or simulated, and
narios where task-agnostic knowledge is extracted from a
where huge compute resources are available. This excludes
family of tasks and used to improve learning of new tasks
many applications where data is intrinsically rare or expen-
from that family [7], [19]; and single-task scenarios where a
sive [5], or compute resources are unavailable [6].
single problem is solved repeatedly and improved over
Meta-learning is the process of distilling the experience
multiple episodes [15], [20], [21], [22]. Successful applications
of multiple learning episodes – often covering a distribution
have been demonstrated in areas spanning few-shot image
of related tasks – and using this experience to improve
recognition [19], [23], unsupervised learning [16], data effi-
future learning performance. This ‘learning-to-learn’ [7] can
cient [24], [25] and self-directed [26] reinforcement learning
lead to a variety of benefits such as improved data and com-
(RL), hyperparameter optimization [20], and neural archi-
pute efficiency, and it is better aligned with human and ani-
tecture search (NAS) [21], [27], [28].
mal learning [8], where learning strategies improve both on
Many perspectives on meta-learning can be found in the
a lifetime and evolutionary timescales [8], [9], [10].
literature, in part because different communities use the term
Historically, the success of machine learning was driven by
differently. Thrun [7] operationally defines learning-to-learn
advancing hand-engineered features [11], [12]. Deep learning
as occurring when a learner’s performance at solving tasks
realised the promise of feature representation learning [13],
drawn from a given task family improves with respect to the
providing a huge improvement in performance for many
number of tasks seen. (cf., conventional machine learning per-
tasks [1], [3] compared to prior hand-designed features. Meta-
formance improves as more data from a single task is seen).
learning in neural networks aims to provide the next step of
This perspective [29], [30], [31] views meta-learning as a tool
integrating joint feature, model, and algorithm learning; that
to manage the ‘no free lunch’ theorem [32] and improve gen-
is, it targets replacing prior hand-designed learners with
eralization by searching for the algorithm (inductive bias) that
learned learning algorithms [7], [14], [15], [16].
is best suited to a given problem, or problem family. How-
ever, this definition can include transfer, multi-task, feature-
selection, and model-ensemble learning, which are not typi-
Timothy Hospedales is with the Samsung AI Centre, CB1 2JD Cambridge, cally considered as meta-learning today. Another usage of
U.K., and also with the University of Edinburgh, EH8 9AB Edinburgh,
U.K. E-mail: [email protected].
meta-learning [33] deals with algorithm selection based on
Antreas Antoniou, Paul Micaelli, and Amos Storkey are with the Univer- dataset features, and becomes hard to distinguish from auto-
sity of Edinburgh, EH8 9AB Edinburgh, U.K. E-mail: {a.antoniou, paul. mated machine learning (AutoML) [34].
micaelli, a.storkey}@ed.ac.uk. In this paper, we focus on contemporary neural-network
Manuscript received 9 Oct. 2020; revised 11 Mar. 2021; accepted 27 Apr. 2021. meta-learning. We take this to mean algorithm learning as per
Date of publication 11 May 2021; date of current version 4 Aug. 2022. [29], [30], but focus specifically on where this is achieved by
(Corresponding author: Timothy Hospedales.)
Recommended for acceptance by L. Wang. end-to-end learning of an explicitly defined objective function (such
Digital Object Identifier no. 10.1109/TPAMI.2021.3079209 as cross-entropy loss). Additionally we consider single-task
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
5150 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 44, NO. 9, SEPTEMBER 2022
meta-learning, and discuss a wider variety of (meta) objectives where L is a loss function that measures the error between
such as robustness and compute efficiency. true labels and those predicted by fu ðÞ. Here, v specifies
This paper provides a unique, timely, and up-to-date assumptions about ‘how to learn’, such as the choice of opti-
survey of the rapidly growing area of neural network meta- mizer for u or function class for f. Generalization is then
learning. In contrast, previous surveys are rather out of date measured by evaluating L on test points with known labels.
and/or focus on algorithm selection for data mining [29], The conventional assumption is that this optimization is
[33], [35], [36], AutoML [34], or particular applications of performed from scratch for every problem D; and that v is
meta-learning such as few-shot learning [37] or NAS [38]. pre-specified. However, the specification of v can drasti-
We address both meta-learning methods and applications cally affect performance measures like accuracy or data effi-
(Fig. 1, Table 1). We first introduce meta-learning through a ciency. Meta-learning seeks to improve these measures by
high-level problem formalization that can be used to under- learning the learning algorithm itself, rather than assuming
stand and position work in this area. We then provide a new it is pre-specified and fixed. This is often achieved by revis-
taxonomy in terms of meta-representation, meta-objective iting the first assumption above, and learning from a distri-
and meta-optimizer. This framework provides a design- bution of tasks rather than from scratch. Note that while the
space for developing new meta learning methods and cus- following formalization of meta-learning takes a supervised
tomizing them for different applications. We survey several perspective for simplicity, all the ideas generalize directly to
popular and emerging application areas including few-shot, a reinforcement learning setting as discussed in Section 5.2.
reinforcement learning, and architecture search; and posi- Meta-Learning: Task-Distribution View. A common view of
tion meta-learning with respect to related topics such as meta-learning is to learn a general purpose learning algo-
transfer and multi-task learning. We conclude by discussing rithm that generalizes across tasks, and ideally enables each
outstanding challenges and areas for future research. new task to be learned better than the last. We can evaluate
the performance of v over a task distribution pðT Þ, where a
2 BACKGROUND task specifies a dataset and loss function T ¼ fD; Lg. At a
Meta-learning is difficult to define, having been used in var- high level, learning how to learn thus becomes
ious inconsistent ways, even within contemporary neural-
min ET pðT Þ LðD; vÞ; (2)
network literature. In this section, we introduce our defini- v
tion and key terminology, and then position meta-learning
where L is a task specific loss. In typical machine-learning
with respect to related topics.
settings, we can split D ¼ ðDtrain ; Dval Þ. We then define the
Meta-learning is most commonly understood as learning
task specific loss to be LðD; vÞ ¼ LðDval ; u ðDtrain ; vÞ; vÞ;
to learn; the process of improving a learning algorithm over
where u is the parameters of the model trained using the
multiple learning episodes. In contrast, conventional ML
‘how to learn’ meta-knowledge v on dataset Dtrain .
improves model predictions over multiple data instances.
More concretely, to solve this problem in practice, we often
During base learning, an inner (or lower/base) learning algo-
assume a set of M source tasks is sampled from pðT Þ. For-
rithm solves a task such as image classification [13], defined
mally, we denote the set of M source tasks used in the meta-
by a dataset and objective. During meta-learning, an outer (or ðiÞ M
training stage as D source ¼ fðDtrain
source ; Dsource Þ gi¼1 where each
val
upper/meta) algorithm updates the inner learning algorithm
task has both training and validation data. Often, the source
such that the model it learns improves an outer objective.
train and validation datasets are respectively called support
For instance this objective could be generalization perfor-
and query sets. The meta-training step of ‘learning how to learn’
mance or learning speed of the inner algorithm.
can be written as:
As defined above, many conventional algorithms such as
random search of hyper-parameters by cross-validation could X
M
fall within the definition of meta-learning. The salient charac- v ¼ arg min LðDðiÞ
source ; vÞ: (3)
teristic of contemporary neural-network meta-learning is an v i¼1
explicitly defined meta-level objective, and end-to-end optimiza-
Now we denote the set of Q target tasks used in the meta-
tion of the inner algorithm with respect to this objective. ðiÞ Q
testing stage as D target ¼ fðDtrain
target ; Dtarget Þ gi¼1 where each
test
Often, meta-learning is conducted on learning episodes sam-
task has both training and test data. In the meta-testing stage
pled from a task family, leading to a base learning algorithm
we use the learned meta-knowledge v to train the base
that performs well on new tasks sampled from this family.
model on each previously unseen target task i:
However, in a limiting case all training episodes can be sam-
pled from a single task. In the following section, we introduce train ðiÞ
u ðiÞ ¼ arg min LðDtarget ; u; v Þ: (4)
these notions more formally. u
2.1 Formalizing Meta-Learning In contrast to conventional learning in Eq. (1), learning on

Conventional Machine Learning. In conventional supervised the training set of a target task i now benefits from meta-
machine learning, we are given a training dataset D ¼ knowledge v about the algorithm to use. This could be an
fðx1 ; y1 Þ; . . . ; ðxN ; yN Þg, such as (input image, output label) estimate of the initial parameters [19], or an entire learning
pairs. We can train a predictive model y^ ¼ fu ðxÞ parameter- model [39] or optimization strategy [14]. We can evaluate
ized by u, by solving: the accuracy of our meta-learner by the performance of u ðiÞ
test ðiÞ
on the test split of each target task Dtarget .
u ¼ arg min LðD; u; vÞ; (1) This setup leads to analogies of conventional underfit-
u ting and overfitting: meta-underfitting and meta-overfitting.
HOSPEDALES ET AL.: META-LEARNING IN NEURAL NETWORKS: A SURVEY 5151
In particular, meta-overfitting is an issue whereby the meta- abstract objective in Eq. (2) to define a toy example for
knowledge learned on the source tasks does not generalize to meta-training linear regression [43].
the target tasks. It is relatively common, especially in the case X h i
where only a small number of source tasks are available. It min ET pðT ÞðDtr ;Dval Þ2T ðxT gv ðDtr Þ yÞ2 : (7)
v
can be seen as learning an inductive bias v that constrains the ðx;yÞ2Dval
hypothesis space of u too tightly around solutions to the
source tasks. Here we meta-train by optimizing over a distribution of
Meta-Learning: Bilevel Optimization View. The previous tasks. For each task a train and validation set is drawn. The
discussion outlines the common flow of meta-learning in a train set Dtr is embedded [44] into a vector gv which defines
multiple task scenario, but does not specify how to solve the the linear regression weights to predict examples x from the
meta-training step in Eq. (3). This is commonly done by validation set. Optimizing Eq. (7) ‘learns to learn’ by train-
casting the meta-training step as a bilevel optimization ing the function gv to map a training set to a weight vector.
problem. While this picture is most accurate for the opti- Thus gv should provide a good solution for novel meta-test
mizer-based methods (see Section 3.1), it is helpful to visual- tasks T te drawn from pðT Þ. Methods in this family vary in
ize the mechanics of meta-learning more generally. Bilevel the complexity of the predictive model g used, and how the
optimization [40] refers to a hierarchical optimization prob- support set is embedded [44] (e.g., by pooling, CNN or
lem, where one optimization contains another optimization RNN). These models are also known as amortized [45]
as a constraint [20], [41]. Using this notation, meta-training because the cost of learning a new task is reduced to a feed-
can be formalised as follows: forward operation through gv ðÞ, with iterative optimization
already paid for during meta-training of v.
X
M
v ¼ arg min Lmeta ðDval ðiÞ ðiÞ
source ; u ðvÞ; vÞ (5) 2.2 Historical Context of Meta-Learning
v i¼1
Meta-learning and learning-to-learn first appear in the liter-
ature in 1987[17]. J. Schmidhuber introduced a family of
methods that can learn how to learn, using self-referential
s.t. u ðiÞ ðvÞ ¼ arg min Ltask ðDtrain ðiÞ
source ; u; vÞ; (6) learning. Self-referential learning involves training neural
u
networks that can receive as inputs their own weights and
where Lmeta and Ltask refer to the outer and inner objectives predict updates for said weights. Schmidhuber proposed to
respectively, such as cross entropy in the case of classifica- learn the model itself using evolutionary algorithms.
tion; and we have dropped explicit dependence of v on D Meta-learning was subsequently extended to multiple
for brevity. Note the leader-follower asymmetry between areas. Bengio et al.[46], [47] proposed to meta-learn biologi-
the outer and inner levels: the inner optimization Eq. (6) is cally plausible learning rules. Schmidhuber et al.continued
conditional on the learning strategy v defined by the outer to explore self-referential systems and meta-learning [48],
level, but it cannot change v during its training. [49]. S. Thrun et al. took care to more clearly define the term
Here v could indicate an initial condition in non-convex learning to learn in [7] and introduced initial theoretical justi-
optimization [19] or a hyper-parameter such as regularization fications and practical implementations. Proposals for train-
strength [20]. Section 4.1 discusses the space of choices for v ing meta-learning systems using gradient descent and
in detail. The outer level optimization trains v to produce backpropagation were first made in 1991 [50] followed by
models u ðiÞ ðvÞ that perform well on their validation sets after more extensions in 2001 [51], [52], with [29] giving an over-
training. Section 4.2 discusses how to optimize v. Note that view of the literature at that time. Meta-learning was used
Lmeta and Ltask are distinctly denoted, because they can be dif- in the context of reinforcement learning in 1995 [53], fol-
ferent functions. For example, v can define the inner task loss lowed by various extensions [54], [55].
Ltask
v (see Section 4.1, [25], [42]); and as explored in Section 4.3,
Lmeta can measure different quantities such as validation per- 2.3 Related Fields
formance, learning speed or robustness. Here we position meta-learning against related areas whose
Finally, we note that the above formalization of meta-train- relation to meta-learning is often a source of confusion.
ing uses the notion of a distribution over tasks. While common Transfer Learning (TL). TL [35], [56] uses past experience
in the meta-learning literature, it is not a necessary condition from a source task to improve learning (speed, data effi-
for meta-learning. More formally, if we are given a single train ciency, accuracy) on a target task. TL refers both to this
and test dataset (M ¼ Q ¼ 1), we can split the training set to problem area and family of solutions, most commonly
get validation data such that D source ¼ ðDtrain source ; Dsource Þ for
val
parameter transfer plus optional fine tuning [57] (although
meta-training, and for meta-testing we can use D target ¼ there are numerous other approaches [35]).
source [ Dsource ; Dtarget Þ. We still learn v over several epi-
ðDtrain val test
In contrast, meta-learning refers to a paradigm that can
sodes, and different train-val splits are usually used during be used to improve TL as well as other problems. In TL the
meta-training. prior is extracted by vanilla learning on the source task
Meta-Learning: Feed-Forward Model View. As we will see, without the use of a meta-objective. In meta-learning, the
there are a number of meta-learning approaches that syn- corresponding prior would be defined by an outer optimi-
thesize models in a feed-forward manner, rather than via an zation that evaluates the benefit of the prior when learn a
explicit iterative optimization as in Eqs. 5-6 above. While new task, as illustrated by MAML [19]. More generally,
they vary in their degree of complexity, it can be instructive meta-learning deals with a much wider range of meta-rep-
to understand this family of approaches by instantiating the resentations than solely model parameters (Section 4.1).
Domain Adaptation (DA) and Domain Generalization (DG). exact due to the choice of conjugate exponential models, in
Domain-shift refers to the situation where source and target others (see e.g., [73]), a stochastic variational approach is used
problems share the same objective, but the input distribution to calculate an approximate posterior, from which a lower
of the target task is shifted with respect to the source task [35], bound to the marginal likelihood is computed.
[58], reducing model performance. DA is a variant of transfer Bayesian hierarchical models provide a valuable view-
learning that attempts to alleviate this issue by adapting the point for meta-learning, by providing a modeling rather
source-trained model using sparse or unlabeled data from the than an algorithmic framework for understanding the meta-
target. DG refers to methods to train a source model to be learning process. In practice, prior work in HBMs has typi-
robust to such domain-shift without further adaptation. cally focused on learning simple tractable models u while
Many knowledge transfer methods have been studied [35], most meta-learning work considers complex inner-loop
[58] to boost target domain performance. However, as for TL, learning processes, involving many iterations. Nonetheless,
vanilla DA and DG don’t use a meta-objective to optimize some meta-learning methods like MAML [19] can be under-
‘how to learn’ across domains. Meanwhile, meta-learning stood through the lens of HBMs [74].
methods can be used to perform both DA [59] and DG [42] AutoML. AutoML [33], [34] is a rather broad umbrella for
(see Section 5.8). approaches aiming to automate parts of the machine learning
Continual Learning (CL). Continual or lifelong learning process that are typically manual, such as data preparation,
[60], [61] refers to the ability to learn on a sequence of tasks algorithm selection, hyper-parameter tuning, and architecture
drawn from a potentially non-stationary distribution, and search. AutoML often makes use of numerous heuristics out-
in particular seek to do so while accelerating learning new side the scope of meta-learning as defined here, and focuses
tasks and without forgetting old tasks. Similarly to meta- on tasks such as data cleaning that are less central to meta-
learning, a task distribution is considered, and the goal is learning. However, AutoML sometimes makes use of end-to-
partly to accelerate learning of a target task. However most end optimization of a meta-objective, so meta-learning can be
continual learning methodologies are not meta-learning seen as a specialization of AutoML.
methodologies since this meta objective is not solved for
explicitly. Nevertheless, meta-learning provides a potential
framework to advance continual learning, and a few recent
3 TAXONOMY
studies have begun to do so by developing meta-objectives 3.1 Previous Taxonomies
that encode continual learning performance [62], [63], [64]. Previous [75], [76] categorizations of meta-learning methods
Multi-Task Learning (MTL) Aims to jointly learn several have tended to produce a three-way taxonomy across opti-
related tasks, to benefit from regularization due to parame- mization-based methods, model-based (or black box) meth-
ter sharing and the diversity of the resulting shared repre- ods, and metric-based (or non-parametric) methods.
sentation [65], [66], as well as compute/memory savings. Optimization. Optimization-based methods include those
Like TL, DA, and CL, conventional MTL is a single-level where the inner-level task (Eq. (6)) is literally solved as an opti-
optimization without a meta-objective. Furthermore, the mization problem, and focuses on extracting meta-knowledge
goal of MTL is to solve a fixed number of known tasks, v required to improve optimization performance. A famous
whereas the point of meta-learning is often to solve unseen example is MAML [19], which aims to learn the initialization
future tasks. Nonetheless, meta-learning can be brought in v ¼ u0 , such that a small number of inner steps produces a
to benefit MTL, e.g., by learning the relatedness between classifier that performs well on validation data. This is also
tasks [67], or how to prioritise among multiple tasks [68]. performed by gradient descent, differentiating through the
Hyperparameter Optimization (HO) is within the remit of updates of the base model. More elaborate alternatives also
meta-learning, in that hyperparameters like learning rate or learn step sizes [77], [78] or train recurrent networks to predict
regularization strength describe ‘how to learn’. Here we steps from gradients [14], [15], [79]. Meta-optimization by gra-
include HO tasks that define a meta objective that is trained dient over long inner optimizations leads to several compute
end-to-end with neural networks, such as gradient-based and memory challenges which are discussed in Section 6. A
hyperparameter learning [67], [69] and neural architecture unified view of gradient-based meta learning expressing
search [21]. But we exclude other approaches like random many existing methods as special cases of a generalized inner
search [70] and Bayesian Hyperparameter Optimization loop meta-learning framework has been proposed [80].
[71], which are rarely considered to be meta-learning. Black Box/Model-Based. In model-based (or black-box)
Hierarchical Bayesian Models (HBM) involve Bayesian learn- methods the inner learning step (Eq. (6), Eq. (4)) is wrapped
ing of parameters u under a prior pðujvÞ. The prior is written up in the feed-forward pass of a single model, as illustrated
as a conditional density on some other variable v which has its in Eq. (7). The model embeds the current dataset D into acti-
own prior pðvÞ. Hierarchical Bayesian models feature strongly vation state, with predictions for test data being made based
as models for grouped data D ¼ fDi ji ¼ 1; 2; . . . ; hMg, QM where on this state. Typical architectures include recurrent net-
each group i has its own ui . The full model is i¼1 pðDi j works [14], [51], convolutional networks [39] or hypernet-
ui Þpðui jvÞpðvÞ. The levels of hierarchy can be increased fur- works [81], [82] that embed training instances and labels of
ther; in particular v can itself be parameterized, and hence a given task to define a predictor for test samples. In this
pðvÞ can be learnt. Learning is usually full-pipeline, but using case all the inner-level learning is contained in the activation
some form of Bayesian marginalisation
Q R to compute the poste- states of the model and is entirely feed-forward. Outer-level
rior over v: P ðvjDÞ pðvÞ M i¼1 du i pðDi jui Þpðui jvÞ. The ease learning is performed with v containing the CNN, RNN or
of doing the marginalisation depends on the model: in some hypernetwork parameters. The outer and inner-level opti-
(e.g., Latent Dirichlet Allocation [72]) the marginalisation is mizations are tightly coupled as v and D directly specify u.
Memory-augmented neural networks [83] use an explicit stor- 4.1 Meta-Representation

age buffer and can be seen as a model-based algorithm [84], Meta-learning methods make different choices about what
[85]. Compared to optimization-based approaches, these enjoy meta-knowledge v should be, i.e., which aspects of the
simpler optimization without requiring second-order gra- learning strategy should be learned; and (by exclusion)
dients. However, it has been observed that model-based which aspects should be considered fixed.
approaches are usually less able to generalize to out-of-distri- Parameter Initialization. Here v corresponds to the initial
bution tasks than optimization-based methods [86]. Further- parameters of a neural network to be used in the inner opti-
more, while they are often very good at data efficient few-shot mization, with MAML being the most popular example
learning, they have been criticised for being asymptotically [19], [96], [97]. A good initialization is just a few gradient
weaker [86] as they struggle to embed a large training set into steps away from a solution to any task T drawn from pðT Þ,
a rich base model. and can help to learn without overfitting in few-shot learn-
Metric-Learning. Metric-learning or non-parametric algo- ing. A key challenge with this approach is that the outer
rithms are thus far largely restricted to the popular but spe- optimization needs to solve for as many parameters as the
cific few-shot application of meta-learning (Section 5.1.1). inner optimization (potentially hundreds of millions in
The idea is to perform non-parametric ‘learning’ at the inner large CNNs). This leads to a line of work on isolating a sub-
(task) level by simply comparing validation points with set of parameters to meta-learn, for example by subspace
training points and predicting the label of matching training [76], [98], by layer [81], [98], [99], or by separating out scale
points. In chronological order, this has been achieved with and shift [100]. Another concern is whether a single initial
siamese [87], matching [88], prototypical [23], relation [89], condition is sufficient to provide fast learning for a wide
and graph [90] neural networks. Here outer-level learning range of potential tasks, or if one is limited to narrow distri-
corresponds to metric learning (finding a feature extractor v butions pðT Þ. This has led to variants that model mixtures
that represents the data suitably for comparison). As before over multiple initial conditions [98], [101], [102].
v is learned on source tasks, and used for target tasks. Optimizer. The above parameter-centric methods usually
Discussion. The common breakdown reviewed above rely on existing optimizers such as SGD with momentum or
does not expose all facets of interest and is insufficient to Adam [103] to refine the initialization when given some
understand the connections between the wide variety of new task. Instead, optimizer-centric approaches [14], [15],
meta-learning frameworks available today. For this reason, [79], [92] focus on learning the inner optimizer by training a
we propose a new taxonomy in the following section. function that takes as input optimization states such as u
and ru Ltask and produces the optimization step for each
3.2 Proposed Taxonomy base learning iteration. The trainable component v can span
We introduce a new breakdown along three independent simple hyper-parameters such as a fixed step size [77], [78]
axes. For each axis we provide a taxonomy that reflects the to more sophisticated pre-conditioning matrices [104], [105].
current meta-learning landscape. Ultimately v can be used to define a full gradient-based
Meta-Representation (“What?”). The first axis is the choice optimizer through a complex non-linear transformation of
of meta-knowledge v to meta-learn. This could be anything the input gradient and other metadata [14], [15], [91], [92].
from initial model parameters [19] to readable code in the The parameters to learn here can be few if the optimizer is
case of program induction [91]. applied coordinate-wise across weights [15]. The initializa-
Meta-Optimizer (“How?”). The second axis is the choice of tion-centric and optimizer-centric methods can be merged
optimizer to use for the outer level during meta-training by learning them jointly, namely having the former learn
(see Eq. (5)). The outer-level optimizer for v can take a vari- the initial condition for the latter [14], [77]. Optimizer learn-
ety of forms from gradient-descent [19], to reinforcement ing methods have both been applied to for few-shot learn-
learning [91] and evolutionary search [25]. ing [14] and to accelerate and improve many-shot learning
Meta-Objective (“Why?”). The third axis is the goal of [15], [91], [92]. Finally, one can also meta-learn zeroth-order
meta-learning which is determined by choice of meta-objec- optimizers [106] that only require evaluations of Ltask rather
tive Lmeta (Eq. (5)), task distribution pðT Þ, and data-flow than optimizer states such as gradients. These have been
between the two levels. Together these can customize meta- shown [106] to be competitive with conventional Bayesian
learning for different purposes such as sample efficient few- Optimization [71] alternatives.
shot learning [19], [39], fast many-shot optimization [91], Feed-Forward Models (FFMs. aka, Black-Box, Amortized).
[92], robustness to domain-shift [42], [93], label noise [94], Another family of models trains learners v that provide a
and adversarial attack [95]. feed-forward mapping directly from the support set to the
Together these axes provide a design-space for meta- parameters required to classify test instances, i.e., u ¼
learning methods that can orient the development of new gv ðDtrain Þ – rather than relying on a gradient-based iterative
algorithms and customization for particular applications. optimization of u. These correspond to black-box model-
Note that the base model representation u isn’t included in based learning in the conventional taxonomy (Section 3.1)
this taxonomy, since it is determined and optimized in a and span from classic [107] to recent approaches such as
way that is specific to the application at hand. CNAPs [108] that provide strong performance on challeng-
ing cross-domain few-shot benchmarks [109].
These methods have connections to Hypernetworks [110]
4 SURVEY: METHODOLOGIES which generate the weights of another neural network con-
In this section we break down existing literature according ditioned on some embedding – and are often used for com-
to our proposed new methodological taxonomy. pression or multi-task learning. Here v is the hypernetwork
Fig. 1. Overview of the meta-learning landscape including algorithm design (meta-optimizer, meta-representation, meta-objective), and applications.
and it synthesises u given the source dataset in a feed-for- or combining with gradient-based meta-learning to train other
ward pass [98], [111]. Embedding the support set is often hyper-parameters such as stochastic regularizers [116].
achieved by recurrent networks [51], [112], [113] convolu- Losses and Auxiliary Tasks. These approaches learn the inner
tion [39], or set embeddings [45], [108]. Research here often task-loss Ltask
v for the base model (in contrast to Lmeta , which
studies architectures for paramaterizing the classifier by the is fixed). Loss-learning approaches typically define a function
task-embedding network: (i) Which parameters should be that inputs relevant quantities (e.g., predictions, labels, or
globally shared across all tasks, versus synthesized per task model parameters) and outputs a scalar to be treated as a loss
by the hypernetwork (e.g., share the feature extractor and by the inner (task) optimizer. This can lead to a learned loss
synthesize the classifier [81], [114]), and (ii) How to parame- that is easier to optimize (less local minima) than standard
terize the hypernetwork so as to limit the number of param- alternatives [22], [25], [117], provides faster learning with
eters required in v (e.g., via synthesizing only lightweight improved generalization [43], [118], [119], [120], robustness to
adapter layers in the feature extractor [108], or class-wise label noise [121], or whose minima corresponds to a model
classifier weight synthesis [45]). more robust to domain shift [42]. Loss learning methods have
Some FFMs can also be understood elegantly in terms of also been used to learn from unlabeled instances [99], [122], or
amortized inference in probabilistic models [45], [107], mak- to learn Ltask
v as a differentiable approximation to a true non-
ing predictions for test data x as: differentiable Lmeta of interest such as area under precision
Z recall curve [123].
Loss learning also arises in generalizations of self-
qv ðyjx; Dtr Þ ¼ pðyjx; uÞqv ðujDtr Þdu; (8)
supervised [124] or auxiliary task [125] learning. In these
problems unsupervised predictive tasks (such as colouris-
where the meta-representation v is a network qv ðÞ that ing pixels in vision [124], or simply changing pixels in RL
approximates the intractable Bayesian inference for parame- [125]) are defined and optimized with the aim of improv-
ters u that solve the task with training data Dtr , and the inte- ing the representation for the main task. In this case the
gral may be computed exactly [107], or approximated by best auxiliary task (loss) to use can be hard to predict in
sampling [45] or point estimate [108]. The model v is then advance, so meta-learning can be used to select among
trained to minimise validation loss over a distribution of several auxiliary losses according to their impact on
training tasks cf. Eq. (7). improving main task learning. I.e., v is a per-auxiliary task
Finally, memory-augmented neural networks, with the weight [68]. More generally, one can meta-learn an auxil-
ability to remember old data and assimilate new data iary task generator that annotates examples with auxiliary
quickly, typically fall in the FFM category as well [84], labels [126].
[85]. Architectures. Architecture discovery has always been an
Embedding Functions (Metric Learning). Here the meta-opti- important area in neural networks [38], [127], and one that is
mization process learns an embedding network v that trans- not amenable to simple exhaustive search. Meta-Learning can
forms raw inputs into a representation suitable for recognition be used to automate this very expensive process by learning
by simple similarity comparison between query and support architectures. Early attempts used evolutionary algorithms to
instances [23], [81], [88], [114] (e.g., with cosine similarity or learn the topology of LSTM cells [128], while later approaches
euclidean distance). These methods are classified as metric leveraged RL to generate descriptions for good CNN architec-
learning in the conventional taxonomy (Section 3.1) but can tures [28]. Evolutionary Algorithms [27] can learn blocks
also be seen as a special case of the feed-forward black-box within architectures modelled as graphs which could mutate
models above. This can easily be seen for methods that pro- by editing their graph. Gradient-based architecture representa-
duce logits based on the inner product of the embeddings of tions have also been visited in the form of DARTS [21] where
support and query images xs and xq , namely gTv ðxq Þgv ðxs Þ the forward pass during training consists in a softmax across
[81], [114]. Here the support image generates ‘weights’ to inter- the outputs of all possible layers in a given block, which are
pret the query example, making it a special case of a FFM weighted by coefficients to be meta learned (i.e., v). During
where the ‘hypernetwork’ generates a linear classifier for the meta-test, the architecture is discretized by only keeping the
query set. Vanilla methods in this family have been further layers corresponding to the highest coefficients. Recent efforts
enhanced by making the embedding task-conditional [99], to improve DARTS have focused on more efficient differentia-
[115], learning a more elaborate comparison metric [89], [90], ble approximations [129], robustifying the discretization step
[130], learning easy to adapt initializations [131], or architec- initial formalization of meta-learning which considers the
ture priors [132]. See Section 5.3 for more details. source datasets to be fixed (Section 2.1, Eqs. (2) and (3)).
Attention Modules. have been used as comparators in met- However, it can be easily understood in the bilevel view of
ric-based meta-learners [133], to prevent catastrophic for- Eqs. (5) and (6). If the validation set in the upper optimiza-
getting in few-shot continual learning [134] and to summarize tion is real and fixed, and a train set in the lower optimiza-
the distribution of text classification tasks [135]. tion is paramaterized by v, the training dataset can be
Modules. Modular meta-learning [136], [137] assumes that tuned by meta-learning to optimize validation performance.
the task agnostic knowledge v defines a set of modules, In dataset distillation [152], [153], [154], the support
which are re-composed in a task specific manner defined by images themselves are learned such that a few steps on
u in order to solve each encountered task. These strategies them allows for good generalization on real query images.
can be seen as meta-learning generalizations of the typical This can be used to summarize large datasets into a handful
structural approaches to knowledge sharing that are well of images, which is useful for replay in continual learning
studied in multi-task and transfer learning [66], [138], and where streaming datasets cannot be stored.
may ultimately underpin compositional learning [139]. Rather than learning input images x for fixed labels y,
Hyper-Parameters. Here v represents hyperparameters of one can also learn the input labels y for fixed images x. This
the base learner such as regularization strength [20], [69], per- can be used in distilling core sets [155] as in dataset distilla-
parameter regularization [93], task-relatedness in multi-task tion; or semi-supervised learning, for example to directly
learning [67], or sparsity strength in data cleansing [67]. learn the unlabeled set’s labels to optimize validation set
Hyperparameters such as step size [69], [77], [78] can be seen performance [156], [157].
as part of the optimizer, leading to an overlap between hyper- In the case of sim2real learning [158] in computer vision
parameter and optimizer learning categories. or reinforcement learning, one uses an environment simula-
Data Augmentation and Noise. In supervised learning it is tor to generate data for training. In this case, as detailed in
common to improve generalization by synthesizing more Section 5.10, one can also train the graphics engine [159] or
training data through label-preserving transformations on the simulator [160] so as to optimize the real-data (validation)
existing data [13]. The data augmentation operation is performance of the downstream model after training on
wrapped up in the inner problem (Eq. (6)), and is convention- data generated by that environment simulator.
ally hand-designed. However, when v defines the data aug- Discussion: Transductive Representations and Methods. Most of
mentation strategy, it can be learned by the outer optimization the representations v discussed above are parameter vectors
in Eq. (5) in order to maximize validation performance [140]. of functions that process or generate data. However a few rep-
Since augmentation operations are often non-differentiable, resentations mentioned are transductive in the sense that the v
this requires reinforcement learning [140], discrete gradient- literally corresponds to data points [152], labels [156], or per-
estimators [141], or evolutionary [142] methods. Recent instance weights [67], [148]. Therefore the number of parame-
attempts use meta-gradient to learn mixing proportions in ters in v to meta-learn scales as the size of the dataset. While
mixup-based augmentation [143]. For stochastic neural net- the success of these methods is a testament to the capabilities
works that exploit noise internally [116], v can define a learn- of contemporary meta-learning [154], this property may ulti-
able noise distribution. mately limit their scalability.
Minibatch Selection, Instance Weights, and Curriculum Learn- Distinct from a transductive representation are methods
ing. When the base algorithm is minibatch-based stochastic that are transductive in the sense that they operate on the
gradient descent, a design parameter of the learning strategy query instances as well as support instances [99], [126].
is the batch selection process. Various hand-designed meth- Discussion: Interpretable Symbolic Representations. A cross-
ods [144] exist to improve on randomly-sampled minibatches. cutting distinction that can be made across many of the meta-
Meta-learning approaches can define v as an instance selec- representations discussed above is between uninterpretable
tion probability [145] or neural network that picks instances (sub-symbolic) and human interpretable (symbolic) represen-
[146] for inclusion in a minibatch. Related to mini-batch selec- tations. Sub-symbolic representations, such as when v param-
tion policies are methods that learn or infer per-instance loss eterizes a neural network [15], are more common and make up
weights for each sample in the training P set [147], [148]. For the majority of studies cited above. However, meta-learning
example defining a base loss as L ¼ i vi ‘ðfðxi Þ; yi Þ. This with symbolic representations is also possible, where v repre-
can be used to learn under label-noise by discounting noisy sents human readable symbolic functions such as optimization
samples [147], [148], discount outliers [67], or correct for class program code [91]. Rather than neural loss functions [42], one
imbalance [147]. can train symbolic losses v that are defined by an expression
More generally, the curriculum [149] refers to sequences of analogous to cross-entropy [119], [121]. One can also meta-
data or concepts to learn that produce better performance learn new symbolic activations [161] that outperform stand-
than learning items in a random order. For instance by focus- ards such as ReLU. As these meta-representations are non-
ing on instances of the right difficulty while rejecting too hard smooth, the meta-objective is non-differentiable and is harder
or too easy (already learned) instances. Instead of defining a to optimize (see Section 4.2). So the upper optimization for v
curriculum by hand [150], meta-learning can automate the typically uses RL [91] or evolutionary algorithms [119]. How-
process and select examples of the right difficulty by defining ever, symbolic representations may have an advantage [91],
a teaching policy as the meta-knowledge and training it to [119], [161] in their ability to generalize across task families. I.
optimize the student’s progress [146], [151]. e., to span wider distributions pðT Þ with a single v during
Datasets, Labels and Environments. Another meta-repre- meta-training, or to have the learned v generalize to an out of
sentation is the support dataset itself. This departs from our distribution task during meta-testing (see Section 6).
Discussion: Amortization. One way to relate some of the local minima that plague gradient-based methods [127]. How-
representations discussed is in terms of the degree of learn- ever, they have a number of disadvantages: (i) The population
ing amortization entailed [45]. That is, how much task-spe- size required increases rapidly with the number of parameters
cific optimization is performed during meta-testing versus to learn. (ii) They can be sensitive to the mutation strategy and
how much learning is amortized during meta-training. may require careful hyperparameter optimization. (iii) Their
Training from scratch, or conventional fine-tuning [57] per- fitting ability is generally inferior to gradient-based methods,
form full task-specific optimization at meta-testing, with no especially for large models such as CNNs.
amortization. MAML [19] provides limited amortization by EAs are relatively more commonly applied in RL appli-
fitting an initial condition, to enable learning a new task by cations [25], [169] (where models are typically smaller, and
few-step fine-tuning. Pure FFMs [23], [88], [108] are fully inner optimizations are long and non-differentiable). How-
amortized, with no task-specific optimization, and thus ever they have also been applied to learn learning rules
enable the fastest learning of new tasks. Meanwhile some [195], optimizers [196], architectures [27], [127] and data
hybrid approaches [98], [99], [109], [162] implement semi- augmentation strategies [142] in supervised learning. They
amortized learning by drawing on both feed-forward and are also particularly important in learning human interpret-
optimization-based meta-learning in a single framework. able symbolic meta-representations [119].
Discussion. These three optimizers are also all used in
conventional learning. However meta-learning compara-
4.2 Meta-Optimizer tively more often resorts to RL and Evolution, e.g., as Lmeta
Given a choice of which facet of the learning strategy to is often non-differentiable with respect to representation v.
optimize, the next axis of meta-learner design is actual outer
(meta) optimization strategy to use for training v.
Gradient. A large family of methods use gradient descent 4.3 Meta-Objective and Episode Design
on the meta parameters v [14], [19], [42], [67]. This requires The final component is to define the meta-learning goal
computing derivatives dLmeta =dv of the outer objective, through choice of meta-objective Lmeta , and associated data
which are typically connected via the chain rule to the flow between inner loop episodes and outer optimizations.
model parameter u, dLmeta =dv ¼ ðdLmeta =duÞðdu=dvÞ. These Most methods define a meta-objective using a performance
methods are potentially the most efficient as they exploit metric computed on a validation set, after updating the task
analytical gradients of v. However key challenges include: model with v. This is in line with classic validation set
(i) Efficiently differentiating through many steps of inner approaches to hyperparameter and model selection. How-
optimization, for example through careful design of differ- ever, within this framework, there are several design options:
entiation algorithms [20], [69], [190] and implicit differentia- Many versus Few-Shot Episode Design. According to
tion [154], [163], [191], and dealing tractably with the whether the goal is improving few- or many-shot perfor-
required second-order gradients [192]. (ii) Reducing the mance, inner loop learning episodes may be defined with
inevitable gradient degradation problems whose severity many [67], [91], [92] or few- [14], [19] examples per-task.
increases with the number of inner loop optimization steps. Fast Adaptation versus Asymptotic Performance. When vali-
(iii) Calculating gradients when the base learner, v, or Ltask dation loss is computed at the end of the inner learning epi-
include discrete or other non-differentiable operations. sode, meta-training encourages better final performance of
Reinforcement Learning. When the base learner includes the base task. When it is computed as the sum of the valida-
non-differentiable steps [140], or the meta-objective Lmeta is tion loss after each inner optimization step, then meta-train-
itself non-differentiable [123], many methods [24] resort to ing also encourages faster learning in the base task [78], [91],
RL to optimize the outer objective Eq. (5). This estimates the [92]. Most RL applications also use this latter setting.
gradient rv Lmeta , typically using the policy gradient theo- Multi versus Single-Task When the goal is to tune the
rem. However, alleviating the requirement for differentia- learner to better solve any task drawn from a given family,
bility in this way is typically extremely costly. High- then inner loop learning episodes correspond to a randomly
variance policy-gradient estimates for rv Lmeta mean that drawn task from pðT Þ [19], [23], [42]. When the goal is to
many outer-level optimization steps are required to con- tune the learner to simply solve one specific task better,
verge, and each of these steps are themselves costly due to then the inner loop learning episodes all draw data from the
wrapping task-model optimization within them. same underlying task [15], [67], [173], [180], [181], [197].
Evolution. Another approach for optimizing the meta- It is worth noting that these two meta-objectives tend to
objective are evolutionary algorithms (EA) [17], [127], [193]. have different assumptions and value propositions. The
Many evolutionary algorithms have strong connections to multi-task objective obviously requires a task family pðT Þ to
reinforcement learning algorithms [194]. However, their work with, which single-task does not. Meanwhile for
performance does not depend on the length and reward multi-task, the data and compute cost of meta-training can
sparsity of the inner optimization as for RL. be amortized by potentially boosting the performance of
EAs are attractive for several reasons [193]: (i) They can multiple target tasks during meta-test; but single-task –
optimize any base model and meta-objective with no differen- without the new tasks for amortization – needs to improve
tiability constraint. (ii) Not relying on backpropagation avoids the final solution or asymptotic performance of the current
both gradient degradation issues and the cost of high-order task, or meta-learn fast enough to be online.
gradient computation of conventional gradient-based meth- Online versus Offline. While the classic meta-learning pipe-
ods. (iii) They are highly parallelizable for scalability. (iv) By line defines the meta-optimization as an outer-loop of the
maintaining a diverse population of solutions, they can avoid inner base learner [15], [19], some studies have attempted to
TABLE 1
Research Papers According to Our Taxonomy
We use color to indicate salient meta-objective or application goal. We focus on the main goal of each paper for simplicity. The color code is: sample efficiency (red),
learning speed (green), asymptotic performance (purple), cross-domain (blue).
preform meta-optimization online within a single base learn- Classification. The most common application of meta-learn-
ing episode [42], [180], [197], [198]. In this case the base model ing is few-shot multi-class image recognition, where the inner
u and learner v co-evolve during a single episode. Since there and outer loss functions are typically the cross entropy over
is now no set of source tasks to amortize over, meta-learning training and validation data respectively [14], [23], [75], [77],
needs to be fast compared to base model learning in order to [78], [88], [90], [98], [99], [102], [105], [201], [202], [203], [204].
benefit sample or compute efficiency. Optimizer-centric [19], black-box [39], [81] and metric learning
Other Episode Design Factors. Other operators can be [88], [89], [90] models have all been considered.
inserted into the episode generation pipeline to customize This line of work has led to a steady improvement in per-
meta-learning for particular applications. For example one formance compared to early methods [19], [87], [88]. How-
can simulate domain-shift between training and validation ever, performance is still far behind that of fully supervised
to meta-optimize for good performance under domain-shift methods, so there is more work to be done. Current research
[42], [59], [93]; simulate network compression such as quan- issues include improving cross-domain generalization [116],
tization [199] between training and validation to meta-opti- recognition within the joint label space defined by meta-
mize for network compressibility; provide noisy labels train and meta-test classes [82], and incremental addition of
during meta-training to optimize for label-noise robustness new few-shot classes [134], [175].
[94], or generate an adversarial validation set to meta-opti- Object Detection. Building on progress in few-shot clas-
mize for adversarial defense [95]. These opportunities are sification, few-shot object detection [175], [205] has been
explored in more detail in the following section. demonstrated, often using feed-forward hypernetwork-
based approaches to embed support set images and syn-
5 APPLICATIONS thesize final layer classification weights in the base model.
Landmark Prediction. aims to locate a skeleton of key
In this section we briefly review the ways in which meta-
points within an image, such as such as joints of a human or
learning has been exploited in computer vision, reinforce-
robot. This is typically formulated as an image-conditional
ment learning, architecture search, and so on.
regression. For example, a MAML-based model was shown
to work for human pose estimation [206], modular-meta-
5.1 Computer Vision and Graphics
learning was successfully applied to robotics [136], while a
Computer vision is a major consumer domain of meta-learn- hypernetwork-based model was applied to few-shot clothes
ing techniques, notably due to its impact on few-shot learning, fitting for novel fashion items [175].
which holds promise to deal with the challenge posed by the Few-Shot Object Segmentation. is important due to the cost
long-tail of concepts to recognise in vision. of obtaining pixel-wise labeled images. Hypernetwork-
based meta-learners have been applied in the one-shot
5.1.1 Few-Shot Learning Methods regime [207], and performance was later improved by
Few-shot learning (FSL) is extremely challenging, especially adapting prototypical networks [208]. Other models tackle
for large neural networks [1], [13], where data volume is cases where segmentation has low density [209].
often the dominant factor in performance [200], and training Actions. Beyond still images, meta-learning has been
large models with small datasets leads to overfitting or non- applied to few-shot action recognition [210], [211] and predic-
convergence. Meta-learning-based approaches are increas- tion [212] using both FFM [210] and optimization-based [211],
ingly able to train powerful CNNs on small datasets in [212] approaches.
many vision problems. We provide a non-exhaustive repre- Image and Video Generation. In [45] an amortized probabi-
sentative summary as follows. listic meta-learner is used to generate multiple views of an
object from just a single image, generative query networks due to sparse rewards, the need for exploration, and the
[213] render scenes from novel views, and talking faces are high-variance [224] of optimization algorithms. However,
generated from little data by learning the initialization of an applications often naturally entail task families which meta-
adversarial model for quick adaptation [214]. In video learning can exploit – for example locomoting-to or reach-
domain, [215] meta-learns a weight generator that synthe- ing-to different positions [185], navigating within different
sizes videos given few example images as cues. Meta-learn- environments [39], traversing different terrains [64], driving
ing has also accelerated style transfer by learning a FFM to different cars [184], competing with different competitor
solve the corresponding optimization problem [216]. agents [62], and dealing with different handicaps such as
Generative Models and Density Estimation. Density estima- failures in individual robot limbs [64]. Thus RL provides a
tors capable of generating images typically require many fertile application area in which meta-learning on task dis-
parameters, and as such overfit in the few-shot regime. Gra- tributions has had significant successes in improving sam-
dient-based meta-learning of PixelCNN generators was ple efficiency over standard RL algorithms. One can
shown to enable their few-shot learning [217]. intuitively understand the efficacy of these methods. For
instance meta-knowledge of a maze layout is transferable
for all tasks that require navigating within the maze.
5.1.2 Few-Shot Learning Benchmarks
Problem Setup. Meta-RL uses essentially the same prob-
Progress in AI and machine learning is often measured, and lem setup outlined in Section 2.1, but in the setting of
spurred, by well designed benchmarks [218]. Conventional sequential decision making within Markov decision pro-
ML benchmarks define a task and dataset for which a model cesses (MDPs). A conventional RL problem is defined by an
should generalize from seen to unseen instances. In meta- MDP M, in which policy pu chooses actions at based on
learning, benchmark design is more complex, since we are states st and obtains rewards rt . The policy is P trained to
often dealing with a learner that should generalize from maximize the expected cumulative return Rt ¼ Tt g t rt of
seen to unseen tasks. Benchmark design thus needs to define an episode t M; t ¼ ðs0 ; a0 ; . . . ; sT ; aT ; rT Þ after T time-
families of tasks from which meta-training and meta-testing steps, u ¼ arg maxu EtM;pu ½Rt . Meta-RL, usually assumes
tasks can be drawn. Established FSL benchmarks include a distribution over MDPs pðMÞ, analogous to the distribu-
miniImageNet [14], [88], Tiered-ImageNet [219], Slimage- tion over tasks in Section 2.1. Some aspect of the learning pro-
Net [220], Omniglot [88] and Meta-Dataset [109]. cess for policy u is then paramaterized by meta-knowledge v,
Dataset Diversity, Bias and Generalization. The standard for example the initial condition [19], or surrogate reward/
benchmarks provide tasks for training and evaluation, but loss function [22], [25]. Then the goal of Meta-RL is to train the
suffer from a lack of diversity (narrow pðT Þ); performance meta-representation v so as to maximize the expected return
on these benchmarks are not reflective of performance on of learning within the MDP distribution pðMÞ:
real-world few shot tasks. For example, switching between
kinds of animal photos in miniImageNet is not a strong test
v ¼ arg max EMpðMÞ EtM;pu ½Rt : (9)
of generalization. Ideally bechmarks would span more v
diverse categories and types of images (satellite, medical,
agricultural, underwater, etc), and even test domain-shifts This is the RL version of the meta-training problem in
between meta-train and meta-test tasks. Eq. (2), and solutions are similarly based on bi-level optimi-
There is work still to be done here as, even in the many- zation (e.g., [19], cf: Eqs 5-6), or feed-forward approaches
shot setting, fitting a deep model to a very wide distribution (e.g., [39], cf: Eq. (7)). Finally, once trained, the meta-learner
of data is itself non-trivial [221], as is generalizing to out-of- v can also be deployed to rapidly learn in a new MDP M
sample data [42], [93]. Similarly, the performance of meta- pðMÞ analogously to the meta-test step in Eq. (4).
learners often drops drastically when introducing a domain
shift between the source and target task distributions [114]. 5.2.1 Methods
This motivates the recent Meta-Dataset [109] and CVPR Several previously discussed meta-representations have been
cross-domain few-shot challenge [222]. Meta-Dataset aggre- explored in RL including learning the initial conditions [19],
gates a number of individual recognition benchmarks to [170], hyperparameters [170], [174], step directions [77] and
provide a wider distribution of tasks pðT Þ to evaluate the step sizes [172]. These enable gradient-based learning of a
ability to fit a wide task distribution and generalize across neural policy with fewer environmental interactions and
domain-shift. Meanwhile, [222] challenges methods to gen- training fast convolutional [39] or recurrent [24], [113] black-
eralize from the everyday ImageNet images to medical, sat- box models to synthesize a policy by embedding the enviro-
ellite and agricultural images. Recent work has begun to try ment experience. Recent work has developed improved
and address these issues by meta-training for domain-shift meta-optimization algorithms [167], [168], [169] for these
robustness as well as sample efficiency [116]. Generalization tasks, and provided theoretical guarantees for meta-RL [225].
issues also arise in applying models to data from under-rep- Exploration. A meta-representation rather unique to RL is
resented countries [223]. the exploration policy. RL is complicated by the fact that the
data distribution is not fixed, but varies according to the
5.2 Meta Reinforcement Learning and Robotics agent’s actions. Furthermore, sparse rewards may mean
Reinforcement learning is typically concerned with learning that an agent must take many actions before achieving a
control policies that enable an agent to obtain high reward reward that can be used to guide learning. As such, how to
after performing a sequential action task within an environ- explore and acquire data for learning is a crucial factor in
ment. RL typically suffers from extreme sample inefficiency any RL algorithm. Traditionally exploration is based on
sampling random actions [226], or hand-crafted heuristics 5.2.2 Benchmarks

[227]. Several meta-RL studies have instead explicitly Meta-learning benchmarks for RL typically define a family
treated exploration strategy or curiosity function as meta- to solve in order to train and evaluate an agent that learns
knowledge v; and modeled their acquisition as a meta- how to learn. These can be tasks (reward functions) to
learning problem [26], [183], [184], [228] – leading to sample achieve, or domains (distinct environments or MDPs).
efficiency improvements by ‘learning how to explore’. Discrete Control RL. An early meta-RL benchmark for
Optimization. RL is a difficult optimization problem vision-actuated control is the arcade learning environment
where the learned policy is usually far from optimal, even (ALE) [235], which defines a set of classic Atari games split
on ‘training set’ episodes. This means that, in contrast to into meta-training and meta-testing. The protocol here is to
meta-SL, meta-RL methods are more commonly deployed evaluate return after a fixed number of timesteps in the meta-
to increase asymptotic performance [25], [174], [180] as well test environment. A challenge is the great diversity (wide
as sample-efficiency, and can lead to significantly better sol- pðT Þ) across games, which makes successful meta-training
utions overall. The meta-objective of many meta-RL frame- hard and leads to limited benefit from knowledge transfer
works is the net return of the agent over a full episode, and [235]. Another benchmark [236] is based on splitting Sonic-
thus both sample efficient and asymptotically performant hedgehog levels into meta-train/meta-test. The task distribu-
learning are rewarded. Optimization difficulty also means tion here is narrower and beneficial meta-learning is relatively
that there has been relatively more work on learning losses easier to achieve. Cobbe et al.[237] proposed two purpose
(or rewards) [22], [120], [180], [229] which an RL agent designed video games for benchmarking meta-RL. CoinRun
should optimize instead of – or in addition to – the conven- game [237] provides 232 procedurally generated levels of
tional sparse reward objective. Such learned losses may be varying difficulty and visual appearance. They show that
easier to optimize (denser, smoother) compared to the true some 10,000 levels of meta-train experience are required
target [25], [229]. This also links to exploration as reward to generalize reliably to new levels. CoinRun is primarily
learning and can be considered to instantiate meta-learning designed to test direct generalization rather than fast adapta-
of learning intrinsic motivation [181]. tion, and can be seen as providing a distribution over MDP
Online Meta-RL A significant fraction of meta-RL studies environments to test generalization rather than over tasks to
addressed the single-task setting, where the meta-knowl- test adaptation. To better test fast learning in a wider task dis-
edge such as loss [22], [180], reward [174], [181], hyperpara- tribution, ProcGen [237] provides a set of 16 procedurally gen-
meters [172], [173], or exploration strategy [182] are trained erated games including CoinRun.
online together with the base policy while learning a single Continuous Control RL. While common benchmarks such as
task. These methods thus do not require task families and gym [238] have greatly benefited RL research, there is less con-
provide a direct improvement to their respective base sensus on meta-RL benchmarks, making existing work hard to
learners’ performance. compare. Most continuous control meta-RL studies have pro-
On- versus Off-Policy Meta-RL. A major dichotomy in con- posed home-brewed benchmarks that are low dimensional
ventional RL is between on-policy and off-policy learning parametric variants of particular tasks such as navigating to
such as PPO [226] versus SAC [230]. Off-policy methods are various locations or velocities [19], [111], or traversing different
usually significantly more sample efficient. However, off-pol- terrains [64]. Several multi-MDP benchmarks [239], [240] have
icy methods have been harder to extend to meta-RL, leading recently been proposed but these primarily test generalization
to more meta-RL methods being built on on-policy RL meth- across different environmental perturbations rather than dif-
ods, thus limiting the absolute performance of meta-RL. Early ferent tasks. The Meta-World benchmark [241] provides a suite
work in off-policy meta-RL methods has led to strong results of 50 continuous control tasks with state-based actuation, vary-
[22], [111], [166], [229]. Off-policy learning also improves the ing from simple parametric variants such as lever-pulling and
efficiency of the meta-train stage [111], which can be expen- door-opening. This benchmark should enable more compara-
sive in meta-RL. It also provides new opportunities to acceler- ble evaluation, and investigation of generalization within and
ate meta-testing by replay buffer sample from meta-training across task distributions. The meta-world evaluation [241] sug-
[166]. gests that existing meta-RL methods struggle to generalize
Other Trends and Challenges. [64] is noteworthy in demon- over wide task distributions and meta-train/meta-test shifts.
strating successful meta-RL on a real-world physical robot. This may be due to our meta-RL models being too weak and/
Knowledge transfer in robotics is often best studied composi- or benchmarks being too small, in terms of number and cover-
tionally [231]. E.g., walking, navigating and object pick/place age tasks, for effective learning-to-learn. Another recent bench-
may be subroutines for a room cleaning robot. However, mark suitable for meta-RL is PHYRE [242] which provides a
developing meta-learners with effective compositional knowl- set of 50 vision-based physics task templates which can be
edge transfer is an open question, with modular meta-learning solved with simple actions but are likely to require model-
[137] being an option. Unsupervised meta-RL variants aim to based reasoning to address efficiently. These also provide
perform meta-training without manually specified rewards within and cross-template generalization tests.
[232], or adapt at meta-testing to a changed environment but Discussion. One complication of vision-actuated meta-RL
without new rewards [233]. Continual adaptation provides an is disentangling visual generalization (as in computer
agent with the ability to adapt to a sequence of tasks within vision) with fast learning of control strategies more gener-
one meta-test episode [62], [63], [64], similar to continual learn- ally. For example CoinRun [237] evaluation showed large
ing. Finally, meta-learning has also been applied to imitation benefit from standard vision techniques such as batch norm
[112] and inverse RL [234]. suggesting that perception is a major bottleneck.
5.3 Neural Architecture Search (NAS) 5.5 Bayesian Meta-Learning

Architecture search [21], [27], [28], [38], [127] can be seen as a Bayesian meta-learning approaches formalize meta-learn-
kind of hyperparameter optimization where v specifies the ing via Bayesian hierarchical modelling, and use Bayesian
architecture of a neural network. The inner optimization inference for learning rather than direct optimization of
trains networks with the specified architecture, and the outer parameters. In the meta-learning context, Bayesian learning
optimization searches for architectures with good validation is typically intractable, and so approximations such as sto-
performance. NAS methods have been analysed [38] accord- chastic variational inference or sampling are used.
ing to ‘search space’, ‘search strategy’, and ‘performance esti- Bayesian meta-learning importantly provides uncer-
mation strategy’. These correspond to the hypothesis space tainty measures for the v parameters, and hence measures
for v, the meta-optimization strategy, and the meta-objective. of prediction uncertainty which can be important for safety
NAS is particularly challenging because: (i) Fully evaluating critical applications, exploration in RL, and active learning.
the inner loop is expensive since it requires training a many- A number of authors have explored Bayesian approaches to
shot neural network to completion. This leads to approxima- meta-learning complex neural network models with competi-
tions such as sub-sampling the train set, early termination of tive results. For example, extending variational autoencoders
the inner loop, and interleaved descent on both v and u [21] as to model task variables explicitly [73]. Neural Processes [176]
in online meta-learning. (ii.) The search space is hard to define, define a feed-forward Bayesian meta-learner inspired by
and optimize. This is because most search spaces are broad, Gaussian Processes but implemented with neural networks.
and the space of architectures is not trivially differentiable. Deep kernel learning is also an active research area that has
This leads to reliance on cell-level search [21], [28] constrain- been adapted to the meta-learning setting [246], and is often
ing the search space, RL [28], discrete gradient estimators coupled with Gaussian Processes [247]. In [74] gradient based
[129] and evolution [27], [127]. meta-learning is recast into a hierarchical empirical Bayes
Topical Issues. While NAS itself can be seen as an inference problem (i.e., prior learning), which models uncer-
instance of hyper-parameter or hypothesis-class meta- tainty in task-specific parameters u. Bayesian MAML [248]
learning, it can also interact with meta-learning in other improves on this model by using a Bayesian ensemble
forms. Since NAS is costly, a topical issue is whether dis- approach that allows non-Gaussian posteriors over u, and later
covered architectures can generalize to new problems work removes the need for costly ensembles [45], [249]. In
[243]. Meta-training across multiple datasets may lead to Probabilistic MAML [96], it is the uncertainty in the meta-
improved cross-task generalization of architectures [132]. knowledge v that is modelled, while a MAP estimate is used
Finally, one can also define NAS meta-objectives to train for u. Increasingly, these Bayesian methods are shown to tackle
an architecture suitable for few-shot learning [244]. Simi- ambiguous tasks, active learning and RL problems.
larly to fast-adapting initial condition meta-learning Separate from the above, meta-learning has also been
approaches such as MAML [19], one can train good initial proposed to aid the Bayesian inference process itself, as in
architectures [131] or architecture priors [132] that are [250] where the authors adapt a Bayesian sampler to pro-
easy to adapt towards specific tasks. vide efficient adaptive sampling methods.
Benchmarks NAS is often evaluated on CIFAR-10, but it is
costly to perform and results are hard to reproduce due to 5.6 Unsupervised and Semi-Supervised
confounding factors such as tuning of hyperparameters. To Meta-Learning
support reproducible and accessible research, the NAS- There are several distinct ways in which unsupervised learn-
benches [245] provide pre-computed performance measures ing can interact with meta-learning, depending on whether
for a large number of network architectures. unsupervised learning in performed in the inner loop or outer
loop, and during meta-train versus meta-test.
5.4 Hyper-Parameter Optimization Unsupervised Learning of a Supervised Learner. The aim
Meta-learning address hyperparameter optimization when here is to learn a supervised learning algorithm (e.g., via
considering v to specify hyperparameters, such as regulari- MAML [19] style initial condition for supervised fine-tun-
zation strength or learning rate. There are two main settings: ing), but do so without the requirement of a large set of
we can learn hyperparameters that improve training over a source tasks for meta-training [251], [252], [253]. To this
distribution of tasks, just a single task. The former case is end, synthetic source tasks are constructed without supervi-
usually relevant in few-shot applications, especially in opti- sion via clustering or class-preserving data augmentation,
mization based methods. For instance, MAML can be and used to define the meta-objective for meta-training.
improved by learning a learning rate per layer per step [78]. Supervised Learning of an Unsupervised Learner. This family of
The case where we wish to learn hyperparameters for a sin- methods aims to meta-train an unsupervised learner. For
gle task is usually more relevant for many-shot applications example, by training the unsupervised algorithm such that it
[69], [154], where some validation data can be extracted works well for downstream supervised learning tasks. One
from the training dataset, as discussed in Section 2.1. End- can train unsupervised learning rules [16] or losses [99], [122]
to-end gradient-based meta-learning has already demon- such that downstream supervised learning performance is
strated promising scalability to millions of parameters (as optimized – after re-using the unsupervised representation for
demonstrated by MAML [19] and Dataset Distillation [152], a supervised task [16], or adapting based on unlabeled data
[154], for example) in contrast to the classic approaches [99], [122]. Alternatively, when unsupervised tasks such as
(such cross-validation by grid or random [70] search, or clustering exist in a family, rather than in isolation, then learn-
Bayesian Optimization [71]) which are typically only suc- ing-to-learn of ‘how-to-cluster’ on several source tasks can pro-
cessful with dozens of hyper-parameters. vide better performance on new clustering tasks in the family
[177], [178], [179], [254], [255]. The methods in this group that 5.8 Domain Adaptation and Domain Generalization
make use of feed-forward models are often known as amortized Domain-shift refers to the statistics of data encountered in
clustering [178], [179], because they amortize the typically itera- deployment being different from those used in training.
tive computation of clustering algorithms into the cost of train- Numerous domain adaptation and generalization algorithms
ing a single inference model, which subsequently performs have been studied to address this issue in supervised, unsu-
clustering using a single feed-froward pass. Overall, these pervised, and semi-supervised settings [58].
methods help to deal with the ill-definedness of the unsuper- Domain Generalization. Domain generalization aims to train
vised learning problem by transforming it into a problem with models with increased robustness to train-test domain shift
a clear supervised (meta) objective. [260], often by exploiting a distribution over training domains.
Semi-Supervised Learning (SSL). This family of methods Using a validation domain that is shifted with respect to the
aim to train a base-learner capable of performing well on training domain [261], different kinds of meta-knowledge
validation data after learning from a mixture of labeled and such as regularizers [93], losses [42], and noise augmentation
unlabeled training examples. In the few-shot regime, [116] can be (meta) learned to maximize the robustness of the
approaches include SSL extensions of metric-learners [219], learned model to train-test domain-shift.
and meta-learning measures of confidence [256] or instance Domain Adaptation. To improve on conventional domain
weights [157] to ensure that reliably labeled instances are adaptation[58], meta-learning can be used to define a meta-
used in self-training. In the many-shot regime, approaches objective that optimizes the performance of a base unsuper-
include direct transductive training of labels [156] , or train- vised DA algorithm [59].
ing a teacher network to generate labels for a student [257]. Benchmarks. Popular benchmarks for DA and DG con-
sider image recognition across multiple domains such as
photo/sketch/cartoon. PACS [262] provides a good starter
5.7 Continual, Online and Adaptive Learning benchmark, with Visual Decathlon [42], [221] and Meta-
Continual Learning. Refers to the human-like capability of Dataset [109] providing larger scale alternatives.
learning tasks presented in sequence. Ideally this is done while
exploiting forward transfer so new tasks are learned better 5.9 Language and Speech
given past experience, without forgetting previously learned
Language Modelling. Few-shot language modelling increas-
tasks, and without needing to store past data [61]. Deep Neural
ingly showcases the versatility of meta-learners. Early
Networks struggle to meet these criteria, especially as they
matching networks showed impressive performances on
tend to forget information seen in earlier tasks – a phenome-
one-shot tasks such as filling in missing words [88]. Many
non known as catastrophic forgetting. Meta-learning can include
more tasks have since been tackled, including text classifica-
the requirements of continual learning into a meta-objective,
tion [135], neural program induction [263] and synthesis
for example by defining a sequence of learning episodes in
[264], English to SQL program synthesis [265], text-based
which the support set contains one new task, but the query set
relationship graph extractor [266], machine translation
contains examples drawn from all tasks seen until now [105],
[267], and quickly adapting to new personas in dialogue
[171]. Various meta-representations can be learned to improve
[268].
continual learning performance, such as weight priors [134],
Speech Recognition. Deep learning is now the dominant
gradient descent preconditioning matrices [105], or RNN
paradigm for state of the art automatic speech recognition
learned optimizers [171], or feature representations [258]. A
(ASR). Meta-learning is beginning to be applied to address
related idea is meta-training representations to support local
the many few-shot adaptation problems that arise within
editing updates [259] for improvement without interference.
ASR including learning how to train for low-resource lan-
Online and Adaptive Learning Also consider tasks arriving
guages [269], cross-accent adaptation [270] and optimizing
in a stream, but are concerned with the ability to effectively
models for individual speakers [271].
adapt to the current task in the stream, more than remem-
bering the old tasks. To this end an online extension of
MAML was proposed [97] to perform MAML-style meta- 5.10 Emerging Topics
training online during a task sequence. Meanwhile others Environment Learning and Sim2Real. In Sim2Real we are
[62], [63], [64] consider the setting where meta-training is interested in training a model in simulation that is able to
performed in advance on source tasks, before meta-testing generalize to the real-world. The classic domain randomiza-
adaptation capabilities on a sequence of target tasks. tion approach simulates a wide distribution over domains/
Benchmarks. A number of benchmarks for continual learn- MDPs, with the aim of training a sufficiently robust model
ing work quite well with standard deep learning methods. to succeed in the real world – and has succeeded in both
However, most cannot readily work with meta-learning vision [272] and RL [158]. Nevertheless tuning the simula-
approaches as their their sample generation routines do not tion distribution remains a challenge. This leads to a meta-
provide a large number of explicit learning sets and an explicit learning setup where the inner-level optimization learns a
evaluation sets. Some early steps were made towards defining model in simulation, the outer-level optimization Lmeta
meta-learning ready continual benchmarks in [97], [171], evaluates the model’s performance in the real-world, and
[258], mainly composed of Omniglot and perturbed versions the meta-representation v corresponds to the parameters of
of MNIST. However, most of those were simply tasks built to the simulation environment. This paradigm has been used
demonstrate a method. More explicit benchmark work can be in RL [160] as well as vision [159], [273]. In this case the
found in [220], which is built for meta and non meta-learning source tasks used for meta-train tasks are not a pre-pro-
approaches alike. vided data distribution, but paramaterized by omega,
D source ðvÞ. However, challenges remain in terms of costly Learning with Label Noise. Commonly arises when large
back-propagation through a long graph of inner task learn- datasets are collected by web scraping or crowd-sourcing.
ing steps; as well as minimising the number of real-world While there are many algorithms hand-designed for this sit-
Lmeta evaluations in the case of Sim2Real. uation, recent meta-learning methods have addressed label
Meta-Learning for Social Good. Meta-learning lends itself to noise. For example by transductively learning sample-wise
challenges that arise in applications of AI for social good weighs to down-weight noisy samples [147], or learning an
such as medical image classification and drug discovery, initial condition robust to noisy label training [94].
where data is often scarce. Progress in the medical domain Adversarial Attacks and Defenses. Deep Neural Networks
is especially relevant given the global shortage of patholo- can be fooled into misclassifying a data point that should be
gists [274]. In [5] an LSTM is combined with a graph neural easily recognizable, by adding a carefully crafted human-
network to predict molecular behaviour (e.g., toxicity) in invisible perturbation to the data [285]. Numerous attack
the one-shot data regime. In [275] MAML is adapted to and defense methods have been published in recent years,
weakly-supervised breast cancer detection tasks, and the with defense strategies usually consisting in carefully hand-
order of tasks are selected according to a curriculum. designed architectures or training algorithms. Analogous to
MAML is also combined with denoising autoencoders to do the case in domain-shift, one can train the learning algo-
medical visual question answering [276], while learning to rithm for robustness by defining a meta-loss in terms of per-
weigh support samples [219] is adapted to pixel wise formance under adversarial attack [95], [286].
weighting for skin lesion segmentation tasks that have noisy Recommendation Systems. Are a mature consumer of
labels [277]. machine learning in the commerce space. However, boot-
Non-Backprop and Biologically Plausible Learners. Most strapping recommendations for new users with little histor-
meta-learning work that uses explicit (non feed-forward/ ical interaction data, or new items for recommendation
black-box) optimization for the base model is based on gra- remains a challenge known as the cold-start problem. Meta-
dient descent by backpropagation. Meta-learning can define learning has applied black-box models to item cold-start
the function class of v so as to lead to the discovery of novel [287] and gradient-based methods to user cold-start [288].
learning rules that are unsupervised [16] or biologically
plausible [46], [278], [279], making use of ideas less com-
monly used in contemporary deep learning such as Heb- 6 CHALLENGES AND OPEN QUESTIONS
bian updates [278] and neuromodulation [279]. Diverse and Multi-Modal Task Distributions The difficulty of
Network Compression. Contemporary CNNs require large fitting a meta-learner to a distribution of tasks pðT Þ can
amounts of memory that may be prohibitive on embedded depend on its width. Many big successes of meta-learning
devices. Thus network compression in various forms such have been within narrow task families, while learning on
as quantization and pruning are topical research areas diverse task distributions can challenge existing methods
[280]. Meta-learning is beginning to be applied to this objec- [109], [221], [241]. This may be partly due to conflicting gra-
tive as well, such as training gradient generator meta-net- dients between tasks [289].
works that allow quantized networks to be trained [199], Many meta-learning frameworks [19] implicitly assume
and weight generator meta-networks that allow quantized that the distribution over tasks pðT Þ is uni-modal, and a single
networks to be trained with gradient [281]. learning strategy v provides a good solution for them all.
Communications. systems are increasingly impacted by However task distributions are often multi-modal; such as
deep learning. For example by learning coding systems that medical versus satellite versus everyday images in computer
surpass hand designed codes for realistic channels [282]. vision, or putting pegs in holes versus opening doors [241] in
Few-shot meta-learning can be used to provide rapid adap- robotics. Different tasks within the distribution may require
tation of codes to changing channel characteristics [283]. different learning strategies, which is hard to achieve with
Active Learning (AL). Methods wrap supervised learning, today’s methods. In vanilla multi-task learning, this phenom-
and define a policy for selective data annotation – typically enon is relatively well studied with, e.g., methods that group
in the setting where annotation can be obtained sequen- tasks into clusters [290] or subspaces [291]. However this is
tially. The goal of AL is to find the optimal subset of data to only just beginning to be explored in meta-learning [292].
annotate so as to maximize performance of downstream Meta-Generalization. Meta-learning poses a new generaliza-
supervised learning with the fewest annotations. AL is a tion challenge across tasks analogous to the challenge of gen-
well studied problem with numerous hand designed algo- eralizing across instances in conventional machine learning.
rithms [284]. Meta-learning can map active learning algo- There are two sub-challenges: (i) The first is generalizing from
rithm design into a learning task by: (i) defining the inner- meta-train to novel meta-test tasks drawn from pðT Þ. This is
level optimization as conventional supervised learning on exacerbated because the number of tasks available for meta-
the annotated dataset so far, (ii) defining v to be a query pol- training is typically low (much less than the number of instan-
icy that selects the best unlabeled datapoints to annotate, ces available in conventional supervised learning), making it
(iii), defining the meta-objective as validation performance difficult to generalize. One failure mode for generalization in
after iterative learning and annotation according to the few-shot learning has been well studied under the guise of
query policy, (iv) performing outer-level optimization to memorisation [201], which occurs when each meta-training
train the optimal annotation query policy [187], [188], [189]. task can be solved directly without performing any task-spe-
However, if labels are used to train AL algorithms, they cific adaptation based on the support set. In this case models
need to generalize across tasks to amortize their training fail to generalize in meta-testing, and specific regularizers
cost [189]. [201] have been proposed to prevent this kind of meta-
overfitting. (ii) The second challenge is generalizing to meta- devices such as smartphones [6], for example to achieve per-
test tasks drawn from a different distribution than the training sonalisation. This is especially so because the embedded
tasks. This is inevitable in many potential practical applica- device versions of contemporary deep learning software
tions of meta-learning, for example generalizing few-shot frameworks typically lack support for backpropagation-based
visual learning from everyday training images of ImageNet to training, which FFMs do not require.
specialist domains such as medical images [222]. From the
perspective of a learner, this is a meta-level generalization of
the domain-shift problem, as observed in supervised learning. 7 CONCLUSION
Addressing these issues through meta-generalizations of reg-
ularization, transfer learning, domain adaptation, and domain The field of meta-learning has seen a rapid growth in inter-
generalization are emerging directions [116]. Furthermore, we est. This has come with some level of confusion, with
have yet to understand which kinds of meta-representations regards to how it relates to neighbouring fields, what it can
tend to generalize better under certain types of domain shifts. be applied to, and how it can be benchmarked. In this sur-
Task Families. Many existing meta-learning frameworks, vey we have sought to clarify these issues by thoroughly
especially for few-shot learning, require task families for meta- surveying the area both from a methodological point of
training. While this indeed reflects lifelong human learning, in view – which we broke down into a taxonomy of meta-
some applications data for such task families may not be avail- representation, meta-optimizer and meta-objective; and
able. Unsupervised meta-learning [251], [252], [253] and sin- from an application point of view. We hope that this survey
gle-task meta-learning methods [42], [173], [180], [181], [197], will help newcomers and practitioners to orient themselves
could help to alleviate this requirement; as can improvements to develop and exploit in this growing field, as well as high-
in meta-generalization discussed above. light opportunities for future research.
Computation Cost & Many-Shot. A naive implementation
of bilevel optimization as shown in Section 2.1 is expensive ACKNOWLEDGMENTS
in both time (because each outer step requires several inner The work of Timothy Hospedales was supported in part by
steps) and memory (because reverse-mode differentiation the Engineering and Physical Sciences Research Council of
requires storing the intermediate inner states). For this rea- the U.K. (EPSRC) under Grant EP/S000631/1, in part by the
son, much of meta-learning has focused on the few-shot U.K. MOD University Defence Research Collaboration
regime [19], where the number of inner gradient steps (aka (UDRC) in Signal Processing, and in part by the EPSRC
the horizon) is small. However, there is an increasing focus under Grant EP/R026173/1.
on methods which seek to extend optimization-based meta-
learning to the many-shot regime, where long horizons are
REFERENCES
inevitable. Popular solutions include implicit differentiation
[1] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for
of v [154], [163], [293], forward-mode differentiation of v
image recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Rec-
[67], [69], [294], gradient preconditioning [105], evolution- ognit., 2016, pp. 770–778.
ary strategies [295], solving for a greedy version of v online [2] D. Silver et al., “Mastering the game of go with deep neural net-
by alternating inner and outer steps [21], [42], [198], trunca- works and tree search,” Nature, vol. 529, pp. 484–489, 2016.
[3] J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: Pre-
tion [296], shortcuts [297] or inversion [190] of the inner training of deep bidirectional transformers for language
optimization. Long horizon meta-learning can also be understanding,” in Proc. Assoc. Comput. Linguistics, 2019,
achieved by learning an initialization that minimizes the pp. 4171–4186.
gradient descent trajectory length over task manifolds [298]. [4] G. Marcus, “Deep learning: A critical pppraisal,” 2018, arXiv:
1801.00631.
Finally, another family of approaches accelerate meta-train- [5] H. Altae-Tran, B. Ramsundar, A. S. Pappu, and V. Pande, “Low
ing via closed-form solvers in the inner loop [164], [165]. data drug discovery with one-shot learning,” ACS Central Sci.,
Each of these strategies provides different trade-offs on vol. 3, no. 4, pp. 283–293, 2017.
[6] A. Ignatov et al. “AI benchmark: All about deep learning on
the following axes: the accuracy at meta-test time, the com- smartphones in 2019,” 2019, arXiv: 1910.06663.
pute and memory cost as the size of v increases, and the [7] S. Thrun and L. Pratt, “Learning to learn: Introduction and over-
computate and memory cost as the horizon gets longer. view,”in, Learning to Learn. Boston, MA, USA: Springer, 1998.
Implicit gradients scale to high-dimensional v; but only pro- [8] H. F. Harlow, “The formation of learning sets,” Psychol. Rev., vol. 56,
pp. 51–65, 1949.
vide approximate gradients for it, and require the inner task [9] J. B. Biggs, “The role of meta-learning in study processes,” Brit. J.
loss to be a function of v. Forward-mode differentiation is Educ. Psychol., vol. 55, pp. 185–212, 1985.
exact and doesn’t have such constraints, but scales poorly [10] A. M. Schrier, “Learning how to learn: The significance and cur-
rent status of learning set formation,” Primates, vol. 25, pp. 95–102,
with the dimension of v. Online methods are cheap in both
1984.
memory and compute but suffer from a short-horizon bias [11] P. Domingos, “A few useful things to know about machine
[299] and thus have lower meta-test accuracy. The same learning,” in Proc. Commun. ACM, 2012, pp. 78–87.
could be said of truncation methods which are cheap but [12] D. G. Lowe, “Distinctive image features from scale-invariant,”
Int. J. Comput. Vis., vol. 60, pp. 91–110, 2004.
suffer from truncation bias [300]. Gradient degradation is [13] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classifi-
also a challenge in the many-shot regime, and solutions cation with deep convolutional neural networks,” in Proc. Com-
include warp layers [105] or gradient averaging [69]. mun. ACM, 2012, pp. 84–90.
In terms of the cost of solving new tasks at the meta-test [14] S. Ravi and H. Larochelle, “Optimization as a model for few-shot
learning,” in Proc. Int. Conf. Learn. Representations, 2016.
stage, FFMs have a significant advantage over optimization- [15] M. Andrychowicz et al., “Learning to learn by gradient descent
based meta-learners, which makes them appealing for applica- by gradient descent,” in Proc. 30th Int. Conf. Neural Inf. Process.
tions involving deployment of learning algorithms on mobile Syst., 2016, pp. 3988–3996.
[16] L. Metz, N. Maheswaranathan, B. Cheung, and J. Sohl-Dickstein, [44] M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Poczos, R. R. Salakhut-
“Meta-learning update rules for unsupervised representation dinov, and A. J. Smola, “Deep sets,” in Proc. 31st Int. Conf. Neural
learning,” in Proc. Int. Conf. Learn. Representations, 2019. Inf. Process. Syst., 2017, pp. 3394–3404.
[17] J. Schmidhuber, “Evolutionary principles in self-referential [45] J. Gordon, J. Bronskill, M. Bauer, S. Nowozin, and R. E. Turner,
learning. On learning now to learn: The meta-meta-meta...- “Meta-learning probabilistic inference for prediction,” Proc. Int.
hook,” Diploma thesis, Technische Universitat Munchen, Conf. Learn. Representations, 2019, pp. 3915–3924.
Munich, Germany, 1987. [46] Y. Bengio, S. Bengio, and J. Cloutier, “Learning a synaptic learn-
[18] J. Schmidhuber, J. Zhao, and M. Wiering, “Shifting inductive bias ing rule,” in Proc. Int. Joint Conf. Neural Netw., 1990, p. 969.
with success-story algorithm, adaptive Levin search, and incremen- [47] S. Bengio, Y. Bengio, and J. Cloutier, “On the search for new
tal self-improvement,” Mach. Learn., vol. 28, pp. 105–130, 1997. learning rules for ANNs,” Neural Process. Lett., vol. 2, pp. 26–30,
[19] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning 1995.
for fast adaptation of deep networks,” in Proc. 34th Int. Conf. [48] J. Schmidhuber, J. Zhao, and M. Wiering, “Simple principles of
Mach. Learn., 2017, pp. 1126–1135. meta-learning,” Istituto Dalle Molle di Studi sull’Intelligenza
[20] L. Franceschi, P. Frasconi, S. Salzo, R. Grazzi, and M. Pontil, Artificiale, Lugano, Switzerland, Tech. Rep. TR-IDSIA-69–96,
“Bilevel programming for hyperparameter optimization and 1996.
meta-learning,” in Proc. 35th Int. Conf. Mach. Learn., 2018, [49] J. Schmidhuber, “A neural network that embeds its own meta-
pp. 1568–1577. levels,” in Proc. IEEE Int. Conf. Neural Netw., 1993, pp. 407–412.
[21] H. Liu, K. Simonyan, and Y. Yang, “DARTS: Differentiable archi- [50] J.-A. Meyer and S. W. Wilson, “A possibility for implementing
tecture search,” in Proc. Int. Conf. Learn. Representations, 2019. curiosity and boredom in model-building neural controllers,” in
[22] W. Zhou, Y. Li, Y. Yang, H. Wang, and T. M. Hospedales, Proc. Animals Animats, 1st Int. Conf. Simul. Adaptive Behav., 1991,
“Online meta-critic learning for off-policy actor-critic methods,” pp. 222–227.
in Proc. 34th Conf. Neural Inf. Process. Syst., 2020. [51] S. Hochreiter, A. S. Younger, and P. R. Conwell, “Learning to
[23] J. Snell, K. Swersky, and R. S. Zemel, “Prototypical Networks for learn using gradient descent,” in Proc. Int. Conf. Artif. Neural
few shot learning,” in Proc. 31st Conf. Neural Inf. Process. Syst., Netw., 2001, pp. 87–94.
2017. [52] A. S. Younger, S. Hochreiter, and P. R. Conwell, “Meta-learning
[24] Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and P. with backpropagation,” in Proc. Int. Joint Conf. Neural Netw.,
Abbeel, “RL2 : Fast reinforcement learning via slow reinforce- 2001, pp. 2001–2006.
ment learning,” 2016, arXiv:1611.02779. [53] J. Storck, S. Hochreiter, and J. Schmidhuber, “Reinforcement
[25] R. Houthooft et al., “Evolved policy gradients,” in Proc. 32nd driven information acquisition in non-deterministic environ-
Conf. Neural Inf. Process. Syst., 2018. ments,” in Proc. Int. Conf. Artif. Neural Netw., 1995, pp. 159–164.
[26] F. Alet, M. F. Schneider, T. Lozano-Perez, and L. PackKaelbling, [54] M. Wiering and J. Schmidhuber, “Efficient model-based
“Meta-learning curiosity algorithms,” in Proc. Int. Conf. Learn. exploration,” in Proc. 5th Int. Conf. Simul. Adaptive Behav., 1998,
Representations, 2020. pp. 223–228.
[27] E. Real, A. Aggarwal, Y. Huang, and Q. V. Le, “Regularized evo- [55] N. Schweighofer and K. Doya, “Meta-learning in reinforcement
lution for image classifier architecture search,” in Proc. 33rd learning,” Neural Netw., vol. 16, pp. 5–9, 2003.
Assoc. Adv. Artif. Intell., 2019, pp. 4780–4789. [56] L. Y. Pratt, J. Mostow, C. A. Kamm, and A. A. Kamm, “Direct
[28] B. Zoph and Q. V. Le, “Neural architecture search with reinforce- transfer of learned information among neural networks,” in Proc.
ment learning,” in Proc. Int. Conf. Learn. Representations, 2017. Assoc. Adv. Artif. Intell., 1991, pp. 584–589.
[29] R. Vilalta and Y. Drissi, “A perspective view and survey of meta- [57] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, “How transferable
learning,” Artif. Intell. Rev., vol. 18, pp. 77–95, 2002. are features in deep neural networks?,” in Proc. 27th Int. Conf.
[30] S. Thrun, “Lifelong learning algorithms,” in Learning to Learn. Neural Inf. Process. Syst., 2014, pp. 3320–3328.
Berlin, Germany: Springer, 1998, pp. 181–209. [58] G. Csurka, Domain Adaptation in Computer Vision Applications.
[31] J. Baxter, “Theoretical models of learning to learn,” in Learning to Berlin, Germany: Springer, 2017.
Learn. Berlin, Germany: Springer, 1998, pp. 71–94. [59] D. Li and T. Hospedales, “Online meta-learning for multi-source
[32] D. H. Wolpert, “The lack of a priori distinctions between learning and semi-supervised domain adaptation,” in Proc. Eur. Conf.
algorithms,” Neural Comput., vol. 8, pp. 1341–1390, 1996. Comput. Vis., 2020.
[33] J. Vanschoren, “Meta-learning: A survey,” 2018, arXiv: 1810. [60] G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, and S. Wermter,
03548. “Continual lifelong learning with neural networks: A review,”
[34] F. Hutter, L. Kotthoff, and J. Vanschoren, Eds.,Automatic Machine Neural Netw., vol. 113, pp. 54–71, 2019.
Learning: Methods, Systems, Challenges. Berlin, Germany: Springer, [61] Z. Chen and B. Liu, “Lifelong machine learning,” Synth. Lectures
2019. Artif. Intell. Mach. Learn., vol. 12, no. 3, pp. 1–207, 2018.
[35] S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE [62] M. Al-Shedivat, T. Bansal, Y. Burda, I. Sutskever, I. Mordatch,
Trans. Knowl. Data Eng., vol. 22, no. 10, pp. 1345–1359, Oct. 2010. and P. Abbeel, “Continuous adaptation via meta-learning in non-
[36] C. Lemke, M. Budka, and B. Gabrys, “Meta-learning: A survey of stationary and competitive environments,” in Proc. Int. Conf.
trends and technologies,” Artif. Intell. Rev., vol. 44, pp. 117–130, Learn. Representations, 2018.
2015, Art. no. 63. [63] S. Ritter et al., “Been there, done that: Meta-learning with episodic
[37] Y. Wang, Q. Yao, J. T. Kwok, and L. M. Ni, “Generalizing from a recall,” in Proc. 35th Int. Conf. Mach. Learn., 2018, pp. 4354–4363.
few examples: A survey on few-shot learning,” ACM Comput. [64] I. Clavera et al., “Learning to adapt in dynamic, real-world envi-
Surv., vol. 53, no. 3, Jun. 2020, Art. no. 63. ronments through meta-reinforcement learning,” in Proc. Int.
[38] T. Elsken, J. H. Metzen, and F. Hutter, “Neural architecture Conf. Learn. Representations, 2019.
search: A survey,” J. Mach. Learn. Res., vol. 20, pp. 1–21, 2019. [65] R. Caruana, “Multitask learning,” Mach. Learn., vol. 28, pp. 41–75,
[39] N. Mishra, M. Rohaninejad, X. Chen, and P. Abbeel, “A simple 1997.
neural attentive meta-learner,” Proc. Int. Conf. Learn. Representa- [66] Y. Yang and T. M. Hospedales, “Deep multi-task representation
tions, 2018. learning: A tensor factorisation approach,” in Proc. Int. Conf.
[40] H. Stackelberg, The Theory of Market Economy. London, U.K.: Learn. Representations, 2017.
Oxford Univ. Press, 1952. [67] L. Franceschi, M. Donini, P. Frasconi, and M. Pontil, “Forward
[41] A. Sinha, P. Malo, and K. Deb, “A review on Bilevel optimiza- and reverse gradient-based hyperparameter optimization,” in
tion: From classical to evolutionary approaches and applica- Proc. Int. Conf. Mach. Learn., 2017, pp. 1165–1173.
tions,” IEEE Trans. Evol. Comput., vol. 22, no. 2, pp. 276–295, Apr. [68] X. Lin, H. Baweja, G. Kantor, and D. Held, “Adaptive auxiliary
2018. task weighting for reinforcement learning,” in Proc. Int. Conf.
[42] Y. Li, Y. Yang, W. Zhou, and T. M. Hospedales, “Feature-critic Neural Inf. Process. Syst., 2019.
networks for heterogeneous domain generalization,” in Proc. [69] P. Micaelli and A. Storkey, “Non-greedy gradient-based hyper-
36th Int. Conf. Mach. Learn., 2019, pp. 3915–3924. parameter optimization over long horizons,” 2020, arXiv:
[43] G. Denevi, C. Ciliberto, D. Stamos, and M. Pontil, “Learning to 2007.07869.
learn around a common mean,” in Proc. 32nd Conf. Neural Inf. [70] J. Bergstra and Y. Bengio, “Random search for hyper-parameter
Process. Syst., 2018. optimization,” J. Mach. Learn. Res., vol. 13, pp. 281–305, 2012.
[71] B. Shahriari, K. Swersky, Z. Wang, R. P. Adams, and N. de Frei- [100] Q. Sun, Y. Liu, T.-S. Chua, and B. Schiele, “Meta-transfer learning
tas, “Taking the human out of the loop: A review of Bayesian for few-shot learning,” in Proc. Conf. Comput. Vis. Pattern Recog-
optimization,” Proc. IEEE, vol. 104, no. 1, pp. 148–175, Jan. 2016. nit., 2019, pp. 403–412.
[72] D. M. Blei, A. Y. Ng, and M. I. Jordan, “Latent Dirchlet [101] R. Vuorio, S.-H. Sun, H. Hu, and J. J. Lim, “Multimodal model-
allocation,” J. Mach. Learn. Res., vol. 3, pp. 993–1022, 2003. agnostic meta-learning via task-aware modulation,” in Proc. 33rd
[73] H. Edwards and A. Storkey, “Towards a neural statistician,” in Int. Conf. Neural Inf. Process. Syst., 2019.
Proc. Int. Conf. Learn. Representations, 2017. [102] H. Yao, Y. Wei, J. Huang, and Z. Li, “Hierarchically structured
[74] E. Grant, C. Finn, S. Levine, T. Darrell, and T. Griffiths, meta-learning,” in Proc. 36th Int. Conf. Mach. Learn., 2019,
“Recasting gradient-based meta-learning as hierarchical Bayes,” pp. 7045–7054.
in Proc. Int. Conf. Learn. Representations, 2018. [103] D. Kingma and J. Ba, “Adam: A method for stochastic opti-
[75] H. Yao et al., “Automated relational meta-learning,” in Proc. Int. mization,” in Proc. Int. Conf. Learn. Representations, 2015, pp. 1–15.
Conf. Learn. Representations, 2020. [104] E. Park and J. B. Oliva, “Meta-curvature,” in Proc. Int. Conf. Neu-
[76] S. C. Yoonho Lee, “Gradient-based meta-learning with learned ral Inf. Process. Syst., 2019.
layerwise metric and subspace,” in Proc. 35th Int. Conf. Mach. [105] S. Flennerhag, A. A. Rusu, R. Pascanu, F. Visin, H. Yin, and
Learn., 2018, pp. 2927–2936. R. Hadsell, “Meta-learning with warped gradient descent,” in
[77] Z. Li, F. Zhou, F. Chen, and H. Li, “Meta-SGD: Learning to learn Proc. Int. Conf. Learn. Representations, 2020.
quickly for few shot learning,” 2017, arXiv: 1707.09835. [106] Y. Chen et al., “Learning to learn without gradient descent by gradi-
[78] A. Antoniou, H. Edwards, and A. J. Storkey, “How to train your ent descent,” in Proc. Int. Conf. Mach. Learn., 2017, pp. 748–756.
MAML,” in Proc. Int. Conf. Learn. Representations, 2018. [107] T. Heskes, “Empirical Bayes for learning to learn,” in Proc. Int.
[79] K. Li and J. Malik, “Learning to optimize,” in Proc. Int. Conf. Conf. Mach. Learn., 2000.
Learn. Representations, 2017. [108] J. Requeima, J. Gordon, J. Bronskill, S. Nowozin, and R. E.
[80] E. Grefenstette et al. “Generalized inner loop meta-learning,” Turner, “Fast and flexible multi-task classification using condi-
2019, arXiv: 1910.01727. tional neural adaptive processes,” in Proc. Int. Conf. Neural Inf.
[81] S. Qiao, C. Liu, W. Shen, and A. L. Yuille, “Few-shot image rec- Process. Syst., 2019.
ognition by predicting parameters from activations,” in Proc. [109] E. Triantafillou et al., “Meta-dataset: A dataset of datasets for
Conf. Comput. Vis. Pattern Recognit., 2018, pp. 7229–7238. learning to learn from few examples,” in Proc. Int. Conf. Learn.
[82] S. Gidaris and N. Komodakis, “Dynamic few-shot visual learn- Representations, 2020.
ing without forgetting,” in Proc. Conf. Comput. Vis. Pattern Recog- [110] D. Ha, A. Dai, and Q. V. Le, “HyperNetworks,” Proc. Int. Conf.
nit., 2018, pp. 4367–4375. Learn. Representations, 2017.
[83] A. Graves, G. Wayne, and I. Danihelka, “Neural turing [111] K. Rakelly, A. Zhou, C. Finn, S. Levine, and D. Quillen, “Efficient
machines,” 2014, arXiv:1410.5401. off-policy meta-reinforcement learning via probabilistic context
[84] A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lilli- variables,” in Proc. 36th Int. Conf. Mach. Learn., 2019, pp. 5331–5340.
crap, “Meta learning with memory-augmented neural networks,” [112] Y. Duan et al., “One-shot imitation learning,” in Proc. Int. Conf.
in Proc. Int. Conf. Mach. Learn., 2016. Neural Inf. Process. Syst., 2017.
[85] T. Munkhdalai and H. Yu, “Meta networks,” in Proc. 34th Int. [113] J. X. Wang et al. “Learning to reinforcement learn,” 2016,
Conf. Mach. Learn., 2017, pp. 2554–2563. arXiv:1611.05763.
[86] C. Finn and S. Levine, “Meta-learning and universality: Deep [114] W.-Y. Chen, Y.-C. Liu, Z. Kira, Y.-C. Wang, and J.-B. Huang, “A
representations and gradient descent can approximate any learn- closer look at few-shot classification,” in Proc. Int. Conf. Learn.
ing algorithm,” in Proc. Int. Conf. Learn. Representations, 2018, Representations, 2019.
pp. 1–12. [115] B. Oreshkin, P. Rodr ıguez L opez, and A. Lacoste, “TADAM:
[87] G. Kosh, R. Zemel, and R. Salakhutdinov, “Siamese neural net- Task dependent adaptive metric for improved few-shot
works for one-shot image recognition,” in Proc. 32nd Int. Conf. learning,” in Proc. 32nd Int. Conf. Neural Inf. Process. Syst., 2018.
Mach. Learn., 2015. [116] H.-Y. Tseng, H.-Y. Lee, J.-B. Huang, and M.-H. Yang, “Cross-
[88] O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra et al., “Matching domain few-shot classification via learned feature-wise trans-
networks for one shot learning,” in Proc. Int. Conf. Neural Inf. Pro- formation,” Proc. Int. Conf. Learn. Representations, Jan. 2020, pp.
cess. Syst., 2016. 1–18.
[89] F. Sung, Y. Yang, L. Zhang, T. Xiang, P. H. S. Torr, and T. M. Hos- [117] F. Sung, L. Zhang, T. Xiang, T. Hospedales, and Y. Yang,
pedales, “Learning to compare: Relation network for few-shot “Learning to learn: Meta-critic networks for sample efficient
learning,” in Proc. Conf. Comput. Vis. Pattern Recognit., 2018, learning,” 2017, arXiv: 1706.09529.
pp. 1199–1208. [118] G. Denevi, D. Stamos, C. Ciliberto, and M. Pontil, “Online-
[90] V. Garcia and J. Bruna, “Few-shot learning with graph neural within-online meta-learning,” in Proc. 33rd Int. Conf. Neural Inf.
networks,” in Proc. Int. Conf. Learn. Representations, 2018, pp. 1–13. Process. Syst., 2019, pp. 1–11.
[91] I. Bello, B. Zoph, V. Vasudevan, and Q. V. Le, “Neural optimizer [119] S. Gonzalez and R. Miikkulainen, “Improved training speed,
search with reinforcement learning,” in Proc. Int. Conf. Mach. accuracy, and data utilization through loss function opti-
Learn., 2017, pp. 459–468. mization,” in Proc. IEEE Congr. Evol. Comput., 2020, pp. 1–8.
[92] O. Wichrowska et al., “Learned optimizers that scale and general- [120] S. Bechtle et al., “Meta-learning via learned loss,” in Proc. Int.
ize,” in Proc. Int. Conf. Mach. Learn., 2017, pp. 3751–3760. Conf. Pattern Recognit., 2020.
[93] Y. Balaji, S. Sankaranarayanan, and R. Chellappa, “MetaReg: [121] B. Gao, H. Gouk, and T. M. Hospedales, “Searching for robust-
Towards domain generalization using meta-regularization,” in ness: Loss learning for noisy classification tasks,” 2021,
Proc. Int. Conf. Neural Inf. Process. Syst., 2018, pp. 1–11. arXiv:2103.00243.
[94] J. Li, Y. Wong, Q. Zhao, and M. S. Kankanhalli, “Learning to [122] A. I. Rinu Boney, “Semi-supervised few-shot learning with
learn from noisy labeled data,” in Proc. Conf. Comput. Vis. Pattern MAML,” in Proc. Int. Conf. Learn. Representations, 2018, pp. 1–4.
Recognit., 2019. [123] C. Huang et al., “Addressing the loss-metric mismatch with
[95] M. Goldblum, L. Fowl, and T. Goldstein, “Adversarially robust adaptive loss alignment,” in Proc. Int. Conf. Mach. Learn., 2019,
few-shot learning: A meta-learning approach,” in Proc. 34th Int. pp. 2891–2900.
Conf. Neural Inf. Process. Syst., 2020. [124] C. Doersch and A. Zisserman, “Multi-task self-supervised visual
[96] C. Finn, K. Xu, and S. Levine, “Probabilistic model-agnostic learning,” in Proc. Int. Conf. Comput. Vis., 2017, pp. 2051–2060.
meta-learning,” in Proc. 32nd Int. Conf. Neural Inf. Process. Syst., [125] M. Jaderberg et al., “Reinforcement learning with unsupervised
2018, pp. 9537–9548. auxiliary tasks,” in Proc. Int. Conf. Learn. Representations, 2017.
[97] C. Finn, A. Rajeswaran, S. Kakade, and S. Levine, “Online meta- [126] S. Liu, A. Davison, and E. Johns, “Self-supervised generalisation
learning,” Proc. Int. Conf. Mach. Learn., 2019. with meta auxiliary learning,” in Proc. Int. Conf. Neural Inf. Pro-
[98] A. A. Rusu et al., “Meta-learning with latent embedding opti- cess. Syst., 2019, pp. 1677–1687.
mization,” in Proc. Int. Conf. Learn. Representations, 2019. [127] K. O. Stanley, J. Clune, J. Lehman, and R. Miikkulainen,
[99] A. Antoniou and A. Storkey, “Learning to learn by self-critique,” “Designing neural networks through neuroevolution,” Nat.
in Proc. Int. Conf. Neural Inf. Process. Syst., 2019. Mach. Intell., vol. 1, pp. 24–35, 2019.
[128] J. Bayer, D. Wierstra, J. Togelius, and J. Schmidhuber, “Evolving [155] O. Bohdal, Y. Yang, and T. Hospedales, “Flexible dataset distilla-
memory cell structures for sequence learning,” in Proc. Int. Conf. tion: Learn labels instead of images,” 2020, arXiv: 2006.08572.
Artif. Neural Netw., 2009, pp. 755–764. [156] W.-H. Li, C.-S. Foo, and H. Bilen, “Learning to impute: A
[129] S. Xie, H. Zheng, C. Liu, and L. Lin, “SNAS: Stochastic neural general framework for semi-supervised learning,” 2019, arXiv:
architecture search,” in Proc. Int. Conf. Learn. Representations, 1912.10364.
2019, pp. 1–17. [157] Q. Sun, X. Li, Y. Liu, S. Zheng, T.-S. Chua, and B. Schiele,
[130] A. Zela, T. Elsken, T. Saikia, Y. Marrakchi, T. Brox, and F. Hutter, “Learning to self-train for semi-supervised few-shot classi-
“Understanding and robustifying differentiable architecture fication,” in Proc. Int. Conf. Neural Inf. Process. Syst., 2019.
search,” in Proc. Int. Conf. Learn. Representations, 2020. [158] O. M. Andrychowicz et al. “Learning dexterous in-hand manipu-
[131] D. Lian et al., “Towards fast adaptation of neural architectures lation,” Int. J. Robot. Res., vol. 39, pp. 3–20, 2020.
with meta learning,” in Proc. Int. Conf. Learn. Representations, [159] N. Ruiz, S. Schulter, and M. Chandraker, “Learning to simulate,”
2020, pp. 1–16. in Proc. Int. Conf. Learn. Representations, 2018, pp. 1–12.
[132] A. Shaw, W. Wei, W. Liu, L. Song, and B. Dai, “Meta architec- [160] Q. Vuong, S. Vikram, H. Su, S. Gao, and H. I. Christensen, “How
ture search,” in Proc. 33rd Int. Conf. Neural Inf. Process. Syst., To pick the domain randomization parameters for sim-to-real
2019. transfer of reinforcement learning policies?,” 2019, arXiv:
[133] R. Hou, H. Chang, M. Bingpeng, S. Shan, and X. Chen, “Cross 1903.11774.
attention network for few-shot classification,” in Proc. Int. Conf. [161] Q. V. Le, P. Ramachandran, and B. Zoph, “Searching for activa-
Neural Inf. Process. Syst., 2019. tion functions,” 2017, arXiv: 1710.05941.
[134] M. Ren, R. Liao, E. Fetaya, and R. Zemel, “Incremental few-shot [162] H. B. Lee et al., “Learning to balance: Bayesian meta-learning for
learning with attention attractor networks,” in Proc. 33rd Int. imbalanced and out-of-distribution tasks,” in Proc. Int. Conf.
Conf. Neural Inf. Process. Syst., 2019, pp. 1–11. Learn. Representations, 2020.
[135] Y. Bao, M. Wu, S. Chang, and R. Barzilay, “Few-shot text classifi- [163] A. Rajeswaran, C. Finn, S. Kakade, and S. Levine, “Meta-learning
cation with distributional signatures,” in Proc. Int. Conf. Learn. with implicit gradients,” in Proc. 33rd Int. Conf. Neural Inf. Process.
Representations, 2020, pp. 1–24. Syst., 2019, pp. 1–12.
[136] F. Alet, T. Lozano-P erez, and L. P. Kaelbling, “Modular meta- [164] K. Lee, S. Maji, A. Ravichandran, and S. Soatto, “Meta-learning
learning,” in Proc. 2nd Conf. Robot Learn., 2018, pp. 856–868. with differentiable convex optimization,” in Proc. Conf. Comput.
[137] F. Alet, E. Weng, T. Lozano-P erez, and L. P. Kaelbling, “Neural Vis. Pattern Recognit., 2019, pp. 10657–10665.
relational inference with fast modular meta-learning,” in Proc. [165] L. Bertinetto, J. F. Henriques, P. H. Torr, and A. Vedaldi, “Meta-
33rd Int. Conf. Neural Inf. Process. Syst., 2019, pp. 1–12. learning with differentiable closed-form solvers,” in Proc. Int.
[138] C. Fernando et al., “PathNet: Evolution channels gradient descent Conf. Learn. Representations, 2019, pp. 1–15.
in super neural networks,” 2017, arXiv: 1701.08734. [166] R. Fakoor, P. Chaudhari, S. Soatto, and A. J. Smola, “Meta-Q-
[139] B. M. Lake, “Compositional generalization through meta learning,” in Proc. Int. Conf. Learn. Representations, 2020.
sequence-to-sequence learning,” in Proc. 33rd Int. Conf. Neural [167] H. Liu, R. Socher, and C. Xiong, “Taming MAML: Efficient unbi-
Inf. Process. Syst., 2019, pp. 1–11. ased meta-reinforcement learning,” in Proc. 36th Int. Conf. Mach.
[140] E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le, Learn., 2019, pp. 4061–4071.
“AutoAugment: Learning augmentation policies from data,” [168] J. Rothfuss, D. Lee, I. Clavera, T. Asfour, and P. Abbeel, “ProMP:
Proc. Conf. Comput. Vis. Pattern Recognit., 2019, pp. 113–123. Proximal meta-policy search,” in Proc. Int. Conf. Learn. Representa-
[141] Y. Li, G. Hu, Y. Wang, T. Hospedales, N. M. Robertson, and tions, 2019.
Y. Yang, “DADA: Differentiable automatic data augmentation,” [169] X. Song, W. Gao, Y. Yang, K. Choromanski, A. Pacchiano, and
in Proc. Eur. Conf. Comput. Vis., 2020, pp. 1–16. Y. Tang, “ES-MAML: Simple Hessian-free meta learning,” in
[142] R. Volpi and V. Murino, “Model vulnerability to distributional Proc. Int. Conf. Learn. Representations, 2020.
shifts over image transformation sets,” in Proc. Int. Conf. Comput. [170] C. Fernando et al., “Meta-learning by the Baldwin effect,” in Proc.
Vis., 2019, pp. 33–35. Genet. Evol. Comput. Conf. Companion, 2018, pp. 109–110.
[143] Z. Mai, G. Hu, D. Chen, F. Shen, and H. T. Shen, “MetaMixUp: [171] R. Vuorio, D.-Y. Cho, D. Kim, and J. Kim, “Meta continual
Learning adaptive interpolation policy of mixup with meta- learning,” 2018, arXiv: 1806.06928.
learning,” 2019, arXiv: 1908.10059. [172] K. Young, B. Wang, and M. E. Taylor, “Metatrace actor-critic:
€
[144] C. Zhang, C. Oztireli, S. Mandt, and G. Salvi, “Active mini-batch Online step-size tuning by meta-gradient descent for reinforce-
sampling using repulsive point processes,” in Proc. 33rd Conf. ment learning control,” in Proc. 28th Inte. Joint Conf. Artif. Intell.,
Assoc. Adv. Artif. Intell., 2019, 5741–5748. 2019, pp. 4185–4191.
[145] I. Loshchilov and F. Hutter, “Online batch selection for faster [173] Z. Xu, H. van Hasselt, and D. Silver, “Meta-gradient reinforce-
training of neural networks,” in Proc. Int. Conf. Learn. Representa- ment learning,” in Proc. 32nd Int. Conf. Neural Inf. Process. Syst.,
tions, 2016, pp. 1–20. 2018, pp. 1–12.
[146] Y. Fan, F. Tian, T. Qin, X. Li, and T. Liu, “Learning to teach,” in [174] M. Jaderberg et al. “Human-level performance in 3D multiplayer
Proc. Int. Conf. Learn. Representations, 2018, pp. 1–16. games with population-based reinforcement learning,” Science,
[147] J. Shu et al., “Meta-weight-net: Learning an explicit mapping for vol. 364, pp. 859–865, 2019.
sample weighting,” in Proc. Int. Conf. Neural Inf. Process. Syst., [175] J.-M. Perez-Rua, X. Zhu, T. Hospedales, and T. Xiang,
2019, pp. 1917–1928. “Incremental few-shot object detection,” in Proc. Conf. Comput.
[148] M. Ren, W. Zeng, B. Yang, and R. Urtasun, “Learning to reweight Vis. Pattern Recognit., 2020, pp. 13846–13855.
examples for robust deep learning,” in Proc. 35th Int. Conf. Mach. [176] M. Garnelo et al., “Conditional neural processes,” Proc. 35th Int.
Learn., 2018, pp. 4334–4343. Conf. Mach. Learn., 2018, pp. 1704–1713.
[149] J. L. Elman, “Learning and development in neural networks: [177] A. Pakman, Y. Wang, C. Mitelut, J. Lee, and L. Paninski, “Neural
The importance of starting small,” Cognition, vol. 48, pp. 71–99, clustering processes,” in Proc. 37th Int. Conf. Mach. Learn., 2019,
1993. pp. 7455–7465.
[150] Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum [178] J. Lee, Y. Lee, J. Kim, A. Kosiorek, S. Choi, and Y. W. Teh, “Set
learning,” in Proc. 26th Int. Conf. Mach. Learn., 2009, pp. 41–48. transformer: A framework for attention-based permutation-
[151] L. Jiang, Z. Zhou, T. Leung, L.-J. Li, and L. Fei-Fei, “MentorNet: invariant neural networks,” in Proc. 36th Int. Conf. Mach. Learn.,
Learning data-driven curriculum for very deep neural networks 2019, pp. 3744–3753.
on corrupted labels,” in Proc. 35th Int. Conf. Mach. Learn., 2018, [179] J. Lee, Y. Lee, and Y. W. Teh, “Deep amortized clustering,” in
pp. 2304–2313. Proc. Int. Conf. Learn. Representations, 2019.
[152] T. Wang, J. Zhu, A. Torralba, and A. A. Efros, “Dataset distil- [180] V. Veeriah et al., “Discovery of useful questions as auxiliary
lation,” 2018, arXiv: 1811.10959. tasks,” in Proc. Int. Conf. Neural Inf. Process. Syst., 2019.
[153] B. Zhao, K. R. Mopuri, and H. Bilen, “Dataset condensation [181] Z. Zheng, J. Oh, and S. Singh, “On learning intrinsic rewards for
with gradient matching,” in Proc. Int. Conf. Learn. Representa- policy gradient methods,” in Proc. 32nd Int. Conf. Neural Inf. Pro-
tions, 2021. cess. Syst., 2018, pp. 1–11.
[154] J. Lorraine, P. Vicol, and D. Duvenaud, “Optimizing millions of [182] T. Xu, Q. Liu, L. Zhao, and J. Peng, “Learning to explore with
hyperparameters by implicit differentiation,” in Proc. 23rd Int. meta-policy gradient,” in Proc. Int. Conf. Mach. Learn., 2018,
Conf. Artif. Intell. Stat., 2020, pp. 1540–1552. pp. 5463–5472.
[183] B. C. Stadie et al., “Some considerations on learning to explore via [210] M. Bishay, G. Zoumpourlis, and I. Patras, “TARN: Temporal
meta-reinforcement learning,” in Proc. 36th Int. Conf. Neural Inf. attentive relation network for few-shot and zero-shot action rec-
Process. Syst., 2018. ognition,” in Proc. Brit. Mach. Vis. Conf., 2019, pp. 1–14.
[184] F. Garcia and P. S. Thomas, “A meta-MDP approach to explora- [211] H. Coskun et al., “Domain-specific priors and meta learning for
tion for lifelong reinforcement learning,” in Proc. 33rd Int. Conf. few-shot first-person action recognition,” IEEE Trans. Pattern
Neural Inf. Process. Syst., 2019, pp. 1–10. Anal. Mach. Intell., early access, Feb. 10, 2021, doi: 10.1109/
[185] A. Gupta, R. Mendonca, Y. Liu, P. Abbeel, and S. Levine, “Meta- TPAMI.2021.3058606.
reinforcement learning of structured exploration strategies,” in [212] L.-Y. Gui, Y.-X. Wang, D. Ramanan, and J. M. Moura, “Few-shot
Proc. 32nd Int. Conf. Neural Inf. Process. Syst., 2018, pp. 5307–5316. human motion prediction via meta-learning,” in Proc. Eur. Conf.
[186] H. B. Lee, T. Nam, E. Yang, and S. J. Hwang, “Meta dropout: Comput. Vis., 2018, pp. 1–19.
Learning to perturb latent features for generalization,” in Proc. [213] S. A. Eslami et al., “Neural scene representation and rendering,”
Int. Conf. Learn. Representations, 2020, pp. 1–11. Science, vol. 360, no. 6394, pp. 1204–1210, 2018.
[187] P. Bachman, A. Sordoni, and A. Trischler, “Learning algorithms for [214] E. Zakharov, A. Shysheya, E. Burkov, and V. S. Lempitsky, “Few-
active learning,” in Proc. Int. Conf. Mach. Learn., 2017, pp. 301–310. shot adversarial learning of realistic neural talking head mod-
[188] K. Konyushkova, R. Sznitman, and P. Fua, “Learning active els,” in Proc. Int. Conf. Comput. Vis., 2019.
learning from data,” in Proc. Int. Conf. Neural Inf. Process. Syst., [215] T.-C. Wang, M.-Y. Liu, A. Tao, G. Liu, J. Kautz, and B. Catanzaro,
2017, pp. 1–11. “Few-shot video-to-video synthesis,” in Proc. Int. Conf. Neural
[189] K. Pang, M. Dong, Y. Wu, and T. M. Hospedales, “Meta-learning Inf. Process. Syst., 2019.
transferable active learning policies by deep reinforcement [216] F. Shen, S. Yan, and G. Zeng, “Neural style transfer via meta
learning,” 2018, arXiv: 1806.04798. networks,” in Proc. Conf. Comput. Vis. Pattern Recognit., 2018,
[190] D. Maclaurin, D. Duvenaud, and R. P. Adams, “Gradient-based pp. 8061–8069.
hyperparameter optimization through reversible learning,” in [217] S. E. Reed et al., “Few-shot autoregressive density estimation:
Proc. Int. Conf. Mach. Learn., 2015, pp. 2113–2122. towards learning to learn distributions,” in Proc. Int. Conf. Learn.
[191] C. Russell, M. Toso, and N. Campbell, “Fixing implicit derivatives: Representations, 2018.
Trust-region based learning of continuous energy functions,” in [218] O. Russakovsky et al., “Imagenet large scale visual recognition
Proc. Int. Conf. Neural Inf. Process. Syst., 2019, pp. 1–11. challenge,” Int. J. Comput. Vis., vol. 115, pp. 211–252, 2015.
[192] A. Nichol, J. Achiam, and J. Schulman, “On first-order meta- [219] M. Ren et al., “Meta-learning for semi-supervised few-shot classi-
learning algorithms,” 2018, arXiv: 1803.02999. fication,” in Proc. Int. Conf. Learn. Represent., 2018, pp. 1–15.
[193] T. Salimans, J. Ho, X. Chen, S. Sidor, and I. Sutskever, “Evolution [220] A. Antoniou, P. Massimiliano, O. Mateusz, and A. Storkey,
strategies as a scalable alternative to reinforcement learning,” “Defining benchmarks for continual few-shot learning,” 2020,
2017, arXiv: 1703.03864. arXiv: 2004.11967.
[194] F. Stulp and O. Sigaud, “Robot skill learning: From reinforce- [221] S.-A. Rebuffi, H. Bilen, and A. Vedaldi, “Learning multiple visual
ment learning to evolution strategies,” Paladyn, J. Behav. Robot., domains with residual adapters,” in Proc. Int. Conf. Neural Inf.
vol. 4, pp. 49–61, 2013. Process. Syst., 2017.
[195] A. Soltoggio, K. O. Stanley, and S. Risi, “Born to learn: The inspi- [222] Y. Guo et al., “A broader study of cross-domain few-shot
ration, progress, and future of evolved plastic artificial neural learning,” in Proc. Eur. Conf. Comput. Vis., 2020, pp. 1–18.
networks,” Neural Netw., vol. 108, pp. 48–67, 2018. [223] T. De Vries, I. Misra, C. Wang, and L. van der Maaten, “Does
[196] Y. Cao, T. Chen, Z. Wang, and Y. Shen, “Learning to optimize in object recognition work for everyone?,” in Proc. Conf. Comput.
swarms,” in Proc. 33rd Int. Conf. Neural Inf. Process. Syst., 2019, Vis. Pattern Recognit., 2019, pp. 52–59.
pp. 1–11. [224] R. J. Williams, “Simple statistical gradient-following algorithms
[197] F. Meier, D. Kappler, and S. Schaal, “Online learning of a mem- for connectionist reinforcement learning,” Mach. Learn., vol. 8,
ory for learning rates,” in Proc. IEEE Int. Conf. Robot. Automat., pp. 229–256, 1992.
2018, pp. 2425–2432. [225] A. Fallah, A. Mokhtari, and A. Ozdaglar, “Provably convergent
[198] A. G. Baydin, R. Cornish, D. Mart ınez-Rubio, M. Schmidt, and F. policy gradient methods for model-agnostic meta-reinforcement
D. Wood, “Online learning rate adaptation with hypergradient learning,” 2020, arXiv: 2002.05135.
descent,” in Proc. Int. Conf. Learn. Representations, 2018, pp. 1–11. [226] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov,
[199] S. Chen, W. Wang, and S. J. Pan, “MetaQuant: Learning to quan- “Proximal policy optimization algorithms,” 2017, arXiv:
tize by learning to penetrate non-differentiable quantization,” in 1707.06347.
Proc. Int. Conf. Neural Inf. Process. Syst., 2019, pp. 1–11. [227] O. Sigaud and F. Stulp, “Policy search in continuous action
[200] C. Sun, A. Shrivastava, S. Singh, and A. Gupta, “Revisiting domains: An overview,” Neural Netw., vol. 113, pp. 28–40, 2019.
unreasonable effectiveness of data in deep learning era,” in Proc. [228] J. Schmidhuber, “What’s interesting?,” Istituto Dalle Molle di
Int. Conf. Comput. Vis., 2017, pp. 843–852. Studi sull’Intelligenza Artificiale, Lugano, Switzerland, Tech.
[201] M. Yin, G. Tucker, M. Zhou, S. Levine, and C. Finn, “Meta-learn- Rep. TR-IDSIA-35–97, 1997.
ing without memorization,” in Proc. Int. Conf. Learn. Representa- [229] L. Kirsch, S. van Steenkiste, and J. Schmidhuber, “Improving
tions, 2020. generalization in meta reinforcement learning using learned
[202] S. W. Yoon, J. Seo, and J. Moon, “TapNet: Neural network aug- objectives,” in Proc. Int. Conf. Learn. Representations, 2020.
mented with task-adaptive projection for few-shot learning,” in [230] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic:
Proc. Int. Conf. Mach. Learn., 2019, pp. 7115–7123. Off-policy maximum entropy deep reinforcement learning with
[203] J. W. Rae, S. Bartunov, and T. P. Lillicrap, “Meta-learning neural a stochastic actor,” in Proc. Int. Conf. Mach. Learn., 2018, pp. 1861–
bloom filters,” Proc. Int. Conf. Mach. Learn., 2019, pp. 5271–5280. 1870.
[204] A. Raghu, M. Raghu, S. Bengio, and O. Vinyals, “Rapid learning [231] O. Kroemer, S. Niekum, and G. Konidaris, “A review of robot
or feature reuse? Towards understanding the effectiveness of learning for manipulation: Challenges, representations, and algo-
MAML,” in Proc. Int. Conf. Learn. Representations, 2020, pp. 1–21. rithms,” J. Mach. Learn. Res., vol. 22, pp. 1–82, 2021.
[205] B. Kang, Z. Liu, X. Wang, F. Yu, J. Feng, and T. Darrell, “Few- [232] A. Jabri, K. Hsu, A. Gupta, B. Eysenbach, S. Levine, and C. Finn,
shot object detection via feature reweighting,” in Proc. Int. Conf. “Unsupervised curricula for visual meta-reinforcement learning,”
Comput. Vis., 2019, pp. 8420–8429. in Proc. Int. Conf. Neural Inf. Process. Syst., 2019.
[206] L.-Y. Gui, Y.-X. Wang, D. Ramanan, and J. Moura, “Few-shot [233] Y. Yang, K. Caluwaerts, A. Iscen, J. Tan, and C. Finn, “NoRML:
human motion prediction via meta-learning,” in Proc. Eur. Conf. No-reward meta learning,” in Proc. Int. Conf. Autonom. Agents
Comput. Vis., 2018, pp. 441–459. Multi-Agent Syst., 2019, pp. 323–331.
[207] A. Shaban, S. Bansal, Z. Liu, I. Essa, and B. Boots, “One-shot [234] S. K. S. Ghasemipour, S. S. Gu, and R. Zemel, “SMILe: Scalable
learning for semantic segmentation,” in Proc. Brit. Mach. Vis. meta inverse reinforcement learning through context-conditional
Conf., 2017, pp. 167.1–167.13. policies,” in Proc. 33rd Int. Conf. Neural Inf. Process. Syst., 2019,
[208] N. Dong and E. P. Xing, “Few-shot semantic segmentation with pp. 1–11.
prototype learning,” in Proc. Brit. Mach. Vis. Conf., 2018, pp. 1–13. [235] M. C. Machado, M. G. Bellemare, E. Talvitie, J. Veness, M. Haus-
[209] K. Rakelly, E. Shelhamer, T. Darrell, A. A. Efros, and S. Levine, knecht, and M. Bowling, “Revisiting the arcade learning environ-
“Few-shot segmentation propagation with guided networks,” in ment: Evaluation protocols and open problems for general
Proc. Int. Conf. Mach. Learn., 2019. agents,” J. Artif. Intell. Res., vol. 61, pp. 523–562, 2018.
[236] A. Nichol, V. Pfau, C. Hesse, O. Klimov, and J. Schulman, “Gotta [264] X. Si, Y. Yang, H. Dai, M. Naik, and L. Song, “Learning a meta-
learn fast: A new benchmark for generalization in RL,” 2018, solver for syntax-guided program synthesis,” Proc. Int. Conf.
arXiv: 1804.03720. Learn. Representations, 2018, pp. 1–11.
[237] K. Cobbe, O. Klimov, C. Hesse, T. Kim, and J. Schulman, [265] P. Huang, C. Wang, R. Singh, W. Yih, and X. He, “Natural lan-
“Quantifying generalization in reinforcement learning,” in Proc. guage to structured query generation via meta-learning,” in Proc.
Int. Conf. Mach. Learn., 2019, pp. 1282–1289. Conf. North Amer. Chapter Assoc. Comput. Linguistics, 2018,
[238] G. Brockman et al. “OpenAI gym,” 2016, arXiv:1606.01540. pp. 732–738.
[239] C. Packer, K. Gao, J. Kos, P. Kr€ahenb€ uhl, V. Koltun, and D. Song, [266] Y. Xie, H. Jiang, F. Liu, T. Zhao, and H. Zha, “Meta learning with
“Assessing generalization in deep reinforcement learning,” 2018, relational information for short sequences,” in Proc. Int. Conf.
arXiv: 1810.12282. Neural Inf. Process. Syst., 2019.
[240] C. Zhao, O. Siguad, F. Stulp, and T. M. Hospedales, “Investigating [267] J. Gu, Y. Wang, Y. Chen, V. O. K. Li, and K. Cho, “Meta-learning
generalisation in continuous deep reinforcement learning,” 2019, for low-resource neural machine translation,” in Proc. Conf.
arXiv: 1902.07015. Empirical Methods Nat. Lang. Process., 2018, pp. 3622–3631.
[241] T. Yu et al., “Meta-world: A benchmark and evaluation for [268] Z. Lin, A. Madotto, C. Wu, and P. Fung, “Personalizing dialogue
multi-task and meta reinforcement learning,” 2019, arXiv: agents via meta-learning,” in Proc. 57th Annu. Meeting Assoc.
1910.10897. Comput. Linguistics, 2019, pp. 5454–5459.
[242] A. Bakhtin, L. van der Maaten, J. Johnson, L. Gustafson, and [269] J.-Y. Hsu, Y.-J. Chen, and H. yi Lee, “Meta learning for end-to-
R. Girshick, “PHYRE: A new benchmark for physical reasoning,” end low-resource speech recognition,” in Proc. Int. Conf. Acoust.,
in Proc. Int. Conf. Neural Inf. Process. Syst., 2019. Speech, Signal Process., 2019.
[243] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, “Learning trans- [270] G. I. Winata et al., “Learning fast adaptation on cross-accented
ferable architectures for scalable image recognition,” in Proc. speech recognition,” in Proc. Interspeech, 2020, pp. 1276–1280.
Conf. Comput. Vis. Pattern Recognit., 2018, pp. 8697–8708. [271] O. Klejch, J. Fainberg, and P. Bell, “Learning to adapt: A meta-
[244] T. Elsken, B. Staffler, J. H. Metzen, and F. Hutter, “Meta-learning learning approach for speaker adaptation,” in Proc. Interspeech,
of neural architectures for few-shot learning,” in Proc. Conf. Com- 2018, pp. 867–871.
put. Vis. Pattern Recognit., 2019, pp. 12365–12375. [272] J. Tremblay et al., “Training deep networks with synthetic data:
[245] C. Ying, A. Klein, E. Christiansen, E. Real, K. Murphy, and Bridging the reality gap by domain randomization,” in Proc.
F. Hutter, “NAS-bench-101: Towards reproducible neural archi- Conf. Comput. Vis. Pattern Recognit., 2018, pp. 969–977.
tecture search,” in Proc. Int. Conf. Mach. Learn., 2019, pp. 7105– [273] A. Kar et al., “Meta-sim: Learning to generate synthetic datasets,”
7114. in Proc. Int. Conf. Comput. Vis., 2019, pp. 4551–4560.
[246] P. Tossou, B. Dura, F. Laviolette, M. Marchand, and A. Lacoste, [274] D. M. Metter, T. J. Colgan, S. T. Leung, C. F. Timmons, and J. Y. Park,
“Adaptive deep kernel learning,” 2019, arXiv: 1905.12131. “Trends in the US and Canadian pathologist workforces from 2007
[247] M. Patacchiola, J. Turner, E. J. Crowley, M. O’Boyle, and A. Stor- to 2017,” JAMA Network Open, vol. 2, 2019, Art. no. e194337.
key, “Deep kernel transfer in Gaussian processes for few-shot [275] G. Maicas, A. P. Bradley, J. C. Nascimento, I. D. Reid, and G. Car-
learning,” in Proc. Int. Conf. Neural Inf. Process. Syst., 2020. neiro, “Training medical image analysis systems like radiolo-
[248] T. Kim, J. Yoon, O. Dia, S. Kim, Y. Bengio, and S. Ahn, “Bayesian gists,” in Proc. Int. Conf. Med. Image Comput. Comput.-Assist.
model-agnostic meta-learning,” in Proc. Int. Conf. Neural Inf. Pro- Interv., 2018.
cess. Syst., 2018. [276] B. D. Nguyen, T.-T. Do, B. X. Nguyen, T. Do, E. Tjiputra, and
[249] S. Ravi and A. Beatson, “Amortized Bayesian meta-learning,” in Q. D. Tran, “Overcoming data limitation in medical visual ques-
Proc. Int. Conf. Learn. Representations, 2019, pp. 1–14. tion answering,” in Proc. Med. Image Comput. Comput.-Assist.
[250] Z. Wang, Y. Zhao, P. Yu, R. Zhang, and C. Chen, “Bayesian meta Interv., 2019.
sampling for fast uncertainty adaptation,” in Proc. Int. Conf. [277] Z. Mirikharaji, Y. Yan, and G. Hamarneh, “Learning to seg-
Learn. Representations, 2020, pp. 1–22. ment skin lesions from noisy annotations,” 2019, arXiv:
[251] K. Hsu, S. Levine, and C. Finn, “Unsupervised learning via meta- 1906.03815.
learning,” in Proc. Int. Conf. Learn. Representations, 2019, pp. 1–24. [278] T. Miconi, J. Clune, and K. O. Stanley, “Differentiable plasticity:
[252] S. Khodadadeh, L. Boloni, and M. Shah, “Unsupervised meta- Training plastic neural networks with backpropagation,” in Proc.
learning for few-shot image classification,” in Proc. 33rd Int. Conf. Int. Conf. Mach. Learn., 2018, pp. 3559–3568.
Neural Inf. Process. Syst., 2019, pp. 10132–10142. [279] T. Miconi, A. Rawal, J. Clune, and K. O. Stanley, “Backpropamine:
[253] A. Antoniou and A. Storkey, “Assume, augment and learn: Training self-modifying neural networks with differentiable neuro-
Unsupervised few-shot meta-learning via random labels and modulated plasticity,” in Proc. Int. Conf. Learn. Representations, 2019.
data augmentation,” 2019, arXiv: 1902.09884. [280] B. Dai, C. Zhu, and D. Wipf, “Compressing neural networks
[254] Y. Jiang and N. Verma, “Meta-learning to cluster,” 2019, arXiv: using the variational information bottleneck,” in Proc. 35th Int.
1910.14134. Conf. Mach. Learn., 2018.
[255] V. Garg and A. T. Kalai, “Supervising unsupervised learning,” in [281] Z. Liu et al., “Metapruning: Meta learning for automatic neural
Proc. Int. Conf. Neural Inf. Process. Syst., 2018. network channel pruning,” in Proc. IEEE/CVF Int. Conf. Comput.
[256] S. M. Kye, H. B. Lee, H. Kim, and S. J. Hwang, “Meta-learned Vis., 2019, pp. 3295–3304.
confidence for few-shot learning,” 2020, arXiv: 2002.12017. [282] T. O’Shea and J. Hoydis, “An introduction to deep learning for
[257] H. Pham, Q. Xie, Z. Dai, and Q. V. Le, “Meta pseudo labels,” the physical layer,” IEEE Trans. Cogn. Commun. Netw., vol. 3,
2020, arXiv: 2003.10580. no. 4, pp. 563–575, Dec. 2017.
[258] K. Javed and M. White, “Meta-learning representations for con- [283] Y. Jiang, H. Kim, H. Asnani, and S. Kannan, “MIND: Model inde-
tinual learning,” in Proc. Int. Conf. Neural Inf. Process. Syst., 2019, pendent neural decoder,” 2019, arXiv: 1903.02268.
pp. 1–11. [284] B. Settles, “Active learning,” Synth. Lectures Artif. Intel. Mach.
[259] A. Sinitsin, V. Plokhotnyuk, D. Pyrkin, S. Popov, and A. Babenko, Learn., vol. 6, pp. 1–114, 2012.
“Editable neural networks,” in Proc. Int. Conf. Learn. Representations, [285] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and har-
2020, pp. 1–12. nessing adversarial examples,” in Proc. Int. Conf. Learn. Represen-
[260] K. Muandet, D. Balduzzi, and B. Sch€ olkopf, “Domain generaliza- tations, 2015.
tion via invariant feature representation,” in Proc. Int. Conf. [286] C. Yin, J. Tang, Z. Xu, and Y. Wang, “Adversarial meta-
Mach. Learn., 2013, pp. I-10–I-18. learning,” 2018, arXiv: 1806.03316.
[261] D. Li, Y. Yang, Y. Song, and T. Hospedales, “Learning to general- [287] M. Vartak, A. Thiagarajan, C. Miranda, J. Bratman, and H. Laro-
ize: Meta-learning for domain generalization,” in Proc. 32nd chelle, “A meta-learning perspective on cold-start recommenda-
Assoc. Adv. Artif. Intell., 2018. tions for items,” in Proc. 31st Conf. Neural Inf. Process. Syst., 2017,
[262] D. Li, Y. Yang, Y.-Z. Song, and T. Hospedales, “Deeper, broader pp. 1–11.
and artier domain generalization,” in Proc. Int. Conf. Comput. [288] H. Bharadhwaj, “Meta-learning for user cold-start recommen-
Vis., 2017, pp. 5542–5550. dation,” in Proc. Int. Joint Conf. Neural Netw., 2019, pp. 1–8.
[263] J. Devlin, R. Bunel, R. Singh, M. J. Hausknecht, and P. Kohli, [289] T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn,
“Neural program meta-induction,” in Proc. 31st Conf. Neural Inf. “Gradient surgery for multi-task learning,” in Proc. Int. Conf.
Process. Syst., 2017, pp. 1–9. Neural Inf. Process. Syst., 2020.
[290] Z. Kang, K. Grauman, and F. Sha, “Learning with whom to share Antreas Antoniou is currently working toward
in multi-task feature learning,” in Proc. 25th Int. Conf. Mach. the PhD degree, supervised by Amos Storkey,
Learn., 2011, pp. 521–528. from the University of Edinburgh. His research
[291] Y. Yang and T. Hospedales, “A unified perspective on multi- contributions in meta-learning and few-shot
domain and multi-task learning,” in Proc. Int. Conf. Learn. Repre- learning are commonly seen as key benchmarks
sentations, 2015, pp. 1–9. in the field. His research interest includes meta-
[292] K. Allen, E. Shelhamer, H. Shin, and J. Tenenbaum, “Infinite mix- learning better priors such as losses, initializa-
ture prototypes for few-shot learning,” in Proc. 36th Int. Conf. tions, and neural network layers, to improve few-
Mach. Learn., 2019, pp. 232–241. shot and life-long learning.
[293] F. Pedregosa, “Hyperparameter optimization with approximate
gradient,” in Proc. 33rd Int. Conf. Mach. Learn., 2016, pp. 737–746.
[294] R. J. Williams and D. Zipser, “A learning algorithm for continu- Paul Micaelli is currently working toward the
ally running fully recurrent neural networks,” Neural Comput., PhD degree, supervised by Amos Storkey and
vol. 1, no. 2, pp. 270–280, 1989. Timothy Hospedales, from the University of Edin-
[295] L. Metz, N. Maheswaranathan, C. D. Freeman, B. Poole, and burgh. His research interests include zero-shot
J. Sohl-Dickstein, “Tasks, stability, architecture, and compute: knowledge distillation and on meta-learning over
Training more effective learned optimizers, and using them to long horizons for many-shot problems.
train themselves,” 2020, arXiv: 2009.11243.
[296] A. Shaban, C.-A. Cheng, N. Hatch, and B. Boots, “Truncated
back-propagation for bilevel optimization,” in Proc. 22nd Int.
Conf. Artif. Intell. Stat., 2019.
[297] J. Fu, H. Luo, J. Feng, K. H. Low, and T.-S. Chua, “DrMAD: Dis-
tilling reverse-mode automatic differentiation for optimizing
hyperparameters of deep neural networks,” in Proc. 25th Int. Joint Amos Storkey is currently a professor of machine
Conf. Artif. Intell., 2016, pp. 1469–1475. learning and AI at the School of Informatics, Univer-
[298] S. Flennerhag, P. G. Moreno, N. Lawrence, and A. Damianou, sity of Edinburgh. He leads a research team
“Transferring knowledge across learning processes,” in Proc. Int. focused on deep neural networks, Bayesian and
Conf. Learn. Representations, 2019, pp. 1–23. probabilistic models, efficient inference and meta-
[299] Y. Wu, M. Ren, R. Liao, and R. Grosse, “Understanding short- learning.
horizon bias in stochastic meta-optimization,” in Proc. Int. Conf.
Learn. Representations, 2018, pp. 1–6.
[300] L. Metz, N. Maheswaranathan, J. Nixon, D. Freeman, and J. Sohl-
Dickstein, “Understanding and correcting pathologies in the
training of learned optimizers,” in Proc. Int. Conf. Mach. Learn.,
2019.
" For more information on this or any other computing topic,
Timothy Hospedales is currently a professor at please visit our Digital Library at www.computer.org/csdl.
the University of Edinburgh, and a principal
researcher with Samsung AI Research. His
research interests include data efficient and
robust learning-to-learn with diverse applications
in vision, language, reinforcement learning, and
beyond.

Meta-Learning in Neural Networks A Survey

Uploaded by

Copyright:

Available Formats

Meta-Learning in Neural Networks A Survey

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Meta-Learning in Neural Networks A Survey

Uploaded by

Copyright:

Available Formats

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 44, NO.

9, SEPTEMBER 2022 5149

Meta-Learning in Neural Networks: A Survey

1 INTRODUCTION Neural network meta-learning has a long history [7], [17],

2.1 Formalizing Meta-Learning In contrast to conventional learning in Eq. (1), learning on

Memory-augmented neural networks [83] use an explicit stor- 4.1 Meta-Representation

sampling random actions [226], or hand-crafted heuristics 5.2.2 Benchmarks

5.3 Neural Architecture Search (NAS) 5.5 Bayesian Meta-Learning

You might also like