Multipath Transmission For The Internet: A Survey

IEEE COMMUNICATIONS SURVEYS & TUTORIALS, VOL. 18, NO.
4, FOURTH QUARTER 2016 2887
Multipath Transmission for the Internet: A Survey

Ming Li, Andrey Lukyanenko, Zhonghong Ou, Antti Yl-Jski,
Sasu Tarkoma, Matthieu Coudron, and Stefano Secci
AbstractSmart devices equipped with multiple network since the birth of the Internet. The reason lies in the fact that,
interfaces are becoming commonplace. Nevertheless, even though by default, the conventional TCP/IP only uses a single best
multiple interfaces can be used to connect to the Internet, their path according to certain routing metrics; the other available
capabilities have not been fully utilized yet because the default
TCP/IP stack supports only a single interface for communication. paths remain standby only for backup and recovery purposes.
This situation is now changing due to the emergence of multi- Nonetheless, this situation has been changing in the past
path protocols on different network stack layers. For example, few years, which is indicated by several trends from the stan-
many IP level approaches have been proposed utilizing tunneling dardization organization, academia, and industry. From the
mechanisms for hiding multipath transmission from the trans- standardization perspective, both IEEE and IETF are active
port protocols. Several working groups under IEEE and IETF
are actively standardizing multipath transmission on the link on concurrent multipath transmission. There have been several
layer and transport layer. Application level approaches enable working groups dedicating on the standardization, for exam-
multipath transmission capability by establishing multiple trans- ple [2], [3], [11], [39], [51], [56], [61], [87], [112], and [127].1
port connections and distributing data over them. Given all these From the academia, hundreds of scientific articles revolving
efforts, it is beneficial and timely to summarize the state-of-the- around multipath transmission have been published, covering
art, compare their pros and cons, and discuss about the future
directions. To that end, we present a survey on multipath trans- different network stack layers on various aspects ranging from
mission and make several major contributions: 1) we present a packet reordering, scheduling to buffer management, fairness,
complete taxonomy pertaining to multipath transmission, includ- Resource Pooling (RP). From the industry, several companies
ing link, network, transport, application, and cross layers; 2) we have implemented their own link layer aggregation schemes,
survey the state-of-the-art for each layer, investigate the prob- such as [15], [32], and [94]. Deutsche Telekom offers hybrid
lems that each layer aims to address, and make comprehensive
assessment of the solutions; and 3) based on the comparison, we access by bundling DSL-line with LTE in its portfolio [43].
identify open issues and pinpoint future directions for multipath Tessares company [1] tried to develop new innovative network
transmission research. services on top of Multipath TCP (MPTCP). For example,
Index TermsMultipath transmission, TCP-friendly, resource its first product aims to aggregate the bandwidths of differ-
pooling, packet reordering. ent infrastructure (LTE/DSL). In 2015, OVH company [138]
announced a new product called Overthebox. This product
combines MPTCP and SOCKS proxies to enable users to bond
different DSL lines together. Apple has implemented a vari-
I. I NTRODUCTION
ant of MPTCP on part of their Siri servers and allows iOS 7
HE INTERNET was originally designed as a two-
T connected net to guarantee that no single failure would
cause any non-failed portion of the network to lose connec-
users to use it in their iPhones [4]. At IETF93 in 2015, KT
Corporation presented Gigapath, a commercial service which
can achieve high bandwidth (800 Mbps and more) by combin-
tivity [113]. In essence, any source-destination pair needs to ing LTE and WiFi networks on Multipath TCP enabled smart
maintain more than one path to assure the reliability and phones [20].
resiliency of the network. Although the rich resources have There already exist a few survey articles focusing on
been existing in the Internet, they have not been fully utilized different aspects of multipath transmission. For exam-
ple, [86], [108], [154], [174], and [193] mainly focused on
Manuscript received October 6, 2015; revised February 24, 2016; accepted the control plane problem (i.e., multipath routing of how to
June 9, 2016. Date of publication June 29, 2016; date of current version compute and select paths). Reference [183] covered the con-
November 18, 2016. This work was supported in part by the Academy of
Finland under Grant 278207, and in part by the HIIT Distributed and Mobile trol plane problem as well as the data plane problem (i.e.,
Cloud Systems Research Programme. how to split the flow on the chosen paths) in wired net-
M. Li, A. Lukyanenko, and A. Yl-Jski are with the Department works. Reference [153] assumed multiple paths had been
of Computer Science, Aalto University, Espoo 02150, Finland (e-mail:
[email protected]; [email protected]; [email protected]). established by routing protocols and focused on load dis-
Z. Ou is with the School of Computer Science, Beijing University tribution in terms of traffic splitting and path selection.
of Posts and Telecommunications, Beijing 100876, China (e-mail: References [45] and [163] considered multipath transmission
[email protected]).
S. Tarkoma is with the Department of Computer Science, University of in wireless and wired networks respectively. Reference [7]
Helsinki, Helsinki 00014, Finland (e-mail: [email protected]).
M. Coudron and S. Secci are with Sorbonne Universits, UPMC
University of Paris 06, UMR 7606, LIP6, 75005 Paris, France (e-mail: 1 Reference [39] gives an overview of bandwidth aggregation mecha-
[email protected]; [email protected]). nisms discussed in the context of Banana mailing list https://www.ietf.org/
Digital Object Identifier 10.1109/COMST.2016.2586112 mailman/listinfo/banana.
1553-877X c 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
2888 IEEE COMMUNICATIONS SURVEYS & TUTORIALS, VOL. 18, NO. 4, FOURTH QUARTER 2016
investigated the common features of various approaches capacity, they tend to cause problems such as congestion
and classified the features into layer-dependent and layer- collapse. RP is a concept that changed the notions of
independent features. In addition, [7] also abstracted common fairness in a way that made multipath communications
design patterns and proposed a unified networking architecture widely acceptable in practice. Instead of handling per
to enable mobile nodes to make context-aware decisions about path resource independently, the RP principle advocates
how and when to use each or a combination of networks. making improved use of multiple path resources by allow-
Nevertheless, to the best of our knowledge, this survey is ing separate paths to act as if they were a single large
the first one to provide a holistic view on the data plane issues resource. This principle is a significant step towards a
of multipath transmission. We have surveyed papers from the practical multipath-aware end system.
year 1975 to 2015, and investigated various research problems Pareto-optimality: it is a state of resource allocation in
from different layers, covering link layer, networking layer, which there is no alternative state that would make some
transmission layer, application layer as well as cross layer. The people better off without making anyone worse off. In the
primary research problems include packet reordering, fairness case of multipath transmission, it means that upgrading
control, RP, Pareto-optimality and path diversity. some regular single-path users to multipath ones can not
All frequently used acronyms in this survey are reported in reduce the throughput of other users with any benefit to
the Appendix. the upgraded users.
Security: as data can be distributed over independent
paths, it will be more complex for a malicious entity to
A. Why Multipath? capture the entire content.
As stated previously, the original Internet was designed as
two-connected net with path diversity in mind. Nevertheless, B. Potential Blocking Points
computers with multiple network interfaces were not an imme- Multipath transmission also has its own challenges we must
diate design priority at the early stage. Only the routers were face. Some of the requirements mentioned in the last sec-
equipped with several physical network interfaces. However, tion can be seen as disadvantages as well. In this section,
the Internet has since then evolved significantly. For example, we investigate some of the potential blocking points from the
most servers have been equipped with more than one network perspective of deployment in practice.
interface nowadays. The abundance of network resources from Packet reordering: it is difficult to schedule data packets
the server domain has spurred the adoption of multipath trans- over heterogeneous paths without causing reordering and
mission in data center networks. In the consumer electronics performance penalties. A robust multipath transmission
domain, the proliferation of mobile devices equipped with cel- solution should be able to cope with any kind of path
lular (e.g., 3G and LTE) and WiFi interfaces, represented by heterogeneity, throughput fluctuations, or jitter. It should
smart phones, brings with it a growing number of multi-homed also be able to deal with persistent reordering of data
hosts onto the Internet. Thus, there exists a mismatch between packets. Otherwise, users would have less incentive to
single-path transport and the multitude of available network upgrade if the solution fails to work properly in certain
paths. Those multi-interface devices require multipath capa- network environments.
bility to improve end-to-end communication performance and Fairness: traditionally, fairness has been one of the obsta-
resilience. cles to the concurrent multipath transmission. It used to
Meanwhile, technological advancement has made multi- be on a per interface basis which is unfair if there is a
path transmission possible in theory. We investigate some of common bottleneck later on. For example, simply utiliz-
the major benefits as well as requirements, and give brief ing multiple flows would result in an unfair share of the
descriptions as follows. bandwidth at the bottleneck; for example, n TCP flows get
Reliability: multipath transmission can enhance the reli- approximately n times throughput as a competing single
ability of data transfer because additional paths can keep TCP flow does.
the connection alive in case of a failing or less performing Compatibility: it is hard to implement a generic multi-
path. In wireless environment, reliability can be further path solution without modifying standardized protocols
improved because signal interference is minimized due to or changing third-party network equipments. For exam-
the use of heterogeneous wireless access techniques. ple, on link layer dedicated setup (even equipment) is
Bandwidth aggregation: it is expected that the band- required on both sides. On other layers above the link
width aggregation can potentially multiply the experi- layer, either hosts or networks (sometimes both) need to
enced throughput by the number of available paths. If upgrade in order to support multipath transmission.
efficient bandwidth aggregation can be achieved in this Pareto-optimality: MPTCP is the first multipath
manner, a multi-homed device can obtain a much better transmission proposal which requires Pareto-
performance. optimality [101], [102] but there is little experience
Fairness and RP: TCP fairness requires that a multi- how well/often it respects this requirement. There are
path transmission protocol receives no larger bandwidth indeed cases where MPTCP may perform worse than
of the shared bottleneck link than a competing TCP flow. normal TCP due to path heterogeneity.
This is important because TCP is the dominant transport Path diversity: it describes the ability of having multi-
protocol on the Internet. If new protocols acquire unfair ple disjoint paths to reach a destination. Users expect to
LI et al.: MULTIPATH TRANSMISSION FOR THE INTERNET: A SURVEY 2889
TABLE I
C LASSIFICATION OF THE R ESEARCH W ORK BASED ON THE I NTERNET P ROTOCOL L AYERS
obtain high throughput from the use of multiple paths. D. Organization, Structure, and Research Problems
But if the paths (or partial of them) travel through a Grouping and discussing multipath transmission approaches
shared bottleneck link, the multiple flows can only get according to their stack position are beneficial for researchers
as much throughput as a comparable TCP flow does due and practitioners to understand the benefits and trade-offs from
to the fairness guarantee. Currently, reliable bottleneck each layer, and make an all-around decision. Therefore, we
detection is really hard in practice. No individual path survey the state-of-the-art multipath transmission from layer-
selection scheme can fit with all network environments. specific perspectives. Table I shows the classification of the
Security: multipath transmission has broken trust mod- research work according to the stack position.
els organizations placed in single network providers. For The structure of the survey is organized as follows.
example, although traffic splitting makes sniffing harder, Section II reviews a number of related surveys and layouts the
firewalls or gateways may miss part of the data delivered position of ours. In Section III, various approaches are clas-
over more than one network providers and thus become sified based on their network stack position and cross-layer
unable to analyze the flow. This would result in broken approaches are discussed separately (see Table I). Figure 1
security solutions including intrusion detection and data illustrates the structure and coverage of Section III. In each
leak prevention. discussion of the layer specific approaches, we investigate the
Note that this survey is neither limited to these problems problems the approaches on that layer aim to address. Some
nor cover all of them. We present the scope of this survey in problems are common to all layers, such as the load sharing
Section II. and packet reordering problems. Some are addressed only on
certain layers. For example, the fairness problem is addressed
C. Contributions only on IP and transport layers. Compared with other lay-
We provide a comprehensive survey of multipath transmis- ers, transport layer approaches have more problems to address
sion, covering various aspects on different layers. Towards that including RP, buffer impact, Pareto-optimality and path diver-
direction, we make several key contributions and summarize sity. The discussion of approaches follows a chronological
them as follows: (1) a complete taxonomy regarding multipath order except that we group some research work which has
transmission is presented, covering various protocol layers similarity or progression. In addition, two tables are used to
including link layer, network layer, transport layer, applica- summarize the key algorithms and approaches respectively.
tion layer and cross layer; (2) the state-of-the-art for each The approach table is connected to the key algorithm table
layer is surveyed, the problems addressed by layer specific by the means of listing the key algorithms used in each
approaches are investigated, and comprehensive comparisons approach as well as the intended network environments of
among them are made; (3) the standardization efforts from var- the algorithms. Note that the same algorithms, which are used
ious parties are summarized, including working groups from on different layers, are not repeatedly described in different
IETF and IEEE; (4) by the means of comparison, open issues key algorithm tables. Instead, we only provide explanation
are identified for future development of multipath transmis- in the table when the algorithm is first discussed. Following
sion. We believe this work will bring insights to the researchers the discussion of the approaches on specific layers, we make
and practitioners working in this field, and foster a set of new a summary to present a comprehensive comparison from
research towards different directions of multipath transmission. five perspectives. In Section IV, we point out the lessons that
Fig. 1. Structure of Section III and research problems (P) to address.
can be learned from this survey. In Section V, we pinpoint adaptiveness to dynamic conditions: non-adaptive and adap-
future research directions. Finally, we conclude this survey in tive approaches. The former ones do not have the ability to
Section VI. adjust the resource allocation and traffic schedule in dynamic
In this article, there are many abbreviations. To help readers network conditions. In contrast, the latter ones take varying
track them easily, we provide a list of frequently used traffic and link conditions into consideration in order to derive
acronyms in the Appendix. optimal resource allocation and scheduling decisions. In each
group, the approaches are further classified according to their
layer position in TCP/IP protocol stack.
Prabhavat et al. [153] presented a literature review of vari-
II. S COPE AND R ELATED S URVEYS ous existing load distribution models for multipath networks,
First of all, we investigate multipath transmission in wired and classified the models in terms of their key functionalities:
and wireless networks but leave its discussion in sensor net- traffic splitting and path selection. A generalized multipath
works out of the scope. For surveys on multipath transmission forwarding mechanism was used to discuss the two key func-
in sensor networks, we refer the readers to [156]. In addi- tionalities without considering which protocol stack position
tion, we focus solely on the data plane problem of how to the mechanism is implemented on. The paper did not address
split data on multiple paths and intentionally leave out all routing to establish multiple paths. Instead, it assumed that
work that focuses on multipath routing, i.e., the control plane multiple paths had been established by routing techniques.
problem of how to compute and select the routes. We refer Domz a et al. [45] considered multipath transmission in
the readers to articles and recent surveys that cover such wired networks. They classified the approaches based on three
work [86], [108], [154], [174], [193]. Furthermore, the secu- different layers in which they operate, i.e., link layer, network
rity issue of multipath transmission and P2P applications are layer and transport layer. Specifically, on the link layer, they
also out of the scope of this survey. discussed a couple of multipath transmission approaches based
Compared with surveys on multipath routing, surveys on on Ethernet. On the network layer, they investigated various
the data plane problems are less popular. There are only a few routing techniques that can be used for the construction and
surveys [7], [45], [67], [153], [154], [163], [183] that touch selection of multiple paths. On the transport layer, they gave a
on the same topic as ours. In a somewhat old but still rele- brief introduction of MPTCP. However, [45] lacks many key
vant survey, Ramaboli et al. [163] reviewed some bandwidth approaches added in this survey. For example, on the network
aggregation approaches in heterogeneous wireless networks layer, all the tunneling based solutions are missing. On the
which consist of a variety of integrated and jointly managed transport layer, they only discussed MPTCP but have left out
radio access technologies. They found that packet reorder- all the other approaches.
ing is the most dominant challenge because it can introduce Addepalli et al.s [7] survey covers some aspects related
undesirable delays for real-time applications and unnecessary to multipath transmission. The authors first reviewed some
retransmissions for TCP applications. In this regard, their protocols and architectures that enable heterogeneous network-
survey focused mainly on the issues caused by packet reordering support, and then abstracted common design patterns and
ing and the approaches to address them accordingly. Those proposed a unified networking architecture to enable mobile
approaches were classified into two groups according to their nodes to make context-aware decisions about how and when
TABLE
II
C OVERAGE OF R ELATED S URVEYS ( M EANS H AVING B EEN C OVERED )
to use each or a combination of networks. However, the scope (host or network). In this survey, we organize the discussion of
of survey [7] is only partially overlapping with our work, in multipath transmission in a different way, for example, inves-
particular the focus is mostly shifted to seamless mobility, tigating the various problems to address on different stack
multihoming, security and other aspects that are out of the positions (see Figure 1). We believe that these two different
scope of our work. perspectives on the same target are complementary and would
Habak et al. [67] investigated the common features of vari- give readers more insights.
ous approaches and classified the features into two categories: As we have discussed previously, most of the existing sur-
layer-dependent and layer-independent features. The layer- veys have classified various solutions based on protocol layers
dependent features include, for example, the common research (excluding [153]). In this regard, we use Table II to compare
problems shared by the approaches on the same layer. In the the coverage of the previous surveys as well as this survey
discussion of these features, they performed a layer by layer on different protocol layers. In this survey, we summarize the
analysis of the available approaches. The layer-independent key algorithms used on all the other layers except the physi-
features include scheduling algorithms, estimation of inter- cal layer (we refer the readers to [29] and [199] for multipath
face/application characteristics, and networking models. In transmission at the physical layer) and associate the algorithms
contrast, in this survey, we argue that some seemingly layer- to the problems they were designed to address. We extract the
independent features are not completely layer independent. For scheduling algorithms and compare their efficiency in terms of
example, certain scheduling algorithms are closely connected packet reordering and load sharing capabilities without con-
to a specific layer so that they may perform differently on sidering their layer dependency. The approaches based on a
different layers. cross-layer design are summarized and discussed in a separate
Singh et al. [183] surveyed both multipath routing and pro- section. The approaches on different layers are also evaluated
visioning. In the discussion of provisioning multiple paths from the viewpoint of compatibility capability. We also discuss
between end hosts, they also followed a layer-based structure, the evolution of the research questions on multipath trans-
from application layer to physical layer, to review the exist- mission and found that only the transport layer approaches
ing approaches. Although [183] has a similar structure as our following a nice evolution.
survey, they have several key differences: 1) Refernce [183]
excluded multipath transmission in wireless networks which
are the primary environment for path diversity. Due to its focus III. M ULTIPATH T RANSMISSION
only on the wired networks, it missed many key references Before we dig into the technical details of multipath trans-
added in this survey; 2) Refernce [183] lacked a detailed dis- mission, we present the timeline of its milestones in Figure 2
cussion, for example, which scheduling algorithms are used to in order to give readers a general picture of its development.
avoid reordering in the receiver and how well these algorithms The first paper on TCP was published in 1974. In the follow-
perform. Instead, this analysis is one of the main focuses in ing year, Dr. Maxemchuk proposed Dispersity Routing [126]
our survey; 3) compared to our survey, [183] has missed some in his Ph.D. dissertation to concurrently transmit data over
important contents including whether the approaches consid- multiple paths. From that point onward, various forms of mul-
ered fairness, implemented the Pareto-efficiency and resource tipath transmission have been proposed. For example, the idea
pooling features, and what compatibility issues the approaches of building multipath capability into TCP was, to the best of
had on each layer. our knowledge, first suggested by Huitema [87] as an Internet
Qadir et al. [154] investigated multipath transmission on draft in IETF in 1995. In 2002, the first 3G network to go com-
the transport layer, despite their main focus on network-layer mercially live was launched in South Korea, which promoted
multipath routing. They organized their investigation by dis- the proliferation of mobile devices equipped with multiple
cussing five questions relating to how the multiple paths are wireless interfaces. In 2006, Key et al. [100] used fluid-flow
used. The five questions have covered: 1) the number of paths modeling to demonstrate that multipath transport can provide
to be used concurrently, 2) the configuration of how multiple not only robustness but also balanced congestion in a sta-
paths work together (backup or concurrent), 3) the load bal- ble manner. In the same year, Shakkottai et al. [176] used a
ancing methods (static or dynamic), 4) the congestion control non-cooperative pricing game to show that multihoming out-
methods (coordinated or uncoordinated) and 5) the controller performs unihoming in terms of throughput and profit to the
entity which performs load balancing and traffic engineering Internet service providers (ISPs). In 2008, Wischik et al. [200]
Fig. 2. Milestones in the evolution of multipath transmission. MT: Multipath Transmission, DCN: Data Center Network, MPTCP: Multipath TCP.
mentioned in Table IV. Note that the algorithms in Table III

may be not only adopted by approaches on the link layer, but
may also be used by those on other layers. In this survey,
we will not elaborate the algorithms that have been discussed
previously. This same rule is applicable for all other algorithm
tables.
Multi-Link PPP (MP) [185], designed for Integrated
Services for Digital Network (ISDN), aggregates multiple
links using the PPP protocol [181]. In order to detect frag-
ment loss and disorder, MP uses a 4-byte re-sequencing header
Fig. 3. Link aggregation between Ethernet switches. SW: Switch.
(RSH) for synchronization and detecting lost fragments at the
receiver. Therefore, a reorder buffer is required at the receiver
to accommodate the out-of-order fragments caused by link
investigated the RP principle, which makes a collection of
aggregation. MP suggests a Weighted Round Robin (WRR)
resources behave like a single pooled resource. This princi-
scheduling scheme so that data can be distributed proportion-
ple is a significant step towards a practical multipath-aware
ally to the transmission rates of the links. To achieve this goal,
end system. From 2009, IETF started to define and stan-
two methods for fragmentation have been proposed. The first
dardize MPTCP, which employs a coupled congestion control
one divides packets into segments with sizes proportional to
algorithm to achieve RP principle.
the transmission rates of different paths. The other method
In the remainder of this section, the state-of-the-art multi-
divides packets into many small equal sized fragments and
path transmission schemes are classified according to which
distributes the number of the fragments proportionally to the
layer of the protocol stack the proposed approach performs at:
transmission rates of different paths.
link layer, network layer, transport layer and application layer.
Adiseshu et al. [9] and Hari et al. [73] added a strIPe
layer, a virtual IP interface below the IP layer and above the
A. Link Layer Bonding data link layer, to aggregate multiple data links. The stripe
High end workstations and data centers can easily satu- layer implements the striping algorithm at the sender and the
rate existing Local Area Networks (LANs). On the link layer, fair queuing algorithm at the receiver. The authors first showed
multipath transmission is typically called bonding or link how a fair-queuing algorithm (FQA) can be transformed to a
aggregation because multiple physical channels are bundled fair load-sharing algorithm (FLSA), and then proposed that
(or aggregated) into a single logical channel. The primary the FQA should run in a reversed manner of the load-sharing
goal of link layer bundling is to coordinate multiple inde- algorithm in order to solve the load sharing issue with variable
pendent links between a fixed pair of systems, providing a packet size. It implies that identical equipment (or of the same
virtual link with a larger bandwidth than what a single link vendor) is required on both sides of the aggregation. They also
can sustain. Figure 3 shows a simplified example of link dealt with the FIFO delivery issue for two separate cases. For
aggregation between two Ethernet switches (SW1 and SW6 ). example, if a RSH can be added to each packet, the issue
These switches can obtain increased throughput by striping can be solved by using the additional reordering number; if
data across multiple interfaces. no header can be added, they proposed a way of synchro-
In the following discussion, we use Table III and Table IV nization in the event of frame loss to provide quasi-FIFO
to summarize the key algorithms and approaches respectively. delivery.
The approaches in Table IV are sorted in chronological order. An implementation of MP in Wide Area Networks (WANs)
The algorithms in Table III are sorted according to their order was discussed by Snoeren [186]. They proposed a Link Quality
TABLE III
K EY A LGORITHMS FOR L INK L AYER B ONDING (S ORTED ACCORDING TO T HEIR O RDER M ENTIONED IN TABLE IV)
TABLE IV
S UMMARY OF L INK L AYER B ONDING A PPROACHES
Balancing (LQB) scheme to bundle multiple channels of the aggregated together to form a Link Aggregation Group (LAG).
same Wide-area Wireless Access Network (WWAN) technol- As such, the Media Access Control (MAC) client can treat the
ogy. In order to adjust traffic striping across bundled links LAG as a single link. LACP allows a network device to nego-
according to their transmission capabilities, LQB adapts the tiate an automatic bundling of links by sending LACP packets
maximum transmission unit (MTU) of each link in proportion to the peer. LACP was initially released as 802.3ad [88] in
to its available bandwidth (short-term averages of the observed 2000. Nearly every network vendor quickly adopted this stan-
throughput). A link layer receive buffer is also required to dard over their proprietary standards. In 2008, the protocol
reorder fragments. was transferred to the 802.1 group with the publication of
FatVAP (an 802.11 driver design) [95] is another work in IEEE 802.1AX-2008 [2]. In LACP, a Frame Collector (FC)
a wireless environment for link bonding. FatVAP aggregates at the receiver is responsible for maintaining any frame order-
the bandwidth available at multiple wireless access points ing constraint. In order to avoid frame reordering, the Frame
(WAPs) that are worth connecting to and balances their loads Distributor (FD) at the sender transmits all frames that com-
by scheduling traffic to different APs according to their avail- pose a given conversation2 only to a single link, which is
able bandwidth. In order to continue delivering the sum of the a Per-Conversation Allocation (PCA) strategy (very similar
bandwidths available across all APs, FatVAP uses a constant to PFA strategy). Therefore, no frame reordering scheme or
estimation of both end-to-end and wireless bandwidth to react reordering buffer is required at the FC. In addition to the
to changes within a few seconds. Note that FatVAP uses a Per- IEEE link aggregation standards, there are a number of propri-
Flow Allocation (PFA) strategy to distribute traffic over APs. etary aggregation schemes, including EtherChannel [32] from
For example, when a new flow arrives, FatVAP determines Cisco, Aggregated Ethernet [94] from Juniper, Multi-Link
which AP to assign this flow to and records the mapping in Trunking [15] from AVAYA. These proprietary aggregation
a hash table. Subsequent packets in the flow are simply sent
2 A set of frames transmitted from one end station to another, where all
through the AP recorded in the hash table.
of the frames form an ordered sequence, and where the communicating
Within IEEE specifications, the Link Aggregation Control end stations require the ordering to be maintained among the set of frames
Protocol (LACP) allows multiple links of Ethernet to be exchanged.
TABLE V
S CHEMES U SED ON IP L EVEL FOR BANDWIDTH AGGREGATION
schemes and IEEE 802.3ad standards are very similar and Protocol (STP). SPB now supports multipath forwarding by
accomplish the same goal. using Equal Cost Multipath (ECMP) and Equal Cost Tree
Since the version 1.1 [33], OpenFlow has supported multi- (ECT) routing strategies. TRILL currently only supports mul-
link aggregation on layer-2. The specification of OpenFlow tipath forwarding by using ECMP routing strategy. In both
switch has introduced Link Aggregation (LA) to obtain the ECMP and ECT strategies, if multiple equal cost paths are
ability for one port to point to a group of other ports. Using present towards a destination, network traffic is distributed
LACP for exchanging dynamic information between LA- over those multiple paths. In order to avoid reordering and
supported devices, the OpenFlow controller has full control path MTU discovery problems, similar to FatVAP, both TRILL
over the switches on how frames are distributed and col- and SPB use per-flow multipath forwarding. Specifically,
lected on multiple links. Nguyen-Duc et al. [135] investigated frames belonging to the same data flow take the same path
the operation of LA in OpenFlow switches and found that and frames belonging to other flows can take the other
LACP on OpenFlow switch provides a slightly lower through- paths.
put than the one on a conventional switch. Thus, OpenFlow From Table III, we find that the main problems the
switches need to be further optimized to achieve equivalent algorithms trying to address are load sharing and packet
performance. Subedi et al. [188] presented an adaptive mul- reordering. Among all the algorithms, the simplest one is
tipath forwarding architecture in a layer-2 OpenFlow data Round Robin (RR) where the sender allocates fragments
center network. In the architecture, all-to-all forwarding paths among all the available links in equal portions and in an
are set up proactively among the edge nodes. Aggregated ordered fashion. In the long run, RR scheduling provides a
bandwidth is achieved by using all the available paths simul- fair share of fragments as long as these fragments are of the
taneously. To avoid the out-of-order delivery issue due to same size. Nevertheless, the basic RR scheduling strategy is
using all available paths, a PCA style scheduling algorithm is rarely used in practice because it provides no load sharing with
used in [135] and [188]. Specifically, the algorithm excludes either variable sized fragments or different link capacities. In
the paths whose path length exceeds the shortest path length order to solve these issues, the Weighted RR variations are the
significantly. more widely used scheduling strategies.
In the last few years, there are notable new protocols The main advantage of link-layer bonding is that the signal-
designed to support multipath forwarding at link-layer in IEEE ing rate of the channel is relatively stable and can be utilized to
and IETF standards, e.g., Shortest Path Bridging (SPB) [57] mitigate reordering. However, link-layer approaches only work
(specified in the IEEE 802.1aq standard [3]) and IETF on a point-to-point link and even require dedicated Ethernet
Transparent Interconnection of a Lot of Links (TRILL) [51]. cards installed on both sides. Thus, they are not applica-
SPB and TRILL are potential successors of the Spanning Tree ble in general scenarios of end-to-end communications where
TABLE VI
K EY A LGORITHMS FOR IP L AYER BANDWIDTH AGGREGATION (S ORTED ACCORDING TO T HEIR O RDER M ENTIONED IN TABLE VII)
the different domains involved are controlled by different

providers.
B. IP Layer Bandwidth Aggregation

The IP layer, originally proposed to handle global address-
ing and routing, is a natural candidate to host the multipath
capability to enhance end-to-end communication. A network
approach has the advantage of being transparent to transport Fig. 4. IP-in-IP tunneling between two multi-homed hosts.
protocols and applications, making wide spread deployment
much easier. In theory, each packet of a TCP flow can be
sent over a different path, and the IP protocol ensures that all Thus, the use of multiple paths with varying characteristics
packets reach their destination. For example, Sun et al. [189] deteriorates the problem.
explored the use of multipath routing to reduce the file In the following discussion, the state-of-the-art is divided
transmission delay in a wireless network. Specifically, they into three categories: IP-in-IP encapsulation, Network Address
proposed taking advantage of packet level erasure code (e.g., Translation (NAT) traversal, and Identity/locator split. We
digital fountain code) to transmit data file with redundancy summarize their features in Table V and discuss each cate-
over a set of paths. They obtained the intuitive understanding gory according to the order they show in the table. Table VI
of the trade-off between the code rate and delay reduction. and Table VII are used to summarize the key algorithms and
Their research was made for a special network environment approaches respectively. The Proxies or Updated Routers in
where a source and destination pair has a rich set of identical Table VII indicates the required number of proxies or updated
and disjoint paths, hence no packet reordering issue intro- routers.
duced. Nevertheless, in most practical network environments, 1) IP-in-IP Encapsulation: A widely used IP layer
when packets inside one connection taking more than one path, approach for aggregating bandwidth of multiple IP paths is
they can experience different propagation delay and arrive out to use tunneling mechanisms which transparently redirect
of order. The TCP receiver sends duplicate acknowledgments packets between two hosts on routing level. For example,
(ACKs) to the sender, which causes the TCP sender mistakenly Phatak et al. [148], [149] proposed using IP-in-IP encap-
interprets packet reordering as packet loss. The results found sulation [144] to split a data flow across multiple network
in [19], [104], and [203] show that TCP suffers significant interfaces. As shown in Figure 4, at the source (A), the trans-
performance degradation due to frequent packet reordering. port layer assembles all packets as if they were going through
TABLE VII
S UMMARY OF IP L EVEL BANDWIDTH AGGREGATION A PPROACHES
A1 and addressed to B1 . The packets going out on interface the WWAN connections from community members. The proxy
A2 get encapsulated in new IP packets each with an extra can be a trusted party or a community member. PRISM uses
header having destination B1 and source A2 . Likewise, each a cross-layer approach that involves support from both trans-
packet going out on interface A3 can be encapsulated in a new port and network layer. We classify PRISM as network layer
IP packet having destination B2 and source A3 . The destina- approach because the PRISM proxy, which is the main entity
tion (B) can then recognize IP-in-IP packets and strip the outer for multipath support, is located on the network layer. PRISM
header. This leaves the original packets with source A1 and uses a packet-scheduling algorithm, i.e., Adaptive Scheduler
destination B1 to be delivered up the network stack to TCP (ADAS), to maintain up-to-date path state. Using the up-
in a transparent manner. The same encapsulation scheme is to-state information, ADAS sends packets according to their
used for tunneling in the mobile IP standard [145]. In order expected arrival time (a variant of EDPF algorithm) to reduce
to avoid fast retransmission, Phatak et al. [148], [149] used a packet reordering. ADAS also uses the path state to adjust
WRR style scheduler, which distributes packets proportionally path weight by using the Additive Increase and Multiplicative
to the effective rates of the paths. Decrease (AIMD) strategy from TCP, so that ADAS can
Chebrolu and Rao [27] presented a network layer architec- dynamically react to congestion from partial paths and control
ture to aggregate bandwidth on multiple paths for real-time the amount of traffic to be allocated on those paths. Moreover,
applications. They made the assumption that an infrastructure PRISM masks the effects of out-of-order delivery by identify-
proxy (like the Home-Agent in Mobile IP [146]) is aware of ing spurious duplicate ACKs and re-sequencing them so that
the multiple interfaces of the client, and tunnels the captured a TCP sender receives correctly sequenced ACKs.
packets to the client using IP-in-IP encapsulation. The advan- Lan and Li [109] designed a different proxy based mul-
tage of a proxy solution is that it is fully controllable and tipath network protocol called Enhancements for TCP On a
allows servers to remain unchanged and hide using multiple Multi-homed mobile router (ETOM) that runs transparently
IPs from TCP. Chebrolu et al. [26] proposed a scheduling to both clients and servers. ETOM involves two proxy com-
algorithm, Earliest Delivery Path First (EDPF), to ensure that ponents instead of one kind: MR (Mobile Router) and HA
packets meet their playback deadlines by scheduling pack- (Home Agent). The client and the MR (as well as the server
ets based on the estimated delivery time of the packets. To and the HA) use normal single-path connection, whereas all
improve the overall performance of IP-in-IP tunneling based packets traveling between MR and HA are IP-in-IP encap-
bandwidth aggregation by the means of minimizing packet sulated. ETOM uses a reordering buffer to eliminate packet
reordering, Chebrolu et al. [26] proposed a two-pronged reordering. For example, out-of-order packets are buffered at
approach. Firstly, a scheduling policy Packet-Pair based EDPF the HA until the missing packets are received, and packets are
for TCP applications (PET) was used to partition traffic onto then sent out in order to the destination. ETOM also uses a
different paths. The design of PET has the same concept of variant of EDPF algorithm to further reduce packet reordering.
EDPF but with idealized delay and bandwidth values replacing Note that unlike other IP layer approaches, ETOM employs
the estimates. Secondly, working together with the scheduling a subflow sequence number in the inner IP header to detect
policy, a receiver-side Buffer Management Policy (BMP) was packet loss between MR and HA.
used to delay forwarding the out-of-order packets to TCP and 2) NAT Traversal: Unlike the previous approach that relies
to detect losses, so that a variety of adverse effects can be on IP encapsulation, there exist several approaches taking
hidden. advantage of NAT instead of tunneling.
Kim and Shin [104], [105] introduced PRISM, another Rodriguez et al. [166] introduced MAR system (see
proxy based approach that enables TCP to efficiently utilize Figure 5), a commuter mobile access NAT router that provides
Fig. 5. MAR [166] system architecture where the MAR router is placed in public mobile vehicles and data traffic is sent from remote servers to local
devices. PDN: Public Data Network, AP: Access Point.
a set of local interfaces and a number of wide-area wireless additional interfaces. The client does the inverse address trans-
interfaces. The former provides access to local mobile devices lation of the packets arriving at non-default interfaces and
and the latter accommodates a variety of wide-area wireless forwards the packets internally. In order to mitigate packet
technologies. The MAR router acts as a NAT box that is reordering, Evensen et al. [52] uses a WRR based sched-
located in the middle and translates IP addresses and ports uler in the NAT to distribute packets according to estimated
of packets for two directions. A MAR router can work alone throughput ratio of available paths.
or cooperate with a MAR proxy-server. With such a proxy- Habak et al. [68] proposed OSCAR architecture that works
server, the Packet-Oriented Scheduling Mode (POSM) is used in a distributed environment. An OSCAR-enabled node can
where the packets of the same TCP flow can be delivered share and use the bandwidth available from its OSCAR-
over multiple paths and a MAR router can implement intelli- enabled neighbors to connect to both legacy and OSCAR-
gent optimization including avoiding TCP 3-way handshake, enabled servers. OSCAR has a NAT module at both sides of
slow-start, spurious timeouts and so on. When the proxy-server a connection. At the sender, the NAT module replaces the
is absent, the Flow-Oriented Scheduling Mode (FOSM) is source and destination IP addresses with the used IPs for
used where a per-flow allocation strategy schedules all pack- transmission. Upon receiving a packet, the NAT module at the
ets belonging to the same TCP flow onto the same path. MAR receiver reverses the source and destination IPs by replacing
provides an API that can accommodate any custom purpose- them with the negotiated ones before delivering the packet to
built scheduling protocol. But the scheduling protocol itself is TCP. When a connection goes through a shared neighbor to a
not part of the MAR architecture. MAR is also designed to legacy server, the neighbor also needs to have a NAT module
determine the weight that should be assigned to each inter- that conducts the address translation operation. OSCAR uses a
face to properly perform load balancing (e.g., dynamically Packet Reordering Module (PRM) to handle packet reordering
shifts load from poor quality to better quality channels). Note issues. Specifically, it delays the packets and their ACKs on
that the MAR router was supposed to be placed in moving both sides respectively before forwarding them to the upper
vehicles, where users can use their devices for Web-browsing layer. OSCAR has two scheduling modes: POSM and FOSM.
and audio/video streaming. Therefore, the traffic load-balanced FOSM is used if the server is a legacy server, where PFA is
is only in one direction (i.e., from remote servers to local used. POSM is used if the server is OSCAR-enabled such that
devices). a WRR style scheduler is used.
Manousakis and Famolari [122] proposed INTELiCON to Some of the IP-in-IP encapsulation and NAT
allow devices to exploit wireless access diversity. At the send- baed approaches, e.g., [26], [27], [52], [104], [105],
ing side, a Packet Processing module manipulates the content [109], and [166], assume the presence of a proxy infrastruc-
of packets, e.g., modifying IP headers and Piggy-Backing for ture in the network. Nevertheless, such approaches work only
Control Signaling (PBCS) (e.g., timestamp and customized for plain-text TCP communication and fail in the presence
sequence numbers). At the receiving side, the piggy-backed of IPsec encryption or authentication mechanisms. When
information can be utilized to smooth out the arrival sequence TCP packets are protected with IPsec, the proxy is not able
of incoming packets. The extra information is stripped out to observe or modify the packet headers. Next we discuss
at the Packet Processing module before the packets are for- certain bandwidth aggregation approaches that use IPSec
warded to the upper layer. Moreover, INTELiCON uses a encapsulation for tunneling.
DATA/ACK separation (SEP) scheme to reduce contention on 3) Identity/Locator Split: Host Identity Protocol
shared media. For example, it transmits the DATA and ACK (HIP) [131]3 and Site Multihoming by IPv6 Intermediation
packets on different paths. (shim6) [137] have been proposed and implemented to
Evensen et al. [52] proposed a Multilink Proxy (MLP)
that makes use of a NAT proxy to rewrite the default des- 3 HIP may not be considered as a strict IP layer approach; however, its
tination IP address and port to the address of the other functions related to multipath transmission are best suited to this layer.
provide multihoming support for failover with the possibility used for packet reordering, the additional sequence number in
of flow-based load balancing. shim6 is architecturally related HIP is used for the purpose of anti-replay.
to HIP in that they both introduce an additional addressing Unlike HIP and shim6 which focus on host-level identity
layer to allow changing IP addresses on network interfaces, and locator separation, Locator/Identifier Separation Protocol
while keeping constant transport-layer identifiers. These two (LISP) [56] is an identity and locator separation protocol
protocols enable IP packet flows to dynamically change working on the network-level to improve the scalability of
paths in the presence of link failure. Thus, they naturally the routing system. LISP creates two numbering spaces and
shield the presence of multiple paths from transport and uses two IP addresses: Endpoint Identifiers (EIDs) (assigned to
application layers, presenting only the global identity of end-hosts) and Routing Locators (RLOCs) (typically assigned
the peer host. Nevertheless, HIP and shim6 do not support to border routers). To achieve the separation of identification
simultaneous multipath transmission without additional and localization, LISP follows a map-and-encapsulate scheme.
extensions. Specifically, upon reception of a packet from the local network
SIMA [150] is an extension of HIP to use multihoming for to an outer EID, the border router is responsible for look-
assigning separate TCP connections independently to different ing up and retrieving the mapping (from a mapping system)
paths. Like FatVAP [95], TRILL [51], SPB [57], MAR [166] between EID and RLOC and this process is invisible to the
and OSCAR [68], SIMA also uses PFA multipath forward- endpoints. Then the router encapsulates the packet with a LISP
ing strategy where flow bonding rules are created to define header and an outer IP header with the destination RLOC as
the usage of the local interfaces. SIMA does not define any the destination IP address. When the packet reaches the border
additional sending or receiving policies to mitigate reordering router assigned with the destination RLOC, the router decap-
issue, instead it uses the IPsec Encapsulating Security Payload sulates the outer headers and forwards the inner packet to the
(ESP) packet processing unit built in HIP to handle each destination EID. LISP has two metrics to support multipath
data packet, as specified in [93]. Gurtov and Polishchuk [65] transmission: RLOC priority and RLOC weight. If equal pri-
designed and implemented Multipath HIP (mHIP), a multi- ority is sent on the RLOCs, the RLOC weight could be used
path scheduler based on HIP, to distribute traffic over multiple for the load-balancing ratio. Under such a setting, an IP-level
available paths. Utilizing a EDPF scheduling algorithm, they aggregate flow (e.g., the same destination prefix) would use
striped packets within a TCP connection to multiple paths different paths.
to mitigate packet reordering. Nevertheless, they found that Locator/Identifier Separation Protocol - Hybrid Access
EDPF algorithm is only effective against packet reordering (LISP-HA) [127] is a mechanism to provide simultaneous
with stable paths in terms of bandwidth and delay. In order to hybrid access (e.g., DSL-line and LTE ) based on LISP tech-
react to dynamic path characteristics, a Marking Technique is nology in both upstream and downstream direction. LISP by
used as a part of the multipath congestion avoidance scheme, itself has basic capabilities to support hybrid access with static
so that changes of path characteristics can be detected in load balancing. However, static load balancing may lead to
one Round-Trip Time (RTT). statistical variations [128] so that some paths are already over-
Polishchuk and Gurtov [151] proposed a TCP-friendly loaded while others are underutilized. Instead, LISP-HA can
congestion control algorithm for mHIP to prevent stealing perform dynamic per-flow load-balancing, which increases the
bandwidth from legacy TCP flows at the shared bottleneck. efficiency of hybrid access. The basic idea is to obtain feed-
Specifically, they proposed a two-level congestion control back about path-specific packet loss and delay, and leverage
scheme (removing the Marking Technique): per-path conges- this information for improved load balancing. In addition,
tion control, and global congestion control on top of it. The LISP-HA also supports dynamic per-packet load-balancing.
global congestion controller coordinates the individual per- Currently, the challenge is the packet reordering problem in
path controllers and balances traffic load among the paths the case that paths have different delay.
based on their available capacity. The per-path controllers As summarized in Table VI, packet reordering is one of the
are connected so that the aggregated congestion window main challenges for all IP layer approaches. These approaches
is the sum of per-flow congestion windows. The goal of use various scheduling algorithms to minimize the reordering
this twofold congestion control scheme is to automatically effect. In Table VII, we have several observations. First, most
redirect traffic from congested paths to the ones that have of the approaches were proposed either for mobile networks or
available capacity. The concept of joint congestion control wireless networks. Second, there are two primary scheduling
algorithm adopted in [151] is also used by certain trans- schemes: EDPF and WRR. Third, some buffer management
port layer approaches (which will be discussed in the next strategies are used to compensate for the inefficiency of the
section). Thus, the concern is that the reordering and conges- scheduling algorithm in the scenario of dynamically changing
tion avoidance algorithms used on the IP layer (or between networks.
IP and TCP, like HIP) may need to repeatedly design addi-
tional mechanisms that are already existing on the transport
layer. C. Transport Layer Multipath Transmission
When ESP is used with HIP, a 64-bit sequence number Compared with IP layer based approaches, transport layer
must be used. Therefore, HIP based bandwidth aggregation approaches have certain inherent benefits because congestion
approaches such as SIMA [150] and mHIP [65], [151] all have control can be used as a mechanism for resource allocation
a double sequence space design. However, instead of being in a network. At this layer, end-systems can easily obtain
TABLE VIII
C OMPARISON OF P ROBLEMS A DDRESSED BY SCTP AND TCP BASED M ULTIPATH T RANSMISSION A PPROACHES
information about each path: capability, latency, loss rate and Table IX and Table X are used to summarize the key algo-
congestion state. This information can then be used to react rithms and approaches of SCTP based multipath transmission
to congestion in the network by moving traffic away from respectively. Likewise, Table XI and Table XII are used to
the congested paths. Current connection-oriented transport summarize the key algorithms and approaches of TCP based
protocols, e.g., TCP, Stream Control Transmission Protocol multipath transmission respectively.
(SCTP) [187], and Datagram Congestion Control Protocol 1) SCTP Based on Multipath Transmission: A standard
(DCCP), transmit data only over a single path between a SCTP is a general-purpose, connection-oriented unicast trans-
source and a destination at any given time. Numerous attempts port protocol. An SCTP connection denotes an associa-
have been made to tune these existing transport protocols for tion. The user data is segmented into units of so called
multipath capability. Currently, Concurrent Multipath Transfer DATA chunks,4 which are identified by unique Transmission
(CMT) for TCP and SCTP are in the process of IETF Sequence Numbers (TSNs).5 The Selective Acknowledgment
standardization. (SACK) [125] mechanism is used as default to acknowl-
Like bandwidth aggregation on network layer, concurrent edge received data chunks and report gaps (i.e., missing data
multipath transmission at the transport layer introduces an chunks indicated by their TSNs) to the sender. In this section,
increase in the occurrence of packet reordering due to different we divide the approaches into two groups differentiated by
path characteristics, including run-time throughput, RTT, loss, whether they have considered fairness.
and error. Specifically, if a connection is striped over mul- a) Multipath transmission without considering fairness:
tiple network paths, the overall throughput may potentially Unlike TCP, SCTP was designed with multihoming in mind
be even worse than the throughput available on any one of that an SCTP association allows multi-homed source and des-
the paths [58], [140]. There are two main causes of it. The tination endpoints. Nevertheless, SCTP uses only one primary
first comes from the impact of heterogeneous RTT. For exam- path and switches to another path for retransmission of lost
ple, TCP expects a first-in-first-out delivery of packets through packets, or as a backup in the case of failure from the primary
the network. Packet reordering at the receiver results in the path. Note that SCTP uses a single buffer structure on both
reception of duplicate ACKs at the sender. The sender will endpoints, but maintains several states per destination: sepa-
fast retransmit the missing packet that may still be on its rate congestion window (cwnd), slow start threshold (ssthresh),
way over a high RTT path. Due to the misunderstanding of retransmission timer, and RTT estimate. Several SCTP exten-
packet reordering, the overall throughput may degrade signif- sions, such as BA-SCTP [13], W-SCTP [24], CMT-SCTP [11],
icantly. The second cause is the receive buffer blocking due [89][91], [120], LS-SCTP [5], [6], WiMP-SCTP [85], cmp-
to path heterogeneity or path failing. We provide more detail SCTP [118], mSCTP-CMT [21] and FPS-SCTP [129], enable
about the receive buffer blocking problem in later discussion SCTP to transmit data over multiple paths simultaneously.
on CMT-SCTP. Argyriou and Madisetti [13] proposed BA-SCTP, a band-
In this section, we classify the state-of-the-art according to width aggregation protocol based on SCTP. BA-SCTP imple-
their base protocols, in the order of SCTP and TCP. In each ments a mechanism for identifying bottlenecks that are shared
discussion of SCTP and TCP based approaches, we further by flows from the same aggregate connection. Based on this
divide them into two categories: with and without consid- mechanism, BA-SCTP performs a unified congestion control
ering fairness. In Table VIII, we make a comparison of the algorithm for the flows that share the same bottleneck (we
general problems addressed by approaches in each category. name this algorithm UCCSB) instead of applying congestion
In addition to the fairness issue, we also analyze the buffer control for each flow separately. This design guarantees that
impact, Pareto-efficiency, and path diversity of the TCP based
approach MPTCP. In the end, we give a comparison between 4 Corresponding to segments in TCP.
two representative approaches based on SCTP and TCP (i.e., 5 TSN serves the same function in SCTP as the sequence number does
CMT-SCTP and MPTCP). in TCP.
TABLE IX
K EY A LGORITHMS FOR SCTP BASED C ONCURRENT M ULTIPATH T RANSMISSION (S ORTED ACCORDING TO T HEIR O RDER M ENTIONED IN TABLE X)
BA-SCTP flows are fair with the other TCP flows sharing the where each subflow pulls data from the shared sending buffer
same bottleneck. BA-SCTP employs a WRR style scheduling whenever the subflow has congestion window space to send
strategy, a congestion window based data allocation strategy data. This strategy assumes the congestion window to be a true
TABLE X
C ONCURRENT M ULTIPATH T RANSFER P ROTOCOLS BASED ON SCTP
representative of the bandwidth-delay product of the subflow

compared to an estimated product. In the rest of the paper,
we use WRR-PULL to denote congestion window based data
allocation strategy.
Casetti and Gaiotto [24] proposed W-SCTP, a
Westwood [124] flavored SCTP to exploit bandwidth
aggregation. The authors believed that Westwood style
congestion control could fully exploit the advantages of
bandwidth estimation which could be utilized for traffic Fig. 6. HLB: the receive buffer cannot accommodate other chunks any more
allocation among multiple flows. W-SCTP uses a EDPF style before the arrival of the head-of-line chunk (chunk 1). HLB: Head-of-Line
Blocking.
scheduler that chooses the path for next packet by predicting
whether it can deliver the packet the fastest to the destination.
Iyengar et al. [89], [91] proposed integrating CMT capabil-
ity into SCTP, namely CMT-SCTP. CMT-SCTP utilizes the receive buffer blocking. This blocking problem is also named
multihoming feature from SCTP to correctly transfer data Head-of-Line Blocking (HLB). Figure 6 illustrates a simplified
between multi-homed end hosts. They identified three neg- example of it. As shown in the figure, a CMT-SCTP receiver
ative side-effects of CMT, and proposed algorithms to solve maintains a single receive buffer which is shared across two
them accordingly. First, they proposed a Split Fast Retransmit sub-association flows in an association. The C1 (chunk 1) is
Changeover Aware Congestion Control (SFR-CACC) algo- transmitted through the path 2 and is lost due to traffic conges-
rithm to eliminate the unnecessary fast retransmissions by tion or path failure. During the time period of C1 s retransmis-
using a different interpretation of SACK information. Second, sion, the receive buffer cannot accommodate any other packets
they used a congestion window (cwnd) growth algorithm to due to flow control so that the overall throughput degrades.
track the earliest outstanding TSN per destination and update To solve the problem, Liu et al. [120] proposed a compound
the cwnd, even in the absence of new cum ACKs. Third, parameter retransmission policy named RTX-LCS. It limits the
they proposed a new Delayed ACK algorithm for CMT-SCTP, retransmission path selection by considering three common
namely Delayed ACK for CMT (DAC). The algorithm allows conditions: cwnd, ssthresh, and loss rate. Adhari et al. [8]
the receiver to delay sending ACK of an out-of-order segment. and Dreibholz et al. [49], [50] examined the challenges of
In [90] and [91], they proposed five retransmission policies for CMT-SCTP over dissimilar paths. They identified the issues
CMT. They demonstrated the occurrence of spurious retrans- of sender and receiver queue blocking, which may lead to
missions with all of those policies, and proposed amendment poor overall performance. In order to improve performance,
algorithms to avoid them. Dreibholz et al. [49] proposed multiple mechanisms accord-
There has been a considerable amount of work on the ingly, including Buffer Splitting, Chunk Rescheduling, and
core CMT-SCTP [91] to overcome its defect and improve Smart Fast Retransmission. Buffer Splitting is used to avoid
performance. Liu et al. [120] found that all of the five retrans- one path occupying too much buffer space, which prevents
mission policies may cause throughput degradation due to other paths from sending out new chunks. Chunk Rescheduling
TABLE XI
K EY A LGORITHMS FOR TCP BASED C ONCURRENT M ULTIPATH T RANSMISSION (S ORTED ACCORDING TO T HEIR O RDER M ENTIONED IN TABLE XII)
copes with the problem of certain delayed or lost chunks it does not consider chunks being moved from another path
stalling the whole transmission. Smart Fast Retransmission in the decision about fast retransmissions on the new path.
deals with spurious fast retransmission bursts. For example, In [8], they presented an optimized buffer handling technique
TABLE XII
C ONCURRENT M ULTIPATH T RANSFER P ROTOCOLS BASED ON TCP
to further improve performance. Specifically, they proposed to The authors in LS-SCTP [5], [6] and cmpSCTP [118]
use the shared buffer space dynamically so that a faster path proposed separating the association flow control from per
can have the possibility to send more data by granting it more path congestion control (denoted as FCCS). The conges-
buffer space. In this paper, we use Buffer Splittingv2 to denote tion control is performed per path, whereas the flow control
this updated version of Buffer Splitting mechanism. Moreover, is performed per association. In order to achieve this goal,
in [50], they proposed using the Multi-streaming feature of LS-SCTP uses two different sequence numbers. The first one
SCTP to mitigate the HLB problem. Specifically, each mes- is the Association Sequence Number (ASN) that is used to
sage is assigned an identifier to indicate a stream. With this reorder the received data at the receiver association buffer.
identifier, the protocol only needs to restore the sequence of The second one is the Path Sequence Number (PSN) that is
messages belonging together. Hence, after a packet loss, only used for reliability and congestion control on each path. The
the messages of the affected streams have to be delayed to scheduling module of LS-SCTP uses the current congestion
restore the sequence. The other messages can be processed window (cwnd) of each path as an estimate of its current
immediately without delay. bandwidth-delay product. For example, it assigns data to the
paths according to the cwnd/RTT of each path. cmpSCTP dis-

tributes data over available paths based on real-time bandwidth
estimation of each path. Similar to LS-SCTP, cmpSCTP also
distributes the data on the available paths based on the esti-
mation of the available bandwidth of each path. Thus, they all
use a WRR style data scheduler. (a) (b)
Sarkar [172] proposed Concurrent Multipath TCP
Fig. 7. RP (a) a single path shares its resource fairly to competing flows (b)
(cmpTCP), an extension of SCTP. cmpTCP splits packets multiple paths are treated as a single pooled resource. RP: Resource Pooling.
concurrently over all available paths from a shared sending
buffer. cmpTCP maintains a virtual retransmission queue
(RTxQ) on each path to control the number of outstanding back to a positive state in the case of successful heartbeat
bytes on the path. The receiver sends back ACKs on the same acknowledgement.
path on which the packets are received. These two designs Mirani et al. [129] proposed a multipath Forward Prediction
may help to ignore spurious gap reports and eliminate unnec- Scheduling (FPS) for SCTP, namely FPS-SCTP. In order to
essary packet retransmissions. Sarkar [172] also developed reduce the number of out-of-order packets, it estimates the
a Markov model in cmpTCP to estimate the data transport arrival time of each packet in advance and decides which
rate on each path when the transmission has reached a steady packet is to be sent through a certain path so that the packets
state. cmpTCP uses a WRR style scheduler by considering can arrive at the destination in order. They used roughly a half
the number of outstanding bytes and congestion window size. of the RTT on a subflow to estimate the one trip time from
Huang and Tsai [85] proposed Wireless Multi-Path SCTP data leaving the send buffer to being received at the receiver.
(WiMP-SCTP), which devised two data transmission modes, This prediction is an approximation considered as too coarse
i.e., Data-striping Mode and Data-duplicating Mode, for multi- because previous studies have shown that a majority of Internet
path transmission in multiple wireless access networks. When connections experience latency asymmetry [141], [197].
the network status is good, the Data-striping Mode is selected b) Multipath transmission considering fairness: The
to aggregate bandwidth. On the other hand, when the network work discussed previously on CMT-SCTP performs indepen-
status becomes bad, the Data-duplicating Mode is selected dent congestion control on each path, and considers little about
to increase destination reachability. To switch between the the fairness against other single-path flows. For example, in
two modes, a mode selection scheme that determines the the case of n CMT-SCTP paths, the association will get n
status of these multiple paths was proposed. Specifically, it times the bandwidth share of a competing non-CMT SCTP or
designed a HEARTBEAT scheme where heartbeat chunks TCP flow over the same bottleneck.
are sent periodically on the paths. The transmission error The concept of RP [200] is a milestone for multipath trans-
counter increases when a heartbeat is not acknowledged mission aggressiveness control. In the context of multipath
within one retransmission timeout interval. If the number transmission, it makes a collection of resources behave like a
of transmission error counter plus the number of consecu- single resource by balancing traffic across multiple paths. As
tive retransmissions exceeds a certain threshold, the Data- shown in Figure 7, when several TCP flows compete for a sin-
duplicating Mode is switched on. Otherwise, the Data-striping gle path, TCP can share the paths capacity fairly among them.
mode is used. Nevertheless, Huang and Tsai [85] did not When the flows go through more than one path, the paths are
present a complete explanation of the scheduling algorithm treated as a single pooled resource. With appropriate coordi-
used by WiMP-SCTP. They only mentioned that the sender nation, traffic can move away from more congested paths to
transmits data as soon as the corresponding receive win- less congested ones, and larger bursts can be accommodated.
dow at the receiver side allows data to be sent. We spec- Dreibholz et al. [48] proposed a RP congestion control
ulate that this is a variant of the WRR style scheduling for CMT-SCTP denoted as CMT/RP-SCTP by combining
algorithm. CMT-SCTP with the concept of RP. The goal of CMT/RP-
Budzisz et al. [21] proposed an mSCTP-CMT protocol to SCTP is to improve the data throughput while still remaining
investigate the applicability of using CMT-SCTP to distribute fair to concurrent single-path flows on the shared bottle-
data between two paths during the handover transition pro- neck. For example, when two paths are used concurrently
cess. They emphasized the consequence of a sender-introduced for data transmission and they share a bottleneck link, the
reordering and its effect on congestion control. The authors overall throughput obtained by a CMT/RP-SCTP associa-
found that in CMT-SCTP the receive buffer may be filled with tion should be similar as that of a standard SCTP. However,
out-of-order data caused by complete or short-term failures CMT/RP-SCTP assumes similar paths, i.e., paths having very
during handover. Although handover is out of the scope of similar characteristics in terms of bandwidth, delay and loss
this paper, the handover scenario is very similar to the worst rate. Dreibholz et al. [47] proposed an updated version of
case in CMT where a path experiences a long delay suddenly. CMT/RP-SCTP (denoted as CMT/RPv2-SCTP) to overcome
To solve this problem, they proposed using Potentially Failed the limitations of CMT/RP-SCTP by considering path band-
(CMT-PF) scheme that a path experiencing a single timeout widths. They studied the behavior of CMT/RPv2-SCTP on
is marked as potentially failed and no further data trans- dissimilar paths and found that CMT/RPv2-SCTP achieves the
mission is allowed on that path. They utilized a heartbeat goals of RP. Furthermore, they observed that compared with
scheme to probe whether the potentially-failed path has got MPTCP (which will be discussed later), CMT/RPv2-SCTP
distributes bandwidth to flows equally when possible regard- data-striping across multiple micro-flows (TCP flows) by con-
less of the number of paths used for transport. sidering their bandwidth difference. Specifically, pTCP uses
Note that both CMT/RP-SCTP and CMT/RPv2-SCTP apply cwnd/RTT ratio, a WRR-PULL scheduler, to allocate traffic
RP principle only during the congestion control phase. proportionally to path capacity. In addition, pTCP has several
Shailendra et al. [175] argued that this strategy is not opti- other strategies addressing specific problems. For example, the
mum on heterogeneous networks with wireless links because congestion window could be an over-estimate especially just
losses in wireless links may happen because of reasons other before congestion occurs. This can result in an undesirable
than congestion. Hence they considered the resources to be as a hold up of data in subflows. Instead of reassigning data to other
single pool of resources during the congestion detection phase subflows later on, pTCP uses a Delayed Binding strategy to
as well. To achieve this goal, they proposed a Bandwidth adapt to instantaneous changes in path capacity. Specifically,
Estimation Based Resource Pooling (BERP) algorithm which it pulls data from the shared sending buffer only when the
applies the RP principle based upon the bandwidth estimates data is scheduled to send out immediately through a subflow.
obtained by observing the data flow on the paths. In order to avoid an overflow of the receive buffer, pTCP uses
2) TCP Based Multipath Transmission: In contrast to a Packet Re-striping strategy to retransmit a packet through a
SCTP, which was designed with multihoming support in different subflow instead of the subflow which transmitted that
nature, TCP is unaware of multiple interfaces and allows packet earlier, and uses a Redundant Striping strategy to send a
only a single IP address per endpoint. Nevertheless, TCP duplicated packet on one subflow to another. Moreover, pTCP
has dominated the Internet traffic and has sparked a lot of uses SACK feedback mechanism to recover a lost packet in a
interests in enabling TCP to support simultaneous multipath much shorter time period.
transmission. In this section, we divide the approaches in Reception Control Protocol (RCP) [81], [106] is a receiver-
four groups. The grouping principle is influenced by the centric transport protocol with a minimized sender design. The
research issues the approaches aim to address, such as fair- receiver controls all the key functions in RCP. To support
ness, buffer impact on performance, Pareto-optimality and CMT, a multi-state extension of RCP, i.e., Radial RCP (R2 CP),
path diversity. Although some approaches may cover more was proposed. R2 CP maintains one RCP pipe (the same as a
than one research issue, we only discuss them in the group TCP flow) per end-to-end path with congestion control being
which we believe the approaches are best fit into. In our handled by individual RCP pipes. Traffic is scheduled to each
study, we found that SCTP and MPTCP share many sim- RCP pipe based on the (estimated) time the requested segment
ilar issues and certain algorithms. At the end of this sec- will arrive through the concerned pipe (a variant of EDPF).
tion, we summarize their common features as well as their Note that each RCP pipe maintains a local sequence number
differences. space internally to facilitate loss detection and recovery. The
a) Multipath transmission without considering fairness: local sequence number can be converted to the global sequence
Magalhaes and Kravets [121] proposed Reliable Multiplexing number, and vice versa.
Transport Protocol (R-MTP), which is a rate-based reliable Chen et al. [28] proposed M-TCP that uses a Duplicate
transport protocol multiplexing data across multiple network Transmission mode for the lossy wireless environment with
interfaces (i.e., a WRR style scheduler). It relies on explicit high interference. In this transmission mode, multiple copies
bandwidth probing via the packet pair method [99] to esti- of the same packet are sent on different paths so that the
mate bandwidth in order to adjust the rate on the available chance that all copies are lost is much reduced. Unfortunately,
paths accordingly. For example, it measures packet inter- they only present sending-side modification without address-
arrival times and jitter to sense bandwidth scarcity. The ing duplicate ACKs due to multiple copies of the same
probing period should occur on a fine time-scale to reflect packet.
the fluctuation of the available bandwidth. Rojviboonchai and Hitoshi [167] proposed a multipath
Lee et al. [111] supported two transmission modes in their Transmission Control Protocol (M/TCP). M/TCP uses One-
work: FOSM and POSM. In POSM, they investigated multiple Way-Trip Time (OWTT) [169], a similar method as FPS, at
schemes to address the spurious retransmissions by modify- the sender to estimate the delay time of the forward path and
ing two TCP operations: 1) Increasing the Fast Retransmit reverse path separately in order to calculate per path RTO
Threshold (IFRT) and 2) enabling Delayed ACKs for out- timer. In addition, M/TCP employs two mechanisms to deal
of-order packets as well as sending immediately ACKs for with packet loss. In the case of fast retransmission, M/TCP
retransmitted packets. The second modified operation is like uses Duplicate Transmission policy, i.e., duplicating the miss-
an advanced version of DAC used in [89] and [91]. Thus, we ing segment and sending each copy through all paths so that
name it DACv2 in this paper. IFRT makes the TCP sender a quick and reliable retransmission can be desirable; in the
wait for more than triple duplicate ACKs, which reduces the case of timeout retransmission, the missing segment on a flow
number of the fast retransmission and the fast recovery events. is sent through the other flows. Moreover, M/TCP uses two
DACv2 enhances performance because when ACKs are being algorithms for the receiver to transmit an ACK in the case of
delayed, new packets may fill the gap and change the out of CMT. Namely, using Duplicated ACK algorithm, an M/TCP
order packets in order. receiver sends an ACK immediately upon the receipt of a data
Parallel TCP (pTCP) [82][84] functions as a wrap- segment to more than one path; using Duplicated & Delayed
per around a modified version of TCP. It opens multiple ACK, the receiver transmits an ACK for every other data seg-
TCP flows, one for each interface in use. pTCP performs ments through more than one path. Rojviboonchai et al. [168]
proposed R-M/TCP as an extension of their previous work According to the feedback, the sender adjusts traffic allocation
M/TCP. R-M/TCP is a rate-based M/TCP that performs con- dynamically according to the estimated transmission rates and
gestion control in a rate-based and loss avoidance manner delays.
(we use RCC to denote it) to avoid packet loss by adjusting Tsao and Sivakumar [191] argued that aggregated band-
the congestion window before buffer overflows. Specifically, width of two wireless interfaces (3G and WiFi) is a Simple
R-M/TCP schedules data packets in a WRR manner while it Aggregation due to path heterogeneity. For example, the low
estimates the queue length at the bottleneck link. If the queue bandwidth interface (e.g., 3G with 100-500Kbps) can only
length grows beyond a predefined threshold, the sender recal- achieve negligible bandwidth compared to the high band-
culates a new congestion window to achieve a fair share at the width interface (e.g., WiFi with 2-54Mbps). They proposed
bottleneck. a super-aggregation to achieve performance that is better
Cetinkaya and Knightly [25] proposed an Opportunistic than the sum of throughput achievable through each of the
Multipath Scheduler (OMS) that follows a traffic splitting interfaces individually by the means of three mechanisms:
policy that favors low-delay high-throughput paths opportunis- Selective Offloading, Proxying, and Mirroring. In spite of the
tically for a short term variations in path quality. To avoid fact that an interface may have a relatively small amount of
violating path weights of the routing protocol and potentially bandwidth, these mechanisms can provide considerable per-
leading to oscillation, OMS ensures that over longer time formance improvement. Specifically, the Selective Offloading
scales traffic is split according to the ratios determined by mechanism is to receive TCP data segments over a compa-
the routing protocol. rably high-speed WiFi and return ACKs over a low-speed
Dong et al. [46] proposed concurrent TCP (cTCP) that uses 3G path to address self-contention in WiFi networks. The
a single congestion window to control the global throughput Proxying mechanism, following IP-in-IP encapsulation, allows
and a single sending buffer to be shared among all paths. a 3G path to notify the TCP sender about blackout events
It uses Credit-Weighted Round-Robin (a variant of WRR) as on the WiFi path. The Mirroring mechanism (i.e., Duplicate
the scheduling algorithm. Each time an ACK comes back to Transmission) establishes an addition TCP connection through
the sender, the capacity estimation of that path is updated, 3G to fetch the missing segments due to random loss on the
and a new sending credit (similar to the congestion window WiFi path.
size) is added to the sender. The new credit is for all the b) Multipath transmission considering fairness: Similar
paths combined, and it is further divided into each path. cTCP to the early development of SCTP based CMT, at the early
uses the path credit (similar to the path capacity) to distribute stage, TCP based multipath transmission was only used for
data among the available paths. Furthermore, cTCP adopts a utilizing multiple TCP flows with intelligent scheduling algo-
duplicated ACK classifier that handles packet reordering by rithms to mitigate packet reordering. Nevertheless, it was
differentiating whether a duplicated ACK is likely caused by found that simply utilizing multiple TCP flows concurrently
CMT or a real duplicated ACK. at a bottleneck would result in a fairness issue, i.e., an unfair
Sharma et al. [178], [179] proposed Multi-Path LOss- share of the bandwidth at a bottleneck link. For example,
Tolerant protocol (MPLOT) to provide multipath transmission NewReno [77] is the most common TCP congestion control
on multiple heterogeneous, highly lossy paths. MPLOT uses variant as it yields an equal share of the congested link. This
erasure based Forward Error Correction (FEC) packet coding. equal share outcome of NewReno results in an unfair share of
The major benefit of packet coding stems from its ability to the bandwidth if more than one TCP flow is active for a sin-
compensate for missing packets from redundancy. This makes gle multipath transmission connection at the bottleneck link.
data transmission over lossy networks robust and efficient. From the literature review we have made, we find that mul-
To counter against packet reordering, MPLOT estimates path tipath transmission approaches based on TCP in recent years
parameters (i.e., loss rate, capacity, and RTT) continuously to have also started to make fairness a necessary feature.
provide adaptive FEC coding. In particular, MPLOT performs To the best of our knowledge, mTCP proposed by
latency-aware packet mapping, a variant of EDPF. For exam- Zhang et al. [204] is among the earliest proposals that
ple, it maps packets that are not required immediately to paths have taken fairness issue into consideration for TCP based
with long delays, while mapping the more immediately use- multipath transmission. To address the fairness issue, they
ful packets to paths with short delays. MPLOT uses Explicit proposed not establishing multiple flows through the same bot-
Congestion Notification (ECN) [164] to distinguish congestion tleneck. For example, to alleviate the aggressiveness problem,
losses from those due to faulty/lossy links. Zhang et al. [204] integrated a shared congestion detec-
Wang et al. [198] proposed a segment-based adaptive Joint tion mechanism into mTCP so as to identify and suppress
Session Scheduling (JOSCH) mechanism. The main goal is subflows that traverse the same set of congested links. For
to restrain the delay difference among multiple Radio Access example, mTCP detects shared congestion by examining the
Networks (RANs) by means of allocating the traffic to dif- correlations among the fast retransmit times of the subflows.
ferent RANs dynamically with reasonable ratios. Specifically, mTCP also uses a scheduler in a WRR manner. For exam-
JOSCH obtains network conditions by a segment-based feed- ple, it maintains a counter, i.e., pipei , to represent the number
back approach, where a segment is defined as a predeter- of outstanding packets on the ith path. pipei is incremented
mined size of data block. Its size is configurable according by 1 when the sender either sends or retransmits a packet
to the delay sensitivities of different services. After each seg- over the ith path, and is decremented by 1 when an incoming
ment transmission, the receiver sends feedback to the sender. ACK indicates that a packet previously sent has been received.
Signal (DSS) option [62] specifies a full mapping from the

connection-level sequence number to the subflow sequence
number. In the early stage of MPTCP, a PUSH style WRR
scheduling strategy is used where the scheduler tries to fill
all subflows when there is data coming from the application.
Later, MPTCP adopts a WRR-PULL manner scheduler [17], a
similar design adopted by pTCP [82][84] where the applica-
tion data stored in a shared connection-level sending buffer
is pulled by subflows whenever they have space in their
congestion window. Both PUSH and WRR-PULL scheduling
strategies are variants of WRR. Their difference lies in the fact
that WRR-PULL strategy uses less time waiting in a subflow
Fig. 8. The architecture of MPTCP, which has the application compatibility
queue before its actual transmission on the wire. Although this
by keeping the standard socket API to legacy applications. MPTCP: Multipath time period seems minor, the path properties may change dur-
TCP, src: Source IP address, dst: Destination IP address, sp: Source Port, dp: ing that time. For the latest development of MPTCP in IETF,
Destination Port, APP: Application.
we refer readers to RFCs such as [61], [61], and [159].
Raiciu et al. [161] were the first proposing a natural evo-
lution of data center transport from TCP to MPTCP. They
The sender associates a score, i.e., pipei /cwndi , for each path. demonstrated that MPTCP could efficiently and seamlessly use
The path with the minimum score has priority to send the next available bandwidth, provide improved throughput and better
packet. fairness compared to single path TCP. The same authors fur-
Instead of avoiding shared bottlenecks, Honda et al. [79] ther investigated what caused these benefits in [157]. They
proposed Bidimensional-Probe Multipath Congestion Control found that using MPTCP allows to rethink data center net-
(BMC) to address the fairness issue. Specifically, BMC uses works and approach them with a different mindset as to the
a Weighted Congestion Control (WCC) approach that applies relationship between transport protocols, routing, and topol-
the weight to each subflow so that the throughput of each ogy. One of the challenges of deploying MPTCP in data cen-
subflow is in proportion to its weight. In addition, WCC main- ters is the Incast effect, a behavior of MPTCP as well as TCP
tains the sum of the weight so that a bundle of subflows in that results in the gross under-utilization of link capacity in
the multipath connection is kept as aggressive as one TCP certain many-to-one traffic patterns [40], [132], [147], [190].
flow. WCC can achieve not only fair resource allocation at Incast collapse is not specific to MPTCP, but is inherited
the shared bottleneck, but also RP [200] along the disjoint from TCP. Li et al. [117] investigated how to share network
bottlenecks. For example, multiple different connections can resources among different MPTCP flows by performing an
obtain fair resource allocation across distinct bottlenecks. additional weighted congestion control based on the coupled
Approximately in 2009, MPTCP [60], [158], [162] was pro- congestion control mechanism.
posed with the fairness property as well as RP feature in Although LIA ensures MPTCP subflows to be no more
mind. Specifically, MPTCP, under discussion of IETF, has the aggressive than a competing TCP flow, LIA is not able to
following set of goals to achieve: differentiate whether the subflows share the same bottleneck
Improve throughput: MPTCP should perform at least as or different ones. This may cause sub-optimal performance of
a single TCP flow running on the best path. MPTCP. Hassayoun et al. [76] proposed a Dynamic Window
Do no harm: MPTCP subflows should not take more Coupling (DWC) algorithm to address it by detecting dis-
capacity than a single TCP flow would get at a shared tinct bottlenecks and only coupling those that share a common
bottleneck. bottleneck, while allowing other subflows to use separate con-
Balance congestion: MPTCP should utilize the least gestion control. For example, DWC uses a loss congestion
congested path the most. event to trigger an alert while using either the Delay or Loss
Figure 8 shows the architecture of MPTCP. It is a major congestion event to group subflows. Singh et al. [182] pro-
extension to TCP and allows a pair of hosts to use several posed an extension of the DWC algorithm, denoted as EDWC.
paths to exchange the segments that carry the data from a sin- This extension uses a delay congestion to trigger an alert while
gle connection. MPTCP presents a standard TCP socket API either delay or loss congestion is used to group and couple
to the upper layer so that legacy applications can run upon subflows. The concept behind DWC and EDWC algorithms is
MPTCP transparently. A Coupled Congestion Control (CCC) similar to that of UCCSB algorithm used in BA-SCTP [13]
algorithm, Linked Increase Algorithm (LIA) [159], is used to because they both identify the shared bottleneck and guarantee
guarantee fair resource allocation on multiple paths and pro- that the aggregated flows on the bottleneck is TCP-friendly.
vide RP feature among them. Its RP feature can shift traffic Diop et al. [44] proposed QoS-oriented MPTCP that takes
away from more congested paths to less congested ones. In advantage of the two sub-layers architecture of MPTCP to
addition to the joint congestion control algorithm, MPTCP also use Quality of Service (QoS) techniques for multimedia appli-
has a few other design features. For example, MPTCP adds cations over multiple paths. MPTCP originally offers a fully
connection-level sequence numbers in order to reassemble the reliable and fully ordered service. Nevertheless, full reliabil-
data stream in-order from multiple subflows. A Data Sequence ity may not be required by certain multimedia applications.
Diop et al. [44] investigated the QoS benefits induced by coding to some but not all subflows. The regular subflows
the implementation of the Partial Reliability [12] feature in deliver original packets while the coding subflows deliver
MPTCP for interactive video applications. Partial Reliability linear coded packets. The coded packets are used to compen-
is an important concept for multimedia transmission over IP sate for the lost and much delayed packets in order to avoid
networks and is defined as the possibility to not recover losses receive buffer blocking. They used an out-of-order scheduler
under a threshold in order to improve QoS. that calculates the expected packet arriving time by taking
Peng et al. [142], [143] presented a fluid model to investi- RTT, throughput, and loss ratio into account. Thus, the pack-
gate a few existing congestion control algorithms designed for ets that are sent out of order are expected to arrive at the
MPTCP, and identified design criteria that guarantees the exis- receiver in order. This scheduling algorithm is the same as
tence, uniqueness, and stability of system equilibrium. They the FPS algorithm used in [129]. Li et al. [115] proposed
characterized algorithm parameters for TCP friendliness and SC-MPTCP to mitigate the packet reordering issue with con-
proved that there is an inevitable trade-off between respon- strained receive buffer. In SC-MPTCP, they proposed to make
siveness and friendliness. Based on the study, in [143] they use of coded packets only as redundancy to compensate
proposed a new congestion control algorithm, Balia (balanced for expensive retransmissions while minimizing the encod-
linked adaptation). This algorithm generalizes existing algo- ing/decoding operations. The redundancy is provisioned in
rithms and strikes a good balance among TCP-friendliness, both proactive and reactive manners. Specifically, SC-MPTCP
responsiveness, and window oscillation. transmits proactive redundancy first and then delivers the origi-
Lim et al. [119] proposed MPTCP-MA to improve MPTCP nal packets. The proactive redundancy is continuously updated
performance during intermittent path connectivity in wireless according to the estimated aggregate retransmission ratio. In
environment. MPTCP-MA exploits MAC-Layer information order to avoid the proactive redundancy being underestimated,
to estimate path status, and suspends/releases a path based on a pre-blocking warning mechanism is utilized to retrieve the
the estimation. By quickly detecting path failure and recov- reactive redundancy from the sender. Cui et al. [37], [38] pro-
ery, MPTCP-MA can avoid unnecessary losses and utilize posed applying the fountain code for multipath scheduling to
recovered paths more quickly. mitigate the impact of path heterogeneity. They also designed
c) Buffer impact on MPTCP: Although MPTCP was a data allocation algorithm based on the expected packet arriv-
designed with several merits such as fairness, RP, and Pareto- ing time and decoding demand to coordinate the transmissions
optimality in mind, buffer size has significant impact on of different subflows. Li et al. [116] dealt with the packet
MPTCP performance. This problem stems from the packet reordering issue from a different perspective. They demon-
reordering issue due to heterogeneous path characteristics. We strated that the traditional Delayed ACK mechanism can lead
now discuss the related work which has explicitly examined to significant performance degradation in the presence of time-
the impact of buffer size on MPTCP. Note that the impact of outs. Thus, they proposed a New Delayed ACK (NDA) for
buffer size is not limited to MPTCP but on all other approaches MPTCP aiming to remove the Minimum RTO constraint at
using POSM. the sender while to reserve the Delayed ACK function at the
Barr et al. [17] evaluated the impact of heterogeneous receiver.
paths on the receive buffer and aggregated throughput. The Raiciu et al. [160] proposed schemes of opportunistic
experiment result shows that losses on one subflow have a retransmission and penalizing slow subflows to avoid the
limited impact on the performance of the other subflows. reordering problem. If a subflow holds up a packet at the
Nevertheless, this observation is based on the assumption trailing edge of the receive window, the opportunistic retrans-
that the reordering buffer is big enough to accommodate all mission allows the sender to resend the packet that is pre-
the out-of-order data. Nguyen et al. [133], [134] evaluated viously sent on another subflow. This scheme is similar to
the performance of MPTCP in terms of load sharing and the Packet Re-striping used in [81][84] and [106] and Pre-
throughput optimization with and without LIA respectively. blocking warning used in [115]. These three similar schemes
The results show that the context of mismatched path char- are used in different scenarios but for the same purpose.
acteristics has a great impact on the performance of MPTCP Opportunistic retransmission is used only if the connection is
with constrained receive buffers. Kim et al. [103] proposed receive-window limited. Packet Re-striping is employed in the
a reordering scheme that considers packet scheduling algo- case of path capacity fluctuations. Pre-blocking warning is
rithm at the sender to reduce the receive buffer. The main triggered if the proactive redundancy is underestimated. The
idea is to estimate packet arrival time and schedule pack- penalizing scheme of [160] is used to slow down the slow
ets accordingly. The sender maintains a per path time table subflows. For example, if a subflow has caused too many
including calculated values of receiving time at the receiver out-of-order packets in the reordering buffer, the congestion
from now for each packet. When the sender has opportunity window of that subflow is reduced by half and its slow-
to send a new packet, it chooses the path that can deliver the start threshold is set to the current congestion window. But
packet faster than others. Thus, it is a EDPF style scheduling if that subflow has been in the slow-start phase already, the
algorithm. reordering problem may become worse because the penaliza-
The impact of buffer size on MPTCP performance has also tion mechanism will set its slow-start threshold to a smaller
been observed in [37], [38], and [114][116], which proposed value. Paasch et al. [140] proposed improving the penalization
packet coding based approaches to address it. For example, mechanism by adjusting the slow-start threshold only when
Li et al. [114] proposed NC-MPTCP that introduces packet a subflow is not in its slow-start phase. However, they also
identified that the penalization mechanism is far from perfect idea is that the sender distributes packets via multiple paths
because a subflow at full sending speed may still overflow the according to their estimated forward delay and throughput dif-
receive buffer while another subflows is in slow-start. ference. This scheduling algorithm is an advanced version of
Ferlin et al. [58] argued that the opportunistic retrans- FPS because it took throughput difference into consideration.
mission scheme does not reduce the effect of extreme RTT Due to the latency asymmetry on the Internet, obtaining a
heterogeneity. Instead, they proposed a Dynamic Relative Path good one-way delay is difficult. Therefore, Coudron et al. [35]
Scoring (DRePaS) algorithm to dynamically score the paths proposed relying to the one-way delay (OWD) difference
relative to the best path and adapt the scheduling accordingly. (OWD) of multiple subflows to improve the scheduling.
Specifically, when the score of a path is less than a threshold, Their estimator functions more like OWTT than FPS because it
no payload is scheduled over that path until its score is larger estimates not only forward delay difference but also backward
than the threshold measured by probing traffic. Note that the delay difference.
standard MPTCP uses the congestion window as an estimation The MPTCP performance is not only impacted by the receive
of path capacity. In contrast, Ferlin et al. [58] believed that the buffer but also by the send buffer. Yang and Amer [201] found
smoothed in-flight data on each path reflects the behavior of that in an MPTCP connection with several high-BDP subflows,
the connection more dynamically than the congestion window. send buffer blocking can occur and seriously decrease the
Chen et al. [30] explored the performance of MPTCP over overall throughput. They introduced Non-Renegable Selective
wireless networks. In order to avoid performance degradation, Acknowledgments (NR-SACKs) to MPTCP. The idea is that
they set the receive buffer up to 8 MB, which is not feasible once a data packet has been sacked, it cant be removed
in practice for many devices. Shamszaman et al. [177] ana- from the receive buffer. Thus, the sender can free the sacked
lyzed the feasibility of MPTCP for big data applications. They data sooner than the advance of the MPTCP level cumulative
found that constrained receive buffer leads to poor perfor- acknowledgement. Arzani et al. [14] found that the send buffer
mance of MPTCP. Zhou et al. [206] proposed CWA-MPTCP size has significant impact on the performance of MPTCP. For
that examines the goodput of MPTCP with bounded receive example, MPTCP provides higher performance gains with a
buffers. They found that if the paths have similar end-to-end larger send buffer. However, they did not propose any method
delays, the MPTCP goodput is near optimal, otherwise the to address the problem.
goodput will be degraded significantly. For a wireless environ- d) Pareto-efficient MPTCP: Khalili et al. [101], [102]
ment, they proposed a Congestion Window Adaptation (CWA) demonstrated that MPTCP is not Pareto-optimal because they
algorithm that can adjust the congestion window dynamically found that MPTCP users can be excessively aggressive toward
for each TCP subflow so as to mitigate the variation of end- TCP users over congested paths even without any benefit to
to-end path delay, maintaining similar end-to-end delays over the MPTCP users. They attributed the problem to the LIA
multiple paths. The primary idea behind CWA is that a large of MPTCP. To deal with the problem, they proposed an
delay ratio indicates that the high-delay path is overloaded. Opportunistic Linked Increases Algorithm (OLIA) as an alter-
Its congestion window needs to be decreased to relieve traf- native for LIA and proved that OLIA is Pareto-optimal and
fic and reduce path delay. For wired environment with stable satisfies the three design goals of MPTCP. Like LIA, OLIA
end-to-end delay they proposed using a delay-aware schedul- is a window-based congestion-control algorithm that couples
ing algorithm to predict the receiving sequence, i.e., a FPS the increase of congestion windows and uses unmodified TCP
manner scheduler, so that packets can arrive at the receiver in behavior in the case of loss. The increase part of OLIA has two
order. terms. The first term provides the Pareto optimality. The sec-
Paasch et al. [139] designed and implemented a generic ond term guarantees non-flappiness6 as MPTCP with LIA and
modular scheduler framework that enables testing of differ- responsiveness (i.e., the rate of algorithm convergence). OLIA
ent schedulers for Multipath TCP. Using this framework, they also compensates for different RTTs by adapting the window
evaluated different schedulers for MPTCP and provided an increases as a function of RTTs. However, Singh et al. [182]
in-depth performance analysis. They identified the impact of found that OLIA of MPTCP still has performance issues. They
scheduling decisions on the performance of MPTCP and illus- presented Adapted Opportunistic Linked Increases Algorithm
trated the underlying root cause for the observed behavior. (AOLIA) to ensure controlled aggressiveness of the MPTCP
For example, they discovered that a bad scheduling deci- subflows. In order to minimize the packet reordering delay,
sion triggers two packet reordering effects. First, EDPF based they also proposed a Push-Pull-Hybrid (PSPLH) scheduler
scheduler is more efficient than simple RR in terms of avoid- where Pull strategy is used to allocate data segments to mul-
ing the HLB problem. Second, receive-window limitation may tiple flows, and Push strategy is used to tune the size of the
prevent the subflows from being fully utilized. segments dynamically.
In the last few years, several articles visited how to use e) Path diversity for MPTCP: By default, MPTCP uses a
delay-aware scheduling algorithms in MPTCP to improve the fullmesh manager to create static full-mesh of possible net-
receive buffer utilization. Yang and Amer [202] used one- work paths among the available IP addresses (see Figure 9).
way communication delay of a TCP connection to design an This path management may not only lead to a large number
MPTCP scheduler that transmits data out-of-order over multi- of subflows being established but also ignore the benefits the
ple paths such that their arrival is in-order. Le and Bui [110]
dealt with the packet reordering problem of MPTCP using 6 Flappiness means that MPTCP would use one path almost exclusively for
a Forward-Delay-based packet scheduling algorithm. Its main a while, then flip to another path, and then repeat.
Therefore, algorithm is not the primary factor that determines

which protocol could become the mainstream because the
algorithms are not specific to any base protocols but could
be used on any of them with minor adaptation. We believe
that the reason mainly comes from their difference in terms
of backward compatibility and sequence number design. For
example, unlike SCTP which modifies the interfaces of legacy
TCP to applications, MPTCP presents a single TCP interface
Fig. 9. Static full-mesh of possible network paths between two MPTCP to the application layer (see Figure 8). This seemingly minor
enabled hosts: A1 :B1 , A1 :B2 , A2 :B1 , A2 :B2 , A3 :B1 , A3 :B2 . difference makes MPTCP compatible with all legacy appli-
cations. As such, the implementation of multipath in TCP,
which dominates Internet traffic, is a much more attractive
path diversity could offer. van der Pol et al. [194] combined deployment strategy.
MPTCP and OpenFlow to dynamically exploit path diversity The sequence space design is another primary difference. To
(choose disjoint paths) between two endpoints to improve sta- make a fair comparison, we assume that SCTP has been widely
bility (in the case of partial path failure) and obtain higher deployed as TCP has, otherwise CMT-SCTP packets would be
throughput. Specifically, in [194] OpenFlow is used to dis- dropped by legacy middleboxes. As summarized in Table X,
cover the topology of the network, calculate multiple disjoint CMT-SCTP keeps using one single sequence space as SCTP
paths and configure these paths. MPTCP is used to distribute does. Therefore, a new SACK mechanism and its interpretation
the traffic across the selected paths. accordingly are required to avoid spurious retransmissions.
In contrast to limiting the number of subflows [194], Moreover, single sequence space makes bytestream discon-
Coudron et al. [34], [36] exploited the path diversity in cloud tinuous on multiple paths so that certain middleboxes may
networks from a different perspective, for example, creating break such associations [160]. In contrast, in MPTCP the
more subflows over disjoint WAN paths (under the assump- sequence numbers carried in the TCP headers are separate
tion that the WAN capacity is the throughput bottleneck). on each path so that the interpretation of out-of-order packets
To achieve this goal, Coudron et al. [36] proposed a cross- and ACKs remain the same as before. Hesmans et al. [78] and
layer approach which combines MPTCP and LISP. LISP is Honda et al. [80] found that MPTCP could traverse most of
used to gather information and give hints to MPTCP about the middleboxes because of its double sequence space design.
the WAN paths diversity between two MPTCP endpoints. Furthermore, with checksums MPTCP can detect middleboxes
MPTCP could use the path manager ndiffports to create interference and fallback to legacy TCP. To allow clients to
additional LISP-enabled subflows which share the same source benefit from MPTCP in its early deployment (e.g., servers have
IP with different TCP ports. In their another work [34], apart not upgraded to support MPTCP), Detal et al. [41] proposed a
from MPTCP and LISP, Coudron et al. [34], [36] introduced protocol converter, MIMBox, to translate MPTCP to TCP and
one more protocol TRILL to support Ethernet level multipath vice versa. MPTCP is not only little influenced by middleboxes
transmission. For example, MPTCP employs coordinated con- but is even extended to explicitly add specified middleboxes
gestion control to achieve RP, LISP handles path diversity at in the middle of an ongoing communication. For example,
the external data center packets between border nodes, and the proposed solution in [136] used MPTCP to implement
TRILL deals with multipath Ethernet-layer communications. connection acrobatics (i.e., the ability to explicitly redirect
f) Comparison between CMT-SCTP and MPTCP: SCTP connections via a middle point and the ability to migrate the
and TCP are the main base protocols widely used to support endpoint of a connection). Therefore, MPTCP connections can
multipath transportation. Currently, CMT-SCTP and MPTCP be redirected to middleboxes located anywhere in the Internet
are in the focus of the IETF and academia. In the following to improve services like load balancing, DDoS filtering and
discussion, we make a comparison of them. anycast.
Although CMT enabled SCTP shares the same issues with One more difference between CMT-SCTP and MPTCP lies
MPTCP in terms of fairness, reordering, and retransmis- in the path management strategy. By default, MPTCP creates
sion policies, moving legacy applications from TCP to SCTP a full-mesh of possible network paths among the available IP
involves a number of challenges such as making SCTP work addresses, whereas CMT-SCTP only uses pairs of addresses
through NATs, the need to modify applications, and the lack to set up communication paths (creating only one additional
of an easy way to negotiate SCTP versus TCP between a path per additional source address). Becke et al. [18] found
client and a server. None of the issues are insurmountable, that MPTCPs path management strategy performs signifi-
but together they make adoption of SCTP as a TCP alterna- cantly better than that of CMT-SCTP in the case of asymmetric
tive a challenge. From the previous discussion, we found that paths.
MPTCP instead of CMT-SCTP has become the main stream
of multipath transportation solution. In this section, we discuss
the reason behind it. D. Application Layer Multipath Capability
By comparing the key algorithms used by CMT-SCTP and Provisioning multipath capability at the application layer
MPTCP (see Table X and Table XII), we found that the two has received a lot of attention because the approaches are
protocols have shared the same or similar key algorithms. almost independent of the underlying access technologies and
TABLE XIII
K EY A LGORITHMS FOR A PPLICATION L AYER M ULTIPATH C APABILITY (S ORTED ACCORDING TO T HEIR O RDER M ENTIONED IN TABLE XIV)
TABLE XIV
C ONCURRENT M ULTIPATH T RANSFER A PPLICATIONS
In the rest of this section, we also divide the approaches

into four groups. The first group discusses approaches using
the same path. The second group discusses approaches using
different paths. The third group investigates approaches based
on HTTP in order to highlight the importance of the combi-
nation of HTTP and multipath transmission. The fourth group
exploits middleware approaches. Table XIII and Table XIV
are used to summarize the key algorithms and approaches
respectively.
1) Multiple Connections Over the Same Path: In the early
stage of research work on application layer multipath transmis-
sion, the focus was on bandwidth aggregation using multiple
TCP connections over the same physical path. For instance,
Fig. 10. Multipath transmission on application layer. It is assumed that each Allman et al. [10] developed a new application called XFTP,
host has two interfaces. src: Source IP address, dst: Destination IP address,
sp: Source Port, dp: Destination Port, APP: Application. a modified version of FTP [152], to improve the poor perfor-
mance of TCP over long-fat satellite channels. XFTP creates
multiple TCP connections. To send a file, XFTP divides the
file into records, reads the file from local storage one record
network-layer routing. It is a common practice that an appli- at a time, and sends each record over whichever connec-
cation establishes multiple transport connections, binds them tion has resource available for transmission. To reassemble
to different IP addresses, and distributes the data in propor- the records into the correct order, XFTP uses an additional 4-
tion to the available path capacity over these connections byte sequence number to each record. GridFTP [92] is another
(see Figure 10). To reassemble the data delivered over differ- extension of the FTP protocol implemented for bulk data trans-
ent connections, each packet or a group of packets are usually fer, where parallel TCP connections are created to increase the
assigned additional sequence numbers. throughput in a bottleneck link. Specifically, GridFTP divides
the data to be transferred into multiple portions and transfers only marginal improvement is gained by further increase in
each portion with a separate TCP connection. When competing the number of paths.
with non-GridFTP connections over a bottleneck link, the Wang et al. [195], [196] proposed Dynamic MPath-
GridFTP connections will be less likely to be selected to Streaming (DMP), a scheme for live streaming over multiple
drop their packets. Hacker et al. [72] examined the effects TCP connections. DMP allocates packets over multiple paths
of using parallel TCP flows to improve end-to-end network according to their current throughput. DMP does not use an
performance. They found that in the absence of congestion, explicit probing scheme for bandwidth estimation on each
the use of parallel TCP connections is equivalent to using path. Instead, it uses the WRR-PULL scheduler to allow each
a large Maximum Segment Size (MSS) on a single con- connection to pull data from a shared queue whenever it
nection. In addition, they addressed the question of how has opportunity to send data. Thus, the paths with higher
to select the maximum number of connections to maximize throughput deliver more packets than others.
the overall throughput while avoiding congestion. For exam- Tullimas et al. [192] proposed MultiTCP for multimedia
ple, if the selected value is too large, the aggregate flow streaming. It aims at providing resilience against short term
may cause network congestion and throughput will not be insufficient bandwidth due to traffic bursts by using multi-
optimized. ple TCP connections for the same application. MultiTCP is
2) Multiple Connections Over Different Paths: The a smart application that allows the application to control
approaches mentioned above aim to increase application the desired sending rate during congestion periods. MultiTCP
throughput by using multiple TCP connections through the achieves rate control by the means of adjusting the receiver
same physical path. Nevertheless, if they are used for strip- window.
ing data over different physical paths, the reordering issue Zhang et al. [205] proposed a general framework of
at the receiver would render them inefficient because they multipath transport system based on application-level relay
do not take into consideration the reordering issue caused (MPTS-AR), currently under the standardization within the
by heterogeneous paths. In the 2000s, researchers started to IETF [112]. This framework defines three logical entities and
seek solutions to provide bandwidth aggregation over differ- two protocols. The entities include user agent, relay server,
ent physical paths. A simple approach to achieve this goal and relay controller. The protocols are OpenPath and MPTP
is to directly add support for multiple interfaces to a given (Multipath Transport Protocol), which are used in control
application by opening multiple TCP sockets (one for each plane to manage relay paths and in data plane to facilitate mul-
active interface), and performing striping of data across dif- tipath data transport respectively. However, they left a few key
ferent sockets. If the interfaces of a client are connected to functions we concern the most out of the scope. For example,
independent networks, the simultaneous use of multiple paths how to split the original data stream into several substreams,
can achieve a total throughput close to the sum of all the how to mitigate the reordering issue at the receiving side, how
throughput from individual interfaces. to provide path diversity among all available paths, and how
Given that popular files are often replicated on multiple to obtain the performance of overlay paths are all out of the
servers, it becomes natural for clients to connect in parallel to scope.
several mirror servers to retrieve a file (i.e., many-to-one fash- 3) HTTP Based Multipath Media Streaming: In addition
ion). Golubchik et al. [64] investigated the potential benefits to multipath approaches for specific applications, HTTP [59]
of an application layer multipath streaming approach between with multipath capability is currently one of the most common
a set of senders and a receiver. They found that multipath protocols for streaming video on the Internet through multiple
streaming exhibits better loss characteristics than single path paths. Kaspar et al. [96], [97] and Evensen et al. [53][55]
streaming. Rodriguez and Biersack [165] described a parallel- presented HTTP-based methods for downloading mul-
access (PA) scheme that allows users to fetch different portions timedia content simultaneously over multiple network
of a file from multiple servers at the same time and reassem- interfaces.
ble the file locally. The PA scheme allows dynamic load Kaspar et al. [97] proposed an HTTP-based on-demand
sharing among all servers so that faster servers will deliver streaming service over multiple wireless access networks.
bigger portions of a file while slower ones will deliver smaller They presented a proof-of-concept implementation of a pro-
portions. gressive download service, which uses HTTP Range Retrieval
Shiwen et al. [123] proposed Multiflow Real-time Transport Request (HTTP-RRR) capability [59] to download specific
Protocol (MRTP) which is a multipath transmission exten- segments of a file from a media server. The drawback of
sion to Realtime Transport Protocol (RTP) [173] for real-time the range retrieval request is that each request must be han-
applications. MRTP supports multimedia services by explor- dled sequentially by the server before the client can send the
ing multipath transport in mobile ad hoc networks, where link next request. Thus, an average time overhead of one RTT
bandwidth may fluctuate and paths are unreliable. The authors is introduced for each request. In order to avoid waiting for
studied the impact of traffic partitioning on congestion at bot- each response, in [96] they presented an improved version of
tleneck links and found that the bandwidth utilization of a their work by using an additional HTTP capability, i.e., HTTP
bottleneck node could be much improved by two strategies Request Pipelining (HTTP-RP) [59]. The request pipelining
(thinning and striping [22]). Furthermore, they showed ana- function of HTTP allows a client to make multiple requests
lytically that most of the performance improvement can be simultaneously. In [97], they studied the effect of file seg-
achieved with a few paths (e.g., two or three paths), while mentation on the buffer requirements and found that there
exists an optimal segment size for which the aggregation effi- Sakakibara et al. [171] proposed a Socket-level Bandwidth
ciency is maximized. In [96], they found that due to the use of Aggregation Mechanism (SBAM) to offer aggregated band-
request pipelining, very small segments can provide efficient width. SBAM is located at the socket layer (close to TCP)
throughput aggregation. so that it can collect system resources efficiently. For exam-
Evensen et al. [55] introduced an adaptive, WRR-PULL ple, it has a network monitoring function to collect delay
based scheduler that achieves smooth playback by schedul- and available bandwidth of each path. Using this informa-
ing requests for video segments of different quality lev- tion, the traffic scheduler decides the amount of data to fill
els over multiple interfaces simultaneously. Like their pre- the bandwidth-delay product of each path.
vious work in [97], they still utilized HTTP-RRR and Like the other middleware approaches, Parallel TCP
HTTP-RP functions to support multipath transmission. In Transfers Helper (PATTHEL) [16] also provides APIs for
order to avoid video deadline misses, they proposed an addi- applications. The difference of PATTHEL lies in two facts.
tional request scheduler in [55]. The scheduler is mainly First, PATTHEL incorporates a separate data connection and
used for estimation of the aggregated throughput for cho- control connection, where the control connection is created
sen video quality level and request of segments over the first to manage the other data connections for the entire com-
available interfaces. However, the weakness of the request munication period. Second, PATTHEL is a cross-layer protocol
scheduler is that the segments are divided into fixed-sized because it adds an entrance to the routing table in order to
subsegments, which may lead to low performance with con- deliver data over a certain channel.
strained receive buffer. Evensen et al. [53], [54] proposed Miyazaki and Oguchi [130] examined how much receive
improving the request scheduler by loosing the segment buffer is needed in various scenarios and found that the buffer
size constraint. For example, the segment sizes are dynam- size is proportional to the ratio of the bandwidth of the two
ically calculated on the fly based on the capability of interfaces. A larger bandwidth difference leads to a bigger
each path. receive buffer and vice versa. The scheduling algorithm used
4) Session Layer Multipath Capability: Unlike application in [130] is EDPF.
layer approaches discussed previously, some approaches open Habak et al. [70], [71] proposed a Deplorable Bandwidth
multiple TCP flows without any change to existing applica- Aggregation System (DBAS) middleware architecture
tions by providing specialized middleware or virtual sockets for multi-interface enabled devices. Like the work
at the session layer between the application and transport layer. in [68], [111], and [166], DBAS also supports both FOSM
Sivakumar et al. [184] proposed PSockets (Parallel Sockets). and POSM. In FOSM where the server is not DBAS enabled,
PSockets is a library that transparently partitions upper layer DBAS schedules different connections to the interfaces such
data into multiple transport streams through the same phys- that a connection can be assigned to only one of the available
ical path. The principal idea is to split data equally across interfaces. If both sides are DBAS enabled, POSM is used
several open sockets without application upgrade. PSockets so that each packet can be scheduled independently on a dif-
has the same Application Programming Interfaces (APIs) as ferent interface. To make better scheduling decisions, DBAS
those of a regular socket. Note that this work was published estimates the characteristics of the applications dynamically
in the year 2000. Back then, the TCP window size had to be based on their behavior and stores them in a database for
tuned manually for high-speed networks at both the source and history track. DBAS focuses on the actual implementation
the destination in order to achieve the maximum throughput. of the basic core system. The authors presented an extended
Differently, PSockets allowed applications to achieve the best work based on DBAS, a Green DBAS (G-DBAS) [69] to
performance without tuning the window size. balance overall throughput with energy consumption. For
Hasegawa et al. [74], [75] proposed an Arrival-Time match- example, they introduced a new utility based scheduler that
ing Load-Balancing (ATLB) layer between the application takes energy consumption of each interface into account in
layer and TCP layer. ATLB consists of a distributed data trans- order to balance the trade-off between maximizing throughput
fer method and a path-failure detection/recovery method. In and minimizing power consumption. Note that G-DBAS
order to mitigate the reordering effect at the receiver, ATLB only works in FOSM. OPERETTA [66] is an extension of
calculates the data arrival time for each path. It considers the G-DBAS to support POSM.
time that data segments spend in the TCP queue at a sender Application layer approaches split a single file or byte
and the time needed for data segments to pass through the stream into segments that are transmitted concurrently over
network. different paths. These kind of approaches seem to be simple
Qureshi et al. [155] presented a prototype system Tavarua. in the sense that the applications are in full control of the strip-
Tavarua is a middleware for providing network striping capa- ing decisions. Thus, it does not require any protocol change
bility to applications with high demands on uplink throughput. at lower layers so that clients and servers can find an optimal
Note that Tavarua runs upon UDP. In an effort to mitigate way to collaborate. Nevertheless, the complexity and over-
the impact of variations in bandwidth, an application can use head at the application layer are considerable. For example, an
feedback information to estimate the maximum available data application-level sequence number has to be included in each
transmission rate. Then the bit-rate at which the video is of the application defined headers. Meanwhile, the application
encoded is adapted dynamically. The middleware also han- has to explicitly ensure that the application layer data units,
dles low-level issues related to the network interfaces (e.g., which carry unique application-level sequence numbers, do
congestion control, disconnections, and reconnections). not get fragmented during transmission. Moreover, a dedicated
TABLE XV
E VALUATION OF E ND - TO -E ND PACKET R EORDERING A LGORITHMS required sequencing at the destination are assigned only to
the same path. However, a multipath transmission protocol
with them usually performs worse than without them in terms
of load sharing. For example, if the number of flows is
less than that of available paths, there would exist paths
which cannot be fully utilized. Therefore, we give - to their
load-sharing capability due to their lower performance than
average.
FPS [37], [38], [74], [75], [110], [114], [115], [129],
resequencing mechanism is required to reassemble the data
[202], [206] breaks the in-order scheduling rule at the source.
at the receiver. In practice, different paths may have diverse
Whenever a path has opportunity to send a new packet, it
characteristics, and the striping ratio may not exactly match
estimates that paths capacity and chooses a new data block
the ratio of data rate from different paths. A large receive
accordingly so that out-of-order sent out packets could arrive
buffer (on the application level) is required to accommodate
at the receiver in-order. We grade FPS a + to its packet reorder-
the out-of-order data. Finally, in order to split intelligently, the
ing capacity because it concerns most on whether the packets
application has to implement a bandwidth estimation mech-
arrive at the receiver, and a ++ to its load-sharing capacity
anism redundantly despite the same mechanism has been
because fully utilizes each paths capacity.
employed by TCP through its congestion control mechanism.
In EDPF [24], [81], [103], [106], [130], [178], [179] and
The middleware approaches are very similar to application-
PET [27], the scheduler first sends data on a path with lowest
layer approaches. They also face the same challenges, for
RTT until it has filled its congestion window. Then, the data is
example, the reordering issue. The advantage of middleware
sent on the subflow with the next lowest RTT. We grade them
approaches is that although it still requires client and server-
o and + to their reordering and load-sharing capacities respec-
side modifications, applications usually are not required to be
tively because EDPF and PET preferentially schedule data on
upgraded.
the available path with lowest RTT. The larger the RTT differ-
ence is, the more out-of-order packets arrive at the destination.
E. Summary Moreover, EDPF and PET consider only path bandwidth
In this section, we summarize the issues that are common to and latency, ignoring packet loss rate caused by congestion.
the approaches from all layers we have covered in this survey. Therefore, it may achieve sub-optimal load-balancing in the
Specifically, we first discuss the packet reordering problem presence of high losses or heavy congestion.
and how effective the widely used scheduling algorithms are WRR [5], [6], [10], [13], [16], [17], [46], [52][56], [66],
to mitigate it. After that, we present the approaches which [68][71], [74][76], [82][85], [96], [97], [101], [102], [118],
have adopted the cross-layer design. Next, we compare the [121], [148], [149], [155], [161], [162], [165][168], [171],
approaches compatibility capability, which is inherently deter- [172], [185], [186], [195], [196], [198], [204] is the most
mined by their stack positions. At last, we summarize the widely used scheduling algorithm on all layers with maxi-
research problem evolution on each layer. mizing the overall throughput as its first priority. It could
1) Packet Reordering: When packets travel through differ- achieve the goal if the multiple paths have similar charac-
ent paths which may have mismatched characteristics, they teristics. Otherwise, it would cause significant out-of-order
may arrive at the destination out of order. All the presented packets at the receiver because it considers little about its
approaches deal, to some extent, with the reordering issue on impact on packet reordering. Therefore, WRR works better
the layer which they are located at. If they ignore or have on link layer than other layers because link layer has rela-
no control over the reordering mechanism on the transport tively stable link state. Like FPS, WRR could fully utilize the
layer, their approach may suffer from performance degrada- available path capacity. Based on the discussion, we give o to
tion because of the misinterpretation of out-of-order packets. WRRs packet reordering capacity and ++ to its load-sharing
Table XV lists the primary algorithms used to mitigate packet capability.
reordering without considering which layers they are located In practical network environments where path characteris-
at. We group and sort them according to their effectiveness in tics may change dynamically, the scheduling algorithms except
terms of packet reordering and load-sharing capability. We use PFA, PCA and Multi-streaming may fail to counter against
four levels (++, +, o, -) to grade the mechanisms only for gen- packet reordering. Several additional mechanisms have been
eral and relative evaluation. The level ++ indicates the most proposed to work coherently with the scheduling algorithms,
efficiency. + comes second, and then o and -. A scheme graded such as ACK manipulation [46], [68], [89], [104], [111], [116],
- does not imply that it is useless. Instead, that scheme may [122], [167], buffering management [8], [26], [49], [68], [109],
work well either in certain network context or with additional packet coding [38], [114][116], [178], blocking warning and
buffer management. fast retrieving [81][84], [106], [115], [160], slow-path penal-
The first group includes PCA [2], [88], [135], [188], ization [58], [140], [160] and so on. The comparison of various
PFA [51], [57], [68][71], [95], [111], [127], [150], [166], combinations of the scheduling algorithms is out of the scope
and Multi-streaming [50]. They are the most effective mech- of our knowledge because they may be used in various network
anisms which can completely eliminate packet reordering environments, triggered by different conditions, and supported
incurred by multipath transmission because the data units by diverse assumptions.
TABLE XVI
C ROSS -L AYER S UPPORT FOR M ULTIPATH T RANSMISSION
To make a conclude on the discussion of packet reorder- of the measurements. Therefore, we argue that the schedul-
ing, the choice of the best scheduling policy depends ing algorithms which rely on the estimation of certain path
on path characteristics. There is no single scheduling characteristics are layer-dependent.
mechanism which can handle all scenarios. To adapt to We discuss WRR as an example to support our argument.
different network environments, several approaches, such The same principle is applicable to other algorithms which
as [68], [70], [71], [111], and [166], could support both flow rely on the estimation of path characteristics. WRR is the most
level scheduling and packet level scheduling strategies. widely used scheduling algorithm. Although each layer adopts
2) Layer-Dependent Scheduling Algorithms: In our many variants of the WRR algorithm, its effectiveness highly
previous discussion, we could find that many scheduling algo- depends on how correctly the path capability is estimated in a
rithms are shared by different approaches on various layers. dynamic fashion. For example, using the congestion window
For example, FPS variants are used on transport (includ- size on the transport layer to estimate the path capability is a
ing [37], [38], [110], [114], [115], [129], [202], and [206]) more lightweight and accurate way than those using various
and application layers (including [74] and [75]). EDPF probing methods on other layers. Furthermore, WRR works
variants are used on network layer (including [27], [65], much better on link layer than upper layers because link layer
[104], [105], and [109]), transport layer (including [24], [81], has more stable link status.
[103], [106], [178], and [179]) and application layer (includ- 3) Cross-Layer Support: In this survey, we define that any
ing [130]). WRR variants are used from link to application attempt to violate the TCP/IP reference model is considered
layers. However, we argue that it does not imply that these a cross-layer design. Among the approaches we have dis-
scheduling algorithms are layer-independent. Instead, they are cussed previously, several of them have explicitly involved
mostly layer-dependent. cross-layer interaction for purposes of estimating path status
Running a scheduling algorithm usually requires measuring to avoid packet delays and losses, scheduling traffic over mul-
bandwidth, delay, loss, or jitter to provide best effort or even tiple paths according to their capacity, exploring path diversity
QoS guarantees. No single measurement on a certain layer to obtain high throughput, achieving better QoS for multimedia
could always give accurate measurements. The measurements applications, and so on. Due to these benefits the cross-layer
may even vary on different layers. Although how to measure design may offer, there has been increased interest in proto-
those metrics is an entire problem unto its own, the effi- cols with interactions between various layers of the network
ciency of the algorithm is closely connected to the correctness stack. In the rest of this section, we discuss the cross-layer
TABLEXVII
C OMPATIBILITY E VALUATION ( M EANS B EING S UPPORTED )
approaches based on our previous discussion and give a few three categories according to how many proxies required in
observations on them without our own judgment. Instead, we each category. We use MPTCP and CMT-SCTP to represent
refer readers to Kawadia and Kumars [98] work which calls the transport layer approaches and evaluate their compatibility
for a cautionary approach to cross-layer design. Although [98] separately. Likewise, we also separate session layer approaches
examined the issue of cross-layer design in wireless networks, from application layer approaches.
we believe the same principle is also applicable for multipath Application compatibility means that the lower layer
transmission. changes do not require the legacy applications to be upgraded.
Table XVI shows approaches based on a cross-layer design. Obviously, the link layer and all the network layer approaches
The Base Layer indicates the layer where the data splitting are compatible with the legacy applications because they do
is initiated, and the Additional Layers indicate which lay- not change the socket interface between the legacy applications
ers are required to provide support to the base layer. From and transport layer protocols. MPTCP presents a standard TCP
the table, we observe that the transport and application lay- socket API to the application so that legacy applications can
ers are the main base layers. From 2005 to 2013, it was run upon MPTCP transparently. However, the legacy appli-
drawn most of the attention to implement the base layer cations running on TCP have to upgrade in order to take
on the application level. Some applications obtain low layer advantage of CMT-SCTP. Session layer approaches usually
information to optimize their behavior in terms of interface keep the same socket API to applications (e.g., PSockets [184])
selection, load balancing, and energy efficiency. The infor- so that they require no changes to the legacy applications.
mation includes throughput history and smoothed RTT from Application layer approaches have a serious application com-
transport level, routing table from IP level, and link status patibility issue because the multipath transmission property
(e.g., energy consumption, available bandwidth, and bit error needs to be implemented for specific applications.
rate). Some applications can even change the low layer proto- An updated approach has backward compatibility if it can
col behaviors such as changing TCP window size to control either work with its communicating peer which uses the
the throughput, modifying the routing table to optimize path standard approach or automatically fallback to the standard
selection, and disconnecting/reconnecting certain interfaces for approach if the communicating peer does not support the
energy efficiency or partial failure. In most recent years, it new features. Link layer approaches generally do not main-
has become more attractive to use transport layer as the base tain the backward compatibility because most of them are
layer with additional support from network and link layers. proprietary approaches and require dedicated setup on both
Although transport layer approaches have advantages from sides. Therefore, we mark that the link layer approaches are
the congestion control mechanism, they lack the choice of not backwards compatible. Network layer approaches with no
path diversity to free the constraint of fairness control. The proxy usually employ some of the fields in protocol headers
path optimization support from network and link layers can (e.g., [65], [68], [122], and [148][150]) to negotiate multiple
compensate for the weakness in network and Ethernet levels IP addresses to be used or piggy-back information so as to pro-
respectively. In addition, some transport layers can suspend or vide backward compatibility. Network layer approaches with
release a path based on the estimated MAC information. one proxy has backward compatibility because the communi-
We can also find that the interaction of cross-layer design cating peer is unaware of the multipath transmission between
may not be limited to adjacent protocol layers. Instead, it the client and the proxy. The network layer approaches requir-
allows vertical communication to take place between nonad- ing two proxies are also backwards compatible because both
jacent layers. The cross-layer design approaches are actually sides are unaware of the multipath transmission between the
not limited to the ones listed in Table XVI because although two proxies. MPTCP is backwards compatible with plain TCP
many approaches above the network layer did not mention or because it can fallback to single-path TCP if the commu-
specify how packets are delivered through multiple interfaces, nicating host does not support the extensions. CMT-SCTP
the cross-layer support from routing function on network layer related articles did mention at all whether it can fallback to
may be assumed implicitly. SCTP if the server is not CMT-SCTP enabled. We believe
4) Compatibility: In this section, we evaluate the com- CMT-SCTP could also fallback to SCTP if the server does
patibility capacity of the approaches on different layers. not support concurrent multipath transfer. But anyway, CMT-
The evaluation is made in general because there may be SCTP has inherited the backward compatibility issue of SCTP
exceptions in certain approaches. Table XVII presents the itself. For example, if the server is only aware of TCP opera-
evaluation where we separate network layer approaches into tions, a CMP-SCTP client may fail to create connection with
TABLEXVIII
R ESEARCH P ROBLEMS ON E ACH L AYER ( C M EANS H AVING T RIED TO A DDRESS )
the server. Session layer and application layer approaches are real deployment. Network layer approaches lack discussion on
generally backwards compatible with the legacy applications. backward compatibility.
When connecting to a server, a client application usually spec- 5) Evolution of Research Problems: Table XVIII presents
ifies two different TCP ports, a probe one and a fallback one. the research problems the approaches on each layer have
First, the client tries to establish a connection using the probe tried to address. In the early stage of multipath transmission
port to check whether the server is upgraded. If the operation research, most approaches emphasized only bandwidth aggre-
fails, another attempt is made by using the fallback port to gation with various scheduling and packet reordering algo-
create a standard connection. rithms. Few of them considered the fairness and RP features.
The compatibility with existing middleboxes, such as fire- Today, these two problems as well as Pareto-optimality prob-
walls and NATs, affects whether the new packets are able lem have become challenges along with the revolutionary
to traverse the legacy middleboxes. First of all, link layer development of multipath transmission.
approaches generally would not have this issue because the The fairness requirement on multipath transmission was
communication paths or trees are well designed between dedi- unclear in the beginning. For example, the early research on
cated multipath-aware devices. Most network layer approaches multipath transmission focused on bandwidth aggregation by
(with and without proxies) split bytestream over multiple taking advantage of the resources through multiple interfaces.
paths. Therefore, the single sequence space across more than The target in the research community matched the potential
one path leaves gaps in the sequence space seen on any indi- expectation of end users because an end user can benefit
vidual path, which may upset certain middleboxes. To solve from the aggregated bandwidth if they have paid for both
this issue, the double sequence space design was proved to accesses. Thus, the fairness emphasized the fairness of each
be an effective solution [80], [160]. However, how to use the individual subflow; for example, each subflow gets as much
double space also influenced the outcome. For example, HIP bandwidth as a standalone TCP flow does. In recent years,
based multipath transmission adopts double sequence space. the research focus was on the fairness of the multiple sub-
But the additional sequence space is used for the purpose of flows as a whole at shared bottlenecks. The principle is that a
anti-replay instead of resequencing. In contrast, MPTCP pack- multipath transmission should behave as a single TCP flow at
ets, which carry double sequence numbers for resequencing on shared bottlenecks. Coordinated congestion control algorithms
two levels, can traverse most of the middleboxes. CMT-SCTP are used as a powerful tool to achieve it.
packets may fail to traverse through certain middleboxes due The concept of RP [200] was proposed in 2008. The early
to its single sequence space design. Session and application approaches before it also proposed using less congested paths
layer approaches create multiple standalone TCP flows so that more, which is one of the RP principles. From the conges-
their TCP flows can travel through various middleboxes as tion control algorithm viewpoint, the difference between these
normal TCP does. approaches and the approaches after 2008 is that the latter
Host compatibility means whether the approaches require ones use coordinated congestion control algorithms instead of
changes in hosts. We found that all approaches need changes independently tuning each paths congestion control behavior.
on hosts except the network layer approaches with two proxies The Pareto-optimality is a state of resource allocation in
because the multipath transmission between the two proxies which there is no alternative state that would make some
are unknown to both communicating peers. people better off without making anyone worse off. MPTCP
Infrastructure compatibility means whether an approach with LIA [17], [60], [158], [162] fails to satisfy the Pareto-
needs additional network infrastructure such as NAT box and optimality because upgrading some regular TCP users to
proxy. We found that only the network layer approaches with MPTCP can reduce the throughput of other users without any
the proxy support require additional infrastructure. benefit to the upgraded users. OLIA [101], [102] as an alter-
According to the previous discussion, it is hard to imple- native for LIA could make MPTCP Pareto-optimal and satisfy
ment a generic multipath solution which can satisfy all the the three design goals of MPTCP.
compatibility requirements. As summarized in Table XVII, In Table XVIII, we observe that the multipath transmission
we have a few more observations. For example, MPTCP follows an evolutionary way mostly on the transport layer.
and session layer approaches have more compatibility sup- We believe this is determined by the stack position at which
port than others. CMT-SCTP has inherited SCTPs application the proposed approaches are located. For example, the fair-
compatibility issue, which becomes the major obstacle of its ness, RP, and Pareto-optimality features are achieved by the
means of congestion control, which as default is managed on did not support it any more. Secondly, although MPTCP has
the transport layer. The link layer and application layer can- been implemented in Linux kernel, people have to manually
not intervene congestion control without breaking the protocol install it (e.g., by the means of apt-repository or compiling
stack layered structure. In our literature review, mHIP [151] is and installing from source) before trying it. It would become
the only protocol which provides fairness feature on network much convenient if some mainstream Linux distributions can
layer. However, mHIP is actually located on a middle layer add the MPTCP kernel directly in their releases. Thirdly, much
between network and transport layers. Therefore, because of work has been done to improve the performance of MPTCP
its closeness to transport layer, mHIP is able to manipulate in terms of packet reordering and Pareto-efficiency. But they
the congestion control operations. The link layer and network are still open questions and need more technical improvement.
layer can provide path diversity for cross-layer approaches.
For example, OpenFlow and TRILL can provide Ethernet level
path diversity [34], [194], and LISP can provide routing level V. O PEN Q UESTIONS
path diversity [34], [36]. We have identified a few open questions and present them
as follows.
IV. L ESSONS L EARNED Multipath transmission in Information-Centric
Networking (ICN): in contrast with the host-centric
From the evolution of end-to-end multipath transmission
paradigm based on perpetual connectivity and the
viewpoint, we observe that multipath has become increasingly
end-to-end principle, ICN was proposed to make the
popular at the transport layer with features such as load balanc-
network content centric allowing nodes to request
ing, fairness control, congestion control and Pareto-optimality.
content that is then delivered by the network to them. In
As a major extension to TCP which has not changed very
this new networking paradigm, information retrieval is
much in the last decades, MPTCP has attracted more and more
pull-based, driven by user requests, point-to-multipoint
attention in recent years. We refer to Figure 2, Table XVII and
and intrinsically coupled with in-network caching. In
Table XVIII to summarize the lessons of its development we
ICN, a content item can be replicated in more than one
have learned from this survey. We believe that they are valu-
node. There is an increased interest in adapting multipath
able lessons that others should learn in order to make their
transport control to ICN in [23], [42], and [170] so
proposals into practice.
that the delivery of the content can follow more than
As shown in Figure 2, MPTCP was proposed at an oppor-
one path to reach the user. The benefits of multipath
tune time to draw the experience gathered in previous work
in ICN include increased resilience and decreased load
and correcting the past mistakes. For example, MPTCP was
to content repositories. The combination of ICN and
designed with the backward compatibility to legacy applica-
multipath transport brings new challenges in terms of
tions and middleboxes (see Table XVII), which makes MPTCP
balancing the performance maximization and network
instead of CMT-SCTP a big step towards being widely accept-
cost minimization. A multipath solution for ICN needs to
able. In addition to solving the compatibility issue, MPTCP
take into account that the content sources are unknown
goes further in addressing issues in fairness and RP by joint
in advance and may vary over time, and that in-network
increase and decrease rules (e.g., the coupled congestion con-
caching may impact the variability of path length and
trol algorithm LIA [162]). Furthermore, OLIA [102] and
the associated delivery time.
AOLIA [182] even make MPTCP meet the requirements of
Currently, there are not many existing multipath trans-
Pareto-efficiency (see Table XVIII). Apart from the technical
mission approaches suitable for ICN. A strategy layer has
aspects, we believe the following factors also contribute to
been proposed for Content Centric Networks (CCN) that
the success of MPTCP: kernel implementation in Linux, sup-
make decisions pertaining to the multipath selection pro-
port from Internet standards organization (e.g., IETF), active
cess. Naive multipath strategies have been reported to
and public academic community,7 and incentives from indus-
negatively impact CCN efficiency [170]. An analyti-
try [1], [4], [20]. According to the previous discussion, we
cal model has been proposed for evaluating different
use the following key words to summarize the key efforts
ICN multipath forwarding strategies. According to the
which has driven the development of MPTCP: collecting previ-
model, a good forwarding strategy that maximizes the
ous experience, correcting past mistakes, continuously finding
receive-rate should control the pending interests injected
and solving new challenges, implementation, standardization,
in the different paths so as to fill the capacity of their
research community and industrial incentives.
pipelines [42]. Joint multipath congestion control and
Although MPTCP is a promising protocol for multipath
request forwarding have been investigated in [23] with
transmission, it still has room for improvement. We give a few
the twofold objective of maximizing user throughput and
recommendations as follows. First, MPTCP needs more mar-
minimizing overall network cost. Relevant research top-
keting work to get more support from industry. Currently,
ics for multipath ICN include forwarding strategies for
only a few companies (e.g., [1], [20], and [138]) make prod-
wireless and dynamic networks, joint design of forward-
ucts or services on MPTCP. Although Apple implemented
ing strategies and cache replacement policies, and routing
MPTCP (in a backup mode) in iOS 7, later iOS upgrades
protocol support for multipath.
7 MPTCP has an active mail list ([email protected]) for Context-aware scheduling: we refer Table XV to sup-
sharing experience. port our discussion on this open question. As shown in
the table, no individual algorithm wins from both the schemes [31], [63], [107], [180] can be potential solutions
packet reordering and load sharing capability. This result to them.
implies that there is no single algorithm that is one-size- Heterogeneity of multipath: there are many different
fits-all. The efficiency of an algorithm depends on how network environments for multipath transmission. For
much it fits to the network environment. For example, example, in modern data centers, there are usually more
although an algorithm may drive better results than others than one path available between any pair of endpoints.
in certain network environment, it cannot beat others all The same applies to access networks, the core of
the time in a dynamically changing network. Each algo- the Internet, and Internet Service Providers (ISPs).
rithm has its intended network environment in which to Nevertheless, these domains are usually autonomous and
work the best. Currently, most work has been using one isolated from each other. There are specialized network
single scheduling algorithm all the time, which would entities, e.g., border gateway, to guarantee the autonomy
definitely lead to low performance in certain network of each domain. The downside of this network topology
conditions. Therefore, context-aware scheduling policy is that even though there might be abundant multipath
is required to dynamically detect the network context resources available locally within each domain, the glob-
and switch to the best performing scheduling algorithm ally available multi-path resources are limited to the
accordingly. The concept of context-aware scheduling border gateways. Thus, how to break the border of each
is not new. Some initial work has been conducted. autonomous domain to enrich the multipath resources sig-
For example, WiMP-SCTP [85] has two data transmis- nificantly is not only a technical problem, but also a
sion scheduling algorithms used for different network management problem. It requires efforts from multiple
conditions. When the network condition is good, the domains, including ISPs, policymakers and end users.
data-striping algorithm is selected to aggregate band- Specialized use of multipath transmission: most of the
width. When the network condition becomes bad, the existing approaches aim to obtain higher throughput from
data-duplicating algorithm is switched on to increase des- the use of multipath transmission. However, from a users
tination reachability. In [166] and [68], there are two perspective, boosting the throughput of a multi-homed
scheduling algorithms which are used for legacy des- mobile device may not be the first-priority goal all the
tination and updated destination respectively. For more time. Instead, some users may be willing to use cheap
similar work, we refer readers to [70], [71], and [111]. subscriptions in order to save a few dollars. Others would
Richer API of transport services: if the application were rather use the low energy-consumption interface(s) to
to explicitly control the congestion control algorithms by save the battery life. Therefore, multipath transmission
the means of APIs provided by transport protocols, then strategies which take price and energy into consideration
it would not only know everything the transport layer should be provided to users so that they can choose from
knows but also what the application knows (e.g., the different transmission strategies to satisfy their demands.
workload and application content type), which helps in
making optimal decisions. This is one of the goals the
TAPS (Transport Services) work group (WG) in IETF VI. C ONCLUSION
plans to achieve. For example, the TAPS WG will identify Conventional TCP/IP always uses a single best path
the services provided by existing IETF transport protocols according to certain routing metrics, even if there may be
and congestion control mechanisms as well as network more than one path between two endpoints. This behavior
requirements of APIs. The application layer approaches results in under-utilization of the available network resources.
could then use the standard APIs to control mechanism The proliferation of mobile devices equipped with multiple
underneath. interfaces, represented by smart phones, brings with it a grow-
Multicast meets multipath: multicast traffic over the ing number of multi-homed hosts onto the Internet. Thus, this
Internet is growing steadily with the increasing number deteriorates the mismatch between single-path transport and
of demanding applications. Many of them require certain the multitude of available network paths. Multipath transmis-
QoS guarantees, and demand that the network resource sion comes into the picture as a natural solution with several
be utilized in an efficient way. To achieve these goals, salient features, such as reliability, fairness, RP and Pareto-
the multipath transmission could be used to effectively optimality. In this article, we make a comprehensive survey
split multicast traffic over multiple paths at the edge of about the state-of-the-art multipath transmission approaches,
the multicast tree. For example, the future of mobile con- intending to provide researchers and practitioners with insight-
tent delivery may use multicast networks for audio/video ful observations. We hope this survey will inspire a series of
streaming applications. The last hop of the multicast net- new research work in this field.
works would be based on various wireless technologies
(e.g., WiFi, 3G, and LTE). Due to the fact that an indi-
vidual wireless path may be unreliable and be unable to A PPENDIX
provide required bandwidth, multipath transmission for L IST OF ACRONYMS
the last hop would improve its resilience and throughput. AOLIA Adapted Opportunistic Linked Increases
Generally, the challenges include packet reordering and Algorithm
in-network caching issues. We believe that packet coding ATLB Arrival-Time matching Load-Balancing
BERP Bandwidth Estimation Based Resource [10] M. Allman, H. Kruse, and S. Ostermann, An application-level solution
Pooling to TCPs satellite inefficiencies, in Proc. 1st Int. Workshop Satellite
Based Inf. Services (WOSBIS), 1996.
BMP Buffer Management Policy [11] P. Amer et al., Load sharing for the stream control transmission pro-
CMT Concurrent Multipath Transfer tocol (SCTP), draft-tuexen-tsvwg-sctp-multipath-10 (Experimental),
DAC Delayed ACK for CMT Internet-Draft, IETF, Nov. 2015.
[12] P. D. Amer, C. Chassot, T. J. Connolly, M. Diaz, and P. Conrad,
DBAS Deplorable Bandwidth Aggregation System Partial-order transport service for multimedia and other applications,
ECMP Equal Cost Multipath IEEE/ACM Trans. Netw., vol. 2, no. 5, pp. 440456, Oct. 1994.
ECT Equal Cost Tree [13] A. Argyriou and V. Madisetti, Bandwidth aggregation with SCTP,
in Proc. IEEE Glob. Telecommun. Conf. (GLOBECOM), vol. 7.
EDPF Earliest Delivery Path First San Francisco, CA, USA, 2003, pp. 37163721.
FOSM Flow-Oriented Scheduling Mode [14] B. Arzani, A. Gurney, S. Cheng, R. Guerin, and B. T. Loo, Impact of
FPS Forward Prediction Scheduling path characteristics and scheduling policies on MPTCP performance,
in Proc. 28th Int. Conf. Adv. Inf. Netw. Appl. Workshops (WAINA),
HIP Host Identity Protocol Victoria, BC, Canada, May 2014, pp. 743748.
HLB Head-of-Line Blocking [15] AVAYA. (2011). Multi-Link Trunking. [Online]. Available:
HTTP-RP HTTP Request Pipelining https://downloads.avaya.com/css/P8/documents/100134063
HTTP-RRR HTTP Range Retrieval Request [16] A. Baldini, L. De Carli, and F. Risso, Increasing performances of
TCP data transfers through multiple parallel connections, in Proc.
LACP Link Aggregation Control Protocol IEEE Symp. Comput. Commun. (ISCC), Sousse, Tunisia, Jul. 2009,
LIA Linked Increase Algorithm pp. 630636.
LISP Locator/Identifier Separation Protocol [17] S. Barr, C. Paasch, and O. Bonaventure, MultiPath TCP: From theory
to practice, in Proc. 10th Int. IFIP TC 6 Conf. Netw. (NETWORKING),
MPTCP Multipath TCP Valencia, Spain, 2011, pp. 444457.
NAT Network Address Translation [18] M. Becke et al., Comparison of multipath TCP and CMT-SCTP based
OLIA Opportunistic Linked Increases Algorithm on intercontinental measurements, in Proc. IEEE Glob. Commun.
Conf. (GLOBECOM), Atlanta, GA, USA, 2013, pp. 13601366.
PCA Per-Conversation Allocation [19] E. Blanton and M. Allman, On making TCP more robust to packet
PET Packet-Pair based EDPF for TCP applications reordering, ACM SIGCOMM Comput. Commun. Rev., vol. 32, no. 1,
PFA Per-Flow Allocation pp. 2030, 2002.
[20] O. Bonaventure. (2015). In Korean, Multipath TCP is Pronounced
POSM Packet-Oriented Scheduling Mode GIGA Path. [Online]. Available: http://blog.multipath-tcp.org/blog/
RP Resource Pooling html/2015/07/24/korea.html
RR Round Robin [21] L. Budzisz, R. Ferrs, F. Casadevall, and P. Amer, On concurrent
multipath transfer in SCTP-based handover scenarios, in Proc. IEEE
RTT Round-Trip Time Int. Conf. Commun. (ICC), Dresden, Germany, 2009, pp. 16.
SACK Selective Acknowledgment [22] D. Bushmitch, S. Panwar, and A. Pal, Thinning, striping and shuffling:
SCTP Stream Control Transmission Protocol Traffic shaping and transport techniques for variable bit rate video, in
Proc. IEEE Glob. Telecommun. Conf. (GLOBECOM), vol. 2. Taipei,
SPB Shortest Path Bridging Taiwan, Nov. 2002, pp. 14851491.
TRILL Transparent Interconnection of a Lot of Links [23] G. Carofiglio, M. Gallo, L. Muscariello, M. Papalini, and S. Wang,
WRR Weighted Round Robin. Optimal multipath congestion control and request forwarding in
information-centric networks, in Proc. 21st IEEE Int. Conf. Netw.
Protocols (ICNP), Gttingen, Germany, 2013, pp. 110.
[24] C. Casetti and W. Gaiotto, Westwood SCTP: Load balancing
R EFERENCES over multipaths using bandwidth-aware source scheduling, in Proc.
Veh. Technol. Conf. (VTC), vol. 4. Los Angeles, CA, USA, 2004,
[1] Hybrid Internet Access Bonding. [Online]. Available: pp. 30253029.
http://www.tessares.net/solutions/dsl-lte [25] C. Cetinkaya and E. W. Knightly, Opportunistic traffic scheduling over
[2] IEEE Standard for Local and Metropolitan Area NetworksLink multiple network paths, in Proc. 23rd Annu. Joint Conf. IEEE Comput.
Aggregation, IEEE Standard 802.1AX-2008, pp. 1163, 2008. Commun. Soc. (INFOCOM), vol. 3. Hong Kong, 2004, pp. 19281937.
[3] IEEE Standard for Local and Metropolitan Area NetworksMedia [26] K. Chebrolu, B. Raman, and R. R. Rao, A network layer approach to
Access Control (MAC) Bridges and Virtual Bridged Local Area enable TCP over multiple interfaces, Wireless Netw., vol. 11, no. 5,
NetworksAmendment 20: Shortest Path Bridging, IEEE Standard pp. 637650, 2005.
802.1aq-2012, pp. 1340, Jun. 2012. [27] K. Chebrolu and R. R. Rao, Bandwidth aggregation for real-time
[4] iOS: Multipath TCP Support in iOS 7. (2015). [Online]. Available: applications in heterogeneous wireless networks, IEEE Trans. Mobile
https://support.apple.com/en-us/HT201373 Comput., vol. 5, no. 4, pp. 388403, Apr. 2006.
[5] A. Abd El Al, T. Saadawi, and M. Lee, Improving throughput and [28] J. Chen, K. Xu, and M. Gerla, Multipath TCP in lossy wire-
reliability in mobile wireless networks via transport layer bandwidth less environment, in Proc. IFIP 3rd Annu. Mediterr. Ad Hoc Netw.
aggregation, Comput. Netw., vol. 46, no. 5, pp. 635649, Dec. 2004. Workshop (Med-Hoc-Net), Bodrum, Turkey, 2004, pp. 263270.
[6] A. Abd El Al, T. Saadawi, and M. Lee, LS-SCTP: A bandwidth aggre- [29] X. Chen, A. Jukan, A. C. Drummond, and N. L. S. da Fonseca,
gation technique for stream control transmission protocol, Comput. A multipath routing mechanism in optical networks with extremely
Commun., vol. 27, no. 10, pp. 10121024, Jun. 2004. high bandwidth requests, in Proc. IEEE Glob. Telecommun.
[7] S. Addepalli, H. G. Schulzrinne, A. Singh, and G. Ormazabal, Conf. (GLOBECOM), Honolulu, HI, USA, 2009, pp. 16.
Heterogeneous access: Survey and design considerations, [30] Y.-C. Chen et al., A measurement-based study of multipath TCP per-
Dept. Comput. Sci., Columbia Univ., New York, NY, formance over wireless networks, in Proc. ACM Conf. Internet Meas.
USA, Tech. Rep. CUCS-28-13, 2013. [Online]. Available: Conf. (IMC), Barcelona, Spain, 2013, pp. 455468.
http://dx.doi.org/10.7916/D8QJ7F8P [31] P. A. Chou, Y. Wu, and K. Jain, Practical network coding, in Proc.
[8] H. Adhari, T. Dreibholz, M. Becke, E. P. Rathgeb, and M. Txen, Allerton Conf. Commun. Control Comput., Monticello, IL, USA, 2003.
Evaluation of concurrent multipath transfer over dissimilar paths, in [32] Cisco. (2003). EtherChannels. [Online]. Available: http://
Proc. IEEE Workshops Int. Conf. Adv. Inf. Netw. Appl. (WAINA), 2011, www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3550/software/
pp. 708714. release/12-1_13_ea1/configuration/guide/3550scg/swethchl.html
[9] H. Adiseshu, G. Parulkar, and G. Varghese, A reliable and scalable [33] OpenFlow Consortium, Openflow switch specification/version
striping protocol, in Proc. Appl. Technol. Archit. Protocols Comput. 1.1.0, Feb. 2011. [Online]. Available: http://archive.openflow.org/
Commun., vol. 26. Stanford, CA, USA, 1996, pp. 131141. documents/openflow-spec-v1.1.0.pdf
[34] M. Coudron, S. Secci, G. Maier, G. Pujolle, and A. Pattavina, Boosting [54] K. Evensen et al., Using bandwidth aggregation to improve the perfor-
cloud communications through a crosslayer multipath protocol archi- mance of quality-adaptive streaming, Signal Process. Image Commun.,
tecture, in Proc. IEEE SDN Future Netw. Services (SDN4FNS), Trento, vol. 27, no. 4, pp. 312328, 2012.
Italy, 2013, pp. 18. [55] K. Evensen, T. Kupka, D. Kaspar, P. Halvorsen, and C. Griwodz,
[35] M. Coudron, S. Secci, and G. Pujolle, Differentiated pacing on mul- Quality-adaptive scheduling for live streaming over multiple access
tiple paths to improve one-way delay estimations, in Proc. IFIP/IEEE networks, in Proc. 20th Int. Workshop Netw. Oper. Syst. Support Digit.
Int. Symp. Integr. Netw. Manag. (IM), Ottawa, ON, Canada, May 2015, Audio Video, Amsterdam, The Netherlands, 2010, pp. 2126.
pp. 672678. [56] D. Farinacci, D. Lewis, D. Meyer, and V. Fuller, The locator/ID
[36] M. Coudron, S. Secci, G. Pujolle, P. Raad, and P. Gallard, Cross-layer separation protocol (LISP), IETF, Fremont, CA, USA, RFC 6830,
cooperation to boost multipath TCP performance in cloud networks, Feb. 2013.
in Proc. IEEE 2nd Int. Conf. Cloud Netw. (CloudNet), San Francisco, [57] D. Fedyk, P. Ashwood-Smith, D. Allan, A. Bragg, and P. Unbehagen,
CA, USA, Nov. 2013, pp. 5866. IS-IS extensions supporting IEEE 802.1 aq shortest path bridging,
[37] Y. Cui, L. Wang, X. Wang, H. Wang, and Y. Wang, FMTCP: A foun- IETF, Fremont, CA, USA, RFC 6329, 2012.
tain code-based multipath transmission control protocol, IEEE/ACM [58] S. Ferlin, T. Dreibholz, and O. Alay, Multi-path transport over het-
Trans. Netw., vol. 23, no. 2, pp. 465478, Apr. 2015. erogeneous wireless networks: Does it really pay off? in Proc. IEEE
[38] Y. Cui, X. Wang, H. Wang, G. Pan, and Y. Wang, FMTCP: A foun- Glob. Commun. Conf. (GLOBECOM), Austin, TX, USA, Dec. 2014,
tain code-based multipath transmission control protocol, in Proc. 32nd pp. 48074813.
IEEE Int. Conf. Distrib. Comput. Syst. (ICDCS), Macau, China, 2012, [59] R. Fielding et al., Hypertext transfer protocolHTTP/1.1, IETF RFC
pp. 366375. 2616, Fremont, CA, USA, Jun. 1999.
[39] M. Cullen et al., Problem statement: Bandwidth aggregation [60] A. Ford, C. Raiciu, M. Handley, and S. Barre, TCP exten-
for Internet access, draft-zhang-banana-problem-statement-01 sions for multipath operation with multiple addresses,
(Informational), Internet-Draft, IETF, Nov. 2015. IETF Internet-Draft, May 2009. [Online]. Available:
[40] T. Das and K. M. Sivalingam, TCP improvements for data center https://tools.ietf.org/html/draft-ford-mptcp-multiaddressed-00
networks, in Proc. 5th Int. Conf. Commun. Syst. Netw. (COMSNETS), [61] A. Ford, C. Raiciu, M. Handley, S. Barr, and J. Iyengar, Architectural
Bengaluru, India, 2013, pp. 110. guidelines for multipath TCP development, IETF, Fremont, CA, USA,
[41] G. Detal, C. Paasch, and O. Bonaventure, Multipath in the mid- RFC 6182, Mar. 2011.
dle(box), in Proc. Workshop Hot Topics Middleboxes Netw. Funct. [62] A. Ford, C. Raiciu, M. Handley, and O. Bonaventure, TCP extensions
Virtualization (HotMiddlebox), Santa Barbara, CA, USA, 2013, for multipath operation with multiple addresses, IETF, Fremont, CA,
pp. 16. USA, RFC 6824, Jan. 2013.
[42] A. Detti, C. Pisa, and N. B. Melazzi, Modeling multipath forward- [63] C. Fragouli, D. Lun, M. Mdard, and P. Pakzad, On feedback for
ing strategies in information centric networks, in Proc. IEEE Conf. network coding, in Proc. 41st Annu. Conf. Inf. Sci. Syst. (CISS),
Comput. Commun. Workshops (INFOCOM WKSHPS), Hong Kong, Baltimore, MD, USA, 2007, pp. 248252.
Apr. 2015, pp. 324329. [64] L. Golubchik et al., Multi-path continuous media streaming: What are
[43] Deutsche Telekom. (2014). Hybrid AccessA Promising Approach the benefits? Perform. Eval., vol. 49, nos. 14, pp. 429449, 2002.
to Increase Bandwidth of DSL-Lines. [Online]. Available: [65] A. Gurtov and T. Polishchuk, Secure multipath transport for legacy
http://www.laboratories.telekom.com/public/english/newsroom/news/ Internet applications, in Proc. 6th Int. Conf. Broadband Commun.
pages/hybrid-access.aspx Netw. Syst., Madrid, Spain, 2009, pp. 18.
[44] C. Diop, G. Dugue, C. Chassot, and E. Exposito, QoS-oriented [66] K. Habak, K. A. Harras, and M. Youssef, OPERETTA: An opti-
MPTCP extensions for multimedia multi-homed systems, in Proc. mal energy efficient bandwidth aggregation system, in Proc. 9th
26th Int. Conf. Adv. Inf. Netw. Appl. Workshops (WAINA), Fukuoka, Annu. IEEE Commun. Soc. Conf. Sensor Mesh Ad Hoc Commun.
Japan, Mar. 2012, pp. 11191124. Netw. (SECON), Seoul, South Korea, 2012, pp. 121129.
[45] J. Domza et al., A survey on methods to provide multipath trans- [67] K. Habak, K. A. Harras, and M. Youssef, Bandwidth aggregation
mission in wired packet networks, Comput. Netw., vol. 77, pp. 1841, techniques in heterogeneous multi-homed devices: A survey, Comput.
Feb. 2015. Netw., vol. 92, pp. 168188, Dec. 2015.
[46] Y. Dong, N. Pissinou, and J. Wang, Concurrency handling in TCP, [68] K. Habak, K. A. Harras, and M. Youssef, OSCAR: A collaborative
in Proc. 5th Annu. Conf. Commun. Netw. Services Res. (CNSR), bandwidth aggregation system, arXiv preprint arXiv:1401.1258, 2014.
Fredericton, NB, Canada, May 2007, pp. 255262. [69] K. Habak, M. Youssef, and K. A. Harras, G-DBAS: A green
[47] T. Dreibholz, M. Becke, H. Adhari, and E. P. Rathgeb, On the impact and deployable bandwidth aggregation system, in Proc. IEEE
of congestion control for concurrent multipath transfer on the transport Wireless Commun. Netw. Conf. (WCNC), Shanghai, China, 2012,
layer, in Proc. 11th IEEE Int. Conf. Telecommun. (ConTEL), Graz, pp. 32903295.
Austria, 2011, pp. 397404. [70] K. Habak, M. Youssef, and K. A. Harras, DBAS: A deployable
[48] T. Dreibholz, M. Becke, J. Pulinthanath, and E. P. Rathgeb, Applying bandwidth aggregation system, in Proc. 5th Int. Conf. New Technol.
TCP-friendly congestion control to concurrent multipath transfer, in Mobility Security (NTMS), Istanbul, Turkey, 2012, pp. 16.
Proc. 24th IEEE Int. Conf. Adv. Inf. Netw. Appl. (AINA), Perth, WA, [71] K. Habak, M. Youssef, and K. A. Harras, An optimal deploy-
Australia, 2010, pp. 312319. able bandwidth aggregation system, Comput. Netw., vol. 57, no. 15,
[49] T. Dreibholz, M. Becke, E. P. Rathgeb, and M. Tuxen, On the use of pp. 30673080, 2013.
concurrent multipath transfer over asymmetric paths, in Proc. IEEE [72] T. J. Hacker, B. D. Athey, and B. Noble, The end-to-end performance
Glob. Telecommun. Conf. (GLOBECOM), Miami, FL, USA, 2010, effects of parallel TCP sockets on a lossy wide-area network, in Proc.
pp. 16. IEEE Int. Parallel Distrib. Process. Symp. (IPDPS), 2002, pp. 434443.
[50] T. Dreibholz, R. Seggelmann, M. Texen, and E. Rathgeb, [73] A. Hari, G. Varghese, and G. Parulkar, An architecture for packet-
Transmission scheduling optimizations for concurrent multipath striping protocols, ACM Trans. Comput. Syst., vol. 17, no. 4,
transfer, in Proc. 8th Int. Workshop Protocols Future Large Scale pp. 249287, Nov. 1999.
Diverse Netw. Transp. (PFLDNeT), vol. 8. Lancaster, PA, USA, 2010, [74] Y. Hasegawa, I. Yamaguchi, T. Hama, H. Shimonishi, and T. Murase,
pp. 16. Improved data distribution for multipath TCP communication, in
[51] D. Eastlake, M. Zhang, A. Ghanwani, V. Manral, and A. Banerjee, Proc. IEEE Glob. Telecommun. Conf. (GLOBECOM), St. Louis, MO,
Transparent interconnection of lots of links (TRILL): Clarifications, USA, Nov. 2005, p. 5.
corrections, and updates, IETF, Fremont, CA, USA, RFC 7180, [75] Y. Hasegawa, I. Yamaguchi, T. Hama, H. Shimonishi, and T. Murase,
May 2014. Deployable multipath communication scheme with sufficient perfor-
[52] K. Evensen et al., A network-layer proxy for bandwidth aggre- mance data distribution method, Comput. Commun., vol. 30, no. 17,
gation and reduction of IP packet reordering, in Proc. IEEE pp. 32853292, 2007.
34th Conf. Local Comput. Netw. (LCN), Zrich, Switzerland, 2009, [76] S. Hassayoun, J. Iyengar, and D. Ros, Dynamic window coupling
pp. 585592. for multipath congestion control, in Proc. 19th IEEE Int. Conf. Netw.
[53] K. Evensen et al., Improving the performance of quality-adaptive Protocols (ICNP), Vancouver, BC, Canada, 2011, pp. 341352.
video streaming over multiple heterogeneous access networks, in Proc. [77] T. Henderson, S. Floyd, A. Gurtov, and Y. Nishida, The NewReno
2nd Annu. ACM Conf. Multimedia Syst., Santa Clara, CA, USA, 2011, modification to TCPs fast recovery algorithm, IETF, Fremont, CA,
pp. 5768. USA, RFC 6582, 2012.
[78] B. Hesmans, F. Duchene, C. Paasch, G. Detal, and O. Bonaventure, [102] R. Khalili, N. Gast, M. Popovic, U. Upadhyay, and J.-Y. Le Boudec,
Are TCP extensions middlebox-proof? in Proc. Workshop Hot Topics MPTCP is not Pareto-optimal: Performance issues and a possible solu-
Middleboxes Netw. Funct. Virtualization, Santa Barbara, CA, USA, tion, in Proc. 8th Int. Conf. Emerg. Netw. Exp. Technol. (CoNEXT),
2013, pp. 3742. Nice, France, 2012, pp. 112.
[79] M. Honda, Y. Nishida, L. Eggert, P. Sarolahti, and H. Tokuda, [103] H. A. Kim, B.-H. Oh, and J. Lee, Improvement of MPTCP per-
Multipath congestion control for shared bottleneck, in Proc. formance in heterogeneous network using packet scheduling mecha-
Workshop Protocols Fast Long-Distance Netw. (PFLDNeT), 2009, nism, in Proc. 18th Asia Pac. Conf. Commun. (APCC), Oct. 2012,
pp. 1924. pp. 842847.
[80] M. Honda et al., Is it still possible to extend TCP? in Proc. [104] K.-H. Kim and K. G. Shin, Improving TCP performance over wire-
ACM SIGCOMM Conf. Internet Meas. Conf., Berlin, Germany, 2011, less networks with collaborative multi-homed mobile hosts, in Proc.
pp. 181194. 3rd Int. Conf. Mobile Syst. Appl. Services, Seattle, WA, USA, 2005,
[81] H.-Y. Hsieh, K.-H. Kim, Y. Zhu, and R. Sivakumar, A receiver-centric pp. 107120.
transport protocol for mobile hosts with heterogeneous wireless inter- [105] K.-H. Kim and K. G. Shin, PRISM: Improving the performance of
faces, in Proc. 9th Annu. Int. Conf. Mobile Comput. Netw. (MobiCom), inverse-multiplexed TCP in wireless networks, IEEE Trans. Mobile
San Diego, CA, USA, 2003, pp. 115. Comput., vol. 6, no. 12, pp. 12971312, Dec. 2007.
[82] H.-Y. Hsieh and R. Sivakumar, A transport layer approach for achiev- [106] K.-H. Kim, Y. Zhu, R. Sivakumar, and H.-Y. Hsieh, A receiver-
ing aggregate bandwidths on multi-homed mobile hosts, in Proc. 8th centric transport protocol for mobile hosts with heterogeneous wireless
Annu. Int. Conf. Mobile Comput. Netw. (MobiCom), Atlanta, GA, USA, interfaces, Wireless Netw., vol. 11, no. 4, pp. 363382, 2005.
2002, pp. 8394. [107] R. Koetter and M. Mdard, An algebraic approach to network coding,
[83] H.-Y. Hsieh and R. Sivakumar, pTCP: An end-to-end transport layer IEEE/ACM Trans. Netw., vol. 11, no. 5, pp. 782795, Oct. 2003.
protocol for striped connections, in Proc. 10th IEEE Int. Conf. Netw. [108] M. Kun, Y. Jingdong, and R. Zhi, The research and simulation
Protocols, Paris, France, 2002, pp. 2433. of multipath-OLSR for mobile ad hoc network, in Proc. IEEE Int.
[84] H.-Y. Hsieh and R. Sivakumar, A transport layer approach for achiev- Symp. Commun. Inf. Technol. (ISCIT), vol. 1. Beijing, China, 2005,
ing aggregate bandwidths on multi-homed mobile hosts, Wireless pp. 540543.
Netw., vol. 11, nos. 12, pp. 99114, 2005. [109] K.-C. Lan and C.-Y. Li, Improving TCP performance over an
[85] C.-M. Huang and C.-H. Tsai, WiMP-SCTP: Multi-path transmission on-board multi-homed network, in Proc. IEEE Wireless Commun.
using stream control transmission protocol (SCTP) in wireless net- Netw. Conf. (WCNC), Shanghai, China, 2012, pp. 29612966.
works, in Proc. 21st IEEE Int. Conf. Adv. Inf. Netw. Appl. Workshops [110] T.-A. Le and L. X. Bui, Forward delay-based packet scheduling
(AINAW), vol. 1. Niagara Falls, ON, Canada, 2007, pp. 209214. algorithm for multipath TCP, arXiv preprint arXiv:1501.03196, 2015.
[86] X. Huang and Y. Fang, Multiconstrained QoS multipath routing in [111] Y. Lee, I. Park, and Y. Choi, Improving TCP performance in mul-
wireless sensor networks, Wireless Netw., vol. 14, no. 4, pp. 465478, tipath packet forwarding networks, J. Commun. Netw., vol. 4, no. 2,
2008. pp. 148157, Jun. 2002.
[87] C. Huitema, Multi-homed TCP, IETF Internet-Draft (Expired), [112] W. Lei, W. Zhang, and S. Liu, A framework of multipath
May 1995. [Online]. Available: https://tools.ietf.org/html/ transport system based on application-level relay (MPTS-AR),
draft-huitema-multi-homed-01 IETF Internet-Draft (Experimental), Jul. 2015. [Online]. Available:
[88] IEEE. (2000). IEEE 802.ad Link Aggregation Task Force. [Online]. https://tools.ietf.org/html/draft-leiwm-tsvwg-mpts-ar-04
Available: http://www.ieee802.org/3/ad/ [113] L. Kleinrock, An early history of the Internet, IEEE Commun. Mag.,
vol. 48, no. 8, pp. 2636, Aug. 2010.
[89] J. Iyengar, K. Shah, P. Amer, and R. Stewart, Concurrent multipath
[114] M. Li, A. Lukyanenko, and Y. Cui, Network coding based multipath
transfer using SCTP multihoming, in Proc. Int. Symp. Perform. Eval.
TCP, in Proc. IEEE Conf. Comput. Commun. Workshops (INFOCOM
Comput. Telecommun. Syst. (SPECTS), 2004.
Workshop), Orlando, FL, USA, Mar. 2012, pp. 2530.
[90] J. R. Iyengar, P. D. Amer, and R. Stewart, Retransmission policies for
[115] M. Li, A. Lukyanenko, S. Tarkoma, Y. Cui, and A. Yl-Jski,
concurrent multipath transfer using SCTP multihoming, in Proc. 12th
Tolerating path heterogeneity in multipath TCP with bounded receive
IEEE Int. Conf. Netw. (ICON), vol. 2. Singapore, 2004, pp. 713719.
buffers, Comput. Netw., vol. 64, pp. 114, May 2014.
[91] J. R. Iyengar, P. D. Amer, and R. Stewart, Concurrent multipath [116] M. Li, A. Lukyanenko, S. Tarkoma, and A. Yl-Jski, The
transfer using SCTP multihoming over independent end-to-end paths, delayed ACK evolution in MPTCP, in Proc. IEEE Glob. Commun.
IEEE/ACM Trans. Netw., vol. 14, no. 5, pp. 951964, Oct. 2006. Conf. (GLOBECOM), Atlanta, GA, USA, 2013, pp. 22822288.
[92] J. Lee et al., Applied techniques for high bandwidth data transfers [117] M. Li, A. Lukyanenko, S. Tarkoma, and A. Yl-Jski, MPTCP incast
across wide area networks, in Proc. Int. Conf. Comput. High Energy in data center networks, China Commun., vol. 11, no. 4, pp. 2537,
Nuclear Phys., Beijing, China, 2001. Apr. 2014.
[93] P. Jokela, R. Moskowitz, and P. Nikander, Using the encapsulating [118] J. Liao, J. Wang, and X. Zhu, cmpSCTP: An extension of SCTP
security payload (ESP) transport format with the host identity protocol to support concurrent multi-path transfer, in Proc. IEEE Int. Conf.
(HIP), IETF, Fremont, CA, USA, RFC 5202, Apr. 2008. Commun. (ICC), Beijing, China, 2008, pp. 57625766.
[94] Juniper. (2014). Aggregated Ethernet. [Online]. Available: [119] Y.-S. Lim, Y.-C. Chen, E. M. Nahum, D. Towsley, and K.-W. Lee,
http://www.juniper.net/documentation/en_US/junos14.1/topics/task/ Cross-layer path management in multi-path transport protocol for
configuration/link-aggregation-cli.html mobile devices, in Proc. Annu. Joint Conf. IEEE Comput. Commun.
[95] S. Kandula, K. C.-J. Lin, T. Badirkhanli, and D. Katabi, FatVAP: Soc. (INFOCOM), Toronto, ON, Canada, 2014, pp. 18151823.
Aggregating AP backhaul capacity to maximize throughput, in [120] J. Liu, H. Zou, J. Dou, and Y. Gao, Rethinking retransmission pol-
Proc. 5th USENIX Symp. Netw. Syst. Design Implement., vol. 8. icy in concurrent multipath transfer, in Proc. Int. Conf. Intell. Inf.
San Francisco, CA, USA, 2008, pp. 89104. Hiding Multimedia Signal Process. (IIHMSP), Harbin, China, 2008,
[96] D. Kaspar, K. Evensen, P. Engelstad, and A. F. Hansen, Using pp. 10051008.
HTTP pipelining to improve progressive download over multiple het- [121] L. Magalhaes and R. Kravets, Transport level mechanisms for band-
erogeneous interfaces, in Proc. IEEE Int. Conf. Commun. (ICC), width aggregation on mobile hosts, in Proc. 9th IEEE Int. Conf. Netw.
Cape Town, South Africa, 2010, pp. 15. Protocols, Riverside, CA, USA, 2001, pp. 165171.
[97] D. Kaspar et al., Enhancing video-on-demand playout over multiple [122] K. Manousakis and D. Famolari, INTELiCON: A framework for the
heterogeneous access networks, in Proc. 7th IEEE Consum. Commun. simultaneous utilization of multiple interfaces and its application on
Netw. Conf. (CCNC), Las Vegas, NV, USA, 2010, pp. 15. TCP, in Proc. Int. Wireless Commun. Mobile Comput. Conf. (IWCMC),
[98] V. Kawadia and P. R. Kumar, A cautionary perspective on cross-layer Aug. 2008, pp. 976981.
design, IEEE Wireless Commun., vol. 12, no. 1, pp. 311, Feb. 2005. [123] S. Mao, D. Bushmitch, S. Narayanan, and S. S. Panwar, MRTP:
[99] S. Keshav, A control-theoretic approach to flow control, ACM A multiflow real-time transport protocol for ad hoc networks, IEEE
SIGCOMM Comput. Commun. Rev., vol. 25, no. 1, pp. 188201, 1995. Trans. Multimedia, vol. 8, no. 2, pp. 356369, Apr. 2006.
[100] P. Key, L. Massouli, and D. Towsley, Combining multipath routing [124] S. Mascolo, L. A. Grieco, R. Ferorelli, P. Camarda, and G. Piscitelli,
and congestion control for robustness, in Proc. 40th Annu. Conf. Inf. Performance evaluation of Westwood+ TCP congestion control,
Sci. Syst., Princeton, NJ, USA, 2006, pp. 345350. Perform. Eval., vol. 55, nos. 12, pp. 93111, 2004.
[101] R. Khalili, N. Gast, M. Popovic, and J.-Y. Le Boudec, MPTCP [125] M. Mathis, J. Mahdavi, S. Floyd, and A. Romanow, TCP selective
is not Pareto-optimal: Performance issues and a possible solution, acknowledgment options, IETF RFC 2018, Fremont, CA, USA, Oct.
IEEE/ACM Trans. Netw., vol. 21, no. 5, pp. 16511665, Oct. 2013. 1996. [Online]. Available: https://tools.ietf.org/html/rfc2018
[126] N. F. Maxemchuk, Dispersity routing in store-and-forward net- [149] D. S. Phatak, T. Goff, and J. Plusquellic, IP-in-IP tunneling to
works, Ph.D. dissertation, Univ. Pennsylvania, Philadelphia, enable the simultaneous use of multiple IP interfaces for network level
PA, USA, 1975. [Online]. Available: http://repository.upenn.edu/ connection striping, Comput. Netw., vol. 43, no. 6, pp. 787804, 2003.
dissertations/AAI7524101 [150] S. Pierrel, P. Jokela, and J. Melen, Simultaneous multi-access exten-
[127] M. Menth, A. Stockmayer, and M. Schmidt, LISP hybrid access, sion to the host identity protocol, draft-pierrel-hip-sima-00 (Work
draft-menth-lisp-ha-00 (Experimental), Internet-Draft, IETF, Jul. 2015. in Process), Internet-Draft, IETF, Jun. 2006. [Online]. Available:
[128] J. Milbrandt, K. Humm, and M. Menth, Adaptive bandwidth allo- https://tools.ietf.org/html/draft-pierrel-hip-sima-00
cation: Impact of routing and load balancing on tunnel capacity [151] T. Polishchuk and A. Gurtov, Improving TCP-friendliness and fairness
requirements, in Proc. Next Gener. Internet Design Eng. (NGI), for mHIP, Infocommun. J., vol. 3, no. 1, pp. 2634, 2011.
Valencia, Spain, 2006, pp. 318325. [152] J. Postel and J. Reynolds, File transfer protocol, IETF, Fremont, CA,
[129] F. H. Mirani, N. Boukhatem, and M. A. Tran, A data-scheduling USA, RFC 959, Oct. 1985.
mechanism for multi-homed mobile terminals with disparate link laten- [153] S. Prabhavat, H. Nishiyama, N. Ansari, and N. Kato, On load distribu-
cies, in Proc. 72nd IEEE Veh. Technol. Conf. Fall (VTC), Ottawa, ON, tion over multipath networks, IEEE Commun. Surveys Tuts., vol. 14,
Canada, 2010, pp. 15. no. 3, pp. 662680, 3rd Quart. 2012.
[130] E. Miyazaki and M. Oguchi, Evaluation of middleware for bandwidth [154] J. Qadir, A. Ali, K.-L. A. Yau, A. Sathiaseelan, and J. Crowcroft,
aggregation using multiple interface in wireless communication, Int. Exploiting the power of multiplicity: A holistic survey of network-
J. Adv. Netw. Services, vol. 4, nos. 34, pp. 343352, 2012. layer multipath, IEEE Commun. Surveys Tuts., vol. 17, no. 4,
[131] R. Moskowitz, P. Nikander, P. Jokela, and T. Henderson, Host identity pp. 21762213, 4th Quart. 2015.
protocol, IETF, Fremont, CA, USA, RFC 5201, Apr. 2008. [155] A. Qureshi, J. Carlisle, and J. Guttag, Tavarua: Video streaming with
[132] D. Nagle, D. Serenyi, and A. Matthews, The Panasas ActiveScale WWAN striping, in Proc. 14th Annu. ACM Int. Conf. Multimedia,
storage cluster: Delivering scalable high bandwidth storage, in Proc. Santa Barbara, CA, USA, 2006, pp. 327336.
ACM/IEEE Conf. Supercomput., Pittsburgh, PA, USA, 2004, p. 53. [156] M. Radi, B. Dezfouli, K. A. Bakar, and M. Lee, Multipath routing
[133] S. C. Nguyen and T. M. T. Nguyen, Evaluation of multipath TCP in wireless sensor networks: Survey and research challenges, Sensors,
load sharing with coupled congestion control option in heterogeneous vol. 12, no. 1, pp. 650685, 2012.
networks, in Proc. Glob. Inf. Infrastruct. Symp. (GIIS), 2011, pp. 15. [157] C. Raiciu et al., Improving datacenter performance and robustness
[134] S. C. Nguyen, X. Zhang, T. M. T. Nguyen, and G. Pujolle, Evaluation with multipath TCP, in Proc. ACM SIGCOMM Conf., vol. 41. Toronto,
of throughput optimization and load sharing of multipath TCP in het- ON, Canada, 2011, pp. 266277.
erogeneous networks, in Proc. 8th Int. Conf. Wireless Opt. Commun. [158] C. Raiciu, M. Handley, and D. Wischik, Coupled multipath-aware con-
Netw. (WOCN), Paris, France, 2011, pp. 15. gestion control, IETF Internet-Draft, Oct. 2009. [Online]. Available:
[135] T. Nguyen-Duc et al., Investigating the performance of link aggrega- https://tools.ietf.org/html/draft-raiciu-mptcp-congestion-00
tion on OpenFlow switches, in Testbeds and Research Infrastructure: [159] C. Raiciu, M. Handly, and D. Wischik, Coupled congestion control for
Development of Networks and Communities. Cham, Switzerland: multipath transport protocols, IETF, Fremont, CA, USA, RFC 6356,
Springer, 2014, pp. 194202. Oct. 2011.
[136] C. Nicutar, C. Paasch, M. Bagnulo, and C. Raiciu, Evolving the [160] C. Raiciu et al., How hard can it be? Designing and implement-
Internet with connection acrobatics, in Proc. Workshop Hot Topics ing a deployable multipath TCP, in Proc. 9th USENIX Conf. Netw.
Middleboxes Netw. Funct. Virtualization, Santa Barbara, CA, USA, Syst. Design Implement. (NSDI), vol. 12. San Jose, CA, USA, 2012,
2013, pp. 712. pp. 399412.
[161] C. Raiciu et al., Data center networking with multipath TCP, in Proc.
[137] E. Nordmark and M. Bagnulo, Shim6: Level 3 multihoming shim
9th ACM SIGCOMM Workshop Hot Topics Netw., Monterey, CA, USA,
protocol for IPv6, IETF, Fremont, CA, USA, RFC 5533, Jun. 2009.
2010, p. 10.
[Online]. Available: http://www.hjp.at/doc/rfc/rfc5533.html/
[162] C. Raiciu, D. Wischik, and M. Handley, Practical congestion con-
[138] OVH Company. (2015). Overthebox. [Online]. Available:
trol for multipath transport protocols, Univ. College London, London,
https://www.ovhtelecom.fr/overthebox
U.K., Tech. Rep., 2009.
[139] C. Paasch, S. Ferlin, O. Alay, and O. Bonaventure, Experimental [163] A. L. Ramaboli, O. E. Falowo, and A. H. Chan, Bandwidth aggre-
evaluation of multipath TCP schedulers, in Proc. ACM SIGCOMM gation in heterogeneous wireless networks: A survey of current
Workshop Capacity Sharing Workshop, Chicago, IL, USA, 2014, approaches and issues, J. Netw. Comput. Appl., vol. 35, no. 6,
pp. 2732. pp. 16741690, 2012.
[140] C. Paasch, R. Khalili, and O. Bonaventure, On the benefits of applying [164] K. Ramakrishnan, S. Floyd, and D. Black, The addition of explicit
experimental design to improve multipath TCP, in Proc. 9th ACM congestion notification (ECN) to IP, IETF, Fremont, CA, USA,
Conf. Emerg. Netw. Exp. Technol. (CoNEXT), Santa Barbara, CA, USA, RFC 3168, Sep. 2001.
2013, pp. 393398. [165] P. Rodriguez and E. W. Biersack, Dynamic parallel access to repli-
[141] A. Pathak, H. Pucha, Y. Zhang, Y. C. Hu, and Z. M. Mao, A mea- cated content in the Internet, IEEE/ACM Trans. Netw., vol. 10, no. 4,
surement study of Internet delay asymmetry, in Passive and Active pp. 455465, Aug. 2002.
Network Measurement (LNCS 4979). Heidelberg, Germany: Springer, [166] P. Rodriguez, R. Chakravorty, J. Chesterfield, I. Pratt, and S. Banerjee,
2008, pp. 182191. MAR: A commuter router infrastructure for the mobile Internet, in
[142] Q. Peng, A. Walid, and S. H. Low, Multipath TCP algorithms: Theory Proc. 2nd Int. Conf. Mobile Syst. Appl. Services, Boston, MA, USA,
and design, ACM SIGMETRICS Perform. Eval. Rev., vol. 41, no. 1, 2004, pp. 217230.
pp. 305316, 2013. [167] K. Rojviboonchai and A. Hitoshi, An evaluation of multi-path
[143] Q. Peng, A. Walid, J. Hwang, and S. H. Low, Multipath TCP: transmission control protocol (M/TCP) with robust acknowledgement
Analysis, design and implementation, IEEE/ACM Trans. Netw., schemes, IEICE Trans. Commun., vol. 87, no. 9, pp. 26992707, 2004.
vol. 24, no. 1, pp. 596609, Feb. 2016. [Online]. Available: [168] K. Rojviboonchai, T. Osuga, and H. Aida, R-M/TCP: Protocol for
http://arxiv.org/abs/1308.3119 reliable multi-path transport over the Internet, in Proc. 19th Int. Conf.
[144] C. Perkins, IP encapsulation within IP, IETF, Fremont, CA, USA, Adv. Inf. Netw. Appl. (AINA), vol. 1. Taipei, Taiwan, 2005, pp. 801806.
RFC 2003, Oct. 1996. [169] K. Rojviboonchai, N. Watanabe, and H. Aida, One-way-trip
[145] C. Perkins, IP mobility support for IPv4, revised, IETF, Fremont, time (OWTT) measurement and retransmission policy for congestion
CA, USA, RFC 5944, Nov. 2010. control in M/TCP, in Proc. Annu. Conf. IPSJ, Okinawa, Japan, 2002.
[146] C. E. Perkins, Mobile IP, IEEE Commun. Mag., vol. 35, no. 5, [170] G. Rossini and D. Rossi, Evaluating CCN multi-path interest for-
pp. 8499, May 1997. warding strategies, Comput. Commun., vol. 36, no. 7, pp. 771778,
[147] A. Phanishayee et al., Measurement and analysis of TCP throughput 2013.
collapse in cluster-based storage systems, in Proc. 6th USENIX Conf. [171] H. Sakakibara, M. Saito, and H. Tokuda, Design and implementation
File Stor. Technol. (FAST), vol. 8. Berkeley, CA, USA, 2008, pp. 114. of a socket-level bandwidth aggregation mechanism for wireless net-
[148] D. S. Phatak and T. Goff, A novel mechanism for data streaming works, in Proc. 2nd Annu. Int. Workshop Wireless Internet, Boston,
across multiple IP links for improving throughput and reliability in MA, USA, 2006, p. 11.
mobile environments, in Proc. 21st Annu. Joint Conf. IEEE Comput. [172] D. Sarkar, A concurrent multipath TCP and its Markov model, in
Commun. Soc. (INFOCOM), vol. 2. New York, NY, USA, 2002, Proc. IEEE Int. Conf. Commun. (ICC), Istanbul, Turkey, Jun. 2006,
pp. 773781. pp. 615620.
[173] H. Schulzrinne, S. Casner, R. Frederick, and V. Jacobson, RTP: A [196] B. Wang, W. Wei, Z. Guo, and D. Towsley, Multipath live streaming
transport protocol for real-time applications, IETF, Fremont, CA, via TCP: Scheme, performance and benefits, ACM Trans. Multimedia
USA, RFC 3550, Jul. 2003. Comput. Commun. Appl., vol. 5, no. 3, p. 25, 2009.
[174] K. Sha, J. Gehlot, and R. Greve, Multipath routing techniques in wire- [197] F. Wang, Z. M. Mao, J. Wang, L. Gao, and R. Bush, A measurement
less sensor networks: A survey, Wireless Pers. Commun., vol. 70, no. 2, study on the impact of routing events on end-to-end Internet path per-
pp. 807829, May 2013. formance, in Proc. Conf. Appl. Technol. Architect. Protocols Comput.
[175] S. Shailendra, R. Bhattacharjee, and S. K. Bose, Improving congestion Commun. (SIGCOMM), Pisa, Italy, 2006, pp. 375386.
control for concurrent multipath transfer through bandwidth estimation [198] X. Wang, Z. Feng, D. Fan, Y. Xue, and V. Le, A segment-based
based resource pooling, in Proc. 8th Int. Conf. Inf. Commun. Signal adaptive joint session scheduling mechanism in heterogeneous wire-
Process. (ICICS), Singapore, 2011, pp. 15. less networks, in Proc. 70th IEEE Veh. Technol. Conf. Fall (VTC),
[176] S. Shakkottai, E. Altman, and A. Kumar, The case for non-cooperative Anchorage, AK, USA, 2009, pp. 15.
multihoming of users to access points in IEEE 802.11 WLANs, in [199] Y. Wen and V. W. S. Chan, Ultra-reliable communication over vul-
Proc. 25th IEEE Int. Conf. Comput. Commun. (INFOCOM), Barcelona, nerable all-optical networks via lightpath diversity, IEEE J. Sel. Areas
Spain, 2006, pp. 112. Commun., vol. 23, no. 8, pp. 15721587, Aug. 2005.
[177] Z. U. Shamszaman, S. S. Ara, and I. Chong, Feasibility considerations [200] D. Wischik, M. Handley, and M. B. Braun, The resource pooling
of multipath TCP in dealing with big data application, in Proc. Int. principle, ACM SIGCOMM Comput. Commun. Rev., vol. 38, no. 5,
Conf. Inf. Netw. (ICOIN), Bangkok, Thailand, Jan. 2013, pp. 708713. pp. 4752, 2008.
[178] V. Sharma, S. Kalyanaraman, K. Kar, K. Ramakrishnan, and [201] F. Yang and P. Amer, Non-renegable selective acknowledgments
V. Subramanian, MPLOT: A transport protocol exploiting multipath (NR-SACKs) for MPTCP, in Proc. 27th Int. Conf. Adv. Inf. Netw. Appl.
diversity using erasure codes, in Proc. 27th IEEE Conf. Comput. Workshops (WAINA), Barcelona, Spain, Mar. 2013, pp. 11131118.
Commun. (INFOCOM), Phoenix, AZ, USA, 2008, pp. 121125. [202] F. Yang and P. Amer, Work in progress: Using one-way commu-
[179] V. Sharma, K. Kar, K. K. Ramakrishnan, and S. Kalyanaraman, A nication delay for in-order arrival MPTCP scheduling, in Proc. 9th
transport protocol to exploit multipath diversity in wireless networks, Int. Conf. Commun. Netw. China (CHINACOM), Maoming, China,
IEEE/ACM Trans. Netw., vol. 20, no. 4, pp. 10241039, Aug. 2012. Aug. 2014, pp. 122125.
[180] H. Shojania and B. Li, Parallelized progressive network coding [203] M. Zhang, B. Karp, S. Floyd, and L. Peterson, RR-TCP: A reordering-
with hardware acceleration, in Proc. 15th IEEE Int. Workshop Qual. robust TCP with DSACK, in Proc. 11th IEEE Int. Conf. Netw.
Service, Evanston, IL, USA, 2007, pp. 4755. Protocols, Atlanta, GA, USA, 2003, pp. 95106.
[181] W. Simpson, The point-to-point protocol (PPP), IETF, Fremont, CA, [204] M. Zhang, J. Lai, A. Krishnamurthy, L. L. Peterson, and R. Y. Wang,
USA, RFC 1661, Jul. 1994. A transport layer approach for improving end-to-end performance and
[182] A. Singh, M. Xiang, A. Konsgen, C. Goerg, and Y. Zaki, robustness using redundant paths, in Proc. Gen. Track USENIX Annu.
Enhancing fairness and congestion control in multipath TCP, in Tech. Conf., Boston, MA, USA, 2004, pp. 99112.
Proc. 6th Joint IFIP Wireless Mobile Netw. Conf. (WMNC), Dubai, [205] W. Zhang, W. Lei, S. Liu, and G. Li, A general framework of
United Arab Emirates, 2013, pp. 18. multipath transport system based on application-level relay, Comput.
Commun., vol. 51, pp. 7080, Sep. 2014.
[183] S. K. Singh, T. Das, and A. Jukan, A survey on Internet multipath
[206] D. Zhou, W. Song, and M. Shi, Goodput improvement for multipath
routing and provisioning, IEEE Commun. Surveys Tuts., vol. 17, no. 4,
TCP by congestion window adaptation in multi-radio devices, in Proc.
pp. 21572175, 4th Quart. 2015.
IEEE Consum. Commun. Netw. Conf. (CCNC), Las Vegas, NV, USA,
[184] H. Sivakumar, S. Bailey, and R. L. Grossman, PSockets: The case
2013, pp. 508514.
for application-level network striping for data intensive applications
using high speed wide area networks, in Proc. ACM/IEEE Conf.
Supercomput. (CDROM), Dallas, TX, USA, 2000, Art. no. 37.
[185] K. Sklower, B. Lloyd, G. McGregor, D. Carr, and T. Coradetti, The
PPP multilink protocol (MP), IETF, Fremont, CA, USA, RFC 1990,
Aug. 1996.
[186] A. C. Snoeren, Adaptive inverse multiplexing for wide-area wireless
networks, in Proc. Glob. Telecommun. Conf. (GLOBECOM), vol. 3.
Ming Li received the M.S. degree in computer
Rio de Janeiro, Brazil, 1999, pp. 16651672.
science from the Beijing University of Posts and
[187] R. Stewart, Stream control transmission protocol, IETF, Fremont,
Communications, China, in 2008, and the Ph.D.
CA, USA, RFC 4960, Sep. 2007.
degree in computer science and engineering from
[188] T. N. Subedi, K. K. Nguyen, and M. Cheriet, OpenFlow-based Aalto University, Finland. His research interests
in-network layer-2 adaptive multipath aggregation in data centers, include network security, network architecture, and
Comput. Commun., vol. 61, pp. 5869, May 2015. network protocols.
[189] J. Sun, Y. Wen, and L. Zheng, On file-based content distribution over
wireless networks via multiple paths: Coding and delay trade-off, in
Proc. IEEE INFOCOM, Shanghai, China, 2011, pp. 381385.
[190] A. S.-W. Tam, K. Xi, Y. Xu, and H. J. Chao, Preventing TCP incast
throughput collapse at the initiation, continuation, and termination,
in Proc. 20th IEEE Int. Workshop Qual. Service, Coimbra, Portugal,
2012, pp. 129.
[191] C.-L. Tsao and R. Sivakumar, On effectively exploiting multiple Andrey Lukyanenko received the M.Sc. degrees
wireless interfaces in mobile hosts, in Proc. 5th Int. Conf. in computer science from the University of Kuopio
Emerg. Netw. Exp. Technol. (CoNEXT), Rome, Italy, 2009, in 2005, the M.Sc. degree in mathematics from
pp. 337348. Petrozavodsk State University in 2005, and the Ph.D.
[192] S. Tullimas, T. Nguyen, R. Edgecomb, and S.-C. Cheung, Multimedia degree in computer science from the University
streaming using multiple TCP connections, ACM Trans. Multimedia of Helsinki in 2010. From 2006 to 2010, he
Comput. Commun. Appl., vol. 4, no. 2, p. 12, 2008. was a Researcher in the networking group with
[193] I. Van Beijnum, J. Crowcroft, F. Valera, and M. Bagnulo, Loop- the Helsinki Institute for Information Technology.
freeness in multipath BGP through propagating the longest path, in Since 2010, he has been a Post-Doctoral Researcher
Proc. IEEE Int. Conf. Commun. Workshops (ICC), Dresden, Germany, with Aalto University, with the Datacommunication
2009, pp. 16. Software Group. In 2013, he was on a research
[194] R. van der Pol et al., Experiences with MPTCP in an intercontinen- visit to ICSI/UC Berkeley. He has worked on problems related to backoff
tal OpenFlow network, in Proc. 29th TERENA Netw. Conf. (TNC), protocols in IEEE802.11, security with host identity protocol, problems of
Maastricht, The Netherlands, 2013, pp. 16171624. denial-of-service attacks, and reputations in peer-to-peer networks. During
[195] B. Wang, W. Wei, Z. Guo, and D. Towsley, Multipath live streaming his research, he actively uses game theory, queuing theory, and data analy-
via TCP: Scheme, performance and benefits, in Proc. 3rd ACM Int. sis methods. His research interests include information centric networking,
Conf. Emerg. Netw. Exp. Technol. (CoNEXT), New York, NY, USA, datacenter architecture, in-network caching mechanisms, and future Internet
2007, Art. no. 11. technologies.
Zhonghong Ou received the Ph.D. degree from Sasu Tarkoma received the Ph.D. degree in com-
the University of Oulu, Finland. Since 2016, he puter science from the University of Helsinki. He
has been an Associate Professor with the School has managed and participated in national and inter-
of Computer Science, Beijing University of Posts national research projects with the University of
and Telecommunications, China. From 2010 to Helsinki, the Helsinki University of Technology, and
2015, he was a Post-Doctoral Researcher with the Helsinki Institute for Information Technology.
Aalto University, Finland. In 2010, he was a He has lectured several courses on middleware and
Visiting Scholar with Internet Real-Time Laboratory, data communications. His research interests include
Columbia University, USA. In 2013, he was a middleware and distributed computing.
Visiting Scholar with Intel Labs, USA. He has a
wide spectrum of research interests, including virtu-
alization and cloud computing platforms, cloud storage, energy optimization
for mobile platforms, and big data analytics.
Matthieu Coudron received the degree in telecom-
munications engineering from Telecom SudParis,
France, in 2012. He is currently pursuing the Ph.D.
degree with University Pierre and Marie Curie (Paris
VI, Sorbonne Universites), France. He works on
Internet multipath forwarding.
Stefano Secci received the Laurea degree in

telecommunications engineering from Politecnico di
Milano, in 2005, and a dual Ph.D. degrees in com-
Antti Yl-Jski was with Nokia from 1994 to puter networks from Politecnico di Milano and
2004 in several research and research manage- Telecom ParisTech, France, in 2009. He has been
ment positions with focus on future Internet, mobile an Associate Professor with the University Pierre
networks, applications, services and service archi- and Marie Curie (Paris VI, Sorbonne Universites),
tectures. He is the Vice Head of the Department France, since 2010. He also worked as a Research
Computer Science, Aalto University. He has super- Fellow with NTNU, George Mason University,
vised 231 masters thesis and 21 doctoral disserta- Ecole Polytechnique de Montreal, and Politecnico
tions during his professorship in Aalto University di Milano, and as a Network Engineer with Fastweb
from 2002 to 2015. He has published over 100 Italia. He has been the current Chair of the Internet Technical Committee,
peer-reviewed articles and holds several international joint between the IEEE Communication Society and the Internet Society, since
patents. His current research interest is focused on 2013. His work spans network optimization, protocol design, Internet rout-
mobile and cloud computing, crowdsourcing, and energy efficient computing ing, and traffic engineering. His current research interests are about Internet
and communications. resiliency and cloud networking.

Multipath Transmission For The Internet: A Survey

Uploaded by

Copyright:

Available Formats

Multipath Transmission For The Internet: A Survey

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Multipath Transmission For The Internet: A Survey

Uploaded by

Copyright:

Available Formats

IEEE COMMUNICATIONS SURVEYS & TUTORIALS, VOL. 18, NO.

4, FOURTH QUARTER 2016 2887

Multipath Transmission for the Internet: A Survey

Fig. 1. Structure of Section III and research problems (P) to address.

mentioned in Table IV. Note that the algorithms in Table III

the different domains involved are controlled by different

B. IP Layer Bandwidth Aggregation

representative of the bandwidth-delay product of the subflow

paths according to the cwnd/RTT of each path. cmpSCTP dis-

Signal (DSS) option [62] specifies a full mapping from the

Therefore, algorithm is not the primary factor that determines

In the rest of this section, we also divide the approaches

Stefano Secci received the Laurea degree in

You might also like