Internet Indirection Infrastructure

Ion Stoica Daniel Adkins Shelley Zhuang Scott Shenker Sonesh Surana


University of California, Berkeley
istoica, dadkins, shelleyz, sonesh  @cs.berkeley.edu

ABSTRACT to the (fixed) location of the desired IP address. The simplicity of

Attempts to generalize the Internet’s point-to-point communication this point-to-point communication abstraction contributed greatly
abstraction to provide services like multicast, anycast, and mobility to the scalability and efficiency of the Internet.
have faced challenging technical problems and deployment barri- However, many applications would benefit from more general
ers. To ease the deployment of such services, this paper proposes communication abstractions, such as multicast, anycast, and host
an overlay-based Internet Indirection Infrastructure ( ) that offers mobility. In these abstractions, the sending host no longer knows
a rendezvous-based communication abstraction. Instead of explic- the identity of the receiving hosts (multicast and anycast) and the
itly sending a packet to a destination, each packet is associated with location of the receiving host need not be fixed (mobility). Thus,
an identifier; this identifier is then used by the receiver to obtain de- there is a significant and fundamental mismatch between the orig-
livery of the packet. This level of indirection decouples the act of inal point-to-point abstraction and these more general ones. All
sending from the act of receiving, and allows  to efficiently sup- attempts to implement these more general abstractions have relied
port a wide variety of fundamental communication services. To on a layer of indirection that decouples the sending hosts from the
demonstrate the feasibility of this approach, we have designed and receiving hosts; for example, senders send to a group address (mul-
built a prototype based on the Chord lookup protocol. ticast or anycast) or a home agent (mobility), and the IP layer of the
network is responsible for delivering the packet to the appropriate
location(s).
Categories and Subject Descriptors Although these more general abstractions would undoubtedly
H.4.3 [Information Systems]: Communication bring significant benefit to end-users, it remains unclear how to
achieve them. These abstractions have proven difficult to imple-
General Terms ment scalably at the IP layer [4, 13, 27]. Moreover, deploying ad-
ditional functionality at the IP layer requires a level of community-
Design wide consensus and commitment that is hard to achieve. In short,
implementing these more general abstractions at the IP layer poses
Keywords difficult technical problems and major deployment barriers.
Indirection, Abstraction, Scalable, Internet, Architecture In response, many researchers have turned to application-layer
solutions (either end-host or overlay mechanisms) to support these
abstractions [4, 15, 24]. While these proposals achieve the desired
1. INTRODUCTION functionality, they do so in a very disjointed fashion in that solu-
The original Internet architecture was designed to provide uni- tions for one service are not solutions for other services; e.g., pro-
cast point-to-point communication between fixed locations. In this posals for application-layer multicast don’t address mobility, and
basic service, the sending host knows the IP address of the receiver vice-versa. As a result, many similar and largely redundant mech-
and the job of IP routing and forwarding is simply to deliver packets anisms are required to achieve these various goals. In addition, if
overlay solutions are used, adding a new abstraction requires the
This research was sponsored by NSF under grant numbers Career deployment of an entirely new overlay infrastructure.
Award ANI-0133811, and ITR Award ANI-0085879. Views and
conclusions contained in this document are those of the authors In this paper, we propose a single new overlay network that
and should not be interpreted as representing the official policies, serves as a general-purpose Internet Indirection Infrastructure ( ).
either expressed or implied, of NSF, or the U.S. government.  offers a powerful and flexible rendezvous-based communication


ICSI Center for Internet Research (ICIR), Berkeley, abstraction; applications can easily implement a variety of commu-
[email protected] nication services, such as multicast, anycast, and mobility, on top
of this communication abstraction. Our approach provides a gen-
eral overlay service that avoids both the technical and deployment
challenges inherent in IP-layer solutions and the redundancy and
Permission to make digital or hard copies of all or part of this work for lack of synergy in more traditional application-layer approaches.
personal or classroom use is granted without fee provided that copies are We thus hope to combine the generality of IP-layer solutions with
not made or distributed for profit or commercial advantage and that copies the deployability of overlay solutions.
bear this notice and the full citation on the first page. To copy otherwise, to The paper is organized as follows. In Sections 2 and 3 we pro-
republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee.
vide an overview of the  architecture and then a general discus-
SIGCOMM ’02 Pittsburgh, Pennsylvania USA sion on how  might be used in applications. Section 4 covers ad-
Copyright 2002 ACM X-XXXXX-XX-X/XX/XX ...$5.00.
 
+ ,)
,) + ,)+ ,+ -)-)-)-
’s Application Programming Interface (API) ()*)
( *)
( *( ,+),)+ ,)+ ,+ .).).). /)0)
/ 0)
/ 0/
 
 
send packet
    ! sender (S) receiver (R) sender (S) (id, data) (R, data) receiver (R)
insert trigger (id, R)
"#%$&'   ! (id, R)
remove trigger
(a)
(b)
(c)

Figure 1: (a)  ’s API. Example illustrating communication between two nodes. (b) The receiver 1 inserts trigger 2 !3 45176 . (c) The
sender sends packet 2 !3 4398;:<8=6 .

ditional aspects of the design such as scalability and efficient rout- inserts the trigger 2 @3 4176 into the network. When a packet is sent
ing. Section 5 describes some simulation results on   performance to identifier !3 , the trigger causes it to be forwarded via IP to 1 .
along with a discussion on an initial implementation. Related work Thus, much as in IP multicast, the identifier !3 represents a log-
is discussed in Section 6, followed by a discussion on future work ical rendezvous between the sender’s packets and the receiver’s
Section 7. We conclude with a summary in Section 8. trigger. This level of indirection decouples the sender from the
receiver. The senders need neither be aware of the number of re-
2. >? OVERVIEW ceivers nor their location. Similarly, receivers need not be aware of
the number or location of senders.
In this section we present an overview of  . We start with the The above description is the simplest form of the abstraction.
basic service model and communication abstraction, then briefly We now describe a generalization that allows inexact matching be-
describe  ’s design. tween identifiers. (A second generalization that replaces identi-
fiers with a stack of identifiers is described in Section 2.5.) We as-
2.1 Service Model sume identifiers are B bits long and that there is some exact-match
The purpose of  is to provide indirection; that is, it decouples threshold D with DFEGB . We then say that an identifier @3=H in a
the act of sending from the act of receiving. The  service model trigger matches an identifier @3 in a packet if and only if
is simple: sources send packets to a logical identifier, and receivers
express interest in packets sent to an identifier. Delivery is best- (a) !3 and !3 H have a prefix match of at least D bits, and
effort like in today’s Internet, with no guarantees about packet de-
livery. (b) there is no trigger with an identifier that has a longer prefix
This service model is similar to that of IP multicast. The cru- match with @3 .
cial difference is that the  equivalent of an IP multicast join is
more flexible. IP multicast offers a receiver a binary decision of In other words, a trigger identifier @3=H matches a packet identi-
whether or not to receive packets sent to that group (this can be fier @3 if and only if @3;H is a longest prefix match (among all other
indicated on a per-source basis). It is up to the multicast infrastruc- trigger identifiers) and this prefix match is at least as long as the
ture to build efficient delivery trees. The  equivalent of a join is exact-match threshold D . The value D is chosen to be large enough
inserting a trigger. This operation is more flexible than an IP mul- so that the probability that two randomly chosen identifiers match
ticast join as it allows receivers to control the routing of the packet. is negligible.1 This allows end-hosts to choose the identifiers inde-
This provides two advantages. First, it allows them to create, at the pendently with negligible chance of collision.
application level, services such as mobility, anycast, and service 2.3 Overview of the Design
composition out of this basic service model. Thus, this one simple
service model can be used to support a wide variety of application- We now briefly describe the infrastructure that supports this ren-
level communication abstractions, alleviating the need for many dezvous communication abstraction; a more in-depth description
parallel and redundant overlay infrastructures. Second, the infras- follows in Section 4.  is an overlay network which consists of
tructure can give responsibility for efficient tree construction to the a set of servers that store triggers and forward packets (using IP)
end-hosts. This allows the infrastructure to remain simple, robust, between  nodes and to end-hosts. Identifiers and triggers have
and scalable. meaning only in this   overlay.
One of the main challenges in implementing   is to efficiently
2.2 Rendezvous-Based Communication match the identifiers in the packets to those in triggers. This is
done by mapping each identifier to a unique  node (server); at
The service model is instantiated as a rendezvous-based com-
any given time there is a single  node responsible for a given @3 .
munication abstraction. In their simplest form, packets are pairs
When a trigger 2 @3 458;3939C6 is inserted, it is stored on the  node
2 @3 453989:<8A6 where !3 is an B -bit identifier and 3989:8 consists of
responsible for @3 . When a packet is sent to @3 it is routed by  to
a payload (typically a normal IP packet payload). Receivers use
the node responsible for @3 ; there it is matched against any triggers
triggers to indicate their interest in packets. In the simplest form,
for that !3 and forwarded (using IP) to all hosts interested in packets
triggers are pairs 2 @3 458;3939C6 , where !3 represents the trigger iden-
sent to that identifier. To facilitate inexact matching, we require that
tifier, and 8;3939C represents a node’s address which consists of an
all !3 ’s that agree in the first D bits be stored on the same  server.
IP address and a port number. A trigger 2 @3 458;3939C'6 indicates that
The longest prefix match required for inexact matching can then be
all packets with an identifier @3 should be forwarded (at the IP
executed at a single node (where it can be done efficiently).
level) by the  infrastructure to the node identified by 8;3939C . More
Note that packets are not stored in  ; they are only forwarded. 
specifically, the rendezvous-based communication abstraction ex-
provides a best-effort service like today’s Internet.  implements
ports three basic primitives as shown in Figure 1(a).
neither reliability nor ordered delivery on top of IP. End-hosts use
Figure 1(b) illustrates the communication between two nodes, I
where receiver 1 wants to receive packets sent to @3 . The receiver In our implementation we choose BKJMLNO and DPJRQLS .
 

 
 



 
  (id, data)
receiver (R) 
 
 
  (id, data)
sender (S) (id, R) (R, data) sender (S) (id, R’) (R’, data)






receiver (R’)
(a) Mobility




 
 
 
 

receiver (R1) receiver (R1)

(id, R1) (R1, data)


 
 
(idp |ids1, R1)
 


 

  (id, data)
(id, R2) (R2, data)










 
  (idp |id s , data)
(idp |ids2, R2) (R2, data) 

 
 
sender (S) (R3, data) sender (S)
(id, R3) receiver (R2) (idp |ids3, R3) receiver (R2)


 
 
  
 
 
 




 



(b) Multicast receiver (R3) (c) Anycast receiver (R3)

Figure 2: Communication abstractions provided by  . (a) Mobility: The change of the receiver’s address from 1 to 1 is transparent 

to the sender. (b) Multicast: Every packet 2 !3 4398;:<8=6 is forwarded to each receiver 1 that inserts the trigger 2 @3 41 6 . (c) Anycast: 
! #"
The packet matches the trigger of receiver 17L . @3 @3 denotes an identifier of size B , where @3 represents the prefix of the D most $
"
significant bits, and !3 represents the suffix of the B &%
D least significant bits.

periodic refreshing to maintain their triggers in  . Hosts contact an any packet that matches @3 is forwarded to all members of the group
 node when sending  packets or inserting triggers. This  node as shown in Figure 2(b). We discuss how to make this approach
then forwards these packets or triggers to the  node responsible scalable in Section 3.4.
for the associated identifiers. Thus, hosts need only know one  Note that unlike IP multicast, with  there is no difference be-
node in order to use the   infrastructure. tween unicast or multicast packets, in either sending or receiving.
Such an interface gives maximum flexibility to the application. An
2.4 Communication Primitives Provided by >? application can switch on-the-fly from unicast to multicast by sim-
We now describe how  can be used by applications to achieve ply having more hosts maintain triggers with the same identifier.
the more general communication abstractions of mobility, multi- For example, in a telephony application this would allow multiple
cast, and anycast. parties to seamlessly join a two-party conversation. In contrast,
with IP, an application has to at least change the IP destination ad-
2.4.1 Mobility dress in order to switch from unicast to multicast.
The form of mobility addressed here is when a host (e.g., a lap-
top) is assigned a new address when it moves from one location to 2.4.3 Anycast
another. A mobile host that changes its address from 1 to 1 as a ' Anycast ensures that a packet is delivered to exactly one receiver
result of moving from one subnet to another can preserve the end- in a group, if any. Anycast enables server selection, a basic building
to-end connectivity by simply updating each of its existing triggers block for many of today’s applications. To achieve this with  , all

from 2 @3 4176 to 2 @3 41 6 , as shown in Figure 2(a). The sending host hosts in an anycast group maintain triggers which are identical in
needs not be aware of the mobile host’s current location or address. the D most significant bits. These D bits play the role of the anycast
Furthermore, since each packet is routed based on its identifier to group identifier. To send a packet to an anycast group, a sender uses
the server that stores its trigger, no additional operation needs to be an identifier whose D -bit prefix matches the anycast group identi-
invoked when the sender moves. Thus,  can maintain end-to-end fier. The packet is then delivered to the member of the group whose
connectivity even when both end-points move simultaneously. trigger identifier best matches the packet identifier according to the
longest prefix matching rule (see Figure 2(c)). Section 3.3 gives
(%
With any scheme that supports mobility, efficiency is a major
concern [25]. With   , applications can use two techniques to two examples of how end-hosts can use the last B D bits of the
achieve efficiency. First, the address of the server storing the trigger identifier to encode their preferences.
is cached at the sender, and thus subsequent packets are forwarded
directly to that server via IP. This way, most packets are forwarded 2.5 Stack of Identifiers
through only one   server in the overlay network. Second, to al- In this section, we describe a second generalization of  , which
leviate the triangle routing problem due to the trigger being stored replaces identifiers with identifier stacks. An identifier stack is a
at a server far away, end-hosts can use off-line heuristics to choose list of identifiers that takes the form 2 !3 I 4 @3 4 @3 '4 4 @3 6 where *) #+ -,.,-, */
triggers that are stored at   servers close to themselves (see Sec- #
@3 is either an identifier or an address. Packets and triggers : are 0
tion 4.5 for details). thus of the form:

2.4.2 Multicast 1 Packet 0 J 2 @3 H " 3254 /

453989:8=6
Creating a multicast group is equivalent to having all members of
the group register triggers with the same identifier !3 . As a result, 1 Trigger :J *" 32-4 /
2 @3 4 @3 <H 6
 0 0

 0 ,  
 recv( ) // upon receiving packet
HTML−WML transcoder (T)
@3 J "8;3 2 !3 :8 DA6 ; // get head of p’s stack

 
  
// is local server responsible for id’s best match?
if 2  8 <2 @36J FALSE 6
0
89:

+%,%+ ,%+ ,+
((T,id), data)
(R, data)
 forward( ); // matching trigger stored elsewhere
((id HTML−WML , id), data)

0 0 0 ,  
(id, R)
return; (id HTML−WML , T)
sender (S) receiver (R)

   

2 @3 :8 D=6 ; // pop id from p’s stack...



: :J : B 8;: 2 @36 ; // get all triggers matching id (a) Service composition

0 ,  

if 2 ": :J 6
if 2 @3 :8 DPJ 6
0
drop( ) // nowhere else to forward
MPEG−H.263 transcoder (T)

'%(%' (%' ('

else
0
((T, R1), data)
(R1, data)

  
 
  forward 2 6 ; (id MPEG−H.263 , T) receiver (R1)

  

while ( : : J ) // forward packet to each matching trigger
)%*%) *%) *) ((id MPEG−H.263 , R1), data)

$%&%$ &%$ &$

(id, data)

0  0 0
: J : :C  "C2 ": :6 ;
sender (S) (id, (id MPEG−H.263 , R1)) (R2, data)
Q J 2 6 ; // create new packet to send
receiver (R2)

0  0 ! 5,   0 ,  
// ... add t’s stack at head of p1’s stack (id, R2)

3 2: @3 :8 D 4 Q @3 :8 D=6 ;

0
AC
(b) Heterogeneous multicast
 forward( Q );

0 0 -

 0 ,  
 forward( ) // send/forward packet
.
Figure 4: (a) Service composition: The sender ( ) specifies that

 0 
@3 J "8;3 2 !3 :8 DA6 ; // get head of p’s stack packets should be transcoded at server before being delivered
if 2: 92 @36 J IP ADDR TYPE6
0
IP send(@3 4 ); // id is an IP address; send p to id via IP
to the destination (1 ). (b) Heterogeneous multicast: Receiver
1 Q specifies that wants to receive H.263 data, while 17L specifies
else
0
forward 2 6 ; // forward via overlay network 0 that wants to receive MPEG data. The sender sends MPEG
data.
Figure 3: Pseudo-code of the receiving and forward operations
executed by an  server. 0 . For each matching trigger , the identifier stack of the trigger is
prepended to 0 ’s identifier stack. The packet 0 is then forwarded
:

based on the first identifier in its stack.

The generalized form of packets allows a source to send a packet

to a series of identifiers, much as in source routing. The general- 3. USING > ?

ized form of triggers allows a trigger to send a packet to another In this section we present a few examples of how  can be used.
identifier rather than to an address. This extension allows for a We discuss service composition, heterogeneous multicast, server
selection, and large scale multicast. In the remainder of the paper,
0 0
much greater flexibility. To illustrate this point, in Sections 3.1, 3.2,
and 4.3, we discuss how identifier stacks can be used to provide we say that packet matches trigger : if the first identifier of ’s
service composition, implement heterogeneous multicast, and in- identifier stack matches : ’s identifier.
crease  ’s robustness, respectively.
0
A packet is always forwarded based on the first identifier @3 3.1 Service Composition
in its identifier stack until it reaches the server who is responsible Some applications may require third parties to process the data
0 0
) +
for storing the matching trigger(s) for . Consider a packet with before it reaches the destination [10]. An example is a wireless
an identifier stack 2 @3 I 4 @3 4 @3 6 . If there is no trigger in   whose application protocol (WAP) gateway translating HTML web pages
identifier matches @3 I , @3 I is popped from the stack. The process is to WML for wireless devices [35]. WML is a lightweight version of
0
repeated until an identifier in ’s identifier stack matches a trigger HTML designed to run on wireless devices with small screens and
0
.
: . If no such trigger is found, packet is dropped. If on the other limited capabilities. In this case, the server can forward the web
hand, there is a trigger : whose identifier matches @3 I , then !3 I is page to a third-party server that implements the HTML-WML

#"
 #"
 ) #+
replaced by : ’s identifier stack. In particular, if : ’s identifier stack transcoding, which in turn processes the data and sends it to the
0
is 2 4 =6 , then ’s identifier stack becomes 2 4 4 @3 '4 @3 "6 . If @3 I destination via WAP.
0
is an IP address, is sent via IP to that address, and the rest of In general, data might need to be transformed by a series of
0 ) #+
) +
’s identifier stack, i.e., 2 @3 4 @3 "6 is forwarded to the application. third-party servers before it reaches the destination. In today’s In-
The semantics of @3 and !3 are in general application-specific. ternet, the application needs to know the set of servers that per-
However, in this paper we consider only examples in which the form transcoding and then explicitly forward data packets via these
application is expected to use these identifiers to forward the packet servers.
after it has processed it. Thus, an application that receives a packet With  , this functionality can be easily implemented by using a
) #+
with identifier stack 2 @3 4 @3 6 is expected to send another packet stack of identifiers. Figure 4(a) shows how data packets containing
) #+
with the same identifier stack 2 !3 '4 @3 6 . As shown in the next HTML information can be redirected to the transcoder, and thus

/10 2435768293
section this allows   to provide support for service composition. arrive at the receiver containing WML information. The sender
Figure 3 shows the pseudo-code of the receiving and forward- associates with each data packet the stack 2 @3 4 @36 ,
ing operations executed by an  node. Upon receiving a packet , 0 where @3 represents the flow identifier. As a result, the data packet
a server first checks whether it is responsible for storing the trig- is routed first to the server which performs the transcoding. Next,
0
ger matching packet . If not, the server forwards the packet at the server inserts packet 2 @3 453989:8=6 into  , which delivers it to the
the  level. If yes, the code returns the set of triggers that match receiver.
3.2 Heterogeneous Multicast  
 
(idg, data) S
Figure 4(b) shows a more complex scenario in which an MPEG
video stream is played back by one H.263 receiver and one MPEG
receiver.
To provide this functionality, we use the ability of the receiver,
instead of the sender (see Section 2.5), to control the transforma- idg idg idg

2
 5 / ) +

tions performed on data packets. In particular, the H.263 receiver id1 id2 R1
1 Q inserts trigger 2 @3 4"2 @3 41 Q"656 , and the sender sends
packets 2 @3 453989:8=6 . Each packet matches 1 Q ’s trigger, and as a

2
 5 /  ) +
.
result the packet’s identifier !3 is replaced by the trigger’s stack
2 @3 4 6 . Next, the packet is forwarded to the MPEG-
id1 id1 id2 id2 id2  
  
H.263 transcoder, and then directly to receiver 1 Q . In contrast, an R4 R5 R6 R1
R2 R3
MPEG receiver 17L only needs to maintain a trigger 2 @3 41 Q"6 in  .
This way, receivers with different display capabilities can subscribe
to the same multicast group.
Another useful application is to have the receiver insist that all

     
    
 
data go through a firewall first before reaching it.   R4 R5 R6
R2 R3
3.3 Server Selection
 provides good support for basic server selection through the
use of the last B %
D bits of the identifiers to encode application Figure 5: Example of a scalable multicast tree with bounded
preferences.2 To illustrate this point consider two examples. degree by using chains of triggers.
In the first example, assume that there are several web servers
and the goal is to balance the client requests among these servers.
This goal can be achieved by setting the B %
D least significant bits
receivers of the multicast group construct and maintain the hierar-
chy of triggers.
of both trigger and packet identifiers to random values. If servers
have different capacities, then each server can insert a number of
triggers proportional to its capacity. Finally, one can devise an 4. ADDITIONAL DESIGN AND PERFOR-
adaptive algorithm in which each server varies the number of trig- MANCE ISSUES
gers as a function of its current load.
In this section we discuss some additional  design and per-
In the second example, consider the goal of selecting a server
formance issues. The  design was intended to be (among other
that is close to the client in terms of latency. To achieve this goal,
each server can use the last B
%
D bits of its trigger identifiers to
properties) robust, self-organizing, efficient, secure, scalable, incre-
encode its location, and the client can use the last B
%
D bits in the
mentally deployable, and compatible with legacy applications. In
this section we discuss these issues and some details of the design
packets’ identifier to encode its own location. In the simplest case,
that are relevant to them.
the location of an end-host (i.e., server or client) can be the zip
Before addressing these issues, we first review our basic design.
code of the place where the end-host is located; the longest pre-
 is organized as an overlay network in which every node (server)
fix matching procedure used by  would result then in the packet
stores a subset of triggers. In the basic design, at any moment of
being forwarded to a server that is relatively close to the client.3
time, a trigger is stored at only one server. Each end-host knows
3.4 Large Scale Multicast about one or more  servers. When a host wants to send a packet
2 @3 453989:8=6 , it forwards the packet to one of the servers it knows. If
The multicast abstraction presented in Section 2.4.2 assumes
the contacted server doesn’t store the trigger matching 2 @3 453989:8=6 ,
that all members of a multicast group insert triggers with identical
the packet is forwarded via IP to another server. This process con-
identifiers. Since triggers with identical identifier are stored at the
tinues until the packet reaches the server that stores the matching
same   server, that server is responsible for forwarding each mul-
trigger. The packet is then sent to the destination via IP.
ticast packet to every member of the multicast group. This solution
obviously does not scale to large multicast groups. 4.1 Properties of the Overlay
One approach to address this problem is to build a hierarchy of
 
The performance of  depends greatly on the nature of the un-
" #" " ) ,-,-,
triggers, where each member 1 of a multicast group @3 replaces

 

its trigger 2 @3 941 6 by a chain of triggers 2 @3 94 I 6 , 2 I 4 6 ,

derlying overlay network. In particular, we need an overlay net-
#"  
, work that exhibits the following desirable properties:
2


451 6 . This substitution is transparent to the sender: a packet

2 @3 943989:8=6 will still reach 1 
via the chain of triggers. Figure 5 1 Robustness: With a high probability, the overlay network
shows an example of a multicast tree with seven receivers in which remains connected even in the face of massive server and
no more than three triggers have the same identifier. This hierarchy communication failures.
of triggers can be constructed and maintained either cooperatively
by the members of the multicast group, or by a third party provider.
1 Scalability: The overlay network can handle the traffic gen-
In [18], we present an efficient distributed algorithm in which the erated by millions of end-hosts and applications.
) Recall that identifiers are B bits long and that D is the exact-
1 Efficiency: Routing a packet to the server that stores the
+
matching threshold.
Here we assume that nodes that are geographically close to each
packet’s best matching trigger involves a small number of
servers.
other are also close in terms of network distances, which is not
always true. One could instead use latency based encoding, much
1 Stability: The mapping between triggers and servers is rela-
as in [20]. tively stable over time, that is, it is unlikely to change during
 Node The main use of a public trigger is to allow an end-host to contact
i finger another end-host. The identifier of a public trigger is known by all
1 2
2 2
end-hosts in the system. An example is a web server that maintains
0 a public trigger to allow any client to contact it. A public trigger can
3 5 7 1
be defined as a the hash of the host’s DNS name, of a web address,
i finger
1 5
or of the public key associated with a web server. Public triggers
6 2 2 5 are long lived, typically days or months. In contrast, private triggers
3 7 are chosen by a small number of end-hosts and they are short lived.
Typically, private triggers exist only during the duration of a flow.
5
i finger 3 To illustrate the difference between public and private triggers,
4 consider a client  accessing a web server  that maintains a public

1 7
2 7 trigger 2 @3  4 6 . First, client  chooses a private trigger iden-
3 2
#2 #2
tifier @3 , inserts trigger 2 @3 4 6 into  , and sends @3 to the $2

web server via the server’s public trigger 2 @3  4 6 . Once con-
tacted, server  selects a private identifier @3  , inserts the associ-
Figure 6: Routing information (finger tables) maintained by
ated trigger 2 @3  4 6 into   , and sends its private trigger identifier
the Chord nodes.
@3  to client  #2
via  ’s private trigger 2 @3 =4 6 . The client and
the server then use both the private triggers to communicate. Once
the duration of a flow. This property allows end-hosts to op- the communication terminates, the private triggers are destroyed.
timize their performance by choosing triggers that are stored Sections 4.5 and 4.10 discuss how private triggers can be used to
on nearby servers. increase the routing efficiency, and the communication security.
Next, we discuss  ’s properties in more detail.
To implement   we have chosen the Chord lookup protocol [26],
which is known to satisfy the above properties. Chord uses an B -
4.3 Robustness
%
bit circular identifier space where follows L
MQ . Each server  inherits much of the robustness properties of the overlay itself
in that routing of packets within  is fairly robust against  node
is associated a unique identifier in this space. In the original Chord
protocol, each identifier !3 is mapped on the server with the clos- failures. In addition, end-hosts use periodic refreshing to maintain
their triggers into  . This soft-state approach allows for a simple
   
est identifier that follows !3 on the identifier circle. This server
is called successor of !3 and it is denoted by  'C2 @36 . Fig- and efficient implementation and frees the  infrastructure from
ure 6 shows an example in which there are three nodes, and BKJ  . having to recover lost state when nodes fail. If a trigger is lost—
Server 2 is responsible for identifiers 0, 1, and 2, server 5 is respon- for example, as a result of an  server failure—the trigger will be
sible for 3, 4 and 5, and server 7 is responsible for 6 and 7. reinserted, possibly at another server, the next time the end-host
refreshes it.
!
To implement the routing operation, each server maintains a rout-

! #5
ing table of size B . The  -th entry in the routing
I
     #!
table of server One potential problem with this approach is that although the

5
contains
L
I the first server that follows  L , i.e., 
!
C2 
6 . This server is called the  -th finger of . Note that the first
triggers are eventually reinserted, the time during which they are
unavailable due to server failures may be too large for some appli-
cations. There are at least two solutions to address this problem.
!
finger is the same as the successor server.
The first solution does not require  -level changes. The idea is to
Upon receiving a packet with identifier @3 , server checks whether
@3 lies between itself and its successor. If yes, the server forwards have each receiver 1 maintain a backup trigger 2 @3  2-4 / -
 4176 in
addition to the primary trigger 2 @3 4176 , and have the sender send
!
the packet to its successor, which should store the packet’s trigger.
If not, sends the packet to the closest server (finger) in its rout- packets with the identifier stack 2 @3 4 !3 2-4 / .
 6 . If the server stor-
ing table that precedes !3 . In this way, we are guaranteed that the ing the primary trigger fails, the packet will be then forwarded via
distance to @3 in the identifier space is halved at each step. As a re- the backup trigger to 1 .4 Note that to accommodate the case when
sult, it takes P  2
 )
 6 hops to route a packet to the server storing the packet is required to match every trigger in its identifier stack
the best matching trigger for the packet, irrespective of the start- (see Section 3.2), we use a flag in the packet header, which, if set,
ing point of the packet, where  is the number of   servers in the causes the packet to be dropped if the identifier at the head of its
system. stack doesn’t find a match. The second solution is to have the over-
In the current implementation, we assume that all identifiers that lay network itself replicate the triggers and manage the replicas.
share the same D -bit prefix are stored on the same server. A simple In the case of Chord, the natural replication policy is to replicate a
way to achieve this is to set the last B % D bits of every node iden- trigger on the immediate successor of the server responsible for that
trigger [5]. Finally, note that when an end-host fails, its triggers are
tifier to zero. As a result, finding the best matching trigger reduces
to performing a longest prefix matching operation locally. automatically deleted from  after they time-out.
While  is implemented on top of Chord, in principle  can use
any of the recently proposed P2P lookup systems such as CAN [22],
4.4 Self-Organizing
Pastry [23] and Tapestry [12].  is an overlay infrastructure that may grow to large sizes. Thus,
it is important that it not require extensive manual configuration or
4.2 Public and Private Triggers human intervention. The Chord overlay network is self-configuring,
Before discussing  ’s properties, we introduce an important tech- in that nodes joining the  infrastructure use a simple bootstrap-
nique that allows applications to use  more securely and effi- ping mechanism (see [26]) to find out about at least one existing  
ciently. With this technique applications make a distinction be- node, and then contacts that node to join the  infrastructure. Sim-

tween two types of triggers: public and private. This distinction Here we implicitly assume that the primary and backup triggers
is made only at the application level:  itself doesn’t differentiate
2-4 / - %
are stored on different servers. The receiver can ensure that this is
between private and public triggers. the case with high probability by choosing @3  7J L
@3 .
ilarly, end-hosts wishing to use   can locate at least one  server tmndec
using a similar bootstrapping technique; knowledge of a single  MPEG−H.263 transcoder (T)

server is all that’s needed to fully utilize the  infrastructure.

i3 i3
Proxy Proxy

4.5 Routing Efficiency mpeg





((T, R1), data) receiver (R1)

As with any network system, efficient routing is important to the (R1, data)
mpeg_play
overall efficiency of  . While  tries to route each packet effi- i3
ciently to the server storing the best matching trigger, the routing Proxy (id, data)
in an overlay network such as   is typically far less efficient than sender (S)
routing the packet directly via IP. To alleviate this problem, the (R2, data) i3
Proxy
sender caches the  server’s IP address. In particular, each data
and trigger packet carry in their headers a refreshing flag . When receiver (R2)
a packet reaches an  server, the server checks whether it stores
the best matching trigger for the packet. If not, it sets the flag in Figure 7: Heterogeneous multicast application. Refer to Fig-
the packet header before forwarding it. When a packet reaches the ure 4(b) for data forwarding in  .
server storing the best matching trigger, the server checks flag in

-
the packet header, and, if is set, it returns its IP address back to the
original sender. In turn, the sender caches this address and uses it a trigger : exceeds a certain threshold, the server storing the trig-
to send the subsequent packets with the same identifier. The sender ger pushes a copy of : to another server. This process can continue

-
can periodically set the refreshing flag as a keep-alive message recursively until the load is spread out. The decision of where to


with the cached server responsible for this trigger. push the trigger is subject to two constraints. First, should push

-
Note that the optimization of caching the server which stores the trigger to the server most likely to route the packets matching

. -
the receiver’s trigger does not undermine the system robustness. that trigger. Second, should try to minimize the state it needs
If the trigger moves to another server (e.g., as the result of a to maintain; at least needs to know the servers to which it has

  
new server joining the system),   will simply route the subsequent already pushed triggers in order to forward refresh messages for

 
packets from to . When the first packet reaches , the receiver these triggers (otherwise the triggers will expire). With Chord, one
will replace with in its cache. If the cached server fails, the simple way to address these problems is to always push the triggers
client simply uses another known   server to communicate. This to the predecessor server.
is the same fall-back mechanism as in the unoptimized case when If there are more triggers that share the same D -bit prefix with a
the client uses only one   server to communicate with all the other popular trigger : , all these triggers need to be cached together with
clients. Actually, the fact that the client caches the  server storing : . Otherwise, if the identifier of a packet matches the identifier
the receiver’s trigger can help reduce the recovery time. When the of a cached trigger : , we cannot be sure that : is indeed the best
sender notices that the server has failed, it can inform the receiver matching trigger for the packet.
to reinsert the trigger immediately. Note that this solution assumes
that the sender and receiver can communicate via alternate triggers 4.7 Scalability
that are not stored at the same  server. Since typically each flow is required to maintain two triggers
While caching the server storing the receiver’s trigger reduces (one for each end-point), the number of triggers stored in  is of the
the number of  hops, we still need to deal with the triangle rout- order of the number of flows plus the number of end-hosts. At first
ing problem. That is, if the sender and the receiver are close by, sight, this would be equivalent to a network in which each router
but the server storing the trigger is far away, the routing can be in- maintains per-flow state. Fortunately, this is not the case. While the

!
efficient. For example, if the sender and the receiver are both in state of a flow is maintained by each router along its path, a trigger

!
Berkeley and the server storing the receiver’s trigger is in London, is stored at only one node at a time. Thus, if there are triggers and
each packet will be forwarded to London before being delivered  servers, each server will store  triggers on the average. This
back to Berkeley! also suggests that  can be easily upgraded by simply adding more
One solution to this problem is to have the receivers choose their servers to the network. One interesting point to note is that these
private triggers such that they are located on nearby servers. This nodes do not need to be placed at specific locations in the network.
would ensure that packets won’t take a long detour before reach-
ing their destination. If an end-host knows the identifiers of the 4.8 Incremental Deployment
nearby  servers, then it can easily choose triggers with identifiers Since  is designed as an overlay network,  is incrementally
that map on these servers. In general, each end-host can sample deployable. At the limit,  may consist of only one node that stores
the identifier space to find ranges of identifiers that are stored at all triggers. Adding more servers to the system does not require
nearby servers. To find these ranges, a node  can insert random any system configuration. A new server simply joins the   system
triggers 2 !3 4  6 into  , and then estimate the RTT to the server
that stores the trigger by simply sending packets, 2 !3 43 
 B B A6 , to
using the Chord protocol, and becomes automatically responsible
for an interval in the identifier space. When triggers with identifiers
itself. Note that since we assume that the mapping of triggers onto in that interval are refreshed/inserted they will be stored at the new
servers is relatively stable over time, this operation can be done server. In this way, the addition of a new server is also transparent
off-line. We evaluate this approach by simulation in Section 5.1. to the end-hosts.

4.6 Avoiding Hot-Spots 4.9 Legacy Applications

Consider the problem of a large number of clients that try to con- The packet delivery service implemented by   is best-effort,
tact a popular trigger such as the CNN’s trigger. This may cause the which allows existing UDP-based applications to work over  eas-
server storing this trigger to overload. The classical solution to this ily. The end-host runs an   proxy that translates between the appli-
problem is to use caching. When the rate of the packets matching cations’ UDP packets and   packets, and inserts/refreshes triggers
on behalf of the applications. The applications do not need to be private trigger !3  , and sends this trigger back to  over  ’s private
modified, and are unaware of the  proxy. Packets are intercepted 2
trigger !3 . Since the sender’s trigger is encrypted, a malicious user
and translated by the  proxy transparently. As a proof of concept, cannot impersonate  .5
we have implemented the heterogeneous multicast application pre-
sented in Section 3.2 over  . The sender sends a MPEG stream, 4.10.2 Trigger hijacking
and one receiver plays back with a MPEG player (mpeg play) and A malicious user can isolate a host by removing its public trigger.
the other with a H.263 player (tmndec), as shown in Figure 7. Similarly, a malicious user in a multicast group can remove other
In [38], we present a solution using   to provide mobility support members from the group by deleting their triggers. While removing
for TCP-based legacy applications. a trigger also requires to specify the IP address of the trigger, this
address is, in general, not hard to obtain.
4.10 Security
-
One possibility to guard against this attack is to add another level

  -
Unlike IP, where an end-host can only send and receive pack- of indirection. Consider a server that wants to advertise a public


" #" - "
ets, in   end-hosts are also responsible for maintaining the routing trigger with identifier @3 . Instead of inserting the trigger 2 @3 4 6 ,

- "
information through triggers. While this allows flexibility for ap- the server can insert two triggers, 2 @3 =4 6 and 2 4 6 , where is an
plications, it also (and unfortunately) creates new opportunities for identifier known only by . Since a malicious user has to know
malicious users. We now discuss several security issues and how in order to remove either of the two triggers, this simple technique

"
 addresses them. provides effective protection against this type of attack. To avoid

 " #" -
We emphasize that our main goal here is not to design a bullet performance penalties, the receiver can choose such that both

 "
proof system. Instead, our goal is to design simple and efficient 2 @3 4 6 and 2 4 6 are stored at the same server. With the current
solutions that make  not worse and in many cases better than implementation this can be easily achieved by having @3 and
today’s Internet. The solutions outlined in this section should be share the same D -bit prefix.
viewed as a starting point towards more sophisticated and better
security solutions that we will develop in the future. 4.10.3 DoS Attacks
The fact that  gives end-hosts control on routing opens new
4.10.1 Eavesdropping possibilities for DoS attacks. We consider two types of attacks: (a)
Recall that the key to enabling multicast functionality is to al- attacks on end-hosts, and (b) attacks on the infrastructure. In the
low multiple triggers with the same identifer. Unfortunately, a ma- former case, a malicious user can insert a hierarchy of triggers (see
licious user that knows a host’s trigger can use this flexibility to Figure 5) in which all triggers on the last level point to the victim.
eavesdrop the traffic towards that host by simply inserting a trig- Sending a single packet to the trigger at the root of the hierarchy
ger with the same identifier and its own address. In addressing this will cause the packet to be replicated and all replicas to be sent
problem, we consider two cases: (a) private and (b) public triggers to the victim. This way an attacker can mount a large scale DoS
(see Section 4.2). attack by simply leveraging the  infrastructure. In the later case,
Private triggers are secretly chosen by the application end-points a malicious user can create trigger loops, for instance by connecting
and are not supposed to be revealed to the outside world. The length the leaves of a trigger hierarchy to its root. In this case, each packet
of the trigger’s identifier makes it difficult for a third party to use sent to the root will be exponentially replicated!
a brute force attack. While other application constraints such as To alleviate these attacks,  uses three techniques:
storing a trigger at a server nearby can limit the identifier choice, 1. Challenges:  assumes implicitly that a trigger that points
the identifier is long enough (i.e., LNO bits), such that the appli- to an end-host 1 is inserted by the end-host itself. An 
cation can always reserve a reasonably large number of bits that server can easily verify this assumption by sending a chal-
are randomly chosen. Assuming that an application chooses QI L'S
random bits in the trigger’s identifier, it will take an attacker L
)

lenge to 1 the first time the trigger is inserted. The challenge
consists of a random nonce that is expected to be returned by
probes on the average to guess the identifier. Even in the face of the receiver. If the receiver fails to answer the challenge the

L
) 5)
a I distributed attack


J L
I  of say one millions of hosts, it will take about
probes per host to guess a private trigger. We
trigger is removed. As a result an attacker cannot use a hier-
archy of triggers to mount a DoS attack (as described above),
note that the technique of using random identifiers as probabilistic since the leaf triggers will be removed as soon as the server
secure capabilities was previously used in [28, 37]. detects that the victim hasn’t inserted them.
Furthermore, end-points can periodically change the private trig- 2. Resource allocation: Each server uses Fair Queueing [7] to
gers associated with a flow. Another alternative would be for the allocate resources amongst the triggers it stores. This way
receiver to associate multiple private triggers to the same flow, and the damage inflicted by an attacker is only proportional to
the sender to send packets randomly to one of these private triggers. the number of triggers it maintains. An attacker cannot sim-
The alternative left to a malicious user is to intercept all private trig- ply use a hierarchy of triggers with loops to exponentially
gers. However this is equivalent to eavesdropping at the IP level or increase its traffic. As soon as each trigger reaches its fair
taking control of the  server storing the trigger, which makes  share the excess packets will be dropped. While this tech-
no worse than IP. nique doesn’t solve the problem, it gives  time to detect
With  , a public trigger is known by all users in the system, and and to eventually break the cycles.
thus anyone can eavesdrop the traffic to such a trigger. To alleviate To increase protection, each server can also put a bound on
this problem, end-hosts can use the public triggers to choose a pair the number of triggers that can be inserted by a particular
of private triggers, and then use these private triggers to exchange end-host. This will preclude a malicious end-host from mo-
the actual data. To keep the private triggers secret, one can use 
public key cryptography to exchange the private triggers. To initiate Note that an attacker can still count the number of connection re-
2
a connection, a host  encrypts its private trigger !3 under the
quests to  . However, this information is of very limited use, if
any, to the attacker. If, in the future, it turns out that this is un-
public key of a receiver  , and then sends it to  via  ’s public
2
acceptable for some applications, then other security mechanisms
trigger.  decrypts  ’s private trigger @3 , then chooses its own such as public trigger authentication will need to be used.
 4.5
power law random graph following network topologies in our simulations:
4
transit−stub
1 A power-law random graph topology generated with the INET
topology generator [16] with 5000 nodes, where the delay of
90th Percentile Latency Stretch

3.5 each link is uniformly distributed in the interval N;4"Q6 ms.

The   servers are randomly assigned to the network nodes.
3
1 A transit-stub topology generated with the GT- ITM topology
generator [11] with 5000 nodes, where link latencies are 100
2.5
ms for intra-transit domain links, 10 ms for transit-stub links
and 1 ms for intra-stub domain links.  servers are randomly
2
assigned to stub nodes.

1.5 5.1 End-to-End Latency

Consider a source  that sends packets to a receiver 1 via trigger

- -
1
0 5 10 15 20 25 30 35 2 @3 4176 . As discussed in Section 4.5, once the first packet reaches
the server storing the trigger 2 @3 4176 ,  caches and sends all
-
Number of Samples

- -
subsequent packets directly to . As a result, the packets will be
Figure 8: The 90th percentile latency stretch vs. number of routed via IP from  to and then from to 1 . The obvious
samples for PLRG and transit-stub with 5000 nodes. -
question is how efficient is routing through as compared to rout-
ing directly from  to 1 . Section 4.5 presents a simple heuristic
in which a receiver 1 samples the identifier space to find an iden-
nopolizing a server’s resources. $4
tifier @3 that is stored at a nearby server. Then 1 inserts trigger
3. Loop detection: When a trigger that doesn’t point to an IP
address is inserted, the server checks whether the new trigger
2 @34 45176 .
Figure 8 plots the 90th percentile latency stretch versus the num-
doesn’t create a loop. A simple procedure is to send a special ber of samples D in a system with QO4 S
 servers. Each point
packet with a random nonce. If the packet returns back to the represents the 90th percentile over 1000 measurements. For each
server, the trigger is simply removed. To increase the robust- measurement, we randomly choose a sender and a receiver. In
ness, the server can invoke this procedure periodically after each case, the receiver generates D triggers with random identi-
such a trigger is inserted. Another possibility to detect loops fiers. Among these triggers, the receiver retains the trigger that is
more efficiently would be to use a Bloom filter to encode the
- -
stored at the closest server. Then we sum the shortest path latency
set of  servers along the packet’s path, as proposed in the from the sender to and from to the receiver, and divide it by
Icarus framework [34]. the shortest path latency from the sender to the receiver to obtain
the latency stretch. Sampling the space of identifiers greatly low-
4.11 Anonymity ers the stretch. While increasing the number of samples decreases
Point-to-point communication networks such as the Internet pro- the stretch further, the improvement appears to saturate rapidly, in-
vide limited support for anonymity. Packets usually carry the des-
tination and the source addresses, which makes it relatively easy
#%
dicating that in practice, just Q"O L samples should suffice. The
receiver does not need to search for a close identifier every time a
for an eavesdropper to learn the sender and the receiver identi- connection is open; in practice, an end-host can sample the space
ties. In contrast, with  , eavesdropping the traffic of a sender will periodically and maintain a pool of identifiers which it can reuse.
not reveal the identity of the receiver, and eavesdropping the traf-
fic of a receiver will not reveal the sender’s identity. The level of 5.2 Proximity Routing in > ?
anonymity can be further enhanced by using chain of triggers or While Section 5.1 evaluates the end-to-end latency experienced
stack of identifiers to route packets. by data packets after the sender caches the server storing the re-
ceiver’s trigger : , in this section, we evaluate the latency incurred
5. SIMULATION RESULTS by the sender’s first packet that matches trigger : . This packet is
routed through the overlay network until it reaches the server stor-
In this section, we evaluate the routing efficiency of  by sim-
ing : . While Chord ensures that the overlay route length is only
ulation. One of the main challenges in providing efficient routing
 2
  6 , where  is the number of  servers, the routing latency
P
is that end-hosts have little control over the location of their trig-
can be quite large. This is because server identifiers are randomly
gers. However, we show that simple heuristics can significantly
chosen, and therefore servers close in the identifier space can be
enhance   ’s performance. The metric we use to evaluate these
very far away in the underlying network. To alleviate this problem,
heuristics is the ratio of the inter-node latency on the   network to
we consider two simple heuristics:
the inter-node latency on the underlying IP network. This is called
the latency stretch. 1 Closest finger replica In addition to each finger, a server
The simulator is based on the Chord protocol and uses iterative
#%
maintains C 7Q immediate successors of that finger. Thus,
style routing [26]. We assume that node identifiers are randomly
distributed. This assumption is consistent with the way the iden-
each node maintains references to about C )
  other
nodes for routing proposes. To route a packet, a server se-
tifiers are chosen in other lookup systems such as CAN [22] and lects the closest node in terms of network distance amongst
Pastry [23]. As discussed in [26], using random node identifiers (1) the finger with the largest identifier preceding the packet’s
increases system robustness and load-balancing. 6 We consider the
 We have also experimented with identifiers that have location se- was shown to approximate the Internet latency well [20]—onto the
one-dimensional Chord identifier space. However, the preliminary
mantics. In particular, we have used space filling curves, such as results do not show significant gains as compared to the heuristics
the Hilbert curve, to map a 3 -dimensional geometric space—which presented in this section, so we omit their presentation here.
 16
16

14
90th Percentile Latency Stretch 14

90th Percentile Latency Stretch

12
12

10
10

8
8

6
6

4 4

default chord default chord

2
closest finger set 2
closest finger set
closest finger replica closest finger replica
0 2 4 6 8 10 12 14 0 2 4 6 8 10 12 14
Number of i3 Servers x 10
4
Number of i3 Servers x 10
4

(a) (b)

Figure 9: The 90th percentile latency stretch in the case of (a) a power-law random network topology with 5000 nodes, and (b) a
transit-stub topology with 5000 nodes. The   servers are randomly assigned to all nodes in case (a), and only to the stub nodes in
case (b).

#%
identifier and (2) the C Q immediate successors of that fin- theory, this allows us to use any of the proposed lookup algorithms
ger. This heuristic was originally proposed in [5]. that performs exact matching.
1 
Both insert trigger requests and data packets share a common

    

Closest finger set Each server chooses    fingers as header of 48 bytes. In addition, data packets can carry a stack of
     6 and E L . To

C=2 6 , where 2  E up to four triggers (this feature isn’t used in the experiments). Trig-
route a packet, server considers only the closest   ) gers need to be updated every 30 seconds or they will expire. The
fingers in terms of network distances among all its    control protocol to maintain the overlay network is minimal. Each
fingers. server performs stabilization every 30 seconds (see [26]). Dur-
ing every stabilization period all servers generate approximately
Figure 9 plots the 90th percentile latency stretch as a function    control messages. Since in our experiments the number
of  ’s size for the baseline Chord protocol and the two heuris- of servers  is in the order of tens, we neglect the overhead due to
tics. The number of replicas C is 10, and is chosen such that the control protocol.
   J C )
  . Thus, with both heuristics, a server con- The testbed used for all of our experiments was a cluster of Pen-
tium III 700 MHz machines running Linux. We ran tests on systems
siders roughly the same number
number of  servers from L
I  of I routing entries. We vary the
to L , and in each case we aver-
 of up to 32 nodes, with each node running on its own processor.
age routing latencies over 1000 routing queries. In all cases the  The nodes communicated over a shared 1 Gbps Ethernet. For time
server identifiers are randomly generated. measurements, we use the Pentium timestamp counter (TSC). This
As shown in Figure 9, both heuristics can reduce the 90th per- method gives very accurate wall clock times, but sometime it in-
#%
centile latency stretch up to L  times as compared to the default cludes interrupts and context switches as well. For this reason, the
high extremes in the data are unreliable.
Chord protocol. In practice, we choose the “closest finger set”
heuristic. While this heuristic achieves comparable latency stretch
with “closest finger replica”, it is easier to implement and does not 5.4 Performance
require to increase the routing table size. The only change in the In the section, we present the overhead of the main operations
Chord protocol is to sample the identifier space using base in-
)
performed by  . Since these results are based on a very prelim-
stead of L , and store only the closest   fingers among the inary implementation, they should be seen as a proof of feasibil-
nodes sampled so far. ity and not as a proof of efficiency. Other Chord related perfor-
mance metrics such as the route length and system robustness are
5.3 Implementation and Experiments presented in [5].
We have implemented a bare-bones version of   using the Chord Trigger insertion: We consider the overhead of handling an
protocol. The control protocol used to maintain the overlay net- insert trigger request locally, as opposed to forwarding a request
work is fully asynchronous and is implemented on top of UDP. The to another server. Triggers are maintained in a hash table, so the
implementation uses 256 bit (B J LNO ) identifiers and assumes time is practically independent of the number of triggers. Inserting
that the matching procedure requires exact matching on the 128 a trigger involves just a hash table lookup and a memory alloca-

 
most significant bits ( DPJRQLS ). This choice makes it very unlikely tion. The average and the standard deviation of the trigger inser-
that a packet will erroneously match a trigger, and at the same time tion operation over 10,000 insertions are 12.5 sec, and 7.12 sec,
gives applications up to 128 bits to encode application specific in- respectively. This is mostly the time it takes the operating system


formation such as the host location (see Section 2.4.3). to process the packet and to hand it to the application. By compar-
For simplicity, in the current implementation we assume that all ison, memory allocation time is just 0.25 sec on the test machine.
triggers that share the first 128 bits are stored on the same server. In Note that since each trigger is updated every 30 sec, a server would
 40 80
50 percentile 50 percentile
Per Data Packet Forwarding Overhead (usec) 25, 75 percentiles 25, 75 percentiles
35 10, 90 percentiles 70 10, 90 percentiles

Per Packet Routing Overhead (usec)

30 60

25 50

20 40

15 30

10 20

5 10

0 0
0 200 400 600 800 1000 1200 1400 0 5 10 15 20 25 30 35
Packet Payload Size (bytes) Number of I3 Servers

Figure 10: Per packet forwarding overhead as a function of Figure 11: Per packet routing overhead as a function of 
payload packet size. In this case, the  header size is 48 bytes. nodes in the system. The packet payload size is zero.

Payload Size Avg. Throughput (std. dev.) Avg. Throughput

(bytes) (pkts/sec) (payload Mbps)
0 35,753 (2,406) 0

be able to maintain up to  S 4  JML
Q triggers. $,  200

400
33,130 (3,035)
28,511 (1,648)
53.00
91.23
Data packet forwarding: Figure 10 plots the overhead of for- 600 28,300 (595) 135.84
800 27,842 (1,028) 178.18
warding a data packet to its final destination. This involves looking 1,000 27,060 (1,127) 216.48
up the matching trigger and forwarding the packet to its destination 1,200 26,164 (1,138) 251.16
addresses. Since we didn’t enable multicast, in our experiments 1,400 23,339 (1,946) 261.39
there was never more than one address. Like trigger insertion,
packet forwarding consists of a hash table lookup. In addition, this Figure 12: The throughput of the data packet forwarding.
measurement includes the time to send the data packet. Packet for-
warding time, in our experiments, increases roughly linearly with
the packet size. This indicates that as packet size increases, mem- 6. RELATED WORK
ory copy operations and pushing the bits through the network dom- The rendezvous-based communication is similar in spirit to the
inate processing time. tuple space work in distributed systems [2, 14, 36]. A tuple space
 routing: Figure 11 plots the overhead of routing a packet to is a shared memory that can be accessed by any node in the system.
another  node. This differs from data packet forwarding in that Nodes communicate by inserting tuples and retrieving them from a
we route the packet using a node’s finger table rather than its trigger tuple space, rather than by point-to-point communication. Tuples
table. This occurs when a data packet’s trigger is stored on some are more general than data packets. A tuple consists of arbitrary
other node. The most costly operation here is a linear finger table typed fields and values, while a packet consists of just an identi-
lookup, as evidenced by the graph. There are two reasons for this fier and a data payload. In addition, tuples are guaranteed to be
seemingly poor behavior. First, we augment the finger table with a stored until they are explicitly removed. Unfortunately, the added
cache containing the most recent servers that have sent control or expressiveness and stronger guarantees of tuple spaces make them
data packets. Since in our experiments this cache is large enough very hard to efficiently implement on a large scale. Finally, with
to store all servers in the system, the number of nodes used to route tuple spaces, a node has to explicitly ask for each data packet. This
a packet (i.e., the fingers plus the cached nodes) increases roughly interface is not effective for high speed communications.
linearly with the number of nodes in the system. Second, the fin-  ’s communication paradigm is similar to the publish-subscribe-
ger table data structure in our implementation is a list. In a more notify (PSN) model. The PSN model itself exists in many pro-
polished implementation, a more efficient data structure is clearly prietary forms already in commercial systems [29, 31]. While the
needed to significantly improve the performance. matching operations employed by these systems are typically much
Throughput: Finally, we ran some experiments to see the max- more powerful than the longest prefix matching used by  , it is not
imum rate at which a node can process data packets. Ideally, this clear how scalable these systems are. In addition, these systems
should be the inverse of overhead. To test throughput, a single don’t provide support for service composition.
node is bombarded with more packets than it can reasonably han- Active Networks aim to support rapid development and deploy-
dle. We measure the time it takes for Q 4  packets to emerge ment of new network applications by downloading and executing
from the node to determine throughput. Not surprisingly, as packet customized programs in the network [33].  provides an alterna-
payload increases, throughput in packets decreases. In addition, we tive design that, while not as general and flexible as Active Net-
calculate the data throughput from the user perspective. Only the works, is able to realize a variety of basic communication services
payload data is considered; headers are overhead to users. The user without the need for mobile code or any heavyweight protocols.
throughput in Mbps increases as the packet payload increases be-  is similar to many naming systems. This should come as no
cause the overhead for headers and processing is roughly the same surprise, as identifiers can be viewed as semantic-less names. One
for both small and large payloads. future research direction is to use  as a unifying framework to
implement various name systems. network realms, IPNL provides more efficient routing by assuming
The Domain Name system (DNS) maps hostnames to IP ad- a hierarchical topology with a single “middle realm”. Packet for-
dresses [19]. A DNS name is mapped to an end-host as a result warding in both TRIAD and IPNL is similar to packet forwarding
of an explicit request at the beginning of a transfer. In  , the based on identifier stacks in  . However, while with TRIAD and
identifier-to-address mapping and the packet forwarding are tightly IPNL the realm-to-realm path of a packet is determined during the
integrated. DNS resolvers form a static overlay hierarchy, while  DNS name resolution by network specific protocols, with  the
servers form a self-organizing overlay. path is determined by end-hosts.
Active Names (AN) map a name to a chain of mobile code re- Multi-Protocol Label Switching (MPLS) was recently proposed
sponsible for locating the remote service, and transporting its re- to speed-up the IP route lookup and to perform route pinning [1].
sponse to the destination [30]. The code is executed on names at Similar to  , each packet carries a stack of labels that specifies the
resolvers. The goals of AN and  are different. In AN, applica- packet route. The first label in the stack specifies the next hop. Be-
tions use names to describe what they are looking for, while in  fore forwarding a packet, a router replaces the label at the head of
identifiers are used primary as a way to abstract away the end-host the stack. There are several key differences between  and MPLS.
location. Also, while the goal of AN is to support extensibility for While  identifiers have global meaning, labels have only local
wide-area distributed services, the goal of  is to support basic meaning. In addition, MPLS requires special protocols to choose
communication primitives such as multicast and anycast. and distribute the labels. In contrast, with   identifier stacks are
The Intentional Naming System (INS) is a resource discovery chosen and maintained by end-hosts.
and service location system for mobile hosts [32]. INS uses an
attribute-based language to describe names. Similar to  identi- 7. DISCUSSION AND FUTURE WORK
fiers, INS names are inserted and refreshed by the application. INS
While we firmly believe in the fundamental purpose of  —
also implements a late biding mechanism that integrates the name
providing a general-purpose indirection service through a single
resolution with message forwarding.  differs from INS in that
overlay infrastructure—the details of our design are preliminary.
from the network’s point of view, an identifier does not carry any
Besides exploring the security and efficiency issues mentioned in
semantics. This simplicity allows for a scalable and efficient imple-
the paper further, there are areas that deserve significant additional
mentation. Another difference is that  allows end-hosts to control
attention.
the application-level path followed by the packets.
A general question is what range of services and applications can
The rendezvous-based abstraction is similar to the IP multicast
be synthesized from the fixed abstraction provided by  . Until now
abstraction [6]. An IP multicast address identifies the receivers of
we have developed two applications on top of  , a mobility solu-
a multicast group in the same way an  identifier identifies the
tion [38], and a scalable reliable multicast protocol [18]. While the
multicast receivers. However, unlike IP which allocates a special
initial experience with developing these applications has been very
range of addresses (i.e., class D) to multicast,  does not put any
promising, it is too early to precisely characterize the limitations
restrictions on the identifier format. This gives   applications the
and the expressiveness of the  abstraction. To answer this ques-
ability to switch on-the-fly from unicast to multicast. In addition,
tion, we need to gain further experience with using and deploying
 can support multicast groups with heterogeneous receivers.
new applications on top of  .
Several solutions to provide the anycast service have been re-
For inexact matching, we have used longest-prefix match. Inex-
cently proposed. IP Anycast aims to provide this service at the IP
act matching occurs locally, on a single node, so one could use any
layer [21]. All members of an anycast group share the same IP
reasonably efficient matching procedure. The question is which
address. IP routing forwards an anycast packet to the member of
inexact matching procedure will best allow applications to choose
the anycast group that is the closest in terms of routing distance.
among several candidate choices. This must work for choosing
Global IP-Anycast (GIA) provides an architecture that addresses
based on feature sets (e.g., selecting printers), location (e.g., se-
the scalability problems of the original IP Anycast by differentiat-
lecting servers), and policy considerations (e.g., automatically di-
ing between rarely used and popular anycast groups [17]. In con-
recting users to facilities that match their credentials). We chose
trast to these proposals,  can use distance metrics that are only
longest-prefix match mostly for convenience and familiarity, and it
available at the application level such as server load, and it sup-
seems to work in the examples we’ve investigated, but there may
ports other basic communication primitives such as multicast and
be superior options.
service composition.
Our initial design decision was to use semanticless identifiers
Estrin et al. have proposed an attribute-based data communi-
and routing; that is, identifiers are chosen randomly and routing is
cation mechanism, called direct diffusion, to disseminate data in
done based on those identifiers. Instead, one could embed location
sensor networks [8]. Data sources and sinks use attributes to iden-
semantics into the node identifiers. This may increase the efficiency
tify what information they provide or are interested in. A user that
of routing, by allowing routes to take lower-latency  -level hops,
wants to receive data inserts an interest into the network in the form
but at the cost of making the overlay harder to deploy, manage, and
of attribute-value pairs. At a high level, attributes are similar to
load-balance.
identifiers, and interests are similar to triggers. However, in di-
Our decision to use Chord [26] to implement  was motivated
rect diffusion, the attributes have a much richer semantic and the
by the protocol simplicity, its provable properties, and by our famil-
rules can be much more complex than in  . At the implementation
iarity with the protocol. However, one could easily implement 
level, in direct diffusion, nodes flood the interests to their neigh-
on top of other lookup protocols such as CAN [22], Pastry [23] and
bors, while   uses a lookup service to store the triggers determined
Tapestry [12]. Using these protocols may present different bene-
based on the trigger identifier.
fits. For instance, using Pastry and Tapestry can reduce the latency
TRIAD [3] and IPNL [9] have been recently proposed to solve
of the first packets of a flow, since these protocols find typically
the IPv4 address scarcity problem. Both schemes use DNS names
routes with lower latencies than Chord. However, note that once
rather than addresses for global identification. However, TRIAD
the sender caches the server storing the receiver’s trigger, there will
and IPNL make different tradeoffs. While TRIAD is more gen-
be little difference between using different lookup protocols, as the
eral by allowing an unlimited number of arbitrarily connected IP
packets will forwarded directly via IP to that server. Studying the
trade-offs involved by using various lookup protocols to implement [6] D EERING , S., AND C HERITON , D. R. Multicast routing in
 is a topic of future research. datagram internetworks and extended LANs. ACM
While these design decisions are important, they may have lit- Transactions on Computer Systems 8, 2 (May 1990), 85–111.
tle to do with whether  is ever deployed. We don’t know what [7] D EMERS , A., K ESHAV, S., AND S HENKER , S. Analysis and
the economic model of  would be and whether its most likely simulation of a fair queueing algorithm. In Journal of
deployment would be as a single provider for-profit service (like Internetworking Research and Experience (Oct. 1990),
content distribution networks), or a multiprovider for-profit service pp. 3–26. (Also in Proc. of ACM SIGCOMM’89, pages
(like ISPs), or a cooperatively managed nonprofit infrastructure. 3-12).
While deployment is always hard to achieve,  has the advantage [8] E STRIN , D., G OVINDAN , R., H EIDEMANN , J., AND
that it can be incrementally deployed (it could even start as a sin- K UMAR , S. Next century challenges: Scalable coordination
gle, centrally located server!). Moreover, it does not require the in sensor networks. In Proc. of ACM/IEEE MOBICOM’99
cooperation of ISPs, so third-parties can more easily provide this (Cambridge, MA, Aug. 1999).
service. Nonetheless,  faces significant hurdles before ever being [9] F RANCIS , P., AND G UMMADI , R. IPNL: A NAT extended
deployed. internet architecture. In Proc. ACM SIGCOMM’01 (San
Diego, 2001), pp. 69–80.
8. SUMMARY [10] G RIBBLE , S. D., W ELSH , M., VON B EHREN , J. R.,
B REWER , E. A., C ULLER , D. E., B ORISOV, N.,
Indirection plays a fundamental role in providing solutions for
C ZERWINSKI , S. E., G UMMADI , R., H ILL , J. R., J OSEPH ,
mobility, anycast and multicast in the Internet. In this paper we pro-
A. D., K ATZ , R. H., M AO , Z. M., ROSS , S., AND Z HAO ,
pose a new abstraction that unifies these solutions. In particular, we
B. Y. The ninja architecture for robust internet-scale systems
propose to augment the point-to-point communication abstraction
and services. Computer Networks 35, 4 (2001), 473–497.
with a rendezvous-based communication abstraction. This level
[11] Georgia tech internet topology model.
of indirection decouples the sender and the receiver behaviors and
http://www.cc.gatech.edu/fac/Ellen.Zegura/graphs.html.
allows us to provide natural support for mobility, anycast and mul-
ticast. [12] H ILDRUM , K., K UBATOWICZ , J. D., R AO , S., AND Z HAO ,
To demonstrate the feasibility of this approach, we have built an B. Y. Distributed Object Location in a Dynamic Network. In
overlay network based on the Chord lookup system. Preliminary Proc. 14th ACM Symp. on Parallel Algorithms and
experience with  suggests that the system is highly flexible and Architectures (Aug. 2002).
can support relatively sophisticated applications that require mo- [13] H OLBROOK , H., AND C HERITON , D. IP multicast channels:
bility, multicast, and/or anycast. In particular, we have developed a EXPRESS support for large-scale single-source applications.
simple heterogeneous multicast application in which MPEG video In Proc. of ACM SIGCOMM’99 (Cambridge, Massachusetts,
traffic is transcoded on the fly to H.263 format. In addition, we Aug. 1999), pp. 65–78.
have recently developed two other applications: providing transpar- [14] Java Spaces. http://www.javaspaces.homestead.com/.
ent mobility to legacy applications [38], and a large scale reliable [15] JANNOTTI , J., G IFFORD , D. K., J OHNSON , K. L.,
multicast protocol [18]. K AASHOEK , M. F., AND J. W. O’T OOLE , J. Overcast:
Reliable multicasting with an overlay network. In Proc. of
the 4th USENIX Symposium on Operating Systems Design
9. ACKNOWLEDGMENTS and Implementation (OSDI 2000) (San Diego, California,
The authors would like to thank Sylvia Ratnasamy, Kevin Lai, October 2000), pp. 197–212.
Karthik Lakshminarayanan, Ananthapadmanabha Rao, Adrian Per- [16] J IN , C., C HEN , Q., AND JAMIN , S. Inet: Internet topology
rig, and Randy Katz for their insightful comments that helped im- generator, 2000. Technical report CSE-TR-433-00,
prove the  design. We thank Hui Zhang, Dawn Song, Volker University of Michigan, EECS dept,
Roth, Lakshminarayanan Subramanian, Steve McCanne, Srinivasan http://topology.eecs.umich.edu/inet.
Keshav, and the anonymous reviewers for their useful comments [17] K ATABI , D., AND W ROCLAWSKI , J. A framework for
that helped improve the paper. scalable global ip-anycast (gia). In Proc. of SIGCOMM 2000
(Stockholm, Sweden, Aug. 2000), pp. 3–15.
10. REFERENCES [18] L AKSHMINARAYANAN , K., R AO , A., S TOICA , I., AND
S HENKER , S. Flexible and robust large scale multicast using
[1] C ALLON , R., D OOLAN , P., F ELDMAN , N., F REDETTE , A., i3. Tech. Rep. CS-02-1187, University of California -
S WALLOW, G., AND V ISWANATHAN , A. A framework for Berkeley, 2002.
multiprotocol label switching, Nov. 1997. Internet Draft, [19] M OCKAPETRIS , P., AND D UNLAP, K. Development of the
draft-ietf-mpls-framework-02.txt. Domain Name System. In Proc. ACM SIGCOMM (Stanford,
[2] C ARRIERO , N. The Implementation of Tuple Space CA, 1988), pp. 123–133.
Machines. PhD thesis, Yale University, 1987. [20] N G , T. S. E., AND Z HANG , H. Predicting internet network
[3] C HERITON , D. R., AND G RITTER , M. TRIAD: A new next distance with coordinates-based approaches. In Proc. of
generation Internet architecture, Mar. 2000. INFOCOM’02 (New York, NY, 2002).
http://www-dsg.stanford.edu/triad/ triad.ps.gz. [21] PARTRIDGE , C., M ENDEZ , T., AND M ILLIKEN , W. Host
[4] C HU , Y., R AO , S. G., AND Z HANG , H. A case for end anycasting service, nov 1993. RFC-1546.
system multicast. In Proc. of ACM SIGMETRICS’00 (Santa [22] R ATNASAMY, S., F RANCIS , P., H ANDLEY, M., K ARP, R.,
Clara, CA, June 2000), pp. 1–12. AND S HENKER , S. A scalable content-addressable network.
[5] DABEK , F., K AASHOEK , F., K ARGER , D., M ORRIS , R., In Proc. ACM SIGCOMM (San Diego, 2001), pp. 161–172.
AND S TOICA , I. Wide-area cooperative storage with cfs. In
Proc. ACM SOSP’01 (Banff, Canada, 2001), pp. 202–215.
[23] ROWSTRON , A., AND D RUSCHEL , P. Pastry: Scalable,
distributed object location and routing for large-scale
peer-to-peer systems. In Proc. of the 18th IFIP/ACM
International Conference on Distributed Systems Platforms
(Middleware 2001) (Nov. 2001), pp. 329–350.
[24] S NOEREN , A. C., AND BALAKRISHNAN , H. An end-to-end
approach to host mobility. In Proc. of ACM/IEEE
MOBICOM’99 (Cambridge, MA, Aug. 1999).
[25] S NOEREN , A. C., BALAKRISHNAN , H., AND K AASHOEK ,
M. F. Reconsidering internet mobility. In Proc. of the 8th
IEEE Workshop on Hot Topics in Operating Systems
(HotOS-VIII) (Elmau/Oberbayern, Germany, May 2001).
[26] S TOICA , I., M ORRIS , R., K ARGER , D., K AASHOEK ,
M. F., AND BALAKRISHNAN , H. Chord: A scalable
peer-to-peer lookup service for internet applications. In Proc.
ACM SIGCOMM’01 (San Diego, 2001), pp. 149–160.
[27] S TOICA , I., N G , T., AND Z HANG , H. REUNITE: A
recursive unicast approach to multicast. In Proc. of
INFOCOM’00 (Tel-Aviv, Israel, Mar. 2000), pp. 1644–1653.
[28] TANENBAUM , A. S., K AASHOEK , M. F., VAN R ENESSE ,
R., AND BAL , H. E. The amoeba distributed operating
system: a status report. Computer Communications 14 (1),
324–335.
[29] Tibco software. http://www.tibco.com.
[30] VAHDAT, A., DAHLIN , M., A NDERSON , T., AND
AGGARWAL , A. Active names: Flexible location and
transport. In Proc. of USENIX Symposium on Internet
Technologies & Systems (Oct. 1999).
[31] Vitria. http://www.vitria.com.
[32] W. A DJIE -W INOTO AND E. S CHWARTZ AND H.
BALAKRISHNAN AND J. L ILLEY. The design and
implementation of an intentional naming system. In Proc.
ACM Symposium on Operating Systems Principles (Kiawah
Island, SC, Dec. 1999), pp. 186–201.
[33] W ETHERALL , D. Active network vision and reality: lessons
form a capsule-based system. In Proc. of the 17th ACM
Symposium on Operating System Principles (SOSP’99)
(Kiawah Island, SC, Nov. 1999), pp. 64–79.
[34] W HITAKER , A., AND W ETHERALL , D. Forwarding without
loops in icarus. In Proc. of OPENARCH 2002 (New York
City, NY, June 2002).
[35] WAP wireless markup language specification (WML).
http://www.oasis-open.org/cover/wap-wml.html.
[36] W YCKOFF , P., M C L AUGHRY, S. W., L EHMAN , T. J., AND
F ORD , D. A. T Spaces. IBM System Journal 37, 3 (1998),
454–474.
[37] YARVIN , C., B UKOWSKI , R., AND A NDERSON , T.
Anonymous rpc: Low-latency protection in a 64-bit address
space. In Proc. of USENIX (June 1993), pp. 175–186.
[38] Z HUANG , S., L AI , K., S TOICA , I., K ATZ , R., AND
S HENKER , S. Host mobility using an internet indirection
infrastructure. Tech. Rep. UCB/CSD-02-1186, Computer
Science Division, U. C. Berkeley, June 2002.