BGP Communities
BGP Communities
BGP Communities
ISP Workshops
These materials are licensed under the Creative Commons Attribution-NonCommercial 4.0 International license
(http://creativecommons.org/licenses/by-nc/4.0/)
Philip Smith 2
Multihoming and Communities
p The BGP community attribute is a very powerful tool for
assisting and scaling BGP Policies and BGP Multihoming
p Most major Network Operators make extensive use of
BGP communities:
n Internal policies
n Inter-provider relationships (MED replacement)
n Customer traffic engineering
3
Using BGP Communities
p Four scenarios are covered:
n Use of RFC1998 traffic engineering
n Extending RFC1998 ideas for even greater customer policy
options
n Community use in Network Operator backbones
n Customer Policy Control (aka traffic engineering)
4
RFC1998
An example of how Network Operators use
communities…
5
RFC1998
p Informational RFC
p Describes how to implement loadsharing and backup on
multiple inter-AS links
n BGP communities used to determine local preference in
upstream’s network
p Gives control to the customer
n Means the customer does not have to phone upstream’s
technical support to adjust traffic engineering needs
p Simplifies upstream’s configuration
n Simplifies network operation!
6
RFC1998
p RFC1998 Community values are defined to have particular
meanings
p ASx:100 set local preference 100
n Make this the preferred path
p ASx :90 set local preference 90
n Make this the backup if dualhomed on ASx
p ASx :80 set local preference 80
n The main link is to another provider with same AS path length
p ASx :70 set local preference 70
n The main link is to another provider
7
RFC1998
p Upstream Provider defines the communities mentioned
p Their customers then attach the communities they want to use to
the prefix announcements they are making
p For example:
n If upstream is AS 100
n To declare a particular path as a backup path, their customer would
announce the prefix with community 100:70 to AS100
n AS100 would receive the prefix with the community 100:70 tag, and then set
local preference to be 70
8
RFC1998
p Sample End-Site Router Configuration
router bgp 130
address-family ipv4
neighbor 100.66.32.1 remote-as 100
neighbor 100.66.32.1 description Backup Provider
neighbor 100.66.32.1 route-map as100-out out
neighbor 100.66.32.1 send-community
neighbor 100.66.32.1 activate
!
ip as-path access-list 20 permit ^$
!
route-map as100-out permit 10
match as-path 20
set community 100:70
!
9
RFC1998
p Sample Upstream Router Configuration
router bgp 100
address-family ipv4
neighbor 100.66.32.2 remote-as 130
neighbor 100.66.32.2 route-map customer-policy-in in
neighbor 100.66.32.2 activate
!
! Homed to another Provider
ip community-list standard rfc1998-70 permit 100:70
! Homed to another Provider with equal ASPATH length
ip community-list standard rfc1998-80 permit 100:80
! Customer backup routes
ip community-list standard rfc1998-90 permit 100:90
!
10
RFC1998
route-map customer-policy-in permit 10
match community rfc1998-70
set local-preference 70
!
route-map customer-policy-in permit 20
match community rfc1998-80
set local-preference 80
!
route-map customer-policy-in permit 30
match community rfc1998-90
set local-preference 90
!
route-map customer-policy-in permit 40
set local-preference 100
!
11
RFC1998
p RFC1998 was the inspiration for a large variety of differing
community policies implemented by Network Operators worldwide
p There are no “standard communities” for what ISPs do
p But best practices today consider that Network Operators should
use BGP communities extensively for multihoming support of
traffic engineering
p Look in the Network Operator AS Object in the IRR for documented
community support
12
RFC1998 Example
Two links to the same AS, one link
primary, the other link backup
Two links to the same AS
primary
C
A
AS 100 AS 65534
E B
D
backup
23
Background
p RFC1998 is okay for “simple” multihoming situations
p Network Operators create backbone support for many
other communities to handle more complex situations
n Simplify Network Operator BGP configuration
n Give customer more policy control
24
Network Operator BGP Communities
p There are no recommended Network Operator BGP communities apart from
n RFC1998
n The well known communities
p www.iana.org/assignments/bgp-well-known-communities
p Efforts have been made to document from time to time
n totem.info.ucl.ac.be/publications/papers-elec-versions/draft-quoitin-bgp-comm-
survey-00.pdf
n But so far… nothing more… L
n Collection of Network Operator communities at www.onesc.net/communities
n NANOG Tutorial:
www.nanog.org/meetings/nanog40/presentations/BGPcommunities.pdf
p Network Operator policy is usually published
n On the Operator’s website
n Referenced in the AS Object in the IRR
Typical Network Operator BGP Communities
p X:80 set local preference 80
n Backup path
p X:120 set local preference 120
n Primary path (over ride BGP path selection default)
p X:1 set as-path prepend X
n Single prepend when announced to X’s upstreams
p X:2 set as-path prepend X X
n Double prepend when announced to X’s upstreams
p X:3 set as-path prepend X X X
n Triple prepend when announced to X’s upstreams
p X:666 set ip next-hop 192.0.2.1
n Blackhole route – very useful for DoS attack mitigation (RFC7999)
26
Sample Router Configuration (1)
router bgp 100
address-family ipv4 Customer BGP
neighbor 100.66.32.2 remote-as 130
neighbor 100.66.32.2 route-map customer-policy-in in
neighbor 100.66.32.2 activate
neighbor 100.65.8.9 remote-as 200
neighbor 100.65.8.9 route-map upstream-out out
neighbor 100.65.8.9 activate
! Upstream BGP
ip community-list standard prepend-1 permit 100:1
ip community-list standard prepend-2 permit 100:2
ip community-list standard prepend-3 permit 100:3
ip community-list standard lp-80 permit 100:80
ip community-list standard lp-120 permit 100:120 Black hole route
ip community-list standard RTBH permit 100:666 (on all routers)
!
ip route 192.0.2.1 255.255.255.255 null0 27
Sample Router Configuration (2)
route-map customer-policy-in permit 10
match community lp-80
set local-preference 80
!
route-map customer-policy-in permit 20
match community lp-120
set local-preference 120
!
route-map customer-policy-in permit 30
match community RTBH
set ip next-hop 192.0.2.1
!
route-map customer-policy-in permit 40
...etc...
28
Sample Router Configuration (3)
route-map upstream-out permit 10
match community prepend-1
set as-path prepend 100
!
route-map upstream-out permit 20
match community prepend-2
set as-path prepend 100 100
!
route-map upstream-out permit 30
match community prepend-3
set as-path prepend 100 100 100
!
route-map upstream-out permit 40
...etc...
29
Example: Sprint
More info at
https://www.sprint.net/index.php?p=policy_bgp
30
Example: NTT
More info at
http://www.us.ntt.net/support/policy/routing.cfm
Example: Verizon Europe
aut-num: AS702
descr: Verizon Business EMEA - Commercial IP service provider in Europe
<snip>
remarks: --------------------------------------------------------------
Verizon Business filters out inbound prefixes longer than /24.
We also filter any networks within AS702:RS-INBOUND-FILTER.
--------------------------------------------------------------
VzBi uses the following communities with its customers:
702:80 Set Local Pref 80 within AS702
702:120 Set Local Pref 120 within AS702
702:20 Announce only to VzBi AS'es and VzBi customers
702:30 Keep within Europe, don't announce to other VzBi AS's
702:1 Prepend AS702 once at edges of VzBi to Peers
702:2 Prepend AS702 twice at edges of VzBi to Peers
702:3 Prepend AS702 thrice at edges of VzBi to Peers
--------------------------------------------------------------
Advanced communities for customers
702:7020 Do not announce to AS702 peers with a scope of
National but advertise to Global Peers, European
Peers and VzBi customers.
702:7001 Prepend AS702 once at edges of VzBi to AS702
peers with a scope of National.
702:7002 Prepend AS702 twice at edges of VzBi to AS702
peers with a scope of National.
<snip> And many more!
--------------------------------------------------------------
Additional details of the VzBi communities are located at: 32
http://www.verizonbusiness.com/uk/customer/bgp/
Example: Telia
aut-num: AS1299
descr: TeliaSonera International Carrier
<snip>
remarks: BGP COMMUNITY SUPPORT FOR AS1299 TRANSIT CUSTOMERS:
remarks:
remarks: Community Action (default local pref 200)
remarks: -----------------------------------------
remarks: 1299:50 Set local pref 50 within AS1299 (lowest possible)
remarks: 1299:150 Set local pref 150 within AS1299 (equal to peer, backup)
remarks:
remarks: European peers
remarks: Community Action
remarks: --------- ------
remarks: 1299:200x All peers Europe incl:
remarks:
remarks: 1299:250x Sprint/1239
remarks: 1299:252x NTT/2914
remarks: 1299:253x Zayo/Abovenet/6461
remarks: 1299:254x Orange/FT/5511
remarks: 1299:256x Level3/3356
remarks: 1299:257x Verizon/702
remarks: 1299:258x AT&T/2686
remarks: 1299:259x Telxius/Telefonica/12956
remarks: 1299:261x Centurylink/Qwest/3910
remarks: 1299:263x TATA/6453
remarks: 1299:264x DTAG/3320 And many
<snip> many more!
remarks: Where x is number of prepends (x=0,1,2,3) or do NOT announce (x=9) 33
Example: BT Ignite
aut-num: AS5400
descr: BT Ignite European Backbone
<snip>
remarks: The following BGP communities can be set by BT
remarks: BGP customers to affect announcements to major peers.
remarks:
remarks: 5400:NXXX
remarks: N=1 not announce
remarks: N=2 prepend an extra "5400 5400" on announcement
remarks: Valid values for XXX:
remarks: 000 All peers and transits
remarks: 500 All transits
remarks: 503 Level3 AS3356
remarks: 509 Telia AS1299
remarks: 510 NTT Verio AS2914
remarks: 002 Sprint AS1239
remarks: 003 Savvis AS3561
remarks: 004 C&W AS1273
remarks: 005 Verizon EMEA AS702
remarks: 014 DTAG AS3320
remarks: 016 Opentransit AS5511
remarks: 018 GlobeInternet Tata AS6453
remarks: 023 Tinet AS3257 And many
remarks: 027 Telia AS1299 more!
remarks: 045 Telecom Italia AS6762
remarks: 073 Eurorings AS286
34
remarks: 169 Cogent AS174
<snip>
Example: Level3
aut-num: AS3356
descr: Level 3 Communications
<snip>
remarks: --------------------------------------------------------
remarks: customer traffic engineering communities - Suppression
remarks: --------------------------------------------------------
remarks: 64960:XXX - announce to AS XXX if 65000:0
remarks: 65000:0 - announce to customers but not to peers
remarks: 65000:XXX - do not announce at peerings to AS XXX
remarks: --------------------------------------------------------
remarks: customer traffic engineering communities - Prepending
remarks: --------------------------------------------------------
remarks: 65001:0 - prepend once to all peers
remarks: 65001:XXX - prepend once at peerings to AS XXX
remarks: 65002:0 - prepend twice to all peers
remarks: 65002:XXX - prepend twice at peerings to AS XXX
<snip>
remarks: --------------------------------------------------------
remarks: customer traffic engineering communities - LocalPref
remarks: --------------------------------------------------------
remarks: 3356:70 - set local preference to 70
remarks: 3356:80 - set local preference to 80
remarks: 3356:90 - set local preference to 90 And many
remarks: more!
--------------------------------------------------------
remarks: customer traffic engineering communities - Blackhole
remarks: -------------------------------------------------------- 35
remarks: 3356:9999 - blackhole (discard) traffic
<snip>
Creating your own community policy
p Consider creating communities to give policy control to
customers
n Reduces technical support burden
n Reduces the amount of router reconfiguration, and the chance
of mistakes
n Use previous Network Operator and configuration examples as a
guideline
36
Using Communities for Backbone
Scaling
Scaling BGP in the Service Provider
backbone…
37
Communities for iBGP
p Network Operators tag prefixes learned from their BGP
and static customers with communities
n To identify services the customer may have purchased
n To identify prefixes which are part of the Provider’s PA space
n To identify PI customer addresses
n To control prefix distribution in iBGP
n To control prefix announcements to customers and upstreams
n (amongst several other reasons)
38
Service Identification
p Network Operator provides:
n Transit via upstreams
n Connectivity via major IXP
n Connectivity to private peers/customers
p Customers can buy all or any of the above access options
n Each option is identified with a unique community
p Network Operator identifies whether address space
comes from their PA block or is their customers’ own PI
space
n One community for each
39
Community Definitions
100:1000 AS100 aggregates
100:1001 AS100 aggregate subprefixes
100:1005 Static Customer PI space
100:2000 Customers who get Transit
100:2100 Customers who get IXP access
100:2200 Customers who get BGP Customer access
100:3000 Routes learned from the IXP
41
Service Identification
p AS100 has four classes of BGP customers
n Full transit (upstream, IXP and BGP customers)
n Upstream only
n IXP only
n BGP Customers only
p For BGP support, easiest IOS configuration is to create a peer-
group for each class (can also use peer-templates to simplify
further)
n Customer is assigned the peer-group of the service they have purchased
n Simple for AS100 customer installation engineer to provision
42
BGP Customers
Creating peer-groups
router bgp 100
address-family ipv4
neighbor full-transit peer-group
neighbor full-transit route-map customers-out out
neighbor full-transit route-map full-transit-in in
neighbor full-transit default-originate
neighbor upstream-only peer-group
neighbor upstream-only route-map customers-out out
neighbor upstream-only route-map upstream-only-in in
neighbor upstream-only default-originate
neighbor ixp-only peer-group
neighbor ixp-only route-map ixp-routes out
neighbor ixp-only route-map ixp-only-in in
neighbor bgpcust-only peer-group
neighbor bgpcust-only route-map bgp-cust-out out
neighbor bgpcust-only route-map bgp-cust-in in
43
BGP Customers
Creating route-maps
route-map customers-out permit 10 Customers only get AS100
match ip community aggregates aggregates and default route
!
route-map full-transit-in permit 10 Full transit go everywhere
set community 100:2000 100:2100 100:2200
! Customers buying IXP
route-map upstream-only-in permit 10
access only get aggregates,
set community 100:2000
! static & full transit
route-map ixp-routes permit 10 customers and IXP routes
match ip community aggregates pi transits ixp-access ixp-routes
!
route-map ixp-only-in permit 10 Customers buying BGP customer
set community 100:2100 access only get aggregates,
! static & full transit customers
route-map bgp-cust-out permit 10 and other BGP customers
match ip community aggregates pi transits bgp-custs
!
route-map bgp-cust-in permit 10 44
set community 100:2200
BGP Customers – configuring customers
router bgp 100
address-family ipv4
neighbor 100.67.3.2 remote-as 200 Customers are
neighbor 100.67.3.2 peer-group full-transit placed into the
neighbor 100.67.3.2 prefix-list as200cust-in appropriate
neighbor 100.67.3.2 activate peer-group
neighbor 100.67.3.6 remote-as 300 depending on
neighbor 100.67.3.6 peer-group upstream-only the service they
neighbor 100.67.3.6 prefix-list as300cust-in paid for
neighbor 100.67.3.6 activate
neighbor 100.67.3.10 remote-as 400 Note the specific
neighbor 100.67.3.10 peer-group ixp-only per-customer
neighbor 100.67.3.10 prefix-list as400cust-in inbound filters
neighbor 100.67.3.10 activate
neighbor 100.67.3.14 remote-as 500
neighbor 100.67.3.14 peer-group bgpcust-only
neighbor 100.67.3.14 prefix-list as500cust-in
neighbor 100.67.3.14 activate
45
BGP Customers – configuring upstream
router bgp 100
address-family ipv4
neighbor 100.66.32.1 remote-as 130
neighbor 100.66.32.1 prefix-list full-routes in
neighbor 100.66.32.1 route-map upstream-out out
neighbor 100.66.32.1 activate
!
route-map upstream-out permit 10
match ip community aggregates pi transits
!
! IP prefix-list full-routes is the standard bogon
! prefix filter - or use a reputable bogon
! route-service such as that offered by Team Cymru
Aggregates, PI customers
and full transit customers
46
are announced to upstream
BGP Customers – configuring IXP peers
router bgp 100
address-family ipv4
neighbor 100.70.0.1 remote-as 901
neighbor 100.70.0.1 route-map ixp-peers-out out
neighbor 100.70.0.1 route-map ixp-peers-in in
neighbor 100.70.0.1 prefix-list AS901-peer in
neighbor 100.70.0.1 activate
neighbor 100.70.0.2 remote-as 902
neighbor 100.70.0.2 route-map ixp-peers-out out
neighbor 100.70.0.2 route-map ixp-peers-in in
neighbor 100.70.0.2 prefix-list AS902-peer in Aggregates, PI
neighbor 100.70.0.2 activate customers full transit
! and IXP customers are
route-map ixp-peers-out permit 10 announced to the IXP
match ip community aggregates pi transits ixp-access
!
route-map ixp-peers-in permit 10
set community 100:3000
47
Service Identification
p While the community set up takes a bit of thought and
planning, once it is implemented:
n eBGP configuration with customers is simply a case of applying
the appropriate peer-group
n eBGP configuration with IXP peers is simply a case of
announcing the appropriate community members to the peers
n eBGP configuration with upstreams is simply a case of
announcing the appropriate community members to the
upstreams
p All BGP policy internally is now controlled by communities
n No prefix-lists, as-path filters, route-maps or other BGP
gymnastics are required 48
What about iBGP itself?
p We’ve made good use of communities to handle
customer requirements
n But what about iBGP?
p Most Network Operators deploy Route Reflectors as a
means of scaling iBGP
p In transit networks:
n Core routers (the Route Reflectors) carry the full BGP table
n Edge/Aggregation routers carry domestic prefixes & customers
49
iBGP core router/route reflector
router bgp 100
address-family ipv4
neighbor rrc peer-group The filter to restrict
neighbor rrc descr Route Reflector Clients client iBGP to just
neighbor rrc remote-as 100 domestic prefixes
neighbor rrc route-reflector-client
neighbor rrc route-map ibgp-filter out
neighbor rrc send-community
neighbor ibgp-peer peer-group Must NOT
neighbor ibgp-peer Standard iBGP peers forget to send
neighbor ibgp-peer remote-as 100 community to
neighbor ibgp-peer send-community iBGP peers
neighbor 100.64.0.1 peer-group ibgp-peer
neighbor 100.64.0.1 activate
neighbor 100.64.0.2 peer-group rrc Allow all prefixes
neighbor 100.64.0.2 activate coming from the
! domestic network
route-map ibgp-filter permit 10 & IXP
match community aggregates subnets pi transits ixp-access bgp-cust ixp-routes
50
!
iBGP in the core
p Notice that the filtering of iBGP from the core to the edge
is again achieved by a simple route-map applying a
community match
n No prefix-lists, as-path filters or any other complicated policy
n Once the prefix belongs to a certain community, it has the
access across the backbone determined by the community
policy in force
51
Using Communities for Customers
Policy
Giving policy control to customers…
52
Customer Policy Control
p Network Operators have a choice on how to handle policy control
for customers
p No delegation of policy options:
n Customer has no choices
n If customer wants changes, the operator’s Technical Support handles it
p Limited delegation of policy options:
n Customer has choices
n The operator’s Technical Support does not need to be involved
p BGP Communities are the only viable way of offering policy control
to customers
53
Policy Definitions
p Typical definitions:
Community Action
Nil: No community set, just announce everywhere
X:1 1x prepend to all BGP neighbours
X:2 2x prepend to all BGP neighbours
X:3 3x prepend to all BGP neighbours
X:80 Local preference set to 80 on customer prefixes
X:120 Local preference set to 120 on customer prefixes
X:666 Black hole this route please! (RFC7999)
X:5000 Don’t announce to any BGP neighbour
X:5MM0 Don’t announce to BGP neighbour MM
X:5MMN Prepend N times to BGP neighbour MM
54
Policy Implementation
p The BGP configuration for the initial communities was discussed at
the start of this slide set
p But the new communities, X:5MMN, are worth covering in more
detail
n The operator in AS X documents the BGP transits and peers that they have
(MM can be 01 to 99)
n The operator in AS X indicates how many prepends they will support (N can
be 1 to 9, but realistically 4 prepends is usually enough on today’s Internet)
n Customers then construct communities to do the prepending or
announcement blocking they desire
p If a customer tags a prefix announcement with:
n 100:5030 don’t send prefix to BGP neighbour 03
n 100:5102 2x prepend prefix announcement to peer 10
55
Community Definitions
p Example: Operator in AS 100 has two upstreams. They create policy based on
previous slide to allow no announce and up to 3 prepends for their customers
56
Creating route-maps – neighbour 1
Don’t announce these
route-map bgp-neigh-01 deny 10
prefixes to neighbour 01
match ip community all-noann peer1-noann
!
route-map bgp-neigh-01 permit 20
match ip community all-pre1 peer1-pre1 Single prepend of these
set as-path prepend 100 prefixes to neighbour 01
!
route-map bgp-neigh-01 permit 30
match ip community all-pre2 peer1-pre2
set as-path prepend 100 100 Double prepend of these
! prefixes to neighbour 01
route-map bgp-neigh-01 permit 40
match ip community all-pre3 peer1-pre3
set as-path prepend 100 100 100
! Triple prepend of these
route-map bgp-neigh-01 permit 50 prefixes to neighbour 01
59
Customer BGP configuration
router bgp 600
address-family ipv4
neighbor 100.69.1.1 remote-as 100
neighbor 100.69.1.1 route-map upstream out
neighbor 100.69.1.1 prefix-list default in
neighbor 100.69.1.1 activate
!
route-map upstream permit 10
match ip address prefix-list blockA
set community 100:5010 100:5023
route-map upstream permit 20
match ip address prefix-list aggregate
p This will:
n 3x prepend of blockA towards their upstream’s 2nd BGP neighbour
n Not announce blockA towards their upstream’s 1st BGP neighbour
n Let the aggregate through with no specific policy 60
Customer Policy Control
p Notice how much flexibility a BGP customer could have with this
type of policy implementation
p Advantages:
n Customer has flexibility
n Operator Technical Support does not need to be involved
p Disadvantages
n Customer could upset the operator’s loadbalancing tuning
p Advice
n This kind of policy control is very useful, but should only be considered if
appropriate for the circumstances
61
Conclusion
62
Communities
p Communities are fun! J
p And they are extremely powerful tools
p Think about community policies, e.g. like the additions
described here
p Supporting extensive community usage makes customer
configuration easy
p Watch out for routing loops!
63
Using BGP Communities
ISP Workshops
64