Virtual Routing in The Cloud
Virtual Routing in The Cloud
Virtual Routing in The Cloud
com
Virtual Routing
in the Cloud
Arvind Durai, CCIE No. 7016
Stephen Lynn, CCIE No. 5507 & CCDE No. 20130056
Amit Srivastava
Cisco Press
800 East 96th Street
Published by:
Cisco Press
800 East 96th Street
Indianapolis, IN 46240 USA
All rights reserved. No part of this book may be reproduced or transmitted in any form or by any means,
electronic or mechanical, including photocopying, recording, or by any information storage and retrieval
system, without written permission from the publisher, except for the inclusion of brief quotations in a
review.
ISBN-13: 978-1-58714-494-3
ISBN-10: 1-58714-494-8
The information is provided on an “as is” basis. The authors, Cisco Press, and Cisco Systems, Inc. shall
have neither liability nor responsibility to any person or entity with respect to any loss or damages
arising from the information contained in this book or from the use of the discs or programs that may
accompany it.
The opinions expressed in this book belong to the author and are not necessarily those of Cisco
Systems, Inc.
Trademark Acknowledgments
All terms mentioned in this book that are known to be trademarks or service marks have been appropri-
ately capitalized. Cisco Press or Cisco Systems, Inc., cannot attest to the accuracy of this information.
Use of a term in this book should not be regarded as affecting the validity of any trademark or service
mark.
Special Sales
For information about buying this title in bulk quantities, or for special sales opportunities (which may
include electronic versions; custom cover designs; and content particular to your business, training goals,
marketing focus, or branding interests), please contact our corporate sales department at
[email protected] or (800) 382-3419.
For questions about sales outside the U.S., please contact [email protected].
www.allitebooks.com
iii
Feedback Information
At Cisco Press, our goal is to create in-depth technical books of the highest quality and value. Each book
is crafted with care and precision, undergoing rigorous development that involves the unique expertise
of members from the professional technical community.
Readers’ feedback is a natural continuation of this process. If you have any comments regarding how we
could improve the quality of this book, or otherwise alter it to better suit your needs, you can contact us
through email at [email protected]. Please make sure to include the book title and ISBN in your
message.
www.allitebooks.com
v
Ray holds a Bachelor of Science degree in Computer Science and Mathematics from the
University of Wisconsin–Madison. He is also a frequent speaker at Cisco Live events.
Matt Bollick has worked in technical marketing at Cisco for the past 19 years, running
an obstacle course of technologies, including SNA, ATM and Ethernet switching, service
provider aggregation, metro Ethernet, network management, and enterprise branch
architectures. He has also worked on a variety of products, including the Cisco 7500,
7200, LS1010, 8540, 7300, and Cisco 10K before finally settling down for the past
several years as the platform architect for the ISR series of branch routers. In his spare
time, Matt is an avid SCUBA diver in North Carolina.
vi Virtual Routing in the Cloud
Dedications
From Arvind:
I am thankful to God for everything. I would like to dedicate this book to my wife,
Monica, and my son, Akhhill, who have been extremely patient and supportive during
my long working hours. I am grateful to my parents for their blessings and for providing
me with strong values.
I would also like to thank my parents-in-law, brother and family, and brother-in-law and
family for all their support and wishes.
From Stephen:
I would like to dedicate this book to my wonderful and beautiful wife, Angela, and to
my two incredible children, Christina and Ethan. Without your love, sacrifice, and sup-
port, this book would not have been possible. Thanks for putting up with the late nights
and weekends I had to spend behind the computer and on conference calls instead of
playing games, building Legos, and doing other fun family activities.
From Amit:
I would like to dedicate this book to my wife, Reshma, my daughter, Aarushi, and my
parents. Without their love and support, I would never have been able to work on this.
I would also like to thank my parents-in-law and my entire extended family. Their love
and support have always been unconditional.
www.allitebooks.com
vii
Acknowledgments
Arvind Durai:
Thanks to my wife, Monica, for encouraging me to write my third book. She inspired
me and helped keep my spirits up all the time and provided her thoughts in multiple sec-
tions of this book. Thank you!!!
It was great working with Amit and Stephen. Their excellent technical knowledge and
passion for writing made this writing experience a pleasure. I am looking forward to
more years of working together as colleagues and friends.
Stephen Lynn:
A debt of gratitude goes to my coauthors, Arvind and Amit. Your knowledge and dedi-
cation to this project are appreciated more than you will ever know.
Amit Srivastava:
Special thanks to Arvind and Stephen, from whom I learned a lot while writing this book.
I look forward to their continued support.
Our Acknowledgement
Many people within Cisco have provided feedback and suggestions to make this a
great book. Thanks to all who have helped in the process, especially Ray Blair and Matt
Falkner, for providing insightful input during the proposal process. A special thank you
goes to our technical editors, Ray Wong and Matt Bollick, for your technical accuracy
and insight into the technologies. Special thanks to Dimitris Vlassopoulos for providing
his NSO lab setup and sharing his insights!
A big thank you goes out to the production team for this book, Brett Bartow, Ellie Bru,
and Tonya Simpson, who have been incredibly professional and a pleasure to work with,
and for making this book possible.
viii Virtual Routing in the Cloud
Contents at a Glance
Introduction xv
Index 319
Reader Services
Register your copy at www.ciscopress.com/title/9781587144943 for convenient access
to downloads, updates, and corrections as they become available. To start the registra-
tion process, go to www.ciscopress.com/register and log in or create an account*. Enter
the product ISBN 9781587144943 and click Submit. Once the process is complete, you
will find any available bonus content under Registered Products.
*Be sure to check the box that you would like to hear from us to receive exclusive
discounts on future editions of this product.
www.allitebooks.com
ix
Contents
Introduction xv
Software-Defined Networking 30
Open Networking Foundation 31
OpenDaylight Project 32
Network Function Virtualization 33
OpenStack 34
Summary 35
www.allitebooks.com
xi
Hybrid Kernels 64
The Cisco IOS Kernel 64
The Boot Process 66
Linux Memory Management 69
Linux Swap Space and Memory Overcommit 69
Linux Caching 71
Understanding Hypervisors 71
How Does a Hypervisor Compare to an Operating System? 72
Type 1 Hypervisor Design 74
Monolithic Architecture 74
Microkernel Architecture 74
Core Partitioning 75
ESXi Hypervisor 75
Architectural Components of ESXi 75
The VMkernel 75
Components of the VMkernel 76
Processes Running on the VMkernel 77
Device Drivers 78
File Systems 79
Management 80
KVM 82
Architectural Components of KVM/QEMU 84
Guest Emulator (QEMU) 85
Management Daemon (Libvirt) 88
User Tools (virsh, virt-manager) 89
Hyper-V 91
Xen 92
Summary 94
www.allitebooks.com
xiii
Index 319
■ Boldface indicates commands and keywords that are entered literally as shown. In
actual configuration examples and output (not general command syntax), boldface
indicates commands that are manually input by the user (such as a show command).
■ Braces within brackets ([{ }]) indicate a required choice within an optional element.
www.allitebooks.com
xv
Introduction
In today’s business environment, enterprise customers are under more pressure than ever
to innovate and adapt to new challenges and market conditions. Enterprises want to
focus their investments on their core business while reducing IT spending.
The cloud offers enterprise customers many benefits, such as lower costs and flexibility.
The cloud’s elastic model enables a company to increase and decrease infrastructure
capacity on demand. The usage-based model offered by the cloud helps governments
and enterprises reduce costs while increasing business agility by moving applications to
the cloud and consuming infrastructure resources from the cloud. This leads to enter-
prises looking at consuming network and IT services from the cloud rather than investing
in in-house operations.
The surge in applications and IT service consumption moving to the cloud highlights the
need for evolved technologies and network elements in the cloud that offer security
and visibility to help businesses with performance and compliance verification. Evolved
networks and network services enable the provider to offer cloud services with security,
performance, and availability. The Cisco Cloud Services Router 1000V (CSR 1000V) is a
fully virtualized software router that offers a platform for enterprises to extend the data
center to the cloud and to enforce their policies in the cloud.
The Cisco CSR 1000V provides a transparent solution for extending IT services into
provider-hosted clouds. The solution offers a rich set of features, including VPN, fire-
wall, Network Address Translation (NAT), application visibility, and WAN optimization.
These functions allow enterprise and cloud providers to build highly secure, scalable,
and extensible cloud networks. In addition, the Cisco CSR 1000V supports a rich set of
application programming interfaces (API), providing robust integration into software-
defined networking (SDN) for automated provisioning of these networks and network
services and allowing simplified management and orchestration, which help in driving
down costs further.
Networks inherently carry vast amounts of information, including user locations, device
capabilities, topologies, and end-to-end performance characteristics. When exposed
appropriately through well-defined APIs, such information can be consumed by cloud
applications to fine-tune and customize their efficient delivery. The future holds the
promise of increasingly rich application–network interactions.
The primary objective of this book is to simplify design aspects and architectural details
in a unified resource, augmenting Cisco’s existing collection of installation and configura-
tion guides for various cloud-related products and solutions. This book covers the key
xvi Virtual Routing in the Cloud
This book also explains network design and deployment scenarios for the Cisco CSR
1000V, which influence its pivotal role in the cloud environment. Furthermore, the book
distills how intelligent networks help providers simplify cloud service management and
reduce costs through efficient scaling and optimized capacity utilization. This book
provides architectural knowledge that contextualizes the roles and capabilities of these
advanced networks and network services, along with discussions of design factors essen-
tial for their insertion into cloud services:
■ The book introduces the readers to the cloud and provides an overview to different
types of cloud operational environments, including a prelude to the evolution of
virtual routers.
■ The book covers the details of the operating systems and hypervisors on which vir-
tual routers run. It provides details pertaining to the operational aspects of virtual
routing.
■ The reader is introduced to the architecture and software design of the Cisco CSR
1000V virtual router. The reader is subsequently introduced to a comprehensive set
of APIs that can be leveraged by SDN.
■ The book focuses on different designs and use cases and configuration examples for
routing, secure extension of enterprises to the cloud, and VM mobility. It illustrates
how the CSR 1000V addresses the challenges that an architect faces in migrating
toward the cloud.
■ This book covers the different management techniques available to simplify opera-
tional and monitoring aspects of cloud services.
This book also caters to the next generation of cloud network operators to implement
enterprise features in the cloud, leveraging the CSR 1000V.
After reading this book, you will have a better understanding of the following:
www.allitebooks.com
xvii
■ Use cases for the CSR 1000V to simplify enterprise routing in the cloud
www.allitebooks.com
Chapter 1
Introduction to Cloud
This chapter introduces the concept of cloud computing. It provides an overview of the
evolution from the physical data center to the concept of virtualization within the data
center. It describes the various cloud models available and how virtualization is enabling
the present-day transition to the cloud. The chapter illustrates multitenant data center
designs and introduces the concept of software-defined networking (SDN).
Traditionally, data centers have relied heavily on physical hardware. The physical size
of a data center is defined by the infrastructure and the space in which the hardware is
hosted. These characteristics in turn define the amount of data that can be stored and
processed in that location.
In the early second half of the 20th century, the early predecessors of the data center
were large mainframes that occupied entire rooms. The first mass-produced mainframe
computer was delivered in the 1950s. This machine, the UNIVAC-I mainframe com-
puter, was the size of a one-car garage 14 feet by 8 feet by 8.5 feet high, and it weighed
29,000 pounds. It could perform 1905 instructions per second. Despite its massive size,
the UNIVAC was only a fraction as powerful as today’s computers. The A9 processor
in an iPhone 6s is more than 6 million (106) times more powerful than the UNIVAC. In
the days when IT decisions revolved around the mainframe, everything from hardware,
operating system, and applications that ran on top of it had to be made on an enterprise
scale. However, all of these components ran within one device, offering limited scalabil-
ity and flexibility.
2 Chapter 1: Introduction to Cloud
During the 1980s, the computer industry experienced a transition that brought
microcomputers and personal computers into wider use in business. This transition
accelerated through the 1990s as microcomputers began filling the old mainframe
computer rooms and functioning as servers. These rooms became known as data
centers. Gone were the mainframes that filled entire rooms. In their place were rack-
mountable servers.
In the early stage of data center evolution, enterprises built distributed data centers based
on their business and IT requirements. As business opportunities continued to expand,
enterprises required better data center services and holistic management strategy. The
demands for higher scalability, increased uptime, and higher data bandwidth made it hard
to satisfy business requirements with the distributed data center model.
These requirements drove enterprises to consolidate their data centers into centralized
locations. They constructed dedicated facilities with dedicated cooling infrastructure,
redundant power distribution equipment, and high-speed network fabric capable of
supporting tens of thousands of servers to enable businesses to host a range of services.
In the process of centralizing the data centers, distributed servers and storage devices
were integrated, along with data and applications. The integration of the systems and data
storage streamlined the infrastructure of the data center, allowing for greater sharing and
utilization of system resources. The exercise also consolidated many arduous processes,
such as system management, disaster recovery, and data backup, and contributed to
enhanced IT services with unified management and increased business continuity.
The latter years of the 2000s saw a shift in the data center paradigm. Studies have shown
that data center resources are underutilized; on average, these studies have found,
data centers are only 20% utilized. Much of that underutilization is due to application
silos with dedicated network, processing, and storage resources. Enterprises have been
increasingly pressed to reduce IT spending while increasing business agility to launch
new services. At the same time, users have increased expectations to be able to access
information from anywhere, anytime, and using any device. These forces together have
ushered in the era of cloud computing.
Today, data centers leverage virtualization technologies and cloud services to improve
resource utilization and increase business agility and flexibility to quickly respond to
rapidly changing business requirements. A variety of data center service models are
available to enterprises. Software as a Service (SaaS), Platform as a Service (PaaS), and
Infrastructure as a Service (IaaS) are data center cloud offerings that offer different
tiers of service-level considerations. After reading this chapter, you will have a better
understanding of these concepts.
www.allitebooks.com
Evolution of the Data Center 3
■ Facility—This block is the building that houses the data center. It provides robust
and redundant power, cooling, and security for the data center. Power consumption
and efficiency are top concerns facing data center managers because the power bill
is generally the largest operating cost. It’s also critical to ensure that a facility has
sufficient power, cooling, and space for future growth. When data centers are built,
they are designed for a specific power budget and cooling capacity. However, over
time these needs tend to increase, and the data center runs out of capacity even
though there is plenty of available space.
■ Storage—This functional block provides data storage services to the data center.
The servers in the compute block either connect to a dedicated storage-area net-
work (SAN) for block-based storage or use network-attached storage (NAS) systems
for file-based storage.
Tip Raw volumes of storage are created in block-based storage, and each block can be
controlled as an individual hard drive. Block-based storage can be used to store files, as
well as special applications such as databases. This type of storage format is much more
efficient and reliable than file-based storage.
File-based storage, the most commonly used type of storage, stores files and folders and
provides the same visibility to all the users accessing the system. This type of storage is
very inexpensive to maintain, and it’s simple to integrate access control with a corporate
directory. However, it is less efficient than block-based storage.
Traditional architectures for data centers leverage discrete tiers of servers for comput-
ing, where each tier uses dedicated servers to execute specific functions. The most well-
known example is the three-tiered architecture, which consists of web, application, and
database servers.
4 Chapter 1: Introduction to Cloud
While the discrete multitiered architecture served as the de facto data center design in
the early 2000s, rapid IT expansion and a desire to quickly roll out new applications
highlight these shortfalls of the architecture:
■ Lack of IT agility
Tip Unstructured data refers to information that does not have a predefined data
model. This type of data is often text centric but may include data such as dates,
numbers, and facts. With a data structure that is not of a predefined format, it is hard for
traditional programs to compare and extract the stored information. This unstructured
data is very prevalent in the age of Facebook and other social media sites. Techniques
such as data mining and text analytics are used to provide different methods for finding
patterns in otherwise unstructured data sets.
Until very recently, storage, computing, and network resources were separate physical
components that were managed separately. Even applications that interacted with these
resources, such as system management and monitoring software, had different security
and access policies.
As the word suggests, virtualization involves creating a virtual version of something that
can be used without it being physically present where it is required. Take, for example, a
server hosting a website. A small business owner, Bob, makes money by selling furniture
online. He has a small server that can host his website and cater to a couple hundred
hits—or shoppers accessing his website—per minute. With Thanksgiving coming, Bob
expects his business to surge, along with the number of hits on his website. Today Bob
has three options:
■ He can buy a larger server for his website that, he is pretty sure, won’t be required
after the Thanksgiving rush.
■ He can do nothing and allow his service to deteriorate during the peak season when
his web users most need it.
■ He can go online and rent a server for his computing needs for the duration he
thinks it will be required. This server is actually a virtualized server sharing hardware
www.allitebooks.com
Introduction to Virtualization in the Data Center 5
resources with virtual servers from other companies that are also trying to handle
the Thanksgiving rush.
You don’t have to be Sherlock to figure out the best option here. Virtualization is the
best option for Bob. He can rent a server and share the compute resource for the dura-
tion of the surge. This way, he doesn’t burn his dollars when his computing requirements
decline, and he gives his users the service they need for the duration of the surge.
Evolution of Virtualization
Let’s look at how our present-day virtualization technology evolved. During the 1950s
and 1960s, the mainframe was the workhorse when it came to processing data or solving
complex equations. Back in the day, the mainframe was one big computer that had all
the answers. The world then moved on to a distributed architecture, with all computing
requirements serviced by different components in the system. Computing, storage, and
networking requirements were serviced by different entities. In some ways, virtualization
technologies bring back many of the shared resource models from the mainframe days.
Today your service provider virtualization cloud can cater to your computing, storage,
and networking requirements. Years of evolution have brought us to a model that is
actually similar to the mainframe architecture.
■ Distributed storage, which offers high availability for data and faster failure
recovery
■ Cost-effectiveness because you pay for the capacity you need when you need it
The lowest layer is the physical layer you wish to virtualize. It can be a server blade,
a storage device, a network of storage devices, a network, or any physical entity you
wish to virtualize. On top of that is a virtualization layer, which is a piece of software or
hardware that controls access to the physical entities and presents these entities to the
instances running on it to ensure efficient utilization of the hardware and fair distribu-
tion of resources. The virtual server, or virtual machine, instances run on top of this piece
6 Chapter 1: Introduction to Cloud
Virtualization is a powerful concept. The lowest physical layer described might not be a
single device. You can extrapolate the same concept to a group of devices that are net-
worked together. You can also run virtualization with the virtualized instances (which is
called nested virtualization). The core concepts, however, do not change.
■ Server virtualization
■ Storage virtualization
■ Network virtualization
Server, storage, and network virtualization together constitute a data center service.
Virtualization of these elements takes care of service virtualization.
Server Virtualization
The instruction sets for the x86 architecture were initially developed for embedded
systems and small computers. However, this architecture grew in features and capabil-
ity over the years. The original 32-bit architecture had a restriction of less than 4GB of
memory per system. However, when AMD burst onto the scene with its Opteron 64-bit
www.allitebooks.com
Introduction to Virtualization in the Data Center 7
version of the x86, it introduced essentially limitless RAM address space. 264 bytes
of RAM will give you memory on the order of 1 billion gigabytes. Throw in a 64-bit
address bus, and you get much better I/O than with a 32-bit version.
Servers—or any hardware, for that matter—equipped with these processors are often
underutilized if they run an application on a single operating system. Virtualization
enables you to exploit the full capability of these modern-day masterpieces.
Without virtualization, an operating system manages the hardware and schedules it for
applications. Software is tightly coupled to the hardware because all of the hardware
resources are managed by a single OS. You run the applications that are supported by
that particular OS only.
With virtualization, a hypervisor runs on the server and provides logical isolation
between virtual instances sharing the same physical hardware. These virtual instances
are logically isolated from one another and behave as if they are running on their own
physical hardware. You run operating systems to manage an instance—that is, the view
the hypervisor gives you of the hardware resource (memory, CPU, networks, and so on).
Typically, instance is used as a synonym for a virtual machine.
With each virtual machine getting its share of the hypervisor-managed hardware resourc-
es, server virtualization ensures that you get the most out of your hardware.
The hypervisor virtualizes the x86 architecture. Memory, CPU, network cards, and other
physical resources are presented to the instances running on the hypervisor. The instanc-
es get their share of the physical resource from the hypervisor (vMem, vCPU, vNIC,
and so on). An instance runs an operating system (that is, a guest OS) using these virtual
resources. Multiple such instances can run on a single hypervisor. Applications run on
this guest OS as tenants, and the virtual resources given to this instance are scheduled by
the guest OS. Figure 1-2 shows the components of server virtualization.
Apps
Guest OS
Hypervisor
Physical Resources
x86 Architecture
www.allitebooks.com
Introduction to Virtualization in the Data Center 9
View of
vMEM vCPU vNIC vDevice vMEM vCPU vNIC vDevice
Consumed
Resources
Chapter 3, “Hypervisor Considerations for the CSR,” covers in detail the different
hypervisor technologies used for CSR in a virtualizated environment.
Storage Virtualization
In its crudest form, storage involves a device connected to your computer storing all
your data. The computer writes and retrieves data from this storage device. The amount
of data that can be written onto the storage device is limited by the storage capacity
of this device. Needless to say, your computer connected to this storage is dependent
on it for all its storage requirements (assuming that the computer is not connected
to any other storage network). It is evident that this basic form of storage has some
shortcomings:
■ The complexity of the storage media must be exposed to the applications/OS or the
device controller.
■ Only one computer can write data to or read data from a storage device.
10 Chapter 1: Introduction to Cloud
Storage virtualization combines a bunch of networked storage devices into what appears
to the users as a single storage entity. Specialized hardware or software manages the
complexity of a storage network and gives each device wanting access to the storage a
view of being directly connected to the storage media.
The act of hiding, abstracting or isolating the internal functions of a storage (sub)
system or service from applications, host computers or general network resources
for the purpose of enabling application and network-independent management of
storage or data.
A storage system can be on a single disk or on an array of disks in any form, and
specialized hardware or software manages the physical storage. For example, RAID
(redundant array of inexpensive [or independent] disks) combines multiple physical disks
into a single logical unit to be used by applications. This gives the user data redundancy
because the data is stored on physically different devices. Two types of RAID are
possible: hardware RAID and software RAID. With hardware RAID, you set up the
multiple disks to function in a RAID configuration. The hardware RAID controller
presents these multiple disks to the operating system as a single disk. The operating
system is not aware of the multiple disks in RAID configuration. With software RAID,
the OS does the heavy lifting. The operating system knows about the RAID storage and
works accordingly.
NFS and RAID are common examples of storage virtualization. Today, NFS and RAID
are so popular that people use them without even being aware that they are using a form
of storage virtualization. Figure 1-5 shows the components of storage virtualization.
Today, storage virtualization is commonly associated with a large SAN (storage area
network) managed by a hardware or software virtualization layer that presents this vast
storage as a single block to the servers requiring storage. The virtualization software and
hardware sitting between the storage and the servers make the applications completely
oblivious to where their data resides. This eases the administrative task because admin-
istrators can manage the storage as if it were an entity. This simplifies the operation and
management of the storage system, offering scalability when there is a need for addi-
tional storage.
www.allitebooks.com
Introduction to Virtualization in the Data Center 11
Server Server
Physical or virtual servers using virtualized
storage as if it were a local disk.
Virtual Virtual
Disk Disk
As is the case with server virtualization, storage virtualization helps enterprises reap a
number of benefits:
Network Virtualization
Traditionally, networks have been meshes of routers and switches that forward data
packets and offer services such as firewalls, access control, QoS, and security. Routers
and switches are hardware devices that run specialized software. This combination of
hardware and software provides network services and packet forwarding. But highly
skilled engineers are needed to manage these resources and ensure that they run correctly.
Cisco and other network equipment manufacturers have long shipped routers and
switches loaded with their proprietary software. The network engineering community
felt the need to separate the control and data planes within this proprietary software
architecture. This simply meant decoupling the intelligent (control plane) piece of soft-
ware from the packet forwarding (data plane) logic. Because packet forwarding is a task
that requires speed, it made sense to have it done in hardware, and changing this meant
software could be separated within the router and switch operating systems. The control
plane becomes a place where all the routing and switching decisions are made. Network
devices have their data plane hardware programmed by the control plane to forward
packets accordingly. This provided a good performance and manageability uplift to
previous monolithic software architectures. Now a piece of software essentially pro-
grammed the hardware for network operations. This software to control hardware fea-
tures was limited within a network device.
Well, why should the software to control the hardware features be limited to a single
networking device? Why not do what the server and storage virtualization folks have
www.allitebooks.com
Introduction to Virtualization in the Data Center 13
done and completely decouple the hardware from the service? This is exactly what
network virtualization does: A piece of software manages the hardware and presents the
applications’ virtual resources and services derived from the hardware. Much as in server
virtualization, the OS is presented with vMem, vCPU, and vNIC, and network virtualization
creates virtual networks with vSwitches, vRouters, and virtual services like vFirewalls.
The VLAN was the first and is by far the most popular virtualization within a network.
It was revolutionary in that it could create broadcast domains with a connected Ethernet
circuit, and during the early 1990s it meant not buying expensive networking equipment.
In 1996, Ipsilon Networks introduced a flow management protocol. The Ipsilon product
IP switch ATM 1600 used Asynchronous Transfer Mode (ATM) with IP routing. It
faded away into oblivion. Cisco said “Why restrict this to ATM?” and went ahead with
the proposal of tag switching. The Cisco proposal was originally called label switching,
14 Chapter 1: Introduction to Cloud
and after the IETF standardized it, Multiprotocol Label Switching (MPLS) was born.
MPLS was developed to give a switching flavor to routed packets and was envisioned
to provide faster networks. However, it was empirically found not to have performance
benefits. The value was (and still is) in the applications built around it, such as Layer 2
and 3 virtual private networks (VPN). MPLS provided a label-switched path within the IP
network that helped build Layer 3 and 2 VPNs around it.
Within a Layer 3 router, there was a need for multiple routing tables. This not only
provided additional security but also, to a certain extent, eliminated the need to have
multiple routing devices for routing segmentation. Having multiple instances of a routing
table within the same router is called virtual routing and forwarding (VRF). VRF with
MPLS gives us our present-day Layer 2 and 3 VPNs.
With network virtualization, servers running virtual machines can now reap the benefits
of virtualized networks, including automation, manageability, and decoupled software
and hardware.
www.allitebooks.com
Introduction to Virtualization in the Data Center 15
Management
WAN
Load Optimiza-
Firewall Balancer Router
tion
VMs
Virtualization: vSwitch
Service Virtualization
Service virtualization is mainly used to virtualize an asset that is difficult to procure
physically during the software development life cycle of a product. Development and
testing teams can use this type of virtual asset instead of waiting for the actual physical
resource to be available.
16 Chapter 1: Introduction to Cloud
For example, consider a small software organization where the development and quality
assurance teams are building an application that requires access to a mainframe in the
back end. It is not cost-effective for the organization to get the engineers access to an
actual mainframe. Even if the organization decides to procure a mainframe, engineering
will have to wait until the resource is available and set up for them to use. The
organization can instead use service virtualization to give the engineering team a virtual
asset. This solution provides a number of benefits:
■ It gives the organization a cost-effective way to procure the asset for its engineers.
■ Engineering is more agile and does not have to wait for the asset to be made
available.
Keep in mind that service virtualization is not the same as network service virtualization.
With service virtualization, the idea is to emulate a specific component within a modular
application to provide software development and testing access to components that
are required to test an application. Network service virtualization, on the other hand,
involves offering a particular network service, such as a firewall, QoS, or DNS, as a
service to a group of authenticated users.
Note A software router does not fit the device-based network virtualization category
exactly. It is a network element but not a complete network. It actually better fits the
server virtualization category because it involves virtualizing the server to provide virtual
handles for memory, CPU, and I/O port, and running the router software over the
virtualized resources.
As virtualization technologies mature and people get creative with deploying them, it will be
increasingly difficult to fit technologies exactly into previously defined categories. The important
thing here is to use the available virtualization tools to enhance a deployment and make it more
cost-effective.
While multitenancy means that some infrastructure is shared, the degree of sharing and
at what infrastructure layer depend on the type of service model. The highest degree
of multitenancy sharing is SaaS. In this model, all customers are served from a single
infrastructure, and every component is shared, all the way down to computing, storage,
www.allitebooks.com
Introduction to Virtualization in the Data Center 17
The concept of multitenancy can be characterized at the different layers within data
center functional blocks:
The concept of virtualization is not new. In 1970, IBM introduced the mainframe
virtualization system VM/370. It was one of the first examples of virtualization
technology applied to a computing environment. The system provided multiuser access
to seemingly separate and independent IBM VM/370 computing systems. Figure
1-7 shows IBM VM/370 virtualization. The control program (CP) together with the
Conversational Monitor System (CMS) formed the virtual machine environment. CP
was the resource manager of the system, and it was similar to the concept of today’s
hypervisor. The CP created virtual machines with guest operating systems.
VM/CMS DOS/VSE
... MVS
Mainframe Hardware
Virtualization can extend beyond computing service and into the entire data center.
Data center virtualization has enabled cost saving by reducing power consumption and
cooling costs. At the same time, it has improved asset utilization by increasing efficiency
18 Chapter 1: Introduction to Cloud
OS OS OS OS
An organization using IaaS still needs to have developers design applications and
operations to manage operating systems in these hosted systems. The cost savings are
huge because the organization doesn’t own the physical device. You can think of this
as renting a home instead of buying it. The consumer’s rent is based on an on-demand
model, and the service is provided based on a best-availability basis. With IaaS, it is
possible to use a specific set of network, computing, and storage resources in the cloud
www.allitebooks.com
Introduction to Cloud Services 19
with a specific service level guarantee. These resources can be leveraged based on a
contractual service level agreement (SLA) that is determined and negotiated between
the consumer and provider. Companies tend to prefer this model because the end user
experience can be managed within contractual boundaries. In this model, scaling can be
provided within a limit for spurts in usage patterns.
The consumer using IaaS bears the licensing cost. The cost of using an IaaS infrastructure
depends on the contractual usage of computing, memory, and storage. Another cost to
be considered is data transfer to multiple instances for the same enterprise (IP address
services, VPN services, security, monitoring, and so on).
Note The National Institute of Standards and Technology (NIST) defines IaaS as follows:
Using PaaS, consumers can leverage middleware services without worrying about
alignment with key hardware and software elements. A developer can get access to
the complete stack of developer tools. PaaS in public cloud environments requires the
clients to use the tool set offered by the provider for application development.
The capability provided to the consumer is to deploy onto the cloud infrastructure
consumer-created or acquired applications created using programming languages,
libraries, services, and tools supported by the provider. The consumer does not manage
or control the underlying cloud infrastructure including network, servers, operating
systems, or storage, but has control over the deployed applications and possibly
configuration settings for the application-hosting environment.
20 Chapter 1: Introduction to Cloud
The capability provided to the consumer is to use the provider’s applications running
on a cloud infrastructure. The applications are accessible from various client devices
through either a thin client interface, such as a web browser (e.g., web-based email), or
a program interface. The consumer does not manage or control the underlying cloud
infrastructure including network, servers, operating systems, storage, or even individual
application capabilities, with the possible exception of limited user-specific application
configuration settings.
■ Public—A public cloud is a multitenant environment that hosts multiple users who
pay for the services they use. The users cannot see any of the other tenants utilizing
the same environment.
■ Private—The private cloud environment has automated provisioning of services for
a single user in a hosted or on-premise location. The usage space is available to only
a single organization. A private cloud reduces the regulatory risk around data secu-
rity because it has a single-tenant deployment model.
■ Hybrid—The hybrid cloud environment, as the name suggests, is a combination of
private and public cloud environments. An organization that leverages two or more
cloud infrastructures, regardless of whether they are private or public, needs to
bind this infrastructure together with standard technology that enables application
portability. By doing so, the organization gains elasticity of the public cloud infra-
structure and can contain data ownership with security ingrained in the asset alloca-
tion model.
www.allitebooks.com
Introduction to Cloud Services 21
The three cloud models can be leveraged for the three service models described earlier
as shown in Figure 1-9.
Deployment Models
Services
Characteristics
■ Elasticity—The user can scale storage, network bandwidth, and computing based
on user demand.
■ Measured services—The user can monitor flow based on consumer billing and has
better management and reporting ability.
22 Chapter 1: Introduction to Cloud
The cloud service model is completely different from a web-hosting environment, which
usually leverages a hosting portal. For the deployment of these discussed features in the
cloud, a stacked framework, such as the Cisco Domain 10 framework (see Figure 1-10),
is necessary.
9. Security * Compliance
8. Applications
Automation * Orchestration 3
Financials
6
7. Platform
Service
SaaS
Catalog
2. Abstraction and Virtualization 5
PaaS
1. Infrastructure * Virtualization
IaaS
Customer
Computing Storage Network Interface
4
Cisco covers the stacked overview of cloud design in its Domain 10 framework. It
is essential for you to understand the various domains to develop an effective cloud
framework; the following sections provide details on the domains. Again, this is an
example for understanding building blocks that must be considered for cloud capability.
The Cisco Domain 10 framework provides a cloud architect focus area and details the
dependencies between different blocks of the framework. Data center infrastructure
for the cloud needs rapid provisioning service, and this requires an automation and
orchestration framework to be aligned with computing, network, and storage domains.
Good orchestration ties to the user catalog to speed up provisioning. The user catalog
is a view for the user to provide available services and used services. The cloud has a
pay-as-you-go model, so it is important for user catalog and finance to be integrated
into a portal used for a customer interface. PaaS and SaaS are aligned to a more mature
model of the cloud infrastructure. Security and governance is the top layer that needs to
encompass all the other domains. The Domain 10 framework provides a sequential way
to build a mature cloud model.
www.allitebooks.com
Introduction to Cloud Services 23
■ Computing:
■ The platforms considered should have good management and should simplify
deployment in the virtualized environment.
■ Network:
■ The platforms considered should provide ease of integration of security for the
tenants.
■ Storage:
■ The platforms considered should provide a single network with the flexibility
to deploy Fibre Channel or IP-based protocols at any point in the path between
server and storage.
jobs. Workflow templates link multiple domains and help in provisioning new services at
the data center.
The customer interface portal should link up with Domains 5 and 6, the service catalog
and financials, so the users have a complete view of the services leveraged, the services
available, and billing.
The application portion is covered under SaaS. This model is commonly used for non-
core enterprise-centric applications such as WebEx and sales force.
■ Threat defense—You should protect the asset resources and deploy efficient
monitoring to detect attacks from internal or external sources.
www.allitebooks.com
Introduction to Cloud Services 25
■ Visibility—You should view each security domain and provide insight for
compliance management. The key goal is to simplify operational visibility into
various security zones and simplify compliance reporting.
After cloud adoption, the governance process should be a single thread for computing,
storage, and network as a converged infrastructure, as shown in Figure 1-11.
New Model
Governance
Converged Infrastructure
Governance
Tools
Infrastructure
The traditional model of governance uses separate tools and processes for computing,
storage, and network. These tools line up to a separate governance model that is
confined to the team managing the respective domain. Governance needs two aspects:
tools and teams responsible for the converged infrastructure. The team looking at
and managing this data also must be collectively representing all the domains of
the converged infrastructure. A core team that includes members and management
representatives from each domain ties the combined governance. This model still
leverages separate tracks of tools, processes, and governance. A data center user has
a unified view of all the domains under a single service umbrella. This unified view is
difficult to manage with the traditional model. Mature cloud governance leverages a
consolidated tool set to view and manage all the domains.
There are two main categories for connecting to the cloud provider:
Accessing the Internet via a VPN has different options, including the following:
■ Hardware VPN access—The customer accesses the cloud provider via VPN tunnel
from the enterprise data center to the VPN edge gateway located at the cloud
provider’s network. After decryption at the VPN edge gateway, the user traffic
traverses to the respective virtual data center (VDC) hosted in the cloud. In this
model, the enterprise can reuse the existing hardware for VPN function and Internet
connection to access the cloud provider. Bandwidth and throughput limitation
needs to be factored here. Figure 1-12 shows the hardware VPN option.
www.allitebooks.com
Introduction to Cloud Services 27
Tenant
Internet Tenant
Space
Tenant
Space
Ent Space
WAN
■ Software VPN access—Software VPN access is popular for small groups of devel-
opers using the cloud infrastructure. To connect to the assets that are being hosted
at the cloud provider, a VPN connectivity option can be leveraged. VPN high avail-
ability is the customer’s responsibility. As shown in Figure 1-13, the termination of a
software VPN is inside the VDC or the tenant zone assigned for the entity.
Enterprise
Network Internet Tenant
Tenant
Space
Tenant
Space
Enterprise Space
WAN
Tenant
DC1 Dedicated Tenant
Space
MPLS L3VPN/ Tenant
Space
L2VPN Space
or
Leased Lines
DC2
Future peer access options will evolve, with close partnerships between the service
providers and cloud providers. The partnerships will enable end users to leverage
geographic cloud provider locations to offload traffic from service providers to a cloud
provider’s localized data centers. This service will benefit large enterprises with global
presence. The enterprise will get better transport SLA in the contractual agreement for
application access through this access approach. The cost of maintaining a transport
infrastructure will be offset to the cloud provider and service provider, thereby reducing
the operating expense for the enterprise customer.
www.allitebooks.com
Introduction to Cloud Services 29
■ Interoperability:
■ In many cases, the enterprise application needs to implement changes before
getting hosted in the cloud space. A common change such an application
needs to adopt is re-IP addressing the application stack. This is common for
organizations adopting the IaaS model.
■ Provider lock-in should be considered for an enterprise planning to adopt
cloud services.
■ Traditionally practiced provisioning standards need to align with cloud
provisioning standards.
■ Visibility:
■ It is also important to manage the operational aspect. The enterprise strives for
operational visibility in transport and security spaces in the cloud environment.
Some of these parameters need to be mentioned in the contractual agreement
with the cloud provider.
The challenges of cloud adoption can be mitigated by a good cloud adoption strategy.
These are the key elements of such a strategy:
■ Scalability requirements
■ Application profiling
■ Strategy to leverage cloud computing (like IaaS) versus cloud services (like SaaS)
■ Governance and business service level objectives (SLO) for vendor selection and
SLAs to be considered by the provider
■ Visibility to the asset elements managing the enterprise tenant in the cloud
30 Chapter 1: Introduction to Cloud
Enterprise architects prefer having control of the services in their tenant space within the
cloud infrastructure. The concept of network function virtualization (NFV) comes up
here. NFV elements are prevalent in IaaS cloud services. NFV brings a simple concept of
implementing network service elements in software such as routing, load balancers, VPN
services, WAN optimization, and firewalls. This is possible due to the new capability
of provisioning memory and server facility to the network service elements. The
provisioning of the network services is aligned with server elements. The NFV elements
can be automated in the same workflow related to the application services. This enables
faster provisioning of service with one orchestration device for network and application
services. The data center design with virtual network services reduces the complexity
of placing firewall or load balancing services because they are now closer to the asset.
These virtual services enable an enterprise to launch these capabilities in an on-premises
data center or in the provider cloud.
Software-Defined Networking
Most people understand that software-defined networking (SDN) is a new paradigm for
networking that will foster agility and innovation in network services. However, trying to
get a concise definition of SDN is like the old story about asking a group of blind men
to describe an elephant when each has a different perspective of the animal. In the story,
a group of six blind men encounter an elephant and try to learn what it is. Each of the
men feels a different part of the elephant, and they all therefore describe the elephant a
bit differently because they are influenced by their individual experiences:
■ The first man approaches the elephant’s side and believes it to be a huge wall.
■ The second man, feeling the legs, believes they are trees trunks.
■ The third man reaches out to the tusk and believes it to be a spear.
■ The fourth man, who finds the elephant’s trunk, believes it to be a snake.
■ The last man, who happens to seize the swinging tail, believes it to be a rope.
The moral of the story is that all the men are partly right, but all of them are also
completely wrong. People can interpret SDN differently depending on their technology
perspective. For a network administrator, SDN may mean automation and orchestration
to simplify network operation. An architect may have a big-picture view of the network
and focus on a controller-based protocol like OpenFlow. Here are some common
interpretations of SDN:
www.allitebooks.com
Software-Defined Networking 31
The physical separation of the network control plane from the forwarding plane,
and where a control plane controls several devices.
ONF focuses on the use of the OpenFlow protocol to drive the decoupling of
network control and the forwarding function. OpenFlow is a standard-based
communication interface between the controller and the forwarding layer of the
SDN network. A controller defines a path that data packets traverse and programs
the forwarding information into the network devices through OpenFlow application
programming interfaces (API). Figure 1-16 shows a logical view of OpenFlow network
programmability.
Northbound API
OpenFlow Controller
Server
OpenFlow began as a Stanford University PhD student’s project call Ethane. Ethane was
intended to centrally manage global policy using a flow-based network controller for
network security. The idea led to what is now known as OpenFlow. Stanford University
released OpenFlow version 1.0 as OpenFlow Switch Specification in 2009. It formed
the basis for the subsequent releases of the protocol. ONF released OpenFlow ver-
sion 1.2 in 2012. It was the first release of a specification by ONF after ONF formally
32 Chapter 1: Introduction to Cloud
charted the project. Since then, ONF has released multiple versions of updates to the
OpenFlow standard.
A network administrator can leverage OpenFlow to define how data traffic flows
through a network infrastructure based on criteria such as bandwidth requirements,
application usage patterns, and resource requirements. OpenFlow gives network
operators very granular control while enabling the network to respond to real-time
changes at the application and session levels.
Most of the time when SDN is discussed, OpenFlow is used as the interface to program
the forwarding behavior of network devices. ONF views the OpenFlow protocol as an
enabler for SDN and says it has the benefit of increasing interoperability of network
equipment across a multivendor environment. However, there are alternatives to the use of
OpenFlow for programming the network infrastructure, such as Network Configuration
Protocol (NetConf) and Extensible Messaging and Presence Protocol (XMPP).
OpenDaylight Project
Organizations such as ONF are giving definitions for SDN that focus on the separation
of the control plane and forwarding plane of the network, but the broader interpretation
of SDN is that it’s a framework that provides programmability to the network
infrastructure. SDN enables IT agility, allowing network administrators and engineers
to respond quickly to changing business requirements. Through SDN, application
developers can leverage infrastructure as a platform to easily instantiate and integrate
network services and applications through the use of APIs and scripting languages.
www.allitebooks.com
Software-Defined Networking 33
OpenDaylight Hydrogen is the first release of the modular open source SDN platform.
This release contains support for protocols such as OpenFlow 1.3, Open vSwitch
Database Management Protocol (OVSDB), Border Gateway Protocol (BGP), and Path
Computation Element Protocol (PCEP). As part of the Hydrogen release, OpenDaylight
also contains a plugin for OpenStack Neutron. OpenDaylight uses northbound APIs to
interact with OpenStack Neutron and uses OVSDB for southbound configuration of
vSwitches on compute nodes.
Beryllium is the fourth software release for the OpenDaylight project. This release
takes another step closer to creating an open industry platform for SDN and network
function virtualization (NFV). OpenDaylight Beryllium provides a tighter integration to
OpenStack and allows centralized network orchestration and management from the con-
troller. Figure 1-17 shows the key standards supported by OpenDaylight Beryllium.
The concept of NFV originated from service providers looking to increase the agility
and flexibility of deploying new network services to support growing customer
demands. NFV is complementary to SDN, and there is no dependency between SDN
and NFV. NFV can be implemented using non-SDN mechanism leveraging techniques
commonly deployed in many data centers. However, combining SDN with NFV
simplifies deployment and operation and maintenance procedures.
OpenStack
OpenStack is an open source solution for building and managing open source cloud
computing platforms that can be used in public and private clouds. OpenStack is
managed by the OpenStack Foundation, a nonprofit organization overseeing both
development and community building of the project.
■ Swift—Swift is a storage system for objects and files that allows developers to refer
to a unique identifier associated with a file or data instead of referring to files by
their location on a disk drive. This allows better scaling and enables the system to
manage how data is backed up.
OpenStack enables users to deploy virtual machines and other component services
previously described to handle various tasks for a cloud data center environment.
OpenStack provides the infrastructure for users to quickly instantiate new services and
allow developers to create software applications that tie into the framework for agile
service delivery to the end users.
www.allitebooks.com
Summary 35
Summary
The intent of this chapter is to provide you with an understanding of the cloud
environment and its key fundamental concepts. In it, you have learned about a number
of concepts, including data center evolution, virtualization aspects for a data center, and
different flavors of cloud services. It is important to understand these concepts to get a
picture of the environment where a CSR 1000V is utilized. The following chapters get
deeper into CSR 1000V.
This page intentionally left blank
www.allitebooks.com
Chapter 2
Now that you have reviewed the concepts of cloud and enterprise trends, this chapter
provides an introduction to CSR 1000V. This chapter starts with some quick background
on Cisco IOS that led to the development of the IOS XE architecture used in the ASR
1000 and ISR 4000 series platforms. This will help you better understand the CSR
1000V architecture.
IOS uses cooperative multitasking that offers a simple and efficient scheduling method.
The IOS scheduler is responsible for managing and scheduling all processes in the
multitasking environment. It employs priority run-to-completion scheduling, allowing
each process a chance to run as long as it needs to run before releasing control back to
the scheduler. Each process is a single thread and is assigned a priority value. The high-
priority processes run before the lower-priority processes. The high-priority processes
can jump to the head of the line for CPU runtime, but high-priority processes may
not take CPU cycles away from running lower-priority processes. The IOS scheduler
maintains four separate queues:
38 Chapter 2: Software Evolution of the CSR 1000
■ Critical—Reserved for system processes such as the scheduler itself and memory
management processes.
■ Low—For all processes running in the background for periodic tasks, such as
logging messages.
To reduce the impact of runaway processes hogging CPU runtime and refusing to
relinquish control back to the scheduler, the IOS scheduler uses a watchdog timer that
can forcefully interrupt and terminate rogue processes.
IOS has a small kernel for CPU scheduling and memory management. Unlike the kernel
for traditional operating system cores that run in a protected CPU environment, the
IOS kernel is a set of components that run in user mode on the CPU with access to
full system resources. There is no special kernel mode for the IOS kernel. The IOS
kernel schedules processes, provides memory resource management, handles hardware
interrupts, maintains buffer resources, traps software exceptions, and manages other
low-level services.
IOS
Processes Memory
Drivers
Buffers
Kernel
Hardware
Over the decades, Cisco IOS has evolved into a feature-rich operating system powering
Cisco hardware. Generic IOS has a 32-bit monolithic architecture that runs as a single
image with all the processes having access to one flat memory address space; the kernel
www.allitebooks.com
IOS XE Architecture 39
does not support memory paging or swapping. The 32-bit architecture limits the memory
allocation of IOS to 4GB (232 bytes), and the single threading of IOS processes prevents
the kernel from taking advantage of the multithreading capability in today’s multicore
CPU hardware architecture. Multithreading enables an operating system to execute
multiple processes simultaneously. To keep up with the hardware advancements and to
take advantage of the multithreading software capability on new multicore CPUs, Cisco
developed IOS XE software.
IOS XE Architecture
IOS XE is part of the continuing evolution of the Cisco IOS operating system. IOS XE
leverages key architectural components of IOS and at the same time overcomes the
limitation of the 32-bit kernel of IOS. IOS XE retains the look and feel of classic IOS,
while offering improved features and functionality. IOS XE separates system functions
into the following components:
■ IOS XE kernel
■ Forwarding Manager
■ Interface Manager
■ Platform Manager
IOS XE
IOS Daemon
(IOSd)
Processes
Platform Forwarding Interface
Manager Manager Manager
Linux Kernel
Device Drivers
Hardware
The IOS XE Linux kernel, sometimes referred as BINOS in Cisco documents, leverages
a scheduler to handle interrupt requests and share processor time among multiple
processes. The IOS XE kernel offers a memory management system to manage process
address space. The kernel resides in an elevated system state compared to regular user
applications. It offers a protected memory management suite, which allows the kernel
and other control plane applications to run in protected memory spaces. The IOS XE
kernel uses symmetric multiprocessing (SMP) architecture, enabling applications to run
across multiple CPU cores for higher performance. Unlike the classic IOS kernel, which
features run-to-completion scheduling, IOS XE has a preemptive kernel that allows
preemption of a task even as another task executes in the kernel. This capability provides
better process response time and prevents a runaway process from hogging the CPU
cycles. IOS XE leverages the Completely Fair Scheduler (CFS) to handle CPU resource
allocation, with the goal of maximizing overall CPU utilization while also maximizing
interactive performance.
IOSd inherits a lot of characteristics from the classic IOS, and because it runs in a
multithreaded 64-bit environment and is responsible for control plane functions, it
enables the system to scale while maintaining backward feature compatibility. Internally,
IOSd provides an environment controlled by its own process scheduler. IOSd runs in a
protected address space that offers fault isolation from other components. IOSd uses
the run-to-completion scheduler model for IOS control plane processes, but the process
IOSd itself can be preempted by the Linux kernel scheduler. The Linux kernel leverages
a CFS low-latency scheduler and preemptable threads to minimize scheduling delays.
www.allitebooks.com
Cisco ASR 1000 System Architecture Overview 41
IOS XE improves the overall security and stability of a system through several
techniques. First, IOSd runs as a user-level process with root permission that has access
only to its own memory and a restricted portion of the file system for enhanced fault
containment. Second, to prevent overloading the IOS daemon, the information sent
to IOSd is filtered and rate limited. Finally, some of the functions that were handled
natively in classic IOS are offloaded to other components. Together, these methods
reduce the likelihood of IOSd becoming overloaded and unable to process critical
control plane packets.
Together, the Forwarding Manager, the Interface Manager, and the Platform Manager
form the middleware that interconnects the major components. These three managers
maintain the state of the overall system in nonvolatile memory and offer the capability
of In-Service Software Upgrade (ISSU) on certain platforms.
modules. It is also the foundation for the virtual routing platform CSR 1000V’s system
architecture.
Now let’s look into the ASR 1000’s high-level system architecture before examining
the CSR 1000V. The ASR 1000 runs on IOS XE and has three main components: route
processor (RP), embedded service processor (ESP), and SPA interface processor (SIP).
Route Processor
A route processor (RP) includes a general-purpose CPU and runs the IOS XE network
operating system. It is an integrated system component for fixed chassis platforms such
as the ASR 1001-X, and it is a separate module on modular systems such as the ASR
1004 and ASR 1006-X.
An RP handles all the control plane traffic and manages the system functions. It runs
routing protocols, builds and distributes forwarding information to the ESP, negotiates
and exchanges encryption keys in IPsec sessions, and monitors and manages power and
temperature for system components including line cards, power supplies, and fans.
■ MAC classification
■ VPNs
■ Load balancing
■ Firewall
■ Hardware-assisted encryption
www.allitebooks.com
Cisco ASR 1000 System Architecture Overview 43
At the heart of an ESP is the Cisco QuantumFlow Processor (QFP). The QFP is a
multicore parallel packet processor on a single silicon chip, designed for scaling and
performance while offering rich data path features and services. The QFP architecture
is non-pipelined, parallel processing with centralized shared memory. The QFP packet
processor engine is responsible for processing all traffic flows in the data-forwarding
path. Inside, the QFP also includes a traffic manager responsible for queuing and
scheduling functions for the forwarding plane. An ESP performs all packet forwarding,
buffering, and output queuing and scheduling for all traffic going through the QFP.
Kernel
Kernel
Control Messaging
SPA Driver
Forwarding Platform
Manager Manager
Kernel
SPA Interface
Processor
The CSR 1000V is an IOS XE software router based on ASR 1001 that runs within a
virtual machine deployed on any general server hardware or x86 server hardware. There
are a lot of commonalities between the system architecture for the CSR 1000V and the
ASR 1000; however, there are some differences as well. Let’s take a look at how the CSR
1000V, which works on the IOS XE software, is different from the ASR 1000:
■ The CSR 1000V is a virtual router that does not have SPA components.
■ The CSR 1000V does not have the hardware interconnects, SPAs, and the few
kernel utilities that relate to SPA.
■ Utilities have been added to the kernel for use by virtual hardware presented by the
virtualization layer, such as vCPU, vMemory, and vConsole.
■ There is no crypto ASIC. The CSR 1000V leverages AES-NI (with compiled
instructions into the crypto library). However, in the future, as chip technology
advances, crypto support can be leveraged from the hardware.
■ Due to the absence of the QFP, it has lower forwarding performance compared to a
traditional ASR 1000.
Note The CSR 1000V can have good performance values with appropriate hardware and
hypervisor tweaking. You learn more about this in Chapter 3, “Hypervisor Considerations for
the CSR.”
The Cisco CSR 1000V series lowers the barriers to enterprise adoption for cloud. The
primary features include the following:
www.allitebooks.com
Deployment Requirements 45
CSR 1000V
Forwarding Manager
Hypervisor
Physical Hardware
Chapter 4, “CSR 1000V Software Architecture,” provides detailed coverage of the soft-
ware architecture for the CSR 1000V.
Deployment Requirements
Deployment of the CSR 1000V requires the following for an infrastructure-agnostic
feature set:
■ vSwitch, OVS, dVS, N1KV (note that there is no dependency on CSR 1000V and
vSwitch)
■ VMware ESXi 5.0 or above, Kernel-based Virtual Machine (KVM), RHEL 6.6,
Ubuntu 14.04 LTS Server, Microsoft Hyper-V, Windows Server 2012 R2, Citrix
XenServer 6.2, Amazon Web Services (AWS)
46 Chapter 2: Software Evolution of the CSR 1000
The CSR 1000V has the following hardware requirements for deployments:
■ Minimum of one vCPU with scalable up to two, four, and eight vCPU for
performance. This requirement will change in future releases. Please refer to the
release notes for minimum requirements on the latest release.
■ 2.5GB on one vCPU (default) and 4GB on four vCPUs. Memory elasticity is
supported with an 8GB license (expansion from 4GB to 8GB).
■ 8GB hard drive with local, SAN, and NAS resources supported.
Table 2-1 lists the support for various hypervisors by CSR 1000V.
www.allitebooks.com
Elastic Performance and Scaling 47
* VMware ESXi 6.0 supported on Cisco IOS XE 3.16.1S and later, and 3.17S and later.
The support for newer versions of the hypervisor will be based on software revisions for
the CSR 1000V. Please refer to the following CCO link to verify support of the hyper-
visor on the latest CSR 1000V release: http://www.cisco.com/c/en/us/td/docs/routers/
csr1000/software/configuration/csr1000Vswcfg/csroverview.html#pgfId-1081607.
Chapter 3 describes the CSR 1000V’s interaction with different types of hypervisors.
There are several throughput options, ranging from 10Mbps all the way up to 10Gbps.
The CSR 1000V applies a license shaper that restricts the aggregate throughput of
the router’s interfaces. For example, if a 100Mbps license is installed, a maximum of
50Mbps of bidirectional traffic is allowed through the CSR 1000V system.
The CSR 1000V uses a license-based shaper to enforce the throughput on the router.
The licensed bandwidth is measured against the aggregate throughput of the traffic
across all the interfaces on the router. The license shaper regulates both priority traffic
and non-priority traffic. The shaper is implemented at the root of the QoS hierarchy. To
ensure that the license shaper doesn’t drop high-priority traffic, QoS (for example, LLQ)
should be configured. All traffic exceeding the bandwidth licensed for the shaper will be
tail-dropped.
48 Chapter 2: Software Evolution of the CSR 1000
■ Cisco IOS XE 3.10S and earlier regulate throughput only on the non-management
interface. The traffic going across the GigabitEthernet 0 management interface does
not count toward the aggregate bandwidth for the system.
■ Cisco IOS XE 3.11S and later enforce the license shaper across all interfaces,
including the GigabitEthernet 0 management interface.
The license throughput shaper enforces globally and not on a per-interface basis. This
means the license shaper does not distinguish the different types of traffic going across
the system. When the aggregate bandwidth exceeds the licensed throughput, the excess
packets are discarded. Figure 2-5 shows the three interfaces on the CSR 1000V passing
an aggregate throughput of 120Mbps. This exceeds the 100Mbps licensed throughput,
which means 20Mbps of traffic is discarded.
20Mbps
Bit Bucket
Table 2-2 shows the server resource requirement for the CSR 1000V for the different
technology packages. Refer to the Cisco.com website for the features covered under the
different technology packages.
www.allitebooks.com
Rapid Deployment and Routing Flexibility in the Cloud 49
Table 2-2 CSR 1000V Resource Requirements for Each Technology Package
Technology Package
Throughput IP Base Security APPX AX
10Mbps 1 vCPU/4GB 1 vCPU/4GB 1 vCPU/4GB 1 vCPU/4GB
50Mbps 1 vCPU/4GB 1 vCPU/4GB 1 vCPU/4GB 1 vCPU/4GB
100Mbps 1 vCPU/4GB 1 vCPU/4GB 1 vCPU/4GB 1 vCPU/4GB
250Mbps 1 vCPU/4GB 1 vCPU/4GB 1 vCPU/4GB 1 vCPU/4GB
500Mbps 1 vCPU/4GB 1 vCPU/4GB 1 vCPU/4GB 1 vCPU/4GB
1Gbps 1 vCPU/4GB 1 vCPU/4GB 1 vCPU/4GB 2 vCPUs/4GB
2.5Gbps 1 vCPU/4GB 1 vCPU/4GB 4 vCPUs/4GB 4 vCPUs/4GB
5Gbps 1 vCPU/4GB 2 vCPUs/4GB 8 vCPUs/4GB —*
10Gbps 2 vCPUs/4GB —* —* —*
* This performance and feature combination is currently not supported as of the IOS XE 3.16S release.
IP Base, Security, APPX, and AX are the technology packages described in Table 2-3.
www.allitebooks.com
CSR 1000V Deployment Examples 51
Many public cloud and virtual private cloud services also provide VPN as a capability;
however, this service is typically offered as a black box with little visibility and
troubleshooting capability into the VPN service. In addition, there is a monthly recurring
charge or per-VPN tunnel fee.
In this case, an enterprise can leverage the CSR 1000V in the cloud for secure
connectivity while maintaining a consistent WAN architecture. Using the CSR 1000V
as the VPN gateway in the cloud provides a familiar platform for monitoring and
troubleshooting problems while avoiding any additional VPN service fees. The CSR
1000V can be used in a hub-and-spoke, partial-mesh, or full-mesh network. By leveraging
DMVPN, an enterprise can dynamically connect its branch sites to its data center in the
cloud, thereby minimizing the latency caused by backhaul through the central site while
overcoming the complexity of managing point-to-point IPsec VPNs.
The following features are leveraged for a secure cloud VPN gateway:
■ Host connectivity—DHCP
The Overlay Transport Virtualization (OTV) and Virtual Private LAN Services (VPLS)
features on the CSR 1000V enable an enterprise to extend Layer 2 network and
VLAN segmentation from its data center into the cloud for virtual machine migration.
This provides the enterprise the capability for server backup, disaster recovery, and
computing on-demand scaling.
The following features are leveraged for network extension from premises to cloud:
■ LISP
■ NAT
■ OTV
■ VPLS
52 Chapter 2: Software Evolution of the CSR 1000
The CSR 1000V can participate in a Virtual Extensible LAN (VXLAN) network service as
a VXLAN tunnel endpoint (VTEP) and provide a termination point for VXLAN network
identifiers (VNI). For larger enterprise data centers and service provider networks,
this feature overcomes the scaling limitation of 4096 VLANs for increased network
scalability. A VXLAN supports millions of network identifiers and allows a service
provider to support a greatly increased number of tenants on its existing infrastructure.
An enterprise can also deploy the CSR 1000V as a dedicated VXLAN gateway to allow
traffic to be routed or bridged to other VXLAN or non-VXLAN networks.
■ MPLS VPN
■ BGP
■ VRF
■ VXLAN
www.allitebooks.com
CSR 1000V Key Features 53
EEM supports more than 20 event detectors that are highly integrated with
different Cisco IOS software components to trigger actions in response to network
events. You can inject your business logic into network operations using IOS EEM
policies to enable creative solutions, such as automated troubleshooting, fault
detection, and device configuration.
■ IP Service Level Agreement (IP SLA)—The Cisco 1000V offers embedded IP SLA
capability, allowing customers to understand IP service levels, increase productivity,
and reduce operational costs in the cloud. The IP SLA feature performs active
monitoring of network performance and can be used for network troubleshooting,
readiness assessment, and health monitoring. IP SLA running on the CSR 1000V
can be configured to actively monitor and measure performance between multiple
network locations or across multiple network paths. The IP SLA sends out probes
that can simulate network data, voice, or video services and collect network
performance information in real time. The information collected includes data about
response time, one-way latency, variation in packet delivery (jitter), packet loss, voice
quality scoring (MOS score), network resource availability, application performance,
and server response time. You can use the measurement and statistics provided by IP
SLA for troubleshooting, problem analysis, and network capacity planning.
■ Device identity
The LISP routing architecture provides separation of the device identity from its
location. This capability brings enhanced scalability and flexibility to the network,
enabling virtual machine IP mobility (VM-Mobility) for geographic dispersion
of data centers and disaster recovery. In addition, LISP simplifies enterprise
multihoming with ingress traffic engineering capability, multitenancy over Internet,
and simplified IPv6 transition support.
The LISP VM-Mobility solution addresses the challenge of route optimization when
a virtual machine moves from one location to another. It does this by keeping the
server identity (its IP address) the same across moves so the clients can continue to
send traffic regardless of the server’s location, and at the same time, it guarantees
optimal routing between clients and the server that moved.
www.allitebooks.com
CSR 1000V Key Features 55
packet. One key benefit of MPLS is that the decision about where the packets
are forwarded is based solely on the label and not on the Layer 3 information the
packet carries; this allows for faster lookups for the forwarding decision.
MPLS VPN extends the capabilities of MPLS and supports creation of VPNs across
an MPLS network. MPLS may be used to deliver VPN solutions at either a Layer 2
VPN (L2VPN) or Layer 3 VPN (L3VPN). All solutions enable a service provider to
deliver a private service over a shared network infrastructure.
■ Virtual Private LAN Services (VPLS) —VPLS is a Layer 2 VPN service that
provides multipoint Ethernet LAN services to multiple sites, offering a single
bridged domain over a managed IP or MPLS network. Enterprises often use VPLS
to provide high-speed any-to-any forwarding at Layer 2 without the need to rely
on Spanning Tree to keep the physical topology loop free. VPLS leverages the
concept of linking virtual Ethernet bridges using MPLS pseudowires to interconnect
sites in a full-mesh topology and form a single logical bridge domain. Compared
to traditional LAN switching technology, VPLS is more flexible in its geographic
scaling, as the sites may be within the same metropolitan domain or may be
geographically dispersed over a region or a nation.
encapsulated into an IP packet and delivered across the transport network. This
eliminates the need to establish virtual circuits, called pseudowires, between the
data center locations.
■ Zone Based Firewall (ZBFW) —Zone Based Policy Firewall, also known as Zone
Based Firewall, is a stateful inspection firewall running on the CSR 1000V that
offers a flexible advanced security model. With ZBFW, router interfaces are
assigned to security zones, and the firewall inspection policy is applied to traffic
moving between the security zones. ZBFW supports many types of application
inspection, including HTTP, Secure HTTP (HTTPS), Secure Shell Protocol (SSH),
www.allitebooks.com
Summary 57
Existing IT staff will be able to configure these features, and using them allows you
to extend existing enterprise security into the cloud. You can apply security policies
between virtual networks or applications in the cloud as well as between cloud and
external locations.
For a complete list of features supported by the CSR 1000V, please refer to the Cisco
Feature Navigator at http://tools.cisco.com/ITDIT/CFN/jsp/index.jsp.
Summary
Now that you have read this chapter, you should have a fundamental understanding
of the system architecture of IOS and the evolution of IOS XE. The chapter provides
detailed discussion of the operations and the system architectures for IOS and IOS
XE network operating system. The components of IOS XE and the ASR 1000 are
fundamental to understand the operation of the CSR 1000V. This chapter has provided
the background information you need to go deeper into the CSR 1000V’s system
architecture and operation.
This page intentionally left blank
www.allitebooks.com
Chapter 3
An operating system is software that manages hardware resources and makes them
available to applications. Application developers can write software that talks to the
hardware. However, software that interacts directly with the hardware is complex and
difficult to code. It is best left to people who understand the inner working of the
hardware to write that piece of code. This way, application developers can do what
they do best: write applications without having to worry about how their code will be
scheduled on the hardware it eventually runs on. That scheduling is the domain of an
operating system, which manages and schedules hardware resources for the applications
running on it. But how does the operating system do it? Let us start with how an
operating system is designed.
60 Chapter 3: Hypervisor Considerations for the CSR
An OS can schedule the CPU in different ways. At a given instant, most CPUs can
service just one process. With multiple applications competing to get a piece of the CPU
cycles, the OS must service the requests in such a way that each application gets a slice
of the CPU cycle. An OS can achieve this in several ways:
■ FIFO (first in, first out) —In this simplest of scheduling algorithms, a process is
serviced by the CPU on a first-come, first-served basis. It is fairly easy to implement
this in the C language. You just create a circular list to have the OS remove the process
from the front of the queue, run it until completion, and allow the next process to
take the CPU. The advantages of this kind of implementation are that it is easy to
implement and simple to understand. On the downside, however, short processes at
the tail of the queue have to wait a long time to get serviced as the non-preemptive
implementation waits for each process to complete before moving on to the next.
■ SPN (Shortest Process Next) —In the SPN implementation, the CPU schedules
the shortest process first, so the process that takes the least time to get executed is
scheduled first. In this algorithm, the OS needs to know the exact time each process
takes to get done, and the user provides this input. So if the input isn’t accurate, the
www.allitebooks.com
Understanding Operating Systems 61
efficiency of the system suffers. With multiple short jobs, the long ones get queued
up for a while.
■ SRT (Shortest Remaining Time) —SRT is a better version of the SPN algorithm.
Here the process that can be completed in the shortest time jumps to the top of the
queue. So whenever there is a process that requires the least amount of CPU time, it
cuts the queue of scheduled processes and goes right to the start of the queue. This
implementation suffers from the same drawbacks as the SPN algorithm.
■ Priority scheduling—In this algorithm, each process is assigned a priority, and the
higher-priority processes are executed first. Some implementations use the time the
process has waited to increase the priority of the process; this prevents the non-
priority processes from being blocked.
As with CPU, memory is another resource that needs to be shared between applications.
The given physical memory (RAM) is managed by the OS. With limited RAM address
space, operating systems will either use segmentation or paging to virtually increase this
address space:
In most Linux versions, the kernel uses the preemption model when it comes to CPU
scheduling. For memory management, the kernel uses paging.
Physical
Page Offset
0 16 31 0 16 31 0 31
Offset
Physical Permission
Page Data
Now that we have discussed the design options available when implementing an OS, let
us discuss the different components that are building blocks for an OS:
■ Libraries provide services to the applications that run on the operating system.
■ A boot mechanism loads the operating system into the main memory.
www.allitebooks.com
Understanding Operating Systems 63
Kernels
A kernel is at the heart of an operating system. It is the piece of code that sits closest
to the hardware. The kernel is an integral part of an operating system, and its duty is to
bring all devices to a known state and make the computer system ready to be used.
The kernel code provides a layer of abstraction between the hardware and the software
running on the system. Through APIs, applications have the kernel perform hardware
jobs. Kernels allow the hardware to be shared between multiple applications, and
kernels overcommit resources to applications. (You’ll learn more about the Linux
implementation later in this chapter.)
When UNIX was originally designed, it had a simple design wherein the entire operating
system code was one big layer, as shown in Figure 3-2. This was a good approach at the
time, given the limited number of programmers, and it kept things simple.
Applications
User Modes
Kernel
Virtual CPU
Drivers Paging
Memory Scheduling
Hardware Controllers
Microkernels
As the demand on the OS increased, there had to be a more modular way to structure
the design of the kernel. Microkernels were introduced to strip off all nonessential
components of the kernel code and move those components to system applications; this
kept the kernel code small, sleek, and efficient. Thus the kernel space and the user space
were segregated from one another; the user space became the code that runs outside
the kernel. The drivers, file systems, and other services are all part of the user space. The
microkernel code provides hardware management (memory and CPU) and messaging
between services. This way, the kernel code is kept intact, and the subsequent enhance-
ments do not involve rebuilding the kernel.
The object-oriented approach to programming brought changes in operating system
design, too. Take, for example, Solaris. It has a small kernel and a set of modules that
can be dynamically linked to the kernel. The kernel can therefore be small and does not
need to take care of interprocess communication (IPC) because the modules are free to
contact each other.
64 Chapter 3: Hypervisor Considerations for the CSR
Hybrid Kernels
Most of today’s operating systems are a hybrid of the legacy architecture and modern
object-oriented approach. Take, for example, Android, shown in Figure 3-3.
Apps
Application Infra
Linux Kernel
In the Android approach, the Linux kernel is used as is. The Linux kernel runs libraries
like OpenGL and WebKit, and it links to certain Android runtime libraries dynamically
at runtime. The runtime libraries allow multiple instances of containers to be created
simultaneously, providing security, isolation, memory management, and threading
support. This runtime support provides improved compilation and runtime efficiency to
applications compared to traditional operating systems.
Fast Switch
Process Packet Buffers
Software
Hardware
The kernel in IOS mainly runs the scheduler and the memory manager.
www.allitebooks.com
Understanding Operating Systems 65
The Scheduler
The IOS scheduler has three main categories of process queues that hold context
information. The scheduler moves the process context from one queue to another, based
on the state of the process. Following are the queues:
■ Dead queue—Holds processes that have been killed but still need to free up
resources.
■ Ready queue—Holds processes that are run ready. There are four ready queues,
based on priority:
■ Critical—The scheduler first empties this queue. The processes in the critical
queue are run until the queue is empty. Only then does the scheduler schedule
processes from the other queues.
■ High—After all the critical processes have completed, the scheduler picks up
processes in the high queue. The scheduler checks the critical queue in between
running processes from the high-priority queue.
■ Medium—After all high processes have been executed, the scheduler picks up
the processes in the medium queue. Here, too, in between running medium-
priority processes, the scheduler looks for processes in the high-priority queue
(and this is repeated if there are processes in the critical queue).
■ Low—The same algorithm is followed for the low-priority queue as for the other
queues. In between running low-priority processes, the scheduler looks for pro-
cesses that are ready in the medium, high, and critical queues.
show process CPU is the command-line command for checking what processes are run-
ning on the CPU.
■ Pool manager—Each malloc or free from a process has to go through the pool
manager. It maintains the free memory information from each pool. The pool man-
ager coagulates discontinuous freed-up areas within a memory pool. It tries to allo-
cate a continuous area of memory for each process that asks for memory. Whenever
a process frees up memory fragments, the pool manager tries to coagulate the freed
memory into continuous memory areas to be made available to processes. Every
memory address managed by the pool manager has 32 bytes of overhead associ-
ated with it, because the pool manager gets its memory from multiple blocks of free
memory of varying sizes.
66 Chapter 3: Hypervisor Considerations for the CSR
IOS scheduling and memory management processes are part of the IOS kernel. However,
all kernel processes and non-kernel processes run in the user space in IOS. IOS was
designed to run on a fixed set of hardware (Cisco routers), and hence this design is easy
to implement and effective, too.
You will see in Chapter 4, “CSR 1000V Software Architecture,” how the IOS XE design
uses a Linux kernel to do all the kernel activities, while IOS runs as a process on a Linux
kernel. The design of IOS XE facilitates portability of code between hardware, and the
implementation is similar to that of a hybrid kernel.
On cold start, the computer system first has the boot loader program loaded into a
known place in memory. The boot loader program is located in the beginning of a hard
drive or partition. (The following section details the steps involved in booting.) The boot
loader performs tasks such as initializing the CPU, identifying key pieces of hardware,
and executing the kernel code. Boot loader code may or may not be part of the operat-
ing system. For example, in Windows NT, the boot loader is a part of the operating sys-
tem. In the case of the Cisco CSR, GNU GRand Unified Bootloader (GRUB) is used to
accomplish boot loader functions and isn’t a part of the operating system.
There are six essential steps involved in loading an operating system to a computer
and making it ready to run the applications. Following are the steps in the Linux boot
process; the CSR boot process follows this sequence as well:
www.allitebooks.com
Understanding Operating Systems 67
Step 1. The Basic Input/Output System (BIOS) runs from the ROM and is OS inde-
pendent. It performs a power-on self-test (POST), which is a basic check for
fundamental hardware on the computer system. BIOS looks for the Master
Boot Record (MBR), loads it, and executes it.
Step 2. The MBR is located in the first sector of the bootable disk. It is /dev/sda on
the Linux kernel that is used with the CSR. On most Linux kernel implemen-
tations, it is smaller than 512 bytes in size. The first 446 bytes provide the
primary boot loader info. The next 64 bytes contain the partition table info,
and the last 2 bytes are an MBR validation check. The main job of the MBR
is to load and execute the GRUB boot loader.
On the CSR, GRUB gets its menu of kernel images from menu.lst, located
under /boot/grub/, as shown in Example 3-1.
cat /boot/grub/menu.lst
default 0
timeout 5
serial --unit=0 --speed=9600 --word=8 --parity=no --stop=1
terminal --timeout=10 serial console
fallback 1
title CSR1000v—packages.conf
root (hd0,0)
kernel /packages.conf rw root=/dev/ram console=ttyS1,9600 max_loop=64
HARDWARE=virtual SR_BOOT=bootflash:packages.conf
So the main job of GRUB is to load and execute the Kernel and initrd images.
Step 4. The kernel mounts the root file system and executes the /sbin/init
program. Because init is the first program to be executed by the Linux
kernel, it gets a process ID of 1:
ps -ef | grep init
root 1 0 0 2014 ? 00:00:18 init [3]
68 Chapter 3: Hypervisor Considerations for the CSR
The main jobs of the kernel are to initialize the devices attached to the
computer system, mount the root file system, and run the init process. To
load the drivers, however, the kernel must have a file system. Before the
root file system is initialized, the kernel needs a file system to make sure
the kernel modules are loaded appropriately. This is done by the initrd
(or initial ramdisk) file system. This is a temporary root file system that the
kernel uses when it boots up. After the kernel boots and loads the main root
file system, it takes initrd offline.
Step 5. The kernel runs the init process, which creates all the processes from the
script located in the file /etc/inittab. The main job of the init process is
to set up the user space. It essentially decides the Linux run level. After it is
done creating all the processes from /etc/inittab, it goes into a wait state
until one of three events occurs:
■ Processes die
■ Power failure
■ Request via /sbin/telint for a change in run level
Step 6. The run level programs/services get started while your Linux system boots.
Depending on your init level setting, the system executes one of the
following for your run level:
www.allitebooks.com
Understanding Operating Systems 69
Under the /etc/rc.d directories you find programs that start with S and K. S stands for
startup, meaning that these are executed during startup. K stands for kill, meaning these
programs are executed during a system shutdown.
On a CSR, the programs shown in Example 3-2 are executed on run level 3 (full-
multiuser mode).
Example 3-2 Programs That Are Executed on the Full-Multiuser Run Level
[CSR:/etc/rc.d/rc3.d]$ ls -lrt
total 212
-r-xr-xr-x 1 root root 4684 Dec 2 15:45 S60nfs
-r-xr-xr-x 1 root root 4131 Dec 2 15:45 S56xinetd
-r-xr-xr-x 1 root root 4958 Dec 2 15:45 S55sshd
-r-xr-xr-x 1 root root 2778 Dec 2 15:45 S28autofs
-r-xr-xr-x 1 root root 2164 Dec 2 15:45 S26pcscd
-r-xr-xr-x 1 root root 1870 Dec 2 15:45 S20virt_support
-r-xr-xr-x 1 root root 8092 Dec 2 15:45 S14netfs
-r-xr-xr-x 1 root root 2615 Dec 2 15:45 S13portmap
-r-xr-xr-x 1 root root 1369 Dec 2 15:45 S12syslog
-r-xr-xr-x 1 root root 10115 Dec 2 15:45 S10network
-r-xr-xr-x 1 root root 7460 Dec 2 15:45 S08iptables
-r-xr-xr-x 1 root root 220 Dec 2 15:45 S99local
-r-xr-xr-x 1 root root 135553 Dec 2 15:45 S80binos
Virtual Memory for Process A Virtual Memory for Process B Virtual Memory for Process C
Figure 3-5 shows 8GB of physical memory available and the memory used by the
virtual process mapped to the physical memory. You can, in theory, create two 4GB
virtual machines on this. However, because two virtual machines will not use the entire
4GB physical address space, it’s better to overcommit and use the memory for hosting
another VM. If there is a surge in the memory requirement, swapping is used to manage
the spike, as shown in Figure 3-6.
0 16 31 0 16 31
Page Permis-
Offset Page
Index sion
Data Swapper
Disk
www.allitebooks.com
Understanding Hypervisors 71
With overcommitting of memory, you run the risk of having a process or virtual machine
want physical memory when the physical RAM space is all used up. In such a case, Linux
has to discard a page residing in RAM to accommodate the new request. If the page to
be discarded is not written into, it can just be removed as it can be easily brought back
when the processes need it. If a page in the cache that has been written into (a dirty
page) needs to be removed (which is done when there are no empty pages left), Linux
removes a dirty page from RAM and puts it in a special file called a swap file. This
is usually on a secondary storage device of a computer system like a hard disk. This
process is called swapping.
Now the obvious question is, how does Linux decide which dirty page to swap? Linux
uses the least recently used algorithm to decide which dirty page can be swapped. Each
page has an age associated with it. The pages that are being accessed are “young” pages,
while the pages that are not accessed a lot get “old.” When Linux has to choose which
dirty page to swap, it chooses the oldest page.
A swap algorithm must be effective; otherwise, you can run in to a situation where your
CPU becomes too busy swapping and is unable to give time to processing real workload.
Such a situation is called thrashing.
Linux Caching
Disk access is time-consuming, and to speed up the access, Linux deploys a cache
mechanism by which it reads from the disk once and then stores the entry in memory
(the Linux swap cache) for subsequent access. A swap cache is a list of page table entries
of the swapped-out pages’ locations in the swap file. Consider a situation where a dirty
page is swapped. It is then brought back to memory unmodified. Now if there is a need
to swap this page again, Linux does not push it back to the swap file. Instead, it simply
discards the page because the swap file already has a copy of the page.
As discussed earlier, an OS maps the virtual address to a physical address using page
tables. These table entries are stored in a hardware cache. This hardware cache has the
translational look-aside buffer (TLB). When a processor wants to map a virtual address
to a physical address, it gets the page table entry from TLB. If the processor finds the
information it is looking for, it gets the physical address from it. However, if it cannot
find the entry there, it asks the OS to update the TLB, using an exception. The OS
subsequently updates the TLB and clears the exception.
Understanding Hypervisors
As discussed in Chapter 1, “Introduction to Cloud,” virtualization allows you to fully
utilize modern hardware that would otherwise not be fully utilized. Just as an operating
system makes hardware resources available to the applications running on it, a manager
is needed to allocate the hardware resources to virtual machines running on it. This is the
job of a virtual machine manager (VMM), which is a software layer that sits between the
hardware resource and the virtual machines. VMM makes sure the virtual machines get
the hardware resources they require.
72 Chapter 3: Hypervisor Considerations for the CSR
Note Computer operating systems provide multiple levels of access to shared resources.
There is therefore a hierarchy of privileged resource access in the system. In x86, the term
ring is used to denote the hierarchy of access. Note that ring 0 is the highest privilege
level, and the lowest privilege level has the highest ring number. These are some important
features of x86 ring architecture:
■ Ring 0 has the most privileges and interacts directly with the physical hardware, such as
the CPU and memory.
■ Ring 1 and 2 aren’t used by most chipset architectures, and most chipsets support just
two modes (such as ring 0 and ring 3).
■ Ring 3 runs the user mode and has the lowest privilege level.
When a process running in the user space (ring 3) wants to make a privileged call to I/O
devices (shared resources), it makes a system call to the kernel. The kernel runs a snippet
in ring 0 after receiving the request from the user space to grant this access based on
the security restriction of the drivers. (This is achieved using the SYSENTER/SYSEXIT
instruction set, available on Pentium II+ processors.)
Now consider the case where there are multiple operating systems running on an x86
platform. All processes in the operating system make system calls as if the hardware
belongs to them. So there needs to be a way to make sure that an application running on
a guest OS does not trample over system calls from another OS. This is where hypervi-
sors come in. Because a hypervisor is aware of all the operating systems running on it,
it ensures that the kernel calls are prioritized and one system call does not conflict with
another, as shown in Figure 3-7. Just as a process in user mode makes system calls to the
kernel, a guest OS calls the hypervisor to execute privileged instructions on the chipset.
These processes are called hypercalls.
www.allitebooks.com
Understanding Hypervisors 73
Ring 3 Ring 0
Hardware
Files
Network
* 9
Para-Virtualized
$
X
' Hypercall
Guest OS
H
S V U Hypervisor
S W L Kernel
V Y
2 H
6 U
The previous discussion comparing operating systems and hypervisors assumes that
the operating systems running as guest operating systems are completely oblivious to
the fact that they are not the only ones running on the x86 platform. However, you
can make an OS aware that it is running on a hypervisor and not bare metal (via para-
virtualization, introduced in Chapter 1). The enlightened OS kernel uses hypercalls to
communicate with the hypervisor. With the knowledge that it is running on a hypervisor,
the enlightened OS kernel has optimized code, and this gives it better performance than
unenlightened guest operating systems.
Note Intel x86 platforms starting with the Pentium II series chipset have a
SYSTENTER/SYSEXIT instruction set that enables faster access to user-land processes to
access the kernel.
Intel’s 64 and IA-32 architecture software developer’s manual describes SYSENTER (a fast
system call) as follows:
Executes a fast call to a level 0 system procedure or routine. SYSENTER is a
companion instruction to SYSEXIT. The instruction is optimized to provide the
maximum performance for system calls from user code running at privilege level 3
to operating system or executive procedures running at privilege level 0.
The same software developer’s manual describes SYSEXIT (the fast return form of a fast
system call) as follows:
Executes a fast return to privilege level 3 user code. SYSEXIT is a companion
instruction to the SYSENTER instruction. The instruction is optimized to provide the
maximum performance for returns from system procedures executing at protection
level 0 to user procedures executing at protection level 3. It must be executed from
code executing at privilege level 0.
74 Chapter 3: Hypervisor Considerations for the CSR
Type 1 hypervisor architectures can be classified broadly into two categories, monolithic
architecture and microkernel architecture, as described in the following sections.
Monolithic Architecture
As is the case with operating system design, monolithic architecture is present in
hypervisor design, too. The hypervisor code using this design includes just one instance
of a virtualization stack and supports multiple instances of guest operating systems run
on it. Because all the device drivers are in the kernel, which is the supervisor area, VMs
are given a common pool of virtualized drivers to choose from. There can be no guest-
specific drivers in this design.
Microkernel Architecture
A microkernel approach strips down the actual hypervisor software to essential calls (like
kernel code) and pushes the other components to a management guest operating system
running on the kernel. The kernel talks to the hardware, and the management guest OS
runs the drivers and virtualization stack. The management guest OS handles I/O calls for
all other guest operating systems. This gives console access to the VM.
Xen and Hyper-V are examples of microkernel design. Figure 3-8 compares monolithic
and microkernel architectures.
VM1(Admin)
VM1 VM2 VM3 Virtualization VM2 VM3
Stack
(Admin)
Drivers Drivers Drivers
Virtualization Stack
Hypervisor Hypervisor
Drivers
Hardware Hardware
www.allitebooks.com
ESXi Hypervisor 75
Core Partitioning
Core management is one of the key aspects to consider when selecting a hypervisor.
The basic approach is static partitioning, wherein each VM is permanently dedicated to
a core. In dynamic partitioning, the VMs are allowed to migrate between cores. This is
critical when there are more VMs than there are cores. The hypervisor must optimally
schedule the cores for the VMs. This means timesharing the cores between multiple
VMs. Take, for example, a two-core system. Without core partitioning, a hypervisor
can allocate two cores to a single VM or single core per VM (assuming that there is no
hyperthreading, in which a single processor is split into two logical processors and each
processor then has the capability of executing instruction sets independently). With core
partitioning, the hypervisor can allocate two cores to two VMs because the cores are
time shared and not dedicated to a single VM.
The following sections cover ESXi and KVM, the two major type 1 hypervisors in the
industry today, and provide a brief overview of Hyper-V and Xen.
ESXi Hypervisor
ESXi from VMware is one of the popular hypervisors in the enterprise data center
environment. As a type 1 hypervisor, it runs on bare metal and provides the user a
platform to create a virtual infrastructure.
■ The VMkernel
■ Device drivers
■ File systems
■ Management
The VMkernel
In the early days, ESXi was called ESX, and at that point it used a Linux kernel. But after
version 4.1 of ESX, VMware stopped the development of ESX in favor of ESXi, which
did not have a Linux kernel. On an ESX hypervisor, the Linux kernel boots first, and
then a script loads the VMkernel. In versions since ESX 3, the Linux kernel loads the
VMkernel from initrd. With ESXi, VMware got rid of the Linux-based service console.
This resulted in a smaller memory footprint (reduced by 90MB) and allowed ESXi to be
embedded within the host flash, thereby eliminating the need for a local boot disk.
76 Chapter 3: Hypervisor Considerations for the CSR
■ CPU scheduler—The CPU scheduler’s role with the VMkernel is the same as that
of a CPU scheduler in an operating system. In a conventional operating system,
the CPU scheduler allocates processes or threads to processors by using fairness,
responsiveness, and scalability as major design criteria. In the VMkernel, the CPU
scheduler does the same thing, but the processor in the case of a hypervisor is a
vCPU (virtual CPU). A vCPU is an execution context or a series of time slots on
a processor. In a multicore environment, the scheduler splits these time slots over
multiple cores. It has to coordinate time slots between multiple physical cores,
and this means overhead. Therefore, just adding more vCPU does not guarantee
improved performance.
www.allitebooks.com
ESXi Hypervisor 77
Guest OS Page
Guest virtual => Guest physical
Table
ESXi
Shadow Page
Table Guest virtual => Host physical
standard that provides a framework for monitoring hardware resources for ESXi.
The CIM framework consists of the following:
■ CIM providers—A CIM provider is a piece of code that enables monitoring and
management capabilities of a piece of hardware. A CIM provider is an extremely
lightweight plugin that is written for the purpose of monitoring and managing
a piece of hardware. Hardware and software makers chose one of the several
predefined XML schemas to give management information about their products.
CIM providers run inside the ESXi system.
■ hostd—This is a very critical process. Together with vpxa (discussed later in this
chapter), it controls management access to the ESXi host. hostd is the process
responsible for communication between the VMkernel and the outside world. It
authenticates users and tracks user privileges. hostd talks to the VMkernel and
invokes all management operations on VMs, storage, and networking. hostd is used
by the vSphere API to make a connection to the ESXi host.
Device Drivers
In the early days of ESXi, VMware used drivers derived from Linux. This meant that
ESXi could support many hardware devices. However, to use these drivers from Linux,
ESXi required a mechanism to make these drivers talk to VMkernel. VMkernel is not
a Linux kernel, and so VMware had to write an additional layer of software to make
these Linux drivers talk to the VMkernel. This additional shim layer, vmklinux, pro-
vided VMkernel with the APIs needed to communicate to the Linux-based drivers. The
downside with this architecture was an additional layer of indirection that degraded the
performance. ESXi was also limited by the capabilities of the Linux drivers.
With ESXi5.5, VMware decided to do away with the vmklinux shim layer. This means
that a driver, native to ESXi now talks to the hardware and has APIs interacting directly
with the VMkernel. The new native driver architecture means better performance as
it removes the overhead processing of the additional shim layer. You also get better
debugging capabilities because VMware can now develop debugging tools for drivers.
ESXi5.5 is backward compatible and supports legacy Linux drivers.
www.allitebooks.com
ESXi Hypervisor 79
vmKernel
vmkLinux vmKernel
Hardware Hardware
File Systems
VMware uses a high-performance clustered file system called Virtual Machine File
System (VMFS). VMFS is custom built for virtualized environments and enables high-
speed storage access for virtual machines. VMFS manages virtual machine storage by
creating a subdirectory for each virtual machine and then storing all its contents within
that directory. The location of the directory is VMhome.
during an operation where metadata needs protection. When the operation completes,
the lock is released, and access to all ESXi hosts resumes. Atomic test-and-set (ATS) is
used in data stores where hardware acceleration is available. Unlike in SCSI reservations,
where you lock the entire volume, ATS allows discrete locking per sector.
Shared
VMFS (LUN)
VMDK VMDK VMDK VMDK Storage
Management
Management of the hypervisor in ESX was done using a service console. However, as
indicated earlier in this chapter, ESXi got rid of the service console, and the management
aspect moved from the service console within the kernel to a remote/central location
that now enables management using standardized APIs rather than the legacy interactive
session service console methodology. In other words, VMware has moved away from
the service console operating system that helped interact with the VMkernel in favor of
a standardized CIM API model that provides APIs to access the VMkernel.
When a system with ESXi hypervisor boots up, ESXi runs immediately and tries to
detect the network by using DHCP. ESXi then spurts out a screen that lets the user
configure networking, administrator access password, and the test management network,
as shown in Figure 3-12. With the networking option, the user can select the vNICs to
assign an IP address and netmask, VLAN, and hostname, for the ESXi host. This enables
remote access to the ESXi host using a vSphere client.
www.allitebooks.com
ESXi Hypervisor 81
A vSphere client runs on a Windows machine, and VMware now has a web client, too.
The vSphere client provides user access to the ESXi host. Through the vSphere client,
users can create and manage virtual machines. vSphere clients can let you access an ESXi
host directly. However, if you want your vSphere client to manage more than a single
instance of ESXi, you need to use vCenter. vCenter is a tool for centrally managing
vSphere hosts. With vCenter, you can manage multiple ESXi servers and VMs through
a single vSphere client. This makes it possible for a network administrator to take
advantage of virtualization features like vMotion, Storage vMotion, fault tolerance, and
DRS (Distributed Resource Scheduler). Figure 3-13 shows the vCenter architecture.
82 Chapter 3: Hypervisor Considerations for the CSR
vCenter Server
KVM
Kernel-based Virtual Machine (KVM) is part of the Linux kernel. It is, in fact, the first
virtualization solution that has made it to the Linux kernel code. Starting with the 2.6.20
version of the kernel, the KVM module is shipped with the Linux kernel. KVM relies
on a virtualization-capable CPU with Intel’s Virtualization Technology (VT) or AMD’s
Secure Virtual Machine (SVM) extension.
www.allitebooks.com
KVM 83
The Linux kernel, sans KVM, is an operating system kernel that runs processes in either
guest mode or kernel mode. As described earlier in the chapter, in kernel mode the OS
operates with critical data structures or tries to access I/O (also referred to as controlled
resources). User mode, on the other hand, runs applications. Kernel mode prevents
applications in user mode from directly accessing the kernel drivers. KVM achieves this
by extending the Linux kernel capability to isolate a process in such a way that it gets its
own kernel and user mode. This is called a guest mode. Thus a process in guest mode can
run its own operating system.
The KVM module uses hardware assists provided by Intel’s VT and AMD’s SVM proces-
sors to execute the guest code directly. KVM treats all its VMs as processes and relies
on the scheduler to assign CPUs to the VM. Virtual CPUs (vCPUs) are threads within this
VM process. KVM allows the guest user to execute on the physical processor but keeps
control of the memory and I/O management. Consider a scenario in which a guest pro-
cess tries to access a controlled resource. KVM takes control from the guest process and
executes the task on the controlled resource on behalf of the guest process. The guest
thinks it is running on a real hardware resource. It can do memory paging, segmentation,
and interrupt manipulation just as it would if it were running on bare metal. It has its own
user and kernel space defined in the guest memory carved for it by KVM.
Linux
Processes
Using KVM User
Linux Linux Guest
Processes Processes Mem Kernel
VM
I/O Ops
Kernel Mode
KVM
Driver
The guest OS has its own user and kernel modes, and all the user mode functions of the
guest OS are executed within this guest-user mode. When the guest OS tries to access
the guest-kernel mode, the process exits from the guest-user mode. The Linux/KVM
user mode then performs I/O on behalf of this guest.
84 Chapter 3: Hypervisor Considerations for the CSR
libvirtd
Management Tools
Guest
qemu-kvm
Drivers
Hardware
The loadable kernel module consists of three files: kvm.ko, kvm_intel.ko (for
Intel processors), and kvm_amd.ko (for AMD processors). Linux kernel 2.6.20 and
above have these modules included as part of the Linux kernel. kvm.ko provides the
core virtualization infrastructure, while kvm_intel.ko and kvm_amd.ko provide the
processor-specific module. These KVM modules are the ones that get the virtualization
capability to the Linux kernel. This kernel module is responsible for resource
management (memory and CPU) for the virtualized environment. KVM natively
performs only downstream functions, such as managing the hardware memory and CPU
scheduling. It provides a control interface through a set of APIs: ioctl() calls for tools
(such as QEMU) to emulate hardware virtualization.
Linux Kernel runs the guest process over KVM as a normal Linux process. It does a
malloc() to allocate memory, and it frees, swaps, and overcommits memory just like a
normal process memory.
Each guest maintains its own page table, and so it believes it is doing its own memory
management. However, KVM/Linux cannot allow guest operating systems to modify
the page table the kernel uses to write to hardware memory. So the host kernel inter-
cepts all the guest memory management unit (MMU) operations and maintains its own
shadow table, which is a replica of the guest’s page table. The host kernel uses this
shadow table to write to its physical memory. This way, the guest’s virtual memory gets
mapped to the host’s physical address. However, KVM memory is transparent to the
Linux kernel, and Linux treats the memory allocated by KVM no differently than the
memory allocated to other Linux processes. Linux tries to swap, free, or replace this
memory just as it does for regular process memory. Trouble comes when the guest tries
to access the memory Linux just freed. To get around this problem, a feedback mecha-
nism exists with the KVM module that updates the guest of any changes to the shadow
tables. mmu_notifiers updates the guest about changes to the shadow tables. Only
www.allitebooks.com
KVM 85
after the guest has updated its page table is the shadow table updated.
1. The Guest OS calls for memory. The KVM calls malloc(), and a virtual address
space is allocated with no physical memory to back it up.
2. When the guest OS process first tries to access this virtual memory, a page fault is
generated on the host because there is no physical memory allocated.
3. The kernel calls do_page_fault() where the malloc() was called and thus allocates
physical memory to it. So now the virtual memory has some physical memory to
back it up.
4. KVM links the malloc()ed virtual address to the physical address allocated on the
host and updates rmaps.
5. The kernel calls mmu notifier to create an entry for the new page created.
6. The host returns from page_fault, and the guest resumes regular operations.
As mentioned earlier, the KVM architectural approach is to utilize most of the Linux
kernel functionality and add only what is required for virtualization support. With CPU
scheduling, each vCPU (virtual CPU—that the guest OS schedules its processes on) is
mapped to a Linux process that uses hardware assistance. This means the vCPU is just
like any other Linux process, and Linux uses its CFS (Completely Fair Scheduler) to
schedule it on the hardware CPU.
Note As the name suggests, as CFS deploys the algorithm, it attempts to be fair to all
processes by providing a fair chance to each process to run on the processor. It keeps an
account of time spent by the processor on a process. It tries to prioritize processes that have
had less time on the CPU and deprioritize processes that spend a lot of time on the CPU.
One tool that enables users to emulate virtual machines is Quick Emulator (QEMU).
When the VMs spawned by QEMU want to perform an I/O operation, they are
intercepted and handled by KVM.
86 Chapter 3: Hypervisor Considerations for the CSR
Note Many people compare QEMU and ESXi host, but this is not really fair. QEMU is
just a machine emulator and not a hypervisor, as ESXi is. QEMU can help you create and
manage VMs when presented with a pool of virtual resources. QEMU, when paired with
KVM within the Linux kernel, is a complete package for running and managing VMs.
When VMware offers functionalities such as creating and managing VMs, it packages
them in ESXi.
QEMU is a free open source tool that emulates the complete hardware of a computer
device. QEMU can run on a variety of operating systems and processor architectures. So,
unlike VMware, it is not limited to an x86 architecture.
As a guest emulator, QEMU should be able to emulate guest operating systems that
run on a physical CPU. In order for QEMU to achieve this, it should be able to do the
following:
■ Handle timers
To achieve this, QEMU must be able to execute guest code and schedule its resources in
a way that does not pause execution when the I/O response takes a while to complete.
There are two architectures available to achieve this:
QEMU uses an architecture that is an amalgamation of the two mentioned here. Its code
is event driven, and at the same time, it also uses threads.
The QEMU core is mainly event driven. It is based on an event loop, which dispatches
events to handler functions. main_loop_wait() is the main event loop in the QEMU
core. This is what it does:
■ Waits for file descriptors to become readable or writable. You use quemu_set_fd_
handler to add file descriptor. This registers the file descriptor with the main loop
and tells the main loop to wake up whenever certain conditions are met.
■ Runs bottom halves, which are timers that expire immediately. These are used to
avoid overflowing the call stack. You add them by using qemu_bh_schedule.
www.allitebooks.com
KVM 87
When any of these three events occur, main_loop_wait() invokes a callback that
responds to the event. The callback should be quick to prevent the system from being
unresponsive.
■ KVM—As discussed earlier, the KVM module virtualizes the hardware resources
and presents the /dev/kvm interface in the Linux file system.
Both TCG and KVM allow the execution control to be given to the guest and allow the
guest to execute its code.
QEMU has one thread per vCPU plus a dedicated event loop thread. This is called the
IOTHREAD model. In the older non-IOTHREAD model, one QEMU thread executed the
guest code and the event loop. In the new IOTHREAD model, each vCPU thread executes
the guest code, while the IOTHREAD runs the event loop.
Figure 3-16 shows the architecture of a system using KVM and QEMU.
QEMU-KVM
Guest
/dev/kvm
Linux Kernel
x86 Architecture
Hardware
Libvirt is designed to mainly manage virtual machines. It offers the following types of
management:
■ Network management—Any host that runs the libvirtd daemon can be used to
manage physical or virtual interfaces and physical or virtual networks. When the
libvirtd system daemon is started, a NAT bridge is created. This is called default
and allows external connectivity. For other network connectivity, you can use the
following:
■ A virtual network that enables you to share data with other virtual machines
■ A macvtap interface that connects directly to the physical interface on the server
on which you host the VM
■ Storage management—Any host that runs the libvirtd daemon can be used to
manage various storage types and file formats.
www.allitebooks.com
KVM 89
libVirt
VMware KVM
x86 Architecture
Table 3-1 shows the command-line tools you can use with virsh to manage guest VMs.
Command Description
dominfo Displays guest information.
domname Displays the guest’s name.
domstate Displays the state of a guest.
quit Quits the interactive terminal.
reboot Reboots a guest.
restore Restores a previously saved guest stored in a file.
resume Resumes a paused guest.
save Saves the present state of a guest to a file.
shutdown Gracefully shuts down a guest.
suspend Pauses a guest.
undefine Deletes all files associated with a guest.
migrate Migrates a guest to another host.
You can also use virsh to manage the resources that are given to the guest or
hypervisor with the commands shown in Table 3-2.
www.allitebooks.com
Hyper-V 91
With these commands you can manage the guest effectively and automate a lot of guest
bring-up and configuration sequences.
virt-manager provides a graphical way to manage VMs and hypervisors. It does the
same things as virsh but via a user-friendly graphical interface.
Hyper-V
Hyper-V, formerly known as Windows Server Virtualization, can create virtual machines
on the x86 platform. In October 2008, Hyper-V Server 2008 was released, and since then
it’s been used as a hypervisor platform for other members of the Windows Server family.
Hyper-V is a type 1 microkernel hypervisor that resides directly on the hardware. The
microkernel architecture optimizes performance and reduces adoptability issues with the
underlying hardware.
The architecture uses a parent VM that hosts the drivers. The guest operating system
interfaces with the parent partition to access hardware resources’ memory, CPU, storage,
and so on. The guest operating system works within the privileged boundary. Figure
3-18 gives a high-level overview of the Hyper-V architecture.
Child Partition
Parent Partition
Virtual
Service
Provider
Drivers
VM Bus
Hyper Calls
Microsoft Hypervisor
Hardware
On the top of the hypervisor is one parent partition and one or more child partitions.
This partitioning creates virtual isolation within the hypervisor for physical memory
address space and virtual processors.
The child partition can host a guest operating system or system functions for Microsoft
Windows Server. For example, when virtual Hyper-V stack management gets installed in
the parent partition, the subsidiary Windows Server functionality is loaded in the child
partition. Each virtual machine has its own security boundary. Microsoft refers to this as
an operating system environment (OSE) that defines the component of a virtual machine.
The VM has its own identity, computer account, IP address, and network resources.
Child partitions have only the virtual view of the hardware resources. The child partition
sends a request to the virtual devices, which gets redirected to the hypervisor in the
parent partition that handles this request.
The parent partition has access to hardware devices and controls the resources used by
the child partition. The child partition accesses the hardware resource using the drivers
in the parent partition space. The parent partition also acts as the broker among the
multiple child partitions and the hardware for accessing the resources. All data and
instructions between the parent and child partition go through the virtual machine bus.
The architecture allows the user to leverage plugin devices, which in turn allow direct
communication between parent and child partitions.
Xen
Xen is another type 1 hypervisor that supports para-virtualization. Xen originated as a
college research project at the University of Cambridge. Ian Pratt, who was a lecturer in
Cambridge, led this project and later cofounded XenSource, Inc. First available in 2004,
Xen was originally supported by XenSource, Inc. Eventually, Xen was moved under the
Linux Foundation as a collaborative project.
Starting with Xen 3.0, all guest VMs can run their own kernels and take advantage of
para-virtualization, which removes the need to emulate virtual hardware resources,
makes the guest aware of the hypervisor, and enables access to the hardware resources
for I/O efficiency. Instead of the guest spending time and extra cycles performing tasks
to get resources from the virtual environment, these guests can use the hooks of para-
virtualization to allow guest and host to communicate and acknowledge these tasks.
www.allitebooks.com
Xen 93
Kemel
Xen Bus
Xen Hypervisor
Hardware
■ Xen hypervisor—The hypervisor sits directly on the hardware. Its key functions
are CPU scheduling and memory partitioning. The VM that requires abstraction of
hardware and resources can leverage the hypervisor for control of shared resources.
The hypervisor does not have knowledge of networking, storage, or the common I/O.
■ Domain 0 —This VM resides on the Xen hypervisor and owns the right to access
physical I/O resources as well as interact with other guest VMs that need access to
these resources. Domain 0 has to be set up before any other guest VMs are spawned.
Summary
Now that you’ve read this chapter, you should have a basic understanding of operating
systems and types of hypervisors. This chapter reviews the details of KVM and ESXi
and provides an overview of Hyper-V and Xen. Understanding the hypervisor types is
important for the deployment of CSR 1000V.
Table 3-3 provides a summary of the hypervisor types you need to know about.
All of these hypervisors support the key features, so the use cases depend on the
environment and operational support criteria.
www.allitebooks.com
Chapter 4
This chapter describes the software design of the CSR 1000V and details the data plane
design. It also illustrates the software implementation and packet flow within the CSR
1000V, as well as how to bring up the CSR 1000V.
System Design
CSR 1000V is a virtualized software router that runs the IOS XE operating system.
IOS XE uses Linux as the kernel, whereas the IOS daemon (IOSd) runs as a Linux
process providing the core set of IOS features and functionality. IOS XE provides a
native Linux infrastructure for distributing the control plane forwarding state into an
accelerated data path. The control and data planes in IOS XE are separated into differ-
ent processes, and the infrastructure to communicate between these processes supports
distribution and concurrent processing. In addition, IOS XE offers inherent multicore
capabilities, allowing you to increase performance by scaling the number of processors.
It also provides infrastructure services for hosting applications outside IOSd.
Originally, IOS XE was designed to run on a system with redundant hardware, which
supports physical separation of the control and data plane units. This design is imple-
mented in the ASR 1006 and ASR 1004 series routers. The original ASR 1000 family
hardware architecture consisted of the following main elements:
■ Chassis
The RP is the control plane, whereas the ESP is the data plane. In an ASR 1006 and
ASR 1004, the RP and ESP processes have separate kernels and run on different sets of
hardware. ASR 1000 was designed for high availability (HA). The ASR 1006 is a fully
96 Chapter 4: CSR 1000V Software Architecture
hardware redundant version of the ASR, and its RP and ESP are physically backed up by
a standby unit. IOSd runs on the RP (as do the majority of the XE processes), and the
RP is backed up by another physical card with its own IOSd process. The ASR 1004 and
fixed ASR 1000s (ASR 1001-X and ASR 1002-X) do not have physical redundancy of
the RP and ESP.
In the hardware-based routing platform for IOS XE, the data plane processing runs out-
side the IOSd process in a separate data plane engine via custom ASIC: QuantumFlow
Processor (QFP). This architecture creates an important framework for the software
design. Because these cards each have independent processors, the system disperses
many elements of software and runs them independently on the different processors.
Tip The ASR 1000 platform first introduced IOS XE. Multiple products run IOS XE,
including the following:
ASR 1000 family:
■ ASR 1001-X
■ ASR 1002-X
■ ASR 1004
■ ASR 1006
■ ASR 1006-X
■ ASR 1009-X
■ ASR 1013
ISR family:
■ ISR 4321
■ ISR 4331
■ ISR 4351
■ ISR 4431
■ ISR 4451-X
IOS XE retains the look and feel of IOS. However, because IOS runs as a Linux process,
it enables the platform-independent code to reside inside the IOSd process running on
the Linux kernel. By moving the platform-dependent code (drivers) outside the IOSd
process, it makes IOS XE a very efficient software delivery model. Different platforms
write their drivers and leverage the existing feature-rich control plane code from IOSd.
Multiple platforms run IOS XE. However, when understanding CSR 1000V architecture
in this chapter, ASR 1000 is used as a hardware example because it was the first platform
to run IOS XE.
www.allitebooks.com
System Design 97
As the need for smaller form factor ASRs arose, a one rack unit (RU) ASR 1000 was con-
ceptualized and developed: ASR 1001. The ASR 1001 is a 64-bit architecture in which all
processes (RP, SIP, and ESP) are controlled by a single CPU. The SPA interface complex,
forwarding engine complex, and IOS XE middleware all access the same Linux kernel.
This is achieved by mapping the RP, ESP, and SIP domains into logical process groups.
The RP’s process domain includes IOSd, a chassis manager process and forwarding man-
ager. The ESP process domain includes the chassis manager process, QFP client/driver
process, and forwarding manager.
The architecture diagram in Figure 4-1 provides a high-level overview of the major
components.
RP
SPA ESP
ESP
Linux Kernel
■ RP—RP mainly contains the IOS daemon (IOSd), the forwarding manager for RP
(FMAN-RP), the chassis manager for RP (CMAN-RP), the kernel, and bootstrap
utilities.
98 Chapter 4: CSR 1000V Software Architecture
■ SIP/SPA—SIP/SPA houses the I/O interface for the chassis. It has its own CMAN
and kernel process to handle the discovery, bootstrapping, and initialization of the
physical interfaces.
■ The kernel utilities have been shared across the RP and ESP software complexes.
■ The kernel utilities use the virtualized resources presented to it by the hypervisor.
The CSR is basically the ASR 1000 design stripped of its hardware components. When
you compare the two designs, you find that the data path implementation is very differ-
ent. This is because the ASR 1001 has a physical processor (the QFP) for running data
path forwarding. In a CSR, the IOS XE data path is implemented as a Linux process.
The CSR 1000V is meant to leverage as much of the ASR 1001 architecture as possible.
There are places in the CSR 1000V system where software emulation for hardware-
specific requirements is needed. In general, the software architecture is kept the same,
using the same grouping approach as for the hardware components. One of the major
engineering efforts has been focused on migrating the QFP custom ASIC network pro-
cessor capabilities onto general-purpose x86 CPU architectures and providing the dis-
tributed data path implementation for IOS XE. This effort creates a unique opportunity
for Cisco to package this high-performance and feature-rich technology into the CSR
1000V. Figure 4-2 shows the high-level architecture of the CSR 1000V.
www.allitebooks.com
System Design 99
CSR 1000V
RP Complex DP Complex
Packet Processing
IOSd
PPE PPE PPE
Gethd Driver FMAN-RP
Client
Driver
Kernel
When a CSR boots up as a virtual machine, interfaces are discovered by parsing the con-
tents of /proc/net/dev on the Linux kernel. The gethd (Guest Ethernet Management
Daemon) process performs the port enumeration at startup and then passes the interface
inventory to the guest Ethernet driver within the IOS complex. The IOSd gethd driver
then instantiates the Ethernet interfaces. This is how the I/O interfaces provided by the
virtual NIC are managed by IOS.
The gethd process manages the interfaces on the CSR VM. It takes care of addition,
removal, configuration, states, and statistics of the Ethernet interfaces on the CSR VM.
Guest VM Boots Up
Figure 4-4 illustrates the CSR 1000V data plane architecture. The HW threads men-
tioned in the figure are packet processing engine (PPE) threads. The terms HW and PPE
can be used interchangeably.
www.allitebooks.com
System Design 101
CSR 1000V
Client
Driver
Kernel
The following is an overview of the three main components that make up the packet-
processing data plane for CSR:
■ Client—The Client is software that ties together the control plane and the data
plane. It is a collection of software modules that transform control plane informa-
tion into various data plane forwarding databases and data structure updates. It is
also responsible for updating the control plane with statistics from the data plane.
It allocates and manages the resources of the uCode, including data structures in
resource memory. The QFP Client is also responsible for restarting the QFP pro-
cess in the event of failure. The Client provides a platform API layer that logically
sits between IOSd and the uCode implementing the corresponding features. The
Client API is called from FMAN-FP and then communicates with the uCode via
both Interprocess Communicator (IPC) and shared memory interfaces provided by
the Driver. Within the Client, feature processing support can be broken down into
functional blocks known as Execution Agents (EA) and Resource Managers (RM).
RMs are responsible for managing physical and logical objects, which are shared
resources. An example of a physical object manager is the TCAM-RM, which man-
ages allocation of TCAM resources, and an example of a logical object manager is
the UIDB-RM, which manages the micro Interface Descriptor Block (uIDB) objects
102 Chapter 4: CSR 1000V Software Architecture
used to represent various forms of interfaces. The data plane (uCode) uses uIDB
objects to see the logical interfaces.
■ QFP uCode (packet processing) —The uCode is where all the feature packet pro-
cessing occurs. The uCode runs as a single process in the same VM/container as the
Client and the Driver processes. IOSd initiates a packet process request through
FMAN-FP. This request is then driven by the Client and the Driver interacting with
the uCode to control the PPE behavior. The QFP uCode is broken up into four
main components: Feature Code, Infrastructure, Platform Abstraction Layer (PAL),
and Hardware Abstraction Layer (HAL). The PAL and HAL are essentially glue for
the portability of software features to different hardware platforms. Originally, the
PAL and HAL were designed for Cisco forwarding ASICs, such as QFP. In order
for uCode software to run on top of x86 in a Linux environment, a new PAL layer
is needed to support the specifics of the CSR 1000V platform. In addition, a new
HAL is introduced for running QFP software on top of x86 in a Linux environment.
The intention is for the CSR 1000V data plane to leverage as much of the existing QFP
code base as possible to produce a full-featured software data plane capable of leverag-
ing the processing capacity and virtualization capabilities of modern multicore CPUs.
One way to minimize changes to the existing QFP software code base is to emulate QFP
hardware ASIC in such a way that the existing Client, Driver, and QFP uCode are not
aware that they are running on a non-QFP platform. However, due to the complexity of
QFP hardware and the differences in platform requirements, a pure emulation is imprac-
tical. There are some cases where we choose to emulate hardware because doing so is
the straightforward approach for code leverage. In other cases, it is best to replace the
corresponding functionality with an implementation that is compatible at an API level
but may be a completely different algorithmic implementation.
www.allitebooks.com
Life of a Packet on a CSR 1000V: The Data Plane 103
The CSR 1000V runs completely on general-purpose CPUs without an offload engine;
therefore, the software implementation of the IPsec/crypto feature path is needed to
support the encryption function. To that end, the CSR 1000V includes a software crypto
engine that uses low-level cryptographic operations for encrypting and decrypting traf-
fic. The software crypto engine is presented to the IOS as a slower crypto engine. One
thing to note is the software crypto engine runs as an independent process within the
CSR 1000V, and it therefore may run as a parallel process in a multicore environment.
To improve the crypto performance of the CSR 1000V software router, the crypto data
path is implemented to take advantage of the latest Advanced Encryption Standard
(AES) crypto instruction set from Intel (AES-NI) for encryption/decryption operations.
The newer Intel processors, such as the Xeon Westmere-EP family and mobile Sandy
Bridge family, provide instruction sets for enhancing Advanced Encryption Standard
(AES-NI) cryptographic operations performance. These instructions are included in the
CSR 1000V crypto library, along with other cryptographic and hash algorithms for low-
level crypto operations. The crypto library is used by the software crypto engine as well
as by other subsystems within IOS that require cryptographic operations. The inclusion
of Intel’s crypto instruction set allows the CSR 1000V to take advantage of the latest
Intel CPUs for encryption and decryption operations in the data path.
User
QFP Data Plane Space
CSR VM
System Call
Netmap Kernel
Software Switch
Physical NIC
From Figure 4-5, you can see that the hypervisor presents a virtual NIC to its guest
VM by using a driver. This driver can either be a para-virtualized driver (for example,
VMXNET3) or a real/emulated driver (for example, e1000). Para-virtualized drivers are
native to hypervisors and perform much better than emulated drivers such as the e1000.
Hypervisors support emulated drivers because they are required for full virtualization.
Recall from Chapter 1, “Introduction to Cloud,” that in full virtualization, guest operat-
ing systems do not require any support from the hypervisor kernel and run as though on
real hardware. Therefore, support for emulated drivers is required. However, the perfor-
mance of emulated drivers is much lower than that of para-virtualized drivers. The CSR
VM supports para-virtualized drivers only.
Netmap I/O
Netmap is an open-source I/O infrastructure package that enables the CSR VM to get
rid of the multiple software layers in the traditional Linux networking stack I/O model.
This results in faster I/O. Understanding the Netmap I/O model will help you better
understand packet flow to and from a CSR VM. This section provides an overview of
the Netmap I/O model and compares it with a Linux I/O model. It is important to under-
stand the I/O model before drilling down to packet flow.
www.allitebooks.com
Life of a Packet on a CSR 1000V: The Data Plane 105
Netmap is designed to strip down software layers and get the frame from the wire to the
data plane process in user space as quickly as possible. Netmap achieves this through the
four building blocks of its I/O architecture:
■ Thin I/O stack—Netmap bypasses the Linux networking stack to reduce overhead.
Since the CSR data plane runs in the user space, when it wants an I/O architecture to
deliver receive (Rx) frames from the NIC to the user space (data plane) and transmit
(Tx) frames from the data plane to the NIC, it leverages Netmap’s thin I/O stack.
■ Zero copy—Netmap maps all memory from rings (pool of memory buffers) in a
way that makes them directly accessible in the data plane (user space). Hence there
is no copy involved in getting the information to the user space. Preventing a copy
operation saves a lot of time in an I/O model, and Netmap’s zero-copy model is
very effective at increasing performance compared to a traditional Linux I/O model.
■ Minimal ring manipulation—In the Netmap I/O architecture, the ring is sized such
that the producer accesses the ring from the head end, while the consumer accesses
it from the tail. (Producer and consumer are terms associated with the process that
tries to initiate the I/O process [producer] and a process that gets affected in trying
to serve the producer [consumer].) The access to the ring is allowed simultaneously
for the producer and the consumer. In a regular Linux I/O scenario, you would
have to wait for the host to fill up the ring with pointers to buffers. When the ring
is being serviced, Linux detaches the buffers from the ring and then replenishes the
ring with new pointers.
System Call
Ring A
Buffer User Space
Shared Memory
System Call System Call
Network Protocol
Packet Flow
There are three major data plane components:
■ Rx thread
■ Tx thread
All these components run on a single process within the QFP process umbrella. Multiple
PPE threads serve requests within this QFP process. The following sections discuss the
flow.
www.allitebooks.com
Life of a Packet on a CSR 1000V: The Data Plane 107
1. During boot-up, the platform code within IOSd discovers all Linux network
interfaces. The platform code then maps these Linux interfaces—eth0, eth1, and
so on—to Gig0, Gig1, and so on. After talking to the kernel, platform code sets
up the interface state (up or down), sets the MTU, sets the ring size, and sets the
MAC address.
2. The FMAN process creates the FMAN interfaces and then reaches out to the QFP
client process to initialize the data-plane interface.
3. After the QFP process receives the initialization message from the Client process
to create an interface, the QFP process then initializes an interface called micro-
interface descriptor block (uIDB) in the data plane.
4. After the uIDB is created in the QFP process, the FMAN process binds this uIDB to
the network interface name.
5. The component of the data-plane process responsible for interacting with the kernel
now has to make sure that the interface created with the QFP process is registered
and enabled within the Netmap component of the kernel.
6. With the new interfaces registered, the Netmap component communicates with the
virtual NIC driver to initialize the physical NIC.
7. The vNIC driver opens the NIC, initiates the rings, and makes the NIC ready for
operation.
TX Flow
The following events take place when there is a packet to be transmitted (Tx) by the CSR
onto the wire:
2. The HQF thread checks congestion on the transmit interface and checks the inter-
face states.
3. If the transmit interface is not congested, HQF sends the frame. HQF can also
wait to accumulate more frames, batch them, and then send them out.
4. The platform code locates the next available slot in the Tx ring and copies the
frame from the source buffer into the Netmap buffer for transmission.
RX Flow
The following events occur whenever a CSR receives a packet to be processed:
1. The Rx thread (the thread that receives frames from the QFP process) issues a poll
system call to wait for the new Rx frames.
2. When a new frame arrives, the NIC (or vNIC, in this case) accesses the vNIC Rx ring
to get a pointer to the next Netmap buffers.
3. The vNIC puts the frame onto the next Netmap buffers.
7. The Rx thread pushes the frame to the PPE thread for processing.
QFP Client
Process
PPE Threads
PPE Threads
HQF Thread
QFP Process
Rx Thread
Kernel
www.allitebooks.com
Life of a Packet on a CSR 1000V: The Data Plane 109
1. The QFP process receives the frame from the Netmap Rx and stores it in Global
Packet Memory (GPM).
2. The Dispatcher copies the packet header from the GPM and looks for free PPE to
assign. The packet remains in the GPM while it is being processed by the PPEs.
3. The Dispatcher assigns a free PPE thread to process the feature on the packet.
4. PPE threads process the packet and gather the packets. The gather process copies
the packets into B4Q memory and sends the HQF thread a notification that there is
a new packet in the B4Q memory.
5. HQF sends the packet by copying it from B4Q into the Netmap Tx ring, and then
releases the B4Q buffer.
6. The Ethernet driver sends the frame and frees the Tx ring once the packet has been
sent out.
7. Multicast IPsec packets are recycled from the HQF thread back to the in/out pro-
cessing of the PPE threads.
Recycled Packet
Netmap Netmap
GPM B4Q
Rx QQ Tx
Packet Output
PPE HQF
DST PPE
Thread Thread
IPC PPE
Thread IPC
Thread
Rx QQ Tx
These phases can be subdivided into the step-by-step procedures described in the fol-
lowing sections. To learn about automated provisioning using the BDEO (build, deploy,
execute OVF), see Chapter 7, “CSR in the SDN Framework.”
The following steps assume ESXi is already installed. Please refer to the VMware ESXi
installation guide for setting up the ESXi if it is not already installed.
Figure 4-9 Installing the OVF Template for the CSR 1000V
www.allitebooks.com
Installing the CSR 1000V on a VMware Hypervisor 111
3. Upload the CSR OVF file you downloaded from cisco.com as shown in
Figure 4-9.
Step 3. When the OVA upload is done, verify the OVF template details on the
screen shown in Figure 4-11.
112 Chapter 4: CSR 1000V Software Architecture
Figure 4-11 Deploying the OVF Template: Verifying the Template Details
The release information, product, size, and so on are received from the meta-
data. Follow the directions for creating the VM.
www.allitebooks.com
Installing the CSR 1000V on a VMware Hypervisor 113
Figure 4-12 Deploying the OVF Template: Selecting the System Memory Profile for
CSR 1000V
2. Select the appropriate type of disk formatting (see Figure 4-13), and
then click Next:
Figure 4-13 Deploying the OVF Template: Choosing the Disk Provisioning Format
Note The OVF used here is for version 3.13. You might see variations in the default
settings with later versions. Please refer to Cisco release documentation.
www.allitebooks.com
Installing the CSR 1000V on a VMware Hypervisor 115
Step 4. When the deployment of the CSR 1000V is complete, boot the router by
selecting the VGA console from the GRUB menu on the Console tab shown
in Figure 4-16.
www.allitebooks.com
Installing the CSR 1000V on a VMware Hypervisor 117
Step 5. At the router prompt, enter platform console serial, as shown in Figure
4-17. (This command causes the VM to send console information on the
serial port from ESXi in the later steps.)
Step 6. To add the serial port for console access, access the vCenter web client
and select Virtual Hardware, Network Adaptor, Serial Port, as shown in
Figure 4-18.
Step 7. Shut down the guest OS as shown in Figure 4-19. (Note that this serial port
will be used for terminal access to the CSR.)
www.allitebooks.com
Installing the CSR 1000V on a VMware Hypervisor 119
Figure 4-19 Configuring the Serial Interface: Shutting Down the Router
Step 8. Select Add New Device, New Serial Port and provide the IP address and ter-
minal port details to access the CSR, as shown in Figure 4-20.
120 Chapter 4: CSR 1000V Software Architecture
Figure 4-20 Configuring the Serial Interface: Setting the Telnet Address
Step 9. Go to vCenter and select Setting, Security Profile. Edit security configuration
ports 23 and 1024 as shown in Figure 4-21. This is needed because by default
ESXi blocks console access.
www.allitebooks.com
Installing the CSR 1000V on a VMware Hypervisor 121
Step 11. Use Telnet to verify the access from the PC. (It’s a good practice to use SSH
for accessing the CSR VM; however, for the sake of simplicity, this example
shows Telnet access setup.) The EXSi hypervisor defaults the network
connections to the VM Network virtual switch connection. The network
122 Chapter 4: CSR 1000V Software Architecture
Step 12. To remap the network adapters to corresponding vNICs, you should perform
the following steps. From the vSphere client in the Edit Settings window, select
New Device Add, Networking and add vNICs to the CSR as assigned inter-
faces (from the vCenter web client), as shown in Figures 4-24 through 4-27.
(Allow all VLANs and create a bridgeForVNIC1 label for this connection.)
Figure 4-24 vNICs and the CSR 1000V: Selecting the Connection Type
www.allitebooks.com
Installing the CSR 1000V on a VMware Hypervisor 123
1. Select the new vNIC, as shown in Figure 4-25, to create a new standard
switch name.
Figure 4-25 vNICs and the CSR 1000V: Creating a Standard Switch
2. Add VLANs and the network label assigned for the vNIC, as shown in
Figure 4-26.
Figure 4-26 vNICs and the CSR 1000V: Setting the Connection Settings
124 Chapter 4: CSR 1000V Software Architecture
3. Complete the configuration of the vNIC with a VLAN and label attach-
ment that can be referenced in a vSwitch. Click Finish to complete this
step, as shown in Figure 4-27.
Figure 4-27 vNICs and the CSR 1000V: Completing the Configuration
Step 13. Go to the vSphere web client and select Virtual Machine, Network Adapter.
In the Networking tab, look for the new bridgeForVNIC1 label you created
earlier, as shown in Figure 4-28. You should note that this label acts as map-
ping between the CSR interface and the vNIC.
www.allitebooks.com
Installing the CSR 1000V on a VMware Hypervisor 125
Figure 4-28 vNICs and the CSR 1000V: Editing the Settings
To map the network adapter to the vNIC created, select the vNIC label cre-
ated in the previous step. The CSR 1000V is now configured and connected
to the physical NIC, as shown in Figure 4-29.
126 Chapter 4: CSR 1000V Software Architecture
Figure 4-29 vNICs and the CSR 1000V: Interface Summary Screen
or like this:
yum install virt-manager
yum install qemu-kvm
yum install bridge-utils
Figure 4-30 shows the installation of packages required for CSR creation.
www.allitebooks.com
Installing the CSR 1000V on a KVM Hypervisor 127
Step 2. Launch Virtual Machine Manager, which is the front end to KVM/QEMU
that allows installation and management of CSR VMs, by selecting
Application, System, Virtual Machine Manager.
Note Virtual Machine Manager could also be on a different path for your Linux server.
Figure 4-31 shows the launch of the virtual machine from QEMU. Make sure you have
XDesktop installed. Also note that VMM is not a mandatory requirement for using
KVM/QEMU, especially when a graphical user interface is not present on a desktop.
Click the Create a New Virtual Machine icon, and the dialog shown in Figure
4-31 appears. Click the Forward button.
128 Chapter 4: CSR 1000V Software Architecture
Step 3. Load the ISO image (which you download from software.cisco.com) for the
CSR 1000V, as shown in Figure 4-32. Click the Forward button.
www.allitebooks.com
Installing the CSR 1000V on a KVM Hypervisor 129
Note Download the ISO CSR 1000V image to your local hard disk. When you
download it, it is named csr1000v-universalk9.<version>.std.iso, but the file is renamed
ultra.iso in the example shown.
Step 4. Allocate hardware resources for the guest VM as shown in Figure 4-33.
(Refer to Table 2-2 in Chapter 2 for further allocation information.) Click
Forward.
Step 5. Select hardware resources, as shown in Figure 4-34, and click Forward.
Note If you do not check Allocate Entire Disk Now, only a small portion of memory
asked for will be allocated. It will keep growing as memory requirements increase.
Checking Allocate Entire Disk Now guarantees that much storage.
Step 6. Look over the hardware resources summary (see Figure 4-35) and make any
changes needed. Click Finish.
Step 7. To apply changes for the guest VM, select Application, System, Virtual
Machine Manager and highlight the CSR installed in the VMM. Then click
the Show Virtual Hardware Details tab and click the Add Hardware button,
as shown in Figure 4-36.
www.allitebooks.com
Installing the CSR 1000V on a KVM Hypervisor 131
Step 8. To create serial connection access for console access, select Serial, and then
select TCP for Device Type and provide the telnet information, as shown in
Figure 4-37.
Step 9. In the Virtual Machine Manager, highlight the guest VM and shut it down (if
it is not down already). (See Figure 4-38.)
Step 10. Access the router from the console, as shown in Figure 4-40. Make sure the
VM is powered up before you try to access it.
www.allitebooks.com
Installing the CSR 1000V on a KVM Hypervisor 133
Step 11. Use the serial interface command for telnet access: platform console
serial and write mem, as shown in Figure 4-41.
Step 12. Access the CSR 1000V via the telnet, as shown in Figure 4-42.
134 Chapter 4: CSR 1000V Software Architecture
Step 13. Ensure that your virtual machine is shut down, and then start vNIC
provisioning by selecting Show Virtual Hardware Details, NIC, as shown in
Figure 4-43.
Step 14. In the Virtual Machine Manager, select virtio as the device model (see Figure
4-44) because it is the para-virtualized driver in Linux. Using virtio is the best
way to exploit the underlying kernel for I/O virtualization. It provides an
efficient abstraction for hypervisors and a common set of I/O drivers.
www.allitebooks.com
Installing the CSR 1000V on a KVM Hypervisor 135
Select the virtual network with NAT to tie all VMs in the same bridge
domain and NAT it to the outgoing physical interface (see Figure 4-45).
Attach the other NIC to the bridge tap.
VM-A VM-B
eth0 eth0
Tap 0 Tap 1
Step 15. Configure the mapping of the vNIC to the physical interface:
www.allitebooks.com
Performance Tuning of the CSR 1000V 137
Step 16. In the Virtual Machine Manager, select Show Virtual Hardware Details.
■ Hypervisor scheduling
■ CPU pining
This section reviews the common tuning practices for an ESXi host. The scheduler for
ESXi is responsible for vCPU, IRQ (interrupt requests), and I/O threads. To provide
equal treatment to all guest VMs, the scheduler provides allocation of equal resources of
vCPU threads for scheduling. Note that you can relax coscheduling of threads to avoid
synchronization latency.
To tweak the scheduling and resource allocation details, you must access the VM setting
using vSphere client and follow these steps:
1. In the vSphere client inventory, right-click the virtual machine and select Edit
Settings.
The Processor Affinity setting (CPU pining) restricts VMs to a particular set of cores
by defining the affinity set. The scheduling algorithm aligns with process affinity for
assigning the resources used for the tasks. Figure 4-49 assumes two tasks: Task 1 and
Task 2. Task 1 has affinity to processor 1 and is using it. When Task 2 needs a resource,
the scheduler uses a second processor. Task 2 then acquires affinity with the second
processor.
pCPU 2 pCPU 3
pCPU 2 pCPU 3
www.allitebooks.com
Summary 139
To tweak these settings, access the vSphere client and follow these steps:
1. In the vSphere client inventory panel, select a virtual machine and select Edit
Settings.
Hyperthreading by definition allows a single physical core to have two logical cores; that
is, a single core can execute two threads at a given time. Each process from the guest
VM can be split into multiple threads to a logical CPU, and the CPU can handle multiple
threads of independent tasks. The main function of hyperthreading is to increase the
number of tasks in the pipeline by creating parallel pipelines. By tweaking the process
affinity option, you can restrict VMs to a particular set of cores and unhook the VM
from processor scheduling. Most of the hypervisors use BIOS settings to modify the
hyperthreading feature.
■ Use CPU pining to allow the guest VMs to dedicate one or more physical hardware
CPUs for processing.
These features are supported in all hypervisors, and it is important to understand the
settings on the hypervisor deployed in order to optimize guest VM performance with
features used on the hypervisor.
Summary
Now that you’ve read this chapter, you should have an understanding of the CSR 1000V
data plane architecture, as well as packet flow. You should also have an understanding of
the steps for bringing up a CSR 1000V on ESXi and KVM hypervisors.
This page intentionally left blank
www.allitebooks.com
Chapter 5
This chapter introduces some of the common deployment scenarios for the CSR 1000V
software router. After reading this chapter, you will be able to apply the CSR 1000V to
multiple deployment scenarios, such as VPN services extension, route reflector design,
branch designs, and Locator/ID Separation Protocol (LISP).
VPN Services
A virtual private network (VPN) is an extension of a private network that enables an orga-
nization to securely deliver data services across a public or untrusted network infrastruc-
ture such as the Internet. A VPN connection is a logical connection and can be made at
either Layer 2 or Layer 3 of the OSI (Open System Interconnect) model. In today’s envi-
ronment, a VPN typically uses encryption for data privacy and integrity protection.
Layer 2 VPNs
Layer 2 VPNs (L2VPN), as the name suggests, operate at Layer 2 of the OSI model. They
provide virtual circuit connections that are either point-to-point or point-to-multipoint.
The forwarding of traffic in L2VPN is based on Layer 2 header information such as
MAC address. One of the advantages of L2VPN is that it is agnostic to the Layer 3 traf-
fic protocol that is carried over it because the L2VPN operates at a lower OSI layer.
Layer 3 VPNs
Layer 3 VPNs (L3VPN) provide IP connectivity between sites and operate at Layer 3 of
the OSI model. L3VPNs have many flavors and can be either point-to-point or multi-
point connections for site-to-site route exchanges.
Provider Core
Router (P)
■ IPsec VPNs—IP Security (IPsec) is a suite of protocols that provides data privacy
and integrity protection. IPsec uses cryptographic security services to ensure that
communications over public networks such as the Internet stay confidential. As
www.allitebooks.com
VPN Services 143
described in the following sections, IPsec VPNs can be classified into two catego-
ries: site-to-site VPNs and remote access VPNs.
Site-to-Site VPNs
Site-to-site VPNs offer secure connections between enterprise branch locations and
enable secure communication between the branch offices, the head office, and data
center. Site-to-site VPNs offer several advantages:
■ The IP addresses of branch LANs and head office/data center’s network information
are hidden inside IPsec-encrypted headers, away from external prying eyes.
■ They offer greater scalability, because it is easy to add a new site or an office to the
network, and it is very cost-effective as well because a new office can leverage pub-
lic Internet circuits for connection over the Internet.
There are several flavors of site-to-site IPsec VPNs, each with unique characteristics. The
following sections cover the various IPsec VPN services supported on the CSR platform.
The crypto access control list (ACL) in the IPsec policy determines what traffic will
be encrypted and what traffic will be left alone, without encryption. The data traffic
matching the permit statement in the crypto ACL is the “interesting traffic” to which the
encryption algorithm is applied. Network administrators must explicitly define a protec-
tion profile for every potential subnet that requires data encryption protection.
Classic IPsec is sometimes referred to as static routing VPN or policy IPsec VPN
because it is statically configured based on the security policy defined by network
administrators. The IPsec protection profile in the crypto map essentially acts as routing
entries and determines which traffic will be sent over the encrypted tunnel, while the
rest of the traffic will follow the existing routing rules. However, there are complexities
to be aware of with IPsec profiles. For example, the addition of a single subnet in the
VPN network requires configuration updates to the other VPN gateways in the network
that interact with the subnet. In addition, classic IPsec with crypto map lacks support for
IP multicast traffic as the original IPsec RFCs did not accommodate multicast traffic in
the IPsec requirement.
144 Chapter 5: CSR 1000V Deployment Scenarios
Because classic IPsec requires a network admin to explicitly define every potential IP
flow on every VPN gateway and because of the lack of multicast traffic support, this is
not a viable method for building a large complex network for an enterprise with hun-
dreds or thousands of subnets. A large IPsec VPN may require N2 IPsec crypto profile
configurations at the head end VPN gateway.
DMVPN offers a scalable IPsec VPN solution with flexible deployment topology. It
supports hub-and-spoke, partial-mesh, or dynamic full-mesh topology. DMVPN com-
bines several standards-based protocols:
■ IPsec
DMVPN uses the mGRE interface, which is a point-to-multipoint interface where a sin-
gle tunnel interface can terminate multiple GRE tunnels from the spokes. This effectively
reduces configuration complexity and allows incoming GRE tunnel connections from
any spoke, including the ones using dynamically allocated tunnel addresses. The mGRE
interface drastically simplifies the hub setup, allowing the hub tunnel interface configu-
ration to stay the same while more spoke routers are incrementally added.
The hub identifies a remote spoke’s tunnel endpoint address through NHRP. When the
remote peer builds the permanent tunnel to the hub router, it also sends an NHRP regis-
tration message to the hub over the tunnel. The NHRP registration message identifies the
tunnel endpoint address of the spoke router, thus preventing the hub router from hav-
ing to know the remote peer’s tunnel IP address in advance. Leveraging NHRP messages
provides the freedom for the spoke router to use dynamically assigned addresses as well
as to build dynamic site-to-site tunnels between the spokes for traffic communication
between the branches. The DMVPN topology is shown in Figure 5-2.
www.allitebooks.com
VPN Services 145
10.0.1.0/24
Dynamic
Spoke-to-Spoke
DMVPN Spoke-1 Tunnel DMVPN Spoke-2
10.0.2.0/24 10.0.3.0/24
DMVPN supports hierarchical hub deployment and allows a network to scale to tens
of thousands of branch nodes while offering any-to-any communication between the
branch sites. Incrementally adding more hubs for additional throughput performance or
scaling requirements can gain additional performance.
GET VPN is a tunnel-less VPN technology that offers end-to-end encryption for data
traffic and maintains full-mesh IPsec VPN topology without prior negotiation of point-
to-point tunnels, as shown in Figure 5-3. Like the other VPN technologies discussed so
far in this chapter, GET VPN leverages the same IPsec protocol suites to provide data
confidentiality and integrity protection. In addition, it introduces Group Domain of
Interpretation (GDOI), an IEFT standards-based protocol (RFC 6407), as the security
key management protocol.
146 Chapter 5: CSR 1000V Deployment Scenarios
Routing Group
Members Member
Group
Member
Group
Group Member
Member
Two group security functions have been introduced for devices participating in GET
VPN communication:
■ A Group Member (GM) is a device that is responsible for securing data traffic. It is
in charge of encrypting and decrypting the data traffic. The GM registers with a Key
Sever to obtain key materials for data encryption and decryption.
■ A Key Server (KS) is responsible for authenticating the GMs, managing the security
policies, creating group keys (GK), and distributing the GK material to the GMs.
On a periodic basis, a KS would send out new key material to refresh the keys prior
to their expiration. The most important function for a KS is the generation of the
encryption keys. A KS generates two types of keys:
■ Key-Encryption-Key (KEK)—KEK is used to secure the control plane messaging
between the KS and the GMs.
■ Traffic Encryption Key (TEK)—This key is used for encrypting the data traffic
between the GMs.
Unlike the traditional IPsec VPN model, which is a bilateral trust model in which a pair
of VPN gateways mutually authenticate each other and set up IPsec sessions between
them, an important aspect of GET VPN is that it does not set up any IPsec tunnel
between the GMs. GET VPN uses a group trust model in which every GM shares the
same security policy and encryption keys obtained from the KS. The security policy
defines what traffic will be encrypted, the encryption algorithm to use, and the encryp-
tion keys to use.
GET VPN uses a tunnel mode called “address preservation,” which copies the original
source and destination IP addresses from the original IP header. Address preservation
allows packets to traverse asymmetric paths in a private MPLS WAN network. In
www.allitebooks.com
VPN Services 147
addition to the IP address, other fields from the original IP header, such as the Type
of Service (ToS), Identity (Id), and Don’t Fragment (DF) fields, are also preserved. The
interesting property of address preservation allows IP multicast packets to be routed
natively over the provider network because GET VPN applies this method to both
multicast and unicast traffic for optimal packet delivery.
High availability is achieved by using a set of KSs running in cooperative mode, where
they jointly accept registration from GMs and distribute GDOI rekeys. The Cooperative
Key Server Protocol enables the KSs to communicate among themselves and exchange
active group policy and encryption keys. If the primary KS becomes unreachable, the
remaining KS continues to distribute group policy and group keys to the GMs. This
ensures that the GET VPN encryption domain is uninterrupted and continues to func-
tion as long as one of the KSs is reachable.
The group trust model for GET VPN is most suitable when the VPN gateways are part
of the same network domain and all VPN gateways are trusted to decrypt any packet
encrypted and forwarded by other GET VPN gateways. It leverages a centralized entity,
the KS, for authentication and distribution of security policy and encryption keys. Most
importantly, Network Address Translation (NAT) is not present along the network paths
between the GMs or KSs. These characteristics are usually found in MPLS networks. If
there are NAT devices present along the network paths between the GMs, then the net-
work is not suitable for deploying GET VPN.
■ Clientless mode provides secure access to web resources only and is useful for
accessing resources that are on corporate web servers.
■ Thin client mode extends the capability of secure web access, with a port-
forwarding Java applet allowing remote access to TCP-based applications such as
POP3, SMTP, IMAP, and SSH that are not web-based protocols.
■ Tunnel mode delivers a lightweight and centrally configured SSL VPN service
that offers extensive application support through a Cisco AnyConnect Secure
Mobility Client. Full tunnel mode provides secure network access to virtually
any applications on the enterprise network. During the establishment of the VPN
with the CSR SSL VPN gateway, the Cisco AnyConnect Secure Mobility Client is
downloaded and installed on the remote user device. When the user logs into the
SSL VPN gateway, it establishes the tunnel connection, and the network access is
determined by the group policy configured on the gateway. After the user closes
148 Chapter 5: CSR 1000V Deployment Scenarios
the connection, the AnyConnect Secure Mobility Client is removed from the cli-
ent device by default; however, there is an option to keep the AnyConnect Secure
Mobility Client installed on the client equipment.
Figure 5-4 illustrates the use of the CSR 1000V as a VPN gateway to extend an enter-
prise network into cloud providers.
Cloud Provider
APP
OS
IPsec/
DMVPN
www.allitebooks.com
Use Cases for the CSR 1000V as a VPN Service Gateway 149
Examples 5-1 and 5-2 show DMVPN hub and spoke configuration examples, respectively.
hostname Hub-1
!
crypto ikev2 keyring VPNKey
peer WANVPN
address 10.10.0.0 255.255.0.0
pre-shared-key cisco123
!
crypto ikev2 profile VPN-PROFILE
match identity remote address 10.10.0.0 255.255.0.0
authentication remote pre-share
authentication local pre-share
keyring local VPNKey
!
crypto ikev2 dpd 60 25 on-demand
!
crypto ipsec security-association idle-time 120
!
crypto ipsec transform-set AES256 esp-aes 256 esp-sha-hmac
mode transport
!
crypto ipsec profile DMVPN
set transform-set AES256
set ikev2-profile VPN-PROFILE
!
interface Tunnel1
description DMVPN
bandwidth 10000
ip address 172.16.1.1 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp authentication VPN123
ip nhrp map multicast dynamic
ip nhrp network-id 101
ip nhrp holdtime 600
ip nhrp redirect
ip tcp adjust-mss 1360
load-interval 30
tunnel source GigabitEthernet1
tunnel mode gre multipoint
tunnel protection ipsec profile DMVPN
!
150 Chapter 5: CSR 1000V Deployment Scenarios
interface GigabitEthernet1
description WAN interface
ip address 10.10.1.2 255.255.255.252
!
interface GigabitEthernet2
description LAN interface
ip address 172.16.1.1 255.255.255.0
!
!
router eigrp WAN
!
address-family ipv4 unicast autonomous-system 100
!
af-interface default
passive-interface
exit-af-interface
!
af-interface Tunnel1
summary-address 172.16.0.0 255.255.0.0
no passive-interface
exit-af-interface
!
topology base
exit-af-topology
network 172.16.0.0 0.0.255.255
exit-address-family
!
end
hostname Spoke-1
!
crypto ikev2 keyring VPNKey
peer WANVPN
address 10.10.0.0 255.255.0.0
pre-shared-key cisco123
!
crypto ikev2 profile VPN-PROFILE
match identity remote address 10.10.0.0 255.255.0.0
authentication remote pre-share
authentication local pre-share
keyring local VPNKey
!
www.allitebooks.com
Use Cases for the CSR 1000V as a VPN Service Gateway 151
!
af-interface Tunnel1
summary-address 172.16.0.0 255.255.0.0
no passive-interface
exit-af-interface
!
topology base
exit-af-topology
network 172.16.0.0 0.0.255.255
exit-address-family
!
end
APP
APP
OS
CSR 1000V CSR 1000V APP
APP
OS
OS OS
APP APP
OS OS
Enterprise Network
www.allitebooks.com
Use Cases for the CSR 1000V as a VPN Service Gateway 153
Secure Inter-Cloud
APP
APP
OS
CSR 1000V Connection CSR 1000V APP
APP
OS
OS OS
APP APP
OS OS
Remote
Access VPN
Figure 5-6 CSR for Remote VPN Access into the Cloud
Example 5-3 Configuration Example: CSR as a Remote Access VPN Server with an
AnyConnect Client
hostname CSR-RA-VPN-Gateway
!
aaa new-model
!
!
radius server FLEXVPN-RADIUS
address ipv4 10.10.1.21 auth-port 1645 acct-port 1646
key 7 01300F175804575D72
154 Chapter 5: CSR 1000V Deployment Scenarios
!
aaa authentication login FLEXVPN-AAA-LIST group radius
aaa authorization network FLEXVPN-AAA-LIST local
!
clock timezone EST -5 0
clock calendar-valid
!
! Generate an RSA Key with key length of 2048 bytes
crypto key generate rsa general-keys label FLEXVPN-KEY modulus 2048
!
ip domain name cisco.com
!
!Enable IOS CA server for local certificate enrollment
crypto pki server CA
no database archive
grant auto
hash sha256
eku server-auth client-auth
no shutdown
!
! Enable HTTP server for certificate enrollment over SCEP
ip http server
!
!Creating a certificate for use of FLEXVPN with AnnyConnect Client
!to support Extended Key Usage (EKU) Requirement.
crypto pki trustpoint FLEXVPN
enrollment url http://10.10.1.1:80
fqdn FLEXVPN-HUB.cisco.com
ip-address none
subject-name CN=FLEXVPN-GWY.CISCO.COM, OU=IT, O=CISCO
revocation-check none
rsakeypair FLEXVPN-KEY 2048 2048
eku request server-auth client-auth
!
crypto pki authenticate FLEXVPN
!
crypto pki trustpoint CA
revocation-check crl
rsakeypair CA
!
!
crypto pki certificate chain FLEXVPN
crypto pki certificate chain CA
certificate ca 01
!
www.allitebooks.com
BGP Route Reflector Use Case for the CSR 155
available on all routers within that AS. There are multiple features and design options
available to reduce this requirement of full mesh, and BGP route reflector is one that is
prevalent in many deployments.
The route reflector function is a part of Cisco IOS but often requires a dedicated hard-
ware router for optimal scale. On the route reflectors, the internal BGP (iBGP) loop-
prevention rule is relaxed for these routers, and they are allowed to re-advertise or
reflect routes from one iBGP speaker to another iBGP speaker. The route reflector is a
control plane function running on the CSR and requires very minimal data-plane func-
tion for sending routing updates to the BGP route-reflector clients. The following rules
for route propagation are applied to a route reflector:
■ Routes received from an iBGP peer (not a part of the route reflector client), locally
generated routes, and routes received from external BGP (eBGP) neighbors are
selected as best routes.
■ The route received from the route reflector client that is selected as the best route is
propagated to all peers.
There are several reasons BGP route reflector design is gaining popularity. BGP is
increasingly being deployed in large networks. The need for BGP has also increased
due to the deployment of MPLS Layer 3 VPNs. The concept of autonomous systems
inheriting the functionality of Layer 3 VPN virtual routing instances between provider
edge routers has increased the usage of BGP deployment within a single AS, and route
reflectors are leveraged to simplify BGP deployments. The use of route reflectors in this
design is mainly to handle the control plane for the BGP routing table. The CSR 1000V
offers a cost-effective way to scale the route reflector functionality and handle the BGP
control plane, removing the need to deploy dedicated route reflector hardware for this
functionality. The actual number of sessions that a route reflector can service depends
on a number of factors, such as the number of routes per session, the use of peer groups,
the CPU power, and the memory resources of the route reflector.
Table 5-1 provides a comparison between the CSR 1000V and Route Processor 2 (RP2)
and shows that the two are similar in terms of handling the route reflector function.
www.allitebooks.com
BGP Route Reflector Use Case for the CSR 157
1000V as a route reflector is not in the data path; you can thereby provide cost-effective
capability to manage the BGP control plane that can scale the number of routes in the
BGP RIB. Table 5-1 should be used as a mere reference point since these numbers can
change based on software release cycle and enhancement in the BGP protocol. Any
use case of the CSR 1000V to replace hardware functionality in the design should be
done with careful tuning of the hypervisor environment, as discussed in Chapter 3,
“Hypervisor Considerations for the CSR.” By default, the IOS XE uses about 50% of the
memory for the IOSd process and the remaining memory for other IOS XE processes
(valid for RPs with 4GB of memory). Because BGP route reflector is a control plane
functionality that runs on the IOSd process only, by default out of 4GB memory avail-
able to the CSR 1000V, only 50% of the memory (2GB) will be used for route reflector
functionality. To optimize this functionality in the CSR 1000V and to enhance the scal-
ability of the route reflector function, the CSR 1000V with a route reflector license uses
the entire 4GB of memory to run the IOSd. The route reflector license available for the
CSR 1000V increases its scalability as a route reflector.
The concept of cluster ID and hierarchy is used in Figure 5-7 to achieve route reflector
redundancy, localization of policies, and route updates within the iBGP domain.
RR 1 RR 2 RR 3 RR 4
lo0 192.168.1.1 lo0 192.168.1.8 lo0 192.168.1.7 lo0 192.168.1.2
East West
Cluster ID 1000 Cluster ID 1001
lo0 192.168.1.3 lo0 192.168.1.4 lo0 192.168.1.9 lo0 192.168.1.10
R1 R2 R3 R4
lo0 192.168.1.5
Figure 5-7 Hierarchical Route Reflector Use Case with the CSR 1000V
158 Chapter 5: CSR 1000V Deployment Scenarios
In this example, AS 65111 is split into two domains: East (cluster ID 1000) and West
(cluster ID 1001). Each subdomain has its own route reflectors for the local route reflec-
tor clients. The CSR 1000V as a route reflector fully utilizes the hardware resource for
the control plane. Very minimum data plane traffic flows between the route reflectors.
The following setup illustrates hierarchical route reflector design:
■ R5-1 and R5-2 are conventional routers and form iBGP relationship with the route
reflectors in the East and West domains.
The configurations for this route reflector design are listed in Examples 5-4 through 5-14.
interface Loopback0
ip address 192.168.1.1 255.255.255.255
router bgp 65111
bgp cluster-id 1000
bgp log-neighbor-changes
neighbor RR-East peer-group
neighbor RR-East remote-as 65111
neighbor RR-East update-source Loopback0
neighbor RR-East route-reflector-client
neighbor IBGP peer-group
neighbor IBGP remote-as 65111
neighbor IBGP update-source Loopback0
neighbor 192.168.1.2 peer-group IBGP
neighbor 192.168.1.3 peer-group RR-East
neighbor 192.168.1.4 peer-group RR-East
neighbor 192.168.1.5 peer-group IBGP
neighbor 192.168.1.6 peer-group IBGP
neighbor 192.168.1.7 peer-group IBGP
neighbor 192.168.1.8 peer-group IBGP
interface Loopback0
ip address 192.168.1.8 255.255.255.255
router bgp 65111
bgp cluster-id 1001
bgp log-neighbor-changes
neighbor RR-West peer-group
neighbor RR-West remote-as 65111
www.allitebooks.com
BGP Route Reflector Use Case for the CSR 159
interface Loopback0
ip address 192.168.1.3 255.255.255.255
router bgp 65111
bgp log-neighbor-changes
network 192.168.40.40 mask 255.255.255.255
neighbor 192.168.1.1 remote-as 65111
neighbor 192.168.1.1 update-source Loopback0
neighbor 192.168.1.1 route-reflector-client
neighbor 192.168.1.2 remote-as 65111
neighbor 192.168.1.2 update-source Loopback0
neighbor 192.168.1.2 route-reflector-client
interface Loopback0
ip address 192.168.1.4 255.255.255.255
router bgp 65111
bgp log-neighbor-changes
neighbor 192.168.1.1 remote-as 65111
neighbor 192.168.1.1 update-source Loopback0
neighbor 192.168.1.1 route-reflector-client
neighbor 192.168.1.2 remote-as 65111
neighbor 192.168.1.2 update-source Loopback0
neighbor 192.168.1.2 route-reflector-client
160 Chapter 5: CSR 1000V Deployment Scenarios
interface Loopback0
ip address 192.168.1.5 255.255.255.255
router bgp 65111
bgp log-neighbor-changes
neighbor IBGP peer-group
neighbor IBGP remote-as 65111
neighbor IBGP update-source Loopback0
neighbor 192.168.1.1 peer-group IBGP
neighbor 192.168.1.2 peer-group IBGP
Interface Loopback0
ip address 192.168.1.6 255.255.255.255
router bgp 65111
bgp log-neighbor-changes
neighbor IBGP peer-group
neighbor IBGP remote-as 65111
neighbor IBGP update-source Loopback0
neighbor 192.168.1.1 peer-group IBGP
neighbor 192.168.1.2 peer-group IBGP
interface Loopback0
ip address 192.168.1.7 255.255.255.255
router bgp 65111
bgp cluster-id 1001
bgp log-neighbor-changes
neighbor RR-West peer-group
neighbor RR-West remote-as 65111
neighbor RR-West update-source Loopback0
neighbor RR-West route-reflector-client
neighbor IBGP peer-group
neighbor IBGP remote-as 65111
neighbor IBGP update-source Loopback0
neighbor 192.168.1.1 peer-group IBGP
neighbor 192.168.1.2 peer-group IBGP
neighbor 192.168.1.5 peer-group IBGP
neighbor 192.168.1.6 peer-group IBGP
neighbor 192.168.1.8 peer-group IBGP
neighbor 192.168.1.9 peer-group RR-West
neighbor 192.168.1.10 peer-group RR-West
www.allitebooks.com
BGP Route Reflector Use Case for the CSR 161
interface Loopback0
ip address 192.168.1.2 255.255.255.255
router bgp 65111
bgp cluster-id 1000
bgp log-neighbor-changes
neighbor RR-East peer-group
neighbor RR-East remote-as 65111
neighbor RR-East update-source Loopback0
neighbor RR-East route-reflector-client
neighbor IBGP peer-group
neighbor IBGP remote-as 65111
neighbor IBGP update-source Loopback0
neighbor 192.168.1.1 peer-group IBGP
neighbor 192.168.1.3 peer-group RR-East
neighbor 192.168.1.4 peer-group RR-East
neighbor 192.168.1.5 peer-group IBGP
neighbor 192.168.1.6 peer-group IBGP
neighbor 192.168.1.7 peer-group IBGP
neighbor 192.168.1.8 peer-group IBGP
interface Loopback0
ip address 192.168.1.9 255.255.255.255
router bgp 65111
bgp log-neighbor-changes
neighbor 192.168.1.7 remote-as 65111
neighbor 192.168.1.7 update-source Loopback0
neighbor 192.168.1.8 remote-as 65111
neighbor 192.168.1.8 update-source Loopback0
interface Loopback0
ip address 192.168.1.10 255.255.255.255
router bgp 65111
bgp log-neighbor-changes
neighbor 192.168.1.7 remote-as 65111
neighbor 192.168.1.7 update-source Loopback0
neighbor 192.168.1.8 remote-as 65111
neighbor 192.168.1.8 update-source Loopback0
162 Chapter 5: CSR 1000V Deployment Scenarios
The highlighted cluster ID in Example 5-14 shows the path of origination of the route in
the BGP AS.
The traditional network components that impact the branch design are routing, firewall,
and encryption. Today, the network is being converged as a platform to launch IT servic-
es. As this trend continues, branch solutions will need to scale these multiple technology
domains, moving a branch from a single node in a high-availability domain to a multiple-
node environment.
In Figure 5-8, notice that the branch router’s role is more than just a normal routing
functionality. The transition in basic routing is seen in the introduction of performance-
based routing (Cisco Performance Routing [PfR]), where the routing concept has evolved
from prefix-based routing to more intelligent path selection. Performance-based routing
allows the router to make an intelligent path decision based on application service level
requirements.
The current WAN branch router supports VPN, voice, WAN optimization engine, a
router-based firewall, integrated switches, and UCS computing blades. All these ele-
ments are added to the router as features in the IOS code and interface cards. The rapid
deployment of these features has driven the need for using services outside the device.
Leveraging computing infrastructure present in the router to spawn network virtual
devices reduces cost factors such as power, space, and cabling.
www.allitebooks.com
Planning for Future Branch Design with the CSR 1000V 163
Router Features
Branch VM
Router VM
Integrated
Switch UCS-E
V
Module
Branch Location
By using NFV, you can spawn virtual devices to scale to new feature requirements. In
Figure 5-9, the branch router has a security gateway (firewall and Sourcefire) that pro-
vides functionalities such as firewall services, Advanced Malware Protection (AMP),
Application Visibility and Control (AVC), and URL filtering. Instead of using firewall
functionality in the router, you have an option of using an NFV element that provides
additional security functionality.
Prevalence
NFV Elements
in Enterprise
WAN Access Types Branch
Deployments
Branch
Router Source VM
vWAAS
Fire
Integrated Switch
V UCS-E
Module
Branch Location
■ One of the internal interfaces is a Layer 2 interface that connects the server blade
directly to the backplane switch for traffic destined to the virtual server. The
other internal interface is a Layer 3 interface that connects the blade to the Cisco
164 Chapter 5: CSR 1000V Deployment Scenarios
ISR route engine for management traffic such as Cisco Integrated Management
Controller (CIMC) and the host operating system configuration.
■ The external Gigabit Ethernet port takes care of external connection use cases. (One
of them is WAN connectivity, or connectivity to the external firewall.)
The UCS E-Series uses a bare metal hypervisor platform that is a joint Cisco and
VMware solution to create a virtualization-ready platform. The ESXi is optimized to
function on the UCS E blade server hardware and is the beginning of a new thought in
designing an NFV solution for the branch. These are some of the benefits of having an
integrated computing platform in the routers:
■ Virtual servers can be provisioned more quickly than their physical counterparts.
The NFV approach to branch virtualization opens up new technology avenues by pro-
viding a platform for customers to deploy virtualized network elements as required.
Coupling this with an easy-to-use end-to-end orchestration and management framework,
enterprises are able to significantly reduce costs and get better return on investment
(ROI) by avoiding expensive truck rolls to enable services at their branches. These are
the key aspects of branch virtualization:
■ Simplicity—You can reduce complexity from services and operations and endorse
more nimble business models. You gain the ability to manage all branches with a
single pane of glass.
www.allitebooks.com
Planning for Future Branch Design with the CSR 1000V 165
Figure 5-10 shows an optimized hypervisor with special drivers for network services
(aligned to the hardware platform) that facilitates performance throughput for NFV ele-
ments. The hardware platform also takes care of special functions like crypto accelera-
tion to provide increased performance.
2
5
&
VM +
CSR ASAv WAAS (
6
7
Optimized Hypervisor 5
$
7
Customized Hardware Platform ,
2
1
An internal virtual switch provides connectivity between VMs or VNFs and is used
to switch traffic between service elements. This internal path offers optimal traffic
forwarding between the VNF elements running inside the platform. Availability of
different types of external I/O interfaces allow the deployment of this device in
different use cases.
One of the challenges in offering new services at a branch is that connecting up the
service chain and deploying new applications takes a great deal of time and effort. It
means racking and stacking network devices and cabling them together in the required
166 Chapter 5: CSR 1000V Deployment Scenarios
sequence. Each new service requires a specialized hardware appliance that has to be indi-
vidually configured. The chances for errors are high, and a problem in one component
could disrupt the entire network.
NFV enables on-demand service and centralized orchestration for integrating the new
service into the existing ones—in essence creating a service chain. For example, a
customer who desires firewall functionality can use a portal to choose among a list of
VNFs (ASAv, vWAAS, and so on), which will then be deployed dynamically on the
platform. Enterprises gain the ability to choose “best of breed” VNFs to implement a
particular service.
When scaling virtual network functions, it is important to consider the performance fac-
tor. The hardware drivers will be available only on limited VNF devices to optimize their
performance. These drivers fit within the hypervisor code, enabling it to interact with the
I/O function. Figure 5-10 calls this optimized hypervisor. The branch virtualization plat-
form has hardware-assisted binding to create and configure the hardware path for the
flow in the data plane between the VNF elements. The future branch design with NFV
will be similar to the high-level concept of ISR design, where a single box takes care of
multiple functions.
Virtual network services need to have simple management and orchestration. These ser-
vices leverage an auto-provisioning agent (for zero-touch deployment) that connects to
the server in the cloud. Zero-touch provisioning plays an important role in a software-
defined WAN (SD-WAN) framework for VNF elements. This offers the capability for a
centralized controller to manage a multitude of VNF elements at scale.
As shown in Figure 5-10, the branch virtualization is not complete without a good
orchestration solution. Some of the key characteristics of the orchestration solution are
to provide role-based access to admins, ability to instantiate new VNF elements, and
configuration management from a centralized management tool. The use of tools like
Cisco Network Service Orchestrator enabled by Tail-f improves the management and
deployment of VNF elements.
The network service orchestrator shown in Figure 5-11 contains a service manager com-
ponent and a device manager component. The service manager maintains information on
different domain components required for the service, and the device manager keeps the
configuration of each device type. The network element drivers provide several options
that can leverage NETCONF, CLI, SNMP, and REST, thereby providing a multitude of
automation coverage.
It is important to understand the new phase of orchestration using NETCONF and
YANG. A detailed use case of NETCONF and YANG with Tail-f is provided in Chapter
8, “CSR 1000V Automation, Orchestration, and Troubleshooting.” NETCONF is a pro-
tocol that is defined by the IETF for installing, modifying, or deleting configuration
from network devices. One can argue that SNMP also does the same functionality on
a network device. The reasons for the new NETCONF protocol were lack of a defined
discovery process, nondefined framework to get the MIB, UDP-based limitation, and
standard security mechanism. NETCONF uses a structured layering concept, as shown in
Figure 5-12.
www.allitebooks.com
Planning for Future Branch Design with the CSR 1000V 167
Administrator
NSO
Service
Service Manager Models
Database
Device
Device Manager Models
Virtual Virtual
Appliance Appliance
Content
Function: Configuration Data
Operations
Function: functions <get>
<Modify> <replace>
RPC
This layering concept provides a structured approach for configuration templates used
in user-defined operations to get or modify the configuration. The delivery method used
here is RPC. An RPC message gets transported using different protocols, such as SSL,
to reach the network device. This layered approach provides the flexibility needed for
multiple use cases.
The YANG data modeling language is used to create user-defined functions for the data
modeling in the NETCONF model. The configuration data in the top layer is modeled
by YANG. The YANG model provides easy representation that can be read by anyone, a
hierarchical configuration model that can be scaled and reused based on user needs, and a
protocol supporting the underlay RPC operations. With the rapid adoption of NETCONF,
the modeling language YANG that provides a simple, structured, and scalable approach for
modeling the data has a promising future in the field of network orchestration.
■ Enterprises will be able to deploy best-of-breed virtual network services from dif-
ferent vendors.
www.allitebooks.com
LISP and CSR 169
DNS
Who is www.cisco.com?
23.211.3.42
The IP routing entries across the Internet are present on all Internet routers. All rout-
ers hold the entire Internet’s routing entries. Summary routes reduce these entries to a
certain extent. However, it is still a very large database. LISP tries to address this prob-
lem by creating two address spaces. The first address is the routing endpoint identifier,
which is the entry the Internet routers use to forward the packet. The second address
is the actual endpoint identifier that sits behind the routing endpoint identifier. Figure
5-14 illustrates the basic LISP concept.
162.151.78.208
Router MAP Resolver
To illustrate using DNS, say you have a router requesting your MAP resolver for an IP
address it does not have a routing entry for. The MAP resolver tells the router to send
the packet to a “routing endpoint,” which can forward the packet to the end host the
router wants to reach. The router then encapsulates the original IP packet to send it to
the routing endpoint the MAP resolver provided.
The fact that a MAP server is instructing a router which routing endpoint connects to
the end device is reminiscent of IP mobility. An end device can keep its IP address intact
and move to a completely different administrative domain. All you need to do is update
the MAP resolver for the new routing locator entry for the endpoint! This mobility use
case is particularly useful when it comes to managing and moving VMs across adminis-
trative domains.
LISP Terminology
Figure 5-15 illustrates IP mobility simplified, using LISP. You can use the scenario illus-
trated in this figure to become more familiar with LISP terminologies. The router that
receives packets from the hosts (in this case, router SJ 162.1.1.5, which receives packets
from host 15.1.1.2) to be sent to remote LISP sites is called the ingress tunnel router
(ITR). The ITR encapsulates packets for remote LISP sites; for non-LISP sites, it just
forwards packets natively. The egress tunnel router (ETR) receives LISP packets, decap-
sulates them, and sends them to the end device. As illustrated in the example in Figure
5-15, the ETR and ITR are usually the same device and can be referred to as xTR. The
host or endpoint (15.1.1.2 in Figure 5-15) is called an endpoint identifier (EID).
170 Chapter 5: CSR 1000V Deployment Scenarios
15.1.1.2
LISP XTR
VM
Router SJ
Map 162.1.1.5
Router A
10.1.1.2 192.1.1.2 Router CH
15.1.1.2 162.1.1.5
172.1.1.8
LISP XTR
Router SJ
Map 162.1.1.5
The ETR’s IP address is called the Routing Locator (RLOC). The RLOC in Figure 5-15 is
162.1.1.5 for administrative domain SJ. The xTR registers this RLOC as the IP address for
reaching 15.1.1.2, which is the EID. This registration is stored in the Map Server (MS). The
MS keeps a table of this EID-to-RLOC mapping, which it receives from the ETR.
The MAP Resolver (MR) is the server that gives you the RLOC-to-EID mapping.
(Customers usually configure the same device as MS and MR.)
An ITR encapsulates traffic for LISP sites and natively forwards traffic for non-LISP
sites. However, in certain cases, an ITR may have to encapsulate packets that are com-
ing in from a non-LISP site but are destined for a LISP site. This is a special functionality
that an ITR needs to support and is referred to as PITR, where P stands for proxy. PITR
is required to connect non-LISP sites to LISP sites.
Similarly, PETR is a functionality supported by an ETR when it accepts LISP-
encapsulated packets from an ITR or PITR for non-LISP sites. The PETR in this case
must decapsulate the LISP packet and forward it natively to the non-LISP sites. The
PETR is useful when dealing with routers that have Unicast Reverse Path Forwarding
(uRPF) configured in strict mode. uRPF drops packets from unknown source addresses.
Consider a situation where a host is sending packets from a LISP site to a non-LISP site.
Because the LISP EIDs are not advertised, the LISP encapsulated packets have a source
address of EIDs that are not known to the routers outside the LISP site domain. Routers
www.allitebooks.com
LISP and CSR 171
with uRPF configured drop these packets. PETR functionality comes in handy in such a
scenario.
The following sections cover the LISP data plane and control plane flows using Figure
5-15.
address space from RFC 1918, they are not routable over the Internet. So the ITR has
the LISP header imposed on it with the routable RLOCs to ensure proper routing over
the WAN.
Three kinds of packets can be used to make this mapping possible: data probe, map
reply, and map request. A data probe is used to send to the MAP server to get an RLOC
for an EID. When an authoritative ETR gets this packet (the data probe packet has the
inner destination address copied to the outer, which means the outer header carries the
EID), it sends a map reply to the ITR. The ITR uses this authoritative ETR’s IP address as
the RLOC IP address and encapsulates the packet and sends it over. An ITR can send a
map request packet to the map server to get the EID-to-RLOC mapping, too.
Figure 5-16 illustrates the LISP packet, including the inner header, the outer header, and
the LISP headers.
Payload
Outer IP Header
DST RLOC
SRC RLOC
www.allitebooks.com
LISP and CSR 173
0 31
Types of Total Length
Version IHL
Service
Identification Flags Fragment Offset
Protocol =
TTL Header Checksum
17
LISP Message
UDP Map-Request: Source port is chosen by the sender; destination port is 4342.
UDP Map-Reply: Source UDP port number is set to 4342. Destination UDP
port number is copied from the source port of either the
Map-Request or the invoking data packet.
Figure 5-18 illustrates the LISP map request message format and the special bits in the
header and their functions:
■ 1 when an ITR wants the destination site to return the map reply rather than the
mapping database system
■ M—This is the map-data-present bit. When set, it indicates that a map reply record
segment is included in the map request.
■ P—This is the probe bit, which indicates that a map request should be treated as a
locator reachability probe. The receiver should respond with a map reply with the
probe bit set, indicating that the map reply is a locator reachability probe reply,
with the nonce copied from the map request.
■ p—This is the PITR bit. It is set to 1 when a PITR sends a map request.
■ s—This is the SMR-invoked bit. It is set to 1 when an xTR is sending a map request
in response to a received SMR-based map request.
174 Chapter 5: CSR 1000V Deployment Scenarios
0 31
Nonce...
...Nonce
....
REC
EID-Prefix
0 31
Type=2 P E S Reserved Record Count
Nonce...
...Nonce
EID-Prefix
Priority Weight M Priority M Weight
LOC
www.allitebooks.com
LISP and CSR 175
RFC 6830 details the packet field descriptions on different LISP packets, such as map
reply.
In the following use cases, the CSR is being used as an MS/MR router, xTR, and PxTR.
IP Mobility
As is evident from the LISP protocol overview in this chapter, LISP encapsulation cre-
ates a dynamic control plane that does not require the user to preconfigure the endpoint
for mobility. The ITR queries the MS/MR and gets the RLOC dynamically for each EID
it needs to reach. This enables the endpoint device to keep its identity and not have a
region-based identity. For example, the endpoint can have an IP address within a data
center in San Jose and keep the same IP address when moved to a New York data center.
All the endpoint needs to do is to update the MS/MR (that is, register itself to the new
MS/MR). This is particularly useful in a virtualized data center environment. These days,
EIDs are virtual machines. With LISP, they can move to a new data center and retain
their IP addresses.
In a cloud environment, to abstract the IP addresses within the cloud, the CSR offers the
capability of using LISP as a feature to accomplish this IP mobility function.
IPv6 Migration
LISP supports both IPv4 and IPv6 addressing. The secret is in the way the LISP header is
designed. In Figure 5-16, it is evident that it does not matter what the inner IP packet looks
like. It can be an IPv6 packet, and LISP will encapsulate it with a UDP header and tunnel it
across to the RLOC, where it is decapsulated and forwarded as a native IPv6 packet. This is
very useful for sites that want to migrate to IPv6 and still have an IPv4 backbone for trans-
port. LISP is therefore a low-cost IPv6 migration option for enterprises.
Network-to-Network Connectivity
LISP enables the reduction of the routing table size by aggregating networks/hosts
behind the RLOC with just the RLOC address. This is ideally suited for network-to-
network connectivity. A large network, when connecting to a smaller network, does not
need to inject its entire routing table into the smaller network. LISP enables smaller net-
works to have just one route to reach the LISP gateway within the larger networks. This
not only prevents smaller networks from being overwhelmed with large routing tables
but also helps the bigger providers encapsulate their TOS value within a UDP packet.
This way, the transit networks are unable to modify the TOS value.
176 Chapter 5: CSR 1000V Deployment Scenarios
In the use case described here, CSR is being used for all the following:
■ Map server (MS)—The MS receives a MAP request from an ITR and finds the cor-
responding ETR RLOC mapping for the EID.
■ Map resolver (MR)—The MR receives map requests from the ITR and uses the
mapping database to find the corresponding ETRs to answer those requests.
■ Route reflector (RR)—The RR is a BGP route reflector that reflects BGP routes
to route reflector clients within an iBGP network, enabling route learning without
requiring full-mesh neighbor adjacency.
MSMR
www.allitebooks.com
LISP and CSR 177
mpls ip
!
interface GigabitEthernet2.100
encapsulation dot1Q 100
vrf forwarding blue
ip address 10.10.11.1 255.255.255.252
!
interface GigabitEthernet2.101
encapsulation dot1Q 101
vrf forwarding Trans_blue
ip address 10.10.1.1 255.255.255.252
interface GigabitEthernet3
no ip address
negotiation auto
!
interface GigabitEthernet3.2
description connected to Partnet ASBR
encapsulation dot1Q 2
vrf forwarding Trans_blue
ip address 10.0.1.2 255.255.0.0
!
router lisp 1
locator-table vrf Trans_blue
eid-table vrf blue instance-id 101
ipv4 route-import map-cache bgp 200 route-map CUST-EID
exit
!
no ipv4 map-cache-persistent
ipv4 proxy-etr
ipv4 proxy-itr 10.100.8.2
ipv4 itr map-resolver 10.100.1.2
exit
!
router ospf 1
!
router bgp 200
bgp log-neighbor-changes
neighbor 101.1.1.1 remote-as 200
neighbor 101.1.1.1 description iBGP
neighbor 101.1.1.1 update-source Loopback0
!
address-family ipv4
network 10.0.0.0 mask 255.255.0.0
network 102.1.1.1 mask 255.255.255.255
neighbor 101.1.1.1 activate
www.allitebooks.com
LISP and CSR 179
Example 5-16 details the MS/MR configuration for the network-to-network connectiv-
ity use case.
www.allitebooks.com
LISP and CSR 181
exit
!
site PCE2
authentication-key cisco123
eid-prefix instance-id 101 14.1.1.0/24
exit
!
ipv4 map-server
ipv4 map-resolver
exit
!
router bgp 300
bgp log-neighbor-changes
redistribute lisp
!
address-family ipv4 vrf Trans_blue
network 10.100.1.2 mask 255.255.255.255
redistribute connected
neighbor 10.10.1.1 remote-as 200
neighbor 10.10.1.1 activate
neighbor 10.10.1.1 send-community both
exit-address-family
!
address-family ipv4 vrf blue
redistribute lisp route-map CUST-EID
neighbor 10.10.11.1 remote-as 200
neighbor 10.10.11.1 activate
neighbor 10.10.11.1 send-community both
neighbor 10.10.11.1 route-map CUST-EID out
neighbor 10.10.11.1 filter-list 1 in
exit-address-family
!
ip route vrf Trans_blue 0.0.0.0 0.0.0.0 10.10.1.1
!
route-map CUST-EID permit 10
set community 991:1177
Example 5-17 details the show commands you need to verify the proper functioning of
LISP.
182 Chapter 5: CSR 1000V Deployment Scenarios
LMGW1#
www.allitebooks.com
Summary 183
Summary
With the transition to virtual data centers and cloud services, enterprises are looking
for a secure and consistent design to extend the network connection to these locations.
With the multitude of security and network segmentation features embedded, the CSR
1000V is in a prime position to tackle these enterprise requirements in the cloud. This
chapter has discussed using the CSR to extend data center networks and to secure inter-
cloud connectivity, for providing remote VPN access into the cloud, and for dynamic
site-to-site VPNs with DMVPN technology.
In addition, this chapter provided a use case for a route reflector with the CSR 1000V.
A route reflector is primarily control plane driven, which makes it a perfect use case for
the CSR 1000V. You have learned about using the CSR 1000V in the NFV framework
and looked at the evolution of the branch architecture that we will soon be seeing. This
chapter’s LISP with the CSR 1000V use case has helped you understand the importance
of IP mobility and segmentation requirements for enterprises in the cloud environment.
This chapter provides an overview of different use case scenarios and the technology
associated with the deployment use cases. The use of the CSR 1000V is not limited to
these use cases, and you can find many more with evolving functionality applicable in
the future of computer networking. Further simplification of management will create
more use cases for deployment of the CSR 1000V.
This page intentionally left blank
www.allitebooks.com
Chapter 6
This chapter describes the landscape you need to understand to fit the CSR 1000V amid
your architecture. It discusses using CSR in a multitenant data center, with OpenStack,
and in a public cloud environment.
Multitenancy refers to logical separation of shared virtual computing, storage, and net-
work resources. A tenant uses a logical separation—which can be a business unit, depart-
ment, or workgroup—on the same physical hardware.
Security-Based
Computing A Zone 2 Isolation
Network B1
Computing B1
Storage A
Storage B1
Most data centers have tenant isolation and zones aligned to security policy to create
further separation within the tenant infrastructure. These zones can be aligned for regu-
latory compliance of lines of business that require data privacy. In Figure 6-1, each zone
in tenant B has separate network, computing, and storage infrastructures. However, in
complex designs, zone resources in a single tenant domain can be shared. For example,
zone 1 and 2 in tenant B could share the same storage domain.
The key design factor in a multitenant data center is the placement of the service block.
This placement affects the characteristic of a virtual zone. Figure 6-2 shows a simple ten-
ant zone with security and load balancing services aligned to it. Tenant A is a completely
isolated, contained environment without services, whereas Tenant B and Tenant C have
load balancing or a combination of load balancing and firewall services aligned with the
contained zone environment.
A service block can be a physical or virtual device in a virtual zone. The segregation of
traffic and isolation of the flow depend on the virtualization technology used in the
zones. To have a simple traffic flow, it is recommended that the service block be placed
near the server (asset). Placing the service block with the firewall close to the server helps
simplify the plan to extend the domain across a single location as is normally done for
disaster recovery. This simplifies the flow across the security infrastructure and prevents
asymmetric flows that would be seen across firewall infrastructure.
Figure 6-3 shows microsegmentation within a single tenant, based on the policy of the
security gateway.
www.allitebooks.com
CSR in a Multitenant Data Center 187
Firewall
Load Balancer Load Balancer
Network
Tenant A
Firewall
Load Balancer
Microsegmentation
The following design methods can be used to achieve segmentation at the service level:
■ A single context service block that aligns the traffic to various logical segments
■ Virtual domains within the service block devices that are aligned to each logical
segment
If the service block is deployed away from the computing and storage assets, network
virtualization technologies such as virtual routing and forwarding (VRF) need to be used.
Network segmentation can be done to carry the traffic with logical isolation to the
computing and storage assets. With the rapid adoption of computing virtualization, the
simplicity and flexibility of service block deployment is due to NFV elements such as
firewalls, routers, or load balancers. The use of the CSR 1000V here allows an architect
188 Chapter 6: CSR Cloud Deployment Scenarios
to provide network elements (with extensive routing features, such as NAT and IPsec)
and security elements (such as zone-based firewalls) in one virtual device instance. One
of the main reasons for data center architects to consider physical service block devices
is data throughput.
Figure 6-4 shows a classic traditional data center deployment with the service block
(security gateway) creating multiple zones extended via VRF (Layer 3 segmentation) and
VLAN (Layer 2 segmentation) technologies to create logical isolation.
WAN I/O
Service Block
Top of Rack
Switch
Non-
Secure
secure
The CSR 1000V can be used to extend the security container to other data centers or
cloud environments. The use of the CSR 1000V helps simplify traffic flow by creating
a symmetric path across the security gateway segments. Features like overlay transport
virtualization (OTV), VxLAN, and MPLS over GRE are used to extend the multitenant
container across the WAN to another data center or cloud environment. This extension
can be for Layer 2 or Layer 3 domains. Figure 6-5 shows connectivity of zones in a mult-
itenant data center environment using the CSR 1000V.
www.allitebooks.com
CSR in a Multitenant Data Center 189
Secure
CSR 1010v
CSR 1000V
Cloud - IaaS
WAN
DC1 DC2
DCI
TOR TOR
Figure 6-5 Connectivity of Zones in a Multitenant Data Center with CSR 1000V
In the example shown in Figure 6-5, the paths for the enterprise traffic to access the
secure zones in the data center (DC1 and DC2) are via the local firewalls. The service
block is localized within the data center. There are multiple options for maintaining
the symmetric traffic flow to take care of the state inspection of the security gateway.
The commonly deployed methods are NAT, selective routing, and firewall clustering.
The different options for the systemic traffic flows between security gateways are not
described in this book. The packet inside the secure tenant can access resources destined
for this tenant. This tenant is extended between two data centers and a virtual data cen-
ter (VDC) provisioned in the cloud. The traffic flow between the secure domains in each
190 Chapter 6: CSR Cloud Deployment Scenarios
of the data centers, through the physical firewall, needs to have policy set for the flow
that increases the operational overhead on provisioning applications in the secure zone
and increases complexity during troubleshooting. By adding NFV components for the
CSR 1000V at these zones, you can isolate the traffic flow between these zones from
the egress security gateway points. The CSR 1000V offers multiple overlay technologies,
including the following:
Apart from overlay, security gateway and VRF-Aware Software Infrastructure (VASI)
functionalities, other technologies can be used to provide a multitude of services in the
security domain. By using the CSR 1000V in the zone design, a data center architect can
build preset logical communication between the zones within the data center owned by
the enterprise and extend it to the cloud to leverage cost-effective provisioning within
the boundaries of the security container defined and controlled by the enterprise fire-
wall policy.
Cloudburst
The concept of leveraging the public cloud for deploying applications needing resources
in spurts is called cloudburst. In this model, the application runs in a private cloud or
data center and, based on demand, provisions assets into a public cloud environment.
This is done to handle sudden spikes in the computing power. This model is also called
the hybrid cloud environment, and an organization pays for extra computing resources
only when they are needed.
Consider the following when planning for a hybrid cloud infrastructure that has cloud-
burst capability:
■ Network connectivity:
■ Unified user access to the asset segment, regardless of where the asset is located
(private or public cloud infrastructure)
■ Secure tenant space extension between the private and public infrastructures (the
policy enforcement of both environments should be handled by a single manage-
ment tool)
www.allitebooks.com
Cloudburst 191
■ Data synchronization:
■ Workload migration:
The user access to the asset, whether hosted in a public or private cloud, needs to be
uniform from a flow perspective, aligned to the application stack (web, middleware, and
database deployment). This needs to be well planned. Two main models can be lever-
aged based on requirements for traffic flow: the direct access model and the redirection
access model.
Application
Asset (Burst)
Public Cloud
User A
Enterprise WAN
WAN Connectivity to
Computing Asset Cloud Provider
Provisioning
Application
Asset
The enterprise connectivity to the public cloud is via the WAN connection from the
service provider to the public cloud environment. In this example, the enterprise traffic
access flows via the enterprise DC (or VDC for the tenant) to the WAN. In this case, the
192 Chapter 6: CSR Cloud Deployment Scenarios
cloudburst model considers only scaling of the computing resources. This model does
not provide a seamless user experience for application access due to hair-pinning of traf-
fic. However, this model does provide scaling of on-demand computing resources to the
public cloud, and it thereby reduces the capital and operational costs for maintaining
hardware resources to take care of the scaled demand. The tenant extension and isola-
tion, if required, are covered by deployment of NFV components.
■ Unified user experience for application access for data in the on-premise/private
data center environment and to the public cloud
Using cloud-based on-demand or static assets for disaster recovery, especially for active/
standby, gives the end user a cost-effective DR strategy that can be offered to multiple
application tiers. Application stack design should be reviewed before cloud adoption.
There might be limits to traditional applications that are not cloud ready to adapt to this
model (see Figure 6-7), and this model applies only for cloud-ready applications that
require locational availability.
Public Cloud
WAN Connectivity to
Cloud Provider
Asset User A
Redirection Enterprise WAN
Computing Asset Block
Provisioning
www.allitebooks.com
Cloudburst 193
Figure 6-7 shows the redirection access model. The asset redirection block is a key ele-
ment that provides redirection of traffic to the private or public cloud. The asset redirec-
tion block is an architectural module that can reside in the private data center environ-
ment or at WAN hub locations at the enterprise data center. The use case of adding the
asset redirection blocks at the enterprise WAN is for networks that are geographically
spread around the world, with the logic abstracted to regional WAN hub locations. If
the logic for redirection is based on DNS and location-based proximity, you can use
hierarchical load balancer design. In this design, a global site selector redirects traffic to
the local load balancer, based on different load balancing algorithms. The local load bal-
ancer can distribute traffic to local assets based on a defined algorithm taking care of the
local load balancing criteria. The load balancer method is beneficial if you use applica-
tion-centric probes to redirect traffic. Another way to achieve the distribution and intel-
ligence from a network perspective is to use LISP. You have already read about the use
case of LISP with the CSR 1000V in Chapter 5, “CSR 1000V Deployment Scenarios.”
You can also use LISP for detecting asset mobility of a host from a private data center to
a public data center. LISP is a more IP-centric approach for redirecting traffic based on
asset location.
There are several use cases for the redirection access model, and two of them are
described here:
■ Allows for a single management portal for all the technical domains
Figure 6-8 shows the important elements of the inter-cloud fabric framework. This sec-
tion describes the functionality of these elements, which contribute to the simplification
of the inter-cloud solution and enable enterprises to cloudburst.
Cisco Prime
Inter-Cloud
Director: ICD
Figure 6-8 shows the components of the inter-cloud fabric. The inter-cloud director is an
OVA file that launches a VM that is used for management and provides a single opera-
tional view of the inter-cloud solution. This file is also used for API mapping between
different cloud providers or private clouds by creating a template. The template is a
user-defined profile for a tenant asset that pulls all APIs into a user-defined grouping.
This grouping has a parent and a child profile. The parent profile is the cloud (public or
www.allitebooks.com
Private Cloud Deployment with CSR in OpenStack 195
private) API index, and the child template is the new inter-cloud template for the tenant
that pulls in all the attributes into one single template.
The inter-cloud exchange (ICX) is an inter-cloud switch (ICS) extension of the tenant trans-
port segment between the private and public clouds on a carrier transport between the
cloud domains that has already been provisioned. The inter-cloud exchange is positioned
on the private cloud that connects to multiple public cloud providers. At the public cloud,
the ICS is provisioned. The data plane for the tenant extension from the private cloud to
the public cloud data flows between the ICX and ICS, across an encrypted path that has
the capability to carry Layer 2 domains. The encrypted domain is maintained for the host
of the driver at the public cloud. The host driver in the Cisco inter-cloud framework at the
remote end encrypts and decrypts the data plane. The communication between the hosts
at the public cloud is contained within the tenant domains maintained at the private cloud.
Each tenant domain has a separate encryption key that isolates the communication in the
public cloud. The communication needs to traverse through the ICS.
The ability to host virtual services for networking and security is possible from the inter-
cloud fabric director, which can instantiate the CSR 1000V or virtual security domain
appliance at the public cloud. The ability to add the CSR 1000V on demand enables a
data center admin to take care of different use cases, where the traffic flow might need
to use the public cloud exit point and where NAT will be necessary for the enterprise IP
address to preserve and use the cloud provider as an exit point. The NAT policy masks
the enterprise IP address to the public cloud’s IP address.
Host A (10.1.1.1) in Figure 6-8 at the private data center needs to communicate with
the 10.1.1.2 host ported in the public cloud. The traffic pattern needs to follow the
same security rule set. The secure tenant zone has a network connection to the cloud
access router; the inter-cloud exchange device builds a Layer 2 encrypted tunnel to
the inter-cloud switch hosted in the public cloud. If the 10.1.1.2 computing asset has
not been provisioned, it is possible to use the inter-cloud director to spawn a new VM
from a single portal. The communication between 10.1.1.1 and 10.1.1.2 is encrypted.
In the future for the same enterprise, if another VM is provisioned outside the secured
container at the public cloud—say 10.1.2.1—the communication between 10.1.1.2 and
10.1.2.1 is not possible because they are in two separate security zones extended to the
public cloud. Both the hosts will have encrypted communication with the ICS. Because
they exist in two separate security domains, the encryption key will be different and will
be considered mutually exclusive.
Introduction to OpenStack
With open source software gaining momentum, customers want software that is free
of cost and open for changes so that they can tailor the software to suit their needs.
OpenStack is a cloud operating system that is a free and open source cloud computing
platform. Written in Python, OpenStack consists of different software components that
were developed as separate projects but come together to work as a single cloud operat-
ing system designed to manage a cloud environment.
Linux is a very powerful operating system, and the KVM module in Linux enables it to
virtualize. As discussed in Chapter 3, “Hypervisor Considerations for the CSR,” Linux
(along with KVM and QEMU) is a fully functional type 1 hypervisor. It enables you to
manage your hardware infrastructure, spawn VMs, and automate. So if you already have
Linux, why do you need OpenStack as an operating system? As mentioned earlier in this
chapter, it’s a cloud operating system, which makes it very distinct from regular operat-
ing systems like Linux.
Linux, along with KVM and QEMU, gives you a fully functional hypervisor and allows
you to create virtual machines. However, Linux can do this only for the single server
blade that it runs on. If you have multiple server blades and you want to pool in the
memory, CPU, storage, and network resources of all of them, you cannot use a single
instance of Linux to do that. Even if you run Linux on all servers, you still do not
have a way to get the resources of all servers together and manage them as one entity.
OpenStack solves this problem. It gives you one holistic view of all the resources at
your disposal. It pools the hardware resources available and manages them for you, as
an operating system would. Since OpenStack has the ability to do this for a bunch of
resources and you can replace hardware at will without impacting the functioning (and
thus scale), it is rightly called a cloud operating system.
Figure 6-9 compares the capabilities of Linux and OpenStack. You can see here that
Linux is used to manage a single server, while OpenStack is used to manage pooled com-
puting, network, storage, and memory resources.
www.allitebooks.com
Private Cloud Deployment with CSR in OpenStack 197
vMEM VM1
vCPU
Linux OS with
vNIC
x86 Server KVM and
QEMU
GUI
RAM CPU I/O
VM2
vStorage
Physical Resources
Linux Being Used as Type 1 Hypervisor
vMEM
x86 Server VM1
vCPU
x86 Server
vNIC
OpenStack
x86 Server GUI
VM2
x86 Server vStorage
OpenStack Components
OpenStack is an operating system that enables you to create, automate, and moni-
tor your cloud. OpenStack is made up of individual components written in Python.
Each component, developed as an individual project, performs a specific role in the
OpenStack framework, as shown in Figures 6-10 and 6-11.
Horizon (Dashboard)
Keystone
Heat (Orchestration) (Authorization
and
Authentication)
Glance Swift Cinder Trove Sahara
Nova Neutron Ceilometer
(Image (Object (Block (Database (Analytics
(Computing) (Network) (Metering)
Store) Store) Store) as a Service) as a Service)
• CLI
• Mgmt Public Network
Services
• GUI Tools
HTTP(S) OpenStack OpenStack OpenStack
OpenStack OpenStack Compute API, Block Storage Networking
Object API Image API EC2 API API API
OpenStack
Object API
Horizon
OpenStack OpenStack Block OpenStack
Image API Storage API Networking API
AMQP
OpenStack OpenStack OpenStack OpenStack OpenStack
Object Store Image Service Compute Block Storage Networking
OpenStack Identity API OpenStack Identity API OpenStack Identity API OpenStack Identity API
Keystone
Figure 6-11 OpenStack Architecture; Each Project Is Accessible via a Project API, and
Users Can Access OpenStack via the Dashboard (Horizon) or Through Project-Specific APIs
Nova (Computing)
Nova sits at the heart of the OpenStack framework. It is the computing engine that is
responsible for creating and maintaining virtual machines. It is a complex and distributed
component of OpenStack. Multiple processes constitute the Nova computing project.
Here is a brief description of some of the processes in the Nova project:
■ nova-api—This process accepts and responds to the end user’s computing API
calls. It is responsible for orchestrating events like running a virtual machine. It is
also responsible for enforcing policy.
www.allitebooks.com
Private Cloud Deployment with CSR in OpenStack 199
Nova-api
VM1 VM1 VM1 VM1
Nova
Computing
Control Node
KVM
ESXi
vSphere Hypervisor
Nova
Control
Computing
■ nova-network—This process picks up a task from the queue and executes the task
in a way similar to nova-compute. It targets the networking piece of OpenStack,
like provisioning Layer 2 and Layer 3 circuits, setting up IP addresses, and handling
NAT and other network functionalities. nova-network functionality moved on into
a separate project, called Neutron. Neutron is a full software-defined networking
stack. However, nova-network is still used for simple use cases where the network-
ing requirements are limited.
■ The queue—The message queue connects daemon processes and serves as a cen-
tral repository for passing messages between processes. It is implemented using
Advanced Message Queuing Protocol (AMQP), a standard message queuing proto-
col for passing messages between applications.
Note There are multiple other processes in the Nova project. Covering all of them is
beyond the scope of this book. OpenStack projects are well documented online at
http://docs.openstack.org.
requests and provides users with a catalog of available offerings, enforces policy, and
administers token and identity services.
Keystone issues a token, which essentially defines the scope of access in OpenStack.
The token issued is based on the user’s credentials. Figure 6-13 illustrates how keystone
works at a very high level. These are the steps shown in the figure:
Step 1. The user sends an unscoped token (that is, a token that it is not tied to any
particular project in OpenStack) to determine which OpenStack projects/ten-
ants the user has access to. This token is strictly used to query Keystone to
determine which projects the user has access to. It is important to note that
only scoped tokens (tokens that have user credentials and the tenant details
where the service is requested) must be used with non-Keystone projects,
such as Nova. Unscoped tokens are only used to query Keystone. The user
provides the credentials (username and password).
www.allitebooks.com
Private Cloud Deployment with CSR in OpenStack 201
Step 3. The user sends the unscoped token to determine which tenants/projects he
has access to. This step typically uses a REST API get with this unscoped
token and sends it to Keystone.
Step 4. Keystone responds with a list of tenants to which the user has access.
Step 5. The user now has access to a list of tenants he can access and must decide
which tenant he wants to use. To work with any tenant, the user must obtain
a scoped token. A scoped token is always for a particular tenant and contains
metadata for the tenant. To obtain a scoped token, the user now does a REST
API post with username and password, such as in Step 1. The difference here
is the tenant/project name. In this post, the user specifies his username and
password that align with the tenant and project name for which the token is
requested.
Step 6. As a response to the post, Keystone sends a scoped token with the metadata
associated with the tenant/project. The user now has a service catalog that
contains the URLs to the tenant/project.
Step 7. The user can now invoke the service by using the scoped token and the target
URL of the service he wants to invoke. (This is the case with UUID-based
tokens only.) The tenant receives the service request with the token. It uses
the metadata of the token to verify the user access rights for the service. The
policy.json file for each tenant determines the role-based access.
Note It’s important that you understand how to relate multitenancy concepts with
OpenStack:
■ Tenants are represented as projects, and a project is the base unit of “ownership” in
OpenStack. A tenant can be defined as a user assigned a fixed set of resources within a
container. This is an important concept in multitenancy for network, storage, and com-
puting domains.
■ A group is a collection of users that are part of a domain.
Glance uses a client/server model. The applications that use Glance are clients, and the
Glance project behaves as a server. It uses REST API as the communication mechanism.
Following are the major components of Glance:
■ DAL (Database Abstraction Layer)—This is an API that sits between Glance com-
ponents and the Glance database.
■ Glance store—This is the middleware between Glance and various data stores that
are supported by Glance.
Figure 6-14 illustrates the architecture of the image service project, Glance, in
OpenStack.
Client
REST API
OpenStack
Image Service Client
(Glance) Glance
Keystone Image API Controller/Auth-Notifier, DB
Policy DB/Registry/
Abstraction
AuthIntf
Image API
API
Keystone Glance
Glance
Store
API Glance Store Drivers
Supported Storages
Neutron
Before Neutron became a separate project, it was Nova’s networking service,
nova-network. You can still use nova-network for basic networking operations. It can
do base Layer 2 network provisioning, IP address management for tenants, DHCP, DNS,
and enable implementation of firewalls and NAT using IPtables.
www.allitebooks.com
Private Cloud Deployment with CSR in OpenStack 203
Neutron was created as a separate project from Nova to address the limitations of nova-
network and bolster the networking capability of OpenStack. nova-network had mul-
tiple limitations; for example, it did not have APIs for consuming networking services,
and it had a VLAN-based network model, which allows only 4096 VLANs and only a
very limited set of features. Perhaps the most striking drawback was that the architec-
ture did not allow plugins to render their functionality to OpenStack. Neutron, which
was promoted to a core project at the Folsom Summit in April 2012, addresses the fol-
lowing functionalities:
REST API
DHCP
Neutron Message Agent
Server Queue
L3 Services
Cisco (Nexus,
More Vendor
Node
Firewall
Plugins
N1Kv)
OVS
VPN
ML2
L2 Agent
OVS on
Compute
Node
Type Mechanism
OpenSwan
Futures
HA Proxy
IPTables
Drivers Drivers
More Vendor
OpenDaylight
Cisco Nexus
VXLAN
Drivers
VLAN
APIC
GRE
OVS
Southbound Interfaces
■ Tenant APIs—Tenants can use API calls to create as many networks as they want.
Whenever you bring up a VM, you just need to define the networks it should con-
nect to.
Block storage is one of the most fundamental requirements for a virtualized infrastruc-
ture. This is because the virtual infrastructure is maintained as files. Each virtual machine
is essentially a file that needs storage. These VM files and the data associated with them
need to have persistent storage so that the VM can be powered down and brought up to
life whenever required.
Cinder is the block storage mechanism within OpenStack. It stores and provides access
to block storage for OpenStack tenants. To OpenStack tenants, this storage appears as a
block device that can be accessed using mechanisms like iSCSI, Fibre Channel, and NFS
for back-end connectivity.
Figure 6-16 illustrates the architecture of Cinder, which has the following components:
Cinder Client
SQL DB REST
Cinder API
AMQP
AMQP
AMQP
Scheduler Cinder Volume Backup
Cinder
■ Cinder API—This piece of code, as with all other software in OpenStack, talks
AMQP with modules inside the project. With modules outside the project, it talks
REST API. Cinder API takes in REST API requests from the north and routes them
to the appropriate Cinder modules talking AMQP.
■ Cinder scheduler—This component schedules the requests for the volume service.
■ Cinder volume—This is the block storage manager that manages back-end storage
devices.
www.allitebooks.com
Private Cloud Deployment with CSR in OpenStack 205
■ Heat (orchestration)—This OpenStack project, like Horizon, interacts with all com-
ponents of OpenStack. It is used to create application stacks from multiple resourc-
es (like servers, floating IPs, security groups, users, etc.). It is an engine that is used
to launch multiple cloud applications based on templates. A Heat template describes
the skeleton for an application in a text file that can be edited.
■ Swift (object storage)—This project provides a mechanism for storing and retriev-
ing unstructured data. Swift makes sure that data is replicated across a cluster of
servers and that the integrity is maintained. It provides distributed object storage.
Storage clusters scale horizontally, and if one of the servers fails, OpenStack repli-
cates the data from other active nodes to new cluster locations. Swift stores objects
on object servers. Swift is implemented using a ring concept, where a ring is a com-
ponent that contributes to the Swift service.
■ Gauge meter—This meters discrete (for example, images) and fluctuating (for
example, disk I/O) values.
This example creates an isolated database environment with computing and storage
resources.
Note OpenStack can manage different types of hypervisors and can also manage con-
tainers. What is container-based virtualization?
Figure 6-17 illustrates how the Neutron services can be hosted by the CSR inside
OpenStack.
www.allitebooks.com
Private Cloud Deployment with CSR in OpenStack 207
Management Network
Nova
Neutron
VM
Routing A
Service Plugin
VM
External Network VLAN=9
B
Cisco Config Agent
CSR
VM
OpenStack is essentially designed to talk to Neutron, and the CSR is a software router
that needs to be created and managed, just like any virtual machine. To create a Neutron
router hosted within a CSR VM, you need to install the following:
■ You need to install a plugin in Neutron to talk to the CSR virtual machine and also
to Nova. This spins up a CSR VM when instructed to do so by Nova.
■ You need to install a configuration agent that talks CLI to the CSR and talks RPC
(implemented using RabbitMQ) to the Neutron module. The routing service plugin
in Neutron will discover the configuration agent (over the management network).
The CSR virtual machines and the configuration agent communicate over the man-
agement network, which may or may not be the same as the OpenStack manage-
ment network. In Figure 6-17, both management networks are the same.
Step 1. To establish connectivity to Keystone, create the admin tenant that houses
the CSR:
$ keystone tenant-create --name L3AdminTenant --description “Special
Admin Tenant”
208 Chapter 6: CSR Cloud Deployment Scenarios
Change roles to admin for the special admin tenant created above:
$ keystone user-role-add --user-id <UUID_Neutron_Service> --role-id
<UUID_admin_role> --tenant-id <UUID_Special_Admin_Tenant>
Use the following Keystone commands to get the Neutron service UUID and
admin role UUID:
UUID_Neutron_Service == keystone user-get neutron
UUID_admin_role == keystone role-get admin
Use the following Keystone commands to get the neutron service UUID and
admin role UUID:
UUID_service_tenant == keystone tenant-get service
Step 3. To get Neutron to talk to the CSR routing service plugin, modify the
Neutron configuration file and make sure you point the service_plugin
variable to the CSR:
service_plugins=networking_cisco.plugins.cisco.service_plugins.cisco_
router_plugin.CiscoRouterPlugin
Create Nexus 1000V (N1kv) network profiles. Execute the following com-
mands as user neutron in the special admin tenant:
$ neutron cisco-network-profile-create --tenant-id <UUID_Special_
Admin_Tenant> --physical_network osn_phy_mgmt_network --segment_range
<vlanIdMgmt>-<vlanIdMgmt> osn_mgmt_np vlan
www.allitebooks.com
Private Cloud Deployment with CSR in OpenStack 209
Now you need to log in to the server running the config agent and configure
ipAddressCfgAgent on the interface and macAddress of the port you just
created, using the previously shown CLI commands.
Step 5. To verify that the installation was successful, create external and internal net-
works:
■ Create a Neutron router and attach the internal and external networks to
it. When you do this, the CSR you set up in Steps 1–4 is created inside
the special admin tenant and should work as a Neutron router.
■ Attach interfaces, remove them, and delete them just as you would for a
Neutron router:
Note If you attach the CSR as a tenant router to the provider/public network, you
do not need a Neutron router to NAT your traffic outside OpenStack. It is a practice
to leverage a plugin when the CSR attaches directly to the provider network to replace
a Neutron router. You will learn more about the CSR plugin in Chapter 7, “CSR in the
SDN Framework.”
In Figure 6-18, a CSR is spawned as a tenant VM in each of the internal networks: 1 and
2. In each network, the CSR is being used as a firewall or VPN head-end. VM A and VM
B use CSR Network 1 and CSR Network 2, respectively. This model gives you tremen-
dous control over the firewall and VPN policies to impose on your networks.
210 Chapter 6: CSR Cloud Deployment Scenarios
10.1.1.1
FW
CSR
VPN
Neutron
11.1.1.1
10.1.1.2
FW
CSR
VPN
VM
11.1.1.2
Internal nNetwork 2
Internal Network 1
External Network
10.1.1.3
VM
11.1.1.3
VM
VM
Vm
Figure 6-19 illustrates how to use the CSR to segregate a single network into multiple
tenant networks.
Private Network A
10.1.1.1
40.1.1.1
VM
CSR
VM
Neutron
CSR VM
External Network 1
Private Network B
External Network
VM
Figure 6-19 CSR Segregating One Network into Multiple Tenant Networks
www.allitebooks.com
CSR 1000V in a Public Cloud 211
To install the CSR as a tenant VM, bring up CSR just like any other tenant VM within
OpenStack. Then just list the flavor and image list, choose one, and use nova boot to
create it. Nova uses Neutron to get an IP address for the CSR you spawned. You can
then configure the CSR using the console access and set up the required service for the
other tenant VMs:
$ nova flavor-list
$ nova image-list
$ nova boot --flavor flavorType --key_name keypairName --image ID
newCSRInstanceName
The Cisco CSR 1000V can be deployed on Amazon Web Services (AWS) for public
cloud solutions. Companies typically connect to their applications through a single VPN
tunnel between their data center and AWS. Deploying the Cisco CSR in AWS opens
up the potential for direct VPN access to AWS-hosted applications from campus and
branch-office locations without back hauling through an existing data center. This design
reduces latency, eliminates the need for expensive private WAN services, avoids the per-
VPN-tunnel costs that Amazon charges, allows AWS hosted applications to participate
in existing route-based VPN topologies, and lets enterprises own their network elements
(which they use to enforce enterprise policies) within a public cloud infrastructure. This
ownership helps the enterprise in different inter-cloud or cloudburst use cases described
earlier in this chapter.
■ Amazon Virtual Private Cloud (VPC)—Amazon VPC is a cloud service that pro-
vides users a virtual private cloud by offering an isolated private section that exists
212 Chapter 6: CSR Cloud Deployment Scenarios
within Amazon’s public cloud. Amazon EC2 instances can run within a VPC, and
enterprise customers can gain access to the EC2 instances over an IPsec VPN con-
nection. Amazon VPC gives users the option of selecting which AWS resources are
public facing and which are private, thus offering more granular control over secu-
rity for instances running in AWS.
■ EC2-Classic—This was introduced in the original release of Amazon EC2. All the
Amazon Machine Instances (AMI) run in a single, flat network shared with other
customers. All nodes running in EC2-Classic are on a shared network and are
addressable to one another. There is no differentiation between public and private
interfaces for AMI running in the EC2-Classic, as each machine instance has only
one network interface. Each instance automatically receives a public IP address and
can access the Internet directly.
To understand the routing capabilities in AWS clouds, you must understand AWS VPC,
the networking layers for Amazon EC2. These are the components of VPC:
www.allitebooks.com
CSR 1000V in a Public Cloud 213
Internet
AWS
Internet
Gateway
APP
OS
APP
OS Router
EC2-Classic
Instances VPC
APP
APP OS
OS
EC2-VPC
Instances
■ Routing table—Each subnet is associated with a single routing table that controls
that subnet’s routing. The routing table contains a set of rules that define how traffic
is forwarded within the VPC. Creating a VPC generates the main routing table auto-
matically, enabling machine instances in the VPC to communicate with one another.
A routing table can contain multiple subnets; however, only one routing table can
be associated with each subnet.
■ Security group—The AWS security group acts as a virtual firewall to protect virtual
machine instances with inbound and outbound rules. Users can configure ports and
protocols to open to traffic and from which source and to which destination. The
security group operates at the instance level as the first layer of defense. It can be
combined with network ACL, which operates at the subnet level, as a second layer
of defense.
ating the VPC with an Internet gateway. The Internet gateway allows the instances
to be reachable via the elastic IP address from the Internet.
AWS
Region Region
Note While a VPC can span multiple availability zones within the AWS cloud, a subnet
can only reside in a single zone and cannot span availability zones.
Amazon has designed its data center networks quite differently from the conventional
enterprise approach. For one thing, a VLAN is not used as a Layer 2 segmentation
www.allitebooks.com
CSR 1000V in a Public Cloud 215
mechanism in Amazon cloud to avoid the 4096 VLAN scaling limitations associated
with the technology. Instead, Amazon uses Layer 3 routing technology throughout
its infrastructure, which means all traffic is forwarded based on IP addresses instead
of Layer 2 MAC addresses. One way of looking at Amazon networking is that it is a
completely flat network without hierarchy, which can be contradictory to traditional
network design. The benefit of a flat network is that the configuration and setup process
for virtual machine instances is greatly simplified. The entire process is automated
to a greater extent than in the traditional hosting environment. In addition, because
customers are not segmented into assigned VLANs, this easily allows for the growth and
shrinking of computing instances based on seasonality or customer demand. Customers
can easily request additional instances through the Amazon web portal and launch the
new instances from the overall IP addresses in a region within minutes. The IP addresses
for the new instances may be quite different from the others assigned previously,
but because the network is flat and all traffic is forwarded based on IP addresses, the
incongruent address assignment is not a problem.
Within AWS, cloud address assignment is not persistent to a customer. AWS dynami-
cally assigns an IP address to an instance’s virtual network interface (vNIC). The vNIC
can have one or more private IP addresses and one public IP address, which can be auto-
matically assigned. Having two IP addresses means that the instance can send and receive
data traffic from within the AWS cloud network using the private IP address while
accessible from the Internet with the public IP address.
Amazon VPC is confined to a single region and cannot span regions. However, within
the region, a VPC can span multiple availability zones, and by launching instances into
separate availability zones, you can protect applications from the failure of a single avail-
ability zone. Within an availability zone, you can create one or more subnets. Each sub-
net must reside entirely within one availability zone and cannot span zones. All subnets
can route to one another by default.
When a VPC is created, it is automatically associated with a main route table. Initially
there is only one entry in the main route table, which is the CIDR block address associ-
ated with the VPC. When new subnets are added in the VPC, they are implicitly associ-
ated with the main route table by default.
With the fundamental networking of AWS in hand, next we look into deploying the
CSR in the AWS cloud network.
216 Chapter 6: CSR Cloud Deployment Scenarios
Internet
Internet Gateway
AWS
VPC
Main Route Table
Destination Target
0.0.0.0/0 igw-id
Router
Subnet 2 Subnet 4
Region
■ Hourly-Billed AMIs—Users are billed for the hourly usage of the CSR 1000V
instance. The hourly usage fee is in addition to the basic instance-type usage fee
billed by AWS. AWS pays Cisco for CSR usage fees it collects. This model does not
require any license to be purchased or installed.
■ Bring Your Own License AMI—Users purchase software licenses directly from
Cisco and launch the Bring Your Own License AMI from the AWS Marketplace.
Users pay only for the basic instance-type fees. Once the CSR AMI is launched
and deployed, users follow the standard Cisco software activation process to install
the license.
From a design standpoint, CSR is logically placed between the VPC router and the
machine instances running within the VPC, as shown in Figure 6-23.
However, the fact that the CSR is running as an AMI within a VPC and because of the
networking property of AWS VPC, in reality the CSR sits in parallel to the machine
instances, as illustrated in Figure 6-24. Therefore, a subnet route pointing to the CSR
must be added to ensure traffic flow through CSR. Alternatively, the administrator must
change the default gateway for each of the instances within the VPC to point to the CSR.
www.allitebooks.com
CSR 1000V in a Public Cloud 217
AWS
VPC
Internet APP
Gateway OS
VPC CSR
Router
Internet
VPC
APP
OS
VPC CSR
Router
AWS
VPC
Internet
Gateway
CSR
Internet APP
OS
VPC
Router
APP
OS
Another property of an AWS VPC cloud network is that link-local multicast, and broad-
cast traffic are not supported. This restricts the use of First Hop Redundancy Protocol
(FSRP), such as HSRP/VRRP, commonly leveraged by network engineers for high avail-
ability and the use of interior gateway protocols, such as OSPF and EIGRP, for route
propagation. The workaround for some of these restrictions is the use of GRE encapsula-
tion on the CSR. GRE encapsulation will transport the unsupported protocols. The GRE
tunnel allows the CSR routers to exchange Bidirectional Forwarding Detection (BFD)
protocol for rapid peer failure detection. When BFD detects a peer-down event, it trig-
gers an Embedded Event Manager (EEM) script to modify the VPC route table, through
the use of the AWS EC2 API, to redirect traffic around the failure. This use case can be
expanded further to create tenants within a single VPC domain, as discussed earlier in
this chapter, in the section “CSR 1000V as a Tenant Router.”
Step 2. Select the Cisco CSR AMI for your deployment. You might have to log in
or create a new account if you don’t already have one. Choose one of the
options for the AMI license:
■ Hourly-Billed AMIs:
www.allitebooks.com
CSR 1000V in a Public Cloud 219
this writing, this is the only version that includes SR-IOV capability.
In future releases of the CSR 1000V, the SR-IOV feature will be inte-
grated into all versions.
■ CSR Direct Connect 1G/CSR Direction Connect Multi-Gig—These
AMIs are used for securing an enterprise-grade hybrid workload
design with AWS Direct Connect circuits.
■ Cisco Cloud Services Router: CSR 1000V, Bring Your Own License
(BYOL)
Step 3. From the AWS EC2 launch page, select either the 1-Click Launch or the
Manual Launch option to start the CSR AMI. Use the Manual Launch option
for more granular control of the setup options. The following example uses
the 1-Click Launch option to show the step-by-step procedure.
Note With 1-Click Launch, the user first creates the VPC, as well as security key pairs
for remote SSH. Refer to the AWS documentation for how to set up VPC and key pairs.
Also note that 1-Click Launch is currently not supported with BYOL AMIs.
Under VPC Settings, click the Set Up button (see Figure 6-26), which will
take you to the VPC Settings page.
Step 4. On the VPC Settings screen, select the VPC for the CSR to use. Next, attach
a public subnet and a private subnet to the CSR network interfaces. The secu-
rity group for the public subnet is automatically created for the VPC. This
security group is predefined, and users can change the security group settings
after the AMI has launched. Click Done (see Figure 6-27) to return to the
launch page.
Note By default, the security group is set up to allow SSH traffic. For IPsec
deployment, IKE and ESP protocols have to be explicitly added to the security group.
Step 5. From the Launch page, go to the bottom and enter the Key Pair information.
Choose from an existing key pair or select Create a New Key Pair if one does
not exist and follow the directions. Next click the Launch with 1-Click but-
ton to start the CSR AMI instance (see Figure 6-28). It takes 5 to 10 minutes
to deploy the instance and get the CSR fully up and running.
www.allitebooks.com
CSR 1000V in a Public Cloud 221
Figure 6-28 AWS Key Pair Selection and Launch with 1-Click
Step 6. To reach the CSR AMI through SSH for console access, enter the following
procedures from a Unix shell command:
ssh –i <key-pair-pem-file-name> ec2-user@<public-ip-address or DNS-name>
Note For first-time login, use username ec2-user to access the instance.
These are the key considerations you need to review with AWS or any other public
cloud provider:
■ Service solutions—The type of service model that is offered by the cloud provider:
You must consider all these factors as you review any public cloud solution, whether
from AWS, Google, Cisco, or Azure.
Summary
The CSR 1000V offers a lot of flexibility and capabilities such as overlay technologies,
security gateway functionalities, and VRF-aware software infrastructure to provide a
multitude of services inside of a multitenant data center. A data center architecture can
leverage the CSR 1000V to effectively create security zone designs inside the data center
and build preset logical communication between the zones within the data center owned
by the enterprise.
Now that you have read this chapter, you should have a clear understanding of using
the CSR 1000V in a multitenant data center, with OpenStack, and in a public cloud envi-
ronment. There are other use cases besides those covered in this chapter, and you can
explore them as needed.
www.allitebooks.com
Chapter 7
SDN creates abstract layers that cover three main components: controller, application,
and data plane. This abstraction enables the creation of a customized network tailored
to deliver the needs of the applications that run on it. It also provides an elastic data
plane to users (an elastic data plane enables the addition of physical or virtual hardware
resources on demand without the modification or disruption of the control plane). SDN
can be categorized into three main functions as shown in Figure 7-1:
Application-Centric
Programmable Fabric Overlay
Infrastructure
SDN has multiple flavors, and its implementation varies based on vendor. In the Cisco
product armada, multiple products are used to achieve abstraction of different layers,
programmability, and orchestration via a central controller. For example, the Cisco
Application Centric Infrastructure (ACI) is one such product solution. It is important to
know what solutions are available and to understand their high-level components. Figure
7-2 shows out-of-the-box SDN for a data center that is vendor centric.
www.allitebooks.com
Deploying OpenStack 225
Another way to achieve this abstraction is with a combination of NFV elements and
cloud management software, such as OpenStack. You have read in previous chapters
about the history and various components of OpenStack.
OpenStack leverages NFV components in its framework to create an environment of
vendor-agnostic cloud operations. OpenStack is focused on accelerating adoption of
SDN by providing a robust SDN platform on which the industry can build and innovate.
A centralized controller provides management of physical and virtual networks. The
open source format makes it flexible for vendors like Cisco, IBM, and Red Hat to con-
tribute to the project goals and platform architecture, as well as to its roadmap.
OpenStack brings together virtual networking approaches with Neutron. Please see
Chapter 6, “CSR Cloud Deployment Scenarios,” for OpenStack and Neutron functional-
ity, which provides virtual networking components to the cloud operating system.
This chapter explores the following:
Deploying OpenStack
This section covers deploying OpenStack in a proof-of-concept environment using
Packstack installation. Packstack is a command-line tool that leverages Puppet modules
for the rapid deployment of OpenStack on host machines. Packstack can be deployed
interactively, using command-line prompts, or non-interactively, using defaults or
a configuration file. This section illustrates deployment examples with Red Hat
Enterprise Linux.
Step 1. Perform a minimum install of Red Hat Enterprise Linux on qualified server
hardware.
Step 2. Change the interface names as needed for consistent and predictable net-
work device naming for the network interfaces. These changes should make
locating and differentiating the interfaces easier. In this case, you want to
convert interface names like enp9s0 to names like eth0 by editing the /etc/
default/grub file. Here is the first edit:
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
226 Chapter 7: CSR in the SDN Framework
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rd.lvm.lv=rhel/swap crashkernel=auto
rd.lvm.lv=rhel/root rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
Step 5. Change the names of the network scripts to ethX, where X is 0, 1, 2, and
so on:
mv /etc/sysconfig/network-scripts/ifcfg-enp2s0 /etc/sysconfig/
network-scripts/ifcfg-eth0
■ ifcfg-eth0 config:
TYPE=Ethernet
DEVICE=eth0
NAME=eth0
BOOTPROTO=none
ONBOOT=yes
USERCTL=no
■ ifcfg-eth1config:
TYPE=Ethernet
DEVICE=eth1
NAME=eth1
BOOTPROTO=none
ONBOOT=yes
USERCTL=no
Note Interfaces on the “data network” of OpenStack are not mandated to have an IP
address defined when tenant networks are VLAN based.
www.allitebooks.com
Deploying OpenStack 227
Step 8. Configure the IP addresses on interfaces that need to be used for OpenStack
management and the external network (for example, eth3 with IP addresses
in the range 192.168.1.22/24):
TYPE=Ethernet
DEVICE=eth3
NAME=eth3
BOOTPROTO=none
ONBOOT=yes
USERCTL=no
IPADDR=192.168.1.22
NETMASK=255.255.255.0
GATEWAY=192.168.1.1
DNS1=8.8.8.8
DNS2=8.8.4.4
Step 9. Once the interfaces have been configured, shut down the server and restart
the network service so the configuration can take effect. To do this, as root,
issue the following command:
/etc/init.d/network restart
Step 10. OpenStack networking currently does not work on systems that have the
NetworkManager service enabled. Use this configuration command, as root
user, to disable the NetworkManager server:
systemctl stop NetworkManager
systemctl disable NetworkManager
Step 12. Each host machine deployed in OpenStack must register to Red Hat
Subscription Management to receive updates from Red Hat Network. To
allow Packstack to install OpenStack on each host, activate the correct sub-
scriptions and repos as follows:
https://access.redhat.com/products/red-hat-enterprise-linux-openstack-
platform/get-started
Subscribe the system via Red Hat Subscription Management and confirm that
an OpenStack subscription is attached:
subscription-manager register
#Username : <userID>
#Password: <password>
Step 13. Clear the repositories that were initially set up and enable the ones needed
for OpenStack:
subscription-manager repos --disable=*
subscription-manager repos --enable=rhel-7-server-rpms
subscription-manager repos --enable=rhel-7-server-rh-common-rpms
Step 14. Install the necessary yum packages, adjust the repository priority, and update:
yum repolist
yum install -y yum-plugin-priorities yum-utils
Figure 7-3 shows an example output from the yum install of OpenStack and
the software collections associated with the operating system.
www.allitebooks.com
Deploying OpenStack 229
Figure 7-4 Sample Output from Enabling Packages from the Red Hat Repository to yum
Figure 7-5 Sample Output for Enabling the Optional OpenStack repository.
Step 16. Execute one of the following two options—but not both:
Figure 7-6 shows an output example of the Packstack installation through the
yum utility.
Step 20. Edit the Packstack answer file as follows (see Appendix A, “Sample Answer
File for Packstack”):
■ Update the Neutron type driver info to be VLAN (instead of the default
VxLAN).
Example 7-1 shows an abbreviated sample answer file for Packstack; see
Appendix A for the entire output.
www.allitebooks.com
Deploying OpenStack 231
CONFIG_NOVA_COMPUTE_PRIVIF=eth1.10
# The name of the Open vSwitch bridge (or empty for linuxbridge) for
# the OpenStack Networking L3 agent to use for external traffic.
# Specify 'provider' if you intend to use a provider network to handle
# external traffic.
CONFIG_NEUTRON_L3_EXT_BRIDGE=br-ex
# Specify 'y' to install OpenStack Networking's Load-Balancing-
# as-a-Service (LBaaS)
CONFIG_LBAAS_INSTALL=y
# Specify 'y' to install OpenStack Networking's L3 Metering agent
CONFIG_NEUTRON_METERING_AGENT_INSTALL=y
# Specify 'y' to configure OpenStack Networking's Firewall-
# as-a-Service (FWaaS)
CONFIG_NEUTRON_FWAAS=y
# list of network-type driver entry points to be
# loaded from the neutron.ml2.type_drivers namespace
CONFIG_NEUTRON_ML2_TYPE_DRIVERS=local,vlan,flat,gre,vxlan
# Network types to allocate as tenant networks (local, vlan, gre, vxlan)
CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vlan
CONFIG_NEUTRON_ML2_MECHANISM_DRIVERS=openvswitch
CONFIG_NEUTRON_ML2_FLAT_NETWORKS=*
# list of <physical_network>:<vlan_min>:<vlan_max> or
# <physical_network> specifying physical_network names usable for VLAN
# provider and tenant networks, as well as ranges of VLAN tags on each
# available for allocation to tenant networks.
CONFIG_NEUTRON_ML2_VLAN_RANGES=physnet1:2:4094
# list of <vni_min>:<vni_max> tuples enumerating ranges of
# VXLAN VNI IDs that are available for tenant network allocation.
CONFIG_NEUTRON_ML2_VNI_RANGES=5001:10000
CONFIG_NEUTRON_L2_AGENT=openvswitch
CONFIG_NEUTRON_OVS_BRIDGE_MAPPINGS=physnet1:br-data
# list of colon-separated Open vSwitch <bridge>:<interface> pairs.
# The interface will be added to the associated bridge. If you desire
# the bridge to be persistent a value must be added to this directive,
# also CONFIG_NEUTRON_OVS_BRIDGE_MAPPINGS must be set in order to
# create the proper port.CONFIG_NEUTRON_OVS_BRIDGE_IFACES=br-data:eth1.10
CONFIG_NEUTRON_OVS_VXLAN_UDP_PORT=4789
CONFIG_HEAT_DB_PW=Lab_P@sswd
CONFIG_HEAT_KS_PW=Lab_P@sswd
CONFIG_HEAT_DOMAIN_ADMIN=heat_admin
CONFIG_HEAT_DOMAIN_PASSWORD=Lab_P@sswd
# Specify 'y' to provision for demo usage and testing
CONFIG_PROVISION_DEMO=y
# Specify 'y' to configure the OpenStack Integration Test Suite
# (tempest) for testing.
CONFIG_PROVISION_TEMPEST=y
CONFIG_PROVISION_DEMO_FLOATRANGE=172.24.4.224/28
www.allitebooks.com
Deploying OpenStack 233
CONFIG_PROVISION_TEMPEST_USER=admin
CONFIG_PROVISION_TEMPEST_USER_PW=Lab_P@sswd
CONFIG_SAHARA_DB_PW=Lab_P@sswd
CONFIG_SAHARA_KS_PW=Lab_P@sswd
CONFIG_TROVE_DB_PW=Lab_P@sswd
CONFIG_TROVE_KS_PW=Lab_P@sswd
CONFIG_TROVE_NOVA_USER=admin
CONFIG_TROVE_NOVA_TENANT=services
CONFIG_TROVE_NOVA_PW=Lab_P@sswd
Figure 7-7 Sample Output from the OpenStack Installation Using Packstack
www.allitebooks.com
CSR as an OpenStack Tenant Deployment 235
Step 3. Upload the CSR 1000V image by navigating to Admin > Images > Create
Image or using the glance CLI. Set both Minimum Disk and Minimum RAM
to 0, as shown in Figure 7-9. Example 7-2 shows the use of the glance CLI
to add a CSR 1000V image.
Note To use the CLI, su to root on the OSP7 server and set source keystonerc_
admin before running the CLI.
Example 7-2 Example of Using the glance CLI to Add a CSR 1000V Image
source keystonerc_admin
csr1kvImageSrc="/home/stack/csr_images/csr1000v-universalk9.03.16.00.S.155-3.S-ext.
qcow2"
csr1kvImageName="csr1000v-3_16"
csr1kvDiskFormat="qcow2"
csr1kvContainerFormat="bare"
csr1kvGlanceExtraParams="--property hw_vif_model=virtio --property hw_disk_
bus=virtio --property hw_cdrom_bus=ide"
tenantId='keystone tenant-list | grep " admin "| cut -f 2 -d'|''
glance image-create --name $csr1kvImageName --owner $tenantId --disk-format \
$csr1kvDiskFormat --container-format $csr1kvContainerFormat \
--file $csr1kvImageSrc $csr1kvGlanceExtraParams --is-public true
Use the glance CLI to verify that the CSR 1000V image is added correctly:
glance image-list
glance image-show csr1000v-3_16
www.allitebooks.com
CSR as an OpenStack Tenant Deployment 237
Alternatively, use the nova CLI to create a flavor with an ID of 100, 4GB of
RAM, zero disks, and two vCPUs, as follows:
nova flavor-create csr.2vcpu.4gb 100 4096 0 2
nova flavor-list
nova flavor-show csr.2vcpu.4gb
Step 5. To create the public network and internal network within OpenStack, navi-
gate to Project > Network > Network Topology > Create Network and fill
out the form as shown in Figure 7-11. Then click Next.
Enter the subnet name, network address, and gateway IP as shown in Figure
7-12, and then click Next.
Enable DHCP as shown in Figure 7-13 and enter the DHCP pool address
range in the Allocation Pools field. Fill in the DNS Name Servers and Host
Routes fields if relevant.
Step 6. Create a neutron router and attach it to the internal and public network.
From the OpenStack dashboard navigate to Project > Network > Network
Topology > Create Router. The Create Router dialog appears, as shown in
Figure 7-14.
www.allitebooks.com
CSR as an OpenStack Tenant Deployment 239
On the network topology page, click on the newly created neutron router
icon and click Add Interface, as shown in Figure 7-15, to attach the router to
the public and internal networks.
Here is an example of using the neutron CLI to create the router and attach
it to the internal and public networks:
neutron router-create neutron-rtr-1
neutron router-interface-add neutron-rtr-1 subnet=subnet-10-20-1
neutron router-interface-add neutron-rtr-1 subnet=subnet-172-16-1
neutron router-gateway-set neutron-rtr-1 public
neutron router-port-list neutron-rtr-1
240 Chapter 7: CSR in the SDN Framework
To verify that the neutron router is active and running, enter the following at
the neutron CLI:
neutron route-list
neutron router-show <router name or uuid>
neutron port-list
neutron port-show <port name or uuid>
Step 7. Create a file for the CSR 1000V default configuration script when it boots
up. Example 7-3 shows a sample of the configuration script.
hostname csr1000v
line con 0
logging synchronous
transport preferred none
line vty 0 4
login local
transport preferred none
transport input ssh
Step 8. Launch the CSR 1000V instance. From the dashboard, go to Project >
Network > Network Topology > Launch Instance. Enter the instance name,
choose the custom flavor created for the CSR 1000V, and select the CSR
1000V qcow2 image in Image Name (see Figure 7-16).
www.allitebooks.com
CSR as an OpenStack Tenant Deployment 241
On the Networking tab, select the networks the CSR 1000V will join (see
Figure 7-17).
You can also launch the CSR 1000V tenant VM by using the following at the
nova CLI:
nova boot --image csr1000v-3_16 --flavor csr.2vcpu.4gb
--nic port-id=$MGMT_PORT_ID --nic port-id=$INTN_PORT_ID
--nic port-id=$EXTN_PORT_ID --config-drive=true
--file iosxe_config.txt=/opt/stack/iosxe_config.txt csr1000v-3_16
Step 9. Access the CSR 1000V console via the network topology page and click on
the CSR 1000V instance to open the console access (see Figure 7-18). From
there you can monitor the boot process of the VM.
www.allitebooks.com
Instantiate CSR Plugin to OpenStack 243
■ Flat Network Manager offers Linux bridge for basic flat layer 2 network functions
■ Flat DHCP Network Manager (dnsmasq) for IP address allocation and management
■ VLAN Network Manager supports VLAN Network mode for direct bridging, direct
bridging with DHCP, and segments based on VLAN boundary
One of the limitations of nova-network is having three simple modes of networking that
only support VLAN-based segmentation (with a max scale of 4094 VLANs). The nova-
network stack lacks support for features like ACL and QoS. In addition, it is unable to
leverage or integrate with third-party network vendors. Project Neutron incubated in
April 2011 and was promoted to a core project at Folsom Summit in April 2012 to take
care of these limitations. Neutron provides the ability to run multiple instances with an
OpenStack-like rich feature set of APIs, plugin architecture for services, and a modular
Layer 2. The plugin architecture of Neutron enables it to leverage more than one plugin
at a given time.
Figure 7-19 shows the framework for the OpenStack Neutron plugin.
REST API
DHCP
Neutron Message Agent
Server Queue
Node
L3Services
Cisco Nexus,
More Vendor
Firewall
Plugins
N1KO
OVS
VPN
ML2
L2Agent
OVS on
Compute
Node
Type Mechanism
OpenSwan
Futures
HA Proxy
IPTables
Drivers Drivers
OpenDayLight
More Vendor
Cisco Nexus
VXLAN
Drivers
VLAN
APIC
GRE
OVS
Southbound Interfaces
Figure 7-19 clearly shows a hierarchical structure for the plugins, as core and services
plugins. You should be familiar with this diagram from Chapter 6, where we discussed
OpenStack and CSR 1000V use cases. Core plugins offer Layer 2 and Layer 3
244 Chapter 7: CSR in the SDN Framework
networking features such as VLAN, VxLAN, and GRE. The services plugin adds
capabilities like firewalls, load balancers, and VPNs to the feature list that the OpenStack
framework can support. There are limitations to the core and service plugins native to
OpenStack. For example, the Neutron NAT function uses Linux IPTables for NAT,
which has severe scale limitations. You can overcome these limitations by leveraging the
CSR 1000V as an OpenStack plugin that offers robust Layer 3 forwarding and security
service capabilities. OpenStack is a framework that can scale to different features and
vendor-specific capabilities through the plugin architecture.
Chapter 6 covered how to deploy a CSR 1000V router as a neutron using plugins. Using
devStack simplifies the process of replacing a neutron router with CSR 1000V. The
plugins required for doing this can be retrieved from GitHub as detailed in the following
steps (GitHub is a web-based Git hosting repository service):
■ If your setup is behind a proxy server, switch git:// to https:// for git to
work properly:
git config --global url."https://".insteadOf git://
git config --global https.insteadOf.git
Step 4. Download the OpenStack release kilo devstack from the OpenStack host
server. After you install it, the directory devstack will be created under
home directory:
cd ~/ git clone https://github.com/openstack-dev/devstack.git
www.allitebooks.com
Summary 245
For CSR VPN-aaS, enter the following. Ensure that CSR Routing-aaS is
enabled.
enable_service cisco_vpn
For CSR FW-aaS, enter the following. Ensure that CSR Routing-aaS is
enabled.
enable_service cisco-fwaas
The CSR 1000V now has the plugin in OpenStack to replace the neutron router that
OpenStack uses by default. CSR 1000V (with a plugin running in OpenStack) offers
a more feature-rich virtual routing functionality that compliments the OpenStack
environment.
Summary
This chapter reviews the importance of SDN concepts in cloud environments. The SDN
architecture has a basic framework that includes APIs, overlay, and controllers. The
deployment and consumption of SDN depends on the user environments and also the
hardware/software vendors involved. OpenStack provides cloud operators a framework
and flexibility to manage a cloud environment. OpenStack also allows admins to leverage
vendor-specific plugins to build new capability in the cloud infrastructure. You should
now be comfortable deploying OpenStack and leveraging Neutron.
This page intentionally left blank
www.allitebooks.com
Chapter 8
The terms orchestration and automation are sometimes used interchangeably. However,
they are different. Automation involves achieving the completion of a single task, such
as configuring a service on a router. Orchestration, on the other hand, involves automat-
ing a series of events to achieve a complete workflow or process.
Let us view this from a cloud perspective now. Cloud automation includes tasks that
are required for deploying a cloud resource, such as spinning a routing instance (a
virtual router, for example) in the cloud. Just this act of spinning a VM in the cloud
should be considered automation. When you configure this instance, make it interact
with other elements/instances and piece together a solution, it is orchestration.
Orchestrating a network solution can be broken down into the following high-level tasks:
■ Managing the entity created using tools like Elastic Service Controller (ESC) that are
popular for lifecycle management of VMs.
With creation, life cycle management, and configuration automated, solutions can be
designed to orchestrate a workflow and give turnkey solutions to customers. Virtual
Managed Services (VMS) and Prime Network Services Controller (PNSC) automation
solutions create a framework and use various tools to achieve service orchestration
functionality.
Automation
The following sections review two common tools used for automation. The effort of
automation varies based on the cloud use case that you intend to automate; the com-
plexity of the tools also depends on this. You will learn about the BDEO tool, which
is used to instantiate a single CSR 1000V instance, and the NSO tool, which is used to
accomplish service chaining and network nodes.
BDEO
The BDEO tool is available to instantiate a CSR 1000V router. This tool is available
when you download a CSR 1000V image from “CCO downloads.” As shown in Figure
8-1, the BDEO tool instantiates a CSR 1000V ISO image with the base configuration on
the hypervisor. An administrator can leverage the BDEO tool to instantiate multiple CSR
1000V instances with the base configuration.
CSR ISO
BDEO Tool
IOS CLIs
hostname CSR RP CSR 1000V
FP
interface GigabitEthernet1
ip address 10.2.1.1 255.255.255.0
no shut Hypervisor
Physical Hardware
interface GigabitEthernet2
ip address 10.2.2.1 255.255.255.0
no shut
■ It takes OVA (or ISO) as input. It outputs custom OVA, pre-provisioned with basic
IOS configuration elements (management IP address, SSH, hostname, and so on). It
is helpful for immediate provisioning.
www.allitebooks.com
Automation 249
■ It is possible to apply a complete IOS config.txt, but you must deploy it via vCenter
and cannot reference the host directly
■ BDEO provides the intelligence to extract the config.info file and pass it to IOS,
and it requires VMware’s Open Virtualization Format (OVF) tool for deployment.
Note that for 3.9S1 and later versions of the CSR 1000V, the recommended tool for
single/standalone installation is the Common OVF Tool (COT) instead of BDEO. You can
use COD for editing the OVF of a virtual appliance with a focus on the Cisco CSR 1000V.
■ Can edit OVF hardware information (CPUs, RAM, NICs, configuration profiles, etc.)
NSO (Tail-f)
The NSO tool is the same as Tail-f, which has been mentioned in earlier chapters. Figure
8-2 shows the architecture of NSO.
NSO Architecture
Service Manager
Templates
Package Core Mapping
AAA
Manager Engine Logic
FastMap
Cache
NSO Engine
Alarm Manager Notification Receiver Device Manager
The following are some of the highlights of the NSO architecture, which is defined using
a layered approach:
■ The connectivity to the user interface (UI) or any interface stack for user-defined
scripts for provisioning is handled with a rich set of tools, including NETCONF,
REST, CLI, SNMP, and Java.
■ The core engine described in Figure 8-2 is a key element of the NSO orchestra-
tion architecture. The core engine ties different blocks in the NSO architecture to
execute a defined function.
■ The NSO has a service manager that integrates the administrator-defined instruc-
tions to the orchestration engine.
■ The AAA layer provides the engine role-based access to the devices and also to
administrators accessing the device inventory.
■ The package manager acts like a plugin that attaches itself to the core engine. The
package manager provides flexibility for different use cases to be added to the
orchestration profile, like GUI deployment for specific functionality that allows the
pictorial representation of the orchestration and overview of health devices man-
aged by NSO.
■ The NSO engine is made up of the database with the mapping logic and fast map
cache. Mapping logic and templates enable you to map intended service operations
to network configuration required to implement the intended services. In other
words, this is where a service intent is mapped to the elements that need to be
modified to get the service implemented. Transactional integrity is maintained by
a database that stores the configuration of each device on-boarded by the device
manager. The unique thing that differentiates NSO from the rest of the orchestra-
tion tools is the capability to run non-real-time configuration event correlation of
the new command function on the CDB (configuration database) baseline templates
at fast Map cache. With FASTMAP you just need to specify the CREATE opera-
tion and not the REDEPLOY, UPDATE, and DELETE (RUD) operations. The RUD
operation is automatically generated by the FASTMAP logic. This is done by having
the NSO database view the actual as well as the intended device configurations for a
particular service.
■ The alarm manager manages alarms generated when there are errors pushing the
configurations to end devices (via the NED) and from simulated results run in the
Fast Map cache. The alarm manager can have a user-defined function written that
takes action when a fault appears during a real-time deployment. The action can be
a rollback, a log, or a priority-based log notification. The alarm manager is synced
with the notification receiver that receives real-time notification.
■ The device manager needs to be populated with a list of devices. This can be done
manually or via automatic discovery.
www.allitebooks.com
Automation 251
■ The network element drivers (NED) allow the tool to communicate with different
device elements. This can also be vendor specific to collaborate with vendor capa-
bilities—for example, if a YANG-based model is not supported, then a NED can
be written to the vendor-specific CLI. These NEDs are part of the package manager
that attaches with the core engine and maps access to devices. The NED also adds
specific rules used in mapping engine services for the operation of NSO tool.
■ The process of configuring upgrades in the NSO tool involves these steps:
3. The NSO uses the device enlisted in the device manager to run the command
in FastMap and mapping service for optimal configuration. Any false alarm is
reviewed and assessed according to admin-defined rules.
4. The appropriate NED pushes the optimal device configuration to the devices.
6. After the new configuration is applied, the NSO adds the configuration to the CDB.
NSO is a very powerful tool and creates an abstraction for various functions. This
tool can be modified and inserted in multiple solutions that can use different UIs,
scripts for particular solutions, and NEDs as configuration management devices. For
orchestration of NFV elements, ESC adds extra elasticity in provisioning and the
life cycle that is helpful in cloud environments.
The deployment use case in NSO is explained in the simple three-tier layer shown in
Figure 8-3.
The northbound APIs can invoke and provide input to a service model written in YANG.
The service model provides a definition to the service functionality that needs to be
deployed. This generic model can be used with multiple tenant instances to derive the
same outcome. The device model is vendor-specific information that provides instruc-
tions for the device to function. This device model can cover CLI or YANG-based func-
tions and interacts with the NED to communicate with the end devices.
252 Chapter 8: CSR 1000V Automation, Orchestration, and Troubleshooting
Northbound APIs
Service Models
Device Models
NED
Northbound APIs
NED (Step 2)
www.allitebooks.com
Automation 253
Step 1. Define the service chain in NSO to get called by the northbound UI. This
means designing the service, describing the intent (functionality that needs to
be performed), and describing the elements needed to facilitate the request
and input parameters that will be applied to the devices (speed, users, band-
width, and so on).
Example 8-1 shows a simple base script written in YANG for service chaining.
Example 8-1 Sample NSO Initiation to Understand Input to Be Used in the Service and
Device Model Framework
root@nso0:/nfv/poc/local/packages/serviceChain/src/yang# cat serviceChain.yang
module serviceChain {
namespace "http://com/example/serviceChain";
prefix serviceChain;
import ietf-inet-types {
prefix inet;
}
import tailf-common {
prefix tailf;
}
import tailf-ncs {
prefix ncs;
}
import resource-allocator {
prefix ralloc;
}
typedef deviceref {
type leafref {
path "/ncs:devices/ncs:device/ncs:name";
}
}
list dctopology {
key dc-name;
leaf dc-name {
type string;
}
leaf dc-type {
type enumeration {
enum core;
enum edge;
}
}
254 Chapter 8: CSR 1000V Automation, Orchestration, and Troubleshooting
leaf esc-os {
type deviceref;
}
leaf openstack {
type deviceref;
}
leaf vts-vc {
type deviceref;
}
container zenoss {
leaf ip {
type inet:ipv4-address;
}
leaf community {
type string;
}
}
container vcenter {
description "Configuration related to VMware, optional";
presence vcenter;
leaf vcenter {
description "vCenter device for this DC";
mandatory true;
type deviceref;
}
container esxi {
container asa {
leaf-list ip {
description "List of ESXi hosts for provisioning ASAv";
min-elements 1;
type inet:ipv4-address;
}
}
container csr {
leaf-list ip {
description "List of ESXi hosts for provisioning CSR";
min-elements 1;
type inet:ipv4-address;
}
}
}
leaf vswitch {
www.allitebooks.com
Automation 255
container mgmt-network {
description "Management interface configuration";
leaf mgmt-ip-pool {
description "resource-allocator ip-address-pool for management IPs";
type leafref {
path "/ralloc:resource-pools/ralloc:ip-address-pool/ralloc:name";
}
}
container mgmt-route-network {
description "Network to use when setting up route to mgmt gateway";
leaf network {
type inet:ipv4-address;
}
leaf netmask {
type inet:ipv4-address;
}
}
}
leaf datastore {
description "Datastore for deployed VNFs";
default datastore1;
type string;
}
}
leaf border-leaf {
description "Border leaf to be used as a gateway";
type inet:ipv4-address;
default 192.168.101.1;
}
}
augment /ncs:services {
list serviceChain {
description "Instantiate the service chaining function";
key name;
256 Chapter 8: CSR 1000V Automation, Orchestration, and Troubleshooting
leaf name {
tailf:info "Unique service id";
tailf:cli-allow-range;
type string;
}
uses ncs:service-data;
ncs:servicepoint serviceChain-servicepoint;
container status {
config false;
tailf:cdb-oper { tailf:persistent true; }
leaf serviceChain-phase {
type enumeration {
enum tenant;
enum network;
enum day0;
enum day1;
enum finished;
}
}
}
list vm {
key name;
leaf name {
type string;
}
leaf dc {
type leafref {
path "/serviceChain:dctopology/serviceChain:dc-name";
}
mandatory true;
}
leaf vim-type {
type enumeration {
enum openstack;
enum vcenter;
}
mandatory true;
www.allitebooks.com
Automation 257
}
leaf vm-type {
type enumeration {
enum asa;
enum csr;
}
}
container ip-pool {
when "../vm-type = 'asa'";
list pool {
max-elements 1;
min-elements 1;
key "start end";
leaf start {
type inet:ipv4-address;
}
leaf end {
type inet:ipv4-address;
}
leaf mask {
mandatory true;
type inet:ipv4-address;
}
}
}
container acl {
when "../vm-type = 'csr'";
list access-list {
ordered-by user;
key name;
leaf name {
type string;
}
leaf protocol {
default ip;
type enumeration {
enum ip;
enum icmp;
enum tcp;
enum udp;
}
}
leaf src-ip {
mandatory true;
258 Chapter 8: CSR 1000V Automation, Orchestration, and Troubleshooting
type inet:ipv4-address;
}
leaf src-mask {
mandatory true;
type inet:ipv4-address;
}
leaf src-port {
when "../protocol = 'tcp' or ../protocol = 'udp'";
type union {
type uint16;
type enumeration {
enum any;
}
}
}
leaf dest-ip {
mandatory true;
type inet:ipv4-address;
}
leaf dest-mask {
mandatory true;
type inet:ipv4-address;
}
leaf dest-port {
when "../protocol = 'tcp' or ../protocol = 'udp'";
type union {
type uint16;
type enumeration {
enum any;
}
}
}
leaf action {
mandatory true;
type enumeration {
enum permit;
enum deny;
}
}
}
}
list interface {
max-elements 3;
min-elements 3;
key name;
leaf name {
www.allitebooks.com
Automation 259
type string;
}
leaf network-type {
type enumeration {
enum mgmt;
enum private;
enum external;
}
}
leaf ip {
type inet:ipv4-address;
}
leaf mask {
type inet:ipv4-address;
}
leaf is-managed {
type empty;
}
leaf network-name {
type string;
}
leaf subnet-prefix {
when "boolean(../is-managed)";
type inet:ip-prefix;
}
leaf gateway-ip {
when "boolean(../is-managed)";
type inet:ip-address;
}
} //interface
} // vm
tailf:action remove-service {
tailf:info "Remove service";
tailf:actionpoint serviceChain-remove-service;
output {
leaf success {
type boolean;
}
leaf message {
type string;
description "Free format message.";
}
}
}
260 Chapter 8: CSR 1000V Automation, Orchestration, and Troubleshooting
} //serviceChain
}
augment "/ncs:devices/ncs:device" {
leaf transition-workaround-ready {
description "Workaround for checking readiness of device - transition package
is weird after redeploy";
config false;
tailf:cdb-oper {
tailf:persistent true;
}
type boolean;
}
}
}
This model provides basic elements needed for service chaining and defines
the parameters of the VNF component and base configuration that can be
orchestrated from the user using REST API or XML structure. Using this
script, the NSO captures the functionality of the service chaining elements.
Translating this to vendor-specific configuration needs special templates that
tie to the NED (vendor-specific attributes). Multiple models can use these
templates. The templates are vendor/device specific and are not covered in
this book. This base template provides NSO a framework that enables it to
receive inputs from the user interface.
Step 2. NSO should have the appropriate NEDs to communicate with the infrastruc-
ture southbound. The command ls –l $NCS_APP/packages helps verify the
installation of the packages for the NEDs, as shown in Example 8-2.
www.allitebooks.com
Automation 261
cisco-ios/src/ncsc-out/modules/fxs/tailf-ned-cisco-ios-id.fxs
cisco-ios/src/ncsc-out/modules/yang
cisco-ios/src/ncsc-out/modules/yang/tailf-ned-cisco-ios-id.yang
cisco-ios/src/ncsc-out/modules/yang/tailf-ned-cisco-ios-stats.yang
cisco-ios/src/ncsc-out/modules/yang/tailf-ned-cisco-ios.yang
...
Step 3. Onboard VNF descriptors detail the characteristics of the VNF, such as
capabilities (like firewall or router), resources required (CPU, storage, and
so on), and artifacts to instantiate it (disk image). This is the device package
in the deployment hierarchy. Example 8-3 provides an example of VNF
descriptor definition. In this example, the ASA and CSR are both defined
as VNF descriptors. These two instances will be reviewed in later service
chaining steps.
+ device-type {
+ cli {
+ ned-id cisco-asa;
+ }
+ }
+ authgroup asa;
+ day0 {
+ source-url http://192.168.100.16/nfv/day0/ASA_941_day0.txt;
+ destination-file day0-config;
+ }
+ }
+ }
+ vnfm {
+ esc esc0;
+ }
+ }
+ vnfd CSR313 {
+ version 3.13.01;
+ connection-points left {
+ flavour small;
+ vdu CSR;
+ nic-id 1;
+ }
+ connection-points right {
+ flavour small;
+ vdu CSR;
+ nic-id 2;
+ }
+ flavours small {
+ vdus CSR {
+ vm-spec {
+ pkg-uri http://192.168.100.16/nfv/qcow/csr1000v-universalk
9.03.13.01.S.154-3.S1-ext.qcow2;
+ disk-format qcow2;
+ }
+ storage root-disk {
+ size-gb 8;
+ storage-type root;
+ }
+ cpu {
+ num-vpu 1;
+ }
+ memory {
+ total-memory-gb 3;
+ }
+ device-type {
www.allitebooks.com
Automation 263
+ cli {
+ ned-id cisco-ios;
+ }
+ }
+ authgroup csr;
+ day0 {
+ source-url http://192.168.100.16/nfv/day0/CSR_day0.txt;
+ destination-file iosxe_config.txt;
+ }
+ disk-bus virtio;
+ serial-console true;
+ e1000-net true;
+ }
+ }
+ vnfm {
+ esc esc0;
+ }
+ }
+ management-ip-pool net_osmgmt;
+ nsd advanced {
+ connection-points a {
+ member-vnf fw;
+ vnfd-connection-point inside;
+ }
+ connection-points z {
+ member-vnf router;
+ vnfd-connection-point right;
+ }
+ flavours small {
+ member-vnfs fw {
+ vnfd ASA941;
+ flavour small;
+ vdu ASA;
+ }
+ member-vnfs router {
+ vnfd CSR313;
+ flavour small;
+ vdu CSR;
+ }
+ }
+ }
+ nsd basic {
+ connection-points a {
+ member-vnf router;
+ vnfd-connection-point left;
+ }
264 Chapter 8: CSR 1000V Automation, Orchestration, and Troubleshooting
+ connection-points z {
+ member-vnf router;
+ vnfd-connection-point right;
+ }
+ flavours small {
+ member-vnfs router {
+ vnfd CSR313;
+ flavour small;
+ vdu CSR;
+ }
+ }
+ }
}
devices {
authgroups {
+ group asa {
+ default-map {
+ remote-name admin;
+ remote-password $4$wIo7Yd068FRwhYYI0d4IDw==;
+ remote-secondary-password $4$wIo7Yd068FRwhYYI0d4IDw==;
+ }
+ }
}
}
Step 4. Service is deployed between VNF and a physical device for Layer 3 con-
nectivity from the OpenStack CSR tenant to an external physical device. This
ties the device package with the service package, creating a customized ser-
vice chained configuration. Example 8-4 is a sample for step 4 to instantiate
VNF for ASA and CSR with service chaining.
vpn {
+ l3 ACME { << A L3 service for
customer ACME
+ as-number 65000;
+ endpoint branch-office { << First endpoint, the
branch office
+ ce-device ce7;
+ virtual { << Denotes we need a
virtual device
+ p-device p3;
+ nsd advanced; << Type of network service
descriptor (Defined in step
3 with the VNFD on-boarding – a service
chain of FW and RT)
www.allitebooks.com
Automation 265
+ p-connection-point a;
+ ce-connection-point z;
+ vnfm esc0; << VNFM that will instantiate
the VNF
+ }
+ ce-interface GigabitEthernet0/1; << From here on, specify L3VPN
configurations
+ ip-network 172.14.1.0/24;
+ bandwidth 50000;
+ }
+ endpoint main-office { << Second endpoint, the main
office - Provider
+ ce-device ce4;
+ ce-interface GigabitEthernet0/1;
+ ip-network 172.15.1.0/24;
+ bandwidth 60000;
+ }
+ }
}
Once you commit step 4 on the NSO, the model is executed from the NSO. The NSO
uses the ESC to spawn the VNF elements.
■ Before executing Step 4, no service is associated with the tenants (see Figure 8-5).
■ Figure 8-6 shows the OpenStack tenant after execution of the script.
The following service chaining components are added after execution of the script:
www.allitebooks.com
Orchestration 267
Orchestration
Orchestration leverages automation engines to drive a user-defined function across mul-
tiple environments. Some of these tools leverage automation engines like NSO. The fol-
lowing sections describe the tools that are commonly used for orchestration.
Service workflows cover the service initiation for the user-defined workflows. This
block consists of a tool that has a user-friendly interface to execute the workflows. The
domain orchestration involves individual tools used for service initiation for different
infrastructure solutions, as defined in Figure 8-9. This framework provides flexibility
in terms of choosing a tool that is relevant to the solution for a specific user interface,
domain orchestrator, or type of device configuration service.
268 Chapter 8: CSR 1000V Automation, Orchestration, and Troubleshooting
Service Workflows
Domain Orchestration
DC (Services/
On-Prem WAN Access Computing/
Network)
Service
CPE Router Metro Chaining
(NFV)
As shown in Figure 8-10, the VMS solution covers the tools used for the tenant portal in
the Cisco Prime catalog: Orchestration is covered by NSO, and the instantiation of the
NFV elements is handled by the Elastic Services Controller (ESC). The NSO signals the
ESC to instantiate a new NFV element. The ESC also stores the initial configuration and
has a rule engine to access the elastic scale. This elastic scale is based on NFV monitoring
via SNMP and custom scripts. This function helps create a new NFV element on demand
if the CPU or load threshold increases in the existing NFV element. This is why the
VMS solution uses the ESC controller instead of using NSO directly for instantiation.
■ VNF Manager—Takes care of life cycle management of VNF instances. ESC in the
VMS framework takes care of VNF manager functionality.
www.allitebooks.com
Orchestration 269
Tenant Portal
OpenStack
The NFV components are grouped together with a specific profile. The service chaining
A and B profiles are used in tenants A and B. The same service chaining A profile can be
used in tenant C (a new tenant). Creating these groups helps provision a tenant within
minutes for a cloud environment. Cisco Prime has multi-hypervisor support and creates
service chaining through NFV components in different hypervisors. Most of the enter-
prise environments have vCenter deployed in their data centers to facilitate computing
virtualization. This tool fits in those environments to facilitate not only CSR provisioning
and configuration but also other NFV elements, like ASA 1000V and VSG. The deploy-
ment of configuration for these NFV instances is GUI driven and largely used for tradi-
tional data center management and operations. These are some of the important features
of the PNSC management solution:
■ In a vCenter environment, PNSC associates with the hypervisor and can view all the
host VMs seen by vCenter.
270 Chapter 8: CSR 1000V Automation, Orchestration, and Troubleshooting
Service Chaining - A
FW LB
Tenant A
Service Chaining - B
IPS FW LB
Tenant B
Note The vPATH function of the Cisco Application Virtual Switch (AVS) or
Nexus 1000V provides abstraction of the forwarding plane by redirecting packets
to appropriate virtual service nodes that offer NFV services. vPATH steers traffic to
optimize the traffic flow between the host and the service chained elements.
These are the main steps to remember for installing PNSC in a vCenter computing
environment:
1. Create a tenant from vCenter and boot it with the correct OVA file. During this cre-
ation, the IP address for management should be added.
2. Register PNSC as a plugin to vCenter and accept the new plugin in vCenter. After
the registration, create VM membership at PNSC, pointing to the vCenter’s manage-
ment IP address.
3. Create service chaining for a workflow with the required NFV element.
www.allitebooks.com
CSR 1000V Troubleshooting 271
■ I/O Configuration—This section gives an overview of the I/O models that are cur-
rently supported with the CSR 1000V virtual machine running on an ESXi hypervi-
sor. It is important for a troubleshooter to understand the deployed I/O model
prior to troubleshooting.
Architecture Overview
Architecture of the CSR 1000V is covered in detail in the earlier chapters. Figure 8-12
illustrates the different components we analyze subsequently in the chapter to assist in
debugging and troubleshooting packet flow issues.
CSR 1000V
App App
RP
OS OS FP
VPC/vDC
Hypervisor
Virtual Switch
Server
From a high level, there are three components in the architecture (covered in detail in
earlier chapters) of the CSR 1000V virtual router:
■ The CSR 1000V VM—As detailed in previous chapters, the CSR VM leverages
the IOS XE software infrastructure. It runs on a Linux kernel with applications
running in the user space. IOSd and other IOS XE processes run in the user space.
272 Chapter 8: CSR 1000V Automation, Orchestration, and Troubleshooting
The CSR 1000V control plane is made up of IOSd and other IOS XE infrastructure
processes:
■ The CSR data plane consists of the RX, PPE, and the HQF threads. All these
components run on a single process within the QFP process, with multiple PPE
threads. The software infrastructure of the CSR data plane ensures efficient
movement of packets between the vNIC interface and the user space. The data
path “punts” packets to the control plane if required, and the control plane
“injects” data path packets to the vNIC interfaces.
■ The IOS XE control plane and data plane share a single vCPU (in the case of a
1vCPU configuration). In the case of a multi-vCPU configuration, the control
plane gets one vCPU. The remaining CPUs are allocated to the data plane.
■ Packet drops and troubleshooting should first be done at the virtual machine
level, using the IOS XE command-line interface that is provided.
■ The host machine—The host machine is the piece of hardware that the hypervisor
manages.
I/O Configuration
On a physical router, the physical interfaces are either built in to the hardware or made
available to the software by means of interface devices such as shared port adapters
(SPA). In a virtualized environment, the VM uses the physical NIC on the host machine
as an interface. There are multiple ways to connect a physical NIC to a CSR VM. It is
important to understand this connection methodology to be able to successfully debug
and troubleshoot an event. Performance and latency of a CSR 1000V will depend on
how the I/O connection is set up.
The most common way a physical NIC is made available to the CSR VM is via a virtual
switch.
vSwitch
A virtualized switch sits between a VM and the vNIC that the hypervisor presents. ESXi
supports VMware’s distributed vSwitch and Cisco’s Nexus 1000V. This chapter covers
VMware’s vSwitch.
www.allitebooks.com
CSR 1000V Troubleshooting 273
CSR 1000V
RP
FP
vSwitch
vNIC
Hypervisor
Physical NIC
The virtual switch offers a Layer 2 connection between the VMs connected to it. It
does not support physical switch features like IGMP snooping and Spanning Tree
Protocol (STP).
Understanding the differences between a virtual switch and a physical switch will help
you better debug issues. These are two important differences that need to be highlighted:
■ Virtual switches do not need to support STP. The reason is because you do not
need STP here to avoid loops. Because it is not possible to connect two virtual
switches together, the only way to create loops in a virtualized switch would be
to run bridging software as a guest VM. Hence STP is not required for a virtual-
ized switch. The vSwitch should function in promiscuous mode. In this mode, only
the objects defined within that port group have the option of receiving all incom-
ing traffic on the vSwitch. Interfaces outside the port group in the vSwitch do not
receive the traffic.
274 Chapter 8: CSR 1000V Automation, Orchestration, and Troubleshooting
PCI Passthrough
In PCI passthrough mode, the hypervisor software that manages the physical NIC
is completely bypassed using a Peripheral Component Interconnect Express (PCI-e)
passthrough configuration; the guest VM gets direct access to the hardware NIC. This
increases the throughput and reduces latency that would be introduced due to the
hypervisor scheduling the NIC. The disadvantage here is that the NIC is dedicated to
the VM, and virtualization for that NIC is not possible. You cannot use virtualization
features (like vMotion) that the hypervisor has to offer but need to use the NICs that are
supported by the guest VM. No driver is required on the ESX host machine. The guest
VM needs to have a driver to support the NIC. This I/O model is also referred to as
VM-DirectPath I/O.
CSR 1000V can work with Intel’s 1G and 10G NICs in a PCI-e passthrough mode, as
shown in Figure 8-14.
CSR 1000V
RP
FP
PCI-e Passthrough Mode
vSwitch
vNIC A
Hypervisor
A PF is a full-featured PCI-e function, which means it is managed like any other PCI-e
device. A PF has a complete configuration resource, which means it completely owns the
PCI-e device and can move data in and out of the device.
www.allitebooks.com
CSR 1000V Troubleshooting 275
VFs work like PFs except for the configuration resource piece. A VF does not have a
configuration resource because you do not want a VF to change the configuration. You
just need the VF to move data in and out. The control for configuration change rests
solely with the PFs, and the VF config should be dictated by the underlying PFs. VFs
are not complete PCI-e devices, and so the hypervisor must be aware of the fact that it
is dealing with an incomplete PCI-e device. This virtualization feature requires software
support at the hypervisor level. For SR-IOV to work, you need BIOS and hardware sup-
port as well as support in software at the hypervisor/OS level.
Figure 8-15 illustrates the SR-IOV model.
CSR 1000V
RP
FP
vSwitch
vNIC A vNIC B
VF PF VF
Physical NIC A
Host Configurations
The hardware on which the CSR runs plays an important role in the throughput perfor-
mance of the VM. The user must be aware of the hardware configurations of the host
when debugging and troubleshooting packet flow and other issues with respect to the
CSR 1000V. Following are some hardware specifications to look out for:
■ CPU—Awareness on what kind of CPU the host machine has is significant because
CPU architectures and throughputs change from hardware vendor to hardware ven-
dor. This has a direct impact on the performance of the VM running on the host
machine.
■ Sockets—Most modern servers have CPUs with multiple sockets. When running an
application over multiple sockets, the user sees degraded performance due to con-
text switching, inefficient memory and cache access, and increased bus utilization.
276 Chapter 8: CSR 1000V Automation, Orchestration, and Troubleshooting
■ BIOS—BIOS settings on the hardware can impact the performance of the CSR.
Following are certain BIOS settings the user should be aware of:
■ Power—This setting may put the processor to sleep if the system is deemed idle.
This is done to save power, and it impacts the CSR’s performance. Therefore,
C-state should be set to C-0 to disable this feature in BIOS.
■ Rx thread
■ Tx thread
All these components run in a single process under the QFP process umbrella. Multiple
PPE threads serve requests within this QFP process.
In Figure 8-16, physical NICs are connected to two separate virtual switches. The packet
comes in on the physical NIC. The hypervisor maps the packet to the vNIC and then for-
wards it to the attached vSwitch. The vSwitch forwards the frame based on the destina-
tion MAC address. The frame is then received by the CSR 1000V’s virtual interface. The
packet then goes to the IOS XE code for processing, as detailed in Chapter 4.
www.allitebooks.com
CSR 1000V Troubleshooting 277
CSR 1000V
RP
FP
vSwitch vSwitch
vNIC A vNIC B
Hypervisor
The following sections discuss certain things you need to check if you see packet loss on
a CSR 1000V VM.
Example 8-5 shows how to check the throughput level and license details.
For the license to be enforced properly, its state should be Active, In Use. It is impor-
tant to note that the CSR license policer is an aggregate policer and not an interface-
level policer. If the license is for 1G, you can send 1G traffic combined through all inter-
faces. This should be the sum of all traffic going in or out of all interfaces from a CSR
virtual machine.
To check whether traffic drops are due to traffic exceeding the license, use the follow-
ing command and look for tail drops in the QFP subsystem:
Router# show platform hardware qfp active statistics drop clear | exc _0_
-------------------------------------------------------------------------
Global Drop Stats Packets Octets
-------------------------------------------------------------------------
TailDrop 2018258 256333010
The license uses a policer that tail drops packets when you exceed the bandwidth
enforced through the license.
www.allitebooks.com
CSR 1000V Troubleshooting 279
The user needs to make sure the interface NIC is not oversubscribed, which means the
user should not try to use the speed command when the physical NIC is a 1G module.
Example 8-7 is sample output from the show interface command on a CSR VM.
280 Chapter 8: CSR 1000V Automation, Orchestration, and Troubleshooting
L2 MTU
The CSR 1000V supports an L2 MTU of 1500–9216. This, as in any other IOS/IOS XE
device, can be changed using the mtu command in interface configuration mode. It is,
however, important to note that just changing the MTU on the CSR does not guarantee
a configured MTU. The user needs to make sure the vSwitch is configured with an MTU
value that equals the MTU configured on the CSR interface.
www.allitebooks.com
CSR 1000V Troubleshooting 281
Interface-to-NIC Mapping
To be sure of what interface on the CSR VM matches the interface configured on ESXi,
the user should run the following command on the CSR VM:
The user should note the MAC address and verify it with the interface configuration on
the ESXi host.
GigabitEthernet1
GigabitEthernet2
GigabitEthernet3
Example 8-8 shows the control-processor CLI that gives the control and data plane
memory and CPU status. IOS XE drops packets when the CPU utilization reaches 100%.
Memory (kB)
Slot Status Total Used(Pct) Free(Pct) Committed(Pct)
RP0 Healthy 8117052 3226704(40%) 4890348(60%) 3711452(46%)
CPU Utilization
Slot CPU User System Nice Idle IRQ SIRQ IOwait
RP0 0 1.93 0.90 0.00 96.36 0.00 0.80 0.00
1 95.01 4.98 0.00 0.00 0.00 0.00 0.00
CSR
Ingress Interface Egress Interface
Make sure a packet is received and transmitted by the vNIC. The first step is to make
sure the packets are properly received and transmitted by the CSR vNIC interface. The
IOS show interfaces command displays the packet statistics for a vNIC interface.
Example 8-9 shows the CLI output for the show interfaces command.
www.allitebooks.com
CSR 1000V Troubleshooting 283
To check the vNIC driver statistics, use the show interface controller command.
The CSR vNIC driver should be fast enough to get packets onto the data plane. This
driver should not be a bottleneck because the packet drops at the driver level are ran-
dom. You can increase the vNIC queue size if there are drops at the driver level.
Example 8-10 shows the show interface controller output. The key part of the CLI
output that needs to be examined during troubleshooting is highlighted.
www.allitebooks.com
CSR 1000V Troubleshooting 285
fcs 0
rx buf alloc fail 0
tx timeout count 0 too many frags 0
giant hdr 0
hdr err 0
The errors in Example 8-10 indicate a completely overloaded CSR data plane.
The packets make it to the CSR data plane. The show interfaces <name> stats com-
mand tells the user whether the data plane received packets from the vNIC interface:
To check the QFP subsystem (data plane) drops, use the statistics drop command:
TailDrop This error is generally due to packets getting dropped at the egress
interface. One of the common reasons for this error is sending
traffic beyond license. Another reason can be overload by packet
fragmentation. This can happen when packets are fragmented due
to the MTU size.
UnconfiguredIpv4Fia This means the IP packet is being received and the IPv4 address is
not set on that interface.
UnconfiguredIpv6Fia This means the IP packet is being received and the IPv6 address is
not set on that interface.
IOS XE allows you to get into the details of the feature drops, as shown in Example
8-11, which shows the debugging options at the QFP level.
www.allitebooks.com
CSR 1000V Troubleshooting 287
4 IN_US_V4_PKT_SA_NOT_FOUND_SPI 2322
7 IN_TRANS_V4_IPSEC_PKT_NOT_FOUND_SPI 0
12 IN_US_V6_PKT_SA_NOT_FOUND_SPI 0
288 Chapter 8: CSR 1000V Automation, Orchestration, and Troubleshooting
The QFP starts dropping packets if the utilization reaches 100%. Pushing in more CPUs
should solve this problem. A high CPU number here indicates that the IOS XE software
needs more QFP processing power.
To check the packet drops on the RX process of the data plane, use the following com-
mand shown in Example 8-12, which provides the data path sw-nic information.
ZNM info:
poll err 0; epochs: znm 3 wait_all 3
poll calls 28079227 priorities 1
im-tx - active
www.allitebooks.com
CSR 1000V Troubleshooting 289
www.allitebooks.com
CSR 1000V Troubleshooting 291
It is recommended that you increase the receive ring buffer size to 4K because the
device driver is not fast enough to retrieve these packets and post new buffers. You do
this by using the following command:
# ethtool –G <vmnic> rx <size>
If this does not help, check the version of the driver by using the following command:
# ethtool –i <vmnic>
From the VMware support site, check whether an updated version of this driver is avail-
able. Sometimes upgrading the driver helps resolve issues.
These same details of packet statistics can also be obtained from vCenter, as shown in
Figure 8-19. They are present at Hosts > Monitor > Performance > Advanced. Using the
chart options, you can select the chart metrics as Network, object type as the VM’s VM
NIC, and other desired counters, such as data transmit rate, data receive rate, and packet
transmit errors.
You can check whether the packets are being received at vmnic to the vSwitch ingress
port. Example 8-15 shows the port stats from an ESXi host.
One of the reasons for the packet drops might be the high CPU Ready Time for the
VM. The CPU Ready Time is the time when the VM is ready to run but the hypervisor is
not able to schedule it on a physical processor. A high value indicates poor performance.
This value can be found by running the following command on the host:
# esxtop
The value returned should ideally be zero. If it is not, there is not enough CPU for the
VM to be scheduled in time, which may lead to packet drops. Reduce the number of
VMs on the ESXi host if you run into this issue.
Summary
This chapter reviews three import topics: automation, orchestration, and troubleshoot-
ing. The first part of this chapter disambiguates the concepts of automation and orches-
tration, using the key tools available. After reading this chapter, you should be able to
appreciate the difference between these two concepts and should be in a position to
choose the correct tool for your virtualized infrastructure.
This chapter also describes how to troubleshoot a CSR VM on an ESXi hypervisor. You
should be able to follow the troubleshooting approach described here and leverage the
command-line outputs to get a better situational awareness of a CSR VM.
www.allitebooks.com
Appendix A
The following output is the complete output of the abbreviated Example 7-1 from
Chapter 7, “CSR in the SDN Framework.” It provides a sample answer file for Packstack.
[general]
www.allitebooks.com
295
# Specify 'y' if you want to use subnet addresses (in CIDR format)
# instead of interface names in following options:
# CONFIG_NOVA_COMPUTE_PRIVIF, CONFIG_NOVA_NETWORK_PRIVIF,
# CONFIG_NOVA_NETWORK_PUBIF, CONFIG_NEUTRON_OVS_BRIDGE_IFACES,
# CONFIG_NEUTRON_LB_INTERFACE_MAPPINGS, CONFIG_NEUTRON_OVS_TUNNEL_IF.
# This is useful for cases when interface names are not same on all
# installation hosts.
CONFIG_USE_SUBNETS=n
CONFIG_VCENTER_CLUSTER_NAME=
www.allitebooks.com
297
# User name to use for Red Hat Subscription Manager's HTTP proxy.
CONFIG_RH_PROXY_USER=
# Access key for the Satellite server; if you intend to use a user
# name and password for Satellite authentication, leave this blank.
CONFIG_SATELLITE_AKEY=
# HTTP proxy to use when connecting to the RHN Satellite server (if
# required).
CONFIG_SATELLITE_PROXY=
CONFIG_SSL_CACERT_FILE=/etc/pki/tls/certs/selfcert.crt
CONFIG_SELFSIGN_CACERT_SUBJECT_MAIL=admin@openstack-csr.cisco.com
www.allitebooks.com
299
# Email address for the Identity service 'admin' user. Defaults to:
CONFIG_KEYSTONE_ADMIN_EMAIL=root@localhost
# User name for the Identity service 'admin' user. Defaults to:
# 'admin'.
CONFIG_KEYSTONE_ADMIN_USERNAME=admin
# User DN for the Identity service LDAP backend. Used to bind to the
# LDAP server if the LDAP server does not allow anonymous
# authentication.
CONFIG_KEYSTONE_LDAP_USER_DN=
# Query scope for the Identity service LDAP backend. Use 'one' for
# onelevel/singleLevel or 'sub' for subtree/wholeSubtree ('base' is
# not actually used by the Identity service and is therefore
# deprecated) (base, one, sub)
CONFIG_KEYSTONE_LDAP_QUERY_SCOPE=one
www.allitebooks.com
301
# User email address attribute for the Identity service LDAP backend.
CONFIG_KEYSTONE_LDAP_USER_MAIL_ATTRIBUTE=
# DN of the group entry to hold enabled LDAP users when using enabled
# emulation.
CONFIG_KEYSTONE_LDAP_USER_ENABLED_EMULATION_DN=
www.allitebooks.com
303
# Specify 'y' if the Identity service LDAP backend should use TLS (n,
# y).
CONFIG_KEYSTONE_LDAP_USE_TLS=n
# database.
CONFIG_GLANCE_DB_PW=e73ab8bc00a94083
# Storage backend for the Image service (controls how the Image
# service stores disk images). Valid options are: file or swift
# (Object Storage). The Object Storage service must be enabled to use
# it as a working backend; otherwise, Packstack falls back to 'file'.
# ['file', 'swift']
CONFIG_GLANCE_BACKEND=file
# Storage backend to use for the Block Storage service; valid options
# are: lvm, gluster, nfs, vmdk, netapp. ['lvm', 'gluster', 'nfs',
# 'vmdk', 'netapp']
CONFIG_CINDER_BACKEND=lvm
# Specify 'y' to create the Block Storage volumes group. That is,
# Packstack creates a raw disk image in /var/lib/cinder, and mounts it
# using a loopback device. This should only be used for testing on a
# proof-of-concept installation of the Block Storage service (a file-
# backed volume group is not suitable for production usage) (y, n).
CONFIG_CINDER_VOLUMES_CREATE=y
www.allitebooks.com
305
# The TCP port to use for communication with the storage system or
# proxy. If not specified, Data ONTAP drivers will use 80 for HTTP and
# 443 for HTTPS; E-Series will use 8080 for HTTP and 8443 for HTTPS.
# Defaults to: 80.
CONFIG_CINDER_NETAPP_SERVER_PORT=80
# Time period (in minutes) that is allowed to elapse after the image
306 Appendix A: Sample Answer File for Packstack
www.allitebooks.com
307
# Password for the NetApp E-Series storage array. Defaults to: ''.
CONFIG_CINDER_NETAPP_SA_PASSWORD=
# The storage family type used on the storage system; valid values
# are ontap_cluster for clustered Data ONTAP. Defaults to:
# 'ontap_cluster'.
CONFIG_MANILA_NETAPP_STORAGE_FAMILY=ontap_cluster
# The TCP port to use for communication with the storage system or
# proxy server. If not specified, Data ONTAP drivers will use 80 for
# HTTP and 443 for HTTPS. Defaults to: '443'.
CONFIG_MANILA_NETAPP_SERVER_PORT=443
www.allitebooks.com
309
# Location of disk image for Manila service instance. Defaults to: '
CONFIG_MANILA_SERVICE_IMAGE_LOCATION=https://www.dropbox.com/s/vi5oeh10q1qkckh/
ubuntu_1204_nfs_cifs.qcow2
# Network mask that will be used. Can be either decimal like '24' or
# binary like '255.255.255.0'. Required. Defaults to: ''.
CONFIG_MANILA_NETWORK_STANDALONE_NETMASK=
310 Appendix A: Sample Answer File for Packstack
# Protocol used for instance migration. Valid options are: tcp and
# ssh. Note that by default, the Compute user is created with the
# /sbin/nologin shell so that the SSH protocol will not work. To make
# the SSH protocol work, you must configure the Compute user on
www.allitebooks.com
311
# The name of the Open vSwitch bridge (or empty for linuxbridge) for
# the OpenStack Networking L3 agent to use for external traffic.
# Specify 'provider' if you intend to use a provider network to handle
# external traffic.
CONFIG_NEUTRON_L3_EXT_BRIDGE=br-ex
www.allitebooks.com
313
# eth1,physnet2:br-eth2,physnet3:br-eth3
CONFIG_NEUTRON_OVS_BRIDGE_MAPPINGS=physnet1:br-data
CONFIG_HORIZON_SSL_CACERT=
www.allitebooks.com
315
# device; Packstack does not create the filesystem, you must do this
# first). If left empty, Packstack creates a loopback device for test
# setup.
CONFIG_SWIFT_STORAGES=
# Specify 'y' to provision for demo usage and testing (y, n).
CONFIG_PROVISION_DEMO=y
www.allitebooks.com
317
CONFIG_PROVISION_TEMPEST_REPO_URI=https://github.com/openstack/tempest.git
www.allitebooks.com
Index
attach-interface command (virsh), 90
A automation, 247-248
BDEO tool, 248-249
A9 processor, 1
management, 247
active/active flow distribution to NSO tool, 249-251
cloudburst, redirection access NFV orchestration with OpenStack,
model, 193 252-253, 260-261, 264-266
adoption, Enterprise cloud, 29-30 versus orchestration, 247
algorithms, scheduling, 60-62 provisioning, 247
Amazon web services deployment, CSR, availability zones, VPC, 214
211-215 AVC (Application Visibility and Control),
Amazon web services deployment, CSR CSR 1000V, 52-53
1000V, 216-222 AVS (Application Virtual Switch), 270
application-centric infrastructure AWS (Amazon web services) deployment
(SDN), 224 CSR, 211-215
Application Virtual Switch (AVS), 270 CSR 1000V, 216-222
architecture, CSR 1000V, troubleshooting,
271-272
ASR (Aggregation Service Router) 1000
B
router, 41, 96-97 BadUidbSubIdx drop type (IOS), 285
architectural elements, 95 Basic Input/Output System (BIOS), 67
ESPs (embedded service processors),
BDEO tool, 248-249
42-43
BIOS (Basic Input/Output System), 67
RP (route processor), 42
SIP (SPA interface processor), 43 BIOS settings, hosts, 276
ASR (Aggregation Service Router) block-based storage, 3
1001, 97 boot process (IOS), 66-67
logical architecture, 97 BPG route reflector, CSR, 155-157
virtualizing into CSR 1000V, 98 hierarchical use, 157-162
ATM (Asynchronous Transfer Mode), 13 BqsOor drop type (IOS), 285
attach-device command (virsh), 90 branch design, CSR 1000V, planning,
attach-disk command (virsh), 90 162-168
320 caching memory, Linux
www.allitebooks.com
CSR 1000V 321
www.allitebooks.com
ESXi 323
www.allitebooks.com
IPsec VPNs 325
www.allitebooks.com
networks 327
www.allitebooks.com
public cloud deployment 329
www.allitebooks.com
swap files 331
www.allitebooks.com
zones, connectivity, multitenant data center with CSR 1000V 333