Hybrid Cloud for Architects: Build robust hybrid cloud solutions using AWS and OpenStack

Hybrid Cloud for Architects

Build robust hybrid cloud solutions using AWS and OpenStack

Alok Shrivastwa

BIRMINGHAM - MUMBAI

Hybrid Cloud for Architects

Copyright © 2018 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Reviewers: David Duncan, Ganesh Raja

Commissioning Editor: Gebin George

Acquisition Editor: Rohit Rajkumar

Content Development Editor: Nithin Varghese

Technical Editor: Mohit Hassija

Copy Editors: Safis Editing, Laxmi Subramanian

Project Coordinator: Virginia Dias

Proofreader: Safis Editing

Indexer: Rekha Nair

Graphics: Tom Scaria

Production Coordinator: Nilesh Mohite

First published: February 2018

Production reference: 1220218

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-78862-351-3

www.packtpub.com

Detailed installation steps (software-wise)

The steps should be listed in a way that it prepares the system environment to be able to test the codes of the book.

Software A:

Step 1

Step 2

Software B

Step a

Step b

Step c

mapt.io

Mapt is an online digital library that gives you full access to over 5,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.

Why subscribe?

Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals

Improve your learning with Skill Plans built especially for you

Get a free eBook or video every month

Mapt is fully searchable

Copy and paste, print, and bookmark content

PacktPub.com

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.

Contributors

About the author

Alok Shrivastwa is a technologist from India, currently working as the director of special projects for Microland in the CMD's office. He currently runs special projects on cloud technologies. Having worked at multiple enterprises of varied sizes, designing and implementing solutions, public and private clouds, and integrations, he has created a myriad number of tools and intellectual properties in the operationalization of emerging technologies. He has authored two books on OpenStack alongside several white papers and blogs on technology, in addition to writing poems in Hindi.

We as humans need contrast, without which we cannot perceive. Because of this, to show something in a good light, something has to be made the villain. This book is about being pragmatic when looking at the cloud. I thank God for the perspective, and my family—my mother, father, sisters and my niece, Aarya—who helped me see it. I am thankful to each and every person who I meet and learn from.

About the reviewer

David Duncan is a partner solutions architect at Amazon Web Services who specializes in enabling open source platform partners. He focuses on enabling Linux support on Amazon EC2, cloud native deployments, and hybrid cloud workloads with operating system partners such as Red Hat OpenShift, SUSE Cloud Application Platform, and the Canonical distribution of Kubernetes. David is a coauthor of the book AWS Quick Start for Red Hat OpenShift.

Packt is searching for authors like you

If you're interested in becoming an author for Packt, please visit authors.packtpub.com and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea.

Table of Contents

Title Page

Copyright and Credits

Hybrid Cloud for Architects

Software Hardware List

Packt Upsell

Why subscribe?

PacktPub.com

Contributors

About the author

About the reviewer

Packt is searching for authors like you

Preface

Who this book is for

What this book covers

To get the most out of this book

Download the example code files

Download the color images

Conventions used

Get in touch

Reviews

Introducing Hybrid Cloud

The cloud's demographics

Based on abstraction

Service down clouds 

Infrastructure up clouds 

Differentiating service down and infrastructure up clouds 

Based on services offered

Based on consumers of the services 

Choosing different cloud combinations

Summary

Hybrid Cloud – Why Does It Matter?

What does the world say? 

Pure-play public cloud strategy 

Public cloud benefits

Need for agility

Ability to experiment without upfront cost

Reducing operational overheads

Ability to consume enhanced services

Shortcomings of a public cloud

Cost 

Control/customizability

Compliance 

Fear of lock-in

Hybrid cloud case study

Summary – maximizing benefits

Hybrid Cloud Building Blocks

The story of a web application 

Transport level 

Case 1 – without a proxy

Case 2 – with a proxy 

Application level 

Web tier 

Application tier 

Database tier 

Putting it all together

Use cases of a hybrid cloud 

Isolated use case 

Distributed use case 

Co-Existent use case 

Cloud bursting 

Using cognitive services 

Supporting application use cases 

Backup and disaster recovery in the cloud

Decoupling the tiers

Case in point – architecture of OpenStack

Services to enable a hybrid cloud 

Network connectivity 

DNS service 

Public cloud services for hybrid deployment

Amazon Web Services (AWS)

Storage gateway

Direct connect

Route 53

Amazon EC2 run command

VMware cloud on AWS 

Microsoft Azure

Azure Stack

Azure Site Recovery (ASR)

Azure Traffic Manager

Summary – setting up hybrid cloud

Architecting the Underpinning Services

Networking

Underlay network

LAN architecture

WAN architecture

Overlay networking

GRE

VXLAN

Virtual Private Network (VPN)

Encrypting data using IPSec and SSL – concepts

IPSec VPN

SSL VPN

MPLS connectivity – direct connect

Routing table

Domain Name System (DNS)

How does DNS work?

Global load balancing

Identity and Access Management (IAM) 

Identity Federation 

Multi-Factor Authentication (MFA)

Application components

Global databases 

Using Cockroach DB in a hybrid cloud environment 

Database log shipping

Choosing the right components

Network connectivity 

DNS services 

IAM and Active Directory 

Conclusion 

Hybrid Cloud Deployment – Architecture and Preparation

Getting started with the public cloud – AWS

AWS terminology 

Account

Region 

Availability zones (AZ)

Virtual private cloud (VPC)

AWS services 

Architecting the AWS environment 

AWS account design

VPC design 

Designing an AWS environment 

Connectivity to the private cloud

Setting up a public cloud – AWS

Creating an account in AWS

Creating a VPC and subnets

Creating the IGW and VGW

Setting up AWS API access 

Setting up the private cloud 

Basics of designing an OpenStack environment

Choosing an OpenStack distribution 

Choosing the deployment method

Installing DevStack 

Configuring DevStack to enable Heat

Summary

Building a Traditional CMP-Based Hybrid Cloud

Supporting applications use case

Traditional operations 

Modern outlook

Using the AWS storage gateway

File gateway

Volume gateways

Tape gateway 

Isolated/distributed application use case

General architecture of CMP

ManageIQ

Installing ManageIQ

Preparing the host environment 

Containerization basics

Understanding and installing Docker

Installing a ManageIQ container

Configuring ManageIQ to connect to AWS and OpenStack 

Adding a new AWS EC2 provider 

Adding our OpenStack endpoint 

Provisioning virtual machines using ManageIQ 

Creating a catalog

Creating a Service Dialog

Creating a catalog item and catalog

Testing the catalog

Policies and user authentication

Creating cloud images

In conclusion – architecting with a CMP

Summary

Building a Containerized Hybrid Cloud

Evolving to containers

Container networking 

None – no networking

Bridge networking

Host networking 

Overlay networking 

Underlay networking 

Container orchestration engine 

Kubernetes architecture 

Basic concepts in Kubernetes

Pod

Controllers

Service 

Volumes

Namespaces

Kubernetes deployment

Introduction to Juju 

Installing the Juju client and bootstrapping clouds

Bootstrapping an AWS Cloud 

Bootstrapping an OpenStack Cloud 

Accessing the Juju controller using a GUI

Deploying Kubernetes with Juju

Deploying a second instance of Kubernetes 

Connecting to the Kubernetes clusters

Federation using Kubernetes

Reasons for consideration 

Application migration – avoiding vendor lock-in

Enforce policies 

High availability and application upgrades

Cloud bursting 

Federation challenges

Implementing a Kubernetes federation

Step 1 – setting up the federation controller 

Step 2 – combining the Kubernetes configuration (optional)

Step 3 – creating the federation 

Creating the DNS provider 

Initializing the federation 

Summary 

Using PreBuilt Hybrid Cloud Solutions

Azure Stack 

Getting the Azure Stack

OpenStack Omni 

Installing OpenStack Omni on DevStack

Removing the DevStack instance

Modifying the local.conf file

Running DevStack 

vCloud Air

Using the different hybrid cloud solutions 

Summary

DevOps in the Hybrid Cloud

The development cycle and DevOps 

The traditional development stages 

Merging the different teams

Creating the infrastructure

Configuring the infrastructure

Templatize

DevOps or NoOps

IaaC with Terraform 

Installing Terraform 

Configuring and using Terraform

Configuration management using Ansible

Installing Ansible

Configuring Ansible and a sample playbook 

Summary

Monitoring the Hybrid Cloud

The traditional concepts in monitoring

Availability monitoring 

ICMP monitoring 

TCP/UDP monitoring 

Enhanced monitoring 

SNMP-based availability monitoring

Performance monitoring 

SNMP monitoring

WMI monitoring and custom agent monitoring

Monitoring the hybrid cloud

Prometheus

The implementation architecture of Prometheus

Installing Prometheus

Downloading Prometheus

Setting up directories

Setting up startup script

Setting up node exporter

Configuring Prometheus

Grafana

Installing Grafana

Configuring Grafana to use Prometheus

Summary

Security in a Hybrid Cloud

Components of security

The CIA triad

Confidentiality

Integrity

Availability

Tools to protect against the breaches

IAM systems

Data encryption in rest and in motion

Network perimeter security

Firewalls

IDS/IPS

Proxies

Host controls

High availability and disaster recovery

Detection and analytics mechanism

Minimizing shared infrastructure

Compliance standards and controls

HIPAA compliance standards

Administrative controls

Physical controls

Technical controls

Security controls consideration in hybrid cloud

Common controls

Implementing the controls on AWS – public cloud

Security – shared responsibility model

Implementing the controls in private cloud

Security – best practices

Implementing a CMDB/asset list

User accounts and authentication

Provisioning and postprovisioning controls

Networks 

Other practices

Summary

Other Books You May Enjoy

Leave a review - let other readers know what you think

Preface

The book takes us on a journey of architecting, building, and operating a hybrid cloud while taking a very pragmatic approach towards it. The book starts by defining the different demographics of the cloud and the different use cases that need to be solved. It then introduces two modes of building a hybrid cloud, with the CMP and the other with containers—along with the use cases that each of them addresses. The book finally drops into operational mode with topics such as DevOps, monitoring, and security considerations in the hybrid cloud.

Who this book is for

This book is targeted at cloud architects, cloud solution providers, DevOps engineers, or any working stakeholder who wants to learn about the hybrid cloud architecture. A basic understanding of public and private clouds is desirable.

What this book covers

Chapter 1, Introducing Hybrid Cloud, deals with the definitions and demographics of the cloud, the differences between service down and infrastructure up cloud, and its examples.

Chapter 2, Hybrid Cloud – Why Does It Matter?, starts with adoption statistics of the hybrid cloud and moves on to drivers for cloud adoption, public cloud benefits, and its shortcomings. Finally, we introduce a case for hybrid cloud and how to maximize the benefits using the best of both worlds. 

Chapter 3, Hybrid Cloud Building Blocks, introduces the building blocks of the hybrid cloud using an example of a web application, use cases that potentially will need a hybrid cloud, making applications suitable for a hybrid cloud using decoupling, and services that are used to enable the hybrid cloud.

Chapter 4, Architecting the Underpinning Services, covers the concepts of networking, DNS systems, IAM systems, application components, and choosing the appropriate components for the use with a hybrid cloud.

Chapter 5, Hybrid Cloud Deployment – Architecture and Preparation, covers the concepts of AWS, architecting an AWS environment, the basic design of an OpenStack environment, setting up a DevStack, and connectivity between the cloud environments. 

Chapter 6, Building a Traditional CMP-Based Hybrid Cloud, starts with AWS's storage gateway and use cases in the hybrid cloud scenario, the concepts of CMP, setting up Docker, and running a ManageIQ container in Docker. 

Chapter 7, Building a Containerized Hybrid Cloud, introduces the basics of container orchestration platforms, an introduction to Kubernetes, deploying Kubernetes using Juju, and closes with using the kubefed project to federate a hybrid cloud based on Kubernetes. 

Chapter 8, Using Prebuilt Hybrid Cloud Solution, introduces products that are available from different providers, including AzureStack and Project Omni. 

Chapter 9, DevOps in the Hybrid Cloud, deals with the traditional development cycle and the steps involved, along with the concepts of DevOps and NoOps. We look at the introduction to IaaC, templatizer, and configuration management systems and their roles in the development cycle. We take an example of Terraform and its deployment with a sample to solidify the concepts of IaaC. Also, deploy Ansible and a sample to solidify the concepts of configuration management.

Chapter 10, Monitoring the Hybrid Cloud, introduces the basics of monitoring, along with Prometheus and Grafana, to help us monitor the hybrid cloud. 

Chapter 11, Security in a Hybrid Cloud, starts with the concepts of security and compliance standards, and moves on to taking HIPAA as an example to elucidate some of the best practices that need to be used. 

To get the most out of this book

While a simple reading of the book will impart the different architectural and cloud concepts to the reader, in order to follow along, ensure that you have the following:

An internet connection to download the software.

A Ubuntu 16.04 machine to act as the management system.

A fully functioning OpenStack deployment or a Ubuntu 16.04 machine to run DevStack.

AWS user account—if you don't have the user account, ensure that you have your credit card ready in order to open a free account. (Remember that while we have taken care to use the free-tier systems in AWS, make sure you use the appropriate instance sizes and AMI IDs if you are creating the environment in a different region). 

Download the example code files

You can download the example code files for this book from your account at www.packtpub.com. If you purchased this book elsewhere, you can visit www.packtpub.com/support and register to have the files emailed directly to you.

You can download the code files by following these steps:

Log in or register at www.packtpub.com.

Select the SUPPORT tab.

Click on Code Downloads & Errata.

Enter the name of the book in the Search box and follow the onscreen instructions.

Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:

WinRAR/7-Zip for Windows

Zipeg/iZip/UnRarX for Mac

7-Zip/PeaZip for Linux

The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Hybrid-Cloud-for-Architects. In case there's an update to the code, it will be updated on the existing GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Download the color images

We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: https://www.packtpub.com/sites/default/files/downloads/HybridCloudforArchitects_ColorImages.pdf.

Conventions used

There are a number of text conventions used throughout this book.

CodeInText: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: Default values are port 80 for HTTP, port 443 for HTTPS.

A block of code is set as follows:

provider aws {

access_key =

secret_key =

region = us-east-1

}

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

provider aws {

access_key =

secret_key =

region = us-east-1

}

Any command-line input or output is written as follows:

sudo cp terraform /usr/local/bin

Bold: Indicates a new term, an important word, or words that you see onscreen. For example, words in menus or dialog boxes appear in the text like this. Here is an example: "Gartner introduced the Bimodal IT concept and coined