AWS SysOps Cookbook: Practical recipes to build, automate, and manage your AWS-based cloud environments, 2nd Edition

AWS SysOps Cookbook

Second Edition

Practical recipes to build, automate, and manage your

AWS-based cloud environments

Eric Z. Beard

Rowan Udell

Lucas Chan

BIRMINGHAM - MUMBAI

AWS SysOps Cookbook Second Edition

Copyright © 2019 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Commissioning Editor: Karan Sadawana

Acquisition Editor: Heramb Bhavsar

Content Development Editor: Alokita Amanna

Technical Editor: Dinesh Pawar

Copy Editor: Safis Editing

Language Support Editor: Rahul Dsouza

Project Coordinator: Vaidehi Sawant

Proofreader: Safis Editing

Indexer: Rekha Nair

Production Designer: Deepika Naik

First published: April 2017

Second edition: September 2019

Production reference: 1260919

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-83855-018-9

www.packt.com

This book is dedicated to the Horde, an extended team of partner solutions architects at AWS. They go above and beyond to work with our emerging partners to help them grow and succeed on AWS. I count everyone in the group among my mentors. They come from a wide array of technical backgrounds and bring an impressive amount of brainpower and creativity to the job. It's a humbling group to work with, and I do my best to try and learn from all of them.

–  Eric Z. Beard

Packt.com

Subscribe to our online digital library for full access to over 7,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.

Why subscribe?

Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals

Improve your learning with Skill Plans built especially for you

Get a free eBook or video every month

Fully searchable for easy access to vital information

Copy and paste, print, and bookmark content

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.packt.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details.

At www.packt.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks. 

Contributors

About the authors

Eric Z. Beard, a former United States Marine, has nearly two decades of experience in technology, leading diverse DevOps and solutions architecture teams. Eric is currently a manager at Amazon Web Services in Seattle, Washington, and holds nine AWS certifications.

First I have to thank my wife, Kate, for being so patient with me while I worked on this book over many nights and weekends. Without her support, I can't imagine how I'd be successful in any of my endeavors. I would also like to thank Rowan Udell and Lucas Chan, authors of the first edition of the book. They gave me a great foundation to work from, and much of the content they created is still in this edition, mostly intact with minor edits to reflect changes made by AWS since that printing. And a big shout out to the people on the service teams at AWS who work so hard to keep innovating on behalf of customers.

Rowan Udell has been working in development and operations for 15 years. His travels have seen him work in start-ups and enterprises in the finance, education, and web industries in both Australia and Canada. He currently works as a Technical Director at Versent, an AWS Premier Consulting Partner, working with teams building cloud-native products on AWS. He specializes in serverless applications and architectures on AWS, and contributes actively in the AWS and serverless communities.

Lucas Chan has been working in tech since 1995 in a variety of development, systems admin, and DevOps roles. He is currently a senior consultant and engineer at Versent and was a technical director at Stax. He's been running production workloads on AWS for over 10 years. He's also a member of the APAC AWS warriors program and holds all five of the available AWS certifications.

About the reviewers

Ian Scofield, a former United States Army Officer, has a background in technology and communications. He is a Solutions Architect Manager at AWS and works with his team to build internal applications. He lives in Austin, Texas with his wife, an adorable labradoodle, and a grumpy cat.

Gajanan Chandgadkar has more than 13 years' IT experience. He has spent over 6 years in the USA, assisting large enterprises in architecting, migrating, and deploying applications in AWS. He's been running production workloads on AWS for over 6 years. He is an AWS certified solutions architect professional and a certified DevOps professional with 7+ certifications in trending technologies. Gajanan is also a technology enthusiast who has an extended interest and experience in a variety of topics, including application development, container technology, and continuous delivery.

Currently, he is working with a product company as a DevOps expert, having worked with the Wipro Limited in the past.

Packt is searching for authors like you

If you're interested in becoming an author for Packt, please visit authors.packtpub.com and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea.

Table of Contents

Title Page

Copyright and Credits

AWS SysOps Cookbook Second Edition

Dedication

About Packt

Why subscribe?

Contributors

About the authors

About the reviewers

Packt is searching for authors like you

Preface

Who this book is for

What this book covers

To get the most out of this book

Download the example code files

Download the color images

Conventions used

Sections

Getting ready

How to do it…

How it works…

There's more…

See also

Get in touch

Reviews

AWS Fundamentals

Signing up for an AWS account

How to do it…

How it works…

There's more…

See also

Understanding AWS's global infrastructure

Regions and availability zones

Global resources

Using the web console

The menu bar

AWS logo

Services

Resource Groups

Pins

Alerts

Account

Region and support

Learning the basics of AWS CloudFormation

What is CloudFormation?

Why is CloudFormation important?

Infrastructure as Code (IaC)

The layer cake

CloudFormation templates

YAML versus JSON

A closer look at CloudFormation templates

Parameters

Resources

Dependencies and ordering

Functions

Fn::Join

Fn::Sub

Conditionals

Permissions and service roles

Cross-stack references

Updating resources

Changesets

Other things to know

Name collisions

Rollback

Limits

Circular dependencies

Credentials

Stack policies

Using the command-line interface (CLI)

Installation

Upgrade

Configuration

Default profile

Named profiles

Environment variables

Instance roles

Usage

Commands

Subcommands

Options

Output

JSON

Table

Text

Querying

Generating a CLI skeleton

Input

Output

Pagination

Autocomplete

There's more…

See also

Account Setup and Management

Setting up an automated landing zone with AWS Control Tower

How to do it…

How it works…

Accounts

There's more…

See also

Setting up a master account with AWS Organizations

How to do it…

How it works…

There's more…

Using the CLI

See also

Creating a member account

Getting ready

How to do it…

How it works…

There's more…

Accessing the member account

Service Control Policies

Root credentials

Deleting accounts

See also

Inviting an account

Getting ready

How to do it…

How it works…

There's more…

Removing accounts

Consolidated billing

See also

Managing your accounts

Getting ready

How to do it…

Getting the root ID for your organization

Creating an OU

Getting the ID of an OU

Adding an account to an OU

Removing an account from an OU

Deleting an OU

How it works…

There's more…

See also

Adding a Service Control Policy (SCP)

Getting ready

How to do it…

How it works…

There's more…

See also

Setting up consolidated billing

How to do it…

How it works…

There's more…

Credits

Support charges

See also

AWS Storage and Content Delivery

Setting up a secure Amazon S3 bucket

How to do it…

Using the web console to create a bucket with versioning enabled

Using the CLI to create a bucket with cross-region replication enabled

Using CloudFormation to create a bucket

How it works…

There's more…

Athena

S3 Select

See alo

Hosting a static website

How to do it…

Creating S3 buckets and hosting content

Creating a hosted zone

Creating DNS records

Uploading website content

How it works…

There's more…

Delegating your domain to AWS

Cross-origin resource sharing (CORS)

See also

Caching a website with CloudFront

Getting ready

About dynamic content

Configuring CloudFront distributions

How to do it…

How it works...

Working with network storage provided by EFS

Getting ready

How to do it…

How it works…

There's more…

Amazon FSx for Windows File Server

Getting ready

How to do it…

How it works...

Backing up data for compliance

How to do it…

How it works…

There's more...

AWS Compute

Creating a key pair

Getting ready

How to do it…

How it works…

Launching an instance

Getting ready

How to do it…

How it works…

There's more…

See also

Attaching storage

Getting ready

How to do it…

How it works…

See also

Autoscaling an application server

Getting ready

How to do it…

How it works…

Scaling policies

Alarms

Creating security groups

Getting ready

How to do it…

There's more…

Differences from traditional firewalls

Creating a load balancer

How to do it…

How it works…

There's more…

HTTPS/SSL

Path-based routing

Using AWS Systems Manager to log in to instances from the console

Getting ready…

How to do it…

How it works…

There's more…

Creating serverless functions with AWS Lambda

How to do it…

How it works…

There's more…

See also

Monitoring the Infrastructure

AWS Trusted Advisor

How to do it…

How it works…

There's more…

Resource tags

How to do it…

How it works…

AWS CloudWatch

Getting ready

How to do it…

How it works…

Widget types

Billing alerts

Getting ready

How to do it…

How it works…

The ELK stack

How to do it…

How it works…

There's more...

AWS CloudTrail

How to do it…

How it works…

There's more…

Network logging and troubleshooting

Getting ready

How to do it…

How it works…

There's more…

Log format

Updates

Omissions

See also

Managing AWS Databases

Creating an RDS database with automatic failover

Getting ready

How to do it...

How it works...

There's more...

Creating an RDS database read replica

Getting ready

How to do it...

How it works...

There's more...

Promoting an RDS read replica to master

Getting ready

How to do it...

How it works...

Creating a one-time RDS database backup

Getting ready

How to do it...

How it works...

Restoring an RDS database from a snapshot

Getting ready

How to do it...

How it works...

There's more...

Managing Amazon Aurora databases

How to do it...

How it works...

There's more...

Managing Amazon Neptune graph databases

How to do it...

How it works...

Create a DynamoDB table with a global secondary index

How to do it...

How it works...

Calculating Amazon DynamoDB capacity

Getting ready

How to do it...

How it works...

There's more...

Burst capacity

Metrics

Eventually consistent reads

See also

AWS Networking Essentials

Creating a VPC and subnets

Getting ready

How to do it...

How it works...

There's more...

See also

Managing a transit gateway

Getting ready

How to do it...

How it works...

Creating a Virtual Private Network (VPN)

How to do it...

How it works...

There's more...

BGP

ASN

Setting up NAT gateways

Getting ready

How to do it...

How it works...

See also

Managing domains with Route 53

Getting ready

How to do it...

How it works...

There's more...

See also

AWS Account Security and Identity

Administering users with IAM

Getting ready

How to do it...

There's more...

See also

Deploying Simple Active Directory service

Getting ready

How to do it...

How it works...

There's more...

See also

Creating instance roles

How to do it...

How it works...

There's more...

Using cross-account roles

Getting ready

How to do it...

How it works...

There's more...

AWS CLI profiles

Storing secrets

How to do it...

How it works...

There's more...

Protecting applications from DDoS

How to do it...

How it works...

There's more...

Configuring AWS WAF

How to do it...

How it works...

There's more...

Setting up intrusion detection

How to do it...

How it works...

There's more...

Managing Costs

Estimating costs with the Simple Monthly Calculator

Getting ready

How to do it...

How it works...

See also

Estimating costs with the Total Cost of Ownership Calculator

Getting ready

How to do it...

How it works...

There's more...

See also

Estimating CloudFormation template costs

Getting ready

How to do it...

How it works...

See also

Reducing costs by purchasing reserved instances

Getting ready

How to do it...

How it works...

There's more...

Advanced AWS CloudFormation

Creating and populating an S3 bucket with custom resources

How to do it...

How it works...

There's more...

Using a macro to create an S3 bucket for CloudTrail logs

How to do it...

How it works...

There's more...

See also

Using mappings to specify regional AMI IDs

How to do it...

How it works...

There's more...

See also

Using StackSets to deploy resources to multiple regions

Getting ready

How to do it...

How it works...

There's more...

See also

Detecting resource drift from templates with drift detection

How to do it...

How it works...

There's more...

Unsupported resources and properties

Using the CLI

See also

AWS Well-Architected Framework

Understanding the five pillars of the Well-Architected Framework

Security

Operational excellence

Performance efficiency

Reliability

Cost optimization

Conducting a technology baseline review self-assessment

How to do it...

How it works...

There's more...

Using the Well-Architected Tool to evaluate a production workload

How to do it...

How it works...

There's more...

Working with Business Applications

Creating a place for employees to share files with WorkDocs

How to do it...

How it works...

There's more...

Hosting desktops in the cloud and allowing users to connect remotely using WorkSpaces

How to do it...

How it works...

There's more...

Giving your users a place to chat and conduct video calls with Chime

How to do it...

How it works...

There's more...

Exploring the use of Alexa for Business

How to do it...

How it works...

There's more...

Hosting your company's email with WorkMail

How to do it...

How it works...

There's more...

AWS Partner Solutions

Creating machine images with Hashicorp's Packer

Getting ready

How to do it...

How it works...

Template

Validating the template

Building the AMI

There's more...

Debugging

Orphaned resources

Deregistering AMIs

Other platforms

Monitoring and optimizing your AWS account with nOps

Getting ready

How to do it...

How it works...

There's more...

Using IOPipe to instrument your lambda functions

How to do it...

How it works...

Metrics dashboards

Alerting

Profiling

Labels and search

There's more...

Other Books You May Enjoy

Leave a review - let other readers know what you think

Preface

The AWS platform is developing at a rapid rate and is being increasingly adopted across all industries and sectors. As the saying goes, friends don't let friends build data centers. No matter how you look at it, the model of pay-as-you-go computing, networking, and storage is here to stay. It's also becoming increasingly hard to argue against standing on the shoulders of giants, especially when you look at the rate at which features and enhancements are being added to the AWS platform compared to what you'd typically get out of other cloud providers or a so-called private cloud.

We work with many technical professionals who are highly knowledgeable in their domain, but who are often completely new to the AWS platform. Alternatively, they may be familiar with AWS, but are new to automation and infrastructure code practices.

We wanted to write a book for these people.

This book is intended to kick start your journey on AWS by providing recipes, patterns, and best practices across the areas we are often asked to help with on our consulting engagements. All the recipes and recommendations contained in this book are based on our personal experiences and observations from our time helping customers on the AWS platform.

CloudFormation is the AWS-native method for automating the (repeatable and reliable) deployment of AWS resources, and we use it extensively throughout this book. The recipes that follow will help you get well acquainted with CloudFormation and you'll soon be on your way to customizing and building your own templates. With so much power at your fingertips, there's a lot of potential for finding yourself in a rabbit hole. This book aims to steer you in the right direction and help you adopt the platform in a sustainable and maintainable way.

Who this book is for

This book is for anyone with a technical background who is interested in using AWS, either for moving existing workloads or deploying entirely new applications. Those who want to learn CloudFormation will also find this book useful.

What this book covers

Chapter 1, AWS Fundamentals, provides an overview of infrastructure as code, CloudFormation, and the AWS CLI tools.

Chapter 2, Account Setup and Management, includes everything you need to know to manage your accounts and get started with AWS organizations.

Chapter 3, AWS Storage and Content Delivery, shows how to back up your data and serve file objects to your users.

Chapter 4, AWS Compute, dives deep into how to run VMs (EC2 instances) on AWS, how to autoscale them, and how to create and manage load balancers.

Chapter 5, Monitoring the Infrastructure, provides an overview of how to audit your account and monitor your infrastructure.

Chapter 6, Managing AWS Databases, shows how to create, manage, and scale databases on the AWS platform.

Chapter 7, AWS Networking Essentials, introduces private networks, routing, and DNS.

Chapter 8, AWS Account Security and Identity, offers advice and practical solutions for managing identities and role-based access.

Chapter 9, Managing Costs, provides an overview of how to estimate your spend on the AWS platform as well as how to reduce your costs by purchasing reserved instance capacity.

Chapter 10, Advanced AWS CloudFormation, explains how to pursue plans that will enable you to customize the behavior of CloudFormation, and apply your scripts over various regions and accounts.

Chapter 11, AWS Well-Architected Framework, introduces the AWS Well-Architected Framework, which was created by AWS following years spent working with clients, to enable them to build secure, highly performant, and reliable systems.

Chapter 12, Working with Business Applications, enables you to gain proficiency with these services so that you can supplant costly on-premises assets with cloud-based options.

Appendix, AWS Partner Solutions, presents a few recipes covering products offered by members of the AWS Partner Network (APN).

To get the most out of this book

The recipes in this book show you how to deploy a wide variety of resources on AWS, so you'll need at least one AWS account with full administrative access. You'll also need a text editor to edit YAML/JSON CloudFormation templates and the AWS CLI tools, which are supported on common operating systems (macOS/Linux/Windows).

Download the example code files

You can download the example code files for this book from your account at www.packt.com. If you purchased this book elsewhere, you can visit www.packtpub.com/support and register to have the files emailed directly to you.

You can download the code files by following these steps:

Log in or register at www.packt.com.

Select the Support tab.

Click on Code Downloads.

Enter the name of the book in the Search box and follow the onscreen instructions.

Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:

WinRAR/7-Zip for Windows

Zipeg/iZip/UnRarX for Mac

7-Zip/PeaZip for Linux

The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/AWS-SysOps-Cookbook-Second-Edition. In case there's an update to the code, it will be updated on the existing GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Download the color images

We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: http://www.packtpub.com/sites/default/files/downloads/9781838550189_ColorImages.pdf.

Conventions used

There are a number of text conventions used throughout this book.

CodeInText: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: Next, we define Resources parameters.

A block of code is set as follows:

Resources:

ExampleEC2Instance:

Type: AWS:EC2::Instance

Any command-line input or output is written as follows:

pip install --upgrade awscli

Bold: Indicates a new term, an important word, or words that you see on screen. For example, words in menus or dialog boxes appear in the text like this. Here is an example: Expand the Create individual IAM users section and click Manage Users.

Warnings or important notes appear like this.

Tips and tricks appear like this.

Sections

In this book, you will find several headings that appear frequently (Getting ready, How to do it..., How it works..., There's more..., and See also).

To give clear instructions on how to complete a recipe, use these sections as follows.

Getting ready

This section tells you what to expect in the recipe and describes how to set up any software or any preliminary settings required for the recipe.

How to do it…

This section contains the steps required to follow the recipe.

How it works…

This section usually consists of a detailed explanation of what happened in the previous section.

There's more…

This section consists of additional information about the recipe in order to make you more knowledgeable about the recipe.

See also

This section provides helpful links to other useful information for the recipe.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, mention the book title in the subject of your message and email us at [email protected].

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata, selecting your book, clicking on the Errata Submission Form link, and entering the details.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in, and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Reviews

Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions, we at Packt can understand what you think about our products, and our authors can see your feedback on their book. Thank you!

For more information about Packt, please visit packt.com.

AWS Fundamentals

Amazon Web Services (AWS) was the pioneer in cloud computing, launching its offering over a decade ago, and it continues to rapidly introduce new services and features based on customer demand. AWS was developed by Amazon.com when the company decided to turn its expertise in building large-scale, reliable, and cost-efficient internet systems into a product that could be used by customers to host their own sites and services.

At the time of writing, AWS has 136 services listed on its web console, ranging from foundational services such as Identity and Access Management (IAM) and Elastic Compute Cloud (EC2) to high-level machine learning services such as Rekognition. The breadth and depth of the services that are available make it possible to implement almost any idea quickly and efficiently – your imagination is the only true limit to what you can do. But all of those services mean that you – as a developer, systems administrator, or solutions architect – have a lot to learn!

Luckily, we are here to help, and if you stick with us throughout the next 12 chapters, you will have a solid foundation for establishing yourself as an AWS expert.

In this chapter, we will cover the following topics:

Signing up for an AWS account

Understanding AWS's global infrastructure

Using the web console

Learning the basics of AWS CloudFormation

Using the AWS CLI

Signing up for an AWS account

To follow along with the recipes in this book, you will need to set up an AWS account. Follow all of these steps to learn how to create an account that you will securely access with an IAM user and a Multi-Factor Authentication (MFA) device.

How to do it…

Follow these steps to create an AWS account:

Create an account at https://aws.amazon.com/ by clicking on the Sign Up button and entering your details:

Creating an AWS account

Even though we will be taking advantage of the free tier wherever possible, you will need a valid credit card to complete the signup process. Go to https://aws.amazon.com/free/ for more information. Note that the free tier only applies for the first year of your account's lifetime.

Before we get started using that shiny new account, let's go over some best practices regarding basic account security. The very first thing you should do as the owner of an AWS account is enable MFA on the root login:

Identity and Access Management

Protect your logins with MFA. Check out this article by Okta on why MFA is a good idea: 

https://www.okta.com/identity-101/why-mfa-is-everywhere/.

As you can see, when you first visit the IAM console, AWS recommends that you Activate MFA as the next step to improve your security status. Expand the Activate MFAsection and click through it to get to your security credentials screen:

Managing