Kubernetes Security - Michael Hausenblas, Liz Rice PDF
Kubernetes Security - Michael Hausenblas, Liz Rice PDF
Kubernetes Security - Michael Hausenblas, Liz Rice PDF
m
pl
im
Kubernetes
en
ts
of
Security
Operating Kubernetes Clusters
and Applications Safely
Building and managing secure Kubernetes clusters is a complex task. Aqua Security
provides a complete solution that leverages native Kubernetes capabilities, makes it easy
to establish policy-driven monitoring and enforcement, and further secures Kubernetes
deployments with runtime protection and compliance controls at the cluster, namespace,
node, pod and container levels.
Enhances Protects
Native Kubernetes Security Controls Applications in Runtime
Aqua Security is the company behind open-source tools that enable you to improve
the security of your Kubernetes cluster:
Check your cluster against 100+ tests Penetration testing tool that “attacks”
of the CIS Kubernetes Benchmark so you your cluster and nodes, looking for
can harden it according to best practices. configuration issues.
github.com/aquasecurity/kube-bench github.com/aquasecurity/kube-hunter
Learn more
www.aquasec.com
Kubernetes Security
Operating Kubernetes Clusters and
Applications Safely
The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Kubernetes Secu‐
rity, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc.
The views expressed in this work are those of the authors, and do not represent the
publisher’s views. While the publisher and the authors have used good faith efforts
to ensure that the information and instructions contained in this work are accurate,
the publisher and the authors disclaim all responsibility for errors or omissions,
including without limitation responsibility for damages resulting from the use of or
reliance on this work. Use of the information and instructions contained in this
work is at your own risk. If any code samples or other technology this work contains
or describes is subject to open source licenses or the intellectual property rights of
others, it is your responsibility to ensure that your use thereof complies with such
licenses and/or rights.
This work is part of a collaboration between O’Reilly and Aqua Security Software.
See our statement of editorial independence.
978-1-492-04600-4
[LSI]
Table of Contents
Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v
3. Authentication. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Identity 15
Authentication Concepts 20
Authentication Strategies 21
Tooling and Good Practices 22
4. Authorization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Authorization Concepts 25
Authorization Modes 26
Access Control with RBAC 27
Tooling and Good Practices 32
iii
CI/CD Best Practices 37
Image Storage 38
Correct Image Versions 39
Image Trust and Supply Chain 40
Minimizing Images to Reduce the Attack Surface 41
7. Secrets Management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
Applying the Principle of Least Privilege 57
Secret Encryption 58
Kubernetes Secret Storage 58
Passing Secrets into Containerized Code 60
Secret Rotation and Revocation 63
Secret Access from Within the Container 64
Secret Access from a Kubelet 64
8. Advanced Topics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
Monitoring, Alerting, and Auditing 67
Host Security 68
Sandboxing and Runtime Protection 69
Multitenancy 70
Dynamic Admission Control 72
Network Protection 72
Static Analysis of YAML 73
Fork Bombs and Resource-Based Attacks 73
Cryptocurrency Mining 74
Kubernetes Security Updates 74
iv | Table of Contents
Introduction
This book will teach you practices to make your Kubernetes deploy‐
ments more secure. It will introduce you to security features in
Kubernetes and tell you about other things you should be aware of
in the context of containerized applications running on Kubernetes;
for example, container image best practices from a security point of
view.
We describe practical techniques and provide an accompanying
website with references and recipes, so if you want to follow along,
check it out!
v
and Sébastien Goasguen (O’Reilly) provides recipes for common
tasks.
In this book, we tackle the technical aspects of Kubernetes security,
but sidestep cultural and organizational issues, such as who should
be responsible for implementing and ensuring the advice we offer.
We do suggest that this is something you pay attention to, as no
amount of technology will fix a broken culture.
A Note on Federation
Federation is the concept of operating multiple Kubernetes clusters
together, with the ability to synchronize and discover resources
across them. At the time of writing, the Kubernetes Federation API
has no clear path to general availability, so we have left the security
of federated clusters out of the scope of this book.
Acknowledgments
A big thank you to the O’Reilly team, especially Virginia Wilson, for
shepherding us through the process of writing this book.
We’re super grateful to our technical reviewers Alban Crequy, Amir
Jerbi, Andrew Martin, Ian Lewis, Jordan Liggitt, Michael Kehoe,
vi | Introduction
Seth Vargo, and Tim Mackey, who provided valuable, actionable
feedback and advice.
Introduction | vii
CHAPTER 1
Approaching Kubernetes Security
Security is a funny, elusive thing. You will rarely hear a security pro‐
fessional describe something as “secure.” You’ll hear that something
may be more or less secure than an alternative, but security is
dependent on context.
In this book, we will show you ways to make your Kubernetes clus‐
ter more secure. Whether you need to apply a particular measure to
make your deployment secure enough for your particular use case is
something for you to assess, depending on the risks you are run‐
ning. We hope that if your Kubernetes cluster holds our bank
account details or our medical records, you will take all the precau‐
tions described herein, at the very least!
We will cover ways that you can configure your Kubernetes cluster
to improve security. Your cluster runs containerized workloads, and
we will discuss ways to make it more likely that you are running the
workloads you expect (and nothing more). We present precautions
you can take to limit the likelihood of a breach by an attacker, and to
limit the likelihood of that breach resulting in data loss.
In addition, you can use plenty of non-Kubernetes-specific security
tools and approaches that are outside the scope of this book. You
can layer traditional network firewalls and intrusion-detection sys‐
tems, in addition to everything that is described here. You may have
an air-gapped deployment. And wherever humans interact with
your system, they may constitute a risk to security, either mali‐
ciously or just due to human error. We don’t pretend to address
those issues in this book. As shown in Figure 1-1, there are various
1
ways that an attacker could attempt to compromise your Kubernetes
cluster and the applications running on it.
Security Principles
In this section, we’ll discuss three important principles that can be
used to increase security: defense in depth, least privilege, and limit‐
ing the attack surface.
Defense in Depth
Picture a medieval castle under siege. It has strong, high walls to
keep undesirables out. The wall is surrounded by a moat, with
access via a drawbridge that is lowered only occasionally to let peo‐
ple in and out. The castle has thick doors, and bars across any win‐
dows. Archers patrol the castle walls, ready to fire at any attacker.
The castle has several layers of defense. Attackers who can swim
might be prepared to cross the moat, but then they have the walls to
scale, and the likelihood of being picked off by an archer. It might be
possible to compromise any given layer in the defensive structure,
but by having several layers, it’s hard for an attacker to successfully
enter the castle.
In the same way, it’s preferable to have several layers of defense
against attacks on your Kubernetes cluster. If you’re relying on a sin‐
gle defensive measure, attackers might find their way around it.
Least Privilege
The principle of least privilege tells us to restrict access so that differ‐
ent components can access only the information and resources they
need to operate correctly. In the event of a component being com‐
promised, an attacker can reach only the subset of information and
resources available to that component. This limits the “blast radius”
of the attack.
Security Principles | 3
Consider an example of an e-commerce store. Let’s assume it is built
using a “microservice” architecture with functionality broken into
small, discrete components. Even if product and user information is
held in the same database, different microservices might each be
granted access to only the appropriate parts of that database. A
product-search microservice needs read-only access to the product
tables, but nothing more. If this microservice somehow gets com‐
promised, or simply has a bug, the broken service can’t overwrite
product information (because it has only read access) or extract user
information (because it has no access to that data at all). Applying
the principle of least privilege means that we make it more difficult
for an attacker to cause damage.
Security Principles | 5
CHAPTER 2
Securing the Cluster
Perhaps it goes without saying, but you don’t want to allow unau‐
thorized folks (or machines!) to have the ability to control what’s
happening in your Kubernetes cluster. Anyone who can run soft‐
ware on your deployment can, at the very least, use your compute
resources (as in the well-publicized case of “cryptojacking” at Tesla);
they could choose to play havoc with your existing services and even
get access to your data.
Unfortunately, in the early days of Kubernetes, the default settings
left the control plane insecure in important ways. The situation is
further complicated by the fact that different installation tools may
configure your deployment in different ways. The default settings
have been improving from a security point of view, but it is well
worth checking the configuration you’re using.
In this chapter, we cover the configuration settings that are impor‐
tant to get right for the Kubernetes control-plane components, con‐
cluding with some advice on tools that can be used to verify the
deployed configuration.
API Server
As its name suggests, the main function of the Kubernetes API
server is to offer a REST API for controlling Kubernetes. This is
powerful—a user who has full permissions on this API has the
equivalent of root access on every machine in the cluster.
7
The command-line tool kubectl is a client for this API, making
requests of the API server to manage resources and workloads. Any‐
one who has write access to this Kubernetes API can control the
cluster in the same way.
By default, the API server will listen on what is rightfully called the
insecure port, port 8080. Any requests to this port bypass authentica‐
tion and authorization checks. If you leave this port open, anyone
who gains access to the host your master is running on has full control
over your entire cluster.
Close the insecure port by setting the API server’s --insecure-port
flag to 0, and ensuring that the --insecure-bind-address is not set.
You can check whether the insecure port is open on the default port
with a simple curl command like the following, where <IP
address> is the host where the API server is running (or localhost
if you can SSH directly to that machine):
$ curl <IP address>:8080
{
"paths": [
"/api",
"/api/v1",
"/apis",
...
If the response lists API endpoints, as in the preceding example,
then the insecure port is open. However, if you see an error message
of Connection refused, it’s good news, as the port is not open.
With the insecure port closed, the API can be accessed only over a
secure, encrypted TLS connection via the secure port. You may want
to further restrict API access to known, authenticated users by set‐
ting --anonymous-auth=false for the API server. However, it is not
reckless to allow anonymous access to the API so long as you are
using RBAC, which we strongly recommend. We discuss this in
more detail in “Access Control with RBAC” on page 27.
Kubelet
The kubelet is the agent on each node that is responsible for inter‐
acting with the container runtime to launch pods, and report node
and pod status and metrics. Each kubelet in the cluster also operates
an API, through which other components ask it to do things like
starting and stopping pods. If unauthorized users can access this
API (on any node) to execute code on the cluster, it’s possible to gain
control of the entire cluster.
Fortunately, layers of defense are now available in Kubernetes that
make it easy to prevent this kind of attack:
Kubelet | 9
• You can limit the API access to authenticated requests; that is,
anonymous requests are ignored.
• You can leverage access control to stop unauthorized actions
from being performed (see “Access Control with RBAC” on
page 27).
Kubernetes Dashboard
The Dashboard has historically been used by attackers to gain con‐
trol of Kubernetes clusters. It’s a powerful tool, and in older versions
of Kubernetes, the default settings made it easy to abuse; for exam‐
ple, prior to 1.7, the Dashboard had full admin privileges by default.
You might want to take several steps to ensure that your Kubernetes
Dashboard is not an easy entry point for attackers, including but not
limited to the following:
Allow only authenticated access
Only known users should be able to access the Dashboard.
Use RBAC
Limit the privileges that users have so they can administer only
the resources they need to.
Make sure the Dashboard service account has limited access
After reaching the Dashboard login screen, users have the
option to Skip. Taking this path means that rather than authen‐
ticating as their own user identity (as discussed in “Identity” on
page 15), they access the Dashboard with the service account
Penetration Testing
Enterprises commonly recruit the services of a “pen-tester,” or pene‐
tration testing company, to probe their deployed software, searching
for ways that an attacker could exploit the software or the platform
on which it runs. A penetration-testing specialist will use creative
approaches to find weak points in your cluster configuration and in
the software running on it.
Additionally, you may like to consider testing with kube-hunter.
This project (also one that Liz maintains) is an open source penetra‐
tion testing tool specifically for Kubernetes.
To learn more about how to secure the Kubernetes control plane,
check out the resources on the accompanying website, in the “Secur‐
ing the Cluster” section.
Now that we have covered configuring the Kubernetes control-plane
components, let’s move on to discussing how to enable access to the
cluster by known users and software entities.
Identity
For the API server to authenticate a request, the request issuer needs
to possess an identity. At the time of writing, Kubernetes doesn’t
have a first-class notion of a human user, but rather assumes that
users are managed outside Kubernetes via a directory service such as
Lightweight Directory Access Protocol (LDAP) or single sign-on
(SSO) login standards like Security Assertion Markup Language
(SAML) or Kerberos. This is the standard approach in production,
but if you’re not using such a system, other authentication strategies
are available.
15
User accounts are considered cluster-wide, so make sure that the
usernames are unique across namespaces.
It’s not just humans who interact with Kubernetes. We often want a
programmatic way for applications to communicate with the Kuber‐
netes API; for example, to query, create, or update resources such as
pods, services, or deployments. To that end, Kubernetes has a top-
level resource to represent the identity of an application: the service
account. A service account is a namespaced resource that you can
use if your application needs to communicate with the API server.
Many business applications don’t need to manipulate Kubernetes
resources in this way, so (following the principle of least privilege)
they can have service accounts with limited permissions.
By default, Kubernetes makes the credentials of the service account
available via a secret that is mounted into the pod (note that all files
shown here are owned by root):
$ kubectl run -it --rm jumpod \
--restart=Never \
--image=alpine -- sh
~ $ ls /var/run/secrets/kubernetes.io/serviceaccount/
ca.crt namespace service-ca.crt token
16 | Chapter 3: Authentication
Figure 3-1. A JSON Web Token provided by a service account
If you don’t explicitly specify a service account in the pod spec, the
default service account for the namespace is used.
The general form of a service account is as follows:
system:serviceaccount:$NAMESPACE:$NAME
In Figure 3-2, you can see a more complex example setup.
Identity | 17
Figure 3-2. Service accounts
18 | Chapter 3: Authentication
What you can learn from the preceding output (also shown in
Figure 3-2) is that the creation of a service account triggers the cre‐
ation of a secret, attached to and managed by the service account.
This secret contains the JSON Web Token discussed earlier.
Now that we have created the service account, we want to use it in a
pod. How can you do that? Simply by using the serviceAccount
Name field in the pod spec to select the service account, in our case,
mysa. Let’s store a pod spec in a file called podwithsa.yaml with the
following content:
apiVersion: v1
kind: Pod
metadata:
name: podwithsa
spec:
serviceAccountName: mysa
containers:
- name: shell
image: alpine:3.7
command:
- "sh"
- "-c"
- "sleep 10000"
You can launch the pod and inspect its properties as follows (the
output has been edited for better readability):
$ kubectl apply -f podwithsa.yaml
pod "podwithsa" created
And indeed, here you see that our podwithsa pod uses its own ser‐
vice account with the token mysa-token-prb4r (allowing it to com‐
municate with the API server) available at the usual file
location /var/run/secrets/kubernetes.io/serviceaccount/token moun‐
ted into the pod.
Identity | 19
At this point, you might be wondering why you would bother at all
messing around with service accounts and not always use the default
service account. This will make more sense when you learn how ser‐
vice accounts are used with RBAC to define permissions for users
and applications in Chapter 4. For now, just remember that service
accounts allow applications to communicate with the API servers (if
they have to at all).
Now that we’ve covered the basics of identity in Kubernetes, let’s
move on to how authentication works.
Authentication Concepts
In Figure 3-3, you can see how the API server conceptually per‐
forms authentication by using one of the available strategies repre‐
sented by the authentication plug-ins (learn more about the
supported strategies in the next section).
20 | Chapter 3: Authentication
4. If the credentials are in order, the API server moves on to check
permissions as described in Chapter 4. Otherwise, it returns an
HTTP 401 Unauthorized client error status response code, and
with that the request fails.
Kubernetes also supports user impersonation; that is, a user can act
as another user. For example, as a cluster admin, you could use
impersonation to debug any authorization issues.
Authentication Strategies
A couple of authentication strategies are available in Kubernetes,
represented by authentication plug-ins. Depending on the size of the
deployment, the target users (human versus processes), and organi‐
zational policies, you as a cluster admin can choose one or more of
the following:
Static password or token file
This strategy uses the Basic HTTP authentication scheme as per
RFC7617. Essentially, the API server requires the client to pro‐
vide the identify via an HTTP header named Authorization
and the value of Basic base64($USER:$PASSWORD) in case of a
static password file or Bearer $TOKEN in case of a static token
file. Since it’s inflexible to maintain a static file with the users
and their passwords and requires direct access to the API server,
this method is not recommended in production.
X.509 certificates
With this strategy, every user has their own X.509 client certifi‐
cate. The API server then validates the client certificate via a
configured certificate authority (CA). If the client certificate is
verified successfully, the common name of the subject is used as
Authentication Strategies | 21
the username for the request, and any organizations defined for
the subject are used as groups. As an admin, you need to man‐
age access to the CA as well as issue the client certificates, and
reissue them as they approach expiry. Kubernetes does not, at
the time of writing, support certificate revocation, and this is
considered a good reason to use an SSO approach where possi‐
ble.
OpenID Connect (OIDC)
OIDC is an identity layer on top of the OAuth 2.0. With this
strategy, you use OIDC to provide the API server with an id-
token in the form of a JSON Web Token after using your pro‐
vider’s login page, such as Google or Azure Active Directory.
Bootstrap tokens
These are an experimental feature targeting the cluster setup
phase and can be used with installers such as kubeadm.
If you want to integrate with other authentication protocols such as
LDAP, SAML, and Kerberos, you can use one of the following meth‐
ods:
Authenticating proxy
The API server can be configured to identify users from request
header values, such as X-Remote-User. You need to take care of
setting up and running the proxy; see, for example, Haoran
Wang’s post of an authentication example.
Webhook token authentication
Essentially, a hook for verifying bearer tokens.
With that, we move on to some good practices and tooling around
authentication.
22 | Chapter 3: Authentication
Online, etc.), or an on-premises deployment. The latter is impor‐
tant, as you may have different options depending on the environ‐
ment and may end up having more or less work with the
authentication bits, based on what authentication strategy you go
for.
Several tools are available to help with this (you may wish to check
the latest list on the website accompanying this book):
Keycloak
An open source IAM solution with built-in support to connect
to existing LDAP servers. Keycloak can authenticate users with
existing OIDC or SAML 2.0 identity providers. A Helm chart is
also available to deploy it in Kubernetes.
Dex
An identity service that uses OIDC to drive authentication for
other applications. Dex acts as a portal to other identity provid‐
ers, allowing you to defer authentication to LDAP servers,
SAML providers, or established identity providers like GitHub,
Google, and Active Directory.
AWS IAM Authenticator for Kubernetes
A tool to use AWS IAM credentials to authenticate to a Kuber‐
netes cluster maintained by Heptio and Amazon.
Guard
A Kubernetes webhook authentication server by AppsCode,
allowing you to log into your Kubernetes cluster by using vari‐
ous identity providers, from GitHub to Google to LDAP.
In the last section of this chapter, we look at good practices in the
context of authentication. Note that because a new Kubernetes
release comes out every couple of months, some tips might be more
relevant than others (as defaults change or new features are intro‐
duced):
Use third-party providers
Unless you have to roll your own thing, integrate Kubernetes
with third-party identity providers such as Azure, Google, or
GitHub.
Don’t use static files
If you can’t use third-party providers, prefer X.509 certificates
over static password or token files.
24 | Chapter 3: Authentication
CHAPTER 4
Authorization
Authorization Concepts
Kubernetes authorizes API requests by using the API server, evalu‐
ating the request attributes against the policies and subsequently
allowing or denying the request. By default, permissions are denied,
unless explicitly allowed by a policy. Conceptually, authorization in
Kubernetes works as depicted in Figure 4-1.
25
Figure 4-1. Authorization concepts
Authorization Modes
Kubernetes offers multiple ways to enforce permissions, represented
by various authorization modes and modules:
26 | Chapter 4: Authorization
Node authorization
A special-purpose authorizer that grants permissions to kube‐
lets based on the pods they are scheduled to run.
Attribute-based access control (ABAC)
An authorizer through which access rights are granted to users
through policies combining attributes (user attributes, resource
attributes, objects, etc.).
Webhook
A webhook is an HTTP callback—an HTTP POST that occurs
when something happens. This mode allows for integration
with Kubernetes-external authorizers.
Role-based access control (RBAC)
This is explained in detail in the following section.
Since RBAC is the most important authorization method for both
developers and admins in Kubernetes, let’s look at it in greater detail.
The actions on a resource that a role uses in its rules are the so-
called verbs, such as the following:
28 | Chapter 4: Authorization
Kubernetes prevents users from escalating privileges by
editing roles or role bindings. Users can create or
update a role only if they already have all the permis‐
sions contained in the role. For example, if user alice
does not have the ability to list secrets cluster-wide,
that user cannot create a cluster role containing that
permission.
As you can see, the view default role would work, but it additionally
allows your application access to many other resources such as
deployments and services. This is a potential security risk and goes
against the principle of least privilege, so let’s create a dedicated role
for it. A role that allows you to retrieve only info about pods.
30 | Chapter 4: Authorization
Since we want to set permissions for an application rather than a
user whose identity is managed outside Kubernetes, we first have to
create a dedicated service account representing the application’s
identity toward the API server. Also, it’s a good practice to not use
the default namespace, so let’s start by creating a namespace coolapp
that our application will live in and then a service account myappid
in this namespace:
$ kubectl create namespace coolapp
namespace "coolapp" created
$ kubectl --namespace=coolapp create serviceaccount myappid
serviceaccount "myappid" created
Now that we have established an identity for our application, we can
define a role podview that allows only viewing and listing pods in its
namespace:
$ kubectl --namespace=coolapp create role podview \
--verb=get --verb=list \
--resource=pods
That looks more like it! The role podview allows only for viewing
pods. Next, we need to attach the role podview to our application,
represented by the service account myappid. We do this by creating a
role binding (which binds a role to a human or machine user) called
mypodviewer, like so:
$ kubectl --namespace=coolapp create rolebinding mypodviewer \
--role=podreader \
--serviceaccount=coolapp:myappid
rolebinding.rbac.authorization.k8s.io "mypodviewer" created
32 | Chapter 4: Authorization
Use RBAC
This should be the standard now—if not, please do upgrade
Kubernetes to a version equal to or greater than 1.8. Pass the --
authorization-mode=RBAC parameter to the API server to
enable this.
Disable automounting of the default service account token
Most applications don’t need to talk to the API server, so they
don’t need an access token. This is especially important if you’re
not using RBAC. You can do this by specifying automountServi
ceAccountToken: false in the PodSpec for your applications,
or you can patch the default service account so that its creden‐
tials are not automatically mounted into pods:
$ kubectl patch serviceaccount default \
-p $'automountServiceAccountToken: false'
serviceaccount "default" patched
Until now, we’ve been discussing things mainly from the point of
view of a Kubernetes cluster administrator. Going forward, we’ll
switch gears and focus more on developers, operators, or even
DevOps teams who want to deploy code to run on the cluster.
The software that you run in your Kubernetes cluster gets there in
the form of container images. In this chapter, we’ll discuss how to
check that your images:
Vulnerabilities
In this context, a vulnerability is a flaw in a piece of code that an
attacker can exploit to cause undesirable consequences, and that
has been publicly disclosed (typically, through the National Vulner‐
ability Database). For example, the renowned Heartbleed vulnera‐
bility was a flaw in the OpenSSL library that allowed attackers to
access system memory, and hence steal encrypted information.
35
Scanning Container Images
To detect vulnerabilities, you need to use a container image scanner.
The basic function of a container image scanner is to inspect the
packages included in an image, and report on any known vulnera‐
bilities included in those packages. At a minimum, this looks at the
packages installed through a package manager (like yum or apt,
depending on the OS distribution). Some scanners may also exam‐
ine files installed at image build time; for example, through ADD,
COPY, or RUN operations in a Dockerfile. Some scanners also
report on known malware (e.g., viruses) or the presence of sensitive
data (like passwords and tokens).
To ensure that you’re not running vulnerable code in your deploy‐
ment, you should scan any third-party container images as well as
the ones built by your own organization.
New vulnerabilities continue to be found in existing software, so it’s
important to rescan your images on a regular basis. In our experi‐
ence, it’s typical for enterprise customers to rescan the images in use
on their production systems every 24 hours, but you should con‐
sider your own risk profile. Depending on the scanning tool you
use, this may be a simple configuration setting, or you may need to
write automation scripting to put this in place.
Several commercial image-scanning tools are available as well as
some open source and/or free-to-use options.
Some registries provide metrics on the health of the container
images they store. For example, the Red Hat Container Catalog
grades images from A–F, and the Google Container Registry and
Docker Trusted Registry also include image scan results.
Image Storage
Container images can be stored in public or private registries. Many
security-conscious organizations use one or more private registries
and require that only images from these registries can be deployed.
Now that you know how to build container images in a secure man‐
ner from Chapter 5, we move on to the topic of running those
images as containers in Kubernetes. In order to run containers
securely in Kubernetes, we aim to do the following:
Say No to Root
As “Mr. SELinux” Dan Walsh pointed out in “Just Say No to Root (in
Containers),” there’s little need to run containers as root. Some
exceptions are as follows:
43
• The container needs to bind to privileged ports on the node
(below 1024—for example, nginx serving on port 80). In prac‐
tice, this can be by-and-large avoided through port mappings
and the service abstraction in Kubernetes.
• Installing software into a container at runtime: traditional pack‐
age management systems might require root to function or
store files in a certain location with a different user ID than the
user executing the program. This approach is generally consid‐
ered bad practice since any code installed at runtime has not
been scanned for vulnerabilities or other policy requirements
(see Chapter 5).
If your container does not fall into one of the preceding categories,
then according to the principle of least privilege, it would make
sense to run it as a nonroot user. You can do this by including a USER
command in the Dockerfile, defining a user identity that the code
should run under.
The advocacy site canihaznonprivilegedcontainers.info has more
background resources on this topic, and Liz explored it in her “Run‐
ning with Scissors” keynote at KubeCon Copenhagen in 2018. How‐
ever, general awareness around this topic is sadly still low, and most
images on Docker Hub are built to run as the root user by default
(having no USER command).
Let’s move on and see how the API server enforces policies.
Admission Control
When a client submits a request to the API server and that request
has been authenticated (Chapter 3) and the client is authorized
(Chapter 4) to carry out the operation, there is one more step the
API server performs before persisting the resource in etcd: admis‐
sion control. A whole slew of admission controllers are included in
the API server, that you, as a cluster admin, can configure. The offi‐
cial docs list explains the more than 30 controllers in great detail;
some relevant ones in the context of running containers securely are
as follows:
AlwaysPullImages
Modifies every new pod to force the image pull policy to
Always, overwriting the default specification. This can be
Security Boundaries
We introduced some security principles in Chapter 1, and one of
those principles is defense in depth. Kubernetes gives you a set of
first-class mechanisms to realize defense in depth. To better under‐
stand what that means, have a look at Figure 6-1, which shows the
security boundaries present by default.
Security Boundaries | 45
Figure 6-1. Security boundaries
Now that you know about the security boundaries, let’s see what
mechanisms you have available to establish and enforce them.
Policies
Kubernetes offers two pod-level security-policy mechanisms allow‐
ing you to restrict what processes can do within a pod (as described
Policies | 47
in the next section) and how pods are allowed to communicate (as
laid out in “Network Policies” on page 52).
• All containers in the pod must run under user 1001, through the
runAsUser setting.
• In the webserver container, prevent setuid binaries from
changing the effective user ID as well as prevent files from ena‐
bling extra capabilities by setting allowPrivilegeEscalation
to false.
Policies | 49
A word on seccomp and AppArmor profiles: by
default, containers running under Docker use a sec‐
comp and AppArmor profile that prevent some system
calls that most containerized applications have no
business trying to run. Docker is used to provide the
runtime layer for many Kubernetes installations, and it
would be easy—but sadly incorrect, at least at the time
of writing—to assume that the same profiles would be
used by default in Kubernetes as well.
Policies | 51
Network Policies
Limiting the traffic that can flow between pods adds a good layer of
security:
A network policy applies to the set of pods that match the podSelec
tor defined in the spec. Typically, a label selector is used to match a
subset of pods; an empty podSelector as in the preceding example
matches all pods.
If a pod is not matched by any network policies, all traffic is allowed
to and from that pod.
The Kubernetes documentation includes example network policies
that you might use to limit traffic to and from all pods by default.
Policies | 53
- protocol: TCP
port: 5432
We then likely want to allow traffic from the internet to access my-
app, which we can achieve with another network policy like this:
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: allow-external
namespace: lockeddown
spec:
podSelector:
matchLabels:
app: my-app
ingress:
- from: []
The combination of these three network policies allows the applica‐
tion traffic to flow as desired from external users to the application
pods, and from the application pods to the database, but traffic is
not permitted anywhere else in this lockeddown namespace.
Ahmet Alp Balkan has put together a set of useful network policy
recipes as well as a good post on the topic of “Securing Kubernetes
Cluster Networking”. Another helpful backgrounder is “Kubernetes
Security Context, Security Policy, and Network Policy” by Mateo
Burillo.
Going forward, service meshes will also play a role in this area. See
“Service Meshes” on page 72 for more on this topic.
Policies | 55
CHAPTER 7
Secrets Management
57
of team members without necessarily giving them full access to
the production credentials.
Secret Encryption
Since secret values protect sensitive data, we want them to be hard
to access. Ideally, they should by protected at rest and in transit:
At rest
Secrets should always be stored on disk in encrypted form, so
that an attacker with access to the filesystem cannot simply read
them from the file. In this chapter, you will see how secrets are
stored in Kubernetes, and our options for encrypting at rest.
In transit
Secrets should be encrypted whenever they are sent as network
traffic, so that an attacker snooping on the network cannot read
them as they pass.
Encryption in transit is achieved by encrypting the traffic between
the Kubernetes control-plane components using TLS (see Chap‐
ter 2). In order to consider how we can store our secrets safely in
encrypted form, let’s look at how Kubernetes manages secrets, and
the options for where they can be stored.
The default storage for secret values is etcd, or you can use third-
party secret storage solutions.
Anyone who gains access to your etcd database will be able to read
base64-encoded secrets from it. You can control access by configur‐
ing secure access only (see Chapter 2), but even then there is a risk:
your data is written to disk unencrypted, and if an attacker gains
access to the filesystem, your data may be compromised.
You can avoid this risk by ensuring that your etcd cluster is also
encrypted on disk. (The API server has access to the encrypted data
in etcd, so you will also want to limit API access as described in
Chapter 2 and Chapter 4.)
It’s good practice to rotate the encryption secret from time to time.
If you have multiple etcd nodes, you should also encrypt the com‐
munication between them, to prevent the secret values from being
passed in the clear.
• Anyone who has access to the image can obtain the secrets it
holds. Bear in mind that the set of people who can access the
image may not be the same set of people who need your pro‐
duction credentials.
Case 1
It’s common for a process to log out its entire environment in the
event of a crash. This may be written to file, or in many deployments
it will make its way to a centralized log aggregation system. Should
the people who have access to this system also have access to your
production database credentials?
Case 3
If you are using Docker as your container runtime, the environment
is accessible using docker inspect <container>, running on the
host as shown here:
$ sudo docker inspect b5ad78e251a3
[
{
"Id": "b5ad78e251a3f94c10b9336ccfe88e576548b4f387f5c7040...",
...
"Config": {
"Hostname": "nginx-env-6c5b7b8ddd-dwpvk",
...
"Env": [
"NOT_SO_SECRET=some_value",
...
]
}
}
]
The fact that environment variables are accessible via logs or via the
command line to a broader set of people than might need access to
secret credentials can be considered a security risk. You should con‐
template whether this is an issue in your application and in your
organization.
Some commercial solutions inject secrets into the environment by
using a proprietary technique that means that the secret is not avail‐
67
logs, but here you also should be aware of where and when your
sensitive data lands on disk; see Chapter 7 for details.
Another useful feature Kubernetes offers via the API server is audit‐
ing, effectively recording the sequence of activities affecting the clus‐
ter. Different strategies are available in the auditing policy (from no
logging to logging event metadata, request and response bodies),
and you can choose between a simple log backend as well as using a
webhook for integrating with third-party systems.
Host Security
Much discussion of Kubernetes security focuses on the setup of
Kubernetes itself, or on securing the containerized applications that
run within it. There is another layer to be secured: the host
machines on which Kubernetes runs.
Whether they are bare-metal or VMs, there are many steps you can
take to restrict access to your hosts that are essentially the same as
you would take in a traditional cluster. For example, you should, of
course, restrict access to the machines, and you might well configure
them to use a dedicated VPN. These general machine security meas‐
ures are outside the scope of this book, but there are some specific
things you would do well to consider when you are running Kuber‐
netes (or any container orchestrator). Resources such as OpenSCAP
and OVAL can help with a broader security assessment.
Node Recycling
In a cloud-native deployment, we treat nodes as “cattle not pets,” and
it should be trivially easy to create a new node or replace a faulty
one, as it should be automated through an infrastructure as code
approach. This enables node recycling, where you tear down and
replace your nodes on a regular (or random) schedule.
When you recycle a node, you know that it has been returned to the
desired state, as determined by your infrastructure as code. If there
has been any “drift”—for example, because an undetected attacker
got a foothold into the system—that drift is removed.
Another benefit of recycling your nodes (especially for small or new
deployments) is that it’s effectively a fire drill. If you are frequently
replacing nodes as a matter of course, you can have more confidence
in the system’s ability to cope through node failure. It’s a baby step
toward chaos engineering!
Multitenancy
In some deployments, multiple “tenants” using the same cluster
don’t fully trust each other or might not be trusted by the cluster
operator. Tenants could be users, groups of users, or applications.
The level of trust between users depends on the environment; for
example, in a platform-as-a-service (PaaS) environment where users
can upload and run their own applications, they should be treated as
entirely untrusted, whereas in an enterprise, the tenants might map
to different organizational teams, who do cooperate and trust each
other to some extent, but who want to limit the risk of someone out‐
side one team maliciously or inadvertently affecting another team’s
application.
Multitenant isolation has two parts:
• The control plane, so that users can’t, for example, run kubectl
commands that impact other tenants’ resources
The Kubernetes namespace gives us the first building block for mul‐
titenancy in the control plane. Typically, there will be one name‐
space per tenant. Users are given RBAC permissions to create,
update, and delete resources within the namespace that maps to
their tenancy (see Chapter 3 and Chapter 4). Resource quotas allow
each namespace (and thereby, each tenant) to be restricted to a limit
of compute and storage resources, and of Kubernetes objects (for
example, upper bounds on the number of services, secrets, and per‐
sistent volume claims allowed within the namespace).
In an enterprise environment, this may be sufficient. Namespace-
based RBAC controls mean that one team can’t update application
code and associated Kubernetes resources that they are not respon‐
sible for. Quotas mean that one team’s application can’t use all avail‐
able resources so that another is starved.
However, this is unlikely to be sufficient protection in a fully untrus‐
ted environment. Here, tenant workloads should be isolated from
each other at a container level so that if there were to be an escape
from one container (perhaps because a user deploys code with a
serious vulnerability, perhaps even deliberately), they can’t affect or
inspect other tenants’ applications or data. Container sandboxing, as
described in “Sandboxing and Runtime Protection” on page 69, is
largely designed to solve this problem (for example, the gVisor
approach is based on the way Google isolates user workloads from
each other in Google App Engine).
Another approach to workload isolation is to assign one or more
nodes to each tenant, and then schedule pods so that workloads are
only ever colocated with other workloads from the same tenant.
This is straightforward (using taints and tolerations), but potentially
wasteful unless each tenant needs a node’s worth of compute resour‐
ces.
In an untrusted multitenant environment, you would want strong
network policies to isolate traffic so that it can’t flow between name‐
spaces. See “Network Policies” on page 52 for more information.
Multitenancy | 71
Dynamic Admission Control
From Kubernetes 1.9, dynamic admission controllers allow for flexi‐
ble, extensible mechanisms for making checks before allowing a
resource to be deployed to a cluster. As an example, check out Kel‐
sey Hightower’s Grafeas tutorial, which includes a validating web‐
hook that ensures that only signed images are admitted.
Network Protection
In a typical traditional deployment, a significant proportion of the
security measures is network based: firewalling and the use of VPNs
come immediately to mind. In the cloud-native world, similar
approaches are dedicated to restricting traffic so that only approved
flows can take place.
Security solutions for containers have for some years talked about
network micro- or nano-segmentation, and these approaches have
been made fairly common practice in Kubernetes deployments
through the use of network policies (as discussed in “Network Poli‐
cies” on page 52).
At the time of writing, service meshes are a popular topic—
although, in our opinion, currently at the stage of “early adopter”
rather than “early majority” market penetration.
Service Meshes
The idea of a service mesh like Istio or Linkerd is to take on much of
the burden of networking communication and control, so that
application developers don’t need to concern themselves with these
nonfunctional capabilities. While they offer several features like load
balancing and routing that are outside the scope of this book, they
offer two features that are of particular interest in relation to secu‐
rity: mutually authenticated TLS connections and service mesh net‐
work policies.
Cryptocurrency Mining
A famous attack on Tesla exploited control-plane insecurities to
allow hackers to use the company’s resources to mine cryptocur‐
rency. Following the advice in Chapter 2 will go a long way to pre‐
venting this from happening in your cluster.
Additional research shows other approaches that would-be miners
are attempting. Ensuring that only trusted images can run in your
cluster would prevent, for example, a bad actor with approved access
to the cluster from running an unexpected mining image. See Chap‐
ter 5 for advice on preventing untrusted or compromised images
from running.
Runtime protection can add another layer of defense to ensure that
even if an approved image has a vulnerability that allows code to be
injected into a running container, that code can’t be executed.
Monitoring for unusual activity, such as unexpected CPU usage and
unexpected resources being scaled out, can help spot when your
resources are being used by an attacker. See “Monitoring, Alerting,
and Auditing” on page 67.