Securing a Kubernetes Cluster

In the early version of kubernetes, the default settings has left the control plane insecure.

This is further complicated since various kubernetes installation tools could configure your deployment in different ways. Later version of kubernetes has improved the default setting from a security point of view. However, it is worth checking the security settings of your deployment.

Kubernetes API server

The kubernetes API server offers REST API for controlling the cluster. This is very powerful as a user with full API server access has an equivalent of root access to all the machines in the cluster.

• kubectl is a client for the API server, which makes requests to the API server to manage resources and workloads
• The API server listens on an “insecure port”, port 8080.
• Any request to this port bypass authentication and authorisation checks.
• If this port 8080 is left open, anyone who gains access to host on your master is running on has full control over your entire cluster.
• Mitigation: Close port 8080 by setting the API server’s –insecure-port flag to 0 and ensure –insecure-bind-address is not set.
• Run the checks below to validate the configuration:

$ curl <IP Address>:8080

where IP Address is the host where the API server is running.

To be sure the port is not opened, a connection refused error message should be returned.

• The API can only be accessed over a secure TLS connection via a secure port. It is therefore necessary to restrict API access to known authenticated user by setting –anonymous-auth=false for the API server.
• It is not always reckless to allow anonymous access to API server, as long as you are using RBAC which permits only limited access to anonymous users. RBAC authorisation module can be enabled by setting the –authorisation-mode on the API server.

Kubelet

• This is an agent on each node that is responsible for interacting with the container runtime to launch pods, and report node and pod status as well as metrics.
• Each kubelet in the cluster has an API through which other components ask it to do things like start and stop the pods.
• If any unauthorised users can access this API (on any node) to execute code on the cluster, it’s possible to gain control of the entire cluster

Layers of defense to prevent attack

• Limit API access to authenticated request and ignore anonymous requests
• Leverage access with RBAC to stop unauthorised actions from being performed.

Configuration options to minimising attack surface

• Disable anonymous access with –anonymous-auth=false
• Ensure requests are authorised by setting –authorisation mode to something other than AlwayAllow.
• Limit the permissions of kubelets by including NodeRestriction in the –admission control settings.
• Set –read-only-port=0 to turn off the read-only port.
• Unless you are using cAdvisor port, you should turn it off to stop it from exposing information about running workloads by setting –cadvisor-port=0
• Run the checks below to valid the configuration:

$ curl -sk https://<IP address>:10250/pods/

• If –anonymous-auth is set to false, you will see an error 401 unauthorised response.
• If –anonymous-auth is true and –authorisation-mode is webhook, you will see error 403 forbidden response.
• If –anonymous-auth is true and –authorisation-mode is AlwaysAllow, you will see a list of pods.
• Kubelet uses client certificate to communicate with the API server. Kubelet supports rotation of these certificates automatically with –rotate-certificates flag so that a new certificate will be requested and issued automatically as the expiration deadline of the previous certificates approaches.

Etcd

• Kubernetes stores configuration and state information in a distributed key-value store called etcd.
• Anyone who can write to etcd can effectively control your kubernetes cluster

How to secure etcd

• Set –cert-file and –key-file to enable HTTPS connections to etcd
• Set –client-cert-auth=true to ensure access to etcd requires authentication
• Set –trusted-ca-file to specify the certificate authority that signed the client certificate
• Set –auto-tls=false to disallow the generation and use of self-signed certificates
• Set –etcd-cafile on the API server to the certificate authority that signed etcd’s certificate
• Specify –etcd-certfile and –etcd-keyfile so that the API server can identify itself to etcd
• Set –peer-client-cert-auth=true and –peer-auto-tls=flase and specify –peer-cert-file, –peer-key-file and –peer-trusted-ca-file
• Additional steps should be taken to encrypt etcd’s data stored on disk. This is particularly important if you are storing kubernetes secrets in etcd rather than an external secret store.

Securing Kubernetes Dashboard

Attackers can gain control of your kubernetes cluster via the dashboard. There are some default settings on this dashboard that made it easy to abuse.

The following steps can ensure the kubernetes dashboard is not an easy entry point for attackers:

• Allow only authenticated access
• Use Role Based Access Control (RBAC)
• Make sure the dashboard service account has limited access
• Don’t expose your dashboard to public internet
• Recommended settings for kubernetes dashboard installation can be found here
• Use kubectl proxy to access the dashboard securely from the local machine

Validating Cluster Configuration

In general, there are two main options to validate whether your cluster is securely configure.

• Configuration testing – validate deployment against a recommended set of settings. The Centre for Internet Security publishes a benchmark for kubernetes best practises for configuring a deployment to use secure settings. It’s a good idea to check your deployment against this benchmark.

Penetration testing – explore the cluster from the attacker’s perspective. Kube-hunter is an open source penetration testing tool specifically for kubernetes.