Kubernetes TLS Setup Howto

Secure communication on Kubernetes cluster.

Kubernetes apiserver supports both insecure HTTP and secure HTTPS/TLS protocol.

The insecure HTTP with port 8080 is the default setup but as the name indicates, it is not secure. So we need to set up secure HTTPS communication on KUBIC cluster. Let’s see how to do it.

SSL vs TLS

Before setting up HTTPS, we talk about SSL vs. TLS since HTTPS uses these technologies.

  • Secure Sockets Layer protocol
  • developed by Netscape
  • v2.0(1995), v3.0(1996)
  • old protocol
  • POODLE/BEAST attack
  • not recommended

POODLE attack made SSL v3.0 no longer secure.

  • Transport Layer Security protocol
  • v1.0 (1999, RFC 2246)
  • v1.1 (2006, RFC 4346)
  • latest v1.2 (2008, RFC 5246)
  • v1.3 is coming soon.
  • More secure than SSL (v1.0 is still vulnerable to BEAST attack)
  • Use TLS whenever possible

It is recommended not to use SSL v2/v3, TLS v1.0 for security, especially SSL v2/v3. So from now on, I’ll only mention TLS.

openssl is used for the standard TLS implementation.

TLS handshake between server and client

How do client and server handshake? There are 9 handshake steps before transfering data between server and client.

  1. TLS client sends a “client hello” message to TLS server.
  • TLS version: supported TLS version
  • CipherSuites: supported cipher methods
  • a random byte string to use in subsequent computations
  1. TLS server responds with a “server hello” message.
  • CipherSuite: chosen from the list of CipherSuites sent by TLS client
  • session ID
  • another random byte string
  • server certificate
  • (optional) client certificate request
    • list of the types of certificates supported
    • Distinguished Name(DN) of acceptable Certification Authorities (CAs)
  1. TLS client verifies the server certificate
  • CA.crt + server.crt
  1. TLS client sends
  • the random byte string to compute the secret key
  • the secret key will be shared on both.
  • The random byte string is itself encrypted with the server’s certificate.
  1. (optional) The client sends
  • a random byte string encrypted with the client private key
  • the client certificate or “no certificate alert”
  1. (optional) TLS server verifies the client certificate
  • CA.crt + client.crt
  1. TLS client sends
  • “finished” message encrypted with the secret key
  1. TLS server sends
  • “finished” message encrypted with the same secret key
  1. For the duration of TLS session, the server and client exchange
  • messages that are symmetrically encrypted with the shared secret key.
 ----------                              -----------
|TLS Client|                            | TLS Server|
|          |1.client hello              |           |
|          |--------------------------->|           |
|          |CipherSuites                |           |
|          |                            |           |
|          |2.server hello              |           |
|          |<-------------------------->|           |
|          |ciphersuite                 |           |
|          |server cert.                |           |
|          |client cert req.(opt.)      |           |
|          |                            |           |
|3. verify |                            |           |
|   server |                            |           |
|   cert.  |                            |           |
|          |4.client key exchange       |           |
|          |--------------------------->|           |
|          |random string(secret key)   |           |
|          |(encrypted with svr cert.)  |           |
|          |5.client certificate(opt.)  |           |
|          |                            |6.verify   |
|          |                            |  client   |
|          |                            |  cert.    |
|          |7.client "finish"           |           |
|          |--------------------------->|           |
|          |8.server "finish"           |           |
|          |<---------------------------|           |
|          |                            |           |
|          |9. exchange messages        |           |
|          |<-------------------------->|           |
|          |(encrypted with secret key) |           |
 ----------                              -----------

The following is the procedure to set up TLS environment for the cluster.

Kubernetes master

Create a directory to store the certificates.

$ sudo mkdir /srv/kubernetes

Create a Certificate Authority(CA) key and certificate.

$ sudo openssl genrsa -out /srv/kubernetes/ca.key 4096
$ sudo openssl req -x509 -new -nodes \
  -key /srv/kubernetes/ca.key \
  -subj "/CN=kubemaster" -days 10000 \
  -out /srv/kubernetes/ca.crt

For the server certificate, I’ll add

* Subject: CN=kubernetes.default.svc (for other addons like prometheus)
* subjectAltName = IP:10.24.0.1 (for kubedns)

http://wiki.cacert.org/FAQ/subjectAltName

Create a server key.

$ sudo openssl genrsa -out /srv/kubernetes/server.key 2048

Generate a Certificate Signing Request(CSR) for a server certificate.

$ cat /etc/ssl/openssl.cnf \
  <(printf "[SAN]\nsubjectAltName='DNS.1:kubernetes.default.svc,DNS.2:kubemaster,IP.1:10.24.0.1'") \
  > /tmp/openssl.cnf
$ sudo openssl req \
  -new -key /srv/kubernetes/server.key \
  -subj "/CN=kubernetes.default.svc" \
  -reqexts SAN \
  -config /tmp/openssl.cnf \
  -out /srv/kubernetes/server.csr

Generate the server certificate from the CSR and sign it with CA key and certificate.

$ sudo openssl x509 -req -in /srv/kubernetes/server.csr \
-CA /srv/kubernetes/ca.crt -CAkey /srv/kubernetes/ca.key \
-CAcreateserial \
-extensions SAN -extfile /tmp/openssl.cnf \
-days 10000 -out /srv/kubernetes/server.crt

View the server certificate(optional).

$ sudo openssl x509 -noout -text -in /srv/kubernetes/server.crt

Make the certificate files secure. Kubernetes master’s three musketeers (apiserver, controller-manager, scheduler) is run by ‘kube’ user in my setup so I make only ‘kube’ user read the certificate files.

$ sudo chown -R kube:nogroup /srv/kubernetes
$ sudo chmod 600 /srv/kubernetes/*

Generate a random token to authentication using bearer token for kubelet If you only want to use client certificate authentication, you can skip it.

$ TOKEN=$(dd if=/dev/urandom bs=128 count=1 2>/dev/null \
  | base64 |tr -d '=+/' |dd bs=32 count=1 2>/dev/null)

Create a token file for api server. This file will be used in /etc/kubernetes/apiserver (–token-auth-file option)

$ echo "${TOKEN},kubelet,kubelet" | \
  sudo tee /srv/kubernetes/tokens.csv

Edit the /etc/kubernetes/apiserver file.

KUBE_API_ARGS="--kubelet-https=true
--client-ca-file=/srv/kubernetes/ca.crt
--tls-cert-file=/srv/kubernetes/server.crt
--tls-private-key-file=/srv/kubernetes/server.key
--token-auth-file=/srv/kubernetes/tokens.csv"

Edit the /etc/kubernetes/controller-manager file.

KUBE_CONTROLLER_MANAGER_ARGS="--root-ca-file=/srv/kubernetes/ca.crt
--service-account-private-key-file=/srv/kubernetes/server.key
--node-monitor-grace-period=20s --pod-eviction-timeout=20s"

Restart kube-apiserver and kube-controller-manager service.

$ sudo systemctl restart kube-{apiserver,controller-manager}.service

Check if TCP 6443 port is opened by kube-apiserver.

$ sudo netstat -ntpl |grep 6443
tcp6  0    0    :::6443      :::*    LISTEN 7609/kube-apiserver

Generate each minion certificate.

$ MINIONS=$(echo kubeminion{1,2,3,4})
$ for MINION in $MINIONS; do \
    sudo openssl req -newkey rsa:2048 -nodes -keyout \
      /srv/kubernetes/${MINION}.key -subj "/CN=${MINION}" -out \
      /srv/kubernetes/${MINION}.csr; \
    sudo openssl x509 -req -days 10000 \
      -in /srv/kubernetes/${MINION}.csr \
      -CA /srv/kubernetes/ca.crt -CAkey /srv/kubernetes/ca.key \
      -CAcreateserial -out /srv/kubernetes/${MINION}.crt; \
  done

Make the certificate files secure again. Make only ‘kube’ user read the certificate files.

$ sudo chown -R kube:nogroup /srv/kubernetes
$ sudo chmod 600 /srv/kubernetes/*

Copy the CA and client certificates to the minions.

$ for MINION in $MINIONS; do \
  ssh ${MINION} sudo rm -fr /srv/kubernetes; \
  ssh ${MINION} sudo mkdir /srv/kubernetes; \
  sudo scp /srv/kubernetes/ca.crt \
    /srv/kubernetes/${MINION}.{crt,key} \
    jijisa@${MINION}:~/; \
  ssh ${MINION} sudo mv ~jijisa/${MINION}.{crt,key} ca.crt \
    /srv/kubernetes/; \
  ssh ${MINION} sudo chown -R root:root /srv/kubernetes/; \
  ssh ${MINION} sudo chmod 600 /srv/kubernetes/*; \
done

The dynamic duo(kubelet and kube-proxy) on each minion run as root user so the ownership of key directory should be owned by root only.

Kubernetes Minions

On each minion, configure the kubelet service to communicate securely with the apiserver. I’ll create kubeconfig file in /var/lib/kubelet on each minion.

$ KUBECONFIG=/var/lib/kubelet/kubeconfig
$ MINION=$(hostname)
$ TOKEN=<the_token_master_created>
$ CLUSTER=<cluster_name>
$ MASTER=<master_hostname>
$ sudo rm -f ${KUBECONFIG}
$ sudo kubectl config --kubeconfig=${KUBECONFIG} \
  set-cluster ${CLUSTER} \
  --server=https://${MASTER}:6443 \
  --certificate-authority=/srv/kubernetes/ca.crt --embed-certs=true
$ sudo kubectl config --kubeconfig=${KUBECONFIG} \
  set-credentials kubelet \
  --client-certificate=/srv/kubernetes/${MINION}.crt \
  --client-key=/srv/kubernetes/${MINION}.key --embed-certs=true \
  --token=${TOKEN}
$ sudo kubectl config --kubeconfig=${KUBECONFIG} \
  set-context kubelet-context --cluster=${CLUSTER} --user=kubelet
$ sudo kubectl config --kubeconfig=${KUBECONFIG} \
  use-context kubelet-context

In my lab, cluster_name is debian-cluster and master is kubernetes.default.svc.

Modify /etc/kubernetes/config file.

KUBE_MASTER="--master=https://kubemaster:6443"

Change /etc/kubernetes/kubelet file.

KUBELET_API_SERVER="--api-servers=https://kubemaster:6443"
KUBELET_ARGS="--cluster-dns=10.24.0.10 --cluster-domain=kubic.local
--enable_server=true --register-node=true
--kubeconfig=/var/lib/kubelet/kubeconfig
--node-status-update-frequency=5s"

Change /etc/kubernetes/proxy file.

KUBE_PROXY_ARGS="--kubeconfig=/var/lib/kubelet/kubeconfig"

Restart the kubernetes minion’s dynamic duo - kubelet and kube-proxy service.

$ sudo systemctl restart kube{let,-proxy}.service

Test

  • using client certificate. Verify you can securely access the kubernetes API using the client certificate.

    $ sudo curl -k --key /srv/kubernetes/${MINION}.key --cert \
    /srv/kubernetes/${MINION}.crt --cacert /srv/kubernetes/ca.crt \
    https://kubemaster:6443/version
    {
      "major": "1",
      "minor": "4",
      "gitVersion": "v1.4.2",
      "gitCommit": "c072cc2ee4dfc21b60196e5994c56c40e1c29b93",
      "gitTreeState": "clean",
      "buildDate": "2016-10-15T00:07:23Z",
      "goVersion": "go1.6.3",
      "compiler": "gc",
      "platform": "linux/amd64"
    }
    

    -k option is needed because the certificate is self-signed.

    -k, --insecure: (SSL) CA certificate self-signed.
    
  • using token

    $ curl -k -H "Authorization: Bearer ${TOKEN}" \
    https://kubemaster:6443/version
    

References