Monthly Archives: August 2017

Installing Kubernetes via Kubeadm on Ubuntu


We have already spent quite some posts on getting a Kubernetes cluster up and running. This post describes the installation of a single node cluster using Minikube (a Single node Kubernetes cluster) and using a CoreOS setup. It also addresses how to setup a multinode cluster using CoreOS. In this post, we used this CoreOS multi-node setup to get acquainted with Kubernetes a bit.

Why spending time on a new post to bring up a multinode Kubernetes cluster then? Well, while the setup in these previous posts work well, I wanted to understand a bit how this KubeAdm tool was working, because it seems to become the default tool to install Kubernetes clusters. In addition, I want (you’ll see that in future posts) to experiment a bit with Contiv which is container networking plugin that is compatible with Kubernetes. As a prerequisite it states it needs to run on either Ubuntu or Centos. As KubeAdm also works nicely with Ubuntu and Centos it seems to be the right way to move.

Anyway, let’s get started!

Description of our setup

Note: we will be largely following this guide.

We would like to setup a Kubernetes cluster using 1 Kubernetes master and 2 Kubernetes worker nodes. I will be running the Kubernetes cluster on 3 Ubuntu 16.04 virtual servers running on a vSphere setup (ESX).

So first, we will create 3 Ubuntu server hosts. In this setup, they will get the following IP addresses:

  • Kubernetes Master (
  • Kubernetes Worker 1 (
  • Kubernetes Worker 2 (

Once the Ubuntu servers are installed, ensure that the servers can reach each other via their domainname. To achieve that, I have simply changed the /etc/hosts file on each server.

wim@k8s-master:~$ cat /etc/hosts   localhost   k8s-master

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Change the /etc/hosts files also on the other two worker nodes and ensure you can ping all the machines.

Installing Docker on all hosts

Next, we can install Docker on each server. While the documentation states one can use simply the apt-get method to install docker, I have deliberately chosen not to do it as this would install Docker version 12 onto your machines while Docker has released version 17 in the meantime already.
So don’t use the following snippet:

wim@k8s-master:~$ apt-get update
wim@k8s-master:~$ apt-get install -y

But instead use the following method:

wim@k8s-master:~$ sudo su
root@k8s-master:/home/wim# apt-get update && apt-get install -y curl apt-transport-https
Hit:1 xenial InRelease
Hit:2 xenial-updates InRelease
Get:3 xenial-backports InRelease [102 kB]
Hit:4 xenial-security InRelease
Fetched 102 kB in 0s (338 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
apt-transport-https is already the newest version (1.2.24).
curl is already the newest version (7.47.0-1ubuntu2.2).
The following packages were automatically installed and are no longer required:
  bridge-utils cgroupfs-mount containerd runc ubuntu-fan
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
root@k8s-worker1:/home/wim# curl -fsSL | apt-key add -
root@k8s-master:/home/wim# cat </etc/apt/sources.list.d/docker.list
> deb$(lsb_release -si | tr '[:upper:]' '[:lower:]') $(lsb_release -cs) stable
root@k8s-master:/home/wim# apt-get update && apt-get install -y docker-ce=$(apt-cache madison docker-ce | grep 17.03 | head -1 | awk '{print $3}')
Get:1 xenial InRelease [49.8 kB]
Hit:2 xenial InRelease
Hit:3 xenial-security InRelease
Hit:4 xenial-updates InRelease
Get:5 xenial-backports InRelease [102 kB]
Get:6 xenial/stable amd64 Packages [2,579 B]
Fetched 155 kB in 0s (219 kB/s)
Reading package lists... Done

If all went well, you will end up with Docker version 17 on the hosts:

wim@k8s-master:~$ docker -v
Docker version 17.03.2-ce, build f5ec1e2

Installing Kubeadm on all hosts

In this section, we will install some Kubernetes related tools:

  • Kubeadm: the installer which will bootstrap a Kubernetes cluster
  • Kubelet: runs on all machines and ensure we can run pods etc
  • Kubectl: CLI tool to talk to the Kubernetes cluster
root@k8s-master:/home/wim# curl -s | apt-key add -
root@k8s-master:/home/wim# cat </etc/apt/sources.list.d/kubernetes.list
> deb kubernetes-xenial main
root@k8s-master:/home/wim# apt-get update
Hit:1 xenial-security InRelease
Hit:2 xenial InRelease
Hit:3 xenial-updates InRelease
Get:4 xenial-backports InRelease [102 kB]
Hit:5 xenial InRelease
Get:6 kubernetes-xenial InRelease [8,942 B]
Get:7 kubernetes-xenial/main amd64 Packages [5,776 B]
Fetched 117 kB in 1s (77.8 kB/s)
Reading package lists... Done
root@k8s-master:/home/wim# apt-get install -y kubelet kubeadm kubectl
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  ebtables kubernetes-cni socat
The following NEW packages will be installed:
  ebtables kubeadm kubectl kubelet kubernetes-cni socat
0 upgraded, 6 newly installed, 0 to remove and 1 not upgraded.
Need to get 51.9 MB of archives.
After this operation, 370 MB of additional disk space will be used.
Get:1 xenial-updates/main amd64 ebtables amd64 [79.4 kB]
Get:2 xenial/universe amd64 socat amd64 [321 kB]
Get:3 kubernetes-xenial/main amd64 kubernetes-cni amd64 0.5.1-00 [5,560 kB]

When finished, all the Kubernetes related tools should be installed. At the end of the output, it will mention that you need to run some commands as a regular user:

Initialize the Kubernetes master

Now that kubeadm is installed, it’s time to bring up the Kubernetes cluster. As the documentation states, you should execute the ‘kubeadm init’ command. However, when I tried to do this, it would complain that certain directories are not empty. Therefore, we will first execute the ‘kubeadm reset’ command. In your setup, I would simply start with ‘kubeadm init’. If it passes the preflight checks you should be fine.

root@k8s-master:/home/wim# kubeadm reset

[preflight] Running pre-flight checks
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Removing kubernetes-managed containers
[reset] No etcd manifest found in "/etc/kubernetes/manifests/etcd.yaml", assuming external etcd.
[reset] Deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes]
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]

root@k8s-master:/home/wim# kubeadm init
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.8.0
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks
Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run (as a regular user):

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:

As indicated in the output of the previous command, execute the commands to start using the cluster.

wim@k8s-master:~$ mkdir -p $HOME/.kube
wim@k8s-master:~$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[sudo] password for wim:
wim@k8s-master:~$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

Add the workers to the cluster

Next, we will need to add our workers to the Kubernetes cluster. The ‘kubeadm join’ command to use will have been output as part of the ‘kubeadm init’ command previously. There is really nothing more to it than executing the command:

root@k8s-worker1:/home/wim# kubeadm join --token 4c327b.5ea7f6d3aea2c700 --discovery-token-ca-cert-hash sha256:79a7c498eeae680e620cf5865e1cef4388a5ce1ead5c42ceb65de6792c17fa31
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[preflight] Running pre-flight checks
[preflight] Starting the kubelet service
[discovery] Trying to connect to API Server ""
[discovery] Created cluster-info discovery client, requesting info from ""
[discovery] Requesting info from "" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server ""
[discovery] Successfully established connection with API Server ""
[bootstrap] Detected server version: v1.8.0
[bootstrap] The server supports the Certificates API (

Node join complete:
* Certificate signing request sent to master and response
* Kubelet informed of new secure connection details.

Run 'kubectl get nodes' on the master to see this machine join.

Additional steps

As indicated above, we should ensure that our workers have been added to the cluster. We can do this by executing some simple Kubernetes commands:

wim@k8s-master:~$ kubectl get nodes
NAME          STATUS     ROLES     AGE       VERSION
k8s-master    Ready      master    2h        v1.8.0
k8s-worker1   Ready          1m        v1.8.0
k8s-worker2   NotReady       18s       v1.8.0

In the above output you will see that the status of the second worker is still NotReady. Usually, it takes about a minute before the status will show Ready.

wim@k8s-master:~$ kubectl get nodes
NAME          STATUS    ROLES     AGE       VERSION
k8s-master    Ready     master    2h        v1.8.0
k8s-worker1   Ready         2m        v1.8.0
k8s-worker2   Ready         1m        v1.8.0