Docker networking: Flannel

By | 20/12/2016

Introduction

Flannel is a networking technology used to connect containers. It is distributed and maintained by CoreOS and was originally designed for Kubernetes. It is a generic overlay network that can be used as a simple alternative to existing docker networking solutions. It provides a configurable virtual overlay network for Docker.

There are many different ways to network containers, all with different architectural approaches. Docker’s native networking scheme creates a virtual layer 3 Ethernet bridge, which automatically forwards packets between containers through a subnet routing scheme. Flannel on the other side is a basic overlay network that works by assigning a range of subnet addresses. Each address corresponds to a container, so that all containers in a system may reside on different hosts. By using packet encapsulation, Flannel enables the entire span of hosts to be addressed, by assigning a separate subnet to each host. Flannel uses the open source etcd key/value store to record the mappings between the addresses assigned to containers by their native hosts, and their addresses in the overlay network.

In all honesty, I struggled to get Flannel installed on my hosts. For some reason, yet unknown to me, every attempt to install Flannel on an Ubuntu host just failed. I logged an issue here so let’s see what comes out of it.

As I just wanted to get my hands dirty with Flannel, I have re-used this post to get a CoreOS cluster up and running easily. In this post, I will look more into the Flannel specifics though.

Looking at the IP configuration

It’s always interesting to have a look at the ip configuration. SSH into the CoreOS hosts and have a look at it:

On the first host:

core@node-01 ~ $ ifconfig
eth0: flags=4163  mtu 1500
        inet 10.0.2.15  netmask 255.255.255.0  broadcast 10.0.2.255
        inet6 fe80::a00:27ff:fe7b:5175  prefixlen 64  scopeid 0x20
        ether 08:00:27:7b:51:75  txqueuelen 1000  (Ethernet)
        RX packets 6897  bytes 8445671 (8.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3634  bytes 245544 (239.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1: flags=4163  mtu 1500
        inet 172.17.8.101  netmask 255.255.255.0  broadcast 172.17.8.255
        inet6 fe80::a00:27ff:fe19:656b  prefixlen 64  scopeid 0x20
        ether 08:00:27:19:65:6b  txqueuelen 1000  (Ethernet)
        RX packets 2885  bytes 288770 (282.0 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2908  bytes 289173 (282.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

flannel0: flags=4305  mtu 1472
        inet 10.1.16.0  netmask 255.255.0.0  destination 10.1.16.0
        inet6 fe80::a98a:9a9f:4164:8419  prefixlen 64  scopeid 0x20
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 500  (UNSPEC)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3  bytes 144 (144.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10
        loop  txqueuelen 1  (Local Loopback)
        RX packets 901  bytes 227935 (222.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 901  bytes 227935 (222.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

On the second host:

core@node-02 ~ $ ifconfig
eth0: flags=4163  mtu 1500
        inet 10.0.2.15  netmask 255.255.255.0  broadcast 10.0.2.255
        inet6 fe80::a00:27ff:fe7b:5175  prefixlen 64  scopeid 0x20
        ether 08:00:27:7b:51:75  txqueuelen 1000  (Ethernet)
        RX packets 6887  bytes 8444854 (8.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3459  bytes 235545 (230.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1: flags=4163  mtu 1500
        inet 172.17.8.102  netmask 255.255.255.0  broadcast 172.17.8.255
        inet6 fe80::a00:27ff:fe26:5cce  prefixlen 64  scopeid 0x20
        ether 08:00:27:26:5c:ce  txqueuelen 1000  (Ethernet)
        RX packets 3024  bytes 300022 (292.9 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3028  bytes 302154 (295.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

flannel0: flags=4305  mtu 1472
        inet 10.1.12.0  netmask 255.255.0.0  destination 10.1.12.0
        inet6 fe80::1a69:f9ce:1e7:dd97  prefixlen 64  scopeid 0x20
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 500  (UNSPEC)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3  bytes 144 (144.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10
        loop  txqueuelen 1  (Local Loopback)
        RX packets 623  bytes 109571 (107.0 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 623  bytes 109571 (107.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

You notice immediately that a flannel0 interface is available. This is the flannel overlay network that has been created automatically (using our Vagrantfile). You can see here also that containers on node 01 will get IP addresses in the 10.1.16.0 network, while containers on node 02 will get IP addresses in the 10.1.12.0 network.

So wait, where was that defined? The easier answer is that it is defined in the user-data file. You can see that in the flannel section we have defined a network 10.1.0.0/16.

- name: flanneld.service
    drop-ins:
    - name: 50-network-config.conf
      content: |
        [Service]
        ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{ "Network": "10.1.0.0/16" }'
    command: start

Another way to verify this is to retrieve the information from the etcd key-value store.

core@node-01 ~ $ /usr/bin/etcdctl get /coreos.com/network/config
{ "Network": "10.1.0.0/16" }

By default, flannel assigns a /24 to each host. So far, it is unclear to me why it selects 10.1.12.0 and 10.1.16.0 initially. Need to reserve some time to find out.

Leave a Reply

Your email address will not be published. Required fields are marked *