Container Network Interfaceについて調べた

社内のKubernetes勉強会で, 前より関心のあった Container Network Interface (CNI) の説明を することになった。その際の資料をここに書いておく。

Container Network Interface (CNI) の基本

CNIとは何か

CNIが生まれてきた背景

CNI plugins

CNI仕様

graph TD subgraph host subgraph container_A if_a1[NIC] if_a2[NIC] end subgraph container_B if_b1[NIC] end end nw1((network 1)) nw2((network 2)) conf1(config) conf2(config) r[runtime] cni[CNI plugin] ipam[IPAM plugin] nw1 --- conf1 nw2 --- conf2 r == invoke ==> cni cni == invoke ==> ipam conf1 -. indicate .-> cni conf1 -. indicate .-> ipam if_a1 --- nw1 if_b1 --- nw2 if_a2 --- nw2 cni -- create --> if_a1 ipam -- setup --> if_a1
sequenceDiagram participant Kernel participant Runtime participant Plugin1 participant IPAM Plugin participant Plugin2 Runtime->>Kernel: create netns Runtime->>Plugin1: "ADD" (netns, I/F name, network config, ...) Plugin1->>IPAM Plugin: "ADD" (netns, I/F name, network config, ...) IPAM Plugin-->>Plugin1: Result Plugin1-->>Runtime: Result Runtime->>Plugin2: "ADD" (netns, I/F name, network config(+prevResult), ...) Plugin2-->>Runtime: Result

実験: Docker container に CNI plugin でインターフェイスを足す

  1. テスト用のbridgeインターフェイスをhost上に準備

    $ sudo ip link add test-bridge type bridge

$ ip link show test-bridge
23: test-bridge: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 46:16:ad:0e:cd:6a brd ff:ff:ff:ff:ff:ff

  2. CNI pluginの準備

    $ git clone https://github.com/containernetworking/plugins.git

$ cd plugins && ./build_linux.sh

  3. テスト用コンテナを動かす(別のターミナルなどで)

    $ docker run -it --rm --name cni_test alpine:latest /bin/sh -c 'watch -n 5 ip addr'

Every 5s: ip addr                                           2019-01-14 13:28:33

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
        valid_lft forever preferred_lft forever
2: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
20: eth0@if21: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP 
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
        valid_lft forever preferred_lft forever

  4. テスト用コンテナのnetnsにアクセスしてみる

    $ docker inspect cni_test | jq -r '.[].NetworkSettings.SandboxKey'
/var/run/docker/netns/bf1e67c802ee

$ sudo mkdir -p /var/run/netns

$ sudo ln -s /var/run/docker/netns/bf1e67c802ee /var/run/netns/cni_test

$ sudo ip netns exec cni_test ip addr

  5. CNI pluginを呼び出してみる

    $ echo '{"cniVersion": "0.4.0", "name": "test", "type": "bridge", "bridge": "test-bridge", "ipam": { "type": "static", "addresses": [ { "address": "192.168.0.42/24" } ] } }' | CNI_COMMAND=ADD CNI_CONTAINERID=hoge CNI_NETNS=/var/run/docker/netns/bf1e67c802ee CNI_IFNAME=test-if CNI_PATH=$PWD/bin sudo -E $PWD/bin/bridge
{
    "cniVersion": "0.4.0",
    "interfaces": [
        {
            "name": "test-bridge",
            "mac": "32:b2:ce:3d:ec:f1"
        },
        {
            "name": "veth38f6229d",
            "mac": "32:b2:ce:3d:ec:f1"
        },
        {
            "name": "test-if",
            "mac": "c6:26:ea:d0:71:8f",
            "sandbox": "/var/run/docker/netns/bf1e67c802ee"
        }
    ],
    "ips": [
        {
            "version": "4",
            "interface": 2,
            "address": "192.168.0.42/24"
        }
    ],
    "dns": {}
}

  6. コンテナ内にインターフェイスが増えていることを確認(手順3の出力を再度確認)

    Every 5s: ip addr                                                                                2019-01-14 13:56:54

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
4: test-if@if24: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP 
    link/ether c6:26:ea:d0:71:8f brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.42/24 scope global test-if
       valid_lft forever preferred_lft forever
20: eth0@if21: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP 
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever

Kubernetes と CNI

Kubernetesのネットワーク

CNIの利用

利用方法

CNI plugin 各論

Flannel

Project Calico

Multus

参考にした情報源まとめ