系统环境
| Node | IP | CentOS | kernel | cpu | memory |
|---|---|---|---|---|---|
| node1 | 192.168.159.4 | CentOS Linux release 7.4.1708 (Core) | 3.10.0-693.el7.x86_64 | Intel® Core™ i5-7500 CPU @ 3.40GHz * 1 | 2G |
| node2 | 192.168.159.5 | CentOS Linux release 7.4.1708 (Core) | 3.10.0-693.el7.x86_64 | Intel® Core™ i5-7500 CPU @ 3.40GHz * 1 | 2G |
Node 软件环境
| Node | IP | docker | flannel |
|---|---|---|---|
| node1 | 192.168.159.4 | 19.03.1 | 0.11.0 |
| node2 | 192.168.159.5 | 19.03.1 | 0.11.0 |
Docker 安装
docker 下载
wget https://download.docker.com/linux/static/stable/x86_64/docker-19.03.1.tgz
tar -zxvf docker-19.03.1.tgz
cp -f docker/* /usr/local/bin/
docker 配置文件
cat > /opt/k8s/node/etc/docker.conf << "EOF"
DOCKER_NETWORK_OPTIONS="-H unix:///var/run/docker.sock \
-H 0.0.0.0:2375"
"EOF"
docker 服务文件
cat > /usr/lib/systemd/system/docker.service << EOF
[Unit]
Description=Docker Engine Service
Documentation=https://docs.docker.com
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin"
EnvironmentFile=-/opt/k8s/node/etc/docker.conf
ExecStart=/usr/local/bin/dockerd $DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
Restart=on-failure
RestartSec=5
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Delegate=yes
KillMode=process
[Install]
WantedBy=multi-user.target
EOF
docker 服务启动
启动
systemctl daemon-reload && systemctl start docker
查看
查看docker版本信息
[root@node1 k8s]# docker version
Client: Docker Engine - Community
Version: 19.03.1
API version: 1.40
Go version: go1.12.5
Git commit: 74b1e89e8a
Built: Thu Jul 25 21:17:37 2019
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.1
API version: 1.40 (minimum version 1.12)
Go version: go1.12.5
Git commit: 74b1e89e8a
Built: Thu Jul 25 21:27:55 2019
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.2.6
GitCommit: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc:
Version: 1.0.0-rc8
GitCommit: 425e105d5a03fabd737a126ad93d62a9eeede87f
docker-init:
Version: 0.18.0
GitCommit: fec3683
查看本地镜像
[root@node1 docker]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
查看容器
[root@node1 docker]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
验证
[root@node1 pki]# docker search centos
NAME DESCRIPTION STARS OFFICIAL AUTOMATED
centos The official build of CentOS. 5524 [OK]
ansible/centos7-ansible Ansible on Centos7 122 [OK]
jdeathe/centos-ssh CentOS-6 6.10 x86_64 / CentOS-7 7.6.1810 x86… 111 [OK]
consol/centos-xfce-vnc Centos container with "headless" VNC session… 99 [OK]
centos/mysql-57-centos7 MySQL 5.7 SQL database server 62
imagine10255/centos6-lnmp-php56 centos6-lnmp-php56 57 [OK]
tutum/centos Simple CentOS docker image with SSH access 44
centos/postgresql-96-centos7 PostgreSQL is an advanced Object-Relational … 39
kinogmt/centos-ssh CentOS with SSH 29 [OK]
pivotaldata/centos-gpdb-dev CentOS image for GPDB development. Tag names… 10
guyton/centos6 From official centos6 container with full up… 9 [OK]
drecom/centos-ruby centos ruby 6 [OK]
pivotaldata/centos Base centos, freshened up a little with a Do… 3
mamohr/centos-java Oracle Java 8 Docker image based on Centos 7 3 [OK]
darksheer/centos Base Centos Image -- Updated hourly 3 [OK]
pivotaldata/centos-gcc-toolchain CentOS with a toolchain, but unaffiliated wi… 2
pivotaldata/centos-mingw Using the mingw toolchain to cross-compile t… 2
ovirtguestagent/centos7-atomic The oVirt Guest Agent for Centos 7 Atomic Ho… 2
miko2u/centos6 CentOS6 日本語環境 2 [OK]
mcnaughton/centos-base centos base image 1 [OK]
indigo/centos-maven Vanilla CentOS 7 with Oracle Java Developmen… 1 [OK]
blacklabelops/centos CentOS Base Image! Built and Updates Daily! 1 [OK]
smartentry/centos centos with smartentry 0 [OK]
pivotaldata/centos7-dev CentosOS 7 image for GPDB development 0
pivotaldata/centos6.8-dev CentosOS 6.8 image for GPDB development 0
[root@node1 pki]# docker pull centos
Using default tag: latest
latest: Pulling from library/centos
d8d02d457314: Pull complete
Digest: sha256:307835c385f656ec2e2fec602cf093224173c51119bbebd602c53c3653a3d6eb
Status: Downloaded newer image for centos:latest
docker.io/library/centos:latest
[root@node1 pki]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
centos latest 67fa590cfc1c 2 days ago 202MB
[root@node1 pki]# docker run -i -t --name test centos /bin/bash
WARNING: IPv4 forwarding is disabled. Networking will not work.
docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \"write /proc/self/attr/keycreate: permission denied\"": unknown.
[root@node1 pki]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5ba3f8995eff centos "/bin/bash" 7 minutes ago Created
sed -i '$a\net.ipv4.ip_forward=1' /usr/lib/sysctl.d/00-system.conf
systemctl daemon-reload && systemctl restart network
docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \"write /proc/self/attr/keycreate: permission denied\"": unknown.
[root@node1 pki]# getenforce
Enforcing
setenforce 0
[root@node1 pki]# getenforce
Permissive
[root@node1 pki]# docker start test
test
[root@node1 pki]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5ba3f8995eff centos "/bin/bash" 14 minutes ago Up 14 seconds
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
reboot
systemctl start docker
docker start test
[root@node1 pki]# getenforce
Disabled
Docker常用命令介绍
docker attach test
Docker跨主机集群网络配置
Node节点网卡查看
docker0[root@node1 ~]# ifconfig
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255
inet6 fe80::42:c5ff:fed3:c41d prefixlen 64 scopeid 0x20<link>
ether 02:42:c5:d3:c4:1d txqueuelen 0 (Ethernet)
RX packets 2901 bytes 119827 (117.0 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3839 bytes 15022256 (14.3 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.159.4 netmask 255.255.255.0 broadcast 192.168.159.255
inet6 fe80::69a7:b207:afb9:9db4 prefixlen 64 scopeid 0x20<link>
inet6 fe80::2b45:985c:3f4e:908b prefixlen 64 scopeid 0x20<link>
ether 00:50:56:34:3a:20 txqueuelen 1000 (Ethernet)
RX packets 65971 bytes 19511878 (18.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 56734 bytes 3494791 (3.3 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 32 bytes 2592 (2.5 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 32 bytes 2592 (2.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
veth9d114c0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::80e1:39ff:fe8e:240e prefixlen 64 scopeid 0x20<link>
ether 82:e1:39:8e:24:0e txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8 bytes 656 (656.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
docker0[root@node2 ~]# ifconfig
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255
inet6 fe80::42:1aff:fe5e:9ea4 prefixlen 64 scopeid 0x20<link>
ether 02:42:1a:5e:9e:a4 txqueuelen 0 (Ethernet)
RX packets 13 bytes 980 (980.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 21 bytes 2014 (1.9 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.159.5 netmask 255.255.255.0 broadcast 192.168.159.255
inet6 fe80::69a7:b207:afb9:9db4 prefixlen 64 scopeid 0x20<link>
ether 00:50:56:24:97:9a txqueuelen 1000 (Ethernet)
RX packets 150770 bytes 220073564 (209.8 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8739 bytes 781644 (763.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 64 bytes 5568 (5.4 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 64 bytes 5568 (5.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
veth7fa5fd1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::6c8e:4dff:fed4:5761 prefixlen 64 scopeid 0x20<link>
ether 6e:8e:4d:d4:57:61 txqueuelen 0 (Ethernet)
RX packets 13 bytes 1162 (1.1 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 21 bytes 2014 (1.9 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
查看node1的docker网络
[root@node1 ~]# docker network ls
NETWORK ID NAME DRIVER SCOPE
899aadb08971 bridge bridge local
2b6bb260df1e host host local
f227670e6a5b none null local
bridgehostnoneoverlay查看node2的docker网络
[root@node2 ~]# docker network ls
NETWORK ID NAME DRIVER SCOPE
16f0b5194fdb bridge bridge local
593efbd057ff host host local
e9e9bbc0bef6 none null local
测试跨主机容器互访
在node1节点启动centos镜像容器node1
[root@0402a3fb29e6 /]# cat /etc/hosts
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.17.0.2 0402a3fb29e6
在node2节点启动centos镜像容器node2
[root@bc6194e6391e /]# cat /etc/hosts
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.17.0.2 bc6194e6391e
主机网络互访
[root@node1 ~]# ping node2
PING node2 (192.168.159.5) 56(84) bytes of data.
64 bytes from node2 (192.168.159.5): icmp_seq=1 ttl=64 time=0.252 ms
64 bytes from node2 (192.168.159.5): icmp_seq=2 ttl=64 time=0.463 ms
64 bytes from node2 (192.168.159.5): icmp_seq=3 ttl=64 time=0.374 ms
64 bytes from node2 (192.168.159.5): icmp_seq=4 ttl=64 time=0.242 ms
64 bytes from node2 (192.168.159.5): icmp_seq=5 ttl=64 time=0.495 ms
[root@node2 ~]# ping node1
PING node1 (192.168.159.4) 56(84) bytes of data.
64 bytes from node1 (192.168.159.4): icmp_seq=1 ttl=64 time=0.557 ms
64 bytes from node1 (192.168.159.4): icmp_seq=2 ttl=64 time=0.441 ms
64 bytes from node1 (192.168.159.4): icmp_seq=3 ttl=64 time=0.413 ms
64 bytes from node1 (192.168.159.4): icmp_seq=4 ttl=64 time=0.284 ms
64 bytes from node1 (192.168.159.4): icmp_seq=5 ttl=64 time=0.301 ms
容器访问主机
[root@0402a3fb29e6 /]# ping 192.168.159.4
PING 192.168.159.4 (192.168.159.4) 56(84) bytes of data.
64 bytes from 192.168.159.4: icmp_seq=1 ttl=64 time=0.261 ms
64 bytes from 192.168.159.4: icmp_seq=2 ttl=64 time=0.250 ms
64 bytes from 192.168.159.4: icmp_seq=3 ttl=64 time=0.129 ms
64 bytes from 192.168.159.4: icmp_seq=4 ttl=64 time=0.126 ms
64 bytes from 192.168.159.4: icmp_seq=5 ttl=64 time=0.129 ms
[root@0402a3fb29e6 /]# ping 192.168.159.5
PING 192.168.159.5 (192.168.159.5) 56(84) bytes of data.
64 bytes from 192.168.159.5: icmp_seq=1 ttl=63 time=1.28 ms
64 bytes from 192.168.159.5: icmp_seq=2 ttl=63 time=0.310 ms
64 bytes from 192.168.159.5: icmp_seq=3 ttl=63 time=0.724 ms
64 bytes from 192.168.159.5: icmp_seq=4 ttl=63 time=0.523 ms
64 bytes from 192.168.159.5: icmp_seq=5 ttl=63 time=0.564 ms
[root@bc6194e6391e /]# ping 192.168.159.5
PING 192.168.159.5 (192.168.159.5) 56(84) bytes of data.
64 bytes from 192.168.159.5: icmp_seq=1 ttl=64 time=0.283 ms
64 bytes from 192.168.159.5: icmp_seq=2 ttl=64 time=0.210 ms
64 bytes from 192.168.159.5: icmp_seq=3 ttl=64 time=0.129 ms
64 bytes from 192.168.159.5: icmp_seq=4 ttl=64 time=0.122 ms
64 bytes from 192.168.159.5: icmp_seq=5 ttl=64 time=0.258 ms
[root@bc6194e6391e /]# ping 192.168.159.4
PING 192.168.159.4 (192.168.159.4) 56(84) bytes of data.
64 bytes from 192.168.159.4: icmp_seq=1 ttl=63 time=0.571 ms
64 bytes from 192.168.159.4: icmp_seq=2 ttl=63 time=0.356 ms
64 bytes from 192.168.159.4: icmp_seq=3 ttl=63 time=0.609 ms
64 bytes from 192.168.159.4: icmp_seq=4 ttl=63 time=0.607 ms
64 bytes from 192.168.159.4: icmp_seq=5 ttl=63 time=0.417 ms
容器互访
ping -c 3 172.17.0.2tcpdumpdocker0veth*ens33tcpdump -i docker0 -n icmp直接路由方式修改node1主机的Docker0的网络地址
[root@node1 ~]# cat /etc/docker/daemon.json
{
"bip": "172.17.1.1/24"
}
[root@node1 ~]# ifconfig
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.17.1.1 netmask 255.255.255.0 broadcast 172.17.1.255
inet6 fe80::42:23ff:fe43:87a4 prefixlen 64 scopeid 0x20<link>
ether 02:42:23:43:87:a4 txqueuelen 0 (Ethernet)
RX packets 2068 bytes 97452 (95.1 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2847 bytes 17131048 (16.3 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.159.4 netmask 255.255.255.0 broadcast 192.168.159.255
inet6 fe80::69a7:b207:afb9:9db4 prefixlen 64 scopeid 0x20<link>
ether 00:50:56:34:3a:20 txqueuelen 1000 (Ethernet)
RX packets 474960 bytes 52578160 (50.1 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 461729 bytes 28211425 (26.9 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
vethf5fadcf: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::404e:afff:feec:c983 prefixlen 64 scopeid 0x20<link>
ether 42:4e:af:ec:c9:83 txqueuelen 0 (Ethernet)
RX packets 187 bytes 17934 (17.5 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 15 bytes 950 (950.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@0402a3fb29e6 /]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.17.1.2 netmask 255.255.255.0 broadcast 172.17.1.255
ether 02:42:ac:11:01:02 txqueuelen 0 (Ethernet)
RX packets 8 bytes 656 (656.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
修改node2主机的Docker0的网络地址
[root@node1 ~]# cat /etc/docker/daemon.json
{
"bip": "172.17.0.1/24"
}
[root@node2 ~]# ifconfig
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.255.0 broadcast 172.17.0.255
inet6 fe80::42:a6ff:feb7:78bf prefixlen 64 scopeid 0x20<link>
ether 02:42:a6:b7:78:bf txqueuelen 0 (Ethernet)
RX packets 4121 bytes 170163 (166.1 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 4782 bytes 17294541 (16.4 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.159.5 netmask 255.255.255.0 broadcast 192.168.159.255
inet6 fe80::69a7:b207:afb9:9db4 prefixlen 64 scopeid 0x20<link>
inet6 fe80::2b45:985c:3f4e:908b prefixlen 64 scopeid 0x20<link>
inet6 fe80::c846:2166:96b5:5f86 prefixlen 64 scopeid 0x20<link>
ether 00:50:56:24:97:9a txqueuelen 1000 (Ethernet)
RX packets 18895 bytes 18246491 (17.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8037 bytes 815344 (796.2 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 68 bytes 5904 (5.7 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 68 bytes 5904 (5.7 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
veth09bb648: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::9069:b6ff:fed1:8797 prefixlen 64 scopeid 0x20<link>
ether 92:69:b6:d1:87:97 txqueuelen 0 (Ethernet)
RX packets 4 bytes 280 (280.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 12 bytes 936 (936.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
容器node1访问容器node2
无法访问
[root@0402a3fb29e6 /]# ping -c 3 172.17.0.2
PING 172.17.0.2 (172.17.0.2) 56(84) bytes of data.
--- 172.17.0.2 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2000ms
tcpdumpens33[root@node1 ~]# tcpdump -i ens33 -n icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens33, link-type EN10MB (Ethernet), capture size 262144 bytes
17:12:19.636080 IP 192.168.159.4 > 172.17.0.2: ICMP echo request, id 37, seq 1, length 64
17:12:20.640152 IP 192.168.159.4 > 172.17.0.2: ICMP echo request, id 37, seq 2, length 64
17:12:21.640212 IP 192.168.159.4 > 172.17.0.2: ICMP echo request, id 37, seq 3, length 64
容器node2访问容器node1
无法访问
[root@bc6194e6391e /]# ping -c 3 172.17.1.2
PING 172.17.1.2 (172.17.1.2) 56(84) bytes of data.
From 172.17.0.2 icmp_seq=1 Destination Host Unreachable
From 172.17.0.2 icmp_seq=2 Destination Host Unreachable
From 172.17.0.2 icmp_seq=3 Destination Host Unreachable
--- 172.17.1.2 ping statistics ---
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 1999ms
pipe 3
添加静态路由
[root@node1 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.159.2 0.0.0.0 UG 100 0 0 ens33
172.17.0.0 192.168.159.5 255.255.255.0 UG 0 0 0 ens33
172.17.1.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0
192.168.159.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33
cat > /etc/sysconfig/static-routes << EOF
any net 172.17.0.0 netmask 255.255.255.0 gw 192.168.159.5 dev ens33
EOF
[root@node2 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.159.2 0.0.0.0 UG 100 0 0 ens33
172.17.0.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0
172.17.1.0 192.168.159.4 255.255.255.0 UG 0 0 0 ens33
192.168.159.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33
cat > /etc/sysconfig/static-routes << EOF
any net 172.17.1.0 netmask 255.255.255.0 gw 192.168.159.4 dev ens33
EOF
容器node1访问容器node2
[root@0402a3fb29e6 /]# ping 172.17.0.2 -c 3
PING 172.17.0.2 (172.17.0.2) 56(84) bytes of data.
64 bytes from 172.17.0.2: icmp_seq=1 ttl=62 time=1.34 ms
64 bytes from 172.17.0.2: icmp_seq=2 ttl=62 time=0.934 ms
64 bytes from 172.17.0.2: icmp_seq=3 ttl=62 time=0.425 ms
--- 172.17.0.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 0.425/0.902/1.347/0.377 ms
[root@bc6194e6391e /]# tcpdump -i eth0 -n icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:49:31.700980 IP 192.168.159.4 > 172.17.0.2: ICMP echo request, id 46, seq 1, length 64
09:49:31.701198 IP 172.17.0.2 > 192.168.159.4: ICMP echo reply, id 46, seq 1, length 64
09:49:32.703187 IP 192.168.159.4 > 172.17.0.2: ICMP echo request, id 46, seq 2, length 64
09:49:32.703231 IP 172.17.0.2 > 192.168.159.4: ICMP echo reply, id 46, seq 2, length 64
09:49:33.704675 IP 192.168.159.4 > 172.17.0.2: ICMP echo request, id 46, seq 3, length 64
09:49:33.704695 IP 172.17.0.2 > 192.168.159.4: ICMP echo reply, id 46, seq 3, length 64
[root@bc6194e6391e /]# ping 172.17.1.2 -c 3
PING 172.17.1.2 (172.17.1.2) 56(84) bytes of data.
64 bytes from 172.17.1.2: icmp_seq=1 ttl=62 time=1.69 ms
64 bytes from 172.17.1.2: icmp_seq=2 ttl=62 time=0.987 ms
64 bytes from 172.17.1.2: icmp_seq=3 ttl=62 time=1.15 ms
--- 172.17.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 0.987/1.279/1.695/0.304 ms
[root@0402a3fb29e6 /]# tcpdump -i eth0 -n icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:52:51.691610 IP 192.168.159.5 > 172.17.1.2: ICMP echo request, id 21, seq 1, length 64
09:52:51.691738 IP 172.17.1.2 > 192.168.159.5: ICMP echo reply, id 21, seq 1, length 64
09:52:52.693258 IP 192.168.159.5 > 172.17.1.2: ICMP echo request, id 21, seq 2, length 64
09:52:52.693361 IP 172.17.1.2 > 192.168.159.5: ICMP echo reply, id 21, seq 2, length 64
09:52:53.695265 IP 192.168.159.5 > 172.17.1.2: ICMP echo request, id 21, seq 3, length 64
09:52:53.695350 IP 172.17.1.2 > 192.168.159.5: ICMP echo reply, id 21, seq 3, length 64
通过配置静态路由的方式达到了跨主机docker容器的网络互访，但是在大规模集群中手动配置过于低效，下面将进一步进行优化。
使用fannel配置docker集群网络
环境准备
[root@node1 pki]# etcdctl -ca-file=ca.pem -cert-file=etcdctl.pem -key-file=etcdctl-key.pem cluster-health
member 46899d42c87d524e is healthy: got healthy result from https://192.168.159.4:2379
member a3ec213779ea2c81 is healthy: got healthy result from https://192.168.159.3:2379
cluster is healthy
[root@node1 etc]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.159.2 0.0.0.0 UG 100 0 0 ens33
192.168.159.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33
[root@node2 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.159.2 0.0.0.0 UG 100 0 0 ens33
192.168.159.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33
添加flannel网络信息到etcd
{
"Network": "172.17.0.0/8",
"SubnetLen": "24",
"SubnetMin": "172.17.0.0",
"SudbnetMax": "172.17.255.0",
"Backend": {
"Type": "vxlan",
"VNI": 1,
"Port": 8472
}
}
[root@node1 pki]# etcdctl -ca-file=ca.pem -cert-file=etcdctl.pem -key-file=etcdctl-key.pem set /k8s/network/config < /opt/k8s/node/etc/flannel-etcd-config.json
{
"Network": "172.17.0.0",
"SubnetLen": 24,
"SubnetMin": "172.17.0.0",
"SudbnetMax": "172.17.255.0",
"Backend": {
"Type": "vxlan",
"VNI": 1,
"Port": 8472
}
}
[root@node1 pki]# etcdctl -ca-file=ca.pem -cert-file=etcdctl.pem -key-file=etcdctl-key.pem get /k8s/network/config
{
"Network": "172.17.0.0/8",
"SubnetLen": 24,
"SubnetMin": "172.17.0.0",
"SudbnetMax": "172.17.255.0",
"Backend": {
"Type": "vxlan",
"VNI": 1,
"Port": 8472
}
}
flannel安装
以下操作在node1和node2同时执行
官方文档
参考博客
flannel下载
wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz
tar -zxvf flannel-v0.11.0-linux-amd64.tar.gz -C flannel
cd flannel
cp -f flanneld mk-docker-opts.sh /usr/local/bin
flannel配置文件
cat > /opt/k8s/node/etc/flanneld.conf << EOF
FLANNELD_INSECURE_OPTS="--etcd-cafile=/opt/etcd/pki/ca.pem \
--etcd-certfile=/opt/etcd/pki/etcdctl.pem \
--etcd-keyfile=/opt/etcd/pki/etcdctl-key.pem \
--etcd-endpoints=https://192.168.159.3:2379,https://192.168.159.4:2379 \
--etcd-prefix=/k8s/network \
--kube-subnet-mgr=false \
--subnet-file=/run/flannel/subnet.env \
--iface=ens33 \
--iptables-resync=5 \
--ip-masq=true"
EOF
--etcd-cafile--etcd-certfile--etcd-keyfile--etcd-endpoints--etcd-prefix--kube-subnet-mgr--net-config-path--subnet-file--iface--iptables-resync--ip-masqflannel服务文件
cat > /usr/lib/systemd/system/flanneld.service << EOF
[Unit]
Description=Flanneld
Documentation=https://github.com/coreos/flannel
After=network.target
After=network-online.target
Before=docker.service
[Service]
Type=notify
EnvironmentFile=/opt/k8s/node/etc/flanneld.conf
ExecStart=/usr/local/bin/flanneld $FLANNELD_INSECURE_OPTS
ExecStartPost=/usr/local/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
TypeBeforeExecStartPostflannel服务启动
systemctl daemon-reload && systemctl start flanneld
[root@node1 pki]# systemctl status flanneld
● flanneld.service - Flanneld
Loaded: loaded (/usr/lib/systemd/system/flanneld.service; disabled; vendor preset: disabled)
Active: active (running) since 二 2019-08-27 15:11:05 CST; 30min ago
Docs: https://github.com/coreos/flannel
Process: 2447 ExecStartPost=/usr/local/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env (code=exited, status=0/SUCCESS)
Main PID: 2432 (flanneld)
Tasks: 8
Memory: 8.4M
CGroup: /system.slice/flanneld.service
└─2432 /usr/local/bin/flanneld --etcd-cafile=/opt/etcd/pki/ca.pem --etcd-certfile=/opt/etcd/pki/etcdctl.pem --etcd-keyfile=/opt/etcd/pki/etcdctl-key.pem --etcd-endpoints=https://192.168.159.3..
[root@node1 pki]# etcdctl -ca-file=ca.pem -cert-file=etcdctl.pem -key-file=etcdctl-key.pem ls /k8s/network/subnets
/k8s/network/subnets/172.17.83.0-24
/k8s/network/subnets/172.17.46.0-24
[root@node1 pki]# etcdctl -ca-file=ca.pem -cert-file=etcdctl.pem -key-file=etcdctl-key.pem get /k8s/network/subnets/172.17.83.0-24
{"PublicIP":"192.168.159.4","BackendType":"vxlan","BackendData":{"VtepMAC":"5a:f0:7a:bf:de:79"}}
[root@node1 pki]# etcdctl -ca-file=ca.pem -cert-file=etcdctl.pem -key-file=etcdctl-key.pem get /k8s/network/subnets/172.17.46.0-24
{"PublicIP":"192.168.159.5","BackendType":"vxlan","BackendData":{"VtepMAC":"fe:32:4e:da:3f:21"}}
[root@node1 pki]# cat /run/flannel/subnet.env
DOCKER_OPT_BIP="--bip=172.17.83.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=false"
DOCKER_OPT_MTU="--mtu=1450"
DOCKER_NETWORK_OPTIONS=" --bip=172.17.83.1/24 --ip-masq=false --mtu=1450"
[root@node2 pki]# cat /run/flannel/subnet.env
DOCKER_OPT_BIP="--bip=172.17.46.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=false"
DOCKER_OPT_MTU="--mtu=1450"
DOCKER_NETWORK_OPTIONS=" --bip=172.17.46.1/24 --ip-masq=false --mtu=1450"
[root@node1 pki]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.159.4 netmask 255.255.255.0 broadcast 192.168.159.255
inet6 fe80::69a7:b207:afb9:9db4 prefixlen 64 scopeid 0x20<link>
ether 00:50:56:34:3a:20 txqueuelen 1000 (Ethernet)
RX packets 430580 bytes 54653587 (52.1 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 431431 bytes 56850150 (54.2 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 172.17.83.0 netmask 255.255.255.255 broadcast 0.0.0.0
inet6 fe80::58f0:7aff:febf:de79 prefixlen 64 scopeid 0x20<link>
ether 5a:f0:7a:bf:de:79 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 8 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 2749 bytes 333239 (325.4 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2749 bytes 333239 (325.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@node2 ~]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.159.5 netmask 255.255.255.0 broadcast 192.168.159.255
inet6 fe80::69a7:b207:afb9:9db4 prefixlen 64 scopeid 0x20<link>
inet6 fe80::2b45:985c:3f4e:908b prefixlen 64 scopeid 0x20<link>
ether 00:50:56:24:97:9a txqueuelen 1000 (Ethernet)
RX packets 1355 bytes 128086 (125.0 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1421 bytes 157618 (153.9 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 172.17.46.0 netmask 255.255.255.255 broadcast 0.0.0.0
inet6 fe80::f828:21ff:fef6:2bd4 prefixlen 64 scopeid 0x20<link>
ether fa:28:21:f6:2b:d4 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 8 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 64 bytes 5568 (5.4 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 64 bytes 5568 (5.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
重新配置docker服务
服务文件配置
EnvironmentFileflanneldocker/run/flannel/subnet.envcat > docker.service << EOF
[Unit]
Description=Docker Engine Service
Documentation=https://docs.docker.com
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin"
EnvironmentFile=-/run/flannel/subnet.env
ExecStart=/usr/local/bin/dockerd $DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
Restart=on-failure
RestartSec=5
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Delegate=yes
KillMode=process
[Install]
WantedBy=multi-user.target
EOF
服务试启动
[root@node1 etc]# systemctl daemon-reload && systemctl start docker
Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.
journalctl -xeunable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives are specified both as a flag and in the configuration file: bip: (from flag: 172.17.83.1/24, from file: 172.17.1.1/24)
直接路由/etc/docker/daemon.json--bipDOCKER_NETWORK_OPTIONS--bip/etc/docker/daemon.jsonrm -f /etc/docker/daemon.json
systemctl daemon-reload && systemctl start docker
网络查看
node1主机网络
[root@node1 etc]# ifconfig
docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.17.83.1 netmask 255.255.255.0 broadcast 172.17.83.255
inet6 fe80::42:e2ff:feaf:525d prefixlen 64 scopeid 0x20<link>
ether 02:42:e2:af:52:5d txqueuelen 0 (Ethernet)
RX packets 156 bytes 9898 (9.6 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 67 bytes 80756 (78.8 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.159.4 netmask 255.255.255.0 broadcast 192.168.159.255
inet6 fe80::69a7:b207:afb9:9db4 prefixlen 64 scopeid 0x20<link>
ether 00:50:56:34:3a:20 txqueuelen 1000 (Ethernet)
RX packets 530560 bytes 67349085 (64.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 533082 bytes 70220830 (66.9 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 172.17.83.0 netmask 255.255.255.255 broadcast 0.0.0.0
inet6 fe80::58f0:7aff:febf:de79 prefixlen 64 scopeid 0x20<link>
ether 5a:f0:7a:bf:de:79 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 8 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 2841 bytes 338023 (330.1 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2841 bytes 338023 (330.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
node2主机网络
[root@node2 etc]# ifconfig
docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.17.46.1 netmask 255.255.255.0 broadcast 172.17.46.255
inet6 fe80::42:d4ff:fe47:66f4 prefixlen 64 scopeid 0x20<link>
ether 02:42:d4:47:66:f4 txqueuelen 0 (Ethernet)
RX packets 200 bytes 12034 (11.7 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 40 bytes 3394 (3.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.159.5 netmask 255.255.255.0 broadcast 192.168.159.255
inet6 fe80::69a7:b207:afb9:9db4 prefixlen 64 scopeid 0x20<link>
inet6 fe80::2b45:985c:3f4e:908b prefixlen 64 scopeid 0x20<link>
ether 00:50:56:24:97:9a txqueuelen 1000 (Ethernet)
RX packets 2201 bytes 199369 (194.6 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2147 bytes 256110 (250.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 172.17.46.0 netmask 255.255.255.255 broadcast 0.0.0.0
inet6 fe80::f828:21ff:fef6:2bd4 prefixlen 64 scopeid 0x20<link>
ether fa:28:21:f6:2b:d4 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 8 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 64 bytes 5568 (5.4 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 64 bytes 5568 (5.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
测试跨主机容器互访
[root@node1 etc]# ifconfig
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 172.17.83.1 netmask 255.255.255.0 broadcast 172.17.83.255
inet6 fe80::42:e2ff:feaf:525d prefixlen 64 scopeid 0x20<link>
ether 02:42:e2:af:52:5d txqueuelen 0 (Ethernet)
RX packets 301 bytes 21598 (21.0 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 95 bytes 82756 (80.8 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.159.4 netmask 255.255.255.0 broadcast 192.168.159.255
inet6 fe80::69a7:b207:afb9:9db4 prefixlen 64 scopeid 0x20<link>
ether 00:50:56:34:3a:20 txqueuelen 1000 (Ethernet)
RX packets 613016 bytes 77826826 (74.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 616778 bytes 81210151 (77.4 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 172.17.83.0 netmask 255.255.255.255 broadcast 0.0.0.0
inet6 fe80::58f0:7aff:febf:de79 prefixlen 64 scopeid 0x20<link>
ether 5a:f0:7a:bf:de:79 txqueuelen 0 (Ethernet)
RX packets 22 bytes 1536 (1.5 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 139 bytes 11712 (11.4 KiB)
TX errors 0 dropped 8 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 2917 bytes 341975 (333.9 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2917 bytes 341975 (333.9 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
veth9c659ff: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet6 fe80::80a2:90ff:fe94:c603 prefixlen 64 scopeid 0x20<link>
ether 82:a2:90:94:c6:03 txqueuelen 0 (Ethernet)
RX packets 145 bytes 13730 (13.4 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 36 bytes 2656 (2.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@node1 etc]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.159.2 0.0.0.0 UG 100 0 0 ens33
172.17.46.0 172.17.46.0 255.255.255.0 UG 0 0 0 flannel.1
172.17.83.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0
192.168.159.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33
[root@0402a3fb29e6 /]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 172.17.83.2 netmask 255.255.255.0 broadcast 172.17.83.255
ether 02:42:ac:11:53:02 txqueuelen 0 (Ethernet)
RX packets 36 bytes 2656 (2.5 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 145 bytes 13730 (13.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 50 bytes 4200 (4.1 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 50 bytes 4200 (4.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@node2 ~]# ifconfig
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 172.17.46.1 netmask 255.255.255.0 broadcast 172.17.46.255
inet6 fe80::42:d4ff:fe47:66f4 prefixlen 64 scopeid 0x20<link>
ether 02:42:d4:47:66:f4 txqueuelen 0 (Ethernet)
RX packets 411 bytes 26542 (25.9 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 79 bytes 6604 (6.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.159.5 netmask 255.255.255.0 broadcast 192.168.159.255
inet6 fe80::69a7:b207:afb9:9db4 prefixlen 64 scopeid 0x20<link>
inet6 fe80::2b45:985c:3f4e:908b prefixlen 64 scopeid 0x20<link>
ether 00:50:56:24:97:9a txqueuelen 1000 (Ethernet)
RX packets 3649 bytes 336880 (328.9 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3511 bytes 407431 (397.8 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 172.17.46.0 netmask 255.255.255.255 broadcast 0.0.0.0
inet6 fe80::f828:21ff:fef6:2bd4 prefixlen 64 scopeid 0x20<link>
ether fa:28:21:f6:2b:d4 txqueuelen 0 (Ethernet)
RX packets 18 bytes 1548 (1.5 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 190 bytes 13560 (13.2 KiB)
TX errors 0 dropped 8 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 64 bytes 5568 (5.4 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 64 bytes 5568 (5.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
veth71ae48a: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet6 fe80::ccb2:8eff:fe58:e9f9 prefixlen 64 scopeid 0x20<link>
ether ce:b2:8e:58:e9:f9 txqueuelen 0 (Ethernet)
RX packets 211 bytes 17462 (17.0 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 47 bytes 3866 (3.7 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@node2 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.159.2 0.0.0.0 UG 100 0 0 ens33
172.17.46.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0
172.17.83.0 172.17.83.0 255.255.255.0 UG 0 0 0 flannel.1
192.168.159.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33
[root@bc6194e6391e /]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 172.17.46.2 netmask 255.255.255.0 broadcast 172.17.46.255
ether 02:42:ac:11:2e:02 txqueuelen 0 (Ethernet)
RX packets 47 bytes 3866 (3.7 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 211 bytes 17462 (17.0 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
firewall-cmd --zone=public --add-port=8472/udp --permanent
firewall-cmd reload
[root@node2 ~]# firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: ens33
sources:
services: ssh dhcpv6-client
ports: 2380/tcp 2379/tcp 8472/udp
protocols:
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
# node1主机访问node1容器
[root@node1 etc]# ping 172.17.83.2 -c 3
PING 172.17.83.2 (172.17.83.2) 56(84) bytes of data.
64 bytes from 172.17.83.2: icmp_seq=1 ttl=64 time=0.103 ms
64 bytes from 172.17.83.2: icmp_seq=2 ttl=64 time=0.116 ms
64 bytes from 172.17.83.2: icmp_seq=3 ttl=64 time=0.171 ms
--- 172.17.83.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.103/0.130/0.171/0.029 ms
# node2主机访问node1容器
[root@node2 ~]# ping 172.17.83.2 -c 3
PING 172.17.83.2 (172.17.83.2) 56(84) bytes of data.
64 bytes from 172.17.83.2: icmp_seq=1 ttl=63 time=0.823 ms
64 bytes from 172.17.83.2: icmp_seq=2 ttl=63 time=0.699 ms
64 bytes from 172.17.83.2: icmp_seq=3 ttl=63 time=0.748 ms
--- 172.17.83.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 0.699/0.756/0.823/0.060 ms
# node1容器访问node1主机
[root@0402a3fb29e6 /]# ping 192.168.159.4 -c 3
PING 192.168.159.4 (192.168.159.4) 56(84) bytes of data.
64 bytes from 192.168.159.4: icmp_seq=1 ttl=64 time=0.140 ms
64 bytes from 192.168.159.4: icmp_seq=2 ttl=64 time=0.120 ms
64 bytes from 192.168.159.4: icmp_seq=3 ttl=64 time=0.132 ms
--- 192.168.159.4 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.120/0.130/0.140/0.015 ms
# node1容器访问node2主机
[root@0402a3fb29e6 /]# ping 192.168.159.5 -c 3
PING 192.168.159.5 (192.168.159.5) 56(84) bytes of data.
64 bytes from 192.168.159.5: icmp_seq=1 ttl=63 time=1.08 ms
64 bytes from 192.168.159.5: icmp_seq=2 ttl=63 time=0.670 ms
64 bytes from 192.168.159.5: icmp_seq=3 ttl=63 time=0.711 ms
--- 192.168.159.5 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 0.670/0.821/1.083/0.187 ms
# node1容器访问node2容器
[root@0402a3fb29e6 /]# ping 172.17.46.2 -c 3
PING 172.17.46.2 (172.17.46.2) 56(84) bytes of data.
64 bytes from 172.17.46.2: icmp_seq=1 ttl=62 time=1.55 ms
64 bytes from 172.17.46.2: icmp_seq=2 ttl=62 time=0.862 ms
64 bytes from 172.17.46.2: icmp_seq=3 ttl=62 time=0.757 ms
--- 172.17.46.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 0.757/1.058/1.556/0.355 ms
# node1容器访问百度
[root@0402a3fb29e6 /]# ping www.baidu.com -c 3
PING www.a.shifen.com (14.215.177.39) 56(84) bytes of data.
64 bytes from 14.215.177.39 (14.215.177.39): icmp_seq=1 ttl=127 time=36.0 ms
64 bytes from 14.215.177.39 (14.215.177.39): icmp_seq=2 ttl=127 time=35.9 ms
64 bytes from 14.215.177.39 (14.215.177.39): icmp_seq=3 ttl=127 time=35.8 ms
--- www.a.shifen.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 35.882/35.951/36.009/0.163 ms
至此Docker跨主机集群网络配置完毕，下一步接着探讨Node节点的其他服务安装。