Kubernetes容器编排技术网络流量分析

资讯 2024-06-24 阅读:235 评论:0

杜平Dupin. （中国联通晋中分公司 030600） (China Utsun-Sin-China Branch & #xa0; 030600) 容器技术近年来在各大企业、互联网...

美化布局示例

欧易(OKX)最新版本

【遇到注册下载问题请加文章最下面的客服微信】永久享受返佣20%手续费！

APP下载全球官网大陆官网

币安(Binance)最新版本

币安交易所app【遇到注册下载问题请加文章最下面的客服微信】永久享受返佣20%手续费！

APP下载官网地址

火币HTX最新版本

火币老牌交易所【遇到注册下载问题请加文章最下面的客服微信】永久享受返佣20%手续费！

APP下载官网地址

杜平

Dupin.

（中国联通晋中分公司 030600）

(China Utsun-Sin-China Branch & #xa0; 030600)

容器技术近年来在各大企业、互联网公司作为最先进的云虚拟化技术发挥了重要的作用，为业务快速迭代和发展提供了有效支持。本文深度分析了Kubernetes容器编排技术在Canal网络方案的不同场景中的数据流向、原理、故障排查方法等。

Container technology has played an important role in recent years in major enterprises, Internet companies, as the state-of-the-art cloud virtualization technology, and has effectively supported rapid business rotation and development. This paper provides an in-depth analysis of data flows, rationales, failure-checking methods, etc., in different scenarios of Kubernetes packagings in the Canal network programme.

关键词：

Keywords: xa0;

Docker Kubernetes Canal容器 overlay Deployment

Docker & #xa0; Kubernetes & #xa0; Canal container & #xa0; overlay #xa0; Development

In recent years, some large enterprise start to adopt container technology as virtualization to serve business grow fast. This thesis analyzes container network technology by Kubernetes and Canal base on experiment environment for data flow, route and troubleshooting methodology.

KEY WORDS:

Docker Kubernetes Canal overlay Deployment

容器云技术已经越来越被企业所采用，助力传统企业业务的数字化转型，应对互联网大潮下的业务冲击。Kubernetes作为容器编排的事实标准已被49%的企业所采用，而另外的企业也在考虑和测试Kubernetes在业务环境里使用的可能性。联通内部互联网类应用越来越多，未来可以使用容器技术提升业务的迭代速度和效率，更好地为客户服务。容器技术中网络是比较复杂的组成部分，本文深度剖析了Kubernetes与Canal构建的容器云网络的数据流向和相关故障排查方法。

The packaging cloud technology has become increasingly used by companies to help digitalize traditional business operations and cope with the impact of Internet operations. Kubernetes, as a container, has been used by 49% of businesses, while others are considering and testing the possibility of Kubernetes using it in the business environment. There is an increasing use of inter-house Internet applications that can be used in the future to increase the speed and efficiency of operations and better serve clients. The network in the container technology is a more complex component, and the data flow and associated fault-checking methods of the container cloud network constructed by Kubernets and Canal have been analysed in depth.

此次研究课题所使用的环境如下图1所示：

The environment used for the study is illustrated in figure 1 below:

图1：研究环境

Figure 1: Research environment

环境说明：

Environmental description:

1. 客户端：1台

2. 服务器：2台，Centos7-1：172.16.240.131，Centos7-2：172.16.240.132

3. 服务器操作系统：Centos7.3

4. 服务器安装的平台服务：Kubernetes（包括：apiserver，scheduler，controller，etcd，kubelet，kubeproxy），Docker，Canal网络组件等。

5. 测试所使用的应用：service：ghost，IP地址：10.250.0.40，应用暴露端口：31266

6. 应用后端的pod的IP地址：10.244.1.5，10.244.0.74

7. 客户端IP：172.16.240.1

###centos7-1###查看ghost的2个副本

# Centos7-1 # # check out two copies of ghost #

[root@centos7-1 ~]# kubectl get pods -o wide | grep ghost

NAME READY STATUS RESTARTS AGE IP NODE

ghost-55669748d5-txmpg 1/1 Running 0 1m 10.244.1.5 centos7-1

ghost-55669748d5-pgnhw 1/1 Running 1 7d 10.244.0.74 centos7-2

###centos7-1###进入pod内部：ghost-55669748d5-txmpg

# #centos7-1# # Inside Pod: ghost-5669748d5-txmpg

[root@centos7-1 ~]# kubectl exec -ti ghost-55669748d5-txmpg bash

###pod：ghost-55669748d5-txmpg###查看ghost-ep1的ip地址是10.244.1.5/32，pod内的网卡eth0

##pod:ghost-5669748d5-txmpg##seegost-p1 at 10.244.1.5/32, netcath0 in Pod

root@ghost-55669748d5-txmpg:/var/lib/ghost# ip a

1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

valid_lft forever preferred_lft forever

3: eth0@if8: mtu 1500 qdisc noqueue state UP group default

link/ether 3e:0c:94:46:7b:98 brd ff:ff:ff:ff:ff:ff

inet 10.244.1.5/32 scope global eth0

valid_lft forever preferred_lft forever

###pod：ghost-55669748d5-txmpg###查看pod内的路由，缺省路由的下一跳是169.254.1.1，通过设备eth0出去

##pod: Gott-5669748d5-txmpg## views the path within Pod, with 169.254.1.1, the next jump for the missing route, going out through device eth0

root@ghost-55669748d5-txmpg:/var/lib/ghost# ip route

default via 169.254.1.1 dev eth0

169.254.1.1 dev eth0 scope link

###pod：ghost-55669748d5-txmpg###查看pod内的网络邻居，发现169.254.1.1的mac地址是 56:aa:dd:0b:e9:03

##pod:ghost-5669748d5-txmpg## views the Internet neighbor in Pod and finds the mac address in 169.254.1.1 #xa0; 56:aaa:dd:0b:e9:03

root@ghost-55669748d5-txmpg:/var/lib/ghost# ip nei

172.16.240.131 dev eth0 lladdr 56:aa:dd:0b:e9:03 STALE

169.254.1.1 dev eth0 lladdr 56:aa:dd:0b:e9:03 STALE

###centos7-1###查看宿主机所有接口，56:aa:dd:0b:e9:03在设备calif0e9df43659上

# #centos7-1## views all host interfaces 56:aa:dd:0b:e9:03 on device calif0e9df436559

[root@centos7-1 ~]# ip addr show

1: lo: mtu 65536 qdisc noqueue state UNKNOWN qlen 1

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

valid_lft forever preferred_lft forever

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

2: ens33: mtu 1500 qdisc pfifo_fast state UP qlen 1000

link/ether 00:0c:29:b2:8d:83 brd ff:ff:ff:ff:ff:ff

inet 172.16.240.131/24 brd 172.16.240.255 scope global ens33

valid_lft forever preferred_lft forever

inet6 fe80::c100:4631:34fa:3256/64 scope link

valid_lft forever preferred_lft forever

3: docker0: mtu 1500 qdisc noqueue state DOWN

link/ether 02:42:fc:fd:bd:65 brd ff:ff:ff:ff:ff:ff

inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0

valid_lft forever preferred_lft forever

4: flannel.1: mtu 1450 qdisc noqueue state UNKNOWN

link/ether 8a:40:90:1c:cf:7e brd ff:ff:ff:ff:ff:ff

inet 10.244.1.0/32 scope global flannel.1

valid_lft forever preferred_lft forever

inet6 fe80::8840:90ff:fe1c:cf7e/64 scope link

valid_lft forever preferred_lft forever

8: calif0e9df43659@if3: mtu 1500 qdisc noqueue state UP

link/ether 56:aa:dd:0b:e9:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0

inet6 fe80::54aa:ddff:fe0b:e903/64 scope link

valid_lft forever preferred_lft forever

###centos7-1###查看设备calif0e9df43659，proxy_arp=1，做了arp代答（代替pod应答arp请求）

# #centos7-1# # view the device calif0e9df436559, proxy_arp=1, an arp response (in place of Pod responding to arp request)

[root@centos7-1 ~]# sysctl -a | grep calif0e9df43659 | grep proxy_arp

sysctl: reading key 'net.ipv6.conf.all.stable_secret'

sysctl: reading key 'net.ipv6.conf.calif0e9df43659.stable_secret'

sysctl: reading key 'net.ipv6.conf.default.stable_secret'

sysctl: reading key 'net.ipv6.conf.docker0.stable_secret'

sysctl: reading key 'net.ipv6.conf.ens33.stable_secret'

sysctl: reading key 'net.ipv6.conf.flannel/1.stable_secret'

sysctl: reading key 'net.ipv6.conf.lo.stable_secret'

net.ipv4.conf.calif0e9df43659.proxy_arp = 1

net.ipv4.conf.calif0e9df43659.proxy_arp_pvlan = 0

###centos7-1###查看宿主机路由表，去往ghost-ep2的路由指示从设备flannel.1发出

# #centos7-1## # check the host route list, send instructions from device flannel.1 to gogost-p2

[root@centos7-1 ~]# ip route

default via 172.16.240.2 dev ens33 proto static metric 100

10.244.0.0/24 via 10.244.0.0 dev flannel.1 onlink

10.244.1.5 dev calif0e9df43659 scope link

172.16.240.0/24 dev ens33 proto kernel scope link src 172.16.240.131 metric 100

172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1

[root@centos7-1 ~]# route -n

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

0.0.0.0 172.16.240.2 0.0.0.0 UG 100 0 0 ens33

10.244.0.0 10.244.0.0 255.255.255.0 UG 0 0 0 flannel.1

10.244.1.5 0.0.0.0 255.255.255.255 UH 0 0 0 calif0e9df43659

172.16.240.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33

172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0

###centos7-1###查看设备flannel.1，vxlan的设备，vni=1，本机地址是172.16.240.131，从设备ens33发出

##centos7-1## views equipment flannel.1, vxlan's equipment, vni = 1, with this aircraft address 172.16.240.131, sent from device ens33

[root@centos7-1 ~]# ip -d link show flannel.1

4: flannel.1: mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT

link/ether 8a:40:90:1c:cf:7e brd ff:ff:ff:ff:ff:ff promiscuity 0

vxlan id 1 local 172.16.240.131 dev ens33 srcport 0 0 dstport 8472 nolearning ageing 300 addrgenmode eui64

###centos7-1###查看宿主机的网络邻居，去往10.244.0.0的数据包，从设备 flannel.1出，邻居的mac是32:a9:79:6d:0b:b1

# #centos7-1## # view the host's network neighbor, go to 10.244.0.0 data packs from equipment #xa0; flannel.1, the neighbour's mac is 32:a9:79:6d:0b:b1

[root@centos7-1 ~]# ip nei

172.16.240.132 dev ens33 lladdr 00:0c:29:b5:b6:92 REACHABLE

10.244.0.0 dev flannel.1 lladdr 32:a9:79:6d:0b:b1 PERMANENT

172.16.240.1 dev ens33 lladdr 00:50:56:c0:00:08 DELAY

10.244.1.4 dev calif7ade911ae8 lladdr 12:9c:95:cd:4e:cd STALE

172.16.240.2 dev ens33 lladdr 00:50:56:fc:f9:04 DELAY

###centos7-1###查看vxlan的fdb表，发现对端vtep的ip是172.16.240.132

# #centos7-1# # look at vxlan's fdb table and find that the ip on the end vtep is 172.16.240.132

FDB表是二层转发表，

The FDB table is republished on the second floor.

[root@centos7-1 ~]# bridge fdb

01:00:5e:00:00:01 dev ens33 self permanent

33:33:00:00:00:01 dev ens33 self permanent

33:33:ff:fa:32:56 dev ens33 self permanent

33:33:00:00:00:01 dev docker0 self permanent

01:00:5e:00:00:01 dev docker0 self permanent

02:42:fc:fd:bd:65 dev docker0 vlan 1 master docker0 permanent

02:42:fc:fd:bd:65 dev docker0 master docker0 permanent

32:a9:79:6d:0b:b1 dev flannel.1 dst 172.16.240.132 self permanent

33:33:00:00:00:01 dev calif0e9df43659 self permanent

01:00:5e:00:00:01 dev calif0e9df43659 self permanent

33:33:ff:0b:e9:03 dev calif0e9df43659 self permanent

###centos7-2###TCP/IP是内核态行为，拆掉数据包，发现是vxlan，本地有路由，设备是cali4a6ea607e95（代答pod的arp）

# #centos7-2## #TCP/IP is kernel behavior, disassembly data packs, vxlan, local route, cali4a6e6a607e95

[root@centos7-2 ~]# route -n

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

0.0.0.0 172.16.240.2 0.0.0.0 UG 100 0 0 ens33

10.244.0.65 0.0.0.0 255.255.255.255 UH 0 0 0 cali185c333c9d5

10.244.0.69 0.0.0.0 255.255.255.255 UH 0 0 0 cali4ea182e3288

10.244.0.72 0.0.0.0 255.255.255.255 UH 0 0 0 cali807b30de140

10.244.0.74 0.0.0.0 255.255.255.255 UH 0 0 0 cali4a6ea607e95

10.244.0.82 0.0.0.0 255.255.255.255 UH 0 0 0 cali1a3e756d696

10.244.0.83 0.0.0.0 255.255.255.255 UH 0 0 0 calic440f455693

10.244.0.84 0.0.0.0 255.255.255.255 UH 0 0 0 cali0dd295201d1

10.244.0.85 0.0.0.0 255.255.255.255 UH 0 0 0 cali1c1c361bfa6

10.244.0.86 0.0.0.0 255.255.255.255 UH 0 0 0 caliccec0fc857b

10.244.0.88 0.0.0.0 255.255.255.255 UH 0 0 0 cali02a732f7d51

10.244.1.0 10.244.1.0 255.255.255.0 UG 0 0 0 flannel.1

172.16.240.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33

172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0

###centos7-2###cali4a6ea607e95转给pod：ghost-55669748d5-pgnhw

# #centos7-2##cali4a6e6e607e95 to Pod: ghost-5669748d5-pgnhw

###centos7-1###查看ghost的service，port是31266

# #centos7-1# # check out the ghost's service, port's 31266

[root@centos7-1 ~]# kubectl get svc ghost -o wide

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR

ghost NodePort 10.250.0.40 2368:31266/TCP 10d run=ghost

###centos7-1###查看ghost的endpoints，共2个ep

# Centos7-1# # check out ghost's endpoints for 2 p

[root@centos7-1 ~]# kubectl get endpoints ghost -o wide

NAME ENDPOINTS AGE

ghost 10.244.0.74:2368,10.244.1.5:2368 10d

###centos7-1###查看pod信息，pod在centos7-2上

# #centos7-1# # viewing Pod information, pod on Centos7-2

[root@centos7-1 ~]# kubectl get pod ghost-55669748d5-pgnhw -o wide

NAME READY STATUS RESTARTS AGE IP NODE

ghost-55669748d5-pgnhw 1/1 Running 1 7d 10.244.0.74 centos7-2

去程发送端PREROUTING: 请求报文 172.16.240.1:任意端口 ----> 172.16.240.131:31266

Go to sender PREROUTING: & #xa0; Request & #xa0; 172.16.240.1: Any port & #xa0; - > #xa0; & #xa0; 172.240.131:3266

###centos7-1###查看PREROUTING的chain的nat表中内容

# #centos7-1 # # look at the contents of the prerooting chain nat

[root@centos7-1 ~]# iptables -vnL PREROUTING -t nat

Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)

pkts bytes target prot opt in out source destination

1749 142K cali-PREROUTING all -- * * 0.0.0.0/0 0.0.0.0/0

1749 142K KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0

1188 74424 DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL

###centos7-1###查看KUBE-SERVICE chain的nat表中的内容，优先匹配Cluster IP的报文，最后默认匹配KUBE-NODEPORTS

# #centos7-1## # view the contents of KUBE-SERVICE chain's nat table, preferred to Cluster IP report, last default match to KUBE-NODEPORTS

[root@centos7-1 ~]# iptables -vnL KUBE-SERVICES -t nat

Chain KUBE-SERVICES (2 references)

pkts bytes target prot opt in out source destination

0 0 KUBE-MARK-MASQ tcp -- * * !10.244.0.0/16 10.250.0.101 tcp dpt:80

0 0 KUBE-SVC-4N57TFCL4MD7ZTDA tcp -- * * 0.0.0.0/0 10.250.0.101 tcp dpt:80

0 0 KUBE-MARK-MASQ udp -- * * !10.244.0.0/16 10.250.0.10 udp dpt:53

0 0 KUBE-SVC-TCOU7JCQXEZGVUNU udp -- * * 0.0.0.0/0 10.250.0.10 udp dpt:53

0 0 KUBE-MARK-MASQ tcp -- * * !10.244.0.0/16 10.250.0.10 tcp dpt:53

0 0 KUBE-SVC-ERIFXISQEP7F7OF4 tcp -- * * 0.0.0.0/0 10.250.0.10 tcp dpt:53

0 0 KUBE-MARK-MASQ tcp -- * * !10.244.0.0/16 10.250.0.151 tcp dpt:80

0 0 KUBE-SVC-WMDDEEP2PS5RXKUQ tcp -- * * 0.0.0.0/0 10.250.0.151 tcp dpt:80

0 0 KUBE-MARK-MASQ tcp -- * * !10.244.0.0/16 10.250.0.40 tcp dpt:2368

0 0 KUBE-SVC-TFV3VPHN2LAGKCVY tcp -- * * 0.0.0.0/0 10.250.0.40 tcp dpt:2368

0 0 KUBE-MARK-MASQ tcp -- * * !10.244.0.0/16 10.250.0.11 tcp dpt:2368

0 0 KUBE-SVC-BXCGFFQYYR74AICZ tcp -- * * 0.0.0.0/0 10.250.0.11 tcp dpt:2368

0 0 KUBE-MARK-MASQ tcp -- * * !10.244.0.0/16 10.250.0.1 tcp dpt:443

0 0 KUBE-SVC-NPX46M4PTMTKRN6Y tcp -- * * 0.0.0.0/0 10.250.0.1 tcp dpt:443

4 240 KUBE-NODEPORTS all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL

###centos7-1###查看KUBE-NODEPORTS chain的nat表中的内容，先打MASQMARK，之后匹配规则

# #centos7-1# # view the contents of Kube-NODEPORTS chain's nat table, hit MASQMARK first, then match the rules

[root@centos7-1 ~]# iptables -vnL KUBE-NODEPORTS -t nat

Chain KUBE-NODEPORTS (1 references)

pkts bytes target prot opt in out source destination

0 0 KUBE-MARK-MASQ tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:30044

0 0 KUBE-SVC-4N57TFCL4MD7ZTDA tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:30044

0 0 KUBE-MARK-MASQ tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:31266

0 0 KUBE-SVC-TFV3VPHN2LAGKCVY tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:31266

###centos7-1###查看KUBE-MARK-MASQ chain的nat表的内容，数据包返回时要做SNAT，对DNAT的报文打0x4000 PKT mark

##centos7-1## # view the contents of KUBE-MARK-MASQ chain's nat table, to be used for SNAT when the data pack returns, with 0x4000xa0 for DNA T; PKT Mark

[root@centos7-1 ~]# iptables -vnL KUBE-MARK-MASQ -t nat

Chain KUBE-MARK-MASQ (18 references)

pkts bytes target prot opt in out source destination

0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK or 0x4000

###centos7-1###查看KUBE-SVC-TFV3VPHN2LAGKCVY chain的nat表中的内容，有2个endpoints，平均分发

##centos7-1## # view the contents of KUBE-SVC-TFV3VPHN2LAGKCVY chain nat tables with 2 endpoints, average distribution

[root@centos7-1 ~]# iptables -vnL KUBE-SVC-TFV3VPHN2LAGKCVY -t nat

Chain KUBE-SVC-TFV3VPHN2LAGKCVY (2 references)

pkts bytes target prot opt in out source destination

0 0 KUBE-SEP-UJXTJ7PNRUN6D2JD all -- * * 0.0.0.0/0 0.0.0.0/0 statistic mode random probability 0.50000000000

0 0 KUBE-SEP-MNOUPYFQRWSOE45J all -- * * 0.0.0.0/0 0.0.0.0/0

###centos7-1###查看KUBE-SEP-UJXTJ7PNRUN6D2JD chain的nat表中的内容，做DNAT，将目的ip和port（172.16.240.131:30266）转换成对应容器的ip和port（10.244.0.74:2368）

##centos7-1## views the contents of KUBE-SEP-UJXJ7PNUN6D2JD chain's nat table, makes DNA T and converts the destination ip and port (172.240.131:30266) to the counterpart container #ip and port (10.244.04.74:2368)

[root@centos7-1 ~]# iptables -vnL KUBE-SEP-UJXTJ7PNRUN6D2JD -t nat

Chain KUBE-SEP-UJXTJ7PNRUN6D2JD (1 references)

pkts bytes target prot opt in out source destination

0 0 KUBE-MARK-MASQ all -- * * 10.244.0.74 0.0.0.0/0

0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp to:10.244.0.74:2368

去程发送端FORWARD: 路由查询出的设备：请求报文 172.16.240.1:任意端口 ----> 10.244.0.74:2368

Go to sender port FORWARD: & #xa0; routing equipment: Request & #xa0; 172.16.240.1: Any port & #xa0; - > #xa0; & #xa0; 10.244.0.74:2368

###centos7-1###查看宿主机路由表，找到10.240.0的路由，需要走到flannel.1接口上

# #centos7-1# # check the host chart, find 10.240.0 route, need to go to the flannel 1 interface

[root@centos7-1 ~]# route -n

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

0.0.0.0 172.16.240.2 0.0.0.0 UG 100 0 0 ens33

10.244.0.0 10.244.0.0 255.255.255.0 UG 0 0 0 flannel.1

10.244.1.5 0.0.0.0 255.255.255.255 UH 0 0 0 calif0e9df43659

172.16.240.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33

172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0

###centos7-1###查看接口设备，vxlan的ep，本端地址：172.16.240.131

# #centos7-1# # view interface device, p for vxlan, home address: 172.16.240.131

[root@centos7-1 ~]# ip -d link show flannel.1

4: flannel.1: mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT

link/ether 8a:40:90:1c:cf:7e brd ff:ff:ff:ff:ff:ff promiscuity 0

vxlan id 1 local 172.16.240.131 dev ens33 srcport 0 0 dstport 8472 nolearning ageing 300 addrgenmode eui64

去程发送端POSTROUTING:

Go to sender POSTROUTING: #xa0;

###centos7-1###查看POSTROUTING的chain的nat表内容

# #centos7-1# # look at the chain nat of POSTROUTING

[root@centos7-1 ~]# iptables -vnL POSTROUTING -t nat

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)

pkts bytes target prot opt in out source destination

60312 3695K cali-POSTROUTING all -- * * 0.0.0.0/0 0.0.0.0/0

0 0 MASQUERADE all -- * !docker0 172.17.0.0/16 0.0.0.0/0

61788 3789K KUBE-POSTROUTING all -- * * 0.0.0.0/0 0.0.0.0/0

35 2716 RETURN all -- * * 10.244.0.0/16 10.244.0.0/16

6 360 MASQUERADE all -- * * 10.244.0.0/16 !224.0.0.0/4

0 0 RETURN all -- * * !10.244.0.0/16 10.244.1.0/24

0 0 MASQUERADE all -- * * !10.244.0.0/16 10.244.0.0/16

###centos7-1###查看KUBE-POSTROUTING的chain的nat表内容

# #centos7-1# # look at the contents of the chain nat of Kube-POSTROUTING

[root@centos7-1 ~]# iptables -vnL KUBE-POSTROUTING -t nat

Chain KUBE-POSTROUTING (1 references)

pkts bytes target prot opt in out source destination

0 0 MASQUERADE all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x4000/0x4000

出接口为flannel.1，MASQUERADE通过inet_select_addr（选择flannel.1网卡的ip作为原ip）从flannel.1上先把IP作SNAT，此时报文请求变为10.244.1.0:任意端口 ----> 10.244.0.74:2368

The interface came from flannel.1, MASQUERADE via nett_select_addr (option of ip from flannel.1 net card as original ip) to use IP as SNAT from flannel.1, at which point the request for submission became 10.244.1.0: any port xa0; - >#xa0; xa0; xa0; 10.244.0.74:2368

去程发送端Vxlan通信：10.244.1.0:任意端口 ----> 10.244.0.74:2368

Vxlan: 10.244.1.0: Any port & #xa0; - > #xa0; & #xa0; 10.244.0.74:2368

从vxlan端口(flannel.1)发送报文，驱动调用vxlan_xmit，通过目的mac查询vxlan_fdb先把remote ip（外层目的ip），即172.16.240.132，从vxlan_dev中获取源local ip 172.16.240.131（&vxlan->cfg.saddr），然后调iptunnel_xmit->ip_local_out从对应物理网口将报文送出去。

Messages were sent from the vxlan (flannel.1) port, driven to vxlan_xmit, via the purpose mac query vxlan_fdb first to remote ip (outside endip), i.e. 172.16.240.132, from the vxlan_dev source local ip 172.21.240.131 (&vxlan->cfg.saddr) and then to call iptunel_xmit->ip_local_out from the corresponding physical portal.

###centos7-1###查看宿主机的网络邻居，去往10.244.0.0的数据包，从设备 flannel.1出，邻居的mac是32:a9:79:6d:0b:b1

# #centos7-1## # view the host's network neighbor, go to 10.244.0.0 data packs from equipment #xa0; flannel.1, the neighbour's mac is 32:a9:79:6d:0b:b1

[root@centos7-1 ~]# ip nei

172.16.240.132 dev ens33 lladdr 00:0c:29:b5:b6:92 REACHABLE

10.244.0.0 dev flannel.1 lladdr 32:a9:79:6d:0b:b1 PERMANENT

172.16.240.1 dev ens33 lladdr 00:50:56:c0:00:08 DELAY

10.244.1.4 dev calif7ade911ae8 lladdr 12:9c:95:cd:4e:cd STALE

172.16.240.2 dev ens33 lladdr 00:50:56:fc:f9:04 DELAY

###centos7-1###查看vxlan的fdb表，发现对端vtep的ip是172.16.240.132

# #centos7-1# # look at vxlan's fdb table and find that the ip on the end vtep is 172.16.240.132

FDB表是二层转发表，

The FDB table is republished on the second floor.

[root@centos7-1 ~]# bridge fdb

01:00:5e:00:00:01 dev ens33 self permanent

33:33:00:00:00:01 dev ens33 self permanent

33:33:ff:fa:32:56 dev ens33 self permanent

33:33:00:00:00:01 dev docker0 self permanent

01:00:5e:00:00:01 dev docker0 self permanent

02:42:fc:fd:bd:65 dev docker0 vlan 1 master docker0 permanent

02:42:fc:fd:bd:65 dev docker0 master docker0 permanent

32:a9:79:6d:0b:b1 dev flannel.1 dst 172.16.240.132 self permanent

33:33:00:00:00:01 dev calif0e9df43659 self permanent

01:00:5e:00:00:01 dev calif0e9df43659 self permanent

33:33:ff:0b:e9:03 dev calif0e9df43659 self permanent

###centos7-1###查看POSTROUTING的chain的nat表内容，由于 POSTROUTING 有如下匹配规则，所以 vxlan 报文不会再做一次 SNAT

##centos7-1## # view the contents of the chain tab in POSTROUTING, #xa0; #xa0; #xa0; vxlan#xa0; no more #xa0; SNAT

[root@centos7-1 ~]# iptables -vnL POSTROUTING -t nat

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)

pkts bytes target prot opt in out source destination

60312 3695K cali-POSTROUTING all -- * * 0.0.0.0/0 0.0.0.0/0

0 0 MASQUERADE all -- * !docker0 172.17.0.0/16 0.0.0.0/0

61788 3789K KUBE-POSTROUTING all -- * * 0.0.0.0/0 0.0.0.0/0

35 2716 RETURN all -- * * 10.244.0.0/16 10.244.0.0/16

6 360 MASQUERADE all -- * * 10.244.0.0/16 !224.0.0.0/4

0 0 RETURN all -- * * !10.244.0.0/16 10.244.1.0/24

0 0 MASQUERADE all -- * * !10.244.0.0/16 10.244.0.0/16

去程接收端：

Go to the receiving end:

Vxlan udp报文处理经过vxlan_rcv (ens3) -> netif_rx(vxlandev skb)

Vxlan udp report processed through vxlan_rcv ens3 ->xa0; netif_rx (vxlandev skb)

其中vxlan_rcv中会使用vxlan_vs_find_vni(vs, vxlan_vni(vxlan_hdr(skb)->vx_vni))通过vxlan报文vni来查找对应的vxlan_dev，所以vxlan网口vni要一致。

Vxlan_rcv uses vxlan_vs_find_vni (vs, & #xa0; vxlan_vni (vxlan_hdr(skb)->vx_vni)) to find the corresponding vxlan_dev via vxlan reportvni, so vxlan web portal vni is consistent.

###centos7-2###查看设备flannel.1，vxlan的设备，vni=1，本机地址是172.16.240.132，从设备ens33发出

##centos7-2## views equipment flannel.1, vxlan's equipment, vni = 1, with this flight address 172.16.240.132, sent from device ens33

[root@centos7-2 ~]# ip -d link show flannel.1

13: flannel.1: mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT

link/ether 32:a9:79:6d:0b:b1 brd ff:ff:ff:ff:ff:ff promiscuity 0

vxlan id 1 local 172.16.240.132 dev ens33 srcport 0 0 dstport 8472 nolearning ageing 300 addrgenmode eui64