概述
基于 Cilium 官网关于 Bandwidth Manage，主要是描述 Cilium 作为唯一 CNI 的情况下，没有太多介绍 bandwidth-manager 是否能够使用在 Chain 模式下，本文主要记录一下 Cilium Chain 模式下的 bandwidth-manager 的部署测试和使用。
部署和测试
在 Kubernetes + Flannel 的集群部署 Cilium，注意是采用 Chain 的方式，这才是我们测试的场景。
1
|
helm install cilium . --version 1.14.4 --namespace=kube-system --set cni.chainingMode=generic-veth --set cni.customConf=true --set cni.configMap=cni-configuration --set routingMode=native --set enableIPv4Masquerade=false --set bandwidthManager.enabled=true
|
部署的情况如下，可以看到 Flannel 和 Cilium 是共存的。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
|
[root@master ~]# k get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
cilium-4rtwr 1/1 Running 35 (23h ago) 46h 192.168.1.201 node1
cilium-8946r 1/1 Running 8 (23h ago) 46h 192.168.1.200 master
cilium-operator-85c7767d4c-wzhxg 1/1 Running 35 (23h ago) 46h 192.168.1.201 node1
cilium-wcmsl 1/1 Running 41 (23h ago) 46h 192.168.1.202 node2
coredns-6d4b75cb6d-c58px 0/1 Running 213 (20m ago) 45h 10.244.2.2 node2
coredns-6d4b75cb6d-gsjcq 1/1 Running 202 (36m ago) 45h 10.244.1.2 node1
etcd-master 1/1 Running 3 (23h ago) 30d 192.168.1.200 master
kube-apiserver-master 1/1 Running 6 (23h ago) 30d 192.168.1.200 master
kube-controller-manager-master 1/1 Running 27 (23h ago) 30d 192.168.1.200 master
kube-flannel-ds-l95kb 1/1 Running 0 46h 192.168.1.201 node1
kube-flannel-ds-p6kcs 1/1 Running 0 46h 192.168.1.202 node2
kube-flannel-ds-vljth 1/1 Running 0 4s 192.168.1.200 master
kube-scheduler-master 1/1 Running 24 (23h ago) 30d 192.168.1.200 master
|
然后再进入 Cilium 的 Pod 里查看一下 bandwidth-manager 的情况，可以看到 BandwidthManager 字段显示是开启的状态的。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
|
[root@master ~]# k exec -it cilium-4rtwr -- cilium status --verbose
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
KVStore: Ok Disabled
Kubernetes: Ok 1.24 (v1.24.8) [linux/amd64]
Kubernetes APIs: ["EndpointSliceOrEndpoint", "cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumCIDRGroup", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement: Partial [eth0 , flannel.1 ]
Host firewall: Disabled
CNI Chaining: generic-veth
Cilium: Ok 1.14.4 (v1.14.4-87dd2b64)
NodeMonitor: Listening for events on 4 CPUs with 64x4096 of shared memory
Cilium health daemon: Ok
IPAM: IPv4: 1/254 allocated from 10.0.2.0/24,
Allocated addresses:
10.0.2.74 (router)
IPv4 BIG TCP: Disabled
IPv6 BIG TCP: Disabled
BandwidthManager: EDT with BPF [CUBIC] [eth0, flannel.1]
Host Routing: Legacy
Masquerading: Disabled
Clock Source for BPF: ktime
Controller Status: 25/25 healthy
...
...
|
最后，来测试一下 BandwidthManager 的作用，部署下面的 netperf 客户端和服务端。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
|
cat <<EOF | kubectl apply -f -
---
apiVersion: v1
kind: Pod
metadata:
annotations:
# Limits egress bandwidth to 10Mbit/s.
kubernetes.io/egress-bandwidth: "10M"
labels:
# This pod will act as server.
app.kubernetes.io/name: netperf-server
name: netperf-server
spec:
containers:
- name: netperf
image: cilium/netperf
ports:
- containerPort: 12865
---
apiVersion: v1
kind: Pod
metadata:
# This Pod will act as client.
name: netperf-client
spec:
affinity:
# Prevents the client from being scheduled to the
# same node as the server.
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values:
- netperf-server
topologyKey: kubernetes.io/hostname
containers:
- name: netperf
args:
- sleep
- infinity
image: cilium/netperf
EOF
|
下面做具体的带宽压测，结果正如我们的配置一样，限制在了 9.51 Mbit/s。
1
2
3
4
5
6
7
8
9
|
[root@master ~]# NETPERF_SERVER_IP=$(kubectl get pod netperf-server -o jsonpath='{.status.podIP}')
[root@master ~]# kubectl exec netperf-client -- netperf -t TCP_MAERTS -H "${NETPERF_SERVER_IP}"
MIGRATED TCP MAERTS TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.244.2.4 (10.244.) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
131072 16384 16384 10.00 9.51
|
另外需要找 server 端的 agent 才能通过 cilium 的客户端查找相关的信息。
1
2
3
4
|
[root@master ~]# kubectl exec -it -n kube-system cilium-wcmsl -- cilium bpf bandwidth list
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
IDENTITY EGRESS BANDWIDTH (BitsPerSec)
1287 10M
|
总结
在 bandwidth-manager 的使用下，实际上是会有更多的类似的应用场景，比如 CFP: QoS for network #24194，这个也是我们在业务上希望探索的方向，就是利用 Cilium Chain 的 BandwidthManager，给容器网络带来更多如 QoS 的管控。
参考资料
- Cilium系列-11-启用带宽管理器
- [译] Cilium：基于 BPF+EDT+FQ+BBR 实现更好的带宽管理（KubeCon, 2022）