kubernetes部署network-operator

a-c-dream 发布于

参考network-operator/example/README.md at master · Mellanox/network-operator · GitHub

实践：使用SR-IOV在k8s集群中提供rdma网卡 - MAAO的博客

Kubernetes Pod支持RDMA网卡的几种方式 - MAAO的博客

Network Operator - NVIDIA Docs

一些想法

思路应该是使用helm安装network-operator其中values.yaml可能是下面的，也可能需要修改。

那些CRD不知道要不要部署

前置条件

GPU operator（暂时未做）

将装有Mellanox Ethernet NICs的节点打上feature.node.kubernetes.io/pci-15b3.present=true标签。

Add NVIDIA NGC repository:

Install Network Operator from the NVIDIA NGC chart using custom values:

values.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
nfd:
  enabled: true
sriovNetworkOperator:
  enabled: false
# NicClusterPolicy CR values:
deployCR: true
ofedDriver:
  deploy: true
nvPeerDriver:
  deploy: false
rdmaSharedDevicePlugin:
  deploy: true
  resources:
    - name: rdma_shared_device_a
      ifNames: [ens1f0]
 
sriovDevicePlugin:
  deploy: false

换源

镜像由k8s.gcr.io/nfd/node-feature-discovery:v0.10.1改为k8s.mirror.nju.edu.cn/nfd/node-feature-discovery:v0.10.1

1
2
3
4
5
helm install network-operator nvidia/network-operator \
  -n nvidia-network-operator \
  --create-namespace \
  --version v23.4.0

helm -n nvidia-network-operator delete network-operator

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
cni0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 172.16.0.1  netmask 255.255.255.0  broadcast 172.16.0.255
        inet6 fe80::a8a4:94ff:fe9d:1ad7  prefixlen 64  scopeid 0x20<link>
        ether aa:a4:94:9d:1a:d7  txqueuelen 1000  (Ethernet)
        RX packets 80110935  bytes 18650620411 (18.6 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 99487744  bytes 28613894788 (28.6 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        inet6 fe80::42:86ff:fe02:d27c  prefixlen 64  scopeid 0x20<link>
        ether 02:42:86:02:d2:7c  txqueuelen 0  (Ethernet)
        RX packets 13066  bytes 1186946 (1.1 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 17922  bytes 63416343 (63.4 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.102.0.232  netmask 255.255.255.192  broadcast 10.102.0.255
        inet6 fe80::1196:cbfd:196c:8a44  prefixlen 64  scopeid 0x20<link>
        ether 3c:ec:ef:af:13:4e  txqueuelen 1000  (Ethernet)
        RX packets 104502502  bytes 25757517277 (25.7 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 115177731  bytes 60906649040 (60.9 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eno2: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether 3c:ec:ef:af:13:4f  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp1s0np0: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST>  mtu 1500
        ether fa:96:48:fe:41:e2  txqueuelen 1000  (Ethernet)
        RX packets 7014  bytes 551988 (551.9 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 78639  bytes 7676560 (7.6 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp218s0np0: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST>  mtu 1500
        ether fa:96:48:fe:41:e2  txqueuelen 1000  (Ethernet)
        RX packets 3748  bytes 381852 (381.8 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 37969  bytes 4708156 (4.7 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp94s0f0np0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether 04:3f:72:c5:bc:2c  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp94s0f1np1: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether 04:3f:72:c5:bc:2d  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 172.16.0.0  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::141a:c0ff:fe1c:fdf1  prefixlen 64  scopeid 0x20<link>
        ether 16:1a:c0:1c:fd:f1  txqueuelen 0  (Ethernet)
        RX packets 2110480  bytes 1344473019 (1.3 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 6488097  bytes 2328670968 (2.3 GB)
        TX errors 0  dropped 84227 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 646196510  bytes 162890707590 (162.8 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 646196510  bytes 162890707590 (162.8 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

nm-bond: flags=5187<UP,BROADCAST,RUNNING,MASTER,MULTICAST>  mtu 1500
        inet 192.168.2.253  netmask 255.255.255.0  broadcast 192.168.2.255
        inet6 fe80::9dc6:903e:8e01:86f6  prefixlen 64  scopeid 0x20<link>
        ether fa:96:48:fe:41:e2  txqueuelen 1000  (Ethernet)
        RX packets 10762  bytes 933840 (933.8 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 116608  bytes 12384716 (12.3 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tailscale0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1280
        inet 100.100.229.13  netmask 255.255.255.255  destination 100.100.229.13
        inet6 fe80::113c:bb91:52c:c40c  prefixlen 64  scopeid 0x20<link>
        inet6 fd7a:115c:a1e0::9601:e50d  prefixlen 128  scopeid 0x0<global>
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 500  (UNSPEC)
        RX packets 93  bytes 9639 (9.6 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 29399  bytes 1763374 (1.7 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth332c833: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::fca8:5eff:fe04:4265  prefixlen 64  scopeid 0x20<link>
        ether fe:a8:5e:04:42:65  txqueuelen 0  (Ethernet)
        RX packets 6197  bytes 430702 (430.7 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 10512  bytes 62653516 (62.6 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth7e2c6a7: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::cc72:ceff:fe8a:50c3  prefixlen 64  scopeid 0x20<link>
        ether ce:72:ce:8a:50:c3  txqueuelen 0  (Ethernet)
        RX packets 6869  bytes 939168 (939.1 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 8475  bytes 860406 (860.4 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethaac0fb59: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::70d2:75ff:fe03:811c  prefixlen 64  scopeid 0x20<link>
        ether 72:d2:75:03:81:1c  txqueuelen 0  (Ethernet)
        RX packets 69425968  bytes 18240947181 (18.2 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 88915769  bytes 23173052947 (23.1 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethb97f0470: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::e4a1:c7ff:fef8:38af  prefixlen 64  scopeid 0x20<link>
        ether e2:70:dc:ca:3a:58  txqueuelen 0  (Ethernet)
        RX packets 463886  bytes 37139415 (37.1 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 560407  bytes 1650409709 (1.6 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethd027d541: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::c8dc:8aff:fea7:e008  prefixlen 64  scopeid 0x20<link>
        ether ca:dc:8a:a7:e0:08  txqueuelen 0  (Ethernet)
        RX packets 10205823  bytes 1492163802 (1.4 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 9998061  bytes 3776117503 (3.7 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
a-c-dream

a-c-dream

贵在坚持

109
13
0
CATEGORYS
TAGS