nvidia-smi

cnmon

注意事项

ib_send_bw测的是结果是client的

ib_write_bw测的是结果是client的

ib_read_bw测的是结果是server的

虚拟化rdma

测试时需要指定对应的master（原因未知）

测试内容

点对点通信（直接用ib perf工具，ib_write_bw和lat）？（这个可以直接测）
1. 英伟达、海光、寒武纪

品牌	ib_send_bw带宽测试	ib_send_lat时延测试	ib_write_bw带宽测试	ib_write_lat时延测试	ib_read_bw带宽测试	ib_read_lat时延测试
英伟达	3001.70	2681.85	3012.43	2678.63	3005.67	2681.04
海光	6086.29	1976.92	6086.33	2078.37	6089.41	1317.90
寒武纪	10943.00	736.30	10943.91	734.52	8661.59	1237.26

是否考虑传输size的影响（有些测试用例默认是2-2^23，先固定最大值测-s，之后看需求是否考虑测size影响）

考虑双边语义send和单边语义write,read测试

虚拟化对点对点通信是否影响，现在只有shared rdma方案，只对比这个虚拟化和裸机（对应1）？
1. 英伟达、海光、寒武纪：在对应的两个node起两个pod挂载rdma，测1中一样的指标：带宽和时延
2. 目前虚拟化，测集合通信，需要装相应的设备，驱动，xccl【这个待定,先不测】

品牌	ib_send_bw带宽测试	ib_send_lat时延测试	ib_write_bw带宽测试	ib_write_lat时延测试	ib_read_bw带宽测试	ib_read_lat时延测试
英伟达	2897.57	2702.51	2376.85	2710.28	2920.75	2722.04
海光	50.78	4238.79	50.97	4298.28	51.05	1318.77
寒武纪	10833.53	738.25	10889.70	738.22	8659.92	929.08

shared后容器中rdma的性能

网卡信息

英伟达233

mlx5_0 port 1 ==> enp1s0np0 (Up)
enp1s0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 08:c0:eb:cb:10:76 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.245/24 brd 192.168.2.255 scope global dynamic noprefixroute enp1s0np0
       valid_lft 363433sec preferred_lft 363433sec
    inet6 fe80::427c:9f32:747:453f/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

mlx5_3 port 1 ==> enp218s0np0 (Up)
enp218s0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 08:c0:eb:c8:97:a0 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.244/24 brd 192.168.2.255 scope global dynamic noprefixroute enp218s0np0
       valid_lft 363433sec preferred_lft 363433sec
    inet6 fe80::fcd2:9078:5727:ada9/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

英伟达232

mlx5_0 port 1 ==> enp1s0np0 (Up)
enp1s0np0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.2.243  netmask 255.255.255.0  broadcast 192.168.2.255
        inet6 fe80::65b:b12d:b223:f868  prefixlen 64  scopeid 0x20<link>
        ether 08:c0:eb:c7:9a:1e  txqueuelen 1000  (Ethernet)
        RX packets 377  bytes 66885 (66.8 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 473  bytes 75361 (75.3 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

mlx5_3 port 1 ==> enp218s0np0 (Up)
enp218s0np0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.2.242  netmask 255.255.255.0  broadcast 192.168.2.255
        inet6 fe80::2967:3c46:51b4:6765  prefixlen 64  scopeid 0x20<link>
        ether 08:c0:eb:cb:11:06  txqueuelen 1000  (Ethernet)
        RX packets 39  bytes 3760 (3.7 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 167  bytes 14142 (14.1 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

寒武纪2

mlx5_0 port 1 ==> ens121f0np0 (Up)
ens121f0np0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.2.247  netmask 255.255.255.0  broadcast 192.168.2.255
        inet6 fe80::ac0:ebff:fef6:193a  prefixlen 64  scopeid 0x20<link>
        ether 08:c0:eb:f6:19:3a  txqueuelen 1000  (Ethernet)
        RX packets 376662  bytes 50024262 (50.0 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 522  bytes 34660 (34.6 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



mlx5_1 port 1 ==> ens121f1np1 (Up)
ens121f1np1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.2.248  netmask 255.255.255.0  broadcast 192.168.2.255
        inet6 fe80::ac0:ebff:fef6:193b  prefixlen 64  scopeid 0x20<link>
        ether 08:c0:eb:f6:19:3b  txqueuelen 1000  (Ethernet)
        RX packets 376823  bytes 50048794 (50.0 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1072  bytes 96910 (96.9 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

寒武纪1

mlx5_0 port 1 ==> ens121f0np0 (Up)
ens121f0np0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.2.251  netmask 255.255.255.0  broadcast 192.168.2.255
        inet6 fe80::ac0:ebff:feb6:e7c4  prefixlen 64  scopeid 0x20<link>
        ether 08:c0:eb:b6:e7:c4  txqueuelen 1000  (Ethernet)
        RX packets 461801  bytes 61249363 (61.2 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 718  bytes 47417 (47.4 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

mlx5_1 port 1 ==> ens121f1np1 (Up)
ens121f1np1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.2.254  netmask 255.255.255.0  broadcast 192.168.2.255
        inet6 fe80::ac0:ebff:feb6:e7c5  prefixlen 64  scopeid 0x20<link>
        ether 08:c0:eb:b6:e7:c5  txqueuelen 1000  (Ethernet)
        RX packets 460924  bytes 61209639 (61.2 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1157  bytes 103323 (103.3 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

海光1

mlx5_0 port 1 ==> ens61f0np0 (Up)
ens61f0np0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.2.241  netmask 255.255.255.0  broadcast 192.168.2.255
        inet6 fe80::7376:ffd5:d1a2:d4b  prefixlen 64  scopeid 0x20<link>
        ether 0c:42:a1:df:c8:44  txqueuelen 1000  (Ethernet)
        RX packets 56  bytes 6579 (6.5 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 134  bytes 13257 (13.2 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



mlx5_1 port 1 ==> ens61f1np1 (Up)
ens61f1np1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.2.240  netmask 255.255.255.0  broadcast 192.168.2.255
        inet6 fe80::44fe:b707:6be6:be40  prefixlen 64  scopeid 0x20<link>
        ether 0c:42:a1:df:c8:45  txqueuelen 1000  (Ethernet)
        RX packets 50  bytes 6139 (6.1 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 88  bytes 9995 (9.9 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

海光2

mlx5_0 port 1 ==> ens61f0np0 (Up)
ens61f0np0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.2.249  netmask 255.255.255.0  broadcast 192.168.2.255
        inet6 fe80::bace:f6ff:fe05:ee32  prefixlen 64  scopeid 0x20<link>
        ether b8:ce:f6:05:ee:32  txqueuelen 1000  (Ethernet)
        RX packets 321138  bytes 42634072 (42.6 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 695  bytes 44455 (44.4 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


mlx5_1 port 1 ==> ens61f1np1 (Up)
ens61f1np1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.2.250  netmask 255.255.255.0  broadcast 192.168.2.255
        inet6 fe80::bace:f6ff:fe05:ee33  prefixlen 64  scopeid 0x20<link>
        ether b8:ce:f6:05:ee:33  txqueuelen 1000  (Ethernet)
        RX packets 321297  bytes 42650409 (42.6 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 845  bytes 55375 (55.3 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

rdma_share测试pod

test-cx5-bond-pod1.yaml

apiVersion: v1
kind: Pod
metadata:
  name: mofed-test-cx5-bond-pod1
  annotations:
    k8s.v1.cni.cncf.io/networks: default/macvlan-cx5-bond-conf
spec:
  nodeName: disai-4090-2
  restartPolicy: OnFailure
  containers:
  - image: registry.cn-hangzhou.aliyuncs.com/szy_is_me/rping-test
    imagePullPolicy: IfNotPresent
    name: mofed-test-ctr
    securityContext:
      privileged: true
      capabilities:
        add: [ "IPC_LOCK" ]
    resources:
      limits:
        rdma/cx5_bond_shared_devices_a: 1
      requests:
        rdma/cx5_bond_shared_devices_a: 1
    command:
    - sh
    - -c
    - |
      ls -l /dev/infiniband /sys/class/infiniband /sys/class/net
      sleep 1000000

test-cx5-bond-pod2.yaml

apiVersion: v1
kind: Pod
metadata:
  name: mofed-test-cx5-bond-pod2
  annotations:
    k8s.v1.cni.cncf.io/networks: default/macvlan-cx5-bond-conf
spec:
  nodeName: disai-4090-3
  restartPolicy: OnFailure
  containers:
  - image: registry.cn-hangzhou.aliyuncs.com/szy_is_me/rping-test
    imagePullPolicy: IfNotPresent
    name: mofed-test-ctr
    securityContext:
      privileged: true
      capabilities:
        add: [ "IPC_LOCK" ]
    resources:
      limits:
        rdma/cx5_bond_shared_devices_a: 1
      requests:
        rdma/cx5_bond_shared_devices_a: 1
    command:
    - sh
    - -c
    - |
      ls -l /dev/infiniband /sys/class/infiniband /sys/class/net
      sleep 1000000

ib_send_bw带宽测试

默认

海光2&

1 2	`ib_send_bw -d mlx5_0 -i 1 -s 8388608 ib_send_bw -d mlx5_0 -i 1 -s 8388608 192.168.2.241`

sunzhongyuan@disai-hygon-2:~$ ib_send_bw -d mlx5_0 -i 1 -s  8388608 192.168.2.241
---------------------------------------------------------------------------------------
                    Send BW Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 TX depth        : 128
 CQ Moderation   : 1
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 3
 Max inline data : 0[B]
 rdma_cm QPs	 : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x00cf PSN 0xd29bec
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:249
 remote address: LID 0000 QPN 0x0074 PSN 0x54bdf5
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:241
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
Conflicting CPU frequency values detected: 2300.097000 != 2485.220000. CPU Frequency is not max.
 8388608    1000             6086.30            6086.29		   0.000761
---------------------------------------------------------------------------------------

寒武纪2&

寒武纪1

寒武纪2

1 2	`ib_send_bw -d mlx5_0 -i 1 -s 8388608 ib_send_bw -d mlx5_0 -i 1 -s 8388608 192.168.2.251`

结果

sunzhongyuan@DisAI-Cambricon-2:~$ ib_send_bw -d mlx5_0 -i 1 -s 8388608 192.168.2.251
---------------------------------------------------------------------------------------
                    Send BW Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 TX depth        : 128
 CQ Moderation   : 1
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 3
 Max inline data : 0[B]
 rdma_cm QPs	 : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0622 PSN 0xb4b3bc
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:247
 remote address: LID 0000 QPN 0x0054 PSN 0x79882c
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:251
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
Conflicting CPU frequency values detected: 3300.000000 != 800.000000. CPU Frequency is not max.
 8388608    1000             10943.14            10943.00		   0.001368
---------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
Conflicting CPU frequency values detected: 3276.155000 != 3105.174000. CPU Frequency is not max.
 33554432    1000             11615.25            11537.82		   0.000361
---------------------------------------------------------------------------------------

NV&

233

232

1 2	`ib_send_bw -d mlx5_0 -i 1 -s 8388608 ib_send_bw -d mlx5_0 -i 1 -s 8388608 192.168.2.245`

结果

sunzhongyuan@DisAI-4090-2:~/rdma/rdma_share$ ib_send_bw -d mlx5_0 -i 1 -s 8388608 192.168.2.245
---------------------------------------------------------------------------------------
                    Send BW Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 TX depth        : 128
 CQ Moderation   : 1
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 3
 Max inline data : 0[B]
 rdma_cm QPs	 : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x01e0 PSN 0x844016
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:243
 remote address: LID 0000 QPN 0x011f PSN 0x7b33e3
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:245
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
Conflicting CPU frequency values detected: 1199.999000 != 3840.929000. CPU Frequency is not max.
 8388608    1000             3001.70            3001.70		   0.000375
---------------------------------------------------------------------------------------

海光

1 2	`ib_send_bw -d mlx5_0 -F --report_gbits -s 8388608 ib_send_bw -d mlx5_0 -F --report_gbits -s 8388608 10.56.217.75`

[root@mofed-test-cx5-bond-pod2 /]# ib_send_bw -d mlx5_0 -F --report_gbits -s 8388608 10.56.217.75
---------------------------------------------------------------------------------------
                    Send BW Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 4
 Max inline data : 0[B]
 rdma_cm QPs	 : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x014c PSN 0x2b47c4
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:76
 remote address: LID 0000 QPN 0x00f3 PSN 0xf62bf2
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:75
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 8388608    1000             51.02              50.78  		   0.000757
---------------------------------------------------------------------------------------

寒武纪

1 2	`ib_send_bw -d mlx5_0 -i 1 -s 8388608 ib_send_bw -d mlx5_0 -i 1 -s 8388608 10.56.217.78`

[root@mofed-test-cx5-bond-pod2 /]# ib_send_bw -d mlx5_0 -i 1 -s  8388608 10.56.217.78
---------------------------------------------------------------------------------------
                    Send BW Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 4
 Max inline data : 0[B]
 rdma_cm QPs	 : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x05ba PSN 0xd728e5
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:74
 remote address: LID 0000 QPN 0x0047 PSN 0xd728e5
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:78
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
Conflicting CPU frequency values detected: 3300.000000 != 800.000000. CPU Frequency is not max.
 8388608    1000             10833.57            10833.53		   0.001354
---------------------------------------------------------------------------------------

NV

233

232

1 2	`ib_send_bw -d mlx5_0 -F -s 8388608 ib_send_bw -d mlx5_0 -F -s 8388608 10.56.217.71`

[root@mofed-test-cx5-bond-pod1 /]# ib_send_bw -d mlx5_0 -F -s 8388608 10.56.217.71
---------------------------------------------------------------------------------------
                    Send BW Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 6
 Max inline data : 0[B]
 rdma_cm QPs	 : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x01ec PSN 0x3caa42
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:72
 remote address: LID 0000 QPN 0x012f PSN 0x9dd6ef
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:71
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 8388608    1000             2897.57            2897.57		   0.000362
---------------------------------------------------------------------------------------

ib_send_lat时延测试

默认

海光2&

1 2	`ib_send_lat -d mlx5_0 -a -F ib_send_lat -d mlx5_0 -i 1 192.168.2.241 -a -F`

结果

sunzhongyuan@disai-hygon-2:~$ ib_send_lat -d mlx5_0 -i 1 192.168.2.241  -a -F 
---------------------------------------------------------------------------------------
                    Send Latency Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 TX depth        : 1
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 3
 Max inline data : 236[B]
 rdma_cm QPs	 : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x00d1 PSN 0xd0a9c9
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:249
 remote address: LID 0000 QPN 0x0076 PSN 0xbf52ef
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:241
---------------------------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]    t_avg[usec]    t_stdev[usec]   99% percentile[usec]   99.9% percentile[usec] 
 2       1000          2.14           7.26         2.17     	       2.17        	0.10   		2.21    		7.26   
 4       1000          2.14           4.30         2.17     	       2.17        	0.03   		2.20    		4.30   
 8       1000          2.14           4.18         2.17     	       2.17        	0.03   		2.20    		4.18   
 16      1000          2.15           5.40         2.18     	       2.18        	0.03   		2.22    		5.40   
 32      1000          2.16           5.70         2.18     	       2.19        	0.03   		2.22    		5.70   
 64      1000          2.21           4.29         2.25     	       2.25        	0.03   		2.28    		4.29   
 128     1000          2.27           6.17         2.30     	       2.31        	0.06   		2.35    		6.17   
 256     1000          2.79           5.33         2.83     	       2.83        	0.00   		2.86    		5.33   
 512     1000          2.88           6.21         2.92     	       2.93        	0.05   		2.96    		6.21   
 1024    1000          3.03           4.74         3.07     	       3.07        	0.00   		3.11    		4.74   
 2048    1000          3.24           7.48         3.29     	       3.30        	0.04   		3.34    		7.48   
 4096    1000          3.57           6.76         3.63     	       3.64        	0.04   		3.68    		6.76   
 8192    1000          4.15           8.46         4.21     	       4.24        	0.08   		4.41    		8.46   
 16384   1000          5.33           5.77         5.39     	       5.39        	0.00   		5.48    		5.77   
 32768   1000          7.71           45.11        7.79     	       7.80        	0.10   		7.97    		45.11  
 65536   1000          12.39          16.24        12.46    	       12.48       	0.09   		12.65   		16.24  
 131072  1000          22.29          23.48        22.35    	       22.36       	0.03   		22.41   		23.48  
 262144  1000          41.45          49.19        41.52    	       41.53       	0.16   		41.60   		49.19  
 524288  1000          79.62          93.10        79.76    	       79.77       	0.21   		79.91   		93.10  
 1048576 1000          155.84         209.94       166.57   	       167.90      	9.08   		186.84  		209.94 
 2097152 1000          308.27         1960.04      320.01   	       317.79      	36.21  		401.02  		1960.04
 4194304 1000          635.66         3636.32      648.92   	       672.78      	188.09 		1270.92 		3636.32
 8388608 1000          1601.17        7919.81      1723.47  	       1976.92     	648.98 		5751.63 		7919.81
---------------------------------------------------------------------------------------

寒武纪2&

寒武纪1

寒武纪2

1 2	`ib_send_lat -R -d mlx5_0 -a -F ib_send_lat -R -d mlx5_0 -i 1 192.168.2.251 -a -F`

结果

sunzhongyuan@DisAI-Cambricon-2:~$ ib_send_lat -R  -d mlx5_0 -i 1 192.168.2.251  -a -F  
---------------------------------------------------------------------------------------
                    Send Latency Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 TX depth        : 1
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 3
 Max inline data : 236[B]
 rdma_cm QPs	 : ON
 Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x064e PSN 0xd3c7c2
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:248
 remote address: LID 0000 QPN 0x0056 PSN 0x3f781
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:251
---------------------------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]    t_avg[usec]    t_stdev[usec]   99% percentile[usec]   99.9% percentile[usec] 
 2       1000          2.14           6.09         2.19     	       2.20        	0.04   		2.37    		6.09   
 4       1000          2.14           5.46         2.19     	       2.20        	0.04   		2.34    		5.46   
 8       1000          2.13           3.89         2.19     	       2.20        	0.05   		2.35    		3.89   
 16      1000          2.14           6.01         2.20     	       2.21        	0.08   		2.37    		6.01   
 32      1000          2.15           5.18         2.21     	       2.22        	0.06   		2.36    		5.18   
 64      1000          2.21           3.89         2.26     	       2.27        	0.03   		2.46    		3.89   
 128     1000          2.25           5.77         2.32     	       2.36        	0.00   		3.02    		5.77   
 256     1000          2.94           6.38         3.01     	       3.02        	0.00   		3.23    		6.38   
 512     1000          3.05           6.59         3.12     	       3.14        	0.05   		3.33    		6.59   
 1024    1000          3.16           5.88         3.22     	       3.28        	0.03   		3.60    		5.88   
 2048    1000          3.28           8.50         3.37     	       3.40        	0.08   		3.68    		8.50   
 4096    1000          3.45           6.97         3.56     	       3.58        	0.07   		3.82    		6.97   
 8192    1000          3.81           4.73         3.91     	       3.91        	0.00   		4.08    		4.73   
 16384   1000          4.50           6.97         4.57     	       4.65        	0.08   		5.06    		6.97   
 32768   1000          5.91           7.72         6.02     	       6.07        	0.00   		6.47    		7.72   
 65536   1000          8.74           11.91        8.84     	       8.92        	0.12   		9.94    		11.91  
 131072  1000          14.64          16.65        14.90    	       14.92       	0.07   		15.32   		16.65  
 262144  1000          26.01          34.99        26.38    	       26.39       	0.11   		27.58   		34.99  
 524288  1000          48.57          51.91        49.15    	       49.18       	0.15   		50.56   		51.91  
 1048576 1000          94.24          98.97        95.15    	       95.20       	0.21   		97.01   		98.97  
 2097152 1000          185.60         195.19       186.69   	       186.77      	0.31   		188.65  		195.19 
 4194304 1000          368.42         379.03       370.11   	       370.19      	0.51   		372.58  		379.03 
 8388608 1000          734.02         740.90       736.16   	       736.30      	0.71   		738.96  		740.90 
---------------------------------------------------------------------------------------

NV&

233

232

1 2	`ib_send_lat -R -d mlx5_0 -a -F ib_send_lat -R -d mlx5_0 -i 1 192.168.2.245 -a -F`

sunzhongyuan@DisAI-4090-2:~/nccl_test/nccl-tests$ ib_send_lat -R  -d mlx5_0 -i 1 192.168.2.245  -a -F  
---------------------------------------------------------------------------------------
                    Send Latency Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 TX depth        : 1
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 3
 Max inline data : 236[B]
 rdma_cm QPs	 : ON
 Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x01e2 PSN 0x55c86e
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:243
 remote address: LID 0000 QPN 0x0121 PSN 0x624ee
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:245
---------------------------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]    t_avg[usec]    t_stdev[usec]   99% percentile[usec]   99.9% percentile[usec] 
 2       1000          2.27           9.83         2.33     	       2.34        	0.16   		2.43    		9.83   
 4       1000          2.25           9.01         2.28     	       2.34        	0.47   		6.85    		9.01   
 8       1000          2.25           9.46         2.28     	       2.32        	0.42   		2.43    		9.46   
 16      1000          2.26           9.52         2.30     	       2.31        	0.23   		2.48    		9.52   
 32      1000          2.29           9.15         2.32     	       2.35        	0.32   		3.18    		9.15   
 64      1000          2.36           8.70         2.40     	       2.43        	0.31   		2.78    		8.70   
 128     1000          2.45           9.30         2.50     	       2.51        	0.20   		2.55    		9.30   
 256     1000          3.32           8.77         3.37     	       3.39        	0.23   		3.58    		8.77   
 512     1000          3.49           10.95        3.54     	       3.62        	0.54   		8.34    		10.95  
 1024    1000          3.86           9.04         3.90     	       3.92        	0.16   		3.96    		9.04   
 2048    1000          4.35           10.50        4.40     	       4.43        	0.31   		4.59    		10.50  
 4096    1000          5.24           13.08        5.34     	       5.41        	0.42   		8.50    		13.08  
 8192    1000          6.53           13.36        6.59     	       6.63        	0.32   		7.04    		13.36  
 16384   1000          9.46           15.45        9.75     	       9.81        	0.42   		13.52   		15.45  
 32768   1000          16.40          22.62        16.48    	       16.59       	0.43   		19.85   		22.62  
 65536   1000          26.33          33.55        26.40    	       26.51       	0.43   		29.10   		33.55  
 131072  1000          46.36          51.55        46.44    	       46.52       	0.30   		48.50   		51.55  
 262144  1000          91.07          97.84        92.16    	       92.24       	0.47   		96.19   		97.84  
 524288  1000          172.58         192.28       174.08   	       174.12      	0.42   		177.00  		192.28 
 1048576 1000          338.52         345.76       339.28   	       339.31      	0.27   		340.43  		345.76 
 2097152 1000          669.73         678.61       670.64   	       670.70      	0.32   		673.13  		678.61 
 4194304 1000          1332.97        1346.43      1334.15  	       1334.43     	0.98   		1340.49 		1346.43
 8388608 1000          2679.61        2705.95      2681.46  	       2681.85     	1.01   		2686.87 		2705.95
---------------------------------------------------------------------------------------

海光

1 2	`ib_send_lat -d mlx5_0 -F --report_gbits -s 8388608 ib_send_lat -d mlx5_0 -F --report_gbits -s 8388608 10.56.217.75`

[root@mofed-test-cx5-bond-pod2 /]# ib_send_lat -d mlx5_0 -F --report_gbits -s 8388608 10.56.217.75
---------------------------------------------------------------------------------------
                    Send Latency Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 TX depth        : 1
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 4
 Max inline data : 236[B]
 rdma_cm QPs	 : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x014d PSN 0xddda08
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:76
 remote address: LID 0000 QPN 0x00f4 PSN 0x4628e4
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:75
---------------------------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]    t_avg[usec]    t_stdev[usec]   99% percentile[usec]   99.9% percentile[usec] 
 8388608 1000          1599.23        9609.64      4285.57  	       4238.79     	383.24 		5101.01 		9609.64
---------------------------------------------------------------------------------------

寒武纪

1 2	`ib_send_lat -R -d mlx5_0 -a -F ib_send_lat -R -d mlx5_0 -i 1 10.56.217.78 -a -F`

[root@mofed-test-cx5-bond-pod2 /]# ib_send_lat -R  -d mlx5_0 -i 1 10.56.217.78  -a -F  
---------------------------------------------------------------------------------------
                    Send Latency Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 TX depth        : 1
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 4
 Max inline data : 236[B]
 rdma_cm QPs	 : ON
 Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x05bc PSN 0x99949a
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:74
 remote address: LID 0000 QPN 0x0049 PSN 0x99949a
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:78
---------------------------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]    t_avg[usec]    t_stdev[usec]   99% percentile[usec]   99.9% percentile[usec] 
 2       1000          2.15           4.65         2.20     	       2.21        	0.09   		2.36    		4.65   
 4       1000          2.16           4.76         2.20     	       2.22        	0.09   		2.46    		4.76   
 8       1000          2.16           6.18         2.20     	       2.21        	0.07   		2.36    		6.18   
 16      1000          2.17           4.49         2.21     	       2.22        	0.07   		2.35    		4.49   
 32      1000          2.17           5.14         2.22     	       2.23        	0.07   		2.39    		5.14   
 64      1000          2.22           5.59         2.27     	       2.28        	0.10   		2.44    		5.59   
 128     1000          2.26           5.88         2.32     	       2.34        	0.13   		2.58    		5.88   
 256     1000          2.96           6.57         3.04     	       3.06        	0.15   		3.30    		6.57   
 512     1000          3.05           6.01         3.13     	       3.16        	0.16   		3.47    		6.01   
 1024    1000          3.18           6.05         3.26     	       3.30        	0.11   		3.55    		6.05   
 2048    1000          3.29           7.79         3.39     	       3.43        	0.14   		3.67    		7.79   
 4096    1000          3.44           5.65         3.61     	       3.65        	0.15   		3.96    		5.65   
 8192    1000          3.82           7.82         3.91     	       3.92        	0.08   		4.13    		7.82   
 16384   1000          4.63           9.46         4.81     	       4.84        	0.18   		5.22    		9.46   
 32768   1000          6.05           9.41         6.25     	       6.26        	0.15   		6.57    		9.41   
 65536   1000          8.87           20.70        9.09     	       9.11        	0.37   		9.54    		20.70  
 131072  1000          14.72          18.14        15.04    	       15.06       	0.22   		16.38   		18.14  
 262144  1000          25.99          28.81        26.47    	       26.47       	0.23   		27.51   		28.81  
 524288  1000          48.76          51.99        49.31    	       49.37       	0.32   		50.85   		51.99  
 1048576 1000          94.59          98.70        95.50    	       95.56       	0.44   		97.32   		98.70  
 2097152 1000          186.29         196.66       187.26   	       187.33      	0.58   		189.59  		196.66 
 4194304 1000          369.17         374.04       371.02   	       371.12      	0.77   		373.40  		374.04 
 8388608 1000          735.49         743.62       738.10   	       738.25      	1.04   		741.17  		743.62 
---------------------------------------------------------------------------------------

NV

233

232

1 2	`ib_send_lat -R -d mlx5_0 -a -F ib_send_lat -R -d mlx5_0 -i 1 10.56.217.71 -a -F`

[root@mofed-test-cx5-bond-pod1 /]# ib_send_lat -R  -d mlx5_0 -i 1 10.56.217.71  -a -F 
---------------------------------------------------------------------------------------
                    Send Latency Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 TX depth        : 1
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 6
 Max inline data : 236[B]
 rdma_cm QPs	 : ON
 Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x01ee PSN 0xc891f6
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:72
 remote address: LID 0000 QPN 0x0131 PSN 0xb9f04a
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:71
---------------------------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]    t_avg[usec]    t_stdev[usec]   99% percentile[usec]   99.9% percentile[usec] 
 2       1000          2.44           9.11         2.57     	       2.59        	0.27   		2.74    		9.11   
 4       1000          2.37           4.84         2.41     	       2.42        	0.06   		2.53    		4.84   
 8       1000          2.43           6.41         2.47     	       2.48        	0.14   		2.58    		6.41   
 16      1000          2.44           9.41         2.48     	       2.50        	0.25   		2.60    		9.41   
 32      1000          2.38           9.57         2.42     	       2.45        	0.32   		2.60    		9.57   
 64      1000          2.46           9.43         2.51     	       2.56        	0.51   		5.89    		9.43   
 128     1000          2.55           10.31        2.60     	       2.66        	0.52   		6.19    		10.31  
 256     1000          3.43           8.41         3.47     	       3.50        	0.29   		3.72    		8.41   
 512     1000          3.58           7.22         3.64     	       3.66        	0.13   		3.89    		7.22   
 1024    1000          3.93           9.15         3.98     	       4.01        	0.23   		4.30    		9.15   
 2048    1000          4.46           10.66        4.52     	       4.54        	0.20   		4.73    		10.66  
 4096    1000          5.36           13.27        5.48     	       5.55        	0.56   		8.80    		13.27  
 8192    1000          6.71           12.64        6.78     	       6.84        	0.39   		9.07    		12.64  
 16384   1000          9.59           15.35        9.94     	       9.99        	0.39   		13.17   		15.35  
 32768   1000          16.73          23.19        16.83    	       16.92       	0.41   		18.51   		23.19  
 65536   1000          26.99          33.98        27.08    	       27.19       	0.53   		29.72   		33.98  
 131072  1000          47.57          54.78        47.68    	       47.75       	0.35   		49.64   		54.78  
 262144  1000          92.16          97.14        93.27    	       93.31       	0.31   		94.97   		97.14  
 524288  1000          175.63         184.12       176.77   	       176.81      	0.60   		180.64  		184.12 
 1048576 1000          343.29         351.31       344.19   	       344.28      	0.55   		347.31  		351.31 
 2097152 1000          678.72         685.71       679.66   	       679.81      	0.63   		682.79  		685.71 
 4194304 1000          1349.68        1361.09      1351.50  	       1352.04     	1.57   		1357.84 		1361.09
 8388608 1000          2698.65        2711.26      2701.94  	       2702.51     	2.45   		2709.45 		2711.26
---------------------------------------------------------------------------------------

ib_write_bw带宽测试

默认

海光2&

1 2	`ib_write_bw -R -d mlx5_0 -a -F ib_write_bw -R -d mlx5_0 -i 1 192.168.2.241 -a -F`

结果

sunzhongyuan@disai-hygon-2:~$ ib_write_bw -R  -d mlx5_0 -i 1 192.168.2.241 -a -F
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 3
 Max inline data : 0[B]
 rdma_cm QPs	 : ON
 Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x01da PSN 0xdc79e0
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:250
 remote address: LID 0000 QPN 0x0078 PSN 0xe4c514
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:241
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 2          5000             5.49               5.48   		   2.873503
 4          5000             10.95              10.95  		   2.869347
 8          5000             21.96              21.82  		   2.860402
 16         5000             43.87              43.83  		   2.872743
 32         5000             87.63              87.11  		   2.854327
 64         5000             175.05             174.70 		   2.862285
 128        5000             352.29             351.91 		   2.882847
 256        5000             702.81             698.53 		   2.861171
 512        5000             1400.37            1397.18		   2.861416
 1024       5000             2807.74            2803.45		   2.870731
 2048       5000             5373.66            5372.01		   2.750467
 4096       5000             5930.53            5930.17		   1.518123
 8192       5000             5989.84            5989.30		   0.766631
 16384      5000             6033.01            6032.08		   0.386053
 32768      5000             6059.52            6059.13		   0.193892
 65536      5000             6072.83            6072.66		   0.097163
 131072     5000             6079.64            6079.55		   0.048636
 262144     5000             6083.11            6083.02		   0.024332
 524288     5000             6084.78            6084.74		   0.012169
 1048576    5000             6085.73            6085.61		   0.006086
 2097152    5000             6086.11            6086.03		   0.003043
 4194304    5000             6086.25            6086.23		   0.001522
 8388608    5000             6086.35            6086.33		   0.000761
---------------------------------------------------------------------------------------

寒武纪2&

寒武纪1

寒武纪2

1 2	`ib_write_bw -R -d mlx5_0 -a -F ib_write_bw -R -d mlx5_0 -i 1 192.168.2.251 -a -F`

结果

sunzhongyuan@DisAI-Cambricon-2:~$ ib_write_bw -R  -d mlx5_0 -i 1 192.168.2.251 -a -F  
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 3
 Max inline data : 0[B]
 rdma_cm QPs	 : ON
 Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0650 PSN 0x8ff343
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:248
 remote address: LID 0000 QPN 0x0058 PSN 0x55ad5b
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:251
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 2          5000             11.14              11.07  		   5.801315
 4          5000             26.82              26.56  		   6.963417
 8          5000             53.49              52.90  		   6.933529
 16         5000             107.55             106.63 		   6.988051
 32         5000             215.11             213.80 		   7.005785
 64         5000             425.59             418.93 		   6.863765
 128        5000             846.63             840.81 		   6.887941
 256        5000             1653.48            1627.24		   6.665165
 512        5000             3255.95            3227.59		   6.610111
 1024       5000             6133.47            6058.19		   6.203585
 2048       5000             9953.37            9920.82		   5.079459
 4096       5000             10906.92            10893.36		   2.788699
 8192       5000             10936.32            10928.30		   1.398822
 16384      5000             10951.12            10944.93		   0.700476
 32768      5000             10962.87            10962.39		   0.350796
 65536      5000             10961.47            10959.98		   0.175360
 131072     5000             10956.56            10955.85		   0.087647
 262144     5000             10954.50            10953.30		   0.043813
 524288     5000             10950.07            10949.63		   0.021899
 1048576    5000             10954.88            10953.37		   0.010953
 2097152    5000             10948.91            10947.91		   0.005474
 4194304    5000             10943.88            10943.35		   0.002736
 8388608    5000             10944.67            10943.91		   0.001368
---------------------------------------------------------------------------------------

NV&

233

232

1 2	`ib_write_bw -R -d mlx5_0 -a -F ib_write_bw -R -d mlx5_0 -i 1 192.168.2.245 -a -F`

sunzhongyuan@DisAI-4090-2:~/nccl_test/nccl-tests$ ib_write_bw -R  -d mlx5_0 -i 1 192.168.2.245  -a -F 
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 3
 Max inline data : 0[B]
 rdma_cm QPs	 : ON
 Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x01e4 PSN 0xab6e7
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:243
 remote address: LID 0000 QPN 0x0123 PSN 0x762bbe
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:245
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 2          5000             6.73               6.67   		   3.496033
 4          5000             20.27              20.07  		   5.260603
 8          5000             40.85              40.72  		   5.337253
 16         5000             100.29             97.25  		   6.373204
 32         5000             202.97             199.41 		   6.534406
 64         5000             401.18             380.60 		   6.235782
 128        5000             783.94             768.07 		   6.292036
 256        5000             1457.57            1433.55		   5.871804
 512        5000             2256.15            2240.20		   4.587924
 1024       5000             2670.32            2662.19		   2.726086
 2048       5000             2843.92            2837.60		   1.452852
 4096       5000             2911.24            2908.66		   0.744618
 8192       5000             2958.11            2957.59		   0.378571
 16384      5000             2986.16            2986.08		   0.191109
 32768      5000             3005.56            3004.86		   0.096156
 65536      5000             3011.61            3011.00		   0.048176
 131072     5000             3014.33            3014.33		   0.024115
 262144     5000             3015.81            3015.69		   0.012063
 524288     5000             3016.75            3016.65		   0.006033
 1048576    5000             3016.59            3016.32		   0.003016
 2097152    5000             3015.23            3015.21		   0.001508
 4194304    5000             3015.19            3015.13		   0.000754
 8388608    5000             3015.22            3012.43		   0.000377
---------------------------------------------------------------------------------------

NV

1 2	`ib_write_bw -d mlx5_0 -F -s 8388608 ib_write_bw -d mlx5_0 -F -s 8388608 10.56.217.71`

[root@mofed-test-cx5-bond-pod1 /]# ib_write_bw -d mlx5_0 -F  -s 8388608 10.56.217.71
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 6
 Max inline data : 0[B]
 rdma_cm QPs	 : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x01f0 PSN 0xb8adee RKey 0x1832f1 VAddr 0x007ebd6678a000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:72
 remote address: LID 0000 QPN 0x0133 PSN 0x4cba6e RKey 0x182bea VAddr 0x0077bc49c17000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:71
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 8388608    5000             2390.86            2376.85		   0.000297
---------------------------------------------------------------------------------------

海光

1 2	`ib_write_bw -d mlx5_0 -F --report_gbits -s 8388608 ib_write_bw -d mlx5_0 -F --report_gbits -s 8388608 10.56.217.75`

[root@mofed-test-cx5-bond-pod2 /]# ib_write_bw -d mlx5_0 -F --report_gbits -s 8388608 10.56.217.75
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 4
 Max inline data : 0[B]
 rdma_cm QPs	 : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x014b PSN 0xf03762 RKey 0x1820df VAddr 0x007fce1c470000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:76
 remote address: LID 0000 QPN 0x00f2 PSN 0xbe7f8a RKey 0x1826e5 VAddr 0x007f9fc7ed6000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:75
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 8388608    5000             51.02              50.97  		   0.000759
---------------------------------------------------------------------------------------

寒武纪

[root@mofed-test-cx5-bond-pod2 /]# ib_write_bw -R  -d mlx5_0 -i 1 10.56.217.78  -a -F 
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 4
 Max inline data : 0[B]
 rdma_cm QPs	 : ON
 Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x05be PSN 0x94df97
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:74
 remote address: LID 0000 QPN 0x004b PSN 0x94df97
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:78
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 2          5000             7.89               7.81   		   4.093379
 4          5000             18.88              18.70  		   4.901774
 8          5000             37.84              37.63  		   4.932182
 16         5000             75.68              75.06  		   4.919003
 32         5000             152.52             151.26 		   4.956602
 64         5000             304.46             300.17 		   4.917929
 128        5000             597.44             591.34 		   4.844218
 256        5000             1217.85            1197.74		   4.905958
 512        5000             2398.80            2367.05		   4.847713
 1024       5000             4806.64            4756.40		   4.870555
 2048       5000             8689.97            8658.55		   4.433178
 4096       5000             10848.55            10833.89		   2.773477
 8192       5000             10842.71            10834.71		   1.386843
 16384      5000             10871.76            10869.13		   0.695624
 32768      5000             10881.97            10881.51		   0.348208
 65536      5000             10881.90            10879.29		   0.174069
 131072     5000             10895.19            10894.35		   0.087155
 262144     5000             10896.20            10895.23		   0.043581
 524288     5000             10896.96            10896.63		   0.021793
 1048576    5000             10900.23            10899.92		   0.010900
 2097152    5000             10893.30            10893.29		   0.005447
 4194304    5000             10891.89            10890.96		   0.002723
 8388608    5000             10890.38            10889.70		   0.001361
---------------------------------------------------------------------------------------

ib_write_lat时延测试

默认

海光2&

1 2	`ib_write_lat -R -d mlx5_0 -a -F ib_write_lat -R -d mlx5_0 -i 1 192.168.2.241 -a -F`

结果

sunzhongyuan@disai-hygon-2:~$ ib_write_lat -R  -d mlx5_0 -i 1 192.168.2.241  -a -F 
---------------------------------------------------------------------------------------
                    RDMA_Write Latency Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 PCIe relax order: OFF
 ibv_wr* API     : ON
 TX depth        : 1
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 3
 Max inline data : 220[B]
 rdma_cm QPs	 : ON
 Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x01dc PSN 0xc73bc3
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:250
 remote address: LID 0000 QPN 0x007a PSN 0x43be5d
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:241
---------------------------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]    t_avg[usec]    t_stdev[usec]   99% percentile[usec]   99.9% percentile[usec] 
 2       1000          2.02           3.71         2.12     	       2.12        	0.03   		2.14    		3.71   
 4       1000          2.02           4.22         2.12     	       2.12        	0.00   		2.14    		4.22   
 8       1000          2.02           5.75         2.12     	       2.12        	0.00   		2.14    		5.75   
 16      1000          2.05           3.83         2.12     	       2.12        	0.00   		2.14    		3.83   
 32      1000          2.07           5.81         2.15     	       2.15        	0.00   		2.17    		5.81   
 64      1000          2.08           3.43         2.17     	       2.17        	0.00   		2.19    		3.43   
 128     1000          2.17           3.02         2.25     	       2.25        	0.00   		2.27    		3.02   
 256     1000          2.67           4.74         2.76     	       2.76        	0.03   		2.78    		4.74   
 512     1000          2.83           4.77         2.85     	       2.85        	0.00   		2.87    		4.77   
 1024    1000          2.91           5.50         3.01     	       3.01        	0.03   		3.04    		5.50   
 2048    1000          3.10           3.33         3.20     	       3.21        	0.00   		3.23    		3.33   
 4096    1000          3.44           4.92         3.56     	       3.56        	0.00   		3.59    		4.92   
 8192    1000          4.04           6.21         4.15     	       4.18        	0.04   		4.35    		6.21   
 16384   1000          5.24           5.54         5.33     	       5.33        	0.00   		5.48    		5.54   
 32768   1000          7.60           10.06        7.81     	       7.79        	0.05   		7.97    		10.06  
 65536   1000          12.27          15.34        12.39    	       12.40       	0.00   		12.52   		15.34  
 131072  1000          22.19          24.76        22.29    	       22.30       	0.08   		22.44   		24.76  
 262144  1000          41.37          66.40        41.45    	       41.46       	0.06   		41.50   		66.40  
 524288  1000          79.58          82.13        79.69    	       79.69       	0.04   		79.81   		82.13  
 1048576 1000          155.75         209.79       162.57   	       164.52      	7.90   		182.61  		209.79 
 2097152 1000          308.16         1834.61      319.90   	       351.04      	203.62 		1585.67 		1834.61
 4194304 1000          637.01         3282.46      650.15   	       666.07      	97.81  		1246.00 		3282.46
 8388608 1000          1613.34        7341.64      1753.70  	       2078.37     	693.13 		5286.40 		7341.64
---------------------------------------------------------------------------------------

寒武纪2&

寒武纪1

寒武纪2

1 2	`ib_write_lat -R -d mlx5_0 -a -F ib_write_lat -R -d mlx5_0 -i 1 192.168.2.251 -a -F`

结果

sunzhongyuan@DisAI-Cambricon-2:~$ ib_write_lat -R  -d mlx5_0 -i 1 192.168.2.251  -a -F
---------------------------------------------------------------------------------------
                    RDMA_Write Latency Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 PCIe relax order: OFF
 ibv_wr* API     : ON
 TX depth        : 1
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 3
 Max inline data : 220[B]
 rdma_cm QPs	 : ON
 Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0652 PSN 0xba0885
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:248
 remote address: LID 0000 QPN 0x005a PSN 0x9df2dd
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:251
---------------------------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]    t_avg[usec]    t_stdev[usec]   99% percentile[usec]   99.9% percentile[usec] 
 2       1000          2.07           3.85         2.10     	       2.10        	0.03   		2.17    		3.85   
 4       1000          2.07           4.06         2.10     	       2.10        	0.00   		2.17    		4.06   
 8       1000          2.07           4.15         2.10     	       2.11        	0.04   		2.17    		4.15   
 16      1000          2.08           4.11         2.10     	       2.11        	0.03   		2.17    		4.11   
 32      1000          2.11           4.38         2.14     	       2.14        	0.03   		2.22    		4.38   
 64      1000          2.12           3.32         2.14     	       2.15        	0.03   		2.21    		3.32   
 128     1000          2.17           3.98         2.20     	       2.21        	0.03   		2.27    		3.98   
 256     1000          2.80           5.41         2.85     	       2.86        	0.04   		3.06    		5.41   
 512     1000          2.90           4.50         2.94     	       2.96        	0.03   		3.19    		4.50   
 1024    1000          3.02           5.91         3.08     	       3.12        	0.09   		3.33    		5.91   
 2048    1000          3.11           6.87         3.18     	       3.21        	0.04   		3.41    		6.87   
 4096    1000          3.28           5.77         3.41     	       3.44        	0.05   		3.71    		5.77   
 8192    1000          3.66           5.28         3.83     	       3.82        	0.03   		3.97    		5.28   
 16384   1000          4.36           6.35         4.51     	       4.53        	0.06   		4.75    		6.35   
 32768   1000          5.78           8.26         5.87     	       5.89        	0.08   		6.12    		8.26   
 65536   1000          8.61           10.20        8.75     	       8.78        	0.06   		9.00    		10.20  
 131072  1000          14.38          16.20        14.66    	       14.68       	0.05   		15.05   		16.20  
 262144  1000          25.64          28.93        26.11    	       26.14       	0.13   		27.27   		28.93  
 524288  1000          48.35          50.50        48.88    	       48.90       	0.12   		50.15   		50.50  
 1048576 1000          94.00          107.43       94.79    	       94.84       	0.21   		96.43   		107.43 
 2097152 1000          185.06         189.19       186.12   	       186.16      	0.20   		187.78  		189.19 
 4194304 1000          367.62         371.61       369.13   	       369.19      	0.39   		371.20  		371.61 
 8388608 1000          732.14         738.09       734.47   	       734.52      	0.68   		737.20  		738.09 
---------------------------------------------------------------------------------------

NV&

233

232

1 2	`ib_write_lat -R -d mlx5_0 -a -F ib_write_lat -R -d mlx5_0 -i 1 192.168.2.245 -a -F`

sunzhongyuan@DisAI-4090-2:~/nccl_test/nccl-tests$ ib_write_lat -R  -d mlx5_0 -i 1 192.168.2.245 -a -F
---------------------------------------------------------------------------------------
                    RDMA_Write Latency Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 PCIe relax order: OFF
 ibv_wr* API     : ON
 TX depth        : 1
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 3
 Max inline data : 220[B]
 rdma_cm QPs	 : ON
 Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x01e6 PSN 0x903f9a
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:243
 remote address: LID 0000 QPN 0x0125 PSN 0x74fffa
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:245
---------------------------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]    t_avg[usec]    t_stdev[usec]   99% percentile[usec]   99.9% percentile[usec] 
 2       1000          2.29           4.48         2.32     	       2.33        	0.00   		2.38    		4.48   
 4       1000          2.23           8.51         2.25     	       2.26        	0.14   		2.28    		8.51   
 8       1000          2.23           8.30         2.25     	       2.26        	0.20   		2.28    		8.30   
 16      1000          2.23           9.18         2.25     	       2.27        	0.22   		2.31    		9.18   
 32      1000          2.30           9.31         2.33     	       2.36        	0.33   		2.49    		9.31   
 64      1000          2.31           9.79         2.34     	       2.35        	0.20   		2.39    		9.79   
 128     1000          2.43           9.21         2.46     	       2.48        	0.31   		2.50    		9.21   
 256     1000          3.25           9.80         3.28     	       3.31        	0.32   		3.87    		9.80   
 512     1000          3.42           9.82         3.45     	       3.49        	0.28   		4.36    		9.82   
 1024    1000          3.86           7.41         3.90     	       3.90        	0.04   		3.98    		7.41   
 2048    1000          4.36           8.17         4.40     	       4.42        	0.07   		4.55    		8.17   
 4096    1000          5.23           12.05        5.33     	       5.38        	0.37   		7.90    		12.05  
 8192    1000          6.54           13.53        6.59     	       6.62        	0.23   		6.78    		13.53  
 16384   1000          9.67           15.91        9.74     	       9.77        	0.27   		10.60   		15.91  
 32768   1000          16.40          20.75        16.45    	       16.49       	0.15   		17.67   		20.75  
 65536   1000          26.32          34.28        26.37    	       26.50       	0.61   		31.00   		34.28  
 131072  1000          46.32          54.15        46.39    	       46.50       	0.55   		50.61   		54.15  
 262144  1000          91.08          98.04        92.14    	       92.21       	0.45   		95.63   		98.04  
 524288  1000          173.19         180.43       174.20   	       174.31      	0.56   		178.82  		180.43 
 1048576 1000          338.55         347.51       339.34   	       339.65      	0.94   		345.47  		347.51 
 2097152 1000          669.53         685.56       670.41   	       670.73      	1.09   		676.73  		685.56 
 4194304 1000          1333.04        1349.62      1334.39  	       1335.44     	2.07   		1343.52 		1349.62
 8388608 1000          2676.42        2709.09      2678.11  	       2678.63     	1.43   		2685.25 		2709.09
---------------------------------------------------------------------------------------

NV

1 2	`ib_write_lat -d mlx5_0 -F -s 8388608 ib_write_lat -d mlx5_0 -F -s 8388608 10.56.217.71`

[root@mofed-test-cx5-bond-pod1 /]# ib_write_lat -d mlx5_0 -F  -s 8388608 10.56.217.71
---------------------------------------------------------------------------------------
                    RDMA_Write Latency Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 TX depth        : 1
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 6
 Max inline data : 220[B]
 rdma_cm QPs	 : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x01f1 PSN 0xf02d45 RKey 0x1832f1 VAddr 0x0077c2fe601000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:72
 remote address: LID 0000 QPN 0x0134 PSN 0x5608bc RKey 0x182bea VAddr 0x007d0535e3c000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:71
---------------------------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]    t_avg[usec]    t_stdev[usec]   99% percentile[usec]   99.9% percentile[usec] 
 8388608 1000          2702.60        2854.49      2707.20  	       2710.28     	19.58  		2852.19 		2854.49
---------------------------------------------------------------------------------------

海光

1 2	`ib_write_lat -d mlx5_0 -F --report_gbits -s 8388608 ib_write_lat -d mlx5_0 -F --report_gbits -s 8388608 10.56.217.75`

[root@mofed-test-cx5-bond-pod2 /]# ib_write_lat -d mlx5_0 -F --report_gbits -s 8388608 10.56.217.75
---------------------------------------------------------------------------------------
                    RDMA_Write Latency Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 TX depth        : 1
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 4
 Max inline data : 220[B]
 rdma_cm QPs	 : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x014e PSN 0xa6ee06 RKey 0x1820df VAddr 0x007f2206273000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:76
 remote address: LID 0000 QPN 0x00f5 PSN 0x7e725a RKey 0x1826e5 VAddr 0x007fddab8ae000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:75
---------------------------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]    t_avg[usec]    t_stdev[usec]   99% percentile[usec]   99.9% percentile[usec] 
 8388608 1000          2388.58        5709.59      4332.45  	       4298.28     	383.91 		5146.24 		5709.59
---------------------------------------------------------------------------------------

寒武纪

#mofed-test-cx5-bond-pod1执行
ib_write_lat -R  -d mlx5_0 -i 1 -a -F 
#mofed-test-cx5-bond-pod2执行，ip为mofed-test-cx5-bond-pod1中net1的ip
ib_write_lat -R  -d mlx5_0 -i 1 10.56.217.78  -a -F

[root@mofed-test-cx5-bond-pod2 /]# ib_write_lat -R  -d mlx5_0 -i 1 10.56.217.78  -a -F 
---------------------------------------------------------------------------------------
                    RDMA_Write Latency Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 TX depth        : 1
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 4
 Max inline data : 220[B]
 rdma_cm QPs	 : ON
 Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x05c0 PSN 0x3d9d90
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:74
 remote address: LID 0000 QPN 0x004e PSN 0x3d9d90
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:78
---------------------------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]    t_avg[usec]    t_stdev[usec]   99% percentile[usec]   99.9% percentile[usec] 
 2       1000          2.16           3.73         2.19     	       2.19        	0.04   		2.24    		3.73   
 4       1000          2.16           3.34         2.19     	       2.19        	0.04   		2.26    		3.34   
 8       1000          2.17           4.67         2.19     	       2.19        	0.07   		2.25    		4.67   
 16      1000          2.16           5.15         2.19     	       2.20        	0.09   		2.28    		5.15   
 32      1000          2.19           3.57         2.22     	       2.23        	0.05   		2.30    		3.57   
 64      1000          2.21           3.72         2.24     	       2.24        	0.03   		2.34    		3.72   
 128     1000          2.27           5.52         2.30     	       2.32        	0.09   		2.51    		5.52   
 256     1000          2.95           5.21         2.98     	       3.00        	0.09   		3.22    		5.21   
 512     1000          3.03           5.05         3.08     	       3.11        	0.10   		3.33    		5.05   
 1024    1000          3.15           6.70         3.22     	       3.26        	0.11   		3.45    		6.70   
 2048    1000          3.23           5.98         3.33     	       3.37        	0.12   		3.68    		5.98   
 4096    1000          3.43           6.20         3.62     	       3.64        	0.17   		3.96    		6.20   
 8192    1000          3.81           5.53         3.91     	       3.92        	0.09   		4.18    		5.53   
 16384   1000          4.52           7.58         4.68     	       4.69        	0.14   		4.87    		7.58   
 32768   1000          5.92           8.61         6.09     	       6.10        	0.10   		6.33    		8.61   
 65536   1000          8.75           11.25        8.90     	       8.93        	0.18   		10.24   		11.25  
 131072  1000          14.64          16.66        14.93    	       14.98       	0.21   		15.68   		16.66  
 262144  1000          26.08          28.90        26.37    	       26.41       	0.24   		27.76   		28.90  
 524288  1000          48.71          51.17        49.25    	       49.31       	0.33   		50.75   		51.17  
 1048576 1000          94.38          146.66       95.43    	       95.48       	0.43   		97.35   		146.66 
 2097152 1000          186.17         190.91       187.29   	       187.35      	0.55   		189.39  		190.91 
 4194304 1000          369.12         374.62       371.04   	       371.12      	0.81   		373.55  		374.62 
 8388608 1000          735.62         743.15       738.11   	       738.22      	1.04   		740.97  		743.15 
---------------------------------------------------------------------------------------

ib_read_bw带宽测试

默认

海光2&

1 2	`ib_read_bw -R -d mlx5_0 -a -F ib_read_bw -R -d mlx5_0 -i 1 192.168.2.249 -a -F`

结果

$ ib_read_bw -R  -d mlx5_0 -i 1 192.168.2.249  -a -F
---------------------------------------------------------------------------------------
                    RDMA_Read BW Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 3
 Outstand reads  : 16
 rdma_cm QPs	 : ON
 Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x007c PSN 0xd315ef
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:241
 remote address: LID 0000 QPN 0x00d3 PSN 0x78aa6e
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:249
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 2          1000             7.39               7.30   		   3.824692
 4          1000             14.80              14.77  		   3.871031
 8          1000             29.64              29.58  		   3.877635
 16         1000             59.18              59.10  		   3.873130
 32         1000             118.37             118.02 		   3.867139
 64         1000             236.74             235.84 		   3.864001
 128        1000             469.52             468.04 		   3.834224
 256        1000             940.61             939.48 		   3.848092
 512        1000             1743.94            1727.22		   3.537343
 1024       1000             3236.58            3235.07		   3.312714
 2048       1000             4748.81            4746.32		   2.430118
 4096       1000             5501.99            5500.83		   1.408213
 8192       1000             5946.23            5946.06		   0.761096
 16384      1000             6071.81            6071.21		   0.388558
 32768      1000             6084.14            6083.66		   0.194677
 65536      1000             6086.46            6086.22		   0.097379
 131072     1000             6089.04            6088.93		   0.048711
 262144     1000             6089.87            6089.84		   0.024359
 524288     1000             6089.94            6089.91		   0.012180
 1048576    1000             6090.02            6090.02		   0.006090
 2097152    1000             6089.79            6089.78		   0.003045
 4194304    1000             6089.76            6089.68		   0.001522
 8388608    1000             6089.41            6089.41		   0.000761
---------------------------------------------------------------------------------------

寒武纪2&

寒武纪2

寒武纪1

1 2	`ib_read_bw -R -d mlx5_0 -a -F ib_read_bw -R -d mlx5_0 -i 1 192.168.2.247 -a -F`

结果

$ ib_read_bw -R  -d mlx5_0 -i 1 192.168.2.247  -a -F 
---------------------------------------------------------------------------------------
                    RDMA_Read BW Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 3
 Outstand reads  : 16
 rdma_cm QPs	 : ON
 Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0149 PSN 0xa52024
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:254
 remote address: LID 0000 QPN 0x0624 PSN 0xd97664
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:247
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 2          1000             7.51               7.46   		   3.912179
 4          1000             15.04              14.88  		   3.899687
 8          1000             29.67              29.62  		   3.882782
 16         1000             58.81              58.67  		   3.845073
 32         1000             118.15             117.79 		   3.859634
 64         1000             233.51             233.04 		   3.818185
 128        1000             458.89             457.18 		   3.745253
 256        1000             923.15             918.43 		   3.761882
 512        1000             1776.37            1768.92		   3.622748
 1024       1000             3328.64            3311.10		   3.390564
 2048       1000             6045.60            6036.33		   3.090602
 4096       1000             9364.63            9357.73		   2.395578
 8192       1000             10365.66            10365.37		   1.326767
 16384      1000             10411.01            10410.24		   0.666256
 32768      1000             10255.49            4331.13		   0.138596
 65536      1000             6341.21            6341.07		   0.101457
 131072     1000             8518.45            8516.08		   0.068129
 262144     1000             9044.75            8901.77		   0.035607
 524288     1000             9081.51            3604.61		   0.007209
 1048576    1000             8703.75            5167.14		   0.005167
 2097152    1000             8459.14            7728.63		   0.003864
 4194304    1000             8668.59            8404.49		   0.002101
 8388608    1000             8910.49            8661.59		   0.001083
---------------------------------------------------------------------------------------

NV&

232

233

1 2	`ib_read_bw -R -d mlx5_0 -a -F ib_read_bw -R -d mlx5_0 192.168.2.243 -a -F`

(base) sunzhongyuan@DisAI-4090-3:~$ ib_read_bw -R  -d mlx5_0 192.168.2.243  -a -F  
---------------------------------------------------------------------------------------
                    RDMA_Read BW Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 5
 Outstand reads  : 16
 rdma_cm QPs	 : ON
 Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0127 PSN 0xddb8be
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:245
 remote address: LID 0000 QPN 0x01e8 PSN 0xaa8334
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:243
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 2          1000             7.15               6.99   		   3.666950
 4          1000             14.40              14.39  		   3.771996
 8          1000             28.71              28.70  		   3.761187
 16         1000             57.66              57.62  		   3.776274
 32         1000             114.70             114.33 		   3.746220
 64         1000             230.63             227.27 		   3.723628
 128        1000             426.71             421.14 		   3.449990
 256        1000             839.80             839.07 		   3.436818
 512        1000             1424.15            1423.56		   2.915453
 1024       1000             2221.26            2220.12		   2.273407
 2048       1000             2604.17            2604.12		   1.333309
 4096       1000             2856.50            2855.84		   0.731094
 8192       1000             2962.92            2962.82		   0.379241
 16384      1000             2991.24            2991.13		   0.191432
 32768      1000             2980.24            2980.21		   0.095367
 65536      1000             2985.35            2985.31		   0.047765
 131072     1000             3001.87            3001.87		   0.024015
 262144     1000             3016.52            3016.52		   0.012066
 524288     1000             3016.78            3016.78		   0.006034
 1048576    1000             3017.30            3017.30		   0.003017
 2097152    1000             3015.62            3015.62		   0.001508
 4194304    1000             3015.49            3015.44		   0.000754
 8388608    1000             3014.89            3005.67		   0.000376
---------------------------------------------------------------------------------------

NV

1 2	`ib_read_bw -d mlx5_0 -F -s 8388608 ib_read_bw -d mlx5_0 -F -s 8388608 10.56.217.72`

[root@mofed-test-cx5-bond-pod2 /]# ib_read_bw -d mlx5_0 -F  -s 8388608 10.56.217.72
---------------------------------------------------------------------------------------
                    RDMA_Read BW Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 6
 Outstand reads  : 16
 rdma_cm QPs	 : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0135 PSN 0x87b40f OUT 0x10 RKey 0x182bea VAddr 0x0075b78204c000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:71
 remote address: LID 0000 QPN 0x01f2 PSN 0x20e8d6 OUT 0x10 RKey 0x1832f1 VAddr 0x0072db34d41000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:72
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 8388608    1000             2920.75            2920.75		   0.000365
---------------------------------------------------------------------------------------

海光

1 2	`ib_read_bw -d mlx5_0 -F --report_gbits -s 8388608 ib_read_bw -d mlx5_0 -F --report_gbits -s 8388608 10.56.217.76`

[root@mofed-test-cx5-bond-pod1 /]# ib_read_bw -d mlx5_0 -F --report_gbits -s 8388608 10.56.217.76
---------------------------------------------------------------------------------------
                    RDMA_Read BW Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 6
 Outstand reads  : 16
 rdma_cm QPs	 : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x00f6 PSN 0xa21140 OUT 0x10 RKey 0x1826e5 VAddr 0x007f3ec0e5d000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:75
 remote address: LID 0000 QPN 0x014f PSN 0x7ab352 OUT 0x10 RKey 0x1820df VAddr 0x007f37a80e4000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:76
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 8388608    1000             51.05              51.05  		   0.000761
---------------------------------------------------------------------------------------

寒武纪

#mofed-test-cx5-bond-pod2执行
ib_read_bw -R  -d mlx5_0 -i 1 -a -F 
#mofed-test-cx5-bond-pod1执行，ip为mofed-test-cx5-bond-pod2中net1的ip
ib_read_bw -R  -d mlx5_0 -i 1 10.56.217.78 -a -F

[root@mofed-test-cx5-bond-pod2 /]# ib_read_bw -R  -d mlx5_0 -i 1 10.56.217.78 -a -F 
---------------------------------------------------------------------------------------
                    RDMA_Read BW Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 4
 Outstand reads  : 16
 rdma_cm QPs	 : ON
 Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x05c2 PSN 0x4ea1c9
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:74
 remote address: LID 0000 QPN 0x0050 PSN 0x4ea1c9
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:78
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 2          1000             7.15               7.14   		   3.745089
 4          1000             14.40              14.31  		   3.750632
 8          1000             28.52              28.31  		   3.710647
 16         1000             57.03              56.55  		   3.705941
 32         1000             114.23             113.31 		   3.712835
 64         1000             228.79             227.23 		   3.722928
 128        1000             455.60             451.56 		   3.699213
 256        1000             902.11             884.42 		   3.622569
 512        1000             1688.75            1667.93		   3.415929
 1024       1000             3272.77            3264.23		   3.342572
 2048       1000             5770.26            5725.07		   2.931235
 4096       1000             9022.77            8996.38		   2.303074
 8192       1000             10355.21            10350.55		   1.324870
 16384      1000             10410.96            10410.20		   0.666253
 32768      1000             10361.77            4716.62		   0.150932
 65536      1000             6322.67            6322.49		   0.101160
 131072     1000             8946.73            8946.49		   0.071572
 262144     1000             9546.63            9090.27		   0.036361
 524288     1000             9448.09            5133.80		   0.010268
 1048576    1000             8773.05            8773.04		   0.008773
 2097152    1000             8481.43            8221.54		   0.004111
 4194304    1000             8896.21            8852.74		   0.002213
 8388608    1000             8679.52            8659.92		   0.001082
---------------------------------------------------------------------------------------

ib_read_lat时延测试

默认

海光2&

1 2	`ib_read_lat -R -d mlx5_0 -a -F ib_read_lat -R -d mlx5_0 -i 1 192.168.2.249 -a -F`

结果

$ ib_read_lat -R  -d mlx5_0 -i 1 192.168.2.249  -a -F  
---------------------------------------------------------------------------------------
                    RDMA_Read Latency Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 TX depth        : 1
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 3
 Outstand reads  : 16
 rdma_cm QPs	 : ON
 Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x007e PSN 0x4f4b7c
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:241
 remote address: LID 0000 QPN 0x00d5 PSN 0xfd1206
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:249
---------------------------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]    t_avg[usec]    t_stdev[usec]   99% percentile[usec]   99.9% percentile[usec] 
 2       1000          3.94           8.28         3.98     	       3.98        	0.00   		4.05    		8.28   
 4       1000          3.93           4.46         3.98     	       3.98        	0.00   		4.04    		4.46   
 8       1000          3.95           5.68         3.99     	       3.99        	0.00   		4.05    		5.68   
 16      1000          3.96           4.43         3.99     	       4.00        	0.00   		4.14    		4.43   
 32      1000          3.95           5.91         3.99     	       3.99        	0.00   		4.05    		5.91   
 64      1000          3.94           4.39         3.98     	       3.98        	0.00   		4.04    		4.39   
 128     1000          4.01           8.40         4.05     	       4.05        	0.00   		4.10    		8.40   
 256     1000          4.05           4.46         4.10     	       4.10        	0.00   		4.15    		4.46   
 512     1000          4.13           7.68         4.17     	       4.17        	0.00   		4.24    		7.68   
 1024    1000          4.28           4.84         4.31     	       4.31        	0.00   		4.37    		4.84   
 2048    1000          4.48           5.22         4.56     	       4.56        	0.00   		4.64    		5.22   
 4096    1000          4.81           9.85         4.92     	       4.92        	0.00   		5.03    		9.85   
 8192    1000          5.45           6.18         5.51     	       5.53        	0.00   		5.68    		6.18   
 16384   1000          6.70           7.43         6.76     	       6.76        	0.00   		6.91    		7.43   
 32768   1000          9.23           11.28        9.32     	       9.32        	0.00   		9.43    		11.28  
 65536   1000          14.20          15.50        14.29    	       14.31       	0.00   		14.56   		15.50  
 131072  1000          24.86          30.12        24.97    	       24.98       	0.15   		25.09   		30.12  
 262144  1000          44.85          49.81        44.99    	       45.00       	0.10   		45.27   		49.81  
 524288  1000          86.28          87.70        86.52    	       86.53       	0.00   		86.84   		87.70  
 1048576 1000          167.64         168.66       167.94   	       167.93      	0.00   		168.11  		168.66 
 2097152 1000          332.08         335.61       332.45   	       332.45      	0.00   		332.77  		335.61 
 4194304 1000          660.82         662.67       661.37   	       661.37      	0.00   		661.77  		662.67 
 8388608 1000          1317.00        1319.29      1317.90  	       1317.90     	0.03   		1318.55 		1319.29
---------------------------------------------------------------------------------------

寒武纪2&

寒武纪2

寒武纪1

1 2	`ib_read_lat -R -d mlx5_0 -i 1 -a -F ib_read_lat -R -d mlx5_0 -i 1 192.168.2.247 -a -F`

结果

$ ib_read_lat -R  -d mlx5_0 -i 1 192.168.2.247  -a -F 
---------------------------------------------------------------------------------------
                    RDMA_Read Latency Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 TX depth        : 1
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 3
 Outstand reads  : 16
 rdma_cm QPs	 : ON
 Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x014b PSN 0x51ff38
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:254
 remote address: LID 0000 QPN 0x0626 PSN 0xed2ff9
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:247
---------------------------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]    t_avg[usec]    t_stdev[usec]   99% percentile[usec]   99.9% percentile[usec] 
 2       1000          4.03           8.16         4.08     	       4.10        	0.06   		4.46    		8.16   
 4       1000          4.03           7.44         4.08     	       4.10        	0.06   		4.45    		7.44   
 8       1000          4.03           6.55         4.09     	       4.11        	0.00   		4.47    		6.55   
 16      1000          4.04           5.40         4.09     	       4.11        	0.00   		4.47    		5.40   
 32      1000          4.04           5.90         4.10     	       4.12        	0.03   		4.49    		5.90   
 64      1000          4.06           6.74         4.11     	       4.13        	0.03   		4.48    		6.74   
 128     1000          4.11           5.93         4.16     	       4.18        	0.03   		4.53    		5.93   
 256     1000          4.16           6.78         4.22     	       4.25        	0.06   		4.66    		6.78   
 512     1000          4.26           5.89         4.31     	       4.34        	0.00   		4.56    		5.89   
 1024    1000          4.35           6.06         4.43     	       4.48        	0.00   		4.89    		6.06   
 2048    1000          4.48           8.72         4.58     	       4.64        	0.07   		5.01    		8.72   
 4096    1000          4.67           6.78         4.76     	       4.81        	0.00   		5.19    		6.78   
 8192    1000          5.05           7.16         5.18     	       5.21        	0.00   		5.51    		7.16   
 16384   1000          5.79           7.45         5.90     	       5.95        	0.00   		6.30    		7.45   
 32768   1000          7.29           11.43        7.48     	       7.53        	0.10   		7.90    		11.43  
 65536   1000          10.28          13.87        10.41    	       10.44       	0.09   		10.74   		13.87  
 131072  1000          16.26          19.45        16.40    	       16.44       	0.10   		16.89   		19.45  
 262144  1000          28.24          31.18        28.37    	       28.42       	0.09   		28.86   		31.18  
 524288  1000          52.20          55.25        52.35    	       52.42       	0.10   		53.45   		55.25  
 1048576 1000          100.10         103.76       100.25   	       100.33      	0.18   		101.53  		103.76 
 2097152 1000          196.35         199.51       196.80   	       196.89      	0.23   		198.83  		199.51 
 4194304 1000          388.76         829.89       540.15   	       559.89      	130.19 		828.78  		829.89 
 8388608 1000          773.29         10731.33      821.09   	       1590.91     	1014.50		6627.52 		10731.33
---------------------------------------------------------------------------------------

NV&

232

233

1 2	`ib_read_lat -R -d mlx5_0 -a -F ib_read_lat -R -d mlx5_0 192.168.2.243 -a -F`

结果

(base) sunzhongyuan@DisAI-4090-3:~$ ib_read_lat -R  -d mlx5_0 192.168.2.243  -a -F  
---------------------------------------------------------------------------------------
                    RDMA_Read Latency Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 TX depth        : 1
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 5
 Outstand reads  : 16
 rdma_cm QPs	 : ON
 Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0129 PSN 0x1cd907
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:245
 remote address: LID 0000 QPN 0x01ea PSN 0xe24d16
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:243
---------------------------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]    t_avg[usec]    t_stdev[usec]   99% percentile[usec]   99.9% percentile[usec] 
 2       1000          4.33           13.32        4.42     	       4.44        	0.24   		4.74    		13.32  
 4       1000          4.32           14.01        4.40     	       4.41        	0.00   		4.72    		14.01  
 8       1000          4.32           14.54        4.41     	       4.42        	0.03   		4.75    		14.54  
 16      1000          4.34           14.36        4.43     	       4.44        	0.00   		4.73    		14.36  
 32      1000          4.36           4.75         4.45     	       4.46        	0.00   		4.70    		4.75   
 64      1000          4.40           11.87        4.49     	       4.49        	0.03   		4.77    		11.87  
 128     1000          4.54           13.76        4.61     	       4.62        	0.07   		4.97    		13.76  
 256     1000          4.62           5.12         4.71     	       4.70        	0.00   		4.96    		5.12   
 512     1000          4.76           10.82        4.86     	       4.87        	0.06   		5.15    		10.82  
 1024    1000          5.12           11.78        5.21     	       5.21        	0.00   		5.35    		11.78  
 2048    1000          5.53           12.48        5.71     	       5.72        	0.00   		6.00    		12.48  
 4096    1000          6.39           14.26        6.50     	       6.50        	0.00   		6.68    		14.26  
 8192    1000          7.81           22.13        7.90     	       7.92        	0.34   		8.19    		22.13  
 16384   1000          11.14          20.23        11.24    	       11.26       	0.20   		11.59   		20.23  
 32768   1000          18.20          28.95        18.28    	       18.30       	0.17   		18.61   		28.95  
 65536   1000          27.58          32.04        27.72    	       27.74       	0.13   		28.00   		32.04  
 131072  1000          47.61          52.24        47.74    	       47.76       	0.22   		48.09   		52.24  
 262144  1000          91.00          94.61        93.19    	       93.32       	0.15   		93.98   		94.61  
 524288  1000          173.31         182.56       174.06   	       174.20      	0.42   		176.37  		182.56 
 1048576 1000          337.79         344.39       338.66   	       338.72      	0.20   		339.83  		344.39 
 2097152 1000          670.59         681.06       672.23   	       672.34      	0.30   		674.34  		681.06 
 4194304 1000          1334.09        1348.28      1337.06  	       1337.18     	0.78   		1342.50 		1348.28
 8388608 1000          2674.01        2690.96      2681.01  	       2681.04     	0.90   		2685.45 		2690.96
---------------------------------------------------------------------------------------

NV

1 2	`ib_read_lat -d mlx5_0 -F -s 8388608 ib_read_lat -d mlx5_0 -F -s 8388608 10.56.217.72`

[root@mofed-test-cx5-bond-pod2 /]# ib_read_lat -d mlx5_0 -F -s 8388608 10.56.217.72
---------------------------------------------------------------------------------------
                    RDMA_Read Latency Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 TX depth        : 1
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 6
 Outstand reads  : 16
 rdma_cm QPs	 : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0136 PSN 0x70ab96 OUT 0x10 RKey 0x182bea VAddr 0x0070e38bef7000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:71
 remote address: LID 0000 QPN 0x01f3 PSN 0x10559d OUT 0x10 RKey 0x1832f1 VAddr 0x007d0e8988a000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:72
---------------------------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]    t_avg[usec]    t_stdev[usec]   99% percentile[usec]   99.9% percentile[usec] 
 8388608 1000          2709.26        2731.17      2722.01  	       2722.04     	0.85   		2724.25 		2731.17
---------------------------------------------------------------------------------------

海光

1 2	`ib_read_lat -d mlx5_0 -F --report_gbits -s 8388608 ib_read_lat -d mlx5_0 -F --report_gbits -s 8388608 10.56.217.76`

[root@mofed-test-cx5-bond-pod1 /]# ib_read_lat -d mlx5_0 -F --report_gbits -s 8388608 10.56.217.76
---------------------------------------------------------------------------------------
                    RDMA_Read Latency Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 TX depth        : 1
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 6
 Outstand reads  : 16
 rdma_cm QPs	 : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x00f7 PSN 0xe65dce OUT 0x10 RKey 0x1826e5 VAddr 0x007f3a88e1d000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:75
 remote address: LID 0000 QPN 0x0150 PSN 0xce186b OUT 0x10 RKey 0x1820df VAddr 0x007f0bee248000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:76
---------------------------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]    t_avg[usec]    t_stdev[usec]   99% percentile[usec]   99.9% percentile[usec] 
 8388608 1000          1317.91        1321.34      1318.71  	       1318.77     	0.40   		1319.99 		1321.34
---------------------------------------------------------------------------------------

寒武纪

#mofed-test-cx5-bond-pod2执行
ib_read_lat -R  -d mlx5_0 -i 1 -a -F 
#mofed-test-cx5-bond-pod1执行，ip为mofed-test-cx5-bond-pod2中net1的ip
ib_read_lat -R  -d mlx5_0 -i 1 10.56.217.78  -a -F

[root@mofed-test-cx5-bond-pod2 /]# ib_read_lat -R  -d mlx5_0 -i 1 10.56.217.78  -a -F 
---------------------------------------------------------------------------------------
                    RDMA_Read Latency Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 TX depth        : 1
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 4
 Outstand reads  : 16
 rdma_cm QPs	 : ON
 Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x05c4 PSN 0xaf7048
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:74
 remote address: LID 0000 QPN 0x0052 PSN 0xaf7048
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:56:217:78
---------------------------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]    t_avg[usec]    t_stdev[usec]   99% percentile[usec]   99.9% percentile[usec] 
 2       1000          4.04           7.35         4.08     	       4.10        	0.07   		4.44    		7.35   
 4       1000          4.04           6.81         4.09     	       4.10        	0.06   		4.43    		6.81   
 8       1000          4.04           6.11         4.09     	       4.11        	0.07   		4.47    		6.11   
 16      1000          4.05           6.59         4.10     	       4.12        	0.10   		4.48    		6.59   
 32      1000          4.05           6.30         4.10     	       4.12        	0.06   		4.42    		6.30   
 64      1000          4.07           6.51         4.11     	       4.13        	0.09   		4.48    		6.51   
 128     1000          4.12           6.70         4.18     	       4.20        	0.09   		4.58    		6.70   
 256     1000          4.18           6.86         4.23     	       4.26        	0.14   		4.65    		6.86   
 512     1000          4.26           5.88         4.31     	       4.35        	0.11   		4.75    		5.88   
 1024    1000          4.36           6.71         4.42     	       4.47        	0.13   		4.87    		6.71   
 2048    1000          4.49           6.35         4.55     	       4.58        	0.09   		4.92    		6.35   
 4096    1000          4.68           7.07         4.75     	       4.81        	0.13   		5.15    		7.07   
 8192    1000          5.03           7.71         5.13     	       5.21        	0.15   		5.55    		7.71   
 16384   1000          5.78           7.64         5.87     	       5.93        	0.13   		6.34    		7.64   
 32768   1000          7.28           8.62         7.42     	       7.49        	0.16   		7.87    		8.62   
 65536   1000          10.28          12.13        10.38    	       10.43       	0.13   		10.79   		12.13  
 131072  1000          16.27          19.09        16.37    	       16.42       	0.17   		16.84   		19.09  
 262144  1000          28.26          31.65        28.36    	       28.43       	0.21   		28.85   		31.65  
 524288  1000          52.21          54.59        52.34    	       52.41       	0.22   		53.54   		54.59  
 1048576 1000          100.11         103.19       100.23   	       100.32      	0.28   		101.67  		103.19 
 2097152 1000          196.38         200.30       196.83   	       196.91      	0.37   		198.39  		200.30 
 4194304 1000          388.75         829.42       389.47   	       512.43      	155.48 		828.86  		829.42 
 8388608 1000          773.27         14315.84      795.45   	       929.08      	338.46 		2429.12 		14315.84
---------------------------------------------------------------------------------------

参考

linux-rdma/perftest: Infiniband Verbs Performance Tests (github.com)

Infiniband 网络性能测试 - EdenLong - 博客园 (cnblogs.com)

https://blog.csdn.net/bandaoyu/article/details/115798045

注意事项

虚拟化rdma

测试内容

网卡信息

英伟达233

英伟达232

寒武纪2

寒武纪1

海光1

海光2

rdma_share测试pod

ib_send_bw带宽测试

默认

海光2&

寒武纪2&

NV&

rdma_share

海光

寒武纪

NV

ib_send_lat时延测试

默认

海光2&

寒武纪2&

NV&

rdma_share

海光

寒武纪

NV

ib_write_bw带宽测试

默认

海光2&

寒武纪2&

NV&

rdma_share

NV

海光

寒武纪

ib_write_lat时延测试

默认

海光2&

寒武纪2&

NV&

rdma_share

NV

海光

寒武纪

ib_read_bw带宽测试

默认

海光2&

寒武纪2&

NV&

rdma_share

NV

海光

寒武纪

ib_read_lat时延测试

默认

海光2&

寒武纪2&

NV&

rdma_share

NV

海光

寒武纪

参考

a-c-dream