There are two servers connected via 10Gbe switch. Cisco sg350xg-24t
Intel Corporation Ethernet Controller 10G X550T Cards (rev 01)
I observe very low performance on UDP, high latency, big losses.
But on TCP everything is fine.
I can not find an explanation for this. Who can tell a thread why so? Below are the test results:
Test for UDP: iperf -c 10.10.10.3 -i 1 -b 10G -u -e

 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Latency avg/min/max/stdev PPS [ 4] local 10.10.10.3 port 5001 connected with 10.10.10.4 port 48233 [ 4] 0.00-1.00 sec 577 MBytes 4.84 Gbits/sec 0.004 ms 250113/662049 (38%) 7.954/ 7.917/ 8.732/ 0.023 ms 411924 pps [ 4] 1.00-2.00 sec 570 MBytes 4.78 Gbits/sec 0.001 ms 245694/651975 (38%) 7.955/ 7.914/ 8.038/ 0.016 ms 406292 pps [ 4] 2.00-3.00 sec 565 MBytes 4.74 Gbits/sec 0.007 ms 277045/679862 (41%) 7.954/ 7.912/ 8.046/ 0.015 ms 402813 pps [ 4] 3.00-4.00 sec 570 MBytes 4.78 Gbits/sec 0.003 ms 250685/657553 (38%) 7.950/ 7.909/ 8.033/ 0.016 ms 406861 pps [ 4] 4.00-5.00 sec 573 MBytes 4.80 Gbits/sec 0.003 ms 263988/672481 (39%) 7.946/ 7.907/ 8.030/ 0.016 ms 408500 pps [ 4] 5.00-6.00 sec 575 MBytes 4.82 Gbits/sec 0.002 ms 255955/665858 (38%) 7.945/ 7.905/ 8.029/ 0.015 ms 409897 pps [ 4] 6.00-7.00 sec 596 MBytes 5.00 Gbits/sec 0.004 ms 241399/666408 (36%) 7.940/ 7.902/ 8.039/ 0.016 ms 424992 pps [ 4] 7.00-8.00 sec 577 MBytes 4.84 Gbits/sec 0.004 ms 250347/662174 (38%) 7.940/ 7.900/ 8.027/ 0.016 ms 411834 pps [ 4] 8.00-9.00 sec 567 MBytes 4.76 Gbits/sec 0.001 ms 263960/668624 (39%) 7.939/ 7.898/ 8.018/ 0.016 ms 404684 pps [ 4] 0.00-10.00 sec 5.60 GBytes 4.81 Gbits/sec 0.002 ms 2571225/6662198 (39%) 7.946/ 7.896/ 8.732/ 0.018 ms 409124 pps


Test on TCP: iperf -c 10.10.10.3 -i 1 -b 10G -e

 [ 4] local 10.10.10.3 port 5001 connected with 10.10.10.4 port 44798 [ ID] Interval Transfer Bandwidth Reads Dist(bin=16.0K) [ 4] 0.00-1.00 sec 1.15 GBytes 9.90 Gbits/sec 15395 378:6617:437:145:67:6:8:7737 [ 4] 1.00-2.00 sec 1.15 GBytes 9.90 Gbits/sec 15318 597:6024:615:192:138:1:3:7748 [ 4] 2.00-3.00 sec 1.15 GBytes 9.90 Gbits/sec 15290 653:5824:707:206:151:5:7:7737 [ 4] 3.00-4.00 sec 1.15 GBytes 9.90 Gbits/sec 15349 503:6182:613:185:115:10:19:7722 [ 4] 4.00-5.00 sec 1.15 GBytes 9.90 Gbits/sec 15322 635:5910:711:204:108:3:2:7749 [ 4] 5.00-6.00 sec 1.15 GBytes 9.90 Gbits/sec 15287 695:5687:773:249:132:6:7:7738 [ 4] 6.00-7.00 sec 1.15 GBytes 9.90 Gbits/sec 15350 555:6168:611:172:92:2:1:7749 [ 4] 7.00-8.00 sec 1.15 GBytes 9.90 Gbits/sec 15338 620:5995:644:207:122:3:9:7738 [ 4] 8.00-9.00 sec 1.15 GBytes 9.90 Gbits/sec 15316 532:6068:627:220:118:1:0:7750 [ 4] 9.00-10.00 sec 1.15 GBytes 9.90 Gbits/sec 15458 179:7168:268:51:41:0:0:7751 [ 4] 0.00-10.01 sec 11.5 GBytes 9.89 Gbits/sec 153457 5347:61660:6007:1831:1084:37:56:77435
  • To begin with, I would find out whether the switch is the cause or not by excluding it from the channel (your tens can be connected directly?) And try a UDP test with a channel width setting of 80-85% of the nominal value. - Alexander Prokoshev

1 answer 1

Partially, I found the problem. A large number of packets are generated between two addresses (approximately 600K per second). They fall into one processing queue, and thus are processed by a single core. The kernel does not cope, losses appear. There are no network problems. I did not understand why tcp does not generate such a load. It doesn't matter now.