XFRM pCPU RSS: Difference between revisions
No edit summary |
No edit summary |
||
| Line 3: | Line 3: | ||
Receive Side Scaling (RSS)[https://www.kernel.org/doc/Documentation/networking/scaling.txt RSS] would steer flow to different ques. The receiver NIC should be able steer different flows, based on SPI, into separate queues to prevent the receiver from getting overwhelmed. We used Mellanex CX4 to test. Some cards initially tested did not seems to support RSS for ESP flows, instead only TCP and UDP. While figuring out RSS for these cards we tried a bit different approch. ESP in UDP encapsulation, along with ESP in UDP GRO patches we could see the flows getting distributed on the receiver. And later on in Nov 2019 kernel version 5.5 ML5 drivers seems to support ESP. [https://community.mellanox.com/s/article/Bluefield-IP-Forwarding-and-IPSEC-SPI-RSS Mellonox RSS]. | Receive Side Scaling (RSS)[https://www.kernel.org/doc/Documentation/networking/scaling.txt RSS] would steer flow to different ques. The receiver NIC should be able steer different flows, based on SPI, into separate queues to prevent the receiver from getting overwhelmed. We used Mellanex CX4 to test. Some cards initially tested did not seems to support RSS for ESP flows, instead only TCP and UDP. While figuring out RSS for these cards we tried a bit different approch. ESP in UDP encapsulation, along with ESP in UDP GRO patches we could see the flows getting distributed on the receiver. And later on in Nov 2019 kernel version 5.5 ML5 drivers seems to support ESP. [https://community.mellanox.com/s/article/Bluefield-IP-Forwarding-and-IPSEC-SPI-RSS Mellonox RSS]. | ||
=== | === config-ntuple Commands === | ||
Enable GRO. ideally you should be able to run the following, | Enable GRO. ideally you should be able to run the following, | ||
<pre> ethtool -N <nic> rx-flow-hash esp4 </pre> | <pre> ethtool -N <nic> rx-flow-hash esp4 </pre> | ||
| Line 10: | Line 10: | ||
<pre> ethtool -N eno2 rx-flow-hash udp4 sdfn </pre> | <pre> ethtool -N eno2 rx-flow-hash udp4 sdfn </pre> | ||
==== Mellanox support ( | ==== Mellanox support (maybe) ==== | ||
could be configured steer the flow to a specific Q | could be configured steer the flow to a specific Q | ||
| Line 61: | Line 61: | ||
The vSphere 6.7 release includes vmxnet3 version 4, which supports some new features. | The vSphere 6.7 release includes vmxnet3 version 4, which supports some new features. | ||
"RSS for ESP – RSS for encapsulating security payloads (ESP) is now available in the vmxnet3 v4 driver. Performance testing of this feature showed a 146% improvement in receive packets per second during a test that used IPSEC and four receive queues." | "RSS for ESP – RSS for encapsulating security payloads (ESP) is now available in the vmxnet3 v4 driver. Performance testing of this feature showed a 146% improvement in receive packets per second during a test that used IPSEC and four receive queues." | ||
==== Marvell octeontx2-af : yes ===== | |||
https://lore.kernel.org/r/1611378552-13288-1-git-send-email-sundeep.lkml@gmail.com | |||
== Future research/ideas == | == Future research/ideas == | ||
Revision as of 11:10, 25 February 2021
Receiver Side Scaling (RSS) support
Receive Side Scaling (RSS)RSS would steer flow to different ques. The receiver NIC should be able steer different flows, based on SPI, into separate queues to prevent the receiver from getting overwhelmed. We used Mellanex CX4 to test. Some cards initially tested did not seems to support RSS for ESP flows, instead only TCP and UDP. While figuring out RSS for these cards we tried a bit different approch. ESP in UDP encapsulation, along with ESP in UDP GRO patches we could see the flows getting distributed on the receiver. And later on in Nov 2019 kernel version 5.5 ML5 drivers seems to support ESP. Mellonox RSS.
config-ntuple Commands
Enable GRO. ideally you should be able to run the following,
ethtool -N <nic> rx-flow-hash esp4
Another argument is if the NIC agnostic the 16 bits of SPI, of ESP packet, is aligned with UDP port number and should provide enough entropy.
ethtool -N eno2 rx-flow-hash udp4 sdfn
Mellanox support (maybe)
could be configured steer the flow to a specific Q
ethtool --config-ntuple enp3s0f0 flow-type esp4 src-ip 192.168.1.1 dst-ip 192.168.1.2 spi 0xffffffff action 4
case ESP_V4_FLOW: return MLX5E_TT_IPV4_IPSEC_ESP;
Intel X710 (notyet)
i40e_ethtool.c case ESP_V4_FLOW: case ESP_V6_FLOW: /* Default is src/dest for IP, no matter the L4 hashing */ cmd->data |= RXH_IP_SRC | RXH_IP_DST; break
AWS ENA (not yet)
case ESP_V4_FLOW: case ESP_V6_FLOW: return -EOPNOTSUPP;
ENA driver mention support CPU indirection may be we can use it as udp.
The default hashing is currently Toeplitz. Starting from ena driver v2.2.1 the driver supports changing the hash key and hash function as well as the indirection table itself. The support is only for instance types that end with "n", for example C5n instances. Please note that changing the indirection table is supported on all instance types.
VMWare RSS ESP support
The vSphere 6.7 release includes vmxnet3 version 4, which supports some new features. "RSS for ESP – RSS for encapsulating security payloads (ESP) is now available in the vmxnet3 v4 driver. Performance testing of this feature showed a 146% improvement in receive packets per second during a test that used IPSEC and four receive queues."
Marvell octeontx2-af : yes =
https://lore.kernel.org/r/1611378552-13288-1-git-send-email-sundeep.lkml@gmail.com
Future research/ideas
- Test with SR IOV and virtualisation(KVM): need systems with NIC that support SR IOV and RSS for ESP or at least UDP.
- Software RSS https://www.linux-kvm.org/page/Multiqueue
- can IKE daemon use other flow distribution methods based on SPI??? DPDK???
- another way of flow control??? https://doc.dpdk.org/dts/test_plans/link_flowctrl_test_plan.html
- RSS/RPS/RFS