XFRM pCPU RSS

Receiver Side Scaling (RSS) support

Receive Side Scaling (RSS)RSS would steer flow to different ques. The receiver NIC should be able steer different flows, based on SPI, into separate queues to prevent the receiver from getting overwhelmed. We used Mellanex CX4 to test. Some cards initially tested did not seems to support RSS for ESP flows, instead only TCP and UDP. While figuring out RSS for these cards we tried a bit different approch. ESP in UDP encapsulation, along with ESP in UDP GRO patches we could see the flows getting distributed on the receiver. And later on in Nov 2019 kernel version 5.5 ML5 drivers seems to support ESP. Mellonox RSS.

RSS Commands

Enable GRO. ideally you should be able to run the following,

 ethtool -N <nic> rx-flow-hash esp4 

Another argument is if the NIC agnostic the 16 bits of SPI, of ESP packet, is aligned with UDP port number and should provide enough entropy.

 ethtool -N eno2 rx-flow-hash udp4 sdfn 

Mellanox support (YES)

could be configured steer the flow to a specific Q

ethtool --config-ntuple enp3s0f0 flow-type esp4 src-ip 192.168.1.1 dst-ip 192.168.1.2 spi 0xffffffff action 4

en_fs_ethtool.c

case ESP_V4_FLOW:
   return MLX5E_TT_IPV4_IPSEC_ESP;

Intel X710 (notyet)

i40e_ethtool.c ESP_V4_FLOW

i40e_ethtool.c
 case ESP_V4_FLOW:
 case ESP_V6_FLOW:
  /* Default is src/dest for IP, no matter the L4 hashing */
  cmd->data |= RXH_IP_SRC | RXH_IP_DST;
  break

AWS ENA (not yet)

ena_ethtool.c

case ESP_V4_FLOW:
case ESP_V6_FLOW:
 return -EOPNOTSUPP;

ENA driver mention support CPU indirection may be we can use it as udp.

The default hashing is currently Toeplitz.

Starting from ena driver v2.2.1 the driver supports changing the hash key and hash function as well as the indirection table itself. The support is only for instance types that end with "n", for example C5n instances.

Please note that changing the indirection table is supported on all instance types.

VMWare RSS ESP support

vmxnet

vmxnet3 version 4 commit

The vSphere 6.7 release includes vmxnet3 version 4, which supports some new features. "RSS for ESP – RSS for encapsulating security payloads (ESP) is now available in the vmxnet3 v4 driver. Performance testing of this feature showed a 146% improvement in receive packets per second during a test that used IPSEC and four receive queues."

Future research/ideas

• Test with SR IOV and virtualisation(KVM): need systems with NIC that support SR IOV and RSS for ESP or at least UDP.
• Software RSS https://www.linux-kvm.org/page/Multiqueue
• can IKE daemon use other flow distribution methods based on SPI??? DPDK???
• another way of flow control??? https://doc.dpdk.org/dts/test_plans/link_flowctrl_test_plan.html
• RSS/RPS/RFS