Difference between revisions of "XFRM pCPU RSS"

From Libreswan
Jump to: navigation, search
 
Line 3: Line 3:
 
Receive Side Scaling (RSS)[https://www.kernel.org/doc/Documentation/networking/scaling.txt RSS] would steer flow to different ques. The receiver NIC should be able steer different flows, based on SPI, into separate queues to prevent the receiver from getting overwhelmed. We used Mellanex CX4 to test. Some cards initially tested did not seems to support RSS for ESP flows, instead only TCP and UDP. While figuring out RSS for these cards we tried a bit different approch. ESP in UDP encapsulation, along with ESP in UDP GRO patches we could see the flows getting distributed on the receiver. And later on in Nov 2019 kernel version 5.5 ML5 drivers seems to support ESP. [https://community.mellanox.com/s/article/Bluefield-IP-Forwarding-and-IPSEC-SPI-RSS Mellonox RSS].  
 
Receive Side Scaling (RSS)[https://www.kernel.org/doc/Documentation/networking/scaling.txt RSS] would steer flow to different ques. The receiver NIC should be able steer different flows, based on SPI, into separate queues to prevent the receiver from getting overwhelmed. We used Mellanex CX4 to test. Some cards initially tested did not seems to support RSS for ESP flows, instead only TCP and UDP. While figuring out RSS for these cards we tried a bit different approch. ESP in UDP encapsulation, along with ESP in UDP GRO patches we could see the flows getting distributed on the receiver. And later on in Nov 2019 kernel version 5.5 ML5 drivers seems to support ESP. [https://community.mellanox.com/s/article/Bluefield-IP-Forwarding-and-IPSEC-SPI-RSS Mellonox RSS].  
  
=== RSS Commands ===  
+
=== config-ntuple Commands ===  
 
Enable GRO. ideally you should be able to run the following,  
 
Enable GRO. ideally you should be able to run the following,  
 
<pre> ethtool -N <nic> rx-flow-hash esp4 </pre>
 
<pre> ethtool -N <nic> rx-flow-hash esp4 </pre>
Line 10: Line 10:
 
<pre> ethtool -N eno2 rx-flow-hash udp4 sdfn </pre>
 
<pre> ethtool -N eno2 rx-flow-hash udp4 sdfn </pre>
  
==== Mellanox support (YES) ====
+
==== Mellanox support (maybe) ====
  
 
could be configured steer the flow to a specific Q
 
could be configured steer the flow to a specific Q
Line 61: Line 61:
 
The vSphere 6.7 release includes vmxnet3 version 4, which supports some new features.  
 
The vSphere 6.7 release includes vmxnet3 version 4, which supports some new features.  
 
"RSS for ESP – RSS for encapsulating security payloads (ESP) is now available in the vmxnet3 v4 driver. Performance testing of this feature showed a 146% improvement in receive packets per second during a test that used IPSEC and four receive queues."
 
"RSS for ESP – RSS for encapsulating security payloads (ESP) is now available in the vmxnet3 v4 driver. Performance testing of this feature showed a 146% improvement in receive packets per second during a test that used IPSEC and four receive queues."
 +
 +
==== Marvell octeontx2-af : yes =====
 +
https://lore.kernel.org/r/1611378552-13288-1-git-send-email-sundeep.lkml@gmail.com
  
 
== Future research/ideas ==
 
== Future research/ideas ==

Latest revision as of 09:10, 25 February 2021

Receiver Side Scaling (RSS) support

Receive Side Scaling (RSS)RSS would steer flow to different ques. The receiver NIC should be able steer different flows, based on SPI, into separate queues to prevent the receiver from getting overwhelmed. We used Mellanex CX4 to test. Some cards initially tested did not seems to support RSS for ESP flows, instead only TCP and UDP. While figuring out RSS for these cards we tried a bit different approch. ESP in UDP encapsulation, along with ESP in UDP GRO patches we could see the flows getting distributed on the receiver. And later on in Nov 2019 kernel version 5.5 ML5 drivers seems to support ESP. Mellonox RSS.

config-ntuple Commands

Enable GRO. ideally you should be able to run the following,

 ethtool -N <nic> rx-flow-hash esp4 

Another argument is if the NIC agnostic the 16 bits of SPI, of ESP packet, is aligned with UDP port number and should provide enough entropy.

 ethtool -N eno2 rx-flow-hash udp4 sdfn 

Mellanox support (maybe)

could be configured steer the flow to a specific Q

ethtool --config-ntuple enp3s0f0 flow-type esp4 src-ip 192.168.1.1 dst-ip 192.168.1.2 spi 0xffffffff action 4

en_fs_ethtool.c

case ESP_V4_FLOW:
   return MLX5E_TT_IPV4_IPSEC_ESP;

Intel X710 (notyet)

i40e_ethtool.c ESP_V4_FLOW

i40e_ethtool.c
 case ESP_V4_FLOW:
 case ESP_V6_FLOW:
  /* Default is src/dest for IP, no matter the L4 hashing */
  cmd->data |= RXH_IP_SRC | RXH_IP_DST;
  break

AWS ENA (not yet)

ena_ethtool.c

case ESP_V4_FLOW:
case ESP_V6_FLOW:
 return -EOPNOTSUPP;

ENA driver mention support CPU indirection may be we can use it as udp.

The default hashing is currently Toeplitz.

Starting from ena driver v2.2.1 the driver supports changing the hash key and hash function as well as the indirection table itself. The support is only for instance types that end with "n", for example C5n instances.

Please note that changing the indirection table is supported on all instance types.

VMWare RSS ESP support

vmxnet

vmxnet3 version 4 commit

The vSphere 6.7 release includes vmxnet3 version 4, which supports some new features. "RSS for ESP – RSS for encapsulating security payloads (ESP) is now available in the vmxnet3 v4 driver. Performance testing of this feature showed a 146% improvement in receive packets per second during a test that used IPSEC and four receive queues."

Marvell octeontx2-af : yes =

https://lore.kernel.org/r/1611378552-13288-1-git-send-email-sundeep.lkml@gmail.com

Future research/ideas