Test Suite: Difference between revisions

From Libreswan
Jump to navigation Jump to search
(fix typo from stripping out KVM)
(point to testing.libreswan.org)
 
(47 intermediate revisions by 2 users not shown)
Line 1: Line 1:
== Nightly Test Tesults ==


Libreswan's testsuite is run nightly.  The results are published [https://testing.libreswan.org/ here], with the most recent result [https://testing.libreswan.org/current here].  The tests are categorized as:


{{ ambox | nocat=true | type=important | text = libvirt 0.9.11 and qemu 1.0 or better are required. RHEL does not support a writable 9p filesystem, so the recommended host/guest OS is Fedora 22 }}
* good: these tests are expected to pass (unfortunately, some still have timing problems and occasionally fail)
* wip: these tests require further work, for instance the result may not be deterministic, or the bug they demonstrate hasn't yet been fixed
* skiptest: these tests require manual intervention to run


[[File:testnet.png]]
To run tests locally, read on.
 
== Test Frameworks ==


This page is an overview of libreswan's testsuite.
== Running tests ==


It can be run using several different frameworks:
The libreswan tests, in testing/pluto, can be run using several different mechanisms:


* [[Test Suite - KVM]] (recommended)
{| class="wikitable"
* [[Test Suite - Docker]]
|+ Test Frameworks
! Framework
! Speed
! Host OS
! Guest OS
! initsystem testing (systemd, rc.d, ...)
! Post-mortem
! Interop testing
! Notes
|- style="vertical-align:top;"
| [[Test Suite - KVM | KVM]]
| slower
| Fedora, Debian <br>(BSD anyone?)
| Alpine, Fedora, FreeBSD, NetBSD, OpenBSD, Debian
| yes
| shutdown, core, leaks, refcnt, selinux
| strongswan (Linux, FreeBSD), iked (OpenBSD), racoon (NetBSD), racoon2 (NetBSD)
| gold standard <br> ideal for BSD builds <br> idea for testing custom kernels <br> used by the [https://testing.libreswan.org Testing] machine <br> requires 9p (virtio anyone?)
|- style="vertical-align:top;"
| [[Test Suite - Namespace | Namespaces]]
| fast
| linux
| uses host's libreswan, kernel, and utilities
| no
| core, leaks
| strongswan (linux)?
| ideal for quick tests <br> requires libreswan to be built/installed on the host <br> requires all dependencies to be installed on the host <br> test results sensitive  differing kernel and utilities
|- style="vertical-align:top;"
| [[Test Suite - Docker | Docker]]
|
| linux
| uses host's kernel <br> uses distro's utilities
| ?
| ?
| ?
| ideal for cross-linux builds (CentOS 6, 7, 8, Fedora 28 - rawhide, Debian, Ubuntu) <br> sensitive to differing kernel and utilities
|}


== Run an individual test (or tests) ==
== How tests work ==


All the test cases involving VMs are located in the libreswan directory under testing/pluto/ . The most basic test case is called basic-pluto-01. Each test case consists of a few files:
All the test cases involving VMs are located in the libreswan directory under <tt>testing/pluto/</tt>. The most basic test case is called basic-pluto-01. Each test case consists of a few files:


* description.txt to explain what this test case actually tests
* description.txt to explain what this test case actually tests
Line 25: Line 63:
* Known good (sanitized) output for each VM (eg west.console.txt, east.console.txt)
* Known good (sanitized) output for each VM (eg west.console.txt, east.console.txt)
* testparams.sh if there are any non-default test parameters
* testparams.sh if there are any non-default test parameters
You can run this test case by issuing the following command on the host:
Either:
<pre>
make kvm-test KVM_TESTS+=testing/pluto/basic-pluto-01/
</pre>
or:
<pre>
./testing/utils/kvmtest.py testing/pluto/basic-pluto-01
</pre>
multiple tests can be selected with:
<pre>
make kvm-test KVM_TESTS+=testing/pluto/basic-pluto-*
</pre>
or:
<pre>
./testing/utils/kvmresults.py testing/pluto/basic-pluto-*
</pre>


Once the test run has completed, you will see an OUTPUT/ directory in the test case directory:
Once the test run has completed, you will see an OUTPUT/ directory in the test case directory:
Line 68: Line 80:
* Any core dumps generated if a pluto daemon crashed
* Any core dumps generated if a pluto daemon crashed


== Debugging inside the VM ==
; testing/baseconfigs/
:  configuration files installed on guest machines
; testing/guestbin/
:  shell scripts used by tests, and run on the guest
; testing/linux-system-roles.vpn/
:  ???
; testing/packaging/
:  ???
; testing/pluto/TESTLIST
:  list of tests, and their expected outcome
; testing/pluto/*/
:  individual test directories
; testing/programs/
:  executables used by tests, and run on the guest
; testing/sanitizers/
:  filters for cleaning up the test output
; testing/utils/
:  test drivers and other host tools
; testing/x509/
:  certificates, scripts are run on a guest


=== Debugging pluto on east ===
== Network Diagrams ==


Terminal 1 - east: log into east, start pluto, and attach gdb
=== Current Network Diagram ===


<pre>
* interface-0 (eth0, vio0, vioif0) is connected to SWANDEFAULT which has a NAT gateway to the internet
make kvmsh-east
** the exceptions are the Fedora test domains: EAST, WEST, ROAD, NORTH; should they?
east# cd /testing/pluto/basic-pluto-01
** the BSD domains always up inteface-0 so that /pool, /source, and /testing can be NFS mounted
east# sh -x ./eastinit.sh
** NIC needs to run DHCP on eth0 manually; how?
east# gdb /usr/local/libexec/ipsec/pluto $(pidof pluto)
** transmogrify does not try to modify interface-0(SWANDEFAULT) (it breaks established network sessions such as NFS)
(gdb) c
* the interface names do not have consistent order (see comment above about Fedora's interface-0 not pointing at SWANDEFAULT)
</pre>
** Fedora has ethN
** OpenBSD has vioN (different order)
** NetBSD has vioifN (different order)


Terminal 2 - west: log into west, start pluto and the test
  LEFT                                                              RIGHT


<pre>
  192.0.3.0/24 -------------------------------------+-- 2001:db8:0:3::/64
make kvmsh-west
                                                    |
west# sh -x ./westinit.sh ; sh -x westrun.sh
                                              2001:db8:0:3::254
</pre>
                                              192.0.3.254(eth0/2)
If pluto wasn't running, gdb would complain: ''<code>--p requires an argument</code>''
                  ROAD                            NORTH
              192.1.3.209(eth0/1)            192.1.3.33(eth1)
              2001:db8:1:3::209              2001:db8:1:3::33
                    |                              |
  192.1.3.0/254 ----+----------------+--------------+-- 2001:db8:1:3::/64
                                    |
                              2001:db8:1:3::254
                                192.1.3.254(eth2)
                                    NIC---swandefault(0)
                                192.1.2.254(eth1)
                              2001:db8:1:2::254
                                    |
  192.1.2.0/24 ---+------------------+-------------+--- 2001:db8:1:2::/64
                  |                                |
            2001:db8:1:2::45                2001:db8:1:2::23
            192.1.2.45(eth1)                192.1.2.23(eth1)
                WEST---[swandefault(0)]          EAST---[swandefault(0)]
            192.0.1.254(eth0/2)              192.0.2.254(eth0/2)
            2001:db8:0:1::254                2001:db8:0:2::254
                  |                                |
                  |              192.0.2.0/255 ---+--- 2001:db8:0:2::/64
                  |
  192.0.1.0/255 --+------------------------------------ 2001:db8:0:1::/64
 
  192.1.4.0/255 --------------------------------------- 2001:db8:1:4::/64


When pluto crashes, gdb will show that and await commands.  For example, the bt command will show a backtrace.
=== Proposed Network Diagram ===


=== Debugging pluto on west ===
Changes:
* add NOC, unlike NIC can run libreswan
* add SUNRISE (behind east) and SUNSET (behind west)
Notes:
* if0 (eth0, vio0, vioif0) is connected to SWANDEFAULT which has a NAT gateway to the internet
** the exceptions are the Fedora test domains: EAST, WEST, ROAD, NORTH; should they?
** the BSD domains use if0 to NFS mount /bench, /pool, /source, and /testing
** NIC needs to run DHCP on eth0 manually; how?
** transmogrify does not try to modify if0(SWANDEFAULT) (it breaks established network sessions such as NFS)
* the interface names still do not have consistent order
** ifN denotes a consistent numbering
** ethN denotes Fedora only assignment
** bsdN denotes a BSD only assignment (FreeBSD has ???N, NetBSD has vioifN, OpenBSD has vioN)
* re-number networks so that they are RFC friendly, these RFCs came up in discussion; alternatively do nothing
** https://www.rfc-editor.org/rfc/rfc1918
** https://www.rfc-editor.org/rfc/rfc5737
** https://www.rfc-editor.org/rfc/rfc3849
** https://www.rfc-editor.org/rfc/rfc5180
** https://www.rfc-editor.org/rfc/rfc2544
** https://www.rfc-editor.org/rfc/rfc6890


See above, but also use virt as a terminal.
  192.0.3.0/24 --------------------------------------+---- 2001:db8:0:3::/64
                                                      |
                                              2001:db8:0:3::254
                                                192.0.3.254
                                                  <eth0/2>
                  ROAD                              NORTH
                <eth0/1>                          <eth1>
              192.1.3.209                        192.1.3.33
            2001:db8:1:3::209                2001:db8:1:3::33
                    |                                |
  192.1.3.0/254 ----+---+-------------------------+---+---- 2001:db8:1:3::/64
                        |                        |
                2001:db8:1:3::254        2001:db8:1:3::253
                192.1.3.254(eth2)          192.1.3.253
                    <eth2>                    <eth2>
                      NIC-<if0>-swandefault    NOC-<if0>-swandefault
                    <eth1>                    <eth1>
                  192.1.2.254              192.1.2.253
                2001:db8:1:2::254        2001:db8:1:2::253
                        |                        |
  192.1.2.0/24 -----+--+-------------------------+--+----- 2001:db8:1:2::/64
                    |                              |
            2001:db8:1:2::45                2001:db8:1:2::23
                192.1.2.45                      192.1.2.23
                  <eth1>                          <eth1>
                  WEST-[<bsd0>-swandefault]      EAST-[<bsd0>-swandefault]
                  <eth0/2>                        <eth0/2>
                192.0.1.254                    192.0.2.254
            2001:db8:0:1::254              2001:db8:0:2::254
                    |                              |
                    |        192.0.2.0/255 ---------+--+-- 2001:db8:0:2::/64
                    |                                  |
                    |                          2001:db8:0:2::252
                    |                            192.0.2.252
                    |                                <eth1>
                    |                                RISE-<eth0>-swandefault
                    |
  192.0.1.0/255 --+--+------------------------------------- 2001:db8:0:1::/64
                  |
          2001:db8:0:1::252
            192.0.1.252
                <if1>
                SET-<if0>-swandefault
 
  192.1.4.0/255 --------------------------------------- 2001:db8:1:4::/64


=== /root/.gdbinit ===
=== Hand Sketch of Current Network ===


If you want to get rid of the warning "warning: File "/testing/pluto/ikev2-dpd-01/.gdbinit" auto-loading has been declined by your `auto-load safe-path'"
[[File:networksketch.png]]


<pre>
=== Original Network Diagram ===
echo "set auto-load safe-path /" >> /root/.gdbinit
</pre>


=== swan-transmogrify ===
[[File:testnet.png]]
 
When the VMs were installed, an XML configuration file from testing/libvirt/vm/ was used to configure each VM with the right disks, mounts and nic cards. Each VM mounts the libreswan directory as /source and the libreswan/testing/ directory as /testing . This makes the /testing/guestbin/ directory available on the VMs. At boot, the VMs run /testing/guestbin/swan-transmogrify. This python script compares the nic of eth0 with the list of known MAC addresses from the XML files. By identifying the MAC, it knows which identity (west, east, etc) it should take on. Files are copied from /testing/baseconfigs/ into the VM's /etc directory and the network service is restarted.
 
=== swan-build, swans-install, swan-update ===
 
These commands are used to build, install or build+install (update) the libreswan userland and kernel code
 
=== swan-prep ===
 
This command is run as the first command of each test case to setup the host. It copies the required files from /testing/baseconfigs/ and the specific test case files onto the VM test machine. It does not start libreswan. That is done in the "init.sh" script.
 
The swan-prep command takes two options.
The --x509 option is required to copy in all the required certificates and update the NSS database.
The --46 /--6 option is used to give the host IPv4 and/or IPv6 connectivity. Hosts per default only get IPv4 connectivity as this reduces the noise captured with tcpdump
 
=== fipson and fipsoff ===
 
These are used to fake a kernel into FIPS mode, which is required for some of the tests.
 
 
== Various notes ==
 
* Currently, only one test can run at a time.
* You can peek at the guests using virt-manager or you can ssh into the test machines from the host.
* ssh may be slow to prompt for the password.  If so, start up the vm "nic"
* On VMs use only one CPU core. Multiple CPUs may cause pexpect to mangle output.
* 2014 Mar: DHR needed to do the following to make things work each time he rebooted the host
<pre>
$ sudo setenforce Permissive
$ ls -ld /var/lib/libvirt/qemu
drwxr-x---. 6 qemu qemu 4096 Mar 14 01:23 /var/lib/libvirt/qemu
$ sudo chmod g+w /var/lib/libvirt/qemu
$ ( cd testing/libvirt/net ; for i in * ; do sudo virsh net-start $i ; done ; )
</pre>
* to make the SELinux enforcement change persist across host reboots, edit /etc/selinux/config
* to remove "169.254.0.0/16 dev eth0  scope link  metric 1002" from "ipsec status output"
<pre> echo 'NOZEROCONF=1' >> /etc/sysconfig/network </pre>
== To improve ==
* install and remove RPM using swantest + make rpm support
* add summarizing script that generate html/json to git repo
* cordump. It has been a mystery :) systemd or some daemon appears to block coredump on the Fedora 20 systems.
* when running multiple tests from TESTLIST shutdown the hosts before copying OUTPUT dir. This way we get leak detect inf. However, for single test runs do not shut down.
 
== IPv6 tests ==
IPv6 test cases seems to work better when IPv6 is disabled on the KVM bridge interfaces the VMs use. The bridges are swanXX and their config files are /etc/libvirt/qemu/networks/192_0_1.xml . Remove the following line from it. Reboot/restart libvirt.
 
<pre>
libvirt/qemu/networks/192_0_1.xml
 
<ip family="ipv6" address="2001:db8:0:1::253" prefix="64"/>
 
</pre>
 
and ifconfig swan01 should have no IPv6 address, no fe:80 or any v6 address. Then the v6 testcases should work. 
 
<br> please give me feedback if this hack work for you. I shall try to add more info about this.
 
== Sanitizers ==
 
* summarize output from tcpdump
* count established IKE, ESP , AH states (there is count at the end of "ipsec status " that is not accurate. It counts instantiated connection as loaded.
 
* dpd ping sanitizer. DPD tests have unpredictable packet loss for ping.
 
== tcpdump permissions on the Host (optional, not needed) ==
 
The experimental kvmrunner --tcpdump option does not require this configuration change.
 
<pre>
XXX: Only swantest uses this, and having swantest use sudo would be better.
 
getent group tcpdump || sudo groupadd tcpdump
#add build to group tcpdump
sudo usermod --append -G tcpdump build
ls -lt /sbin/tcpdump
sudo chown root:tcpdump /sbin/tcpdump
sudo setcap "CAP_NET_RAW+eip" /sbin/tcpdump
 
# check tcpdump group users
getent group tcpdump
tcpdump:x:72:build
 
#when the installation is complete the following should work
tcpdump -i swan12
 
</pre>

Latest revision as of 18:08, 4 March 2024

Nightly Test Tesults

Libreswan's testsuite is run nightly. The results are published here, with the most recent result here. The tests are categorized as:

  • good: these tests are expected to pass (unfortunately, some still have timing problems and occasionally fail)
  • wip: these tests require further work, for instance the result may not be deterministic, or the bug they demonstrate hasn't yet been fixed
  • skiptest: these tests require manual intervention to run

To run tests locally, read on.

Running tests

The libreswan tests, in testing/pluto, can be run using several different mechanisms:

Test Frameworks
Framework Speed Host OS Guest OS initsystem testing (systemd, rc.d, ...) Post-mortem Interop testing Notes
KVM slower Fedora, Debian
(BSD anyone?) 		Alpine, Fedora, FreeBSD, NetBSD, OpenBSD, Debian yes shutdown, core, leaks, refcnt, selinux strongswan (Linux, FreeBSD), iked (OpenBSD), racoon (NetBSD), racoon2 (NetBSD) gold standard
ideal for BSD builds
idea for testing custom kernels
used by the Testing machine
requires 9p (virtio anyone?)
Namespaces fast linux uses host's libreswan, kernel, and utilities no core, leaks strongswan (linux)? ideal for quick tests
requires libreswan to be built/installed on the host
requires all dependencies to be installed on the host
test results sensitive differing kernel and utilities
Docker linux uses host's kernel
uses distro's utilities 		? ? ? ideal for cross-linux builds (CentOS 6, 7, 8, Fedora 28 - rawhide, Debian, Ubuntu)
sensitive to differing kernel and utilities

How tests work

All the test cases involving VMs are located in the libreswan directory under testing/pluto/. The most basic test case is called basic-pluto-01. Each test case consists of a few files:

  • description.txt to explain what this test case actually tests
  • ipsec.conf files - for host west is called west.conf. This can also include configuration files for strongswan or racoon2 for interop testig
  • ipsec.secret files - if non-default configurations are used. also uses the host syntax, eg west.secrets, east.secrets.
  • An init.sh file for each VM that needs to start (eg westinit.sh, eastinit.sh, etc)
  • One run.sh file for the host that is the initiator (eg westrun.sh)
  • Known good (sanitized) output for each VM (eg west.console.txt, east.console.txt)
  • testparams.sh if there are any non-default test parameters

Once the test run has completed, you will see an OUTPUT/ directory in the test case directory: 

$ ls OUTPUT/
east.console.diff  east.console.verbose.txt  RESULT       west.console.txt          west.pluto.log
east.console.txt   east.pluto.log            swan12.pcap  west.console.diff  west.console.verbose.txt
  • RESULT is a text file (whose format is sure to change in the next few months) stating whether the test succeeded or failed.
  • The diff files show the differences between this testrun and the last known good output.
  • Each VM's serial (sanitized) console log (eg west.console.txt)
  • Each VM's unsanitized verbose console output (eg west.console.verbose.txt)
  • A network capture from the bridge device (eg swan12.pcap)
  • Each VM's pluto log, created with plutodebug=all (eg west.pluto.log)
  • Any core dumps generated if a pluto daemon crashed
testing/baseconfigs/
configuration files installed on guest machines
testing/guestbin/
shell scripts used by tests, and run on the guest
testing/linux-system-roles.vpn/
???
testing/packaging/
???
testing/pluto/TESTLIST
list of tests, and their expected outcome
testing/pluto/*/
individual test directories
testing/programs/
executables used by tests, and run on the guest
testing/sanitizers/
filters for cleaning up the test output
testing/utils/
test drivers and other host tools
testing/x509/
certificates, scripts are run on a guest

Network Diagrams

Current Network Diagram

  • interface-0 (eth0, vio0, vioif0) is connected to SWANDEFAULT which has a NAT gateway to the internet
    • the exceptions are the Fedora test domains: EAST, WEST, ROAD, NORTH; should they?
    • the BSD domains always up inteface-0 so that /pool, /source, and /testing can be NFS mounted
    • NIC needs to run DHCP on eth0 manually; how?
    • transmogrify does not try to modify interface-0(SWANDEFAULT) (it breaks established network sessions such as NFS)
  • the interface names do not have consistent order (see comment above about Fedora's interface-0 not pointing at SWANDEFAULT)
    • Fedora has ethN
    • OpenBSD has vioN (different order)
    • NetBSD has vioifN (different order)
 LEFT                                                              RIGHT

 192.0.3.0/24 -------------------------------------+-- 2001:db8:0:3::/64
                                                   |
                                             2001:db8:0:3::254
                                              192.0.3.254(eth0/2)
                 ROAD                            NORTH
              192.1.3.209(eth0/1)             192.1.3.33(eth1)
             2001:db8:1:3::209               2001:db8:1:3::33
                   |                               |
 192.1.3.0/254 ----+----------------+--------------+-- 2001:db8:1:3::/64
                                    |
                              2001:db8:1:3::254
                               192.1.3.254(eth2)
                                   NIC---swandefault(0)
                               192.1.2.254(eth1)
                              2001:db8:1:2::254
                                    |
 192.1.2.0/24 ---+------------------+-------------+--- 2001:db8:1:2::/64
                 |                                |
           2001:db8:1:2::45                 2001:db8:1:2::23
            192.1.2.45(eth1)                 192.1.2.23(eth1)
               WEST---[swandefault(0)]          EAST---[swandefault(0)]
            192.0.1.254(eth0/2)              192.0.2.254(eth0/2)
           2001:db8:0:1::254                2001:db8:0:2::254
                 |                                |
                 |               192.0.2.0/255 ---+--- 2001:db8:0:2::/64
                 |
 192.0.1.0/255 --+------------------------------------ 2001:db8:0:1::/64
 
 192.1.4.0/255 --------------------------------------- 2001:db8:1:4::/64

Proposed Network Diagram

Changes:

  • add NOC, unlike NIC can run libreswan
  • add SUNRISE (behind east) and SUNSET (behind west)

Notes: 

  192.0.3.0/24 --------------------------------------+---- 2001:db8:0:3::/64
                                                     |
                                              2001:db8:0:3::254
                                               192.0.3.254
                                                  <eth0/2>
                 ROAD                              NORTH
                <eth0/1>                          <eth1>
             192.1.3.209                        192.1.3.33
            2001:db8:1:3::209                 2001:db8:1:3::33
                   |                                 |
 192.1.3.0/254 ----+---+-------------------------+---+---- 2001:db8:1:3::/64
                       |                         |
               2001:db8:1:3::254         2001:db8:1:3::253
                192.1.3.254(eth2)           192.1.3.253
                    <eth2>                    <eth2>
                      NIC-<if0>-swandefault     NOC-<if0>-swandefault
                    <eth1>                    <eth1>
                  192.1.2.254               192.1.2.253
               2001:db8:1:2::254         2001:db8:1:2::253
                       |                         |
  192.1.2.0/24 -----+--+-------------------------+--+----- 2001:db8:1:2::/64
                    |                               |
            2001:db8:1:2::45                2001:db8:1:2::23
               192.1.2.45                      192.1.2.23
                 <eth1>                          <eth1>
                  WEST-[<bsd0>-swandefault]       EAST-[<bsd0>-swandefault]
                 <eth0/2>                        <eth0/2>
               192.0.1.254                     192.0.2.254
            2001:db8:0:1::254               2001:db8:0:2::254
                    |                               |
                    |        192.0.2.0/255 ---------+--+-- 2001:db8:0:2::/64
                    |                                  |
                    |                          2001:db8:0:2::252
                    |                             192.0.2.252
                    |                                <eth1>
                    |                                 RISE-<eth0>-swandefault
                    |
 192.0.1.0/255 --+--+------------------------------------- 2001:db8:0:1::/64
                 |
          2001:db8:0:1::252
           192.0.1.252
               <if1>
                SET-<if0>-swandefault
 
 192.1.4.0/255 --------------------------------------- 2001:db8:1:4::/64

Hand Sketch of Current Network

Networksketch.png

Original Network Diagram

Testnet.png

Retrieved from "https://libreswan.org/wiki/index.php?title=Test_Suite&oldid=22341"