Test Suite: Difference between revisions

From Libreswan
Jump to navigation Jump to search
No edit summary
(point to testing.libreswan.org)
 
(230 intermediate revisions by 7 users not shown)
Line 1: Line 1:
== Nightly Test Tesults ==


Libreswan comes with an extensive test suite, written mostly in python, that uses KVM virtual machines and virtual networks. It has replaced the old UML test suite.
Libreswan's testsuite is run nightly.  The results are published [https://testing.libreswan.org/ here], with the most recent result [https://testing.libreswan.org/current here]. The tests are categorized as:
Apart from KVM, the test suite uses libvirtd and qemu. It is strongly recommended to run the test suite natively on the OS (not in a VM itself) on a machine that has a CPU wth virtualization instructions.
The PLAN9 filesystem (9p) is used to mount host directories in the guests - NFS is avoided to prevent network lockups when an IPsec test case would cripple the guest's networking.


{{ ambox | nocat=true | type=important | text = libvirt 0.9.11 and qemu 1.0 or better are required. RHEL does not support a writable 9p filesystem, so the recommended host/guest OS is Fedora 20 }}
* good: these tests are expected to pass (unfortunately, some still have timing problems and occasionally fail)
* wip: these tests require further work, for instance the result may not be deterministic, or the bug they demonstrate hasn't yet been fixed
* skiptest: these tests require manual intervention to run


[[File:testnet.png]]
To run tests locally, read on.
 
== Preparing the host machine ==
 
Nothing apart from the system services requires root access. However, it does require that the user you are using is allowed to run various commands as root via sudo. Additionally, libvirt assumes the VMs are running under the qemu uid, but because we want to share files using the 9p filesystem between host and guests, we want the VMs to run under our own uid. The easiest solution to accomplish all of these is to add your user (for example the username "build") to the kvm, qemu and wheel groups. These are the changed lines from /etc/groups:
 
<pre>
wheel:x:10:root,build
kvm:x:36:qemu,root,qemu,build
qemu:x:107:root,qemu,build
</pre>
 
And the file /etc/sudoers would have a line:
 
<pre>
%wheel ALL=(ALL) NOPASSWD: ALL
</pre>


You might need to relogin for all group changes to take effect.
== Running tests ==


Now we are ready to install the various components of libvirtd, qemu and kvm and then start the libvirtd service.
The libreswan tests, in testing/pluto, can be run using several different mechanisms:


<pre>
{| class="wikitable"
sudo yum install virt-manager qemu-system-x86 libvirt-daemon-driver-qemu qemu qemu-kvm \
|+ Test Frameworks
                libvirt-daemon-qemu qemu-img qemu-user libvirt-daemon-qemu libvirt-python
! Framework
sudo systemctl enable libvirtd.service
! Speed
sudo systemctl start  libvirtd.service
! Host OS
</pre>
! Guest OS
! initsystem testing (systemd, rc.d, ...)
! Post-mortem
! Interop testing
! Notes
|- style="vertical-align:top;"
| [[Test Suite - KVM | KVM]]
| slower
| Fedora, Debian <br>(BSD anyone?)
| Alpine, Fedora, FreeBSD, NetBSD, OpenBSD, Debian
| yes
| shutdown, core, leaks, refcnt, selinux
| strongswan (Linux, FreeBSD), iked (OpenBSD), racoon (NetBSD), racoon2 (NetBSD)
| gold standard <br> ideal for BSD builds <br> idea for testing custom kernels <br> used by the [https://testing.libreswan.org Testing] machine <br> requires 9p (virtio anyone?)
|- style="vertical-align:top;"
| [[Test Suite - Namespace | Namespaces]]
| fast
| linux
| uses host's libreswan, kernel, and utilities
| no
| core, leaks
| strongswan (linux)?
| ideal for quick tests <br> requires libreswan to be built/installed on the host <br> requires all dependencies to be installed on the host <br> test results sensitive  differing kernel and utilities
|- style="vertical-align:top;"
| [[Test Suite - Docker | Docker]]
|
| linux
| uses host's kernel <br> uses distro's utilities
| ?
| ?
| ?
| ideal for cross-linux builds (CentOS 6, 7, 8, Fedora 28 - rawhide, Debian, Ubuntu) <br> sensitive to differing kernel and utilities
|}


{{ ambox | nocat=true | type=important | text = Because our VMs don't run as qemu, /var/lib/libvirt/qemu needs to be changed using chmod to make it writable for the qemu group. This needs to be repeated if the libvirtd package is updated on the system }}
== How tests work ==


 
All the test cases involving VMs are located in the libreswan directory under <tt>testing/pluto/</tt>. The most basic test case is called basic-pluto-01. Each test case consists of a few files:
Various tools are used or convenient to have when running tests:
 
<pre>
sudo yum install python-pexpect git tcpdump
</pre>
 
The libreswan source tree includes all the components that are used on the host and inside the test VMs. To get the latest source code using git:
 
<pre>
git clone https://github.com/libreswan/libreswan
cd libreswan
</pre>
 
== Creating the VMs ==
 
A configuration file called kvmsetup.sh is used to configure a few parameters for the test suite:
 
<pre>
cp kvmsetup.sh.sample kvmsetup.sh
</pre>
 
This file contains various environment variables used for creating and running the tests. In the example version, the KVMPREFIX= is set to the home directory of the user "build". The POOLSPACE= is where all the VM images will be stored. There should be at least 16GB of free disk space in the pool/ directory. You can change the OSTYPE= if you prefer to use ubuntu guests over fedora guests. We recommend that the host and guest run the same OS - it makes things like running gdb on the host for core dumps created in the guests much easier. The OSMEDIA= can be changed to point to a local distribution mirror.
 
Once the kvmsetup.sh file has been edited, we can create the VMs:
 
<pre>
sh testing/libvirt/install.sh
</pre>
 
First, a new VM is added to the system called "fedorabase" (or "ubuntubase"). This is an automated minimal install using kickstart. Once installed, the disk image is converted to QCOW and copied for each test VM, west, east, north, road and nic. A few virtual networks are created to hook up the VMs in isolation. These virtual networks have names like "192_1_2_0" and use bridge interfaces names like "swan12". Fincally, the actual VMs are added to the system's libvirt/KVM system and the "fedorabase" VM is deleted.
 
Since you will be using ssh a lot to login to these machines, it is recommended to put their names in /etc/hosts:
 
<pre>
# /etc/hosts entries for libreswan test suite
192.1.2.45 west
192.1.2.23 east
192.0.3.254 north
192.1.3.209 road
192.1.2.254 nic
</pre>
 
== Preparing the VMs ==
 
The VMs came pre-installed with everything, except libreswan. We do not want to use the OS libreswan package because we want to run our own version to test our code changes. Some of the test cases use the NETKEY/XFRM IPsec stack but most test cases use the KLIPS IPsec stack. Login to the first VM and compile and install the libreswan userland and KLIPS ipsec kernel module:
 
<pre>
[biuld@host:~/libreswan $ ssh root@west
[...]
[root@west source]# swan-update
</pre>
 
swan-update first builds libreswan and then installs libreswan. For the other VMs (except "nic" which never runs IPsec)  we only need to install, as libreswan is already built:
 
<pre>
for host in east north road; do ssh root@$host swan-install
</pre>
 
All VMs are now fully provisioned to run test cases.
 
{{ ambox | nocat=true | type=important | text = The directories /source and /testing inside any VM are automatically mounted from the host's libreswan directory. Do not move the libreswan or the pool space directory on the host }}
 
== Running a test case ==
 
All the test cases involving VMs are located in the libreswan directory under testing/pluto/ . The most basic test case is called basic-pluto-01. Each test case consists of a few files:


* description.txt to explain what this test case actually tests
* description.txt to explain what this test case actually tests
Line 112: Line 64:
* testparams.sh if there are any non-default test parameters
* testparams.sh if there are any non-default test parameters


 
Once the test run has completed, you will see an OUTPUT/ directory in the test case directory:
You can run this test case by issuing the following command on the host:
 
<pre>
cd testing/pluto/basic-pluto-01/
../../utils/utils/dotest.sh
</pre>
 
Once the testrun has completed, you will see an OUTPUT/ directory in the test case directory:


<pre>
<pre>
Line 128: Line 72:
</pre>
</pre>


* RESULT is a text file (whose format is sure to change in the next few months) stating wether the test succeeded or failed.
* RESULT is a text file (whose format is sure to change in the next few months) stating whether the test succeeded or failed.
* The diff files show the differences between this testrun and the last known good output.
* The diff files show the differences between this testrun and the last known good output.
* Each VM's serial (sanitized) console log  (eg west.console.txt)
* Each VM's serial (sanitized) console log  (eg west.console.txt)
Line 136: Line 80:
* Any core dumps generated if a pluto daemon crashed
* Any core dumps generated if a pluto daemon crashed


== Diagnosing inside the VM ==
; testing/baseconfigs/
:  configuration files installed on guest machines
; testing/guestbin/
:  shell scripts used by tests, and run on the guest
; testing/linux-system-roles.vpn/
:  ???
; testing/packaging/
:  ???
; testing/pluto/TESTLIST
:  list of tests, and their expected outcome
; testing/pluto/*/
:  individual test directories
; testing/programs/
:  executables used by tests, and run on the guest
; testing/sanitizers/
:  filters for cleaning up the test output
; testing/utils/
:  test drivers and other host tools
; testing/x509/
:  certificates, scripts are run on a guest


Once a testrun has completed, the VMs shut down the ipsec subsystem. You can use ssh to login as root on any host (password "swan") and rerun the testcase manually. This gives you a change to repeat a crasher while using gdb:
== Network Diagrams ==


<pre>
=== Current Network Diagram ===
ssh root@east
ipsec setup start
pidof pluto
cd /source/OBJ*
gdb programs/pluto/pluto
gdb> attach <pid>
gdb> cont
</pre>


in another window, you can login to east and re-trigger the failure. You can either use the root command history using the arrow keys to start ipsec and load the right connection, or you can re-run the "init.sh" file:
* interface-0 (eth0, vio0, vioif0) is connected to SWANDEFAULT which has a NAT gateway to the internet
** the exceptions are the Fedora test domains: EAST, WEST, ROAD, NORTH; should they?
** the BSD domains always up inteface-0 so that /pool, /source, and /testing can be NFS mounted
** NIC needs to run DHCP on eth0 manually; how?
** transmogrify does not try to modify interface-0(SWANDEFAULT) (it breaks established network sessions such as NFS)
* the interface names do not have consistent order (see comment above about Fedora's interface-0 not pointing at SWANDEFAULT)
** Fedora has ethN
** OpenBSD has vioN (different order)
** NetBSD has vioifN (different order)


<pre>
  LEFT                                                              RIGHT
ssh root@west
cd /testing/pluto/basic-pluto-01
sh ./westinit.sh
</pre>


You can either continue with running "westrun.sh" or you can look at westrun.sh and issue the commands manually.
  192.0.3.0/24 -------------------------------------+-- 2001:db8:0:3::/64
                                                    |
                                              2001:db8:0:3::254
                                              192.0.3.254(eth0/2)
                  ROAD                            NORTH
              192.1.3.209(eth0/1)            192.1.3.33(eth1)
              2001:db8:1:3::209              2001:db8:1:3::33
                    |                              |
  192.1.3.0/254 ----+----------------+--------------+-- 2001:db8:1:3::/64
                                    |
                              2001:db8:1:3::254
                                192.1.3.254(eth2)
                                    NIC---swandefault(0)
                                192.1.2.254(eth1)
                              2001:db8:1:2::254
                                    |
  192.1.2.0/24 ---+------------------+-------------+--- 2001:db8:1:2::/64
                  |                                |
            2001:db8:1:2::45                2001:db8:1:2::23
            192.1.2.45(eth1)                192.1.2.23(eth1)
                WEST---[swandefault(0)]          EAST---[swandefault(0)]
            192.0.1.254(eth0/2)              192.0.2.254(eth0/2)
            2001:db8:0:1::254                2001:db8:0:2::254
                  |                                |
                  |              192.0.2.0/255 ---+--- 2001:db8:0:2::/64
                  |
  192.0.1.0/255 --+------------------------------------ 2001:db8:0:1::/64
 
  192.1.4.0/255 --------------------------------------- 2001:db8:1:4::/64


== Running all test cases ==
=== Proposed Network Diagram ===


To run all test cases, you need to be able to compile libreswan on the host (not for any good reason, but "make check" runs "make programs" first). You might need to install some build requirements:
Changes:
* add NOC, unlike NIC can run libreswan
* add SUNRISE (behind east) and SUNSET (behind west)
Notes:
* if0 (eth0, vio0, vioif0) is connected to SWANDEFAULT which has a NAT gateway to the internet
** the exceptions are the Fedora test domains: EAST, WEST, ROAD, NORTH; should they?
** the BSD domains use if0 to NFS mount /bench, /pool, /source, and /testing
** NIC needs to run DHCP on eth0 manually; how?
** transmogrify does not try to modify if0(SWANDEFAULT) (it breaks established network sessions such as NFS)
* the interface names still do not have consistent order
** ifN denotes a consistent numbering
** ethN denotes Fedora only assignment
** bsdN denotes a BSD only assignment (FreeBSD has ???N, NetBSD has vioifN, OpenBSD has vioN)
* re-number networks so that they are RFC friendly, these RFCs came up in discussion; alternatively do nothing
** https://www.rfc-editor.org/rfc/rfc1918
** https://www.rfc-editor.org/rfc/rfc5737
** https://www.rfc-editor.org/rfc/rfc3849
** https://www.rfc-editor.org/rfc/rfc5180
** https://www.rfc-editor.org/rfc/rfc2544
** https://www.rfc-editor.org/rfc/rfc6890


<pre>
  192.0.3.0/24 --------------------------------------+---- 2001:db8:0:3::/64
sudo yum install flex bison gmp-devel nss-devel nspr-devel openldap-devel curl-devel pam-devel unbound-devel fipscheck-devel libcap-ng-devel
                                                      |
</pre>
                                              2001:db8:0:3::254
                                                192.0.3.254
                                                  <eth0/2>
                  ROAD                              NORTH
                <eth0/1>                          <eth1>
              192.1.3.209                        192.1.3.33
            2001:db8:1:3::209                2001:db8:1:3::33
                    |                                |
  192.1.3.0/254 ----+---+-------------------------+---+---- 2001:db8:1:3::/64
                        |                        |
                2001:db8:1:3::254        2001:db8:1:3::253
                192.1.3.254(eth2)          192.1.3.253
                    <eth2>                    <eth2>
                      NIC-<if0>-swandefault    NOC-<if0>-swandefault
                    <eth1>                    <eth1>
                  192.1.2.254              192.1.2.253
                2001:db8:1:2::254        2001:db8:1:2::253
                        |                        |
  192.1.2.0/24 -----+--+-------------------------+--+----- 2001:db8:1:2::/64
                    |                              |
            2001:db8:1:2::45                2001:db8:1:2::23
                192.1.2.45                      192.1.2.23
                  <eth1>                          <eth1>
                  WEST-[<bsd0>-swandefault]      EAST-[<bsd0>-swandefault]
                  <eth0/2>                        <eth0/2>
                192.0.1.254                    192.0.2.254
            2001:db8:0:1::254              2001:db8:0:2::254
                    |                              |
                    |        192.0.2.0/255 ---------+--+-- 2001:db8:0:2::/64
                    |                                  |
                    |                          2001:db8:0:2::252
                    |                            192.0.2.252
                    |                                <eth1>
                    |                                RISE-<eth0>-swandefault
                    |
  192.0.1.0/255 --+--+------------------------------------- 2001:db8:0:1::/64
                  |
          2001:db8:0:1::252
            192.0.1.252
                <if1>
                SET-<if0>-swandefault
 
  192.1.4.0/255 --------------------------------------- 2001:db8:1:4::/64


To run all test cases (which include non-VM based test cases), run:
=== Hand Sketch of Current Network ===


<pre>
[[File:networksketch.png]]
make check
</pre>


== Updating the VMs ==
=== Original Network Diagram ===


Sometimes you want to update a VM's system or add a package to assist with debugging. This requires an internet connection. While the VMs are completely isolated, the "nic" VM can be configured to give internet access to the machines:
[[File:testnet.png]]
 
<pre>
ifup eth3
iptables -I POSTROUTING -t nat -o eth3 -j MASQUERADE
</pre>
 
On the other VMs, change the nameserver entry in /etc/resolv.conf to point to a valid resolver (eg 8.8.8.8 or 193.110.157.123) and the VM will have full internet connectivity.
 
{{ ambox | nocat=true | type=important | text = Do not enable eth3 on "nic" per default, as it will affect the actual test cases that are run. }}
 
== Various notes ==
 
* Currently, only one test can run at a time.
* You can peek at the guests using virt-manager or you can ssh into the test machines from the host.
* All VMs are shutdown and restarted between tests to ensure a clean start

Latest revision as of 18:08, 4 March 2024

Nightly Test Tesults

Libreswan's testsuite is run nightly. The results are published here, with the most recent result here. The tests are categorized as:

  • good: these tests are expected to pass (unfortunately, some still have timing problems and occasionally fail)
  • wip: these tests require further work, for instance the result may not be deterministic, or the bug they demonstrate hasn't yet been fixed
  • skiptest: these tests require manual intervention to run

To run tests locally, read on.

Running tests

The libreswan tests, in testing/pluto, can be run using several different mechanisms:

Test Frameworks
Framework Speed Host OS Guest OS initsystem testing (systemd, rc.d, ...) Post-mortem Interop testing Notes
KVM slower Fedora, Debian
(BSD anyone?) 		Alpine, Fedora, FreeBSD, NetBSD, OpenBSD, Debian yes shutdown, core, leaks, refcnt, selinux strongswan (Linux, FreeBSD), iked (OpenBSD), racoon (NetBSD), racoon2 (NetBSD) gold standard
ideal for BSD builds
idea for testing custom kernels
used by the Testing machine
requires 9p (virtio anyone?)
Namespaces fast linux uses host's libreswan, kernel, and utilities no core, leaks strongswan (linux)? ideal for quick tests
requires libreswan to be built/installed on the host
requires all dependencies to be installed on the host
test results sensitive differing kernel and utilities
Docker linux uses host's kernel
uses distro's utilities 		? ? ? ideal for cross-linux builds (CentOS 6, 7, 8, Fedora 28 - rawhide, Debian, Ubuntu)
sensitive to differing kernel and utilities

How tests work

All the test cases involving VMs are located in the libreswan directory under testing/pluto/. The most basic test case is called basic-pluto-01. Each test case consists of a few files:

  • description.txt to explain what this test case actually tests
  • ipsec.conf files - for host west is called west.conf. This can also include configuration files for strongswan or racoon2 for interop testig
  • ipsec.secret files - if non-default configurations are used. also uses the host syntax, eg west.secrets, east.secrets.
  • An init.sh file for each VM that needs to start (eg westinit.sh, eastinit.sh, etc)
  • One run.sh file for the host that is the initiator (eg westrun.sh)
  • Known good (sanitized) output for each VM (eg west.console.txt, east.console.txt)
  • testparams.sh if there are any non-default test parameters

Once the test run has completed, you will see an OUTPUT/ directory in the test case directory: 

$ ls OUTPUT/
east.console.diff  east.console.verbose.txt  RESULT       west.console.txt          west.pluto.log
east.console.txt   east.pluto.log            swan12.pcap  west.console.diff  west.console.verbose.txt
  • RESULT is a text file (whose format is sure to change in the next few months) stating whether the test succeeded or failed.
  • The diff files show the differences between this testrun and the last known good output.
  • Each VM's serial (sanitized) console log (eg west.console.txt)
  • Each VM's unsanitized verbose console output (eg west.console.verbose.txt)
  • A network capture from the bridge device (eg swan12.pcap)
  • Each VM's pluto log, created with plutodebug=all (eg west.pluto.log)
  • Any core dumps generated if a pluto daemon crashed
testing/baseconfigs/
configuration files installed on guest machines
testing/guestbin/
shell scripts used by tests, and run on the guest
testing/linux-system-roles.vpn/
???
testing/packaging/
???
testing/pluto/TESTLIST
list of tests, and their expected outcome
testing/pluto/*/
individual test directories
testing/programs/
executables used by tests, and run on the guest
testing/sanitizers/
filters for cleaning up the test output
testing/utils/
test drivers and other host tools
testing/x509/
certificates, scripts are run on a guest

Network Diagrams

Current Network Diagram

  • interface-0 (eth0, vio0, vioif0) is connected to SWANDEFAULT which has a NAT gateway to the internet
    • the exceptions are the Fedora test domains: EAST, WEST, ROAD, NORTH; should they?
    • the BSD domains always up inteface-0 so that /pool, /source, and /testing can be NFS mounted
    • NIC needs to run DHCP on eth0 manually; how?
    • transmogrify does not try to modify interface-0(SWANDEFAULT) (it breaks established network sessions such as NFS)
  • the interface names do not have consistent order (see comment above about Fedora's interface-0 not pointing at SWANDEFAULT)
    • Fedora has ethN
    • OpenBSD has vioN (different order)
    • NetBSD has vioifN (different order)
 LEFT                                                              RIGHT

 192.0.3.0/24 -------------------------------------+-- 2001:db8:0:3::/64
                                                   |
                                             2001:db8:0:3::254
                                              192.0.3.254(eth0/2)
                 ROAD                            NORTH
              192.1.3.209(eth0/1)             192.1.3.33(eth1)
             2001:db8:1:3::209               2001:db8:1:3::33
                   |                               |
 192.1.3.0/254 ----+----------------+--------------+-- 2001:db8:1:3::/64
                                    |
                              2001:db8:1:3::254
                               192.1.3.254(eth2)
                                   NIC---swandefault(0)
                               192.1.2.254(eth1)
                              2001:db8:1:2::254
                                    |
 192.1.2.0/24 ---+------------------+-------------+--- 2001:db8:1:2::/64
                 |                                |
           2001:db8:1:2::45                 2001:db8:1:2::23
            192.1.2.45(eth1)                 192.1.2.23(eth1)
               WEST---[swandefault(0)]          EAST---[swandefault(0)]
            192.0.1.254(eth0/2)              192.0.2.254(eth0/2)
           2001:db8:0:1::254                2001:db8:0:2::254
                 |                                |
                 |               192.0.2.0/255 ---+--- 2001:db8:0:2::/64
                 |
 192.0.1.0/255 --+------------------------------------ 2001:db8:0:1::/64
 
 192.1.4.0/255 --------------------------------------- 2001:db8:1:4::/64

Proposed Network Diagram

Changes:

  • add NOC, unlike NIC can run libreswan
  • add SUNRISE (behind east) and SUNSET (behind west)

Notes: 

  192.0.3.0/24 --------------------------------------+---- 2001:db8:0:3::/64
                                                     |
                                              2001:db8:0:3::254
                                               192.0.3.254
                                                  <eth0/2>
                 ROAD                              NORTH
                <eth0/1>                          <eth1>
             192.1.3.209                        192.1.3.33
            2001:db8:1:3::209                 2001:db8:1:3::33
                   |                                 |
 192.1.3.0/254 ----+---+-------------------------+---+---- 2001:db8:1:3::/64
                       |                         |
               2001:db8:1:3::254         2001:db8:1:3::253
                192.1.3.254(eth2)           192.1.3.253
                    <eth2>                    <eth2>
                      NIC-<if0>-swandefault     NOC-<if0>-swandefault
                    <eth1>                    <eth1>
                  192.1.2.254               192.1.2.253
               2001:db8:1:2::254         2001:db8:1:2::253
                       |                         |
  192.1.2.0/24 -----+--+-------------------------+--+----- 2001:db8:1:2::/64
                    |                               |
            2001:db8:1:2::45                2001:db8:1:2::23
               192.1.2.45                      192.1.2.23
                 <eth1>                          <eth1>
                  WEST-[<bsd0>-swandefault]       EAST-[<bsd0>-swandefault]
                 <eth0/2>                        <eth0/2>
               192.0.1.254                     192.0.2.254
            2001:db8:0:1::254               2001:db8:0:2::254
                    |                               |
                    |        192.0.2.0/255 ---------+--+-- 2001:db8:0:2::/64
                    |                                  |
                    |                          2001:db8:0:2::252
                    |                             192.0.2.252
                    |                                <eth1>
                    |                                 RISE-<eth0>-swandefault
                    |
 192.0.1.0/255 --+--+------------------------------------- 2001:db8:0:1::/64
                 |
          2001:db8:0:1::252
           192.0.1.252
               <if1>
                SET-<if0>-swandefault
 
 192.1.4.0/255 --------------------------------------- 2001:db8:1:4::/64

Hand Sketch of Current Network

Networksketch.png

Original Network Diagram

Testnet.png

Retrieved from "https://libreswan.org/wiki/index.php?title=Test_Suite&oldid=22341"