Test Suite

Nightly Test Tesults

Libreswan's testsuite is run nightly. The results are published here, with the most recent result here. The tests are categorized as:

• good: these tests are expected to pass (unfortunately, some still have timing problems and occasionally fail)
• wip: these tests require further work, for instance the result may not be deterministic, or the bug they demonstrate hasn't yet been fixed
• skiptest: these tests require manual intervention to run

To run tests locally, read on.

Running tests

The libreswan tests, in testing/pluto, can be run using several different mechanisms:

Test Frameworks

Framework	Speed	Host OS	Guest OS	initsystem testing (systemd, rc.d, ...)	Post-mortem	Interop testing	Notes
KVM	slower	Fedora, Debian (BSD anyone?)	Alpine, Fedora, FreeBSD, NetBSD, OpenBSD, Debian	yes	shutdown, core, leaks, refcnt, selinux	strongswan (Linux, FreeBSD), iked (OpenBSD), racoon (NetBSD), racoon2 (NetBSD)	gold standard ideal for BSD builds idea for testing custom kernels used by the Testing machine requires 9p (virtio anyone?)
Namespaces	fast	linux	uses host's libreswan, kernel, and utilities	no	core, leaks	strongswan (linux)?	ideal for quick tests requires libreswan to be built/installed on the host requires all dependencies to be installed on the host test results sensitive differing kernel and utilities
Docker		linux	uses host's kernel uses distro's utilities	?	?	?	ideal for cross-linux builds (CentOS 6, 7, 8, Fedora 28 - rawhide, Debian, Ubuntu) sensitive to differing kernel and utilities

How tests work

All the test cases involving VMs are located in the libreswan directory under testing/pluto/. The most basic test case is called basic-pluto-01. Each test case consists of a few files:

• description.txt to explain what this test case actually tests
• ipsec.conf files - for host west is called west.conf. This can also include configuration files for strongswan or racoon2 for interop testig
• ipsec.secret files - if non-default configurations are used. also uses the host syntax, eg west.secrets, east.secrets.
• An init.sh file for each VM that needs to start (eg westinit.sh, eastinit.sh, etc)
• One run.sh file for the host that is the initiator (eg westrun.sh)
• Known good (sanitized) output for each VM (eg west.console.txt, east.console.txt)
• testparams.sh if there are any non-default test parameters

Once the test run has completed, you will see an OUTPUT/ directory in the test case directory:

$ ls OUTPUT/
east.console.diff  east.console.verbose.txt  RESULT       west.console.txt          west.pluto.log
east.console.txt   east.pluto.log            swan12.pcap  west.console.diff  west.console.verbose.txt

• RESULT is a text file (whose format is sure to change in the next few months) stating whether the test succeeded or failed.
• The diff files show the differences between this testrun and the last known good output.
• Each VM's serial (sanitized) console log (eg west.console.txt)
• Each VM's unsanitized verbose console output (eg west.console.verbose.txt)
• A network capture from the bridge device (eg swan12.pcap)
• Each VM's pluto log, created with plutodebug=all (eg west.pluto.log)
• Any core dumps generated if a pluto daemon crashed

testing/baseconfigs/

configuration files installed on guest machines

testing/guestbin/

shell scripts used by tests, and run on the guest

testing/linux-system-roles.vpn/

???

testing/packaging/

???

testing/pluto/TESTLIST

list of tests, and their expected outcome

testing/pluto/*/

individual test directories

testing/programs/

executables used by tests, and run on the guest

testing/sanitizers/

filters for cleaning up the test output

testing/utils/

test drivers and other host tools

testing/x509/

certificates, scripts are run on a guest

Network Diagrams

Current Network Diagram

• interface-0 (eth0, vio0, vioif0) is connected to SWANDEFAULT which has a NAT gateway to the internet
  • the exceptions are the Fedora test domains: EAST, WEST, ROAD, NORTH; should they?
  • the BSD domains always up inteface-0 so that /pool, /source, and /testing can be NFS mounted
  • NIC needs to run DHCP on eth0 manually; how?
  • transmogrify does not try to modify interface-0(SWANDEFAULT) (it breaks established network sessions such as NFS)
• the interface names do not have consistent order (see comment above about Fedora's interface-0 not pointing at SWANDEFAULT)
  • Fedora has ethN
  • OpenBSD has vioN (different order)
  • NetBSD has vioifN (different order)

 LEFT                                                              RIGHT

 192.0.3.0/24 -------------------------------------+-- 2001:db8:0:3::/64
                                                   |
                                             2001:db8:0:3::254
                                              192.0.3.254(eth0/2)
                 ROAD                            NORTH
              192.1.3.209(eth0/1)             192.1.3.33(eth1)
             2001:db8:1:3::209               2001:db8:1:3::33
                   |                               |
 192.1.3.0/254 ----+----------------+--------------+-- 2001:db8:1:3::/64
                                    |
                              2001:db8:1:3::254
                               192.1.3.254(eth2)
                                   NIC---swandefault(0)
                               192.1.2.254(eth1)
                              2001:db8:1:2::254
                                    |
 192.1.2.0/24 ---+------------------+-------------+--- 2001:db8:1:2::/64
                 |                                |
           2001:db8:1:2::45                 2001:db8:1:2::23
            192.1.2.45(eth1)                 192.1.2.23(eth1)
               WEST---[swandefault(0)]          EAST---[swandefault(0)]
            192.0.1.254(eth0/2)              192.0.2.254(eth0/2)
           2001:db8:0:1::254                2001:db8:0:2::254
                 |                                |
                 |               192.0.2.0/255 ---+--- 2001:db8:0:2::/64
                 |
 192.0.1.0/255 --+------------------------------------ 2001:db8:0:1::/64
 
 192.1.4.0/255 --------------------------------------- 2001:db8:1:4::/64

Proposed Network Diagram

Changes:

• add NOC, unlike NIC can run libreswan
• add SUNRISE (behind east) and SUNSET (behind west)

Notes:

• if0 (eth0, vio0, vioif0) is connected to SWANDEFAULT which has a NAT gateway to the internet
  • the exceptions are the Fedora test domains: EAST, WEST, ROAD, NORTH; should they?
  • the BSD domains use if0 to NFS mount /bench, /pool, /source, and /testing
  • NIC needs to run DHCP on eth0 manually; how?
  • transmogrify does not try to modify if0(SWANDEFAULT) (it breaks established network sessions such as NFS)
• the interface names still do not have consistent order
  • ifN denotes a consistent numbering
  • ethN denotes Fedora only assignment
  • bsdN denotes a BSD only assignment (FreeBSD has ???N, NetBSD has vioifN, OpenBSD has vioN)
• re-number networks so that they are RFC friendly, these RFCs came up in discussion; alternatively do nothing
  • https://www.rfc-editor.org/rfc/rfc1918
  • https://www.rfc-editor.org/rfc/rfc5737
  • https://www.rfc-editor.org/rfc/rfc3849
  • https://www.rfc-editor.org/rfc/rfc5180
  • https://www.rfc-editor.org/rfc/rfc2544
  • https://www.rfc-editor.org/rfc/rfc6890

  192.0.3.0/24 --------------------------------------+---- 2001:db8:0:3::/64
                                                     |
                                              2001:db8:0:3::254
                                               192.0.3.254
                                                  <eth0/2>
                 ROAD                              NORTH
                <eth0/1>                          <eth1>
             192.1.3.209                        192.1.3.33
            2001:db8:1:3::209                 2001:db8:1:3::33
                   |                                 |
 192.1.3.0/254 ----+---+-------------------------+---+---- 2001:db8:1:3::/64
                       |                         |
               2001:db8:1:3::254         2001:db8:1:3::253
                192.1.3.254(eth2)           192.1.3.253
                    <eth2>                    <eth2>
                      NIC-<if0>-swandefault     NOC-<if0>-swandefault
                    <eth1>                    <eth1>
                  192.1.2.254               192.1.2.253
               2001:db8:1:2::254         2001:db8:1:2::253
                       |                         |
  192.1.2.0/24 -----+--+-------------------------+--+----- 2001:db8:1:2::/64
                    |                               |
            2001:db8:1:2::45                2001:db8:1:2::23
               192.1.2.45                      192.1.2.23
                 <eth1>                          <eth1>
                  WEST-[<bsd0>-swandefault]       EAST-[<bsd0>-swandefault]
                 <eth0/2>                        <eth0/2>
               192.0.1.254                     192.0.2.254
            2001:db8:0:1::254               2001:db8:0:2::254
                    |                               |
                    |        192.0.2.0/255 ---------+--+-- 2001:db8:0:2::/64
                    |                                  |
                    |                          2001:db8:0:2::252
                    |                             192.0.2.252
                    |                                <eth1>
                    |                                 RISE-<eth0>-swandefault
                    |
 192.0.1.0/255 --+--+------------------------------------- 2001:db8:0:1::/64
                 |
          2001:db8:0:1::252
           192.0.1.252
               <if1>
                SET-<if0>-swandefault
 
 192.1.4.0/255 --------------------------------------- 2001:db8:1:4::/64

Hand Sketch of Current Network

Original Network Diagram