Difference between revisions of "Test Suite"
Jump to navigation
Jump to search
(strip off KVM stuff) |
(point to testing.libreswan.org) |
||
(49 intermediate revisions by 2 users not shown) | |||
Line 1: | Line 1: | ||
+ | == Nightly Test Tesults == | ||
+ | Libreswan's testsuite is run nightly. The results are published [https://testing.libreswan.org/ here], with the most recent result [https://testing.libreswan.org/current here]. The tests are categorized as: | ||
− | + | * good: these tests are expected to pass (unfortunately, some still have timing problems and occasionally fail) | |
+ | * wip: these tests require further work, for instance the result may not be deterministic, or the bug they demonstrate hasn't yet been fixed | ||
+ | * skiptest: these tests require manual intervention to run | ||
− | + | To run tests locally, read on. | |
− | |||
− | |||
− | |||
− | |||
− | + | == Running tests == | |
− | The | + | The libreswan tests, in testing/pluto, can be run using several different mechanisms: |
− | + | {| class="wikitable" | |
+ | |+ Test Frameworks | ||
+ | ! Framework | ||
+ | ! Speed | ||
+ | ! Host OS | ||
+ | ! Guest OS | ||
+ | ! initsystem testing (systemd, rc.d, ...) | ||
+ | ! Post-mortem | ||
+ | ! Interop testing | ||
+ | ! Notes | ||
+ | |- style="vertical-align:top;" | ||
+ | | [[Test Suite - KVM | KVM]] | ||
+ | | slower | ||
+ | | Fedora, Debian <br>(BSD anyone?) | ||
+ | | Alpine, Fedora, FreeBSD, NetBSD, OpenBSD, Debian | ||
+ | | yes | ||
+ | | shutdown, core, leaks, refcnt, selinux | ||
+ | | strongswan (Linux, FreeBSD), iked (OpenBSD), racoon (NetBSD), racoon2 (NetBSD) | ||
+ | | gold standard <br> ideal for BSD builds <br> idea for testing custom kernels <br> used by the [https://testing.libreswan.org Testing] machine <br> requires 9p (virtio anyone?) | ||
+ | |- style="vertical-align:top;" | ||
+ | | [[Test Suite - Namespace | Namespaces]] | ||
+ | | fast | ||
+ | | linux | ||
+ | | uses host's libreswan, kernel, and utilities | ||
+ | | no | ||
+ | | core, leaks | ||
+ | | strongswan (linux)? | ||
+ | | ideal for quick tests <br> requires libreswan to be built/installed on the host <br> requires all dependencies to be installed on the host <br> test results sensitive differing kernel and utilities | ||
+ | |- style="vertical-align:top;" | ||
+ | | [[Test Suite - Docker | Docker]] | ||
+ | | | ||
+ | | linux | ||
+ | | uses host's kernel <br> uses distro's utilities | ||
+ | | ? | ||
+ | | ? | ||
+ | | ? | ||
+ | | ideal for cross-linux builds (CentOS 6, 7, 8, Fedora 28 - rawhide, Debian, Ubuntu) <br> sensitive to differing kernel and utilities | ||
+ | |} | ||
− | == | + | == How tests work == |
− | All the test cases involving VMs are located in the libreswan directory under testing/pluto/ . The most basic test case is called basic-pluto-01. Each test case consists of a few files: | + | All the test cases involving VMs are located in the libreswan directory under <tt>testing/pluto/</tt>. The most basic test case is called basic-pluto-01. Each test case consists of a few files: |
* description.txt to explain what this test case actually tests | * description.txt to explain what this test case actually tests | ||
Line 26: | Line 63: | ||
* Known good (sanitized) output for each VM (eg west.console.txt, east.console.txt) | * Known good (sanitized) output for each VM (eg west.console.txt, east.console.txt) | ||
* testparams.sh if there are any non-default test parameters | * testparams.sh if there are any non-default test parameters | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
Once the test run has completed, you will see an OUTPUT/ directory in the test case directory: | Once the test run has completed, you will see an OUTPUT/ directory in the test case directory: | ||
Line 69: | Line 80: | ||
* Any core dumps generated if a pluto daemon crashed | * Any core dumps generated if a pluto daemon crashed | ||
− | + | ; testing/baseconfigs/ | |
+ | : configuration files installed on guest machines | ||
+ | ; testing/guestbin/ | ||
+ | : shell scripts used by tests, and run on the guest | ||
+ | ; testing/linux-system-roles.vpn/ | ||
+ | : ??? | ||
+ | ; testing/packaging/ | ||
+ | : ??? | ||
+ | ; testing/pluto/TESTLIST | ||
+ | : list of tests, and their expected outcome | ||
+ | ; testing/pluto/*/ | ||
+ | : individual test directories | ||
+ | ; testing/programs/ | ||
+ | : executables used by tests, and run on the guest | ||
+ | ; testing/sanitizers/ | ||
+ | : filters for cleaning up the test output | ||
+ | ; testing/utils/ | ||
+ | : test drivers and other host tools | ||
+ | ; testing/x509/ | ||
+ | : certificates, scripts are run on a guest | ||
− | == | + | == Network Diagrams == |
− | + | === Current Network Diagram === | |
− | + | * interface-0 (eth0, vio0, vioif0) is connected to SWANDEFAULT which has a NAT gateway to the internet | |
− | + | ** the exceptions are the Fedora test domains: EAST, WEST, ROAD, NORTH; should they? | |
− | + | ** the BSD domains always up inteface-0 so that /pool, /source, and /testing can be NFS mounted | |
− | + | ** NIC needs to run DHCP on eth0 manually; how? | |
− | + | ** transmogrify does not try to modify interface-0(SWANDEFAULT) (it breaks established network sessions such as NFS) | |
− | ( | + | * the interface names do not have consistent order (see comment above about Fedora's interface-0 not pointing at SWANDEFAULT) |
− | + | ** Fedora has ethN | |
+ | ** OpenBSD has vioN (different order) | ||
+ | ** NetBSD has vioifN (different order) | ||
− | + | LEFT RIGHT | |
− | + | 192.0.3.0/24 -------------------------------------+-- 2001:db8:0:3::/64 | |
− | + | | | |
− | + | 2001:db8:0:3::254 | |
− | + | 192.0.3.254(eth0/2) | |
− | + | ROAD NORTH | |
+ | 192.1.3.209(eth0/1) 192.1.3.33(eth1) | ||
+ | 2001:db8:1:3::209 2001:db8:1:3::33 | ||
+ | | | | ||
+ | 192.1.3.0/254 ----+----------------+--------------+-- 2001:db8:1:3::/64 | ||
+ | | | ||
+ | 2001:db8:1:3::254 | ||
+ | 192.1.3.254(eth2) | ||
+ | NIC---swandefault(0) | ||
+ | 192.1.2.254(eth1) | ||
+ | 2001:db8:1:2::254 | ||
+ | | | ||
+ | 192.1.2.0/24 ---+------------------+-------------+--- 2001:db8:1:2::/64 | ||
+ | | | | ||
+ | 2001:db8:1:2::45 2001:db8:1:2::23 | ||
+ | 192.1.2.45(eth1) 192.1.2.23(eth1) | ||
+ | WEST---[swandefault(0)] EAST---[swandefault(0)] | ||
+ | 192.0.1.254(eth0/2) 192.0.2.254(eth0/2) | ||
+ | 2001:db8:0:1::254 2001:db8:0:2::254 | ||
+ | | | | ||
+ | | 192.0.2.0/255 ---+--- 2001:db8:0:2::/64 | ||
+ | | | ||
+ | 192.0.1.0/255 --+------------------------------------ 2001:db8:0:1::/64 | ||
+ | |||
+ | 192.1.4.0/255 --------------------------------------- 2001:db8:1:4::/64 | ||
− | + | === Proposed Network Diagram === | |
− | + | Changes: | |
+ | * add NOC, unlike NIC can run libreswan | ||
+ | * add SUNRISE (behind east) and SUNSET (behind west) | ||
+ | Notes: | ||
+ | * if0 (eth0, vio0, vioif0) is connected to SWANDEFAULT which has a NAT gateway to the internet | ||
+ | ** the exceptions are the Fedora test domains: EAST, WEST, ROAD, NORTH; should they? | ||
+ | ** the BSD domains use if0 to NFS mount /bench, /pool, /source, and /testing | ||
+ | ** NIC needs to run DHCP on eth0 manually; how? | ||
+ | ** transmogrify does not try to modify if0(SWANDEFAULT) (it breaks established network sessions such as NFS) | ||
+ | * the interface names still do not have consistent order | ||
+ | ** ifN denotes a consistent numbering | ||
+ | ** ethN denotes Fedora only assignment | ||
+ | ** bsdN denotes a BSD only assignment (FreeBSD has ???N, NetBSD has vioifN, OpenBSD has vioN) | ||
+ | * re-number networks so that they are RFC friendly, these RFCs came up in discussion; alternatively do nothing | ||
+ | ** https://www.rfc-editor.org/rfc/rfc1918 | ||
+ | ** https://www.rfc-editor.org/rfc/rfc5737 | ||
+ | ** https://www.rfc-editor.org/rfc/rfc3849 | ||
+ | ** https://www.rfc-editor.org/rfc/rfc5180 | ||
+ | ** https://www.rfc-editor.org/rfc/rfc2544 | ||
+ | ** https://www.rfc-editor.org/rfc/rfc6890 | ||
− | + | 192.0.3.0/24 --------------------------------------+---- 2001:db8:0:3::/64 | |
+ | | | ||
+ | 2001:db8:0:3::254 | ||
+ | 192.0.3.254 | ||
+ | <eth0/2> | ||
+ | ROAD NORTH | ||
+ | <eth0/1> <eth1> | ||
+ | 192.1.3.209 192.1.3.33 | ||
+ | 2001:db8:1:3::209 2001:db8:1:3::33 | ||
+ | | | | ||
+ | 192.1.3.0/254 ----+---+-------------------------+---+---- 2001:db8:1:3::/64 | ||
+ | | | | ||
+ | 2001:db8:1:3::254 2001:db8:1:3::253 | ||
+ | 192.1.3.254(eth2) 192.1.3.253 | ||
+ | <eth2> <eth2> | ||
+ | NIC-<if0>-swandefault NOC-<if0>-swandefault | ||
+ | <eth1> <eth1> | ||
+ | 192.1.2.254 192.1.2.253 | ||
+ | 2001:db8:1:2::254 2001:db8:1:2::253 | ||
+ | | | | ||
+ | 192.1.2.0/24 -----+--+-------------------------+--+----- 2001:db8:1:2::/64 | ||
+ | | | | ||
+ | 2001:db8:1:2::45 2001:db8:1:2::23 | ||
+ | 192.1.2.45 192.1.2.23 | ||
+ | <eth1> <eth1> | ||
+ | WEST-[<bsd0>-swandefault] EAST-[<bsd0>-swandefault] | ||
+ | <eth0/2> <eth0/2> | ||
+ | 192.0.1.254 192.0.2.254 | ||
+ | 2001:db8:0:1::254 2001:db8:0:2::254 | ||
+ | | | | ||
+ | | 192.0.2.0/255 ---------+--+-- 2001:db8:0:2::/64 | ||
+ | | | | ||
+ | | 2001:db8:0:2::252 | ||
+ | | 192.0.2.252 | ||
+ | | <eth1> | ||
+ | | RISE-<eth0>-swandefault | ||
+ | | | ||
+ | 192.0.1.0/255 --+--+------------------------------------- 2001:db8:0:1::/64 | ||
+ | | | ||
+ | 2001:db8:0:1::252 | ||
+ | 192.0.1.252 | ||
+ | <if1> | ||
+ | SET-<if0>-swandefault | ||
+ | |||
+ | 192.1.4.0/255 --------------------------------------- 2001:db8:1:4::/64 | ||
− | === | + | === Hand Sketch of Current Network === |
− | + | [[File:networksketch.png]] | |
− | + | === Original Network Diagram === | |
− | |||
− | |||
− | + | [[File:testnet.png]] | |
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− |
Latest revision as of 17:08, 4 March 2024
Nightly Test Tesults
Libreswan's testsuite is run nightly. The results are published here, with the most recent result here. The tests are categorized as:
- good: these tests are expected to pass (unfortunately, some still have timing problems and occasionally fail)
- wip: these tests require further work, for instance the result may not be deterministic, or the bug they demonstrate hasn't yet been fixed
- skiptest: these tests require manual intervention to run
To run tests locally, read on.
Running tests
The libreswan tests, in testing/pluto, can be run using several different mechanisms:
Framework | Speed | Host OS | Guest OS | initsystem testing (systemd, rc.d, ...) | Post-mortem | Interop testing | Notes |
---|---|---|---|---|---|---|---|
KVM | slower | Fedora, Debian (BSD anyone?) |
Alpine, Fedora, FreeBSD, NetBSD, OpenBSD, Debian | yes | shutdown, core, leaks, refcnt, selinux | strongswan (Linux, FreeBSD), iked (OpenBSD), racoon (NetBSD), racoon2 (NetBSD) | gold standard ideal for BSD builds idea for testing custom kernels used by the Testing machine requires 9p (virtio anyone?) |
Namespaces | fast | linux | uses host's libreswan, kernel, and utilities | no | core, leaks | strongswan (linux)? | ideal for quick tests requires libreswan to be built/installed on the host requires all dependencies to be installed on the host test results sensitive differing kernel and utilities |
Docker | linux | uses host's kernel uses distro's utilities |
? | ? | ? | ideal for cross-linux builds (CentOS 6, 7, 8, Fedora 28 - rawhide, Debian, Ubuntu) sensitive to differing kernel and utilities |
How tests work
All the test cases involving VMs are located in the libreswan directory under testing/pluto/. The most basic test case is called basic-pluto-01. Each test case consists of a few files:
- description.txt to explain what this test case actually tests
- ipsec.conf files - for host west is called west.conf. This can also include configuration files for strongswan or racoon2 for interop testig
- ipsec.secret files - if non-default configurations are used. also uses the host syntax, eg west.secrets, east.secrets.
- An init.sh file for each VM that needs to start (eg westinit.sh, eastinit.sh, etc)
- One run.sh file for the host that is the initiator (eg westrun.sh)
- Known good (sanitized) output for each VM (eg west.console.txt, east.console.txt)
- testparams.sh if there are any non-default test parameters
Once the test run has completed, you will see an OUTPUT/ directory in the test case directory:
$ ls OUTPUT/ east.console.diff east.console.verbose.txt RESULT west.console.txt west.pluto.log east.console.txt east.pluto.log swan12.pcap west.console.diff west.console.verbose.txt
- RESULT is a text file (whose format is sure to change in the next few months) stating whether the test succeeded or failed.
- The diff files show the differences between this testrun and the last known good output.
- Each VM's serial (sanitized) console log (eg west.console.txt)
- Each VM's unsanitized verbose console output (eg west.console.verbose.txt)
- A network capture from the bridge device (eg swan12.pcap)
- Each VM's pluto log, created with plutodebug=all (eg west.pluto.log)
- Any core dumps generated if a pluto daemon crashed
- testing/baseconfigs/
- configuration files installed on guest machines
- testing/guestbin/
- shell scripts used by tests, and run on the guest
- testing/linux-system-roles.vpn/
- ???
- testing/packaging/
- ???
- testing/pluto/TESTLIST
- list of tests, and their expected outcome
- testing/pluto/*/
- individual test directories
- testing/programs/
- executables used by tests, and run on the guest
- testing/sanitizers/
- filters for cleaning up the test output
- testing/utils/
- test drivers and other host tools
- testing/x509/
- certificates, scripts are run on a guest
Network Diagrams
Current Network Diagram
- interface-0 (eth0, vio0, vioif0) is connected to SWANDEFAULT which has a NAT gateway to the internet
- the exceptions are the Fedora test domains: EAST, WEST, ROAD, NORTH; should they?
- the BSD domains always up inteface-0 so that /pool, /source, and /testing can be NFS mounted
- NIC needs to run DHCP on eth0 manually; how?
- transmogrify does not try to modify interface-0(SWANDEFAULT) (it breaks established network sessions such as NFS)
- the interface names do not have consistent order (see comment above about Fedora's interface-0 not pointing at SWANDEFAULT)
- Fedora has ethN
- OpenBSD has vioN (different order)
- NetBSD has vioifN (different order)
LEFT RIGHT
192.0.3.0/24 -------------------------------------+-- 2001:db8:0:3::/64 | 2001:db8:0:3::254 192.0.3.254(eth0/2) ROAD NORTH 192.1.3.209(eth0/1) 192.1.3.33(eth1) 2001:db8:1:3::209 2001:db8:1:3::33 | | 192.1.3.0/254 ----+----------------+--------------+-- 2001:db8:1:3::/64 | 2001:db8:1:3::254 192.1.3.254(eth2) NIC---swandefault(0) 192.1.2.254(eth1) 2001:db8:1:2::254 | 192.1.2.0/24 ---+------------------+-------------+--- 2001:db8:1:2::/64 | | 2001:db8:1:2::45 2001:db8:1:2::23 192.1.2.45(eth1) 192.1.2.23(eth1) WEST---[swandefault(0)] EAST---[swandefault(0)] 192.0.1.254(eth0/2) 192.0.2.254(eth0/2) 2001:db8:0:1::254 2001:db8:0:2::254 | | | 192.0.2.0/255 ---+--- 2001:db8:0:2::/64 | 192.0.1.0/255 --+------------------------------------ 2001:db8:0:1::/64 192.1.4.0/255 --------------------------------------- 2001:db8:1:4::/64
Proposed Network Diagram
Changes:
- add NOC, unlike NIC can run libreswan
- add SUNRISE (behind east) and SUNSET (behind west)
Notes:
- if0 (eth0, vio0, vioif0) is connected to SWANDEFAULT which has a NAT gateway to the internet
- the exceptions are the Fedora test domains: EAST, WEST, ROAD, NORTH; should they?
- the BSD domains use if0 to NFS mount /bench, /pool, /source, and /testing
- NIC needs to run DHCP on eth0 manually; how?
- transmogrify does not try to modify if0(SWANDEFAULT) (it breaks established network sessions such as NFS)
- the interface names still do not have consistent order
- ifN denotes a consistent numbering
- ethN denotes Fedora only assignment
- bsdN denotes a BSD only assignment (FreeBSD has ???N, NetBSD has vioifN, OpenBSD has vioN)
- re-number networks so that they are RFC friendly, these RFCs came up in discussion; alternatively do nothing
192.0.3.0/24 --------------------------------------+---- 2001:db8:0:3::/64 | 2001:db8:0:3::254 192.0.3.254 <eth0/2> ROAD NORTH <eth0/1> <eth1> 192.1.3.209 192.1.3.33 2001:db8:1:3::209 2001:db8:1:3::33 | | 192.1.3.0/254 ----+---+-------------------------+---+---- 2001:db8:1:3::/64 | | 2001:db8:1:3::254 2001:db8:1:3::253 192.1.3.254(eth2) 192.1.3.253 <eth2> <eth2> NIC-<if0>-swandefault NOC-<if0>-swandefault <eth1> <eth1> 192.1.2.254 192.1.2.253 2001:db8:1:2::254 2001:db8:1:2::253 | | 192.1.2.0/24 -----+--+-------------------------+--+----- 2001:db8:1:2::/64 | | 2001:db8:1:2::45 2001:db8:1:2::23 192.1.2.45 192.1.2.23 <eth1> <eth1> WEST-[<bsd0>-swandefault] EAST-[<bsd0>-swandefault] <eth0/2> <eth0/2> 192.0.1.254 192.0.2.254 2001:db8:0:1::254 2001:db8:0:2::254 | | | 192.0.2.0/255 ---------+--+-- 2001:db8:0:2::/64 | | | 2001:db8:0:2::252 | 192.0.2.252 | <eth1> | RISE-<eth0>-swandefault | 192.0.1.0/255 --+--+------------------------------------- 2001:db8:0:1::/64 | 2001:db8:0:1::252 192.0.1.252 <if1> SET-<if0>-swandefault 192.1.4.0/255 --------------------------------------- 2001:db8:1:4::/64