Test Suite: Difference between revisions
No edit summary |
(draft renumber) |
||
(145 intermediate revisions by 5 users not shown) | |||
Line 1: | Line 1: | ||
== Nightly Test Tesults == | |||
Libreswan | Libreswan's testsuite is run nightly. The results are published [https://testing.libreswan.org/ here], with the most recent result [https://testing.libreswan.org/current here]. The tests are categorized as: | ||
The | |||
* good: these tests are expected to pass (unfortunately, some still have timing problems and occasionally fail) | |||
* wip: these tests require further work, for instance the result may not be deterministic, or the bug they demonstrate hasn't yet been fixed | |||
* skiptest: these tests require manual intervention to run | |||
To run tests locally, read on. | |||
== Running tests == | |||
The libreswan tests, in testing/pluto, can be run using several different mechanisms: | |||
== | {| class="wikitable" | ||
|+ Test Frameworks | |||
! Framework | |||
! Speed | |||
! Host OS | |||
! Guest OS | |||
! initsystem testing (systemd, rc.d, ...) | |||
! Post-mortem | |||
! Interop testing | |||
! Notes | |||
|- style="vertical-align:top;" | |||
| [[Test Suite - KVM | KVM]] | |||
| slower | |||
| Fedora, Debian <br>(BSD anyone?) | |||
| Alpine, Fedora, FreeBSD, NetBSD, OpenBSD, Debian | |||
| yes | |||
| shutdown, core, leaks, refcnt, selinux | |||
| strongswan (Linux, FreeBSD), iked (OpenBSD), racoon (NetBSD), racoon2 (NetBSD) | |||
| gold standard <br> ideal for BSD builds <br> idea for testing custom kernels <br> used by the [https://testing.libreswan.org Testing] machine <br> requires 9p (virtio anyone?) | |||
|- style="vertical-align:top;" | |||
| [[Test Suite - Namespace | Namespaces]] | |||
| fast | |||
| linux | |||
| uses host's libreswan, kernel, and utilities | |||
| no | |||
| core, leaks | |||
| strongswan (linux)? | |||
| ideal for quick tests <br> requires libreswan to be built/installed on the host <br> requires all dependencies to be installed on the host <br> test results sensitive differing kernel and utilities | |||
|- style="vertical-align:top;" | |||
| [[Test Suite - Docker | Docker]] | |||
| | |||
| linux | |||
| uses host's kernel <br> uses distro's utilities | |||
| ? | |||
| ? | |||
| ? | |||
| ideal for cross-linux builds (CentOS 6, 7, 8, Fedora 28 - rawhide, Debian, Ubuntu) <br> sensitive to differing kernel and utilities | |||
|} | |||
== How tests work == | |||
All the test cases involving VMs are located in the libreswan directory under <tt>testing/pluto/</tt>. The most basic test case is called basic-pluto-01. Each test case consists of a few files: | |||
All the test cases involving VMs are located in the libreswan directory under testing/pluto/ . The most basic test case is called basic-pluto-01. Each test case consists of a few files: | |||
* description.txt to explain what this test case actually tests | * description.txt to explain what this test case actually tests | ||
Line 216: | Line 64: | ||
* testparams.sh if there are any non-default test parameters | * testparams.sh if there are any non-default test parameters | ||
Once the test run has completed, you will see an OUTPUT/ directory in the test case directory: | |||
Once the | |||
<pre> | <pre> | ||
Line 240: | Line 80: | ||
* Any core dumps generated if a pluto daemon crashed | * Any core dumps generated if a pluto daemon crashed | ||
; testing/baseconfigs/ | |||
: configuration files installed on guest machines | |||
; testing/guestbin/ | |||
: shell scripts used by tests, and run on the guest | |||
; testing/linux-system-roles.vpn/ | |||
: ??? | |||
; testing/packaging/ | |||
: ??? | |||
; testing/pluto/TESTLIST | |||
: list of tests, and their expected outcome | |||
; testing/pluto/*/ | |||
: individual test directories | |||
; testing/programs/ | |||
: executables used by tests, and run on the guest | |||
; testing/sanitizers/ | |||
: filters for cleaning up the test output | |||
; testing/utils/ | |||
: test drivers and other host tools | |||
; testing/x509/ | |||
: certificates, scripts are run on a guest | |||
=== | == Network Diagram == | ||
* interface-0 (eth0, vio0, vioif0) is connected to SWANDEFAULT which has a NAT gateway to the internet | |||
** the exceptions are the Linux test domains: EAST, WEST, ROAD, NORTH; should they? | |||
** the BSD domains always up inteface-0 so that /pool, /source, and /testing can be NFS mounted | |||
** NIC needs to run DHCP on eth0 manually; how? | |||
** transmogrify does not try to modify interface-0(SWANDEFAULT) (doing so would break established network sessions such as NFS) | |||
* the interface names do not have consistent order (see comment above about Fedora's interface-0 not pointing at SWANDEFAULT) | |||
** Fedora has ethN | |||
** OpenBSD has vioN (different order) | |||
** NetBSD has vioifN (different order) | |||
LEFT RIGHT | |||
192.0.3.0/24 -------------------------------------+-- 2001:db8:0:3::/64 (198.18.33) | |||
| | |||
2001:db8:0:3::254 | |||
192.0.3.254(eth0) | |||
ROAD NORTH | |||
192.1.3.209(eth0) 192.1.3.33(eth1) | |||
2001:db8:1:3::209 2001:db8:1:3::33 | |||
| | | |||
192.1.3.0/24 -----+----------------+--------------+-- 2001:db8:1:3::/64 (198.18.3 | |||
| | |||
2001:db8:1:3::254 | |||
192.1.3.254(eth2) | |||
NIC---swandefault(0) | |||
192.1.2.254(eth1) | |||
2001:db8:1:2::254 | |||
| | |||
192.1.2.0/24 -----------------+----+----------------------+----- 2001:db8:1:2::/64 (198.18.2) | |||
| | | |||
2001:db8:1:2::45 2001:db8:1:2::23 | |||
(eth1)192.1.2.45 (eth1)192.1.2.23 | |||
WEST EAST | |||
(eth0)192.0.1.254 (eth0)192.0.2.254 | |||
2001:db8:0:1::254 2001:db8:0:2::254 | |||
| | | |||
| | | |||
| | | |||
| TEST-NET-1 | |||
| 192.0.2.0/24 -------+---+-- 2001:db8:0:2::/64 (198.18.23) | |||
| | | |||
| 2001:db8:0:2::12/64 | |||
| 192.0.2.12/24(if1) | |||
| ${OS}RISE--198.18.12.12/24 | |||
| 2001:db8:1::12/64 | |||
| 198.18.1.12/24(if1) | |||
| | | |||
192.0.1.0/24 -------------+---+------ 2001:db8:0:1::/64 | (198.18.45) | |||
| | | |||
001:db8:0:1::15/64 | | |||
192.0.1.15/24(if1) | | |||
192.18.15.15/24--${OS}SET | | |||
2001:db8:1::15/64 | | |||
198.18.1.15/24(if1) | | |||
| | | |||
198.18.1.0/24 ------------+-----------------------------------+----- 2001:db8:1::/64 | |||
== Problems with the existing Network == | |||
The current network has a number of limitations. This section identifies those problems and proposes changes to address them: | |||
* the gateway had only 128 DHCP addresses | |||
* public networks are being used: 192.0.1.0/24 is owned by elevatedcomputing.com; 192.1.2.0/24 is owned by raytheon.com; 192.1.3.0/24 is owned by raytheon.com | |||
* 192.168.234.0/24 (gateway) is reserved for private use networks and known to clash with toronto airport | |||
* using public interfaces means that they can't be used in documentation | |||
* can't test two hosts where each is behind a gateway VPN | |||
see | |||
Ref https://www.rfc-editor.org/rfc/rfc5737 https://www.rfc-editor.org/rfc/rfc3849 https://www.rfc-editor.org/rfc/rfc6890 | |||
The suggestion is: | |||
* use the benchmarking network 198.18.0.0/15 | |||
* reserve 198.19/16 for the gateway | |||
* revive [sun]RISE (behind EAST) and [sun]SET (behind WEST) | |||
* reserve a number N for each machine / network: EAST: 23; WEST: 45; RISE: 123; SET: 145; NORTH: 33; NIC: 254 | |||
* use 198.18.2N.N/24 for IPsec Interfaces | |||
* use 192.18.1N.1N/24 for RISE and SET | |||
LEFT RIGHT | |||
198.18.254.0/24 ----------------------------------+-------------- 2001:db8:254::/64 | |||
| | |||
2001:db8:254::254 | |||
198.18.254.254(eth0) | |||
ROAD NORTH | |||
198.18.3.209(eth0) 198.18.3.33(eth1) | |||
2001:db8:3::209 2001:db8:3::33 | |||
| | | |||
198.18.3.0/254 --------+----------------+---------+---------------- 2001:db8:3::/64 | |||
| | |||
2001:db8:3::254 | |||
198.18.3.254(eth2) | |||
NIC---swandefault(0) | |||
198.18.2.254(eth1) | |||
2001:db8:2::254 | |||
| | |||
| | |||
198.18.2.0/24 ----------------+---------+-----------------+-------- 2001:db8:2::/64 | |||
| | | |||
2001:db8:2::45 2001:db8:2::23 | |||
198.18.2.45(eth1) 198.18.2.23(eth1) | |||
192.18.45.45/24--WEST EAST--198.18.23.23/24 | |||
192.0.1.45(eth0/2) 192.0.2.23(eth0/2) | |||
001:db8:0:1::45 2001:db8:0:2::23 | |||
| | | |||
| | | |||
| TEST-NET-1 | |||
| 192.0.2.0/24 -------+---+-- 2001:db8:0:2::/64 | |||
| | | |||
| 2001:db8:0:2::12/64 | |||
| 192.0.2.12/24(if1) | |||
| ${OS}RISE--198.18.12.12/24 | |||
| 2001:db8:1::12/64 | |||
| 198.18.1.12/24(if1) | |||
| | | |||
192.0.1.0/24 -------------+---+------ 2001:db8:0:1::/64 | | |||
| | | |||
001:db8:0:1::15/64 | | |||
192.0.1.15/24(if1) | | |||
192.18.15.15/24--${OS}SET | | |||
2001:db8:1::15/64 | | |||
198.18.1.15/24(if1) | | |||
| | | |||
198.18.1.0/24 ------------+-----------------------------------+----- 2001:db8:1::/64 | |||
The changes are as follows: | |||
=== Hand Sketch of Current Network === | |||
[[File:networksketch.png]] | |||
=== Original Network Diagram === | |||
[[File:testnet.png]] | |||
Latest revision as of 15:43, 11 October 2024
Nightly Test Tesults
Libreswan's testsuite is run nightly. The results are published here, with the most recent result here. The tests are categorized as:
- good: these tests are expected to pass (unfortunately, some still have timing problems and occasionally fail)
- wip: these tests require further work, for instance the result may not be deterministic, or the bug they demonstrate hasn't yet been fixed
- skiptest: these tests require manual intervention to run
To run tests locally, read on.
Running tests
The libreswan tests, in testing/pluto, can be run using several different mechanisms:
Framework | Speed | Host OS | Guest OS | initsystem testing (systemd, rc.d, ...) | Post-mortem | Interop testing | Notes |
---|---|---|---|---|---|---|---|
KVM | slower | Fedora, Debian (BSD anyone?) |
Alpine, Fedora, FreeBSD, NetBSD, OpenBSD, Debian | yes | shutdown, core, leaks, refcnt, selinux | strongswan (Linux, FreeBSD), iked (OpenBSD), racoon (NetBSD), racoon2 (NetBSD) | gold standard ideal for BSD builds idea for testing custom kernels used by the Testing machine requires 9p (virtio anyone?) |
Namespaces | fast | linux | uses host's libreswan, kernel, and utilities | no | core, leaks | strongswan (linux)? | ideal for quick tests requires libreswan to be built/installed on the host requires all dependencies to be installed on the host test results sensitive differing kernel and utilities |
Docker | linux | uses host's kernel uses distro's utilities |
? | ? | ? | ideal for cross-linux builds (CentOS 6, 7, 8, Fedora 28 - rawhide, Debian, Ubuntu) sensitive to differing kernel and utilities |
How tests work
All the test cases involving VMs are located in the libreswan directory under testing/pluto/. The most basic test case is called basic-pluto-01. Each test case consists of a few files:
- description.txt to explain what this test case actually tests
- ipsec.conf files - for host west is called west.conf. This can also include configuration files for strongswan or racoon2 for interop testig
- ipsec.secret files - if non-default configurations are used. also uses the host syntax, eg west.secrets, east.secrets.
- An init.sh file for each VM that needs to start (eg westinit.sh, eastinit.sh, etc)
- One run.sh file for the host that is the initiator (eg westrun.sh)
- Known good (sanitized) output for each VM (eg west.console.txt, east.console.txt)
- testparams.sh if there are any non-default test parameters
Once the test run has completed, you will see an OUTPUT/ directory in the test case directory:
$ ls OUTPUT/ east.console.diff east.console.verbose.txt RESULT west.console.txt west.pluto.log east.console.txt east.pluto.log swan12.pcap west.console.diff west.console.verbose.txt
- RESULT is a text file (whose format is sure to change in the next few months) stating whether the test succeeded or failed.
- The diff files show the differences between this testrun and the last known good output.
- Each VM's serial (sanitized) console log (eg west.console.txt)
- Each VM's unsanitized verbose console output (eg west.console.verbose.txt)
- A network capture from the bridge device (eg swan12.pcap)
- Each VM's pluto log, created with plutodebug=all (eg west.pluto.log)
- Any core dumps generated if a pluto daemon crashed
- testing/baseconfigs/
- configuration files installed on guest machines
- testing/guestbin/
- shell scripts used by tests, and run on the guest
- testing/linux-system-roles.vpn/
- ???
- testing/packaging/
- ???
- testing/pluto/TESTLIST
- list of tests, and their expected outcome
- testing/pluto/*/
- individual test directories
- testing/programs/
- executables used by tests, and run on the guest
- testing/sanitizers/
- filters for cleaning up the test output
- testing/utils/
- test drivers and other host tools
- testing/x509/
- certificates, scripts are run on a guest
Network Diagram
- interface-0 (eth0, vio0, vioif0) is connected to SWANDEFAULT which has a NAT gateway to the internet
- the exceptions are the Linux test domains: EAST, WEST, ROAD, NORTH; should they?
- the BSD domains always up inteface-0 so that /pool, /source, and /testing can be NFS mounted
- NIC needs to run DHCP on eth0 manually; how?
- transmogrify does not try to modify interface-0(SWANDEFAULT) (doing so would break established network sessions such as NFS)
- the interface names do not have consistent order (see comment above about Fedora's interface-0 not pointing at SWANDEFAULT)
- Fedora has ethN
- OpenBSD has vioN (different order)
- NetBSD has vioifN (different order)
LEFT RIGHT 192.0.3.0/24 -------------------------------------+-- 2001:db8:0:3::/64 (198.18.33) | 2001:db8:0:3::254 192.0.3.254(eth0) ROAD NORTH 192.1.3.209(eth0) 192.1.3.33(eth1) 2001:db8:1:3::209 2001:db8:1:3::33 | | 192.1.3.0/24 -----+----------------+--------------+-- 2001:db8:1:3::/64 (198.18.3 | 2001:db8:1:3::254 192.1.3.254(eth2) NIC---swandefault(0) 192.1.2.254(eth1) 2001:db8:1:2::254 | 192.1.2.0/24 -----------------+----+----------------------+----- 2001:db8:1:2::/64 (198.18.2) | | 2001:db8:1:2::45 2001:db8:1:2::23 (eth1)192.1.2.45 (eth1)192.1.2.23 WEST EAST (eth0)192.0.1.254 (eth0)192.0.2.254 2001:db8:0:1::254 2001:db8:0:2::254 | | | | | | | TEST-NET-1 | 192.0.2.0/24 -------+---+-- 2001:db8:0:2::/64 (198.18.23) | | | 2001:db8:0:2::12/64 | 192.0.2.12/24(if1) | ${OS}RISE--198.18.12.12/24 | 2001:db8:1::12/64 | 198.18.1.12/24(if1) | | 192.0.1.0/24 -------------+---+------ 2001:db8:0:1::/64 | (198.18.45) | | 001:db8:0:1::15/64 | 192.0.1.15/24(if1) | 192.18.15.15/24--${OS}SET | 2001:db8:1::15/64 | 198.18.1.15/24(if1) | | | 198.18.1.0/24 ------------+-----------------------------------+----- 2001:db8:1::/64
Problems with the existing Network
The current network has a number of limitations. This section identifies those problems and proposes changes to address them:
- the gateway had only 128 DHCP addresses
- public networks are being used: 192.0.1.0/24 is owned by elevatedcomputing.com; 192.1.2.0/24 is owned by raytheon.com; 192.1.3.0/24 is owned by raytheon.com
- 192.168.234.0/24 (gateway) is reserved for private use networks and known to clash with toronto airport
- using public interfaces means that they can't be used in documentation
- can't test two hosts where each is behind a gateway VPN
see Ref https://www.rfc-editor.org/rfc/rfc5737 https://www.rfc-editor.org/rfc/rfc3849 https://www.rfc-editor.org/rfc/rfc6890
The suggestion is:
- use the benchmarking network 198.18.0.0/15
- reserve 198.19/16 for the gateway
- revive [sun]RISE (behind EAST) and [sun]SET (behind WEST)
- reserve a number N for each machine / network: EAST: 23; WEST: 45; RISE: 123; SET: 145; NORTH: 33; NIC: 254
- use 198.18.2N.N/24 for IPsec Interfaces
- use 192.18.1N.1N/24 for RISE and SET
LEFT RIGHT 198.18.254.0/24 ----------------------------------+-------------- 2001:db8:254::/64 | 2001:db8:254::254 198.18.254.254(eth0) ROAD NORTH 198.18.3.209(eth0) 198.18.3.33(eth1) 2001:db8:3::209 2001:db8:3::33 | | 198.18.3.0/254 --------+----------------+---------+---------------- 2001:db8:3::/64 | 2001:db8:3::254 198.18.3.254(eth2) NIC---swandefault(0) 198.18.2.254(eth1) 2001:db8:2::254 | | 198.18.2.0/24 ----------------+---------+-----------------+-------- 2001:db8:2::/64 | | 2001:db8:2::45 2001:db8:2::23 198.18.2.45(eth1) 198.18.2.23(eth1) 192.18.45.45/24--WEST EAST--198.18.23.23/24 192.0.1.45(eth0/2) 192.0.2.23(eth0/2) 001:db8:0:1::45 2001:db8:0:2::23 | | | | | TEST-NET-1 | 192.0.2.0/24 -------+---+-- 2001:db8:0:2::/64 | | | 2001:db8:0:2::12/64 | 192.0.2.12/24(if1) | ${OS}RISE--198.18.12.12/24 | 2001:db8:1::12/64 | 198.18.1.12/24(if1) | | 192.0.1.0/24 -------------+---+------ 2001:db8:0:1::/64 | | | 001:db8:0:1::15/64 | 192.0.1.15/24(if1) | 192.18.15.15/24--${OS}SET | 2001:db8:1::15/64 | 198.18.1.15/24(if1) | | | 198.18.1.0/24 ------------+-----------------------------------+----- 2001:db8:1::/64
The changes are as follows: