Test Suite - Namespace

This is a quick guide to run libreswan tests using namespaces.

• Note that the commands below install a lot of development packages. It is recommended to use a user that can use sudo without password prompting.
• Note the page currently describes testing on rpm based systems, such as Fedora, RHEL and CentOS. However, recent Debian/Ubuntu could also work for basic testing if you know how to install the dependencies. And the the exact output may vary slightly.

Prepare host for libreswan testing

The namespace based tests can be run on a real machine (server, laptop) or in a virtual machine (kvm,libvirt,qemu etc). Since namespace tests can run in parallel, having more CPU's will allow you to run more tests in parallel.

sudo without password

To run sudo without password, your user needs to be in the wheel group. Ensure you have the following line enabled in /etc/sudoers

# /etc/sudoers
%wheel 	ALL=(ALL)	NOPASSWD: ALL

Check if you can run sudo without password prompt:

sudo bash -c true

If this command asks for a password, check /etc/sudoers and fix it.

install test RPM dependencies

On fedora, no additional repositories are needed.

On CentOS/RHEL, you might need to enable the EPEL repository.

sudo dnf install epel-release

Run the following command to install all dependencies:

sudo make install-testing-rpm-dep

install libresswan to run tests in namespace

There are 2 ways to test a specific libreswan build.

• use a known released package (deb or rpm) from the official repository. Just install the regular libreswan package on the host.
• install a libreswan rpm or deb on the host based on the latest libreswan version or git code. Create the package using "make rpm" or "make deb" and then install it. Note that uncommited changes won't be part of an rpm build.
• install from source in /usr/local using "make" followed by "sudo make install". Ensure any rpm or deb package is removed before running make install.

Note: you cannot have both an rpm/deb and a /usr/local install on the same homst, as two installs would conflict. The testing infrastructure will detect this an error out. If you are developer who want to change code and re-compile "make install" could be fastest turnaround for changes. An alternative is commit the code in local git and create rpm using "make rpm" or "make kvm-rpm".

git clone https://github.com/libreswan/libreswan libreswan.git
cd libreswan.git

installing the libreswan rpm to test

When libreswan is installed from rpm and if you want gdb backtrace support in case a test run causes pluto.. to crash, also install the source/debuginfo libreswan rpms that belong to the binary rpm.

testing-rpm package or /usr/local install

First, clone the libreswan repository even if you are going to test a libreswan rpm. We will use this to get the latest testing infrastructure and the latest available tests.

installing the libreswan from source to test

If you are testing from a libreswan git directory, first ensure no rpm is installed, then install libreswan:

rpm -e libreswan
make clean
make base 
make install-base
# if your are running on a machine with SElinux enable, also run the following:
restorecon -Rv /usr/local/sbin/ipsec /usr/local/libexec/ipsec

To install with INITSYSTEM=docker

make nsinstall

# make changes to code and to remake

make nsreinstll

These are convenient when working on code and to quickly install new code for a quick test. Note that any changes to code that are not committed are still being used when running make install. As opposed to "make rpm" or "make kvm-rpm" build from committed code.

Creating custom libreswan rpm for testing

You can get the best of both worlds by using the libreswan.git directory to write code, and then create and install this code based via an rpm. Be sure to commit your changes as uncommitted will not be packaged into the rpm. You can use one commit and whenever making more changes, run "git commit -a --amend". Once your code is committed, run:

make rpm
# use the rpm version created by the above command
rpm -Uhv libreswan-3.31rc827_gc9aa82b8a6_master.x86_64.rpm

Advanced use: Creating custom libreswan rpm on Fedora for debugging

NOTE : this need a git repository. Any non committed code will be ignored.

• This rpm will install to /usr/local/ as opposed to standard location. It compiled with -g and no optimization, O2 or O3.
• The RPM build porcess will also take Makefile.inc.local any additional configs in there will be used.
• with with_cavstests 0
• version number is from "make showrpmversion"
• with_efence 1

Few extra options to help debug add the following to Makefile.inc.local

SD_RESTART_TYPE=no
SD_WATCHDOGSEC=0

make kvm-rpm
ls ~/rpmbuild/RPMS/x86_64/
rpm -Uhv ~/rpmbuild/RPMS/x86_64/libreswan-3.30_1691_gbbe8ccd4c1_main-1.fc30.x86_64.rpm
rpm -Uhv ~/rpmbuild/RPMS/x86_64/libreswan-debuginfo-3.30_1691_gbbe8ccd4c1_main-1.fc30.x86_64.rpm
<pre>

== Generating X.509 certificates and DNSSEC zones ==

Some tests use certificates or DNSSEC zones and these must be created once:

<pre>
./testing/x509/dist_certs.py
./testing/baseconfigs/all/etc/bind/generate-dnssec.sh

general setup of tests

TODO

Running a single test

All the tests can be found in the testing/pluto directory. Each test has its own directory. It will contain ipsec configuration files, shell scripts to direct the IPsec nodes being tested and the reference output (the "known good output"). Once a test is run, there will be an OUTPUT directory containing the full pluto logfiles, and the console output of each IPsec node (eg east.console.txt). Since there is some always changing output, such as dates, time, random numbers etc, each console output is "sanitized". The unsanitized output is available as east.console.verbose.txt

To run a single test, for example the basic-pluto-01 test, issue:

cd testing/pluto/basic-pluto-01
../../utils/nsrun --ns

You will be told if the test passed or failed, and all output files will be in the OUTPUT directory.

Note: There seems to be a bug that prevents us from creating the namespace occasionally. If you see a python error about creating a namespace, just ignore it and rerun the test. We hope to fix this issue in the near future.

Doing a partial or full test run

A list of available tests are located in the file testing/pluto/TESTLIST. Tests marked with "good" should pass. Tests marked with "wip" are Work In Progress. To run a (partial/full) test run, issue:

make nsrun

Note: we are working on extending support to all tests. Some of these are not complete yet.

Test results

When doing a full testrun, the test results can be published on a website. This is not yet working for namespace based testing. To see the libvirt/kvm based website results of full test runs, see [testing.libreswan.org testing.libreswan.org].

unsupported tests

As of 2019 December, several tests are hard to run in namespaces. Some can possibly be made to run, with varying amount of effort and motivation (patches welcome!). The KLIPS tests should be ignored as these use an alternative kernel stack that will be completely removed from libreswan in version 3.31.

• SELINUX testing (possible). It need more attention and work in theory. Though won't co-exist with SELinux enabled and disabled.

• audit tests (possible) : auditd and kernel messages are all going to the host audit log outside the namespace. It will require some work to filter these out from the regular host audit log.

• FIPS tests and non fips tests at the same time (impossible?)

We do not know yet if we can turn on (or fake) FIPS mode within a namespace to test the userland part of FIPS. Obviously, the kernel can only be in FIPS or non-FIPS mode and we cannot run tests where the nodes require to be in different FIPS modes.

• Ignored tests: "audit", "dnsoe", "fips", "ipseckey", "dnssec", "interop", "ocsp", "seccomp",

The tests with above name are ignored by nsrun. Some of these tests likely can run using namespaces. Some of these tests need extra software. We need tricks to run unbound, nsd, ocsp etc tests run. Some are working and for others -- Again, patches are welcome :)

• testing kernel netlink messages or kernel crash would be hard. tests that cause kernel crash, bug or warn that will break the testrun and hard to trace which namespace caused the error.

• xfrmi (possible) there seems some issues when the devices are not cleaned up. You would need to reboot after a test.

• testing libreswan code that involve systemd and/or NetworkManager (hard to impossible). This requires extended namespaces running a lot more than just libreswan, and the effort might not be worth it compared to just using the existing libvirt/kvm/qemu based testing for those tests.

• can not run xfrmi and systemd-networkd tests.

future ideas

• Tracking coredumps.

Currently when a process such as pluto, addcon or whack, carsh the core file will be in /tmp/ on the host. However in namespaces these coredumps are mixed up. The tmp is remounted? and it is hard to track core dump by east or west or of another test case. One idea is strictly track the pid. When we start whack we have follow the pid and track it to the core.

• testing with different libreswan version. In theory this should be easy however need more work. One idea is bind mount the "libexec/ipsec" directory on say east and west from different version, binaries of libreswan.
• Output changes due to parallelism, aka multiple CPU. In KVM testing it is assumed there is only one CPU, while in namespace testing pluto and other process are likely to get more than one CPU. This may bring up more corner cases of pluto, such as events firing up in out of order etc, forked process ending in out of order. This can cause output changes or expose bugs.
• also use pid namespace ??? then pidof pluto would work.
• use "ip netns identify" and match to testname to see if it is in the right namespace. This could be used in startup script too.
• better cleanup of previous run. If you use NSENTER and stay inside the namespace, re-running test won't able to clean up previous run. Is there a way to detect this situation?

Debugging a test

Sometimes it can be useful to have a shell within the namespace to do some manual commands, such as "ipsec status". You can enter the namespace of a node (east, west, road etc) of a specific test (eg basicpluto-01, ikev2-04-basic-x509) using this bash function in your .bashrc file:

NSENTER ()
{
    ns=$1;
    nsargs="--mount=/run/mountns/${ns} --net=/run/netns/${ns} --uts=/run/utsns/${ns}";
    NSENTER_CMD="/usr/bin/nsenter ${nsargs} ";
    sudo ${NSENTER_CMD} /bin/bash
}

to enter the east namespace of the completed test basic-pluto-01, run:

NSENTER east-basic-pluto-01

NOTE: exit the namespace before re-running the test. Otherwise next test run will not be able to cleanup. Previous run will consume resources. If you are in gdb pluto will not exit..

attaching gdb inside namespace

e.g

NSENTER east-basic-pluto-01 

will drop you into the namespace. Inside that name space

e.g (note the location on pluto)
gdb /usr/local/libexec/ipsec/pluto -p $(cat /run/pluto/pluto.pid)

or bash alias:
alias gdbp='gdb /usr/local/libexec/ipsec/pluto -p $(cat /run/pluto/pluto.pid)'

NOTE: "pidof pluto" may return several pids, including pluto running in other namesapces

note well when creating tests

• Do not create temp files in /tmp. /tmp is shared between testruns (sockets..). e.g tpcpdump pid file should be full name. avoid east.pid. It would be overwritten.

• Do not use pidof pluto in tests. or pidof tcpdump. pidof will return processes running in other namespaces too.

running namespace tests on Debian

https://lists.libreswan.org/pipermail/swan-dev/2020-June/003848.html