Test Suite - KVM
Libreswan comes with an extensive test suite, written mostly in python, that uses KVM virtual machines and virtual networks. It has replaced the old UML test suite. Apart from KVM, the test suite uses libvirtd and qemu. It is strongly recommended to run the test suite natively on the OS (not in a VM itself) on a machine that has a CPU wth virtualization instructions. The PLAN9 filesystem (9p) is used to mount host directories in the guests - NFS is avoided to prevent network lockups when an IPsec test case would cripple the guest's networking.
|libvirt 0.9.11 and qemu 1.0 or better are required. RHEL does not support a writable 9p filesystem, so the recommended host/guest OS is Fedora 22|
- 1 Test Frameworks
- 2 Preparing the host machine
- 3 Download and configure libreswan
- 4 Set up KVM and run the Testsuite (for the impatient)
- 5 Running the testsuite
- 6 Shell and Console Access (Logging In)
- 7 Run an individual test (or tests)
- 8 Debugging inside the VM
- 9 Updating the VMs
- 10 Publishing Results on the web: http://testing.libreswan.org/results/
This page describes the make kvm framework.
Instead of using virtual machines, it is possible to use Docker instances.
More information is found in Test Suite - Docker in this Wiki
Preparing the host machine
In the following it is assumed that your account is called "build".
Add Yourself to sudo
Some of the test scrips need to be run as root. The test environment assumes this can be done using sudo without a password vis:
XXX: Surely qemu can be driven without root?
This is done by creating a no-pasword rule to /etc/sudoers.d/.
To set this up, add your account to the wheel group and permit wheel to have no-password access. Issue the following commands as root:
echo '%wheel ALL=(ALL) NOPASSWD: ALL' > /etc/sudoers.d/swantest chmod 0440 /etc/sudoers.d/swantest chown root.root /etc/sudoers.d/swantest usermod -a -G wheel build
SELinux blocks some actions that we need. We have not created any SELinux rules to avoid this. The options are:
- set SELinux to permissive (recommended)
sudo sed --in-place=.ORIG -e 's/^SELINUX=.*/SELINUX=permissive/' /etc/selinux/config sudo setenforce Permissive
- disable SELinux
sudo sed --in-place=.ORIG -e 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config sudo reboot
- (experimental) label source tree for SELinux
The source tree on the host is shared with the virtual machines. SELinux considers this a bug unless the tree is labelled with type svirt_image_t.
sudo dnf install policycoreutils-python-utils sudo semanage fcontext -a -t svirt_image_t "$(pwd)"'(/.*)?' sudo restorecon -vR /home/build/libreswan
There may be other things that SELinux objects to.
Install Required Dependencies
Now we are ready to install the various components of libvirtd, qemu and kvm and then start the libvirtd service.
= Fedora Install
While virt-manager isn't strictly required it's useful on a desktop:
sudo dnf install -y qemu virt-manager virt-install libvirt-daemon-kvm libvirt-daemon-qemu sudo dnf install -y python3-pexpect sudo dnf install -y make jq
Once all is installed start libvirtd and then check it is running:
sudo systemctl enable libvirtd sudo systemctl start libvirtd sudo systemctl status libvirtd
There should be no errors and warnings.
On testing and F29, this failed with the error:
error : virQEMUCapsNewForBinaryInternal:4664 : internal error: Failed to probe QEMU binary with QMP: /usr/bin/qemu-system-xtensa: error while loading shared libraries: libbrlapi.so.0.6: cannot open shared object file: No such file or directory
and it was found that 'brlapi' needed to be manually installed.
Install Utilities (Optional)
Various tools are used or convenient to have when running tests:
Optional packages to install on Fedora
sudo dnf -y install git patch tcpdump expect python-setproctitle python-ujson pyOpenSSL python3-pyOpenSSL sudo dnf install -y python2-pexpect python3-setproctitle diffstat
Optional packages to install on Ubuntu
apt-get install python-pexpect git tcpdump expect python-setproctitle python-ujson \ python3-pexpect python3-setproctitle
|do not install strongswan-libipsec because you won't be able to run non-NAT strongswan tests!|
Setting Users and Groups
You need to add yourself to the qemu group. For instance:
sudo usermod -a -G qemu $(id -u -n)
You will need to re-login for this to take effect.
The path to your build needs to be accessible (executable) by root:
chmod a+x ~
|Because our VMs don't run as qemu, /var/lib/libvirt/qemu needs to be changed using chmod g+w to make it writable for the qemu group. This needs to be repeated if the libvirtd package is updated on the system|
sudo chmod g+w /var/lib/libvirt/qemu
Several virtio modules need to be loaded into the host's kernel. This could be done by modprobe ahead of running any virtual machines but it is easier to install them whenever the host boots. This is arranged by listing the modules in a file within /etc/modules-load.d. The host must be rebooted for this to take effect.
sudo dd <<EOF of=/etc/modules-load.d/virtio.conf virtio_blk virtio-rng virtio_console virtio_net virtio_scsi virtio virtio_balloon virtio_input virtio_pci virtio_ring 9pnet_virtio EOF
As of Fedora 28, several of these modules are now built into the kernel and will not show up in /proc/modules (virtio, virtio_rng, virtio_pci, virtio_ring).
Ensure that the host has enough entropy
With KVM, a guest systems uses entropy from the host through the kernel module "virtio_rng" in the guest's kernel (set above). This has advantages:
- entropy only needs to be gathered on one machine (the host) rather than all machines (the host and the guests)
- the host is in the Real World and thus has more sources of real entropy
- any hacking to make entropy available need only be done on one machine
To ensure the host has enough randomness, run either jitterentropy-rngd or havegd.
Fedora commands for using jitterentropy-rngd (broken on F26, service file specifies /usr/local for path):
sudo dnf install jitterentropy-rngd sudo systemctl enable jitterentropy-rngd sudo systemctl start jitterentropy-rngd
Fedora commands for using havegd:
sudo dnf install haveged sudo systemctl enable haveged sudo systemctl start haveged
Download and configure libreswan
The libreswan source tree includes all the components that are used on the host and inside the test VMs. To get the latest source code using git:
git clone https://github.com/libreswan/libreswan cd libreswan
Create the Pool directory - KVM_POOLDIR
The pool directory is used used to store KVM disk images and other configuration files. By default $(top_srcdir)/../pool is used (that is, adjacent to your source tree).
To change the location of the pool directory, set the KVM_POOLDIR make variable in Makefile.inc.local. For instance:
$ grep KVM_POOLDIR Makefile.inc.local KVM_POOLDIR=/home/libreswan/pool
Make tests run in parallel - KVM_PREFIXES
By default only one test is run at a time. This can be changed using KVM_PREFIXES make variable which specifies the prefix to prepend to test domains. The default value is:
which creates the domains east, west, et.al. (i.e., after expansion east, west, et.al.).
Multiple tests can be run in parallel by specifying more prefixes - a rule of thumb is one prefix per two CPU cores. For instance, on a 4-core machine, two prefixes can be specified using:
which creates, after expansion, the domains east, west, et.al. and 1.east, 1.west, et.al.
Serve test results as HTML pages on the test server (optional)
If you want to be able to see the results of testruns in HTML, you can enable a webserver:
dnf install httpd systemctl enable httpd systemctl start httpd mkdir /var/www/html/results/ chown build /var/www/html/results/ chmod 755 /var/www/html/results/ cd ~ ln -s /var/www/html/results
If you want it to be the main page of the website, you can create the file /var/www/html/index.html containing:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> <html> <head> <meta http-equiv="REFRESH" content="0;url=/results/"></HEAD> <BODY> </BODY> </HTML>
and then add:
Set up KVM and run the Testsuite (for the impatient)
If you're impatient, and want to just run the testsuite using kvm then:
- install (or update) libreswan (if needed this will create the test domains):
- make kvm-install
- run the testsuite:
- make kvm-test
- list the kvm make targets:
- make kvm-help
After that, the following make targets are useful:
- clean the kvm build tree
- make kvm-clean
- clean the kvm build tree and all kvm domains
- make kvm-purge
Running the testsuite
The full testsuite requires a number of certificates. The virtual domains are configured for this purpose. Just use:
( Before pyOpenSSL version 0.15 you couldn't run dist_certs.py without a patch to support creating SHA1 CRLs. A patch for this can be found at https://github.com/pyca/pyopenssl/pull/161 )
Run the testsuite
To run all test cases (which include compiling and installing it on all vms, and non-VM based test cases), run:
make kvm-install kvm-test
Stopping pluto tests (gracefully)
If you used "make kvm-test", type control-C; possibly repeatedly.
Shell and Console Access (Logging In)
There are several different ways to gain shell access to the domains.
Each method, depending on the situation, has both advantages and disadvantages. For instance:
- while make kvmsh-host provide quick access to the console, it doesn't support file copy
- while SSH takes more to set up, it supports things like proper terminal configuration and file copy
Serial Console access using "make kvmsh-HOST" (kvmsh.py)
"kvmsh", is a wrapper around "virsh". It automatically handles things like booting the machine, logging in, and correctly configuring the terminal:
$ ./testing/utils/kvmsh.py east [...] Escape character is ^] [root@east ~]# printenv TERM xterm [root@east ~]# stty -a ...; rows 52; columns 185; ... [root@east ~]#
"kvmsh.py" can also be used to script remote commands (for instance, it is used to run "make" on the build domain):
$ ./testing/utils/kvmsh.py east ls [root@east ~]# ls anaconda-ks.cfg
Finally, "make kvmsh-HOST" provides a short cut for the above; and if your using multiple build trees (see further down), it will connect to the DOMAIN that corresponds to HOST. For instance, notice how the domain "a.east" is passed to kvmsh.py in the below:
$ make kvmsh-east /home/libreswan/pools/testing/utils/kvmsh.py --output ++compile-log.txt --chdir . a.east Escape character is ^] [root@east source]#
- no file transfer but files can be accessed via /testing
Graphical Console access using virt-manager
"virt-manager", a gnome tool can be used to access individual domains.
While easy to use, it doesn't support cut/paste or mechanisms for copying files.
Shell access using SSH
While requiring slightly more effort to set up, it provides full shell access to the domains.
Since you will be using ssh a lot to login to these machines, it is recommended to either put their names in /etc/hosts:
# /etc/hosts entries for libreswan test suite 220.127.116.11 west 18.104.22.168 east 22.214.171.124 north 126.96.36.199 road 188.8.131.52 nic
or add entries to .ssh/config such as:
Host west Hostname 184.108.40.206
If you wish to be able to ssh into all the VMs created without using a password, add your ssh public key to testing/baseconfigs/all/etc/ssh/authorized_keys. This file is installed as /root/.ssh/authorized_keys on all VMs
Using ssh becomes easier if you are running ssh-agent (you probably are) and your public key is known to the virtual machine. This command, run on the host, installs your public key on the root account of the guest machines west. This assumes that west is up (it might not be, but you can put this off until you actually need ssh, at which time the machine would need to be up anyway). Remember that the root password on each guest machine is "swan".
You can use ssh-copy for any VM. Unfortunately, the key is forgotten when the VM is restarted.
Run an individual test (or tests)
All the test cases involving VMs are located in the libreswan directory under testing/pluto/ . The most basic test case is called basic-pluto-01. Each test case consists of a few files:
- description.txt to explain what this test case actually tests
- ipsec.conf files - for host west is called west.conf. This can also include configuration files for strongswan or racoon2 for interop testig
- ipsec.secret files - if non-default configurations are used. also uses the host syntax, eg west.secrets, east.secrets.
- An init.sh file for each VM that needs to start (eg westinit.sh, eastinit.sh, etc)
- One run.sh file for the host that is the initiator (eg westrun.sh)
- Known good (sanitized) output for each VM (eg west.console.txt, east.console.txt)
- testparams.sh if there are any non-default test parameters
You can run this test case by issuing the following command on the host:
make kvm-test KVM_TESTS+=testing/pluto/basic-pluto-01/
multiple tests can be selected with:
make kvm-test KVM_TESTS+=testing/pluto/basic-pluto-*
Once the test run has completed, you will see an OUTPUT/ directory in the test case directory:
$ ls OUTPUT/ east.console.diff east.console.verbose.txt RESULT west.console.txt west.pluto.log east.console.txt east.pluto.log swan12.pcap west.console.diff west.console.verbose.txt
- RESULT is a text file (whose format is sure to change in the next few months) stating whether the test succeeded or failed.
- The diff files show the differences between this testrun and the last known good output.
- Each VM's serial (sanitized) console log (eg west.console.txt)
- Each VM's unsanitized verbose console output (eg west.console.verbose.txt)
- A network capture from the bridge device (eg swan12.pcap)
- Each VM's pluto log, created with plutodebug=all (eg west.pluto.log)
- Any core dumps generated if a pluto daemon crashed
Debugging inside the VM
Debugging pluto on east
Terminal 1 - east: log into east, start pluto, and attach gdb
make kvmsh-east east# cd /testing/pluto/basic-pluto-01 east# sh -x ./eastinit.sh east# gdb /usr/local/libexec/ipsec/pluto $(pidof pluto) (gdb) c
Terminal 2 - west: log into west, start pluto and the test
make kvmsh-west west# sh -x ./westinit.sh ; sh -x westrun.sh
If pluto wasn't running, gdb would complain:
--p requires an argument
When pluto crashes, gdb will show that and await commands. For example, the bt command will show a backtrace.
Debugging pluto on west
See above, but also use virt as a terminal.
If you want to get rid of the warning "warning: File "/testing/pluto/ikev2-dpd-01/.gdbinit" auto-loading has been declined by your `auto-load safe-path'"
echo "set auto-load safe-path /" >> /root/.gdbinit
Updating the VMs
- delete all the copies of the base VM:
- $ make kvm-purge
- install again
- $ make kvm-install
Publishing Results on the web: http://testing.libreswan.org/results/
This is experimental and uses:
Two scripts are available:
- sets up the directory ~/results adding any dependencies
- runs the testsuite and then copies the results to ~/results
To view this, use file:///.
To get this working with httpd (Apache web server):
sudo systemctl enable httpd sudo systemctl start httpd sudo ln -s ~/results /var/www/html/ sudo sh -c 'echo "AddType text/plain .diff" >/etc/httpd/conf.d/diff.conf'
To view the results, use http://localhost/results.