Test Suite - KVM

KVM Test framework

Libreswan's test framework can be run using KVM guests, and the ./kvm script. It is strongly recommended to run the test suite on a host machine that has a CPU with virtualisation instructions.

To access files on the host file system:

• Fedora uses the PLAN9 filesystem (9p)
• Other guests (Alpine, Debian, FreeBSD, NetBSD, OpenBSD) use NFS via the NAT interface

For an overview of the network and testing see Test_Suite

Preparing the host machine

Check Virtualization is enabled in the BIOS

Virtualization needs to be enabled by the BIOS during boot.

 grep -e vmx -e svm /proc/cpuinfo

Add yourself to sudo

Some of the test scrips need to be run as root. The test environment assumes this can be done using sudo without a password vis:

sudo pwd

XXX: Surely qemu can be driven without root?

This is setup by adding an entry under /etc/sudoers.d/ specifying that your account does not need a password to become root:

echo "$(id -u -n) ALL=(ALL) NOPASSWD: ALL" | sudo dd of=/etc/sudoers.d/$(id -u -n)

Fight SELinux

SELinux blocks some actions that we need. We have not created any SELinux rules to avoid this. To check the current settings:

 getenforce

The options are:

• set SELinux to permissive (recommended)

sudo sed --in-place=.ORIG -e 's/^SELINUX=.*/SELINUX=permissive/' /etc/selinux/config
sudo setenforce Permissive

• disable SELinux

sudo sed --in-place=.ORIG -e 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config
sudo reboot

• (experimental) label source tree for SELinux

The source tree on the host is shared with the virtual machines. SELinux considers this a bug unless the tree is labelled with type svirt_image_t.

sudo dnf install policycoreutils-python-utils
sudo semanage fcontext -a -t svirt_image_t "$(pwd)"'(/.*)?'
sudo restorecon -vR /home/build/libreswan

There may be other things that SELinux objects to.

Check that the host has enough entropy

As a rough guide run:

while true ; do cat /proc/sys/kernel/random/entropy_avail ; sleep 3 ; done

it should have values in the hundrets if not thousands. If it is in the units or tens then see Entropy matters

Install Dependencies

Why	Fedora	Mint (debian)
Basics	sudo dnf install -y make git gitk patch xmlto python3-pexpect curl tar	sudo apt-get install -y make make-doc git gitk xmlto python3-pexpect curl tar
Virtualization	sudo dnf install -y qemu virt-install libvirt-daemon-kvm libvirt-daemon-qemu	sudo apt install -y qemu virtinst libvirt-clients libvirt-daemon libvirt-daemon-system libvirt-daemon-driver-qemu libosinfo-query qemu-system-x86?
Build BSD Boot CDs	sudo dnf install -y dvd+rw-tools	sudo apt-get install -y dvd+rw-tools
Build Web Pages	sudo dnf install -y jq typescript	sudo apt-get install -y jq node-typescript
Serve Web Server (optional)	sudo dnf install -y httpd	sudo apt-get install -y ????
NFS	sudo dnf install -y nfs-utils # ???	sudo apt-get install -y nfs-kernel-server rpcbind
Broken makefiles	sudo dnf install -y nss-devel # make file invokes pkg-config nss

Enable libvirt

If you're switching from the old libvirtd see https://libvirt.org/daemons.html#switching-to-modular-daemons for how to shut down the old daemons.

Start the "collection of modular daemons that replace functionality previously provided by the monolithic libvirtd daemon":

for drv in qemu network nodedev nwfilter secret storage interface
do
   sudo systemctl unmask virt${drv}d.service
   sudo systemctl unmask virt${drv}d{,-ro,-admin}.socket
   sudo systemctl enable virt${drv}d.service
   sudo systemctl enable virt${drv}d{,-ro,-admin}.socket
done
for drv in qemu network nodedev nwfilter secret storage
do
   sudo systemctl start virt${drv}d{,-ro,-admin}.socket
done

There should be no errors and warnings.

Stop libvirt daemons shutting down

By default the libvirt daemons timeout and shutdown after 120 seconds (surely systemd will restart them!). It turns out this hasn't worked so well:

• | libvirt clients hang because virtnetworkd.service misses when virtnetworkd is dead

systemd doesn't restart the daemon

• error: Disconnected from qemu:///system due to keepalive timeout

the restart is painfully slow with lots of networks which causes the timeout

• | libvirtd deadlocks (fixed)
• | virtqemud gets slower and slower

Disabling the timeout and just leaving the daemons running seems to help. Add the following:

echo VIRTNETWORKD_ARGS= | sudo dd of=/etc/sysconfig/virtnetworkd
echo VIRTQEMUD_ARGS=    | sudo dd of=/etc/sysconfig/virtqemud
echo VIRTSTORAGED_ARGS= | sudo dd of=/etc/sysconfig/virtstoraged

the standard libvirt systemd config files read these settings using EnvironmentFile=

Add yourself to the KVM/QEMU group

You need to add yourself to the group that QEMU/KVM uses when writing to /var/lib/libvirt/qemu. On Fedora it is 'qemu', and on Debian it is 'kvm'. Something like:

sudo usermod -a -G $(stat --format %G /var/lib/libvirt/qemu) $(id -u -n)

After this you will will need to re-login (or run sudo su - $(id -u -n)

Make certain that root can access the build

The path to your build needs to be accessible (executable) by root, assuming things are under home:

chmod a+x $HOME

Fix /var/lib/libvirt/qemu

sudo chmod g+w /var/lib/libvirt/qemu

Arguably we should run libvirt as a normal user instead.

Enable Tab Completion of ./kvm

If this:

complete -o filenames -C './kvm' ./kvm

is added to .bashrc then tab completion with ./kvm will include both commands and directories.

Set up a Web Server (optional)

If the machine is to run nightly test runs then it can be set up as a web server. See the nightly test results for an example.

See above for dependencies. See below for how to configure libreswan.

To set up the server:

sudo mkdir /var/www/html/results/
sudo chown $(id -un) /var/www/html/results/
sudo chmod 755 /var/www/html/results/
sudo sh -c 'echo "AddType text/plain .diff" >/etc/httpd/conf.d/diff.conf'

to run the web server until the next reboot:

sudo firewall-cmd --add-service=http
sudo systemctl start httpd

to make the web server permanent:

sudo systemctl enable httpd
sudo firewall-cmd --add-service=http --permanent

If you want it to be the main page of the website, you can create the file /var/www/html/index.html containing:

cat <<EOF

 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
 <html>
   <head>
     <meta http-equiv="REFRESH" content="0;url=/results/">
  </head>
   <BODY>
  </BODY>
 </HTML>
 

EOF

Override python?

On Debian slack based systems (i.e., Linux Mint 20.3), the default python is too old. Fortunately python 3.9 is also available vis:

sudo apt-get install python3.9

in addition, the make variable KVM_PYTHON will need to be added to Makefile.inc.local:

echo KVM_PYTHON=python3.9 >> Makefile.inc.local

BSD

Anyone?

Download and configure libreswan

Fetch Libreswan

The libreswan source tree includes all the components that are used on the host and inside the test VMs. To get the latest source code using git:

git clone https://github.com/libreswan/libreswan
cd libreswan

Developers can use Makefile.inc.local to override default build setttings. Create the file:

touch Makefile.inc.local

(packaging systems should not use this, and instead explicitly pass the make variables to the make command)

Create $(KVM_POOLDIR) for storing VM disk images

The pool directory is used used to store:

• VM disk images
• install CD/DVD images
• downloaded packages installed into the VMs
• other files

and can get quite large. It can and should be shared between build trees (this reflects libvirt which has a single name space for domains). $(KVM_PREFIX) (see further down) addresses the lack of name spaces.

By default $(top_srcdir)/../pool (../pool) is used (that is, adjacent to your source tree). It will need to be created.

Alternatively the shared pool directory can be specified explicitly by setting the make variable KVM_POOLDIR in Makefile.inc.local vis:

mkdir KVM_POOLDIR=/home/libreswan/pool
echo KVM_POOLDIR=/home/libreswan/pool >> Makefile.inc.local

Configure $(KVM_LOCALDIR) to store test domain disks in /tmp/pool (tmpfs) (optional)

By default, all disk mages are stored in $(KVM_POOLDIR) (see above). Since the test VM disk images do not need long-term storage (i.e., survive a reboot), $(KVM_LOCALDIR) can be used to specify that test VM disk images are stored in /tmp vis:

echo KVM_LOCALDIR=/tmp/pool >> Makefile.inc.local

This has the advantage of eliminating physical disk I/O as a bottle neck when accessing VM disk images; but the disadvantage of needing to re-build the test disk images after a reboot.

Note: now that the domains are 100% transient this may have zero benefit.

Configure $(KVM_PREFIX) to allow allow multiple build trees on a machine (optional)

By default the domains and networks are assigned names such as linux, east, 198_18_1, et.al.. The problem is that these names are not unique between build trees, and as a result, all build trees try to use the same domains and networks.

The "fix" is to define $(KVM_PREFIX) in Makefile.inc.local, giving it a different value in each build tree. For instance:

$ cat libreswan-a/Makefile.inc.local
KVM_PREFIX=a.
$ cat libreswan-b/Makefile.inc.local
KVM_PREFIX=b.

will use names such as a.linux et.al. in the first tree and b.linux et.al. in the second tree.

For convenience, commands such as:

libreswan-a$ ./kvm sh linux

will log into the current build tree's domain (here a.linux).

Note: due to limitations in the network stack (interfaces have a limit of 16 characters) (the prefix needs to be short).

Configure $(KVM_WORKERS) to run things in parallel (Optional)

By default all operations (building and testing) is serialized (even the VMs are given only one CPU!). If the host has plenty of cores then the parallelism can be increased using $(KVM_WORKERS). It does the following:

- assigns $(KVM_WORKERS) CPUs to the build VMs - runs make -j $(KVM_WORKERS) when building and installing libreswan - runs $(KVM_WORKERS) tests in parallel

To make running tests in parallel possible $(KVM_PREFIX) and the numbers 1..$(KVM_WORKERS) are combined to generate unique domain and network names. For instance, with:

KVM_PREFIX=a.
KVM_WORKERS=3

the prefixes a., a2, a3 are used generating the names a.east, a2east, a3east, et.al.

Note: $(KVM_WORKERS) is ignored when $(KVM_PREFIX) is not set. This might be a bug.

Generate a web page of the test results (optional)

See the nightly test results for an example and how to set up a web server so results can be viewed remotely.

To initially create the web directory RESULTS/ and populate it with the current test results use:

make web

Further test runs will update the RESULTS/ directory. The files can the be viewed using http://file.

To disable web page generation, delete the directory RESULTS/.

To instead publish the results on the web, point $(WEB_SUMMARYDIR) at the web directory:

$ WEB_SUMMARYDIR=/var/www/html/results >> Makefile.inc.local

Running the testsuite

The testsuite is driven using the top-level script ./kvm

For the impatient: ./kvm install check

To build the VMs, and build and install (or update) libreswan, and then run the tests, use:

./kvm install check

Running the testsuite

./kvm install

update the KVMs ready for a new test run

./kvm check

run the testsuite, previous results are saved in BACKUP/-date-

./kvm recheck

run the testsuite, but skip tests that already passed

./kvm results

list the results from the test run

./kvm diffs

display differences between the test results and the expected results, exit non-zero if there are any

./kvm test-clean

delete the current test results

the operations can be combined on a single line:

./kvm test-clean install check recheck diff

and individual tests can be selected (see Running a Single Test, below):

./kvm install check diff testing/pluto/*ikev2*

To stop ./kvm use control-c or ./kvm kill from another terminal.

Updating Certificates

The full testsuite requires a number of certificates. If not present, then ./kvm check will automatically generate them using the domain linux. Just note that the certificates have a limited lifetime. Should the test system detects out-of-date certificates then ./kvm check will barf.

To rebuild the certificates:

./kvm keys

can be used to force the generation of new certificates.

Maintaining (rebuilding and updating) the Domains

In normal operation, the only domains of interest are:

build domains (linux, netbsd, ...)

./kvm install uses these for incremental builds

to force a scratch build run ./kvm uninstall

test domains (east, west, ...)

./kvm install always rebuilds these

since these domains are transient, they disappear after a reboot

And to clean up everything:

./kvm clean

Finally, to upgrade the domains:

./kvm upgrade

Per above, these can be combined:

./kvm test-clean install check
./kvm upgrade install check

Internally, additional domains are created.

The table below lists all the domains and how to manipulate them. There's no need to delete a domain before rebuilding it. For instance:

./kvm test-clean upgrade install check

is equivalent to:

./kvm test-clean
./kvm downgrade
./kvm upgrade
./kvm transmogrify
./kvm install
./kvm check

There are two variants of each command. The first creates all the domains, the second only creates the specified domain.

step	new domain	create	cloned from	mounts	networks	delete delete	notes
base	linux-base	./kvm base ./kvm base-linux	ISOs	/pool /bench	gateway	./kvm purge./kvm demolish	installs the bare minimum needed to get a domain on the network root's account is hacked so that exit codes appear in the prompt demolish also deletes the gateway
upgrade	linux-upgrade	./kvm upgrade ./kvm upgrade-linux	linux-base	/pool /bench	gateway	./kvm downgrade	installs and/or upgrades all packages needed to build and test libreswan using a local cache
transmogrify	linux	./kvm transmogrify ./kvm transmogrify-linux	linux-upgrade	/pool /bench /source /testing	gateway	./kvm uninstall ./kvm clean	transmogrify the domain adding configuration and other files needed to build and test if necessary install custom kernels and/or save kernels for direct boot
install	east et.al.	./kvm install ./kvm install-linux	linux	/source /testing	test networks possibly gateway	./kvm uninstall ./kvm clean	install linux and then clone creating test domains

Mount Points

In normal operation, the only mount points of interest within a domain are /source and /testing. These are configured to point at the current source tree.

Internally, the following additional mount points are used:

mount	variable	default	use when ...	notes
/testing	$(KVM_TESTDIR)	libreswan/testing	running tests	the tests to run
/source	$(KVM_SOURCEDIR)	libreswan/	during install	the source code to build and install
/bench	$(KVM_SOURCEDIR)	libreswan/	building VMs	the scripts driving the tests
/pool	$(KVM_POOLDIR)	pool/	building VMs	KVMs and caches

It is possible, although unusual, to point these at different source trees. For instance: testing.libreswan uses benchdir (/bench) for the scripts, and rutdir (/source, /testing) for the directory being tested; when testing old code /source can be pointed at an alternative directory that contains the sources that are to be built and tested.

Shell and Console Access (Logging In)

There are several different ways to gain shell access to the domains.

Each method, depending on the situation, has both advantages and disadvantages. For instance:

• while make kvmsh-host provide quick access to the console, it doesn't support file copy
• while SSH takes more to set up, it supports things like proper terminal configuration and file copy

Serial Console access using ./kvm sh HOST (kvmsh.py)

./kvm sh HOST is a wrapper around "virsh" that automatically handles things like booting the machine, logging in, and correctly configuring the terminal. It's big advantage is that it always works. For instance:

$ ./testing/utils/kvmsh.py east
[...]
Escape character is ^]
[root@east ~]# printenv TERM
xterm
[root@east ~]# stty -a
...; rows 52; columns 185; ... 
[root@east ~]#

The script "kvmsh.py" can also be used directly to invoke commands on a guest (this is how ./kvm install works):

$ ./testing/utils/kvmsh.py east ls
[root@east ~]# ls
anaconda-ks.cfg

When $(KVM_PREFIX) (and $(KVM_WORKERS)) is defined ./kvm sh east can be used to log into $(KVM_PREFIX)east.

Limitations:

• no file transfer but files can be accessed via /pool and /testing

Graphical Console access using virt-manager

"virt-manager", a gnome tool can be used to access individual domains.

While easy to use, it doesn't support cut/paste or mechanisms for copying files.

Shell access using SSH

While requiring more effort to set up, it provides full shell access to the domains.

Since you will be using ssh a lot to login to these machines, it is recommended to either put their names in /etc/hosts:

# /etc/hosts entries for libreswan test suite
192.1.2.45 west
192.1.2.23 east
192.0.3.254 north
192.1.3.209 road
192.1.2.254 nic

or add entries to .ssh/config such as:

Host west
       Hostname 192.1.2.45

If you wish to be able to ssh into all the VMs created without using a password, add your ssh public key to testing/baseconfigs/all/etc/ssh/authorized_keys. This file is installed as /root/.ssh/authorized_keys on all VMs

Using ssh becomes easier if you are running ssh-agent (you probably are) and your public key is known to the virtual machine. This command, run on the host, installs your public key on the root account of the guest machines west. This assumes that west is up (it might not be, but you can put this off until you actually need ssh, at which time the machine would need to be up anyway). Remember that the root password on each guest machine is "swan".

ssh-copy-id root@west

You can use ssh-copy for any VM. Unfortunately, the key is forgotten when the VM is restarted.

Limitations:

• this only works with the default east, et.al. (it does not work with $(KVM_PREFIX) and/or multiple test directories)

kvm workflows

(seeing as everyone has a "flow", why not kvm) here are some common workflows, the following commands are used:

./kvm modified

list the test directories that have been modified

./kvm baseline

compare test results against a baseline

./kvm patch

update the expected test results

./kvm add

git add the modified test results

./kvm status

show the status of the currently running testsuite

./kvm kill

kill the currently running testsuite

Running a single test

There are two ways to run an individual test:

1. the test to run can be specified on the command line:

   kvm check testing/pluto/basic-pluto-01

2. the test is implied when running kvm from a test directory:

   cd testing/pluto/basic-pluto-01

   ../../../kvm

   ../../../kvm diff

But there's a catch:

• in batch mode pluto is shutdown at the end of the test

this way additional post-mortem checks, such as for memory leaks and core dumps that rely on pluto being stopped, can be performed

• in single test mode the system is left running

this way it is possible to log in and look around the running system and attach a debugger to pluto before it is shutdown

To instead force post-mortem, add:

KVMRUNNER_FLAGS += --run-post-mortem

to Makefile.inc.local.

Working on individual tests

The modified command can be used to limit the test run to just tests with modified files (according to git):

./kvm modified install check diff

install libreswan and then run the testsuite against just the modified tests, display differences differences

./kvm modified recheck diff

re-run the modified tests that are failing, display differences

./kvm modified patch add

update the modified tests applying the latest output and add them to git

this workflow comes into its own, when updating tests en-mass using sed, for instance:

sed -i -e 's/PARENT_//' testing/pluto/*/*.console.txt
./kvm modified check

Controlling a test run remotely

Start the testsuite in the background:

./kvm nohup check

To determine if the testsuite is still running:

./kvm status

and to stop the running testsuite:

./kvm kill

Debugging inside the VM (pluto on east)

Terminal 1 - east: log into east, start pluto, and attach gdb

./kvm sh east
east# cd /testing/pluto/basic-pluto-01
east# sh -x ./eastinit.sh
east# gdb /usr/local/libexec/ipsec/pluto $(pidof pluto)
(gdb) c

If pluto isn't running then gdb will complain with: --p requires an argument

Terminal 2 - west: log into west, start pluto and the test

./kvm sh west
west# sh -x ./westinit.sh ; sh -x westrun.sh

When pluto crashes, gdb will show that and await commands. For example, the bt command will show a backtrace.

TODO:

• stop watchdog eventually killing pluto
• notes for west

Running a Custom Kernel

Custom NetBSD Kernel

Build the kernel per upstream documentation and then copy it to:

$(KVM_POOLDIR)/$(KVM_PREFIX)netbsd-kernel

During transmogrify the stock kernel will be replaced with the above.

Custom Linux Kernel

The linux domains (east, west, et.al.) test domains boot the kernel directly using:

$(KVM_POOLDIR)/$(KVM_PREFIX)linux-upgrade.vmlinuz
$(KVM_POOLDIR)/$(KVM_PREFIX)linux-upgrade.initramfs

These files are re-created whenever upgrade is run. To boot a different kernel, replace the above (or edit the corresponding east.xml et.al. file with the new location).

Building and testing an old branch

Old branches have two problems:

• the KVM codebase is out-of-date
• the OS releases are gone

Here are two ways to get around it:

Using a test-bench

This workflow works best when working on an old branch (lets say v4.11)

Two repositories are used:

1. repo under test aka RUTDIR

   this contains both the sources and the tests

2. testbench

   this contains the test scripts used to drive ${RUT}

Start by checking out the two repositories (existing repositories can also be used, carefully):

RUTDIR=$PWD/v4_maint ; export RUTDIR
git clone https://github.com/libreswan/libreswan.git -r v4_maint ${RUTDIR}
git clone https://github.com/libreswan/libreswan.git testbench

Next, configure testbench so that it compiles, installs, and runs tests from ${RUTDIR} by setting the $(KVM_RUTDIR) make variable:

echo KVM_RUTDIR=$(realpath $RUTDIR)           >> testbench/Makefile.inc.local

($(KVM_SOURCEDIR) and $(KVM_TESTINGDIR) default to $(KVM_RUTDIR); you can also set $(KVM_SOURCEDIR) and $(KVM_TESTINGDIR) explicitly).

Now, (re-)transmogrify the testbench so that, within the domains, /source points at ${RUT} and /testing points at ${RUT}/testing:

./testbench/kvm transmogrify

in the command building the fedora domain look for output like:

--filesystem=target=bench,type=mount,accessmode=squash,source=/.../testbench \
--filesystem=target=source,type=mount,accessmode=squash,source=${RUTDIR} \
--filesystem=target=testing,type=mount,accessmode=squash,source=${RUTDIR}/testing \

Finally install and then run a test:

./testbench/kvm install check diff $RUT/testing/pluto/basic-pluto-01

If you prefer you can run testbench/kvm:

• from the testbench directory as ./kvm
• from the ${RUTDIR} directory as ../testbench/kvm

just do not run $RUTDIR/kvm.

Reviving the dead OS

Again looking at v4_maint branch. Check it out:

 git checkout ... -b v4_maint

add the following to Makefile.inc.local:

 KVM_PREFIX=v4
 KVM_FEDORA_ISO_URL = https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/35/Server/x86_64/iso/Fedora-Server-dvd-x86_64-35-1.2.iso

build fedora-base:

 ./kvm base-fedora

login to the base domain:

 ./kvm sh fedora-base

and edit the repos per:

 /etc/yum.repos.d/fedora.repo:name=Fedora $releasever - $basearch
 /etc/yum.repos.d/fedora.repo:baseurl=https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/35/Everything/x86_64/os
 /etc/yum.repos.d/fedora.repo:name=Fedora $releasever - $basearch - Debug
 /etc/yum.repos.d/fedora.repo:baseurl=https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/35/Everything/x86_64/debug/tree/
 /etc/yum.repos.d/fedora.repo:name=Fedora $releasever - Source
 /etc/yum.repos.d/fedora.repo:baseurl=https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/35/Everything/source/tree/
 /etc/yum.repos.d/fedora-updates.repo:name=Fedora $releasever - $basearch - Updates
 /etc/yum.repos.d/fedora-updates.repo:baseurl=https://archives.fedoraproject.org/pub/archive/fedora/linux/updates/35/Everything/x86_64/
 /etc/yum.repos.d/fedora-updates.repo:name=Fedora $releasever - $basearch - Updates - Debug
 /etc/yum.repos.d/fedora-updates.repo:baseurl=https://archives.fedoraproject.org/pub/archive/fedora/linux/updates/35/Everything/x86_64/debug/
 /etc/yum.repos.d/fedora-updates.repo:name=Fedora $releasever - Updates Source
 /etc/yum.repos.d/fedora-updates.repo:baseurl=https://archives.fedoraproject.org/pub/archive/fedora/linux/updates/35/Everything/source/tree/

after that:

 ./kvm install check

might work

Tracking down regressions (using git bisect)

The easy way

This workflow works best when the regression is recent (i.e., the last few commits) and nothing significant has happened in the meantime (for instance, os upgrade, test rename, ...).

The command ./kvm install check diff exits with a git bisect friendly status codes which means it can be combined with git bisect run to automate regression testing.

For instance:

git bisect start main ^<suspect-commit>
git bisect run ./kvm install check diff testing/pluto/basic-pluto-01
git bisect visualize
# finally
git bisect reset

The hard way

This workflow works best when trying to track down a regression in an older version of libreswan.

Two repositories are used:

1. repo-under-test

   this contains the sources that will be built and installed into the test domains and is what git bisect will manipulate

2. testbench

   this contains the test scripts used to drive repo-under-test

Start by checking out the two repositories (existing repositories can also be used, carefully):

git clone https://github.com/libreswan/libreswan.git repo-under-test
git clone https://github.com/libreswan/libreswan.git testbench

and then cd to the repo-under-test directory:

 cd repo-under-test

Next, configure testbench so that it compiles and installs libreswan from repo-under-test but runs tests from testbench. Do this by pointing the testbench KVM_SOURCEDIR (/source) at repo-under-test vis:

# remember $PWD is repo-under-test
echo KVM_SOURCEDIR=$(realpath ../repo-under-test)    >>../testbench/Makefile.inc.local
echo KVM_TESTINGDIR=$(realpath ../testbench/testing) >>../testbench/Makefile.inc.local

Now, (re-)transmogrify the testbench so that, within the domains, /source points at repo-under-test:

../testbench/kvm transmogrify

in the command building the fedora domain look for output like:

--filesystem=target=bench,type=mount,accessmode=squash,source=/.../testbench \
--filesystem=target=source,type=mount,accessmode=squash,source=/.../repo-under-test \
--filesystem=target=testing,type=mount,accessmode=squash,source=/.../testbench/testing \

Finally run the tests (remember testing/pluto/basic-pluto-01 is the test that started failing):

# start with the bad commit
git bisect start main
# next checkout and confirm the good commit
# NOTE: run testbench/kvm from repo-under-test directory
git checkout <good-commit>
../testbench/kvm install check diff testing/pluto/basic-pluto-01
git bisect good

if you're lucky, the test requires no manual intervention and:

git bisect run ../testbench/kvm install check diff testing/pluto/basic-pluto-01

also works:

# finally
git bisect visualize
git bisect reset

TODO: figure out how to get ../testbench/kvm diff to honour KVM_TESTINGDIR so that it can handle a test somewhere other than in <tt?testbench