Test Suite - 2017 NG

Requirements to extend testing as of 2017 Spring

Scripts that run the test on the guest/instance

000-<host name>.sh Each host that is part of this test need one of these. This could run in parallel on each host.
nnn-<host name>.sh will be executed serially on the host.
nnn-<host name>-debug.sh will be executed in serial. However the output or the commands will not be part <host>.console.txt. The output will go OUTPUT/<host name>.verbose.txt

Installing libreswan

The testing could install libreswan in various ways. Each would support a specific use case.

Install with "make install-base" =

This is the most simple one. It will be useful for automated test runs ad developers to quickly test something.

Install using rpm =

This is more desirable for archived test result. This way if you notice an issue share the rpm. The console output sanitizers are capable of distinguishing run from /usr/local/libexec/ipsec or rpm installed location.

If is useful to test interoperatbility with older versions. In such case you may want to able to specify rpm per host. So one or more host, say east and west would run, with different rpm.

Some tests support its own Makefile.inc.local file

Then there is are odd tests which need specific compile time options set or unset. For example DNSEC defined or not, AUDIT is enabled or not. One solution that comes to mind is add a <host>.Makefile.inc.local in the directory. If you see that then rebuild on this specific host. However after the test run you may have rebuild the whole vm from clone image to reset.

Possibly this could be hard to combine with rpms.

One way this could be supported is specify a custom rpm package?