Test Suite - 2017 NG: Difference between revisions

From Libreswan
Jump to navigation Jump to search
No edit summary
No edit summary
 
Line 1: Line 1:
Requirements to extend testing, test suite and tests themslef as of Fall 2017 =
Requirements to extend testing, test suite and tests themslef as of Fall 2017 =


== Test detials ==
== Test specifications ==
Test run scripts in the test directory and support single domain(guest vm) test and multi-domain tests


==== single domain test ====
<pre>
<hostnamae>.sh


== Test suite requirements/mechanics ==
</pre>
=== Scripts that run the test on the guest/instance ===
==== multi domain test ====
<pre>
<pre>
000-<host name>.sh Each host that is part of this test need one of these. This could run in parallel on each host.
00-<host name>.sh Each host that is part of this test need one of these. These scripts could run in parallel on each test host.
nnn-<host name>.sh will be executed serially on the host.
nn-<host name>.sh will be executed serially on the <host> ordered by <nn> or <nnn>.
nnn-<host name>-debug.sh will be executed in serial. However the output or the commands will not be part <host>.console.txt. The output will go OUTPUT/<host name>.verbose.txt
</pre>
</pre>
=== Output ===
=== Output ===
==== Reference output: ====
==== Reference output: ====
===== verbose output =====
==== verbose output =====
 
==== tcpdump output =====
==== gdb output =====
 
==== iptables ====  


===== tcpdump output =====
Ideally no iptables rules on the host. It could interfear with host xfrm rules and may interfear with conntrack rules. Such rules shoild be on the nic. This is deviation from the current model of east-west tests. Where I think we see iptable  influencing the tests.
===== gdb output =====


=== Installing libreswan ===  
=== Installing libreswan ===  
The testing could install libreswan in various ways. Each would support a specific use case.
The testing could install libreswan in various ways. Each would support a specific use case.
Line 23: Line 33:
This is the most simple one. It will be useful for automated test runs ad developers to quickly test something.
This is the most simple one. It will be useful for automated test runs ad developers to quickly test something.
==== Install using rpm =====
==== Install using rpm =====
This is more desirable for archived test result. This way if you notice an issue share the rpm.
 
for e.g. "make kvm-rpm" can create appropriate rpm files from the git repository for testing. These could be installed on each host.
 
The console output sanitizers are capable of distinguishing run from /usr/local/libexec/ipsec or rpm installed location.
The console output sanitizers are capable of distinguishing run from /usr/local/libexec/ipsec or rpm installed location.


Line 35: Line 47:
Possibly this could be hard to combine with rpms.
Possibly this could be hard to combine with rpms.


One way this could be supported is specify a custom rpm package?
One way this could be supported is specify a custom rpm package.

Latest revision as of 12:42, 24 October 2017

Requirements to extend testing, test suite and tests themslef as of Fall 2017 =

Test specifications

Test run scripts in the test directory and support single domain(guest vm) test and multi-domain tests

single domain test


<hostnamae>.sh

multi domain test

00-<host name>.sh Each host that is part of this test need one of these. These scripts could run in parallel on each test host.
nn-<host name>.sh will be executed serially on the  <host> ordered by <nn> or <nnn>.

Output

Reference output:

verbose output =

tcpdump output =

gdb output =

iptables

Ideally no iptables rules on the host. It could interfear with host xfrm rules and may interfear with conntrack rules. Such rules shoild be on the nic. This is deviation from the current model of east-west tests. Where I think we see iptable influencing the tests.


Installing libreswan

The testing could install libreswan in various ways. Each would support a specific use case.

Install with "make install-base" =

This is the most simple one. It will be useful for automated test runs ad developers to quickly test something.

Install using rpm =

for e.g. "make kvm-rpm" can create appropriate rpm files from the git repository for testing. These could be installed on each host.

The console output sanitizers are capable of distinguishing run from /usr/local/libexec/ipsec or rpm installed location.

If is useful to test interoperatbility with older versions. In such case you may want to able to specify rpm per host. So one or more host, say east and west would run, with different rpm.

Some tests support its own Makefile.inc.local file

Then there is are odd tests which need specific compile time options set or unset. For example DNSEC defined or not, AUDIT is enabled or not. One solution that comes to mind is add a <host>.Makefile.inc.local in the directory. If you see that then rebuild on this specific host. However after the test run you may have rebuild the whole vm from clone image to reset.

Possibly this could be hard to combine with rpms.

One way this could be supported is specify a custom rpm package.