Merge branch '1452-system-test-framework-cleanup-tweaks' into 'master'

System test framework: cleanup tweaks

Closes #1452

See merge request isc-projects/bind9!2717

bin/tests/system/.gitignore

@@ -5,7 +5,7 @@ named.lock
 named.pid
 named.run
 /feature-test
-**/test.output
+/test.output.*
 /systests.output
 /random.data
 parallel.mk

bin/tests/system/Makefile.in

@@ -64,7 +64,7 @@ parallel.mk:
 
 test: parallel.mk subdirs
 	@$(MAKE) -f parallel.mk check
-	@$(SHELL) ./runsequential.sh -r
+	@$(SHELL) ./runsequential.sh
 	@$(SHELL) ./testsummary.sh
 
 check: test

bin/tests/system/README

@@ -98,19 +98,6 @@ Optional flags are:
                     ports 7900 through 7999).  If not specified, the test will
                     have ports 5000 to 5099 available to it.
 
-    -r              The "runall" flag.  This is related to cleaning up after
-                    the tests (see "Maintenance Notes" below).  If specified,
-                    it prevents a copy of the test's output listing from being
-                    deleted when the directory is cleaned up after the test
-                    completes.  (The test's output listing comprises messages
-                    produced by the test during its execution; it does not
-                    include the output files produced by utilities such as
-                    "dig" or "rndc", nor any logging output from named itself.)
-                    It is usually only used when "run.sh" is being called
-                    during a run of the entire test suite.  Note that if "-n"
-                    is specified on the "run.sh" command line, the test output
-                    is retained even if this option is omitted.
-
 Arguments are:
 
     test-name       Mandatory.  The name of the test, which is the name of the
@@ -253,22 +240,12 @@ Re-Running the Tests
 ---
 If there is a requirement to re-run a test (or the entire test suite), the
 files produced by the tests should be deleted first.  Normally, these files are
-deleted if the test succeeds but are retained on error.
+deleted if the test succeeds but are retained on error.  The run.sh script
-
+automatically calls a given test's clean.sh script before invoking its setup.sh
-Deletion of files produced by an individual test can be done with the command:
+script.
-
-    sh clean.sh [-r] <test-name>
-
-The optional flag is:
-
-    -r              The "runall" flag.  This is related to cleaning up after
-                    the tests (see "Maintenance Notes" below).  If specified,
-                    it prevents a copy of the test's output listing from being
-                    deleted when the directory is cleaned after the test
-                    completes.
 
 Deletion of the files produced by the set of tests (e.g. after the execution
-of "runall.sh") can be deleted by the command:
+of "runall.sh") can be carried out using the command:
 
     sh cleanall.sh
 
@@ -709,7 +686,7 @@ the ports are assigned when the tests are run.  This is achieved by having the
 when "make check" is run, and contains a target for each test of the form:
 
     <test-name>:
-        @$(SHELL) run.sh -r -p <baseport> <test-name>
+        @$(SHELL) run.sh -p <baseport> <test-name>
 
 The <baseport> is unique and the values of <baseport> for each test are
 separated by at least 100 ports.
@@ -717,52 +694,27 @@ separated by at least 100 ports.
 
 Cleaning Up From Tests
 ---
-When a test is run, files are created in the test directory.  These files fall
+When a test is run, up to three different types of files are created:
-into three categories:
 
-1. Files generated by the test itself, e.g. output from "dig" and "rndc".
+1. Files generated by the test itself, e.g. output from "dig" and "rndc", are
+stored in the test directory.
 
 2. Files produced by named which may not be cleaned up if named exits
-abnormally, e.g. core files, PID files etc.
+abnormally, e.g. core files, PID files etc., are stored in the test directory.
 
-3. The file "test.output" containing the text written to stdout by the test.
+3. A file "test.output.<test-name>" containing the text written to stdout by the
-This is only produced when the test is run as part of the entire test suite
+test is written to bin/tests/system/.  This file is only produced when the test
-(e.g. via "runall.sh").
+is run as part of the entire test suite (e.g. via "runall.sh").
 
 If the test fails, all these files are retained.  But if the test succeeds,
 they are cleaned up at different times:
 
 1. Files generated by the test itself are cleaned up by the test's own
-"clean.sh".  This is called from the system's "clean.sh", which in turn is
+"clean.sh", which is called from "run.sh".
-called from "run.sh".
 
-2. Files that may not be cleaned up if named exits abnormally are removed
+2. Files that may not be cleaned up if named exits abnormally can be removed
-by the system's "clean.sh".
+using the "cleanall.sh" script.
 
-3. "test.output" files are deleted when the test suite ends.  At this point,
+3. "test.output.*" files are deleted when the test suite ends.  At this point,
-the file "testsummary.sh" is called which concatenates all the "test.output"
+the file "testsummary.sh" is called which concatenates all the "test.output.*"
-files into a single "systest.output" file before deleting them.
+files into a single "systests.output" file before deleting them.
-
-A complication arises with the "test.output" file however:
-
-1. "clean.sh" is called by "run.sh" if the test ends successfully.  For
-this reason, "clean.sh" cannot delete "test.output" as, if the test is
-being run as part of a test suite, the file must be retained.
-
-2. If the deletion of "test.output" were to be solely the responsibility of
-"testsummary.sh", should a test suite terminate abnormally, cleaning up a test
-directory with "sh clean.sh <test-directory>" would leave the file present.
-
-3. An additional step could be added to "cleanall.sh" (which calls the
-system's "clean.sh" for each test) to remove the "test.output" file.  However,
-although the file would be deleted should all test directories be cleaned,
-it would still mean that running "clean.sh" on a particular test directory
-could leave the file present.
-
-To get round this, the system's "clean.sh" script takes an optional flag, "-r"
-(the "runall" flag).  When the test suite is run, each invocation of "run.sh"
-is passed the runall flag.  In turn, "run.sh" passes the flag to "clean.sh",
-which causes that script not to delete the "tests.output" file.  In other
-words, when the system's "clean.sh" is invoked standalone on a test directory
-(or as part of a run of "cleanall.sh"), it will delete the "test.output" if it
-is present.  When invoked during a run of the entire test suite, it won't.

bin/tests/system/addzone/setup.sh

@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 cp -f ns1/redirect.db.1 ns1/redirect.db
 cp -f ns2/redirect.db.1 ns2/redirect.db
 cp -f ns3/redirect.db.1 ns3/redirect.db

bin/tests/system/allow-query/setup.sh

@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 copy_setports ../common/controls.conf.in ns2/controls.conf
 copy_setports  ns1/named.conf.in ns1/named.conf
 copy_setports  ns2/named01.conf.in ns2/named.conf

bin/tests/system/auth/setup.sh

@@ -12,6 +12,5 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf

bin/tests/system/autosign/setup.sh

@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf

bin/tests/system/builtin/setup.sh

@@ -12,7 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf

bin/tests/system/cacheclean/setup.sh

@@ -12,6 +12,5 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf

bin/tests/system/case/setup.sh

@@ -10,8 +10,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 cp -f ns1/dynamic.db.in ns1/dynamic.db
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf

bin/tests/system/cds/setup.sh

@@ -14,8 +14,6 @@ set -eu
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 touch empty
 
 Z=cds.test

bin/tests/system/chain/setup.sh

@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns5/named.conf.in ns5/named.conf

bin/tests/system/checkconf/setup.sh

@@ -1,15 +0,0 @@
-#!/bin/sh
-#
-# Copyright (C) Internet Systems Consortium, Inc. ("ISC")
-#
-# This Source Code Form is subject to the terms of the Mozilla Public
-# License, v. 2.0. If a copy of the MPL was not distributed with this
-# file, You can obtain one at http://mozilla.org/MPL/2.0/.
-#
-# See the COPYRIGHT file distributed with this work for additional
-# information regarding copyright ownership.
-
-# shellcheck source=conf.sh
-. "$SYSTEMTESTTOP/conf.sh"
-
-$SHELL clean.sh

bin/tests/system/checkds/setup.sh

@@ -1,15 +0,0 @@
-#!/bin/sh
-#
-# Copyright (C) Internet Systems Consortium, Inc. ("ISC")
-#
-# This Source Code Form is subject to the terms of the Mozilla Public
-# License, v. 2.0. If a copy of the MPL was not distributed with this
-# file, You can obtain one at http://mozilla.org/MPL/2.0/.
-#
-# See the COPYRIGHT file distributed with this work for additional
-# information regarding copyright ownership.
-
-SYSTEMTESTTOP=..
-. $SYSTEMTESTTOP/conf.sh
-
-$SHELL clean.sh

bin/tests/system/checkzone/setup.sh

@@ -10,8 +10,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 ln -s $CHECKZONE named-compilezone
 
 ./named-compilezone -D -F raw -o good1.db.raw example \

bin/tests/system/clean.sh

@@ -1,53 +0,0 @@
-#!/bin/sh
-#
-# Copyright (C) Internet Systems Consortium, Inc. ("ISC")
-#
-# This Source Code Form is subject to the terms of the Mozilla Public
-# License, v. 2.0. If a copy of the MPL was not distributed with this
-# file, You can obtain one at http://mozilla.org/MPL/2.0/.
-#
-# See the COPYRIGHT file distributed with this work for additional
-# information regarding copyright ownership.
-
-#
-# Clean up after a specified system test.
-#
-
-SYSTEMTESTTOP="$(cd -P -- "$(dirname -- "$0")" && pwd -P)"
-. $SYSTEMTESTTOP/conf.sh
-
-export SYSTEMTESTTOP
-
-# See if the "-r" flag is present.  This will usually be set when all the tests
-# are run (e.g. from "runall.sh") and tells the script not to delete the
-# test.output file created by run.sh.  This is because the script running all
-# the tests will call "testsummary.sh", which will concatenate all test output
-# files into a single systests.output.
-
-runall=0
-
-while getopts "r" flag; do
-    case $flag in
-	r) runall=1 ;;
-    esac
-done
-shift `expr $OPTIND - 1`
-
-if [ $# -eq 0 ]; then
-    echo "usage: $0 [-r] test-directory" >&2
-    exit 1
-fi
-
-systest=$1
-shift
-
-if [ $runall -eq 0 ]; then
-    rm -f $systest/test.output
-fi
-
-if [ -f $systest/clean.sh ]; then
-    ( cd $systest && $SHELL clean.sh "$@" )
-else
-    echo "Test directory $systest does not exist" >&2
-    exit 1
-fi

bin/tests/system/cleanall.sh

@@ -30,6 +30,6 @@ rm -f $SYSTEMTESTTOP/random.data
 for d in $SUBDIRS
 do
    test ! -f $d/clean.sh || ( cd $d && $SHELL clean.sh )
-   test -f $d/test.output && rm $d/test.output
+   rm -f test.output.$d
    test -d $d && find $d -type d -exec rmdir '{}' \; 2> /dev/null
 done

bin/tests/system/cookie/setup.sh

@@ -12,7 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf

bin/tests/system/database/setup.sh

@@ -12,5 +12,4 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
 copy_setports ns1/named1.conf.in ns1/named.conf

bin/tests/system/dialup/setup.sh

@@ -1,13 +0,0 @@
-# Copyright (C) Internet Systems Consortium, Inc. ("ISC")
-#
-# This Source Code Form is subject to the terms of the Mozilla Public
-# License, v. 2.0. If a copy of the MPL was not distributed with this
-# file, You can obtain one at http://mozilla.org/MPL/2.0/.
-#
-# See the COPYRIGHT file distributed with this work for additional
-# information regarding copyright ownership.
-
-# shellcheck source=conf.sh
-. "$SYSTEMTESTTOP/conf.sh"
-
-$SHELL clean.sh

bin/tests/system/digdelv/setup.sh

@@ -14,7 +14,6 @@
 
 set -e
 
-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf

bin/tests/system/dlzexternal/setup.sh

@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 $DDNSCONFGEN -q -z example.nil > ns1/ddns.key
 
 copy_setports ns1/named.conf.in ns1/named.conf

bin/tests/system/dns64/setup.sh

@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 

bin/tests/system/dnstap/setup.sh

@@ -12,7 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf

bin/tests/system/dscp/setup.sh

@@ -12,7 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf

bin/tests/system/dsdigest/setup.sh

@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf

bin/tests/system/dyndb/setup.sh

@@ -12,5 +12,4 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf

bin/tests/system/ecdsa/setup.sh

@@ -12,6 +12,4 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 cd ns1 && $SHELL sign.sh

bin/tests/system/eddsa/setup.sh

@@ -12,6 +12,4 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 cd ns1 && $SHELL sign.sh

bin/tests/system/ednscompliance/setup.sh

@@ -12,5 +12,4 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf

bin/tests/system/fetchlimit/setup.sh

@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh 
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named1.conf.in ns3/named.conf

bin/tests/system/filter-aaaa/setup.sh

@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 copy_setports ns1/named1.conf.in ns1/named.conf
 copy_setports ns2/named1.conf.in ns2/named.conf
 copy_setports ns3/named1.conf.in ns3/named.conf

bin/tests/system/formerr/setup.sh

@@ -12,5 +12,4 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf

bin/tests/system/glue/setup.sh

@@ -12,5 +12,4 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf

bin/tests/system/inline/setup.sh

@@ -10,8 +10,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 cp ns1/root.db.in ns1/root.db
 
 touch ns2/trusted.conf

bin/tests/system/integrity/setup.sh

@@ -12,5 +12,4 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf

bin/tests/system/ixfr/setup.sh

@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 cat <<EOF >ns1/named.conf
 options {
 	query-source address 10.53.0.1;

bin/tests/system/keepalive/setup.sh

@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf

bin/tests/system/keymgr/setup.sh

@@ -14,8 +14,6 @@ SYSTEMTESTTOP=..
 
 KEYGEN="$KEYGEN -q"
 
-$SHELL clean.sh
-
 # Test 1: KSK goes inactive before successor is active
 dir=01-ksk-inactive
 echo_i "set up $dir"

bin/tests/system/legacy/setup.sh

@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 copy_setports ns1/named1.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf

bin/tests/system/limits/setup.sh

@@ -12,5 +12,4 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf

bin/tests/system/metadata/setup.sh

@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 pzone=parent.nil
 czone=child.parent.nil
 

bin/tests/system/mkeys/setup.sh

@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 copy_setports ns1/named1.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf

bin/tests/system/names/setup.sh

@@ -10,6 +10,4 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf

bin/tests/system/notify/setup.sh

@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf

bin/tests/system/nzd2nzf/setup.sh

@@ -12,6 +12,4 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf

bin/tests/system/padding/setup.sh

@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf

bin/tests/system/parallel.sh

@@ -29,6 +29,6 @@ port=${STARTPORT:-5000}
 for directory in $PARALLELDIRS ; do
         echo
         echo "test-`echo $directory | tr _ -`: check_interfaces"
-        echo "	@${SHELL} ./run.sh -r -p $port $directory 2>&1 | tee $directory/test.output"
+        echo "	@${SHELL} ./run.sh -p $port $directory 2>&1 | tee test.output.$directory"
         port=`expr $port + 100`
 done

bin/tests/system/pkcs11/setup.sh

@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 infile=ns1/example.db.in
 
 /bin/echo -n ${HSMPIN:-1234}> pin

bin/tests/system/qmin/setup.sh

@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns5/named.conf.in ns5/named.conf
 copy_setports ns6/named.conf.in ns6/named.conf

bin/tests/system/reclimit/setup.sh

@@ -12,6 +12,5 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns3/named1.conf.in ns3/named.conf

bin/tests/system/redirect/setup.sh

@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf

bin/tests/system/resolver/setup.sh

@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 cp ns4/tld1.db ns4/tld.db
 cp ns6/to-be-removed.tld.db.in ns6/to-be-removed.tld.db
 cp ns7/server.db.in ns7/server.db

bin/tests/system/rndc/setup.sh

@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 $SHELL ../genzone.sh 2 >ns2/nil.db
 $SHELL ../genzone.sh 2 >ns2/other.db
 $SHELL ../genzone.sh 2 >ns2/static.db

bin/tests/system/rootkeysentinel/setup.sh

@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf

bin/tests/system/rrchecker/setup.sh

@@ -1,15 +0,0 @@
-#!/bin/sh
-#
-# Copyright (C) Internet Systems Consortium, Inc. ("ISC")
-#
-# This Source Code Form is subject to the terms of the Mozilla Public
-# License, v. 2.0. If a copy of the MPL was not distributed with this
-# file, You can obtain one at http://mozilla.org/MPL/2.0/.
-#
-# See the COPYRIGHT file distributed with this work for additional
-# information regarding copyright ownership.
-
-# shellcheck source=conf.sh
-. "$SYSTEMTESTTOP/conf.sh"
-
-$SHELL clean.sh

bin/tests/system/rsabigexponent/setup.sh

@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf

bin/tests/system/run.sh

@@ -37,13 +37,12 @@ while getopts "knp:r-:" flag; do
 	k) stopservers=false ;;
 	n) clean=false ;;
 	p) baseport=$OPTARG ;;
-	r) runall="-r" ;;
     esac
 done
 shift `expr $OPTIND - 1`
 
 if [ $# -eq 0 ]; then
-    echofail "Usage: $0 [-k] [-n] [-p <PORT>] [-r] test-directory [test-options]" >&2;
+    echofail "Usage: $0 [-k] [-n] [-p <PORT>] test-directory [test-options]" >&2;
     exit 1
 fi
 
@@ -155,6 +154,12 @@ else
     exit 0
 fi
 
+# Clean up files left from any potential previous runs
+if test -f $systest/clean.sh
+then
+   ( cd $systest && $SHELL clean.sh "$@" )
+fi
+
 # Set up any dynamically generated test data
 if test -f $systest/setup.sh
 then
@@ -219,7 +224,7 @@ else
         echopass "R:$systest:PASS"
         if $clean
         then
-            $SHELL clean.sh $runall $systest "$@"
+            ( cd $systest && $SHELL clean.sh "$@" )
             if test -d ../../../.git
             then
                 git status -su --ignored $systest 2>/dev/null | \

bin/tests/system/runall.sh

@@ -80,7 +80,7 @@ if [ "$NOPARALLEL" = "" ]; then
         # of parallel execution of system tests, and use that.
         $SHELL parallel.sh > parallel.mk
         make -f parallel.mk -j $numproc check
-        $SHELL ./runsequential.sh -r
+        $SHELL ./runsequential.sh
         $SHELL ./testsummary.sh || status=1
     fi
 else

bin/tests/system/runsequential.sh

@@ -21,5 +21,5 @@ SYSTEMTESTTOP=.
 
 for d in $SEQUENTIALDIRS
 do
-    $SHELL run.sh "${@}" $d 2>&1 | tee $d/test.output
+    $SHELL run.sh "${@}" $d 2>&1 | tee test.output.$d
 done

bin/tests/system/sfcache/setup.sh

@@ -14,8 +14,6 @@
 
 set -e
 
-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns5/named.conf.in ns5/named.conf

bin/tests/system/smartsign/setup.sh

@@ -1,15 +0,0 @@
-#!/bin/sh -e
-#
-# Copyright (C) Internet Systems Consortium, Inc. ("ISC")
-#
-# This Source Code Form is subject to the terms of the Mozilla Public
-# License, v. 2.0. If a copy of the MPL was not distributed with this
-# file, You can obtain one at http://mozilla.org/MPL/2.0/.
-#
-# See the COPYRIGHT file distributed with this work for additional
-# information regarding copyright ownership.
-
-SYSTEMTESTTOP=..
-. $SYSTEMTESTTOP/conf.sh
-
-$SHELL clean.sh

bin/tests/system/sortlist/setup.sh

@@ -12,5 +12,4 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf

bin/tests/system/spf/setup.sh

@@ -12,5 +12,4 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf

bin/tests/system/staticstub/setup.sh

@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in tmp
 sed 's/SERVER_CONFIG_PLACEHOLDER/server-names { "ns.example.net"; };/' tmp > ns2/named.conf

bin/tests/system/statistics/setup.sh

@@ -12,7 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf

bin/tests/system/statschannel/setup.sh

@@ -12,8 +12,6 @@
 # shellcheck source=conf.sh
 . "$SYSTEMTESTTOP/conf.sh"
 
-$SHELL clean.sh
-
 copy_setports ns2/named.conf.in ns2/named.conf
 
 (

bin/tests/system/stress/setup.sh

@@ -12,8 +12,6 @@
 # shellcheck source=conf.sh
 . "$SYSTEMTESTTOP/conf.sh"
 
-$SHELL clean.sh
-
 #
 # Set up test data for zone transfer quota tests.
 #

bin/tests/system/stub/setup.sh

@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf

bin/tests/system/synthfromdnssec/setup.sh

@@ -14,8 +14,6 @@
 
 set -e
 
-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf

bin/tests/system/testsummary.sh

@@ -9,7 +9,7 @@
 # See the COPYRIGHT file distributed with this work for additional
 # information regarding copyright ownership.
 
-# Creates the system tests output file from the various test.output files.  It
+# Creates the system tests output file from the various test.output.* files.  It
 # then searches that file and prints the number of tests passed, failed, not
 # run.  It also checks whether the IP addresses 10.53.0.[1-8] were set up and,
 # if not, prints a warning.
@@ -17,7 +17,7 @@
 # Usage:
 #    testsummary.sh [-n]
 #
-# -n	Do NOT delete the individual test.output files after concatenating
+# -n	Do NOT delete the individual test.output.* files after concatenating
 #	them into systests.output.
 #
 # Status return:
@@ -35,13 +35,13 @@ while getopts "n" flag; do
     esac
 done
 
-if [ `ls */test.output 2> /dev/null | wc -l` -eq 0 ]; then
+if [ `ls test.output.* 2> /dev/null | wc -l` -eq 0 ]; then
-    echowarn "I:No 'test.output' files were found."
+    echowarn "I:No 'test.output.*' files were found."
     echowarn "I:Printing summary from pre-existing 'systests.output'."
 else
-    cat */test.output > systests.output
+    cat test.output.* > systests.output
     if [ $keepfile -eq 0 ]; then
-        rm -f */test.output
+        rm -f test.output.*
     fi
 fi
 

bin/tests/system/tkey/setup.sh

@@ -12,6 +12,4 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 cd ns1 && $SHELL setup.sh

bin/tests/system/unknown/setup.sh

@@ -10,8 +10,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh 
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf

bin/tests/system/upforwd/setup.sh

@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 cp -f ns1/example1.db ns1/example.db
 cp -f ns3/nomaster.db ns3/nomaster1.db
 

bin/tests/system/verify/setup.sh

@@ -12,6 +12,4 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh 
-
 (cd zones && $SHELL genzones.sh)

bin/tests/system/views/setup.sh

@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 cp -f ns2/example1.db ns2/example.db
 
 copy_setports ns1/named.conf.in ns1/named.conf

bin/tests/system/wildcard/setup.sh

@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf

bin/tests/system/xfer/setup.sh

@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 $SHELL ../genzone.sh 1 6 7 >ns1/slave.db
 $SHELL ../genzone.sh 1 6 7 >ns1/edns-expire.db
 $SHELL ../genzone.sh 2 3 >ns2/example.db

bin/tests/system/xferquota/setup.sh

@@ -16,8 +16,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 $PERL setup.pl
 
 cp -f ns1/changing1.db ns1/changing.db

bin/tests/system/zero/setup.sh

@@ -10,8 +10,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh
 
-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf

util/copyrights

@@ -421,7 +421,6 @@
 ./bin/tests/system/checkconf/dnssec.2		X	2011,2016,2018,2019
 ./bin/tests/system/checkconf/dnssec.3		X	2011,2016,2017,2018,2019
 ./bin/tests/system/checkconf/good.zonelist	X	2016,2017,2018,2019
-./bin/tests/system/checkconf/setup.sh		SH	2019
 ./bin/tests/system/checkconf/tests.sh		SH	2005,2007,2010,2011,2012,2013,2014,2015,2016,2017,2018,2019
 ./bin/tests/system/checkds/clean.sh		SH	2012,2013,2014,2016,2017,2018,2019
 ./bin/tests/system/checkds/dig.bat		BAT	2016,2018,2019
@@ -435,7 +434,6 @@
 ./bin/tests/system/checkds/ok.example.ds.db	X	2012,2018,2019
 ./bin/tests/system/checkds/prep.example.db	X	2017,2018,2019
 ./bin/tests/system/checkds/prep.example.ds.db	X	2017,2018,2019
-./bin/tests/system/checkds/setup.sh		SH	2012,2013,2014,2016,2018,2019
 ./bin/tests/system/checkds/tests.sh		SH	2012,2013,2014,2016,2017,2018,2019
 ./bin/tests/system/checkds/wrong.example.dnskey.db	X	2012,2018,2019
 ./bin/tests/system/checkds/wrong.example.ds.db	X	2012,2018,2019
@@ -447,7 +445,6 @@
 ./bin/tests/system/checkzone/tests.sh		SH	2011,2012,2013,2014,2015,2016,2017,2018,2019
 ./bin/tests/system/checkzone/zones/.gitattributes	X	2015,2018,2019
 ./bin/tests/system/checkzone/zones/bad-badclass.raw	X	2015,2018,2019
-./bin/tests/system/clean.sh			SH	2015,2016,2018,2019
 ./bin/tests/system/cleanall.sh			SH	2000,2001,2004,2007,2012,2014,2015,2016,2017,2018,2019
 ./bin/tests/system/cleanpkcs11.sh		SH	2010,2012,2014,2016,2018,2019
 ./bin/tests/system/common/rndc.key		X	2011,2013,2016,2018,2019
@@ -492,7 +489,6 @@
 ./bin/tests/system/delzone/setup.sh		SH	2016,2018,2019
 ./bin/tests/system/delzone/tests.sh		SH	2016,2018,2019
 ./bin/tests/system/dialup/clean.sh		SH	2019
-./bin/tests/system/dialup/setup.sh		SH	2000,2001,2004,2007,2012,2016,2018,2019
 ./bin/tests/system/dialup/tests.sh		SH	2000,2001,2004,2007,2012,2016,2018,2019
 ./bin/tests/system/digcomp.pl			PERL	2000,2001,2004,2007,2012,2013,2016,2018,2019
 ./bin/tests/system/digdelv/ans4/startme		X	2017,2018,2019
@@ -956,7 +952,6 @@
 ./bin/tests/system/rrchecker/classlist.good	X	2013,2018,2019
 ./bin/tests/system/rrchecker/clean.sh		SH	2013,2014,2016,2017,2018,2019
 ./bin/tests/system/rrchecker/privatelist.good	X	2013,2018,2019
-./bin/tests/system/rrchecker/setup.sh		SH	2019
 ./bin/tests/system/rrchecker/tests.sh		SH	2013,2014,2015,2016,2017,2018,2019
 ./bin/tests/system/rrchecker/typelist.good	X	2013,2014,2015,2016,2017,2018,2019
 ./bin/tests/system/rrl/clean.sh			SH	2012,2013,2014,2016,2018,2019
@@ -1027,7 +1022,6 @@
 ./bin/tests/system/sfcache/setup.sh		SH	2014,2016,2017,2018,2019
 ./bin/tests/system/sfcache/tests.sh		SH	2014,2016,2017,2018,2019
 ./bin/tests/system/smartsign/clean.sh		SH	2010,2012,2014,2016,2018,2019
-./bin/tests/system/smartsign/setup.sh		SH	2010,2012,2014,2016,2017,2018,2019
 ./bin/tests/system/smartsign/tests.sh		SH	2010,2011,2012,2014,2016,2017,2018,2019
 ./bin/tests/system/sortlist/clean.sh		SH	2000,2001,2004,2007,2009,2012,2014,2015,2016,2018,2019
 ./bin/tests/system/sortlist/setup.sh		SH	2018,2019