Merge branch '1452-system-test-framework-cleanup-tweaks' into 'master'

System test framework: cleanup tweaks Closes #1452 See merge request isc-projects/bind9!2717
2025-09-01 06:55:30 +00:00 · 2019-12-06 13:32:07 +00:00
parent 34fb70b17c d8905b7a9c
commit dd6f9391c3
81 changed files with 38 additions and 325 deletions
--- a/bin/tests/system/.gitignore
+++ b/bin/tests/system/.gitignore
@@ -5,7 +5,7 @@ named.lock
 named.pid
 named.run
 /feature-test
-**/test.output
+/test.output.*
 /systests.output
 /random.data
 parallel.mk
--- a/bin/tests/system/Makefile.in
+++ b/bin/tests/system/Makefile.in
@@ -64,7 +64,7 @@ parallel.mk:

 test: parallel.mk subdirs
 	@$(MAKE) -f parallel.mk check
-	@$(SHELL) ./runsequential.sh -r
+	@$(SHELL) ./runsequential.sh
 	@$(SHELL) ./testsummary.sh

 check: test
--- a/bin/tests/system/README
+++ b/bin/tests/system/README
@@ -98,19 +98,6 @@ Optional flags are:
                    ports 7900 through 7999).  If not specified, the test will
                    have ports 5000 to 5099 available to it.

-    -r              The "runall" flag.  This is related to cleaning up after
-                    the tests (see "Maintenance Notes" below).  If specified,
-                    it prevents a copy of the test's output listing from being
-                    deleted when the directory is cleaned up after the test
-                    completes.  (The test's output listing comprises messages
-                    produced by the test during its execution; it does not
-                    include the output files produced by utilities such as
-                    "dig" or "rndc", nor any logging output from named itself.)
-                    It is usually only used when "run.sh" is being called
-                    during a run of the entire test suite.  Note that if "-n"
-                    is specified on the "run.sh" command line, the test output
-                    is retained even if this option is omitted.
-
 Arguments are:

    test-name       Mandatory.  The name of the test, which is the name of the
@@ -253,22 +240,12 @@ Re-Running the Tests
 ---
 If there is a requirement to re-run a test (or the entire test suite), the
 files produced by the tests should be deleted first.  Normally, these files are
-deleted if the test succeeds but are retained on error.
-
-Deletion of files produced by an individual test can be done with the command:
-
-    sh clean.sh [-r] <test-name>
-
-The optional flag is:
-
-    -r              The "runall" flag.  This is related to cleaning up after
-                    the tests (see "Maintenance Notes" below).  If specified,
-                    it prevents a copy of the test's output listing from being
-                    deleted when the directory is cleaned after the test
-                    completes.
+deleted if the test succeeds but are retained on error.  The run.sh script
+automatically calls a given test's clean.sh script before invoking its setup.sh
+script.

 Deletion of the files produced by the set of tests (e.g. after the execution
-of "runall.sh") can be deleted by the command:
+of "runall.sh") can be carried out using the command:

    sh cleanall.sh

@@ -709,7 +686,7 @@ the ports are assigned when the tests are run.  This is achieved by having the
 when "make check" is run, and contains a target for each test of the form:

    <test-name>:
-        @$(SHELL) run.sh -r -p <baseport> <test-name>
+        @$(SHELL) run.sh -p <baseport> <test-name>

 The <baseport> is unique and the values of <baseport> for each test are
 separated by at least 100 ports.
@@ -717,52 +694,27 @@ separated by at least 100 ports.

 Cleaning Up From Tests
 ---
-When a test is run, files are created in the test directory.  These files fall
-into three categories:
+When a test is run, up to three different types of files are created:

-1. Files generated by the test itself, e.g. output from "dig" and "rndc".
+1. Files generated by the test itself, e.g. output from "dig" and "rndc", are
+stored in the test directory.

 2. Files produced by named which may not be cleaned up if named exits
-abnormally, e.g. core files, PID files etc.
+abnormally, e.g. core files, PID files etc., are stored in the test directory.

-3. The file "test.output" containing the text written to stdout by the test.
-This is only produced when the test is run as part of the entire test suite
-(e.g. via "runall.sh").
+3. A file "test.output.<test-name>" containing the text written to stdout by the
+test is written to bin/tests/system/.  This file is only produced when the test
+is run as part of the entire test suite (e.g. via "runall.sh").

 If the test fails, all these files are retained.  But if the test succeeds,
 they are cleaned up at different times:

 1. Files generated by the test itself are cleaned up by the test's own
-"clean.sh".  This is called from the system's "clean.sh", which in turn is
-called from "run.sh".
+"clean.sh", which is called from "run.sh".

-2. Files that may not be cleaned up if named exits abnormally are removed
-by the system's "clean.sh".
+2. Files that may not be cleaned up if named exits abnormally can be removed
+using the "cleanall.sh" script.

-3. "test.output" files are deleted when the test suite ends.  At this point,
-the file "testsummary.sh" is called which concatenates all the "test.output"
-files into a single "systest.output" file before deleting them.
-
-A complication arises with the "test.output" file however:
-
-1. "clean.sh" is called by "run.sh" if the test ends successfully.  For
-this reason, "clean.sh" cannot delete "test.output" as, if the test is
-being run as part of a test suite, the file must be retained.
-
-2. If the deletion of "test.output" were to be solely the responsibility of
-"testsummary.sh", should a test suite terminate abnormally, cleaning up a test
-directory with "sh clean.sh <test-directory>" would leave the file present.
-
-3. An additional step could be added to "cleanall.sh" (which calls the
-system's "clean.sh" for each test) to remove the "test.output" file.  However,
-although the file would be deleted should all test directories be cleaned,
-it would still mean that running "clean.sh" on a particular test directory
-could leave the file present.
-
-To get round this, the system's "clean.sh" script takes an optional flag, "-r"
-(the "runall" flag).  When the test suite is run, each invocation of "run.sh"
-is passed the runall flag.  In turn, "run.sh" passes the flag to "clean.sh",
-which causes that script not to delete the "tests.output" file.  In other
-words, when the system's "clean.sh" is invoked standalone on a test directory
-(or as part of a run of "cleanall.sh"), it will delete the "test.output" if it
-is present.  When invoked during a run of the entire test suite, it won't.
+3. "test.output.*" files are deleted when the test suite ends.  At this point,
+the file "testsummary.sh" is called which concatenates all the "test.output.*"
+files into a single "systests.output" file before deleting them.
--- a/bin/tests/system/addzone/setup.sh
+++ b/bin/tests/system/addzone/setup.sh
@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 cp -f ns1/redirect.db.1 ns1/redirect.db
 cp -f ns2/redirect.db.1 ns2/redirect.db
 cp -f ns3/redirect.db.1 ns3/redirect.db
--- a/bin/tests/system/allow-query/setup.sh
+++ b/bin/tests/system/allow-query/setup.sh
@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 copy_setports ../common/controls.conf.in ns2/controls.conf
 copy_setports  ns1/named.conf.in ns1/named.conf
 copy_setports  ns2/named01.conf.in ns2/named.conf
--- a/bin/tests/system/auth/setup.sh
+++ b/bin/tests/system/auth/setup.sh
@@ -12,6 +12,5 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
--- a/bin/tests/system/autosign/setup.sh
+++ b/bin/tests/system/autosign/setup.sh
@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf
--- a/bin/tests/system/builtin/setup.sh
+++ b/bin/tests/system/builtin/setup.sh
@@ -12,7 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf
--- a/bin/tests/system/cacheclean/setup.sh
+++ b/bin/tests/system/cacheclean/setup.sh
@@ -12,6 +12,5 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
--- a/bin/tests/system/case/setup.sh
+++ b/bin/tests/system/case/setup.sh
@@ -10,8 +10,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 cp -f ns1/dynamic.db.in ns1/dynamic.db
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
--- a/bin/tests/system/cds/setup.sh
+++ b/bin/tests/system/cds/setup.sh
@@ -14,8 +14,6 @@ set -eu
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 touch empty

 Z=cds.test
--- a/bin/tests/system/chain/setup.sh
+++ b/bin/tests/system/chain/setup.sh
@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns5/named.conf.in ns5/named.conf
--- a/bin/tests/system/checkconf/setup.sh
+++ b/bin/tests/system/checkconf/setup.sh
@@ -1,15 +0,0 @@
-#!/bin/sh
-#
-# Copyright (C) Internet Systems Consortium, Inc. ("ISC")
-#
-# This Source Code Form is subject to the terms of the Mozilla Public
-# License, v. 2.0. If a copy of the MPL was not distributed with this
-# file, You can obtain one at http://mozilla.org/MPL/2.0/.
-#
-# See the COPYRIGHT file distributed with this work for additional
-# information regarding copyright ownership.
-
-# shellcheck source=conf.sh
-. "$SYSTEMTESTTOP/conf.sh"
-
-$SHELL clean.sh
--- a/bin/tests/system/checkds/setup.sh
+++ b/bin/tests/system/checkds/setup.sh
@@ -1,15 +0,0 @@
-#!/bin/sh
-#
-# Copyright (C) Internet Systems Consortium, Inc. ("ISC")
-#
-# This Source Code Form is subject to the terms of the Mozilla Public
-# License, v. 2.0. If a copy of the MPL was not distributed with this
-# file, You can obtain one at http://mozilla.org/MPL/2.0/.
-#
-# See the COPYRIGHT file distributed with this work for additional
-# information regarding copyright ownership.
-
-SYSTEMTESTTOP=..
-. $SYSTEMTESTTOP/conf.sh
-
-$SHELL clean.sh
--- a/bin/tests/system/checkzone/setup.sh
+++ b/bin/tests/system/checkzone/setup.sh
@@ -10,8 +10,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 ln -s $CHECKZONE named-compilezone

 ./named-compilezone -D -F raw -o good1.db.raw example \
--- a/bin/tests/system/clean.sh
+++ b/bin/tests/system/clean.sh
@@ -1,53 +0,0 @@
-#!/bin/sh
-#
-# Copyright (C) Internet Systems Consortium, Inc. ("ISC")
-#
-# This Source Code Form is subject to the terms of the Mozilla Public
-# License, v. 2.0. If a copy of the MPL was not distributed with this
-# file, You can obtain one at http://mozilla.org/MPL/2.0/.
-#
-# See the COPYRIGHT file distributed with this work for additional
-# information regarding copyright ownership.
-
-#
-# Clean up after a specified system test.
-#
-
-SYSTEMTESTTOP="$(cd -P -- "$(dirname -- "$0")" && pwd -P)"
-. $SYSTEMTESTTOP/conf.sh
-
-export SYSTEMTESTTOP
-
-# See if the "-r" flag is present.  This will usually be set when all the tests
-# are run (e.g. from "runall.sh") and tells the script not to delete the
-# test.output file created by run.sh.  This is because the script running all
-# the tests will call "testsummary.sh", which will concatenate all test output
-# files into a single systests.output.
-
-runall=0
-
-while getopts "r" flag; do
-    case $flag in
-	r) runall=1 ;;
-    esac
-done
-shift `expr $OPTIND - 1`
-
-if [ $# -eq 0 ]; then
-    echo "usage: $0 [-r] test-directory" >&2
-    exit 1
-fi
-
-systest=$1
-shift
-
-if [ $runall -eq 0 ]; then
-    rm -f $systest/test.output
-fi
-
-if [ -f $systest/clean.sh ]; then
-    ( cd $systest && $SHELL clean.sh "$@" )
-else
-    echo "Test directory $systest does not exist" >&2
-    exit 1
-fi
--- a/bin/tests/system/cleanall.sh
+++ b/bin/tests/system/cleanall.sh
@@ -30,6 +30,6 @@ rm -f $SYSTEMTESTTOP/random.data
 for d in $SUBDIRS
 do
   test ! -f $d/clean.sh || ( cd $d && $SHELL clean.sh )
-   test -f $d/test.output && rm $d/test.output
+   rm -f test.output.$d
   test -d $d && find $d -type d -exec rmdir '{}' \; 2> /dev/null
 done
--- a/bin/tests/system/cookie/setup.sh
+++ b/bin/tests/system/cookie/setup.sh
@@ -12,7 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf
--- a/bin/tests/system/database/setup.sh
+++ b/bin/tests/system/database/setup.sh
@@ -12,5 +12,4 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
 copy_setports ns1/named1.conf.in ns1/named.conf
--- a/bin/tests/system/dialup/setup.sh
+++ b/bin/tests/system/dialup/setup.sh
@@ -1,13 +0,0 @@
-# Copyright (C) Internet Systems Consortium, Inc. ("ISC")
-#
-# This Source Code Form is subject to the terms of the Mozilla Public
-# License, v. 2.0. If a copy of the MPL was not distributed with this
-# file, You can obtain one at http://mozilla.org/MPL/2.0/.
-#
-# See the COPYRIGHT file distributed with this work for additional
-# information regarding copyright ownership.
-
-# shellcheck source=conf.sh
-. "$SYSTEMTESTTOP/conf.sh"
-
-$SHELL clean.sh
--- a/bin/tests/system/digdelv/setup.sh
+++ b/bin/tests/system/digdelv/setup.sh
@@ -14,7 +14,6 @@

 set -e

-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf
--- a/bin/tests/system/dlzexternal/setup.sh
+++ b/bin/tests/system/dlzexternal/setup.sh
@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 $DDNSCONFGEN -q -z example.nil > ns1/ddns.key

 copy_setports ns1/named.conf.in ns1/named.conf
--- a/bin/tests/system/dns64/setup.sh
+++ b/bin/tests/system/dns64/setup.sh
@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf

--- a/bin/tests/system/dnstap/setup.sh
+++ b/bin/tests/system/dnstap/setup.sh
@@ -12,7 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf
--- a/bin/tests/system/dscp/setup.sh
+++ b/bin/tests/system/dscp/setup.sh
@@ -12,7 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf
--- a/bin/tests/system/dsdigest/setup.sh
+++ b/bin/tests/system/dsdigest/setup.sh
@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf
--- a/bin/tests/system/dyndb/setup.sh
+++ b/bin/tests/system/dyndb/setup.sh
@@ -12,5 +12,4 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf
--- a/bin/tests/system/ecdsa/setup.sh
+++ b/bin/tests/system/ecdsa/setup.sh
@@ -12,6 +12,4 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 cd ns1 && $SHELL sign.sh
--- a/bin/tests/system/eddsa/setup.sh
+++ b/bin/tests/system/eddsa/setup.sh
@@ -12,6 +12,4 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 cd ns1 && $SHELL sign.sh
--- a/bin/tests/system/ednscompliance/setup.sh
+++ b/bin/tests/system/ednscompliance/setup.sh
@@ -12,5 +12,4 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf
--- a/bin/tests/system/fetchlimit/setup.sh
+++ b/bin/tests/system/fetchlimit/setup.sh
@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh 
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named1.conf.in ns3/named.conf
--- a/bin/tests/system/filter-aaaa/setup.sh
+++ b/bin/tests/system/filter-aaaa/setup.sh
@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 copy_setports ns1/named1.conf.in ns1/named.conf
 copy_setports ns2/named1.conf.in ns2/named.conf
 copy_setports ns3/named1.conf.in ns3/named.conf
--- a/bin/tests/system/formerr/setup.sh
+++ b/bin/tests/system/formerr/setup.sh
@@ -12,5 +12,4 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf
--- a/bin/tests/system/glue/setup.sh
+++ b/bin/tests/system/glue/setup.sh
@@ -12,5 +12,4 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf
--- a/bin/tests/system/inline/setup.sh
+++ b/bin/tests/system/inline/setup.sh
@@ -10,8 +10,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 cp ns1/root.db.in ns1/root.db

 touch ns2/trusted.conf
--- a/bin/tests/system/integrity/setup.sh
+++ b/bin/tests/system/integrity/setup.sh
@@ -12,5 +12,4 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf
--- a/bin/tests/system/ixfr/setup.sh
+++ b/bin/tests/system/ixfr/setup.sh
@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 cat <<EOF >ns1/named.conf
 options {
 	query-source address 10.53.0.1;
--- a/bin/tests/system/keepalive/setup.sh
+++ b/bin/tests/system/keepalive/setup.sh
@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf
--- a/bin/tests/system/keymgr/setup.sh
+++ b/bin/tests/system/keymgr/setup.sh
@@ -14,8 +14,6 @@ SYSTEMTESTTOP=..

 KEYGEN="$KEYGEN -q"

-$SHELL clean.sh
-
 # Test 1: KSK goes inactive before successor is active
 dir=01-ksk-inactive
 echo_i "set up $dir"
--- a/bin/tests/system/legacy/setup.sh
+++ b/bin/tests/system/legacy/setup.sh
@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 copy_setports ns1/named1.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf
--- a/bin/tests/system/limits/setup.sh
+++ b/bin/tests/system/limits/setup.sh
@@ -12,5 +12,4 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf
--- a/bin/tests/system/metadata/setup.sh
+++ b/bin/tests/system/metadata/setup.sh
@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 pzone=parent.nil
 czone=child.parent.nil

--- a/bin/tests/system/mkeys/setup.sh
+++ b/bin/tests/system/mkeys/setup.sh
@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 copy_setports ns1/named1.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf
--- a/bin/tests/system/names/setup.sh
+++ b/bin/tests/system/names/setup.sh
@@ -10,6 +10,4 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
--- a/bin/tests/system/notify/setup.sh
+++ b/bin/tests/system/notify/setup.sh
@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf
--- a/bin/tests/system/nzd2nzf/setup.sh
+++ b/bin/tests/system/nzd2nzf/setup.sh
@@ -12,6 +12,4 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
--- a/bin/tests/system/padding/setup.sh
+++ b/bin/tests/system/padding/setup.sh
@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf
--- a/bin/tests/system/parallel.sh
+++ b/bin/tests/system/parallel.sh
@@ -29,6 +29,6 @@ port=${STARTPORT:-5000}
 for directory in $PARALLELDIRS ; do
        echo
        echo "test-`echo $directory | tr _ -`: check_interfaces"
-        echo "	@${SHELL} ./run.sh -r -p $port $directory 2>&1 | tee $directory/test.output"
+        echo "	@${SHELL} ./run.sh -p $port $directory 2>&1 | tee test.output.$directory"
        port=`expr $port + 100`
 done
--- a/bin/tests/system/pkcs11/setup.sh
+++ b/bin/tests/system/pkcs11/setup.sh
@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 infile=ns1/example.db.in

 /bin/echo -n ${HSMPIN:-1234}> pin
--- a/bin/tests/system/qmin/setup.sh
+++ b/bin/tests/system/qmin/setup.sh
@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns5/named.conf.in ns5/named.conf
 copy_setports ns6/named.conf.in ns6/named.conf
--- a/bin/tests/system/reclimit/setup.sh
+++ b/bin/tests/system/reclimit/setup.sh
@@ -12,6 +12,5 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns3/named1.conf.in ns3/named.conf
--- a/bin/tests/system/redirect/setup.sh
+++ b/bin/tests/system/redirect/setup.sh
@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf
--- a/bin/tests/system/resolver/setup.sh
+++ b/bin/tests/system/resolver/setup.sh
@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 cp ns4/tld1.db ns4/tld.db
 cp ns6/to-be-removed.tld.db.in ns6/to-be-removed.tld.db
 cp ns7/server.db.in ns7/server.db
--- a/bin/tests/system/rndc/setup.sh
+++ b/bin/tests/system/rndc/setup.sh
@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 $SHELL ../genzone.sh 2 >ns2/nil.db
 $SHELL ../genzone.sh 2 >ns2/other.db
 $SHELL ../genzone.sh 2 >ns2/static.db
--- a/bin/tests/system/rootkeysentinel/setup.sh
+++ b/bin/tests/system/rootkeysentinel/setup.sh
@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf
--- a/bin/tests/system/rrchecker/setup.sh
+++ b/bin/tests/system/rrchecker/setup.sh
@@ -1,15 +0,0 @@
-#!/bin/sh
-#
-# Copyright (C) Internet Systems Consortium, Inc. ("ISC")
-#
-# This Source Code Form is subject to the terms of the Mozilla Public
-# License, v. 2.0. If a copy of the MPL was not distributed with this
-# file, You can obtain one at http://mozilla.org/MPL/2.0/.
-#
-# See the COPYRIGHT file distributed with this work for additional
-# information regarding copyright ownership.
-
-# shellcheck source=conf.sh
-. "$SYSTEMTESTTOP/conf.sh"
-
-$SHELL clean.sh
--- a/bin/tests/system/rsabigexponent/setup.sh
+++ b/bin/tests/system/rsabigexponent/setup.sh
@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf
--- a/bin/tests/system/run.sh
+++ b/bin/tests/system/run.sh
@@ -37,13 +37,12 @@ while getopts "knp:r-:" flag; do
 	k) stopservers=false ;;
 	n) clean=false ;;
 	p) baseport=$OPTARG ;;
-	r) runall="-r" ;;
    esac
 done
 shift `expr $OPTIND - 1`

 if [ $# -eq 0 ]; then
-    echofail "Usage: $0 [-k] [-n] [-p <PORT>] [-r] test-directory [test-options]" >&2;
+    echofail "Usage: $0 [-k] [-n] [-p <PORT>] test-directory [test-options]" >&2;
    exit 1
 fi

@@ -155,6 +154,12 @@ else
    exit 0
 fi

+# Clean up files left from any potential previous runs
+if test -f $systest/clean.sh
+then
+   ( cd $systest && $SHELL clean.sh "$@" )
+fi
+
 # Set up any dynamically generated test data
 if test -f $systest/setup.sh
 then
@@ -219,7 +224,7 @@ else
        echopass "R:$systest:PASS"
        if $clean
        then
-            $SHELL clean.sh $runall $systest "$@"
+            ( cd $systest && $SHELL clean.sh "$@" )
            if test -d ../../../.git
            then
                git status -su --ignored $systest 2>/dev/null | \
--- a/bin/tests/system/runall.sh
+++ b/bin/tests/system/runall.sh
@@ -80,7 +80,7 @@ if [ "$NOPARALLEL" = "" ]; then
        # of parallel execution of system tests, and use that.
        $SHELL parallel.sh > parallel.mk
        make -f parallel.mk -j $numproc check
-        $SHELL ./runsequential.sh -r
+        $SHELL ./runsequential.sh
        $SHELL ./testsummary.sh || status=1
    fi
 else
--- a/bin/tests/system/runsequential.sh
+++ b/bin/tests/system/runsequential.sh
@@ -21,5 +21,5 @@ SYSTEMTESTTOP=.

 for d in $SEQUENTIALDIRS
 do
-    $SHELL run.sh "${@}" $d 2>&1 | tee $d/test.output
+    $SHELL run.sh "${@}" $d 2>&1 | tee test.output.$d
 done
--- a/bin/tests/system/sfcache/setup.sh
+++ b/bin/tests/system/sfcache/setup.sh
@@ -14,8 +14,6 @@

 set -e

-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns5/named.conf.in ns5/named.conf
--- a/bin/tests/system/smartsign/setup.sh
+++ b/bin/tests/system/smartsign/setup.sh
@@ -1,15 +0,0 @@
-#!/bin/sh -e
-#
-# Copyright (C) Internet Systems Consortium, Inc. ("ISC")
-#
-# This Source Code Form is subject to the terms of the Mozilla Public
-# License, v. 2.0. If a copy of the MPL was not distributed with this
-# file, You can obtain one at http://mozilla.org/MPL/2.0/.
-#
-# See the COPYRIGHT file distributed with this work for additional
-# information regarding copyright ownership.
-
-SYSTEMTESTTOP=..
-. $SYSTEMTESTTOP/conf.sh
-
-$SHELL clean.sh
--- a/bin/tests/system/sortlist/setup.sh
+++ b/bin/tests/system/sortlist/setup.sh
@@ -12,5 +12,4 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf
--- a/bin/tests/system/spf/setup.sh
+++ b/bin/tests/system/spf/setup.sh
@@ -12,5 +12,4 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf
--- a/bin/tests/system/staticstub/setup.sh
+++ b/bin/tests/system/staticstub/setup.sh
@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in tmp
 sed 's/SERVER_CONFIG_PLACEHOLDER/server-names { "ns.example.net"; };/' tmp > ns2/named.conf
--- a/bin/tests/system/statistics/setup.sh
+++ b/bin/tests/system/statistics/setup.sh
@@ -12,7 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf
--- a/bin/tests/system/statschannel/setup.sh
+++ b/bin/tests/system/statschannel/setup.sh
@@ -12,8 +12,6 @@
 # shellcheck source=conf.sh
 . "$SYSTEMTESTTOP/conf.sh"

-$SHELL clean.sh
-
 copy_setports ns2/named.conf.in ns2/named.conf

 (
--- a/bin/tests/system/stress/setup.sh
+++ b/bin/tests/system/stress/setup.sh
@@ -12,8 +12,6 @@
 # shellcheck source=conf.sh
 . "$SYSTEMTESTTOP/conf.sh"

-$SHELL clean.sh
-
 #
 # Set up test data for zone transfer quota tests.
 #
--- a/bin/tests/system/stub/setup.sh
+++ b/bin/tests/system/stub/setup.sh
@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf
--- a/bin/tests/system/synthfromdnssec/setup.sh
+++ b/bin/tests/system/synthfromdnssec/setup.sh
@@ -14,8 +14,6 @@

 set -e

-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf
--- a/bin/tests/system/testsummary.sh
+++ b/bin/tests/system/testsummary.sh
@@ -9,7 +9,7 @@
 # See the COPYRIGHT file distributed with this work for additional
 # information regarding copyright ownership.

-# Creates the system tests output file from the various test.output files.  It
+# Creates the system tests output file from the various test.output.* files.  It
 # then searches that file and prints the number of tests passed, failed, not
 # run.  It also checks whether the IP addresses 10.53.0.[1-8] were set up and,
 # if not, prints a warning.
@@ -17,7 +17,7 @@
 # Usage:
 #    testsummary.sh [-n]
 #
-# -n	Do NOT delete the individual test.output files after concatenating
+# -n	Do NOT delete the individual test.output.* files after concatenating
 #	them into systests.output.
 #
 # Status return:
@@ -35,13 +35,13 @@ while getopts "n" flag; do
    esac
 done

-if [ `ls */test.output 2> /dev/null | wc -l` -eq 0 ]; then
-    echowarn "I:No 'test.output' files were found."
+if [ `ls test.output.* 2> /dev/null | wc -l` -eq 0 ]; then
+    echowarn "I:No 'test.output.*' files were found."
    echowarn "I:Printing summary from pre-existing 'systests.output'."
 else
-    cat */test.output > systests.output
+    cat test.output.* > systests.output
    if [ $keepfile -eq 0 ]; then
-        rm -f */test.output
+        rm -f test.output.*
    fi
 fi

--- a/bin/tests/system/tkey/setup.sh
+++ b/bin/tests/system/tkey/setup.sh
@@ -12,6 +12,4 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 cd ns1 && $SHELL setup.sh
--- a/bin/tests/system/unknown/setup.sh
+++ b/bin/tests/system/unknown/setup.sh
@@ -10,8 +10,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh 
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf
--- a/bin/tests/system/upforwd/setup.sh
+++ b/bin/tests/system/upforwd/setup.sh
@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 cp -f ns1/example1.db ns1/example.db
 cp -f ns3/nomaster.db ns3/nomaster1.db

--- a/bin/tests/system/verify/setup.sh
+++ b/bin/tests/system/verify/setup.sh
@@ -12,6 +12,4 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh 
-
 (cd zones && $SHELL genzones.sh)
--- a/bin/tests/system/views/setup.sh
+++ b/bin/tests/system/views/setup.sh
@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 cp -f ns2/example1.db ns2/example.db

 copy_setports ns1/named.conf.in ns1/named.conf
--- a/bin/tests/system/wildcard/setup.sh
+++ b/bin/tests/system/wildcard/setup.sh
@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf
--- a/bin/tests/system/xfer/setup.sh
+++ b/bin/tests/system/xfer/setup.sh
@@ -12,8 +12,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 $SHELL ../genzone.sh 1 6 7 >ns1/slave.db
 $SHELL ../genzone.sh 1 6 7 >ns1/edns-expire.db
 $SHELL ../genzone.sh 2 3 >ns2/example.db
--- a/bin/tests/system/xferquota/setup.sh
+++ b/bin/tests/system/xferquota/setup.sh
@@ -16,8 +16,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 $PERL setup.pl

 cp -f ns1/changing1.db ns1/changing.db
--- a/bin/tests/system/zero/setup.sh
+++ b/bin/tests/system/zero/setup.sh
@@ -10,8 +10,6 @@
 SYSTEMTESTTOP=..
 . $SYSTEMTESTTOP/conf.sh

-$SHELL clean.sh
-
 copy_setports ns1/named.conf.in ns1/named.conf
 copy_setports ns2/named.conf.in ns2/named.conf
 copy_setports ns3/named.conf.in ns3/named.conf
--- a/util/copyrights
+++ b/util/copyrights
@@ -421,7 +421,6 @@
 ./bin/tests/system/checkconf/dnssec.2		X	2011,2016,2018,2019
 ./bin/tests/system/checkconf/dnssec.3		X	2011,2016,2017,2018,2019
 ./bin/tests/system/checkconf/good.zonelist	X	2016,2017,2018,2019
-./bin/tests/system/checkconf/setup.sh		SH	2019
 ./bin/tests/system/checkconf/tests.sh		SH	2005,2007,2010,2011,2012,2013,2014,2015,2016,2017,2018,2019
 ./bin/tests/system/checkds/clean.sh		SH	2012,2013,2014,2016,2017,2018,2019
 ./bin/tests/system/checkds/dig.bat		BAT	2016,2018,2019
@@ -435,7 +434,6 @@
 ./bin/tests/system/checkds/ok.example.ds.db	X	2012,2018,2019
 ./bin/tests/system/checkds/prep.example.db	X	2017,2018,2019
 ./bin/tests/system/checkds/prep.example.ds.db	X	2017,2018,2019
-./bin/tests/system/checkds/setup.sh		SH	2012,2013,2014,2016,2018,2019
 ./bin/tests/system/checkds/tests.sh		SH	2012,2013,2014,2016,2017,2018,2019
 ./bin/tests/system/checkds/wrong.example.dnskey.db	X	2012,2018,2019
 ./bin/tests/system/checkds/wrong.example.ds.db	X	2012,2018,2019
@@ -447,7 +445,6 @@
 ./bin/tests/system/checkzone/tests.sh		SH	2011,2012,2013,2014,2015,2016,2017,2018,2019
 ./bin/tests/system/checkzone/zones/.gitattributes	X	2015,2018,2019
 ./bin/tests/system/checkzone/zones/bad-badclass.raw	X	2015,2018,2019
-./bin/tests/system/clean.sh			SH	2015,2016,2018,2019
 ./bin/tests/system/cleanall.sh			SH	2000,2001,2004,2007,2012,2014,2015,2016,2017,2018,2019
 ./bin/tests/system/cleanpkcs11.sh		SH	2010,2012,2014,2016,2018,2019
 ./bin/tests/system/common/rndc.key		X	2011,2013,2016,2018,2019
@@ -492,7 +489,6 @@
 ./bin/tests/system/delzone/setup.sh		SH	2016,2018,2019
 ./bin/tests/system/delzone/tests.sh		SH	2016,2018,2019
 ./bin/tests/system/dialup/clean.sh		SH	2019
-./bin/tests/system/dialup/setup.sh		SH	2000,2001,2004,2007,2012,2016,2018,2019
 ./bin/tests/system/dialup/tests.sh		SH	2000,2001,2004,2007,2012,2016,2018,2019
 ./bin/tests/system/digcomp.pl			PERL	2000,2001,2004,2007,2012,2013,2016,2018,2019
 ./bin/tests/system/digdelv/ans4/startme		X	2017,2018,2019
@@ -956,7 +952,6 @@
 ./bin/tests/system/rrchecker/classlist.good	X	2013,2018,2019
 ./bin/tests/system/rrchecker/clean.sh		SH	2013,2014,2016,2017,2018,2019
 ./bin/tests/system/rrchecker/privatelist.good	X	2013,2018,2019
-./bin/tests/system/rrchecker/setup.sh		SH	2019
 ./bin/tests/system/rrchecker/tests.sh		SH	2013,2014,2015,2016,2017,2018,2019
 ./bin/tests/system/rrchecker/typelist.good	X	2013,2014,2015,2016,2017,2018,2019
 ./bin/tests/system/rrl/clean.sh			SH	2012,2013,2014,2016,2018,2019
@@ -1027,7 +1022,6 @@
 ./bin/tests/system/sfcache/setup.sh		SH	2014,2016,2017,2018,2019
 ./bin/tests/system/sfcache/tests.sh		SH	2014,2016,2017,2018,2019
 ./bin/tests/system/smartsign/clean.sh		SH	2010,2012,2014,2016,2018,2019
-./bin/tests/system/smartsign/setup.sh		SH	2010,2012,2014,2016,2017,2018,2019
 ./bin/tests/system/smartsign/tests.sh		SH	2010,2011,2012,2014,2016,2017,2018,2019
 ./bin/tests/system/sortlist/clean.sh		SH	2000,2001,2004,2007,2009,2012,2014,2015,2016,2018,2019
 ./bin/tests/system/sortlist/setup.sh		SH	2018,2019