2
0
mirror of https://gitlab.isc.org/isc-projects/bind9 synced 2025-08-30 14:07:59 +00:00

Detect missing system test results

At the end of each system test suite run, the system test framework
collects all existing test.output files from system test subdirectories
and produces bin/tests/system/systests.output from those files.
However, it does not check whether a test.output file was found for
every executed test.  Thus, if the test.output file is accidentally
deleted by the system test itself (e.g. due to an overly broad file
removal wildcard present in clean.sh), its output will not be included
in bin/tests/system/systests.output.  Since the result of each system
test suite run is determined by bin/tests/system/testsummary.sh, which
only operates on the contents of bin/tests/system/systests.output, this
can lead to test failures being ignored.  Fix by ensuring the number of
test results found in bin/tests/system/systests.output is equal to the
number of tests run and triggering a system test suite failure in case
of a discrepancy between these two values.
This commit is contained in:
Michał Kępień
2019-12-06 14:11:01 +01:00
parent 26ee43da1b
commit 3c3085be3c

View File

@@ -56,4 +56,11 @@ if [ -n "${FAILED_TESTS}" ]; then
status=1
fi
RESULTS_FOUND=`grep -c 'R:[a-z0-9_-][a-z0-9_-]*:[A-Z][A-Z]*' systests.output`
TESTS_RUN=`echo "${SUBDIRS}" | wc -w`
if [ "${RESULTS_FOUND}" -ne "${TESTS_RUN}" ]; then
echofail "I:Found ${RESULTS_FOUND} test results, but ${TESTS_RUN} tests were run"
status=1
fi
exit $status