Previously, JUnit files were not generated or were generated empty for various reasons for some system/unit test runs.
Now, the number of tests collected for a MR is up from about 4k to 5.8k in the "Tests" tab of a pipeline.
Additionally, there is a check that ensures that [a somewhat sane](c5a271eb8b) `junit.xml` file is generated after every system/unit test job and fails the job otherwise.
Closes#5316
Backport of MR !10556
Merge branch 'backport-5316-ensure-junit-xml-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10649
There might be more than one :test-result: and they are collated into
the :global-test-result: field.
This only happens when system tests are run with `make check`.
In some cases the report wasn't generated, sometimes it wasn't kept
properly. This unifies the way artifacts are generated and kept.
(cherry picked from commit 4ec1a37ca0)
In the past artifacts of different types of system test jobs were
treated differently but this is no longer the case.
(cherry picked from commit c61ff639b3)
In the tumbleweed image, we utilize LibreSSL. Several BIND 9 libraries
are linked against LibreSSL's libcrypto.so.55, and when Kerberos is
enabled, we link against libk5crypto.so.3, which in turn links against
OpenSSL's libcrypto.so.3. This might theoretically lead to a symbol
conflict.
Closes#5394
Backport of MR !10643
Merge branch 'backport-5394-disable-kerberos-in-tumbleweed-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10651
In the tumbleweed image, we utilize LibreSSL. Several BIND 9 libraries
are linked against LibreSSL's libcrypto.so.55, and when Kerberos is
enabled, we link against libk5crypto.so.3, which in turn links against
OpenSSL's libcrypto.so.3. This might theoretically lead to a symbol
conflict.
(cherry picked from commit 1b2c191bed)
The prep_doc_mr.py script of the bind9-qa repo needs a way to know that
gitchangelog.py did not produce entries. In the case of release notes,
it dies with "No commits matching given revlist". For changelog entries
it used to warn about "Empty changelog", but did not return non-zero
exit code.
Backport of MR !10591
Merge branch 'backport-mnowak/make-empty-changelog-fatal-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10641
The prep_doc_mr.py script of the bind9-qa repo needs a way to know that
gitchangelog.py did not produce entries. In the case of release notes,
it dies with "No commits matching given revlist". For changelog entries
it used to warn about "Empty changelog", but did not return non-zero
exit code.
(cherry picked from commit 4d0ae4068f)
Previous CPU test relied on either missing default named.conf or the
missing permissions to write into its default directory. In short that
default configuration would be unusable with current user. It would hang
indefinitely at cpu test if the named user could write into directory
specified in default configuration.
Change it instead to explicitly try non-existent configuration file.
It will still fail immediately, but will not rely on running user or
presence of file at default configuration file path.
(cherry picked from commit 8e789ea62f)
We need disable clang-format here to preserve the brackets around
the string concatenation to prevent -Wstring-concatenation -Werror
breaking the build.
(cherry picked from commit eeafcee7ad)
We need to turn off clang-format to preserve the brackets as
'attribute' can be an expression and we need it to be evaluated
first.
Similarly we need the entire result to be evaluated independent of
the adjoining code.
(cherry picked from commit 3620db5ea6)
In case the changelog file doesn't have an empty line at the end of the
file, the job may fail with the following error:
WARNING: Bullet list ends without a blank line; unexpected unindent.
This typically happens in MRs targeting the -S edition, as those
changelogs usually don't have an empty newline. This change ensures the
changelog job can pass and verify the title/desc contents even in those
cases.
Backport of MR !10628
Merge branch 'backport-nicki/ci-changelog-add-missing-newline-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10632
In case the changelog file doesn't have an empty line at the end of the
file, the job may fail with the following error:
WARNING: Bullet list ends without a blank line; unexpected unindent.
This typically happens in MRs targeting the -S edition, as those
changelogs usually don't have an empty newline. This change ensures the
changelog job can pass and verify the title/desc contents even in those
cases.
(cherry picked from commit ebf155ecc8)
There is an ongoing debate about the usefulness of the extra artifacts
check. While it might be useful to detect unexpected behaviour in some
tests, it feels extraneous in many cases. This change provides a middle
ground by making the artifact checking optional. This might be
especially useful for writing new tests, since the author gets to decide
whether the check is useful -- and can utilize it, or can skip it for
sake of brevity.
Backport of MR !10622
Merge branch 'backport-nicki/make-extra-artifacts-check-optional-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10629
There is an ongoing debate about the usefulness of the extra artifacts
check. While it might be useful to detect unexpected behaviour in some
tests, it feels extraneous in many cases. This change provides a middle
ground by making the artifact checking optional. This might be
especially useful for writing new tests, since the author gets to decide
whether the check is useful -- and can utilize it, or can skip it for
sake of brevity.
(cherry picked from commit c06dc71cd5)
Move the util/generate-stress-test-configs.py script from the BIND 9
source repository to the BIND 9 QA repository. This simplifies the
maintenance of that script by eliminating the need to backport every
change applied to it to multiple branches.
Backport of MR !10585
Merge branch 'backport-michal/move-stress-test-generation-script-to-qa-repo-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10612
Move the util/generate-stress-test-configs.py script from the BIND 9
source repository to the BIND 9 QA repository. This simplifies the
maintenance of that script by eliminating the need to backport every
change applied to it to multiple branches.
(cherry picked from commit 4f7f420534)
Add a 1 second wait before updating verify-axfr.db so that the
modification time of the file changes.
Closes#5376
Backport of MR !10586
Merge branch 'backport-5376-verify-axfr-db-gets-updated-too-fast-in-mirror-test-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10609
In some rare cases, the softhsm2 utility reports failure to delete the
token directory, despite the token being found. Subsequent attempts to
delete the token again indicate that the token was deleted.
Ignore this cleanup error, as it doesn't prevent our tests from working
properly. There is also an attempt to delete the token before the test
starts which ensures a clean state before the test is executed, in case
there's actually a leftover token.
Closes#5244
Backport of MR !10607
Merge branch 'backport-5244-ignore-softhsm2util-delete-token-error-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10608
In some rare cases, the softhsm2 utility reports failure to delete the
token directory, despite the token being found. Subsequent attempts to
delete the token again indicate that the token was deleted.
Ignore this cleanup error, as it doesn't prevent our tests from working
properly. There is also an attempt to delete the token before the test
starts which ensures a clean state before the test is executed, in case
there's actually a leftover token.
(cherry picked from commit e786a2e950)
Allow use of exception (and by extension, assert statements) in the
called function in order to extract essential debug information about
the type of failure that was encountered.
In case the called function fails to succeed on the last retry and
raised an exception, log it as error and set it as the assert message to
propagate it through the pytest framework.
Closes#5324
Backport of MR !10580
Merge branch 'backport-5324-pytest-isctest-run-logging-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10605
For duration measurements, i.e. deadlines and timeouts, it's more
suitable to use monotonic time as it's guaranteed to only go forward,
unlike time.time() which can be affected by local clock settings.
(cherry picked from commit 069e4ef0f7)
Allow use of exception (and by extension, assert statements) in the
called function in order to extract essential debug information about
the type of failure that was encountered.
In case the called function fails to succeed on the last retry and
raised an exception, log it as error and set it as the assert message to
propagate it through the pytest framework.
(cherry picked from commit 620c884133)
Previously, when a DNSSEC key was purged by one zone view, other zone views would return an error about missing key files. This has been fixed.
Closes#5315
Backport of MR !10550
Merge branch 'backport-5315-fix-spurious-some-key-files-are-missing-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10598
This happens because old key is purged by one zone view, then the other
is freaking out about it.
Keys that are unused or being purged should not be taken into account
when verifying key files are available.
The keyring is maintained per zone. So in one zone, a key in the
keyring is being purged. The corresponding key file is removed.
The key maintenance is done for the other zone view. The key in that
keyring is not yet set to purge, but its corresponding key file is
removed. This leads to "some keys are missing" log errors.
We should not check the purge variable at this point, but the
current time and purge-keys duration.
This commit fixes this erroneous logic.
(cherry picked from commit d494698852)
Create a test scenario where a signed zone is in multiple views and
then a key may be purged. This is a bug case where the key files are
removed by one view and then the other view starts complaining.
(cherry picked from commit 752d8617f5)
These test cases involve a reconfig, dnssec policy changes.
Backport of MR !10295
Merge branch 'backport-matthijs-pytest-rewrite-kasp-system-test-6-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10594
When going insecure, we publish CDS and CDNSKEY DELETE records. Update
the check_apex function to test this.
Also, skip some tests in the 'check_rollover_step()' function. If
we change the DNSSEC Policy, keys that no longer match the policy will
be retired. When this exactly happens is hard to determine, as it
happens on the reconfigure. So for these tests, we skip the key timing
metadata checks.
Also, the zone becomes unsigned, so don't call 'check_zone_is_signed'
in those cases.
(cherry picked from commit b1d8217d1a)
These test cases involve a reconfiguration. The first one is a zone
that changes from dynamic to inline-signing. The others are tests that
key lifetimes are updated correctly after changing them.
(cherry picked from commit de3c0970eb)
The state files need to be written before trying to identify zsk/ksk
keys. Wait for "keymgr: manual-rollover.kasp done" to appear in
named.run first.
Closes#5371
Backport of MR !10587
Merge branch 'backport-5371-unstable-rollover-tests_rollover-py-test_rollover_manual-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10589
The state files need to be written before trying to identify zsk/ksk
keys. Wait for "keymgr: manual-rollover.kasp done" to appear in
named.run first.
(cherry picked from commit 80fedf7fcf)
Add support to display the CO (Compact denial of existence Ok flag) when displaying messages.
Add support to set the CO flag when making queries in dig (+coflag).
Closes#5319
Backport of MR !10482
Merge branch 'backport-5319-add-support-to-set-and-display-the-co-flag-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10578
