Now, it is also run in schedules and most annoyingly on push which means
that it is run twice on a push to a branch where a MR exists and `.gitlab-ci.yml` is changed.
This was an oversight in https://gitlab.isc.org/isc-projects/bind9/-/merge_requests/10654
Backport of MR !10668
Merge branch 'backport-stepan/remove-additional-pipeline-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10669
Now, it is also run in schedules and most annoyingly on push which means
that it is run twice on a push to a branch where a MR exists.
(cherry picked from commit 7ca18df58a)
- increase clarity of multiline messages
- support `isc.query.*()` query&response logging
- replace use of `print()` statement with proper logging
- omit empty lines from test result output
Backport of MR !10590
Merge branch 'backport-nicki/improve-pytest-logging-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10660
The extra messages are typically traceback from assertion failures.
Previously, they'd be printed only after all individual test case
results have been printed. That made it difficult to pair the traceback
to the failing test in some cases, as the node information (aka test
name) might not always be present.
Instead, log any extra messages related to a particular test failure
directly after reporting its result, making the failure details more
readily available and easy to connect with a particular test case.
(cherry picked from commit fcf31417dd)
The command's stdout may provide useful debug info, so make sure we log
it by default. It doesn't seem to have a significant impact on the log
size.
(cherry picked from commit 9f3f6ec38e)
Make sure the queries and responses are logged at the DEBUG level, which
may provide useful information in case of failing tests.
This doesn't seem to significantly increase the overall artifacts size.
Previously, pytest.log.txt files from all system tests would take around
3 MB, with this change, it's around 8 MB).
(cherry picked from commit 56fec9ba04)
In some cases, it's useful to log the sent and received DNS messages.
Add options to enable this on demand. Query is only logged the first
time it's sent, since it doesn't change. If response logging is turned
on, then each response is logged, since it might be different every
time.
(cherry picked from commit 1e87b5ffc6)
When multiline message is logged, indent all but the first line (which
will be preceeded by the LOG_FORMAT). This improves the clarity of logs,
as it's immediately clear which lines are regular log output, and which
ones are multiline debug output.
Adjust the isctest.run.cmd() stdout/stderr logging to this new format.
(cherry picked from commit 23e6b49cc5)
The messages obtained from test results may contain stuff like detailed
failure/error information, tracebacks etc. In many cases, the message
will be empty, in which case it doesn't need to be logged.
For an example, run test with many test cases, e.g.
verify/test_verify.py, and inspect the tail of the pytest.log.txt before
and after this commit.
(cherry picked from commit 0a6b0cf68c)
Use isctest.log logging facility for consistent and predictable logging
output rather than using print(). Remove writes of stderr, as that
output will be logged in the debug log in case the commands called with
isctest.run.cmd() fails.
(cherry picked from commit 4b8998e4ad)
If an new orphan anchor is (`.anchor: &anchor` with no corresponding `*anchor` elsewhere in the file) is introduced the CI job will.
Depends on https://gitlab.isc.org/isc-projects/bind9-qa/-/merge_requests/101 (merge that first and then drop the `--branch` commit).
Backport of MR !10654
Merge branch 'backport-stepan/ci-orphaned-anchors-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10666
This test doesn't require artifact checking but when bundled in the same
directory with the shell based tests, the `system:clang:tsan` job was
failing non-deterministically.
(cherry picked from commit d5874d5df9)
This is a test for #5380.
Backport of MR !10596
Merge branch 'backport-stepan/mirror-root-zone-from-the-internet-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10657
We skip those by default as:
a) we don't want to stress the upstream servers in every CI pipeline
b) system tests need to be runnable in a isolated environment by default
(cherry picked from commit 3a8ffc74df)
Previously, JUnit files were not generated or were generated empty for various reasons for some system/unit test runs.
Now, the number of tests collected for a MR is up from about 4k to 5.8k in the "Tests" tab of a pipeline.
Additionally, there is a check that ensures that [a somewhat sane](c5a271eb8b) `junit.xml` file is generated after every system/unit test job and fails the job otherwise.
Closes#5316
Backport of MR !10556
Merge branch 'backport-5316-ensure-junit-xml-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10649
There might be more than one :test-result: and they are collated into
the :global-test-result: field.
This only happens when system tests are run with `make check`.
In some cases the report wasn't generated, sometimes it wasn't kept
properly. This unifies the way artifacts are generated and kept.
(cherry picked from commit 4ec1a37ca0)
In the past artifacts of different types of system test jobs were
treated differently but this is no longer the case.
(cherry picked from commit c61ff639b3)
In the tumbleweed image, we utilize LibreSSL. Several BIND 9 libraries
are linked against LibreSSL's libcrypto.so.55, and when Kerberos is
enabled, we link against libk5crypto.so.3, which in turn links against
OpenSSL's libcrypto.so.3. This might theoretically lead to a symbol
conflict.
Closes#5394
Backport of MR !10643
Merge branch 'backport-5394-disable-kerberos-in-tumbleweed-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10651
In the tumbleweed image, we utilize LibreSSL. Several BIND 9 libraries
are linked against LibreSSL's libcrypto.so.55, and when Kerberos is
enabled, we link against libk5crypto.so.3, which in turn links against
OpenSSL's libcrypto.so.3. This might theoretically lead to a symbol
conflict.
(cherry picked from commit 1b2c191bed)
The prep_doc_mr.py script of the bind9-qa repo needs a way to know that
gitchangelog.py did not produce entries. In the case of release notes,
it dies with "No commits matching given revlist". For changelog entries
it used to warn about "Empty changelog", but did not return non-zero
exit code.
Backport of MR !10591
Merge branch 'backport-mnowak/make-empty-changelog-fatal-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10641
The prep_doc_mr.py script of the bind9-qa repo needs a way to know that
gitchangelog.py did not produce entries. In the case of release notes,
it dies with "No commits matching given revlist". For changelog entries
it used to warn about "Empty changelog", but did not return non-zero
exit code.
(cherry picked from commit 4d0ae4068f)
Previous CPU test relied on either missing default named.conf or the
missing permissions to write into its default directory. In short that
default configuration would be unusable with current user. It would hang
indefinitely at cpu test if the named user could write into directory
specified in default configuration.
Change it instead to explicitly try non-existent configuration file.
It will still fail immediately, but will not rely on running user or
presence of file at default configuration file path.
(cherry picked from commit 8e789ea62f)
We need disable clang-format here to preserve the brackets around
the string concatenation to prevent -Wstring-concatenation -Werror
breaking the build.
(cherry picked from commit eeafcee7ad)
We need to turn off clang-format to preserve the brackets as
'attribute' can be an expression and we need it to be evaluated
first.
Similarly we need the entire result to be evaluated independent of
the adjoining code.
(cherry picked from commit 3620db5ea6)
In case the changelog file doesn't have an empty line at the end of the
file, the job may fail with the following error:
WARNING: Bullet list ends without a blank line; unexpected unindent.
This typically happens in MRs targeting the -S edition, as those
changelogs usually don't have an empty newline. This change ensures the
changelog job can pass and verify the title/desc contents even in those
cases.
Backport of MR !10628
Merge branch 'backport-nicki/ci-changelog-add-missing-newline-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10632
In case the changelog file doesn't have an empty line at the end of the
file, the job may fail with the following error:
WARNING: Bullet list ends without a blank line; unexpected unindent.
This typically happens in MRs targeting the -S edition, as those
changelogs usually don't have an empty newline. This change ensures the
changelog job can pass and verify the title/desc contents even in those
cases.
(cherry picked from commit ebf155ecc8)
There is an ongoing debate about the usefulness of the extra artifacts
check. While it might be useful to detect unexpected behaviour in some
tests, it feels extraneous in many cases. This change provides a middle
ground by making the artifact checking optional. This might be
especially useful for writing new tests, since the author gets to decide
whether the check is useful -- and can utilize it, or can skip it for
sake of brevity.
Backport of MR !10622
Merge branch 'backport-nicki/make-extra-artifacts-check-optional-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10629
There is an ongoing debate about the usefulness of the extra artifacts
check. While it might be useful to detect unexpected behaviour in some
tests, it feels extraneous in many cases. This change provides a middle
ground by making the artifact checking optional. This might be
especially useful for writing new tests, since the author gets to decide
whether the check is useful -- and can utilize it, or can skip it for
sake of brevity.
(cherry picked from commit c06dc71cd5)
Move the util/generate-stress-test-configs.py script from the BIND 9
source repository to the BIND 9 QA repository. This simplifies the
maintenance of that script by eliminating the need to backport every
change applied to it to multiple branches.
Backport of MR !10585
Merge branch 'backport-michal/move-stress-test-generation-script-to-qa-repo-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10612
Move the util/generate-stress-test-configs.py script from the BIND 9
source repository to the BIND 9 QA repository. This simplifies the
maintenance of that script by eliminating the need to backport every
change applied to it to multiple branches.
(cherry picked from commit 4f7f420534)
