This is a complement to the already present system test "stress" test.
Backport of MR !9474
Merge branch 'backport-mnowak/generate-tsan-unit-stress-tests-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10094
GitLab CI Runner's $builds_dir variable is set to "/builds" by default.
For technical reasons, the FreeBSD Runners, using the "instance"
executor, sets the path differently.
The value of $CI_PROJECT_DIR is based on $builds_dir, so if the
generate-stress-test-configs.py script generates jobs with
$CI_PROJECT_DIR (or variables like $INSTALL_PATH that are based on it)
evaluated, it is calcified to whatever was the value in the particular
environment, disregarding the FreeBSD "instance" executor specifics in
the child pipeline.
Instead of evaluating $CI_PROJECT_DIR in the script, evaluate it in the
runtime environment.
Backport of MR !10075
Merge branch 'backport-mnowak/fix-CI_PROJECT_DIR-variable-evaluation-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10077
GitLab CI Runner's $builds_dir variable is set to "/builds" by default.
For technical reasons, the FreeBSD Runners, using the "instance"
executor, sets the path differently.
The value of $CI_PROJECT_DIR is based on $builds_dir, so if the
generate-stress-test-configs.py script generates jobs with
$CI_PROJECT_DIR (or variables like $INSTALL_PATH that are based on it)
evaluated, it is calcified to whatever was the value in the particular
environment, disregarding the FreeBSD "instance" executor specifics in
the child pipeline.
Instead of evaluating $CI_PROJECT_DIR in the script, evaluate it in the
runtime environment.
(cherry picked from commit dab7d28b09)
The new command is `rndc memprof`. The memory profiling status is also
reported inside `rndc status`. The status also shows whether named can
toggle memory profiling or not and if the server is built with jemalloc.
Closes#4759
Backport of MR !9370
Merge branch 'backport-4759-add-a-trigger-to-dump-jeprof-data-or-memory-statistics-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10000
The new command is `rndc memprof`. The memory profiling status is also
reported inside `rndc status`. The status also shows whether named can
toggle memory profiling or not and if the server is built with jemalloc.
(cherry picked from commit b495e9918e)
Instead of having many node_lock_count * sizeof(<member>) arrays, pack all
the members into a qpcache_bucket_t that is cacheline aligned to prevent
false sharing between RWLocks.
Backport of MR !10072
Merge branch 'backport-ondrej/prevent-nodelock-false-sharing-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10074
Instead of using on hash of the name modulo number of the buckets,
assign the locknum randomly with isc_random_uniform(). This makes
the locknum assignment aligned with qpcache and allows the bucket
number to be non-prime in the future.
(cherry picked from commit 732fc338a9)
Reduce the number of qpzone_ref() and qpzone_unref() calls in
qpzone_detachnode() by relying on the call_rcu to delay
the destruction of the lock buckets.
(cherry picked from commit 1fa5219fdf)
Instead of having many node_lock_count * sizeof(<member>) arrays, pack
all the members into a qpzone_bucket_t that is cacheline aligned and have
a single array of those.
(cherry picked from commit 6dcc398726)
Instead of having many node_lock_count * sizeof(<member>) arrays, pack
all the members into a qpcache_bucket_t struct that is cacheline aligned
and have a single array of those.
Additionaly, make both the head and the tail of isc_queue_t padded, not
just the head, to prevent false sharing of the lock-free structure with
the lock that follows it.
(cherry picked from commit c602d76c1f)
Print the expiration time of the stale RRsets in the cache dump.
Backport of MR !10057
Merge branch 'backport-ondrej/print-expiration-time-of-stale-records-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10061
In #1870, the expiration time of ANCIENT records were printed, but
actually the ancient records are very short lived, and the information
carries a little value.
Instead of printing the expiration of ANCIENT records, print the
expiration time of STALE records.
(cherry picked from commit 355fc48472)
In some cases, the numeric identifier doesn't correspond to the
directory name (i.e. `resolver` server in `shutdown` test, which is
supposed to be 10.53.0.3). These are typically servers that shouldn't be
auto-started by the runner, thus avoiding the typical `*ns<X>` name.
Support these server by allowing a fallback initialization with custom
numeric identifier in case it can't be parsed from the directory name.
(cherry picked from commit a24f71bae4)
The start()/stop() functions can be used in the pytests in the same way
as start_server and stop_server functions were used in shell tests. Note
that the servers obtained through the servers fixture are still started
and stopped by the test runner at the start and end of the test. This
makes these functions mostly useful for restarting the server(s)
mid-test.
(cherry picked from commit 37699ad84b)
Previously, these functions have been provided as fixtures. This was
limiting re-use, because it wasn't possible to call these outside of
tests / other fixtures without passing these utility functions around.
Move them into isctest.run package instead.
(cherry picked from commit b6d645410c)
Closes#5157
Backport of MR !10067
Merge branch 'backport-5157-suppress-lsan-libxml2-leak-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10068
The ANS servers were not to written to handle NS queries at the QNAME, resulting in gratuitous protocol errors that will break tests when NS requests are made for the QNAME: i.e., NXDOMAIN for NS vs data for expected type, CNAME not being returned for all query types.
Prerequisite for !9155Closes#5062
Backport of MR !9786
Merge branch 'backport-5062-fix-ans-servers-ns-at-qname-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10064
The ANS servers were not to written to handle NS queries at the
QNAME resulting in gratuitious protocol errors that will break tests
when NS requests are made for the QNAME.
(cherry picked from commit 0680eb6f64)
Under rare circumstances, the RRSet that expired at the time of
the query could be returned with TTL far in the future. This
has been fixed.
As a side-effect, the expiration time of expired RRSets are no
longer printed out in the cache dump.
Closes#5094
Backport of MR !10048
Merge branch 'backport-5094-fix-timestamp-in-ttl-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10059
When the mark_ancient() helper function was introduced, couple of places
with duplicate (or almost duplicate) code was missed. Move the
mark_ancient() function closer to the top of the file, and correctly use
it in places that mark the header as ANCIENT.
(cherry picked from commit 58179e6a19)
If we know that the header has ZEROTTL set, the server should never send
stale records for it and the TTL should never be anything else than 0.
The comment was already there, but the code was not matching the
comment.
(cherry picked from commit cfee6aa565)
When the header has been marked as ANCIENT, but the ttl hasn't been
reset (this happens in couple of places), the rdataset TTL would be
set to the header timestamp instead to a reasonable TTL value.
Since this header has been already expired (ANCIENT is set), set the
rdataset TTL to 0 and don't reuse this field to print the expiration
time when dumping the cache. Instead of printing the time, we now
just print 'expired (awaiting cleanup'.
(cherry picked from commit 1bbb57f81b)
The search for the deepest known zone cut in the cache could improperly reject a node if it contained any stale data, regardless of whether it was the NS RRset that was stale.
Closes#5155
Backport of MR !10047
Merge branch 'backport-5155-fix-findzonecut-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10050
the search for the deepest known zone cut in the cache could
improperly reject a node containing stale data, even if the
NS rdataset wasn't the data that was stale.
this change also improves the efficiency of the search by
stopping it when both NS and RRSIG(NS) have been found.
(cherry picked from commit 1f095b902c)
Backport of MR !10029
Backport of MR !10042
Merge branch 'backport-pspacek/no-cross-project-after-merge-jobs-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10043
We need to avoid double-triggering of post-merge jobs in the following
scenario:
1. A private MR gets merged into the private BIND 9 repository.
2. This merge operation triggers a "push" pipeline in the private
repository, which correctly runs post-merge jobs, e.g. to set MR
metadata in the private project.
3. When a release is published, a script is run to change the
automatically assigned milestone value ("Not released yet") to
something else.
4. Shortly afterwards, the result of the merge from step 1 is merged
back into a maintenance branch in the public repository.
5. The push operation triggers another "push" pipeline, this time in
the public project.
At this point there are two problems:
- If the script is dumb (like it currently is), it will extract the
merge request ID from the merge commit description and change the
milestone for a merge request in the wrong project namespace.
- Even if the script was fixed to extract and use the correct GitLab
project reference, it would reset the milestone for the merge
request in the private repository back to "Not released yet" - while
the milestone set in step 3 should be retained.
An alternative would be to change the order of operations so that
post-release milestoning happens at a later stage, while also fixing the
script to correctly follow cross-project references, but that approach
seems more fragile than simply failing on all cross-project pushes. The
rule to enforce is: each project should only take care of its own
post-merge tasks.
(cherry picked from commit 6014060774)
With shallow fetching working reliably in pygit2 1.17.0+, there is no
longer any need for GitLab CI runners to clone the BIND 9 repository
with a fixed depth of 1000 during every "danger" CI job as Hazard is now
able to fetch remote refs with an arbitrary depth, controlled by the
HAZARD_FETCH_DEPTH environment variable. The latter can be defined via
GitLab project's CI settings and adjusted as needed over time, without
the need to update .gitlab-ci.yml every time its value needs to be
changed.
Backport of MR !9946
Merge branch 'backport-michal/use-default-cloning-depth-for-the-danger-ci-job-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10037
With shallow fetching working reliably in pygit2 1.17.0+, there is no
longer any need for GitLab CI runners to clone the BIND 9 repository
with a fixed depth of 1000 during every "danger" CI job as Hazard is now
able to fetch remote refs with an arbitrary depth, controlled by the
HAZARD_FETCH_DEPTH environment variable. The latter can be defined via
GitLab project's CI settings and adjusted as needed over time, without
the need to update .gitlab-ci.yml every time its value needs to be
changed.
(cherry picked from commit e39e7afc16)
Clean up the pattern in the newref() and decref() functions in QP and RBTDB databases. Replace the `db_nodelock_t` structure with plain reference counting for every active database node in QPDB.
Related to #5134
Backport of MR !10006
Merge branch 'backport-5134-refactor-decref-functions-in-qpdb-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10035
Change the names of the node reference counting functions
and add comments to make the mechanism easier to understand:
- dns__rbtdb_newref() and dns__rbtdb_decref() are now called
dns__rbtnode_acquire() and dns__rbtnode_release()
respectively; this reflects the fact that they modify both
the internal and external reference counters for a node.
- rbtnode_newref() and rbtnode_decref are now called
rbtnode_erefs_increment() and rbtnode_erefs_decrement(),
to reflect that they only increase and decrease the node's
external reference counters, not internal.
Refactor the pattern in the newref() and decref() functions in rbtdb.c
following the pattern, so it follows the similar pattern we already have
for QPDB.
Change the names of the node reference counting functions
and add comments to make the mechanism easier to understand:
- newref() and decref() are now called qpcnode_acquire()/
qpznode_acquire() and qpcnode_release()/qpznode_release()
respectively; this reflects the fact that they modify both
the internal and external reference counters for a node.
- qpcnode_newref() and qpznode_newref() are now called
qpcnode_erefs_increment() and qpznode_erefs_increment(), and
qpcnode_decref() and qpznode_decref() are now called
qpcnode_erefs_decrement() and qpznode_erefs_decrement(),
to reflect that they only increase and decrease the node's
external reference counters, not internal.
(cherry picked from commit d4f791793e)
This removes the db_nodelock_t structure and changes the node_locks
array to be composed only of isc_rwlock_t pointers. The .reference
member has been moved to qpdb->references in addition to
common.references that's external to dns_db API users. The .exiting
members has been completely removed as it has no use when the reference
counting is used correctly.
(cherry picked from commit 431513d8b3)
The origin_node in qpcache was always NULL, so we can remove the
getoriginode() function and origin_node pointer as the
dns_db_getoriginnode() correctly returns ISC_R_NOTFOUND when the
function is not implemented.
(cherry picked from commit 36a26bfa1a)
Cleanup the pattern in the decref() functions in both qpcache.c and
qpzone.c, so it follows the similar patter as we already have in
newref() function.
(cherry picked from commit 814b87da64)
Extended DNS error 22 (No reachable authority) was previously detected when `fctx_expired` fired. It turns out this function is used as a "safety net" and the timeout detection should be caught earlier.
It was working though, because of another issue fixed by !9927. But then, the recursive request timed out detection occurs before `fctx_expired` making impossible to raise the EDE 22 error.
This fixes the problem by triggering the EDE 22 in the part of the code detecting the (TCP or UDP) time out and taking the decision to cancel the whole fetch (i.e. There is no other server to attempt to contact).
Note this is not targeting users (no release note) because there is no release versions of BIND between !9927 and this changes. Thus a release note would be confusing.
Closes#5137
Backport of MR !9985
Merge branch 'backport-5137-ede22-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10001
When EDE 3 (stale answer) was added the serve-stale tests were checking
for those exclusively, i.e. grepping for no "EDE" in the dig output when
no stale answer was expected.
However, some stale tests disable stale answers and make the
authoritative server unresponsive, effectively triggering a timed out
request thus an EDE 22. Update those tests so they still tests the
absence of EDE 3 error, but also the presence of EDE 22.
(cherry picked from commit 27f3b8950a)
This re-do a previously existing EDE 22 system test as well as add
another one making sure the timed out flow detection works also on UDP
when the resolver is contacting the authoritative server. (the existing
test was using TCP to contact the authoritative servers).
(cherry picked from commit 7cb8a028fe)
