Is there a time when new_qp(c|z)node() would not be followed by
assignment of the namespace? No, so let's add the assignment to the
function that creates the node.
Now that we have to code working, rename 'dns_qp_lookup2' back to
'dns_qp_lookup' and adjust all remaining 'dns_qp_lookup' occurrences
to take a space=0 parameter.
For now we only allow DNS_DB_NSEC_* values so it makes sense to change
the type to an enum.
Rename 'denial' to the more intuitive 'space', indicating the namespace
of the keyvalue pair.
The dbiterator can take three modes: full, nsec3only and nonsec3.
Previously, in full mode the dbiterator requires special logic to jump
from one qp-trie to the other. Now everything is in one trie, other
special logic is needed.
The qp-trie is now sorted in such a way that all the normal nodes come
first, followed by NSEC nodes, and finally the NSEC3 nodes. NSEC nodes
are empty nodes and need to be skipped when iterating.
We add an additional auxiliary node to the trie, an NSEC origin, so
we can easily find the point in the trie where we need to continue
iterating.
If zone and denial data are going to be stored in the same qp storage,
the unit tests need to be updated to reflect this change. The code
changes mainly affect name to qpkey conversion, lookups, and
predecessors.
A note on predecessors: since the denial and zone data are now in the
same qp storage, the predecessor of the first name in the zone data will
consequently be the last name in the denial data.
In preparation to merge the three qp tries (tree, nsec, nsec3) into
one, add the piece of information into the qpkey. This is the most
significant bit of information, so prepend the denial type to the qpkey.
This means we need to pass on the denial type when constructing the
qpkey from a name, or doing a lookup.
Reuse the the DNS_DB_NSEC_* values. Most qp tries in the code we just
pass on 0 (nta, rpz, zt, etc.), because there is no need for denial of
existence, but for qpzone and qpcache we must pass the right value.
Change the code, so that node->nsec no longer can have the value
DNS_DB_NSEC_HAS_NSEC, instead track this in a new attribute 'havensec'.
Since we use node->nsec to convert names to keys, the value MUST be set
before inserting the node into the qp-trie.
Update the fuzzing and unit tests accordingly. This only adds a few
extra test cases, more are needed.
In the qp_test.c we can remove test code for empty keys as this is
no longer possible.
When used with the ``+keepopen`` option with a TCP connection, iscman:`dig`
could terminate unexpectedly in rare situations. Additionally, iscman:`dig`
could hang and fail to shutdown properly when interrupted during a query.
These have been fixed.
Closes#5381
Merge branch '5381-dig-keepalive-crash' into 'main'
See merge request isc-projects/bind9!10681
When send_done() is called with a ISC_R_CANCELED status (e.g. because
of a signal from ctrl+c), dig can fail to shutdown because
check_if_done() is not called in the branch. Add a check_if_done()
call.
When reusing a TCP connection (because of the '+keepopen' option),
dig detaches from the query after launching it. This can cause a
crash in dig in rare cases when the "receive" callback is called
earlier than the "send" callback.
The '_cancel_lookup()' function detaches a query only if it's
found in the 'lookup->q' list. Before this commit, with one
additional detach happening before recv_done() -> _cancel_lookup()
is called, it didn't cause problems because an earlier _query_detach()
was unlinking the query from 'lookup->q' (because it was the last
reference), so the additional detach and the skipped detach were
undoing each other.
That is unless the "receive" callback was called earlier than the
"send" callback, in which case the additional detach wasn't destroying
the query (and wasn't unlinking it from 'lookup->q') because the "send"
callback's attachment was still there, and so _cancel_lookup() was
trying to "steal" the "send" callback's attachment and causing an
assertion on 'INSIST(query->sendhandle == NULL);'.
Delete the detachment which caused the described situation.
The util/meson-dist-package.sh script hardcodes the date it is run on
into the man pages it creates in the dist tarball. This causes pkgdiff
to report discrepancies if the util/release-tarball-comparison.sh script
is run on a different day than the one the dist tarball was generated
on.
Fix by using the exact same solution as in BIND 9.20: generating the man
page stubs with a @RELEASE_DATE@ placeholder instead of a specific date
and only replacing that placeholder with a specific date during the
build process.
Closes#5412
Merge branch '5412-do-not-hardcode-release-date-in-man-pages' into 'main'
See merge request isc-projects/bind9!10716
The util/meson-dist-package.sh script hardcodes the date it is run on
into the man pages it creates in the dist tarball. This causes pkgdiff
to report discrepancies if the util/release-tarball-comparison.sh script
is run on a different day than the one the dist tarball was generated
on.
Fix by using the exact same solution as in BIND 9.20: generating the man
page stubs with a @RELEASE_DATE@ placeholder instead of a specific date
and only replacing that placeholder with a specific date during the
build process.
RRset ordering is now an enum inside struct rdataset attributes. This
was done to keep size to of the structure to its original value before
this MR.
I expect zero performance impact but it should be easier to deal with
attributes in debuggers and language servers.
Responses which were dropped or slipped because of RRL (Response Rate
Limiting) were logged in the ``rate-limit`` category instead of the
``query-errors`` category, as documented in ARM. This has been fixed.
Closes#5388
Merge branch '5388-rrl-log-category-fix' into 'main'
See merge request isc-projects/bind9!10676
As mentioned in the comments block before the changed code block,
the dropped or slipped responses should be logged in the query
category (or rather query-errors category as done in lib/ns/client.c),
so that requests are not silently lost.
Also fix a couple of errors/typos in the code comments.
The ns_client_t struct is reset and zeroed out on every query,
but some fields (query, message, manager) are preserved.
We observe two things:
- The sendbuf field is going to be overwritten anyway, there's
no need to zero it out.
- The fields are copied out when the struct is zero-ed out, and
then copied back in. For the query field (which is 896 bytes)
this is very inefficient.
This commit makes the reset more efficient by avoiding the unnecessary
zeroing and copying.
Merge branch 'alessio/experimental-ns-client-noinit' into 'main'
See merge request isc-projects/bind9!10463
The ns_client_t struct is reset and zero-ed out on every query,
but some fields (query, message, manager) are preserved.
We observe two things:
- The sendbuf field is going to be overwritten anyway, there's
no need to zero it out.
- The fields are copied out when the struct is zero-ed out, and
then copied back in. For the query field (which is 896 bytes)
this is very inefficient.
This commit makes the reset more efficient avoiding to unnecessary
zero-ing and copy.
This MR reduces lock contention and increases scalability in the ADB by:
a) Using SIEVE algorithm instead of classical LRU;
b) Replacing rwlocked isc_hashmap with RCU cds_lfht table;
c) Replace the single LRU table per-object with per-loop LRU tables per-object.
Merge branch 'ondrej/use-urcu-lfht-for-ADB-tables' into 'main'
See merge request isc-projects/bind9!10645
Replace the read-write locked isc_hashmap with lock-free cds_lfht
hashtable and replace the singular LRU tables for ADB names and entries
with a per-thread LRU tables. These changes allowed to remove all the
read-write locking on the names and entries tables.
Instead of having hand crafted attach/detach/destroy functions, replace
them with the standard ISC_REFCOUNT macro. This also have advantage
that delayed netmgr detach (from dns_dispatch) now doesn't cause
assertion failure. This can happen with delayed (call_rcu) shutdown of
dns_adb.
The dns_adb cleaning is little bit muddled as it mixes the "TTL"
based cleaning (.expire_v4 and .expire_v6 for adbname, .expires for
adbentry) with overmem cleaning.
Rewrite the LRU based cleaning to use SIEVE algorithm and to be overmem
cleaning only with a requirement to always cleanup at least 2-times the
size of the newly added entry.
Qpzone employs a locking strategy where rwlocks are grouped into
buckets, and each zone gets 17 buckets.
This strategy is suboptimal in two ways:
- If named is serving a single zone or a zone is the majority of the
traffic, this strategy pretty much guarantees contention when using
more than a dozen threads.
- If named is serving many small zones, it causes substantial memory
usage.
This commit switches the locking to a global table initialized at start
time. This should have three effects:
- Performance should improve in the single zone case, since now we are
selecting from a bigger pool of locks.
- Memory consumption should go down significantly in the many zone
cases.
- Performance should not degrade substantially in the many zone cases.
The reason for this is that, while we could have substantially more
zones than locks, we can query/edit only O(num threads) at the same
time. So by making the global table much bigger than the expected
number of threads, we can limit contention.
Merge branch 'alessio/global-qpzone-lock-table' into 'main'
See merge request isc-projects/bind9!10446
Qpzone employs a locking strategy where rwlocks are grouped into
buckets, and each zone gets 17 buckets.
This strategy is suboptimal in two ways:
- If named is serving a single zone or a zone is the majority of the
traffic, this strategy pretty much guarantees contention when using
more than a dozen threads.
- If named is serving many small zones, it causes substantial memory
usage.
This commit switches the locking to a global table initialized at start
time. This should have three effects:
- Performance should improve in the single zone case, since now we are
selecting from a bigger pool of locks.
- Memory consumption should go down significantly in the many zone
cases.
- Performance should not degrade substantially in the many zone cases.
The reason for this is that, while we could have substantially more
zones than locks, we can query/edit only O(num threads) at the same
time. So by making the global table much bigger than the expected
number of threads, we can limit contention.
In the current implementation, the resigning heap is part of the zone
database. This leads to a cycle, as the database has a reference to its
nodes, but each node needs a reference to the database.
This MR splits the resigning heap into its own separate struct, in order
to help breaking the cycle.
Merge branch 'alessio/split-qpzone-heap-from-qpdb' into 'main'
See merge request isc-projects/bind9!10706
In the current implementation, the resigning heap is part of the zone
database. This leads to a cycle, as the database has a reference to its
nodes, but each node needs a reference to the database.
This MR splits the resigning heap into its own separate struct, in order
to help breaking the cycle.
Recovering the node lock from a pointer to the header and a pointer to
the db is a common operation. This commit abstracts it away into a
function, so that the node lock selection logic may be modified more
easily.
A query for a zone that was not yet loaded may yield an unexpected result such as a CNAME or DNAME, triggering an assertion failure. This has been fixed.
Closes#5357
Merge branch '5357-resume-qmin-cname' into 'main'
See merge request isc-projects/bind9!10562
Print optionally a bit more details not passed to event in case
dns_view_findzonecut returns unexpected result. Result would be
visible later in foundevent, but found fname would be lost. Print it
into the log.
When authoritative zone is loaded when query minimization query for the
same zone is already pending, it might receive unexpected result codes.
Normally DNS_R_CNAME would follow to query_cname after processing sent
events, but dns_view_findzonecut does not fill CNAME target into
event->foundevent. Usual lookup via query_lookup would always have that
filled.
Ideally we would restart the query with unmodified search name, if
unexpected change from recursing to local zone cut were detected. Until
dns_view_findzonecut is modified to export zone/cache source of the cut,
at least fail queries which went into unexpected state.
The man page for :iscman:`named-makejournal` was erroneously not
installed when building from a source tarball. This has been fixed.
See #5379
Merge branch '5379-fix-named-makejournal-man-page-installation' into 'main'
See merge request isc-projects/bind9!10709
The man page for named-makejournal is erroneously not installed when
building from a source tarball. Add that man page to the appropriate
lists in the build system so that it is installed both when building
from a Git repository and from a source tarball.
System tests' after_script missed the PYTHON environmental variable
setup.
$ find -name 'tsan.*' -exec "$PYTHON" util/parse_tsan.py {} \;
find: '': No such file or directory
Merge branch 'mnowak/fix-parse_tsan-invocation' into 'main'
See merge request isc-projects/bind9!10683
System tests' after_script missed the PYTHON environmental variable
setup.
$ find -name 'tsan.*' -exec "$PYTHON" util/parse_tsan.py {} \;
find: '': No such file or directory
The ADB memory cleaning is opportunistic even when we are under
memory pressure (in the overmem condition). Split the opportunistic
LRU cleaning and overmem cleaning and make the overmem cleaning
always cleanup double of the newly allocated adbname/adbentry to
ensure we never allocate more memory than the assigned limit.
Merge branch 'ondrej/enforce-memory-cleanup-in-ADB-when-overmem' into 'main'
See merge request isc-projects/bind9!10637
The purge_stale_names()/purge_stale_entries() is opportunistic even when
we are under memory pressure (overmem). Split the opportunistic LRU
cleaning and overmem cleaning. This makes the stale purging much
simpler as we don't have to try that hard and makes the overmem cleaning
always cleanup double the amount of the newly allocated ADB name/entry.
Use the native shared library extension when build loadable
libaries. For most platforms this is ".so" but for Darwin it
is ".dylib".
Closes#5375
Merge branch '5375-use-native-shared-library-extension' into 'main'
See merge request isc-projects/bind9!10588
