Before this update, BIND would attempt to do a full recursive resolution
process for each query received if the requested rrset had its ttl
expired. If the resolution fails for any reason, only then BIND would
check for stale rrset in cache (if 'stale-cache-enable' and
'stale-answer-enable' is on).
The problem with this approach is that if an authoritative server is
unreachable or is failing to respond, it is very unlikely that the
problem will be fixed in the next seconds.
A better approach to improve performance in those cases, is to mark the
moment in which a resolution failed, and if new queries arrive for that
same rrset, try to respond directly from the stale cache, and do that
for a window of time configured via 'stale-refresh-time'.
Only when this interval expires we then try to do a normal refresh of
the rrset.
The logic behind this commit is as following:
- In query.c / query_gotanswer(), if the test of 'result' variable falls
to the default case, an error is assumed to have happened, and a call
to 'query_usestale()' is made to check if serving of stale rrset is
enabled in configuration.
- If serving of stale answers is enabled, a flag will be turned on in
the query context to look for stale records:
query.c:6839
qctx->client->query.dboptions |= DNS_DBFIND_STALEOK;
- A call to query_lookup() will be made again, inside it a call to
'dns_db_findext()' is made, which in turn will invoke rbdb.c /
cache_find().
- In rbtdb.c / cache_find() the important bits of this change is the
call to 'check_stale_header()', which is a function that yields true
if we should skip the stale entry, or false if we should consider it.
- In check_stale_header() we now check if the DNS_DBFIND_STALEOK option
is set, if that is the case we know that this new search for stale
records was made due to a failure in a normal resolution, so we keep
track of the time in which the failured occured in rbtdb.c:4559:
header->last_refresh_fail_ts = search->now;
- In check_stale_header(), if DNS_DBFIND_STALEOK is not set, then we
know this is a normal lookup, if the record is stale and the query
time is between last failure time + stale-refresh-time window, then
we return false so cache_find() knows it can consider this stale
rrset entry to return as a response.
The last additions are two new methods to the database interface:
- setservestale_refresh
- getservestale_refresh
Those were added so rbtdb can be aware of the value set in configuration
option, since in that level we have no access to the view object.
There were several problems with rbt hashtable implementation:
1. Our internal hashing function returns uint64_t value, but it was
silently truncated to unsigned int in dns_name_hash() and
dns_name_fullhash() functions. As the SipHash 2-4 higher bits are
more random, we need to use the upper half of the return value.
2. The hashtable implementation in rbt.c was using modulo to pick the
slot number for the hash table. This has several problems because
modulo is: a) slow, b) oblivious to patterns in the input data. This
could lead to very uneven distribution of the hashed data in the
hashtable. Combined with the single-linked lists we use, it could
really hog-down the lookup and removal of the nodes from the rbt
tree[a]. The Fibonacci Hashing is much better fit for the hashtable
function here. For longer description, read "Fibonacci Hashing: The
Optimization that the World Forgot"[b] or just look at the Linux
kernel. Also this will make Diego very happy :).
3. The hashtable would rehash every time the number of nodes in the rbt
tree would exceed 3 * (hashtable size). The overcommit will make the
uneven distribution in the hashtable even worse, but the main problem
lies in the rehashing - every time the database grows beyond the
limit, each subsequent rehashing will be much slower. The mitigation
here is letting the rbt know how big the cache can grown and
pre-allocate the hashtable to be big enough to actually never need to
rehash. This will consume more memory at the start, but since the
size of the hashtable is capped to `1 << 32` (e.g. 4 mio entries), it
will only consume maximum of 32GB of memory for hashtable in the
worst case (and max-cache-size would need to be set to more than
4TB). Calling the dns_db_adjusthashsize() will also cap the maximum
size of the hashtable to the pre-computed number of bits, so it won't
try to consume more gigabytes of memory than available for the
database.
FIXME: What is the average size of the rbt node that gets hashed? I
chose the pagesize (4k) as initial value to precompute the size of
the hashtable, but the value is based on feeling and not any real
data.
For future work, there are more places where we use result of the hash
value modulo some small number and that would benefit from Fibonacci
Hashing to get better distribution.
Notes:
a. A doubly linked list should be used here to speedup the removal of
the entries from the hashtable.
b. https://probablydance.com/2018/06/16/fibonacci-hashing-the-optimization-that-the-world-forgot-or-a-better-alternative-to-integer-modulo/
Both clang-tidy and uncrustify chokes on statement like this:
for (...)
if (...)
break;
This commit uses a very simple semantic patch (below) to add braces around such
statements.
Semantic patch used:
@@
statement S;
expression E;
@@
while (...)
- if (E) S
+ { if (E) { S } }
@@
statement S;
expression E;
@@
for (...;...;...)
- if (E) S
+ { if (E) { S } }
@@
statement S;
expression E;
@@
if (...)
- if (E) S
+ { if (E) { S } }
Using isc_mem_put(mctx, ...) + isc_mem_detach(mctx) required juggling with the
local variables when mctx was part of the freed object. The isc_mem_putanddetach
function can handle this case internally, but it wasn't used everywhere. This
commit apply the semantic patching plus bit of manual work to replace all such
occurrences with proper usage of isc_mem_putanddetach().
allows named to provide stale cached answers when
the authoritative server is under attack.
See max-stale-ttl, stale-answer-enable,
stale-answer-ttl. [RT #44790]
provisioning secondary servers in which a list of
zones to be served is stored in a DNS zone and can
be propagated to slaves via AXFR/IXFR. [RT #41581]
4375. [func] Add support for automatic reallocation of isc_buffer
to isc_buffer_put* functions. [RT #42394]
commit aea73609ac5d41ed091360e94370798965f28f05
commit eef7f44c57a060b24a426eb8888e16176a0a69b1
commit a88a26d864ad399fa2d40e3b9659b4d26f454ca1
commit 1b90d59568e7e3b65690c6bd075cf4d60b03e454
Merge: 74d8f73 cd02924
commit 74d8f73ed553bb64a305e284905762f7ff0029aa
commit 9a59ef6bbd4befe91e5691e8b85afe1cb7ab0706
commit c63606a53b4f1bb7066b37d3cfe588e9dc21a119
commit 2c392a840c8838455d144ce163bd873bee400c97
commit 0241f53563e6e7bed462a883d98a8931f01e0980
commit 79fe22b5d6f04bdaa3073cf54d41952194e879e1
commit 351b3049625f2edd39729dd85413e961b97d4b3b
commit 7207674fc77c9a10d84c0cb94e36d1c09bb31459
commit 543ad34cf08f901c20b438c9d2f45482cff13d5e
commit fc45b99ce4438627fdcbeb4365695ba0065fa46f
commit c425207f57e0a5157372aa7edbb79b13170563e5
commit ef8c5e23ca284e0ea02f69ce1f356d537c19d93b
commit ba0d4e3aa51efe412cfa1d031651f949442d1802
commit 41c7969c7cb6884b93011f7ace3fd9522efc021e
and more from CVS
for rt26172
Add
- optional "recursive-only yes|no" to the response-policy statement
- optional max-policy-ttl to limit the lies that "recursive-only no"
can introduce into resolvers' caches
- test that queries with RD=0 are not rewritten by default
- performance smoke test
Change encoding of PASSTHRU action to "rpz-passthru".
(The old encoding is still accepted.)
Fix rt26180 assert botch in zone_findrdataset() in this branch
as well.
Fix missing signatures on NOERROR results despite RPZ hits
when there are signatures and the client asks for DNSSEC,
This includes the following changes:
3326. [func] Added task list statistics: task model, worker
threads, quantum, tasks running, tasks ready.
[RT #27678]
3325. [func] Report cache statistics: memory use, number of
nodes, number of hash buckets, hit and miss counts.
[RT #27056]
3324. [test] Add better tests for ADB stats [RT #27057]
3323. [func] Report the number of buckets the resolver is using.
[RT #27020]
3322. [func] Monitor the number of active TCP and UDP dispatches.
[RT #27055]
3321. [func] Monitor the number of recursive fetches and the
number of open sockets, and report these values in
the statistics channel. [RT #27054]
3320. [func] Added support for monitoring of recursing client
count. [RT #27009]
3319. [func] Added support for monitoring of ADB entry count and
hash size. [RT #27057]
- fix precedence among competing rules
- improve ARM text including documenting rule precedence
- try to rewrite CNAME chains until first hit
- new "rpz" logging channel
- same fix for "NS ." as in RT 24985
update are now fully supported and no longer require
defines to enable. We now no longer overload the
NSEC3PARAM flag field, nor the NSEC OPT bit at the
apex. Secure to insecure changes are controlled by
by the named.conf option 'secure-to-insecure'.
Warning: If you had previously enabled support by
adding defines at compile time to BIND 9.6 you should
ensure that all changes that are in progress have
completed prior to upgrading to BIND 9.7. BIND 9.7
is not backwards compatible.
dnssec-* tools. Major changes:
- all dnssec-* tools now take a -K option to
specify a directory in which key files will be
stored
- DNSSEC can now store metadata indicating when
they are scheduled to be published, acttivated,
revoked or removed; these values can be set by
dnssec-keygen or overwritten by the new
dnssec-settime command
- dnssec-signzone -S (for "smart") option reads key
metadata and uses it to determine automatically
which keys to publish to the zone, use for
signing, revoke, or remove from the zone
[RT #19816]
