mir/bind
2
0
Fork 0
mirror of https://gitlab.isc.org/isc-projects/bind9 synced 2025-08-24 11:08:45 +00:00
Code Issues Releases Wiki Activity
bind/lib/dns/adb.c

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

3567 lines
89 KiB
C
Raw Normal View History

Start on address.c
1999-09-21 22:46:42 +00:00
/*
formally cast to int
2018-02-14 19:13:56 +11:00
* Copyright (C) Internet Systems Consortium, Inc. ("ISC")
Start on address.c
1999-09-21 22:46:42 +00:00
*
* SPDX-License-Identifier: MPL-2.0
Update the copyright information in all files in the repository This commit converts the license handling to adhere to the REUSE specification. It specifically: 1. Adds used licnses to LICENSES/ directory 2. Add "isc" template for adding the copyright boilerplate 3. Changes all source files to include copyright and SPDX license header, this includes all the C sources, documentation, zone files, configuration files. There are notes in the doc/dev/copyrights file on how to add correct headers to the new files. 4. Handle the rest that can't be modified via .reuse/dep5 file. The binary (or otherwise unmodifiable) files could have license places next to them in <foo>.license file, but this would lead to cluttered repository and most of the files handled in the .reuse/dep5 file are system test files.
2021-06-03 08:37:05 +02:00
*
Start on address.c
1999-09-21 22:46:42 +00:00
* This Source Code Form is subject to the terms of the Mozilla Public
* License, v. 2.0. If a copy of the MPL was not distributed with this
* file, you can obtain one at https://mozilla.org/MPL/2.0/.
Update license headers to not include years in copyright in all applicable files
2018-02-23 09:53:12 +01:00
*
Start on address.c
1999-09-21 22:46:42 +00:00
* See the COPYRIGHT file distributed with this work for additional
2947. [func] Add new zone type "static-stub". It's like a stub zone, but the nameserver names and/or their IP addresses are statically configured. [RT #21474] (for 9.8.0)
2010-12-16 09:51:30 +00:00
* information regarding copyright ownership.
Start on address.c
1999-09-21 22:46:42 +00:00
*/
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
/*! \file */
Start flushing out the address.c stuff
1999-09-23 00:43:10 +00:00
Replace custom isc_u?intNN_t types with C99 u?intNN_t types
2018-03-28 14:19:37 +02:00
#include <inttypes.h>
add limits.h for INT_MAX
2000-01-21 02:13:38 +00:00
#include <limits.h>
Replace custom isc_boolean_t with C standard bool type
2018-04-17 08:29:14 -07:00
#include <stdbool.h>
Megacommit of dozens of files. Cleanup of redundant/useless header file inclusion. ISC style lint, primarily for function declarations and standalone comments -- ie, those that appear on a line without any code, which should be written as follows: /* * This is a comment. */
2000-05-08 14:38:29 +00:00
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
#include <isc/async.h>
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
#include <isc/atomic.h>
Refactor isc_hashmap to accept custom match function Refactor isc_hashmap to allow custom matching functions. This allows us to have better tailored keys that don't require fixed uint8_t arrays, but can be composed of more fields from the stored data structure.
2023-08-28 10:08:59 +02:00
#include <isc/hash.h>
Use isc_hashmap instead of isc_ht in the dns_adb API Replace the use of isc_ht API with isc_hashmap API in the dns_adb database implementation. This requires extending the dns_adbnamebucket_t and dns_adbentrybucket_t structures to include keysize and copy of the key because the isc_hashmap API needs the raw key in case of resizing the hashmap table.
2022-06-27 12:46:06 +02:00
#include <isc/hashmap.h>
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
#include <isc/list.h>
Create per-thread task for dns_adb resolver fetches The dns_adb would serialize all fetches on a single task. Create a per-thread task, so the fetches will stay local to the thread that initiated the fetch.
2022-10-13 08:11:30 +02:00
#include <isc/loop.h>
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
#include <isc/mutex.h>
dump the ADB in a more user-friendly format (unless a debug flag is set)
2000-12-18 20:03:33 +00:00
#include <isc/netaddr.h>
snapshot
1999-10-20 23:26:06 +00:00
#include <isc/random.h>
Make isc_result a static enum Remove the dynamic registration of result codes. Convert isc_result_t from unsigned + #defines into 32-bit enum type in grand unified <isc/result.h> header. Keep the existing values of the result codes even at the expense of the description and identifier tables being unnecessary large. Additionally, add couple of: switch (result) { [...] default: break; } statements where compiler now complains about missing enum values in the switch statement.
2021-10-04 17:14:53 +02:00
#include <isc/result.h>
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
#include <isc/rwlock.h>
2537. [func] Added more statistics counters including those on socket I/O events and query RTT histograms. [RT #18802]
2009-01-27 22:30:00 +00:00
#include <isc/stats.h>
Clean up #include <isc/string.h> It isn't just about HP/UX any more.
2022-04-11 15:53:34 +01:00
#include <isc/string.h>
Create per-thread task for dns_adb resolver fetches The dns_adb would serialize all fetches on a single task. Create a per-thread task, so the fetches will stay local to the thread that initiated the fetch.
2022-10-13 08:11:30 +02:00
#include <isc/tid.h>
move util.h to <isc/util.h>
1999-12-16 22:24:22 +00:00
#include <isc/util.h>
snapshot address.[ch]
1999-09-22 00:26:20 +00:00
s/address/adb/
1999-10-22 01:02:38 +00:00
#include <dns/adb.h>
database lookups
1999-10-21 01:18:06 +00:00
#include <dns/db.h>
print v6 hooks too; use logging system; general cleanup
1999-10-29 22:28:57 +00:00
#include <dns/log.h>
snapshot
1999-10-05 23:50:43 +00:00
#include <dns/rdata.h>
#include <dns/rdataset.h>
clean up suspicious looking and incorrect uses of dns_name_fromregion
2000-10-07 00:09:28 +00:00
#include <dns/rdatastruct.h>
1889. [func] The lame cache is now done on a <qname,qclass,qtype> basis as some servers only appear to be lame for certain query types. [RT #14916]
2005-06-23 04:22:02 +00:00
#include <dns/rdatatype.h>
fix
1999-10-25 21:39:08 +00:00
#include <dns/resolver.h>
2355. [func] Extend the number statistics counters available. [RT #17590]
2008-04-03 05:55:52 +00:00
#include <dns/stats.h>
Resolver query forwarding to DoT-enabled upstream servers Implement TLS transport usage in the resolver. Use the configured TLS transport for the forwarders in the resolver.
2022-12-08 14:18:22 +00:00
#include <dns/transport.h>
Start flushing out the address.c stuff
1999-09-23 00:43:10 +00:00
commit lruttl to the mainline. A tag was set called skan_lruttl-mainline-base, and I will tag this as skan_lruttl-mainline-merge after this commit
2007-10-19 17:15:53 +00:00
#define DNS_ADB_MAGIC ISC_MAGIC('D', 'a', 'd', 'b')
#define DNS_ADB_VALID(x) ISC_MAGIC_VALID(x, DNS_ADB_MAGIC)
#define DNS_ADBNAME_MAGIC ISC_MAGIC('a', 'd', 'b', 'N')
#define DNS_ADBNAME_VALID(x) ISC_MAGIC_VALID(x, DNS_ADBNAME_MAGIC)
#define DNS_ADBNAMEHOOK_MAGIC ISC_MAGIC('a', 'd', 'N', 'H')
use isc/magic.h, and (unfortunately) export dns_adbhandle_t's definition
1999-09-25 01:25:46 +00:00
#define DNS_ADBNAMEHOOK_VALID(x) ISC_MAGIC_VALID(x, DNS_ADBNAMEHOOK_MAGIC)
commit lruttl to the mainline. A tag was set called skan_lruttl-mainline-base, and I will tag this as skan_lruttl-mainline-merge after this commit
2007-10-19 17:15:53 +00:00
#define DNS_ADBENTRY_MAGIC ISC_MAGIC('a', 'd', 'b', 'E')
#define DNS_ADBENTRY_VALID(x) ISC_MAGIC_VALID(x, DNS_ADBENTRY_MAGIC)
#define DNS_ADBFETCH_MAGIC ISC_MAGIC('a', 'd', 'F', '4')
#define DNS_ADBFETCH_VALID(x) ISC_MAGIC_VALID(x, DNS_ADBFETCH_MAGIC)
#define DNS_ADBFETCH6_MAGIC ISC_MAGIC('a', 'd', 'F', '6')
#define DNS_ADBFETCH6_VALID(x) ISC_MAGIC_VALID(x, DNS_ADBFETCH6_MAGIC)
1851. [doc] Doxygen comment markup. [RT #11398]
2005-04-27 04:57:32 +00:00
/*!
Ignore ipv6 mapped ipv4 addresses. Set a default TTL for type 3 negative cache elements. This is local to the ADB, and cannot leak to clients, so no specs are violated. Change a logging level to be less verbose.
1999-12-01 02:05:16 +00:00
* For type 3 negative cache entries, we will remember that the address is
change macro name from ADB_NCACHE_MINIMUM to ADB_CACHE_MINIMUM since it's used as a general purpose minimum ttl, and actually set the TTL to the minimum when re get a negative cache entry.
1999-12-11 02:30:24 +00:00
* broken for this long. XXXMLG This is also used for actual addresses, too.
1402. [cleanup] A6 has been moved to experimental and is no longer fully supported. developer: jinmei reviewer: marka
2002-11-27 09:52:58 +00:00
* The intent is to keep us from constantly asking about A/AAAA records
change macro name from ADB_NCACHE_MINIMUM to ADB_CACHE_MINIMUM since it's used as a general purpose minimum ttl, and actually set the TTL to the minimum when re get a negative cache entry.
1999-12-11 02:30:24 +00:00
* if the zone has extremely low TTLs.
Ignore ipv6 mapped ipv4 addresses. Set a default TTL for type 3 negative cache elements. This is local to the ADB, and cannot leak to clients, so no specs are violated. Change a logging level to be less verbose.
1999-12-01 02:05:16 +00:00
*/
commit lruttl to the mainline. A tag was set called skan_lruttl-mainline-base, and I will tag this as skan_lruttl-mainline-merge after this commit
2007-10-19 17:15:53 +00:00
#define ADB_CACHE_MINIMUM 10 /*%< seconds */
#define ADB_CACHE_MAXIMUM 86400 /*%< seconds (86400 = 24 hours) */
#define ADB_ENTRY_WINDOW 1800 /*%< seconds */
Start flushing out the address.c stuff
1999-09-23 00:43:10 +00:00
Use isc_hashmap instead of isc_ht in the dns_adb API Replace the use of isc_ht API with isc_hashmap API in the dns_adb database implementation. This requires extending the dns_adbnamebucket_t and dns_adbentrybucket_t structures to include keysize and copy of the key because the isc_hashmap API needs the raw key in case of resizing the hashmap table.
2022-06-27 12:46:06 +02:00
#ifndef ADB_HASH_BITS
#define ADB_HASH_BITS 12
#endif /* ifndef ADB_HASH_BITS */
commit lruttl to the mainline. A tag was set called skan_lruttl-mainline-base, and I will tag this as skan_lruttl-mainline-merge after this commit
2007-10-19 17:15:53 +00:00
/*%
* The period in seconds after which an ADB name entry is regarded as stale
* and forced to be cleaned up.
* TODO: This should probably be configurable at run-time.
*/
#ifndef ADB_STALE_MARGIN
#define ADB_STALE_MARGIN 1800
#endif /* ifndef ADB_STALE_MARGIN */
start on cleanup timer/task
1999-10-25 22:53:15 +00:00
silence compiler warnings
2013-03-05 23:41:22 +11:00
#define DNS_ADB_MINADBSIZE (1024U * 1024U) /*%< 1 Megabyte */
1073. [bug] The adb cache should also be space driven, max-adb-size. [RT #1915, #1938]
2001-10-25 04:57:46 +00:00
adjust what points to what. You are lost in a maze of twisty little pointers, all alike.
1999-09-22 01:01:14 +00:00
typedef ISC_LIST(dns_adbname_t) dns_adbnamelist_t;
snapshot address.[ch]
1999-09-22 00:26:20 +00:00
typedef struct dns_adbnamehook dns_adbnamehook_t;
more v4/v6 splitting out
1999-10-29 01:26:45 +00:00
typedef ISC_LIST(dns_adbnamehook_t) dns_adbnamehooklist_t;
Start flushing out the address.c stuff
1999-09-23 00:43:10 +00:00
typedef ISC_LIST(dns_adbentry_t) dns_adbentrylist_t;
start on fetch infrastructure
1999-10-25 21:02:54 +00:00
typedef struct dns_adbfetch dns_adbfetch_t;
a6 checkpoint
1999-10-30 01:22:16 +00:00
typedef struct dns_adbfetch6 dns_adbfetch6_t;
Start on address.c
1999-09-21 22:46:42 +00:00
1851. [doc] Doxygen comment markup. [RT #11398]
2005-04-27 04:57:32 +00:00
/*% dns adb structure */
Start on address.c
1999-09-21 22:46:42 +00:00
struct dns_adb {
update copyright notice
2008-02-07 23:46:54 +00:00
unsigned int magic;
commit lruttl to the mainline. A tag was set called skan_lruttl-mainline-base, and I will tag this as skan_lruttl-mainline-merge after this commit
2007-10-19 17:15:53 +00:00
update copyright notice
2008-02-07 23:46:54 +00:00
isc_mutex_t lock;
isc_mem_t *mctx;
Exclude the ADB hashmaps from ADB overmem accounting The ADB overmem accounting would include the memory used by hashtables thus vastly reducing the space that can be used for ADB names and entries when the hashtables would grow. Create own memory context for the ADB names and entries hash tables.
2022-12-13 14:14:21 +01:00
isc_mem_t *hmctx;
update copyright notice
2008-02-07 23:46:54 +00:00
dns_view_t *view;
attach the resolver to the ADB the ADB depends on the resolver, but previously only accessed it via the view. as view->resolver may now be detached before the ADB finishes, a shutdown race was possible. attaching to the resolver directly prevents this.
2022-05-12 15:51:10 -07:00
dns_resolver_t *res;
commit lruttl to the mainline. A tag was set called skan_lruttl-mainline-base, and I will tag this as skan_lruttl-mainline-merge after this commit
2007-10-19 17:15:53 +00:00
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
isc_refcount_t references;
update copyright notice
2008-02-07 23:46:54 +00:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
dns_adbnamelist_t names_lru;
isc_stdtime_t names_last_update;
isc_hashmap_t *names;
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
isc_rwlock_t names_lock;
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
dns_adbentrylist_t entries_lru;
isc_stdtime_t entries_last_update;
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
isc_hashmap_t *entries;
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
isc_rwlock_t entries_lock;
Use clang-format to reformat the source files
2020-02-12 13:59:18 +01:00
move ADB and resolver stats out of the view object for better object separation, ADB and resolver statistics counters are now stored in the ADB and resolver objects themsevles, rather than in the associated view.
2022-05-11 15:38:54 -07:00
isc_stats_t *stats;
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
atomic_bool exiting;
Use clang-format to reformat the source files
2020-02-12 13:59:18 +01:00
Replace custom isc_u?intNN_t types with C99 u?intNN_t types
2018-03-28 14:19:37 +02:00
uint32_t quota;
uint32_t atr_freq;
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
double atr_low;
double atr_high;
double atr_discount;
Start on address.c
1999-09-21 22:46:42 +00:00
};
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
/*%
* dns_adbname structure:
*
* This is the structure representing a nameserver name; it can be looked
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
* up via the adb->names hash table. It holds references to fetches
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
* for A and AAAA records while they are ongoing (fetch_a, fetch_aaaa), and
* lists of records pointing to address information when the fetches are
* complete (v4, v6).
review checkpoint
1999-11-25 01:23:32 +00:00
*/
snapshot address.[ch]
1999-09-22 00:26:20 +00:00
struct dns_adbname {
update copyright notice
2008-02-07 23:46:54 +00:00
unsigned int magic;
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
isc_refcount_t references;
update copyright notice
2008-02-07 23:46:54 +00:00
dns_adb_t *adb;
Refactor isc_hashmap to accept custom match function Refactor isc_hashmap to allow custom matching functions. This allows us to have better tailored keys that don't require fixed uint8_t arrays, but can be composed of more fields from the stored data structure.
2023-08-28 10:08:59 +02:00
dns_fixedname_t fname;
dns_name_t *name;
update copyright notice
2008-02-07 23:46:54 +00:00
unsigned int partial_result;
unsigned int flags;
dns_name_t target;
isc_stdtime_t expire_target;
isc_stdtime_t expire_v4;
isc_stdtime_t expire_v6;
dns_adbnamehooklist_t v4;
dns_adbnamehooklist_t v6;
dns_adbfetch_t *fetch_a;
dns_adbfetch_t *fetch_aaaa;
unsigned int fetch_err;
unsigned int fetch6_err;
dns_adbfindlist_t finds;
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
isc_mutex_t lock;
update copyright notice
2008-02-07 23:46:54 +00:00
isc_stdtime_t last_used;
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
/* for LRU-based management */
update copyright notice
2008-02-07 23:46:54 +00:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
ISC_LINK(dns_adbname_t) link;
snapshot address.[ch]
1999-09-22 00:26:20 +00:00
};
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
#if DNS_ADB_TRACE
#define dns_adbname_ref(ptr) dns_adbname__ref(ptr, __func__, __FILE__, __LINE__)
#define dns_adbname_unref(ptr) \
dns_adbname__unref(ptr, __func__, __FILE__, __LINE__)
#define dns_adbname_attach(ptr, ptrp) \
dns_adbname__attach(ptr, ptrp, __func__, __FILE__, __LINE__)
#define dns_adbname_detach(ptrp) \
dns_adbname__detach(ptrp, __func__, __FILE__, __LINE__)
ISC_REFCOUNT_TRACE_DECL(dns_adbname);
#else
ISC_REFCOUNT_DECL(dns_adbname);
#endif
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
/*%
* dns_adbfetch structure:
* Stores the state for an ongoing A or AAAA fetch.
*/
start on fetch infrastructure
1999-10-25 21:02:54 +00:00
struct dns_adbfetch {
update copyright notice
2008-02-07 23:46:54 +00:00
unsigned int magic;
dns_fetch_t *fetch;
dns_rdataset_t rdataset;
[master] limit recursion depth and iterative queries 4006. [security] A flaw in delegation handling could be exploited to put named into an infinite loop. This has been addressed by placing limits on the number of levels of recursion named will allow (default 7), and the number of iterative queries that it will send (default 50) before terminating a recursive query (CVE-2014-8500). The recursion depth limit is configured via the "max-recursion-depth" option. [RT #35780]
2014-11-17 23:24:44 -08:00
unsigned int depth;
start on fetch infrastructure
1999-10-25 21:02:54 +00:00
};
1851. [doc] Doxygen comment markup. [RT #11398]
2005-04-27 04:57:32 +00:00
/*%
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
* dns_adbnamehook structure:
*
snapshot address.[ch]
1999-09-22 00:26:20 +00:00
* This is a small widget that dangles off a dns_adbname_t. It contains a
* pointer to the address information about this host, and a link to the next
* namehook that will contain the next address this host has.
*/
struct dns_adbnamehook {
update copyright notice
2008-02-07 23:46:54 +00:00
unsigned int magic;
dns_adbentry_t *entry;
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
ISC_LINK(dns_adbnamehook_t) name_link;
ISC_LINK(dns_adbnamehook_t) entry_link;
Start on address.c
1999-09-21 22:46:42 +00:00
};
1851. [doc] Doxygen comment markup. [RT #11398]
2005-04-27 04:57:32 +00:00
/*%
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
* dns_adbentry structure:
*
* This is the structure representing a nameserver address; it can be looked
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
* up via the adb->entries hash table. Also, each dns_adbnamehook and
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
* and dns_adbaddrinfo object will contain a pointer to one of these.
*
* The structure holds quite a bit of information about addresses,
* including edns state (in "flags"), RTT, and of course the address of
removed unused edns_level member from ADB entry
2001-01-22 22:53:13 +00:00
* the host.
snapshot address.[ch]
1999-09-22 00:26:20 +00:00
*/
Start on address.c
1999-09-21 22:46:42 +00:00
struct dns_adbentry {
update copyright notice
2008-02-07 23:46:54 +00:00
unsigned int magic;
Use clang-format to reformat the source files
2020-02-12 13:59:18 +01:00
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
dns_adb_t *adb;
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
isc_mutex_t lock;
isc_stdtime_t last_used;
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
isc_refcount_t references;
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
dns_adbnamehooklist_t nhs;
Use clang-format to reformat the source files
2020-02-12 13:59:18 +01:00
Replace some ADB entry locking with atomics to reduce ADB contention Use atomics on couple of ADB entry members (.srtt, .flags, .expires, and .lastage) to remove ADB entry locking from couple of hot spots. The most prominent place is copy_namehook_lists() that gets called under ADB name lock and if the namehook list is long it acquires-releases quite a few ADB entry locks. Changing those ADB entry members to atomics allowed us to new_adbaddrinfo() not require locked ADB entry and since adbentry_overquota() already used atomics and handling lame information was dropped in the previous commit, we could not make the copy_namehook_lists() lockless. The other hotspot is dns_adb_adjustsrtt() and dns_adb_agesrtt() that can now use atomics because .srtt is already atomic_uint. And the last place that could now use atomics is dns_adb_changeflags().
2023-09-21 11:59:01 +02:00
atomic_uint flags;
atomic_uint srtt;
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
unsigned int completed;
unsigned int timeouts;
3593. [func] Update EDNS processing to better track remote server capabilities. [RT #30655]
2013-06-12 11:31:30 +10:00
unsigned char plain;
unsigned char plainto;
unsigned char edns;
Simplify the EDNS buffer size logic for DNS Flag Day 2020 The DNS Flag Day 2020 aims to remove the IP fragmentation problem from the UDP DNS communication. In this commit, we implement the required changes and simplify the logic for picking the EDNS Buffer Size. 1. The defaults for `edns-udp-size`, `max-udp-size` and `nocookie-udp-size` have been changed to `1232` (the value picked by DNS Flag Day 2020). 2. The probing heuristics that would try 512->4096->1432->1232 buffer sizes has been removed and the resolver will always use just the `edns-udp-size` value. 3. Instead of just disabling the PMTUD mechanism on the UDP sockets, we now set IP_DONTFRAG (IPV6_DONTFRAG) flag. That means that the UDP packets won't get ever fragmented. If the ICMP packets are lost the UDP will just timeout and eventually be retried over TCP.
2020-09-23 14:47:26 +02:00
unsigned char ednsto;
Reduce padding in ADB address entries Saves a couple of bytes in `struct dns_adbentry`.
2022-04-06 18:37:54 +01:00
uint16_t udpsize;
Use clang-format to reformat the source files
2020-02-12 13:59:18 +01:00
Replace custom isc_u?intNN_t types with C99 u?intNN_t types
2018-03-28 14:19:37 +02:00
uint8_t mode;
lib/dns/adb.c: Use atomics for adb quota values and reference counting
2019-01-22 10:47:18 +01:00
atomic_uint_fast32_t quota;
atomic_uint_fast32_t active;
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
double atr;
update copyright notice
2008-02-07 23:46:54 +00:00
isc_sockaddr_t sockaddr;
4152. [func] Implement DNS COOKIE option. This replaces the experimental SIT option of BIND 9.10. The following named.conf directives are avaliable: send-cookie, cookie-secret, cookie-algorithm and nocookie-udp-size. The following dig options are available: +[no]cookie[=value] and +[no]badcookie. [RT #39928]
2015-07-06 09:44:24 +10:00
unsigned char *cookie;
Replace custom isc_u?intNN_t types with C99 u?intNN_t types
2018-03-28 14:19:37 +02:00
uint16_t cookielen;
Use clang-format to reformat the source files
2020-02-12 13:59:18 +01:00
Replace some ADB entry locking with atomics to reduce ADB contention Use atomics on couple of ADB entry members (.srtt, .flags, .expires, and .lastage) to remove ADB entry locking from couple of hot spots. The most prominent place is copy_namehook_lists() that gets called under ADB name lock and if the namehook list is long it acquires-releases quite a few ADB entry locks. Changing those ADB entry members to atomics allowed us to new_adbaddrinfo() not require locked ADB entry and since adbentry_overquota() already used atomics and handling lame information was dropped in the previous commit, we could not make the copy_namehook_lists() lockless. The other hotspot is dns_adb_adjustsrtt() and dns_adb_agesrtt() that can now use atomics because .srtt is already atomic_uint. And the last place that could now use atomics is dns_adb_changeflags().
2023-09-21 11:59:01 +02:00
_Atomic(isc_stdtime_t) expires;
_Atomic(isc_stdtime_t) lastage;
update copyright notice
2008-02-07 23:46:54 +00:00
/*%<
* A nonzero 'expires' field indicates that the entry should
* persist until that time. This allows entries found
* using dns_adb_findaddrinfo() to persist for a limited time
* even though they are not necessarily associated with a
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
* entry.
update copyright notice
2008-02-07 23:46:54 +00:00
*/
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
ISC_LINK(dns_adbentry_t) link;
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
};
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
#if DNS_ADB_TRACE
#define dns_adbentry_ref(ptr) \
dns_adbentry__ref(ptr, __func__, __FILE__, __LINE__)
#define dns_adbentry_unref(ptr) \
dns_adbentry__unref(ptr, __func__, __FILE__, __LINE__)
#define dns_adbentry_attach(ptr, ptrp) \
dns_adbentry__attach(ptr, ptrp, __func__, __FILE__, __LINE__)
#define dns_adbentry_detach(ptrp) \
dns_adbentry__detach(ptrp, __func__, __FILE__, __LINE__)
ISC_REFCOUNT_TRACE_DECL(dns_adbentry);
#else
ISC_REFCOUNT_DECL(dns_adbentry);
#endif
snapshot address.[ch]
1999-09-22 00:26:20 +00:00
/*
Start flushing out the address.c stuff
1999-09-23 00:43:10 +00:00
* Internal functions (and prototypes).
*/
Remove use of the inline keyword used as suggestion to compiler Historically, the inline keyword was a strong suggestion to the compiler that it should inline the function marked inline. As compilers became better at optimising, this functionality has receded, and using inline as a suggestion to inline a function is obsolete. The compiler will happily ignore it and inline something else entirely if it finds that's a better optimisation. Therefore, remove all the occurences of the inline keyword with static functions inside single compilation unit and leave the decision whether to inline a function or not entirely on the compiler NOTE: We keep the usage the inline keyword when the purpose is to change the linkage behaviour.
2021-10-11 13:43:12 +02:00
static dns_adbname_t *
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
new_adbname(dns_adb_t *adb, const dns_name_t *, bool start_at_zone);
Remove use of the inline keyword used as suggestion to compiler Historically, the inline keyword was a strong suggestion to the compiler that it should inline the function marked inline. As compilers became better at optimising, this functionality has receded, and using inline as a suggestion to inline a function is obsolete. The compiler will happily ignore it and inline something else entirely if it finds that's a better optimisation. Therefore, remove all the occurences of the inline keyword with static functions inside single compilation unit and leave the decision whether to inline a function or not entirely on the compiler NOTE: We keep the usage the inline keyword when the purpose is to change the linkage behaviour.
2021-10-11 13:43:12 +02:00
static void
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
destroy_adbname(dns_adbname_t *);
Refactor isc_hashmap to accept custom match function Refactor isc_hashmap to allow custom matching functions. This allows us to have better tailored keys that don't require fixed uint8_t arrays, but can be composed of more fields from the stored data structure.
2023-08-28 10:08:59 +02:00
static bool
match_adbname(void *node, const void *key);
static uint32_t
hash_adbname(const dns_adbname_t *adbname);
Remove use of the inline keyword used as suggestion to compiler Historically, the inline keyword was a strong suggestion to the compiler that it should inline the function marked inline. As compilers became better at optimising, this functionality has receded, and using inline as a suggestion to inline a function is obsolete. The compiler will happily ignore it and inline something else entirely if it finds that's a better optimisation. Therefore, remove all the occurences of the inline keyword with static functions inside single compilation unit and leave the decision whether to inline a function or not entirely on the compiler NOTE: We keep the usage the inline keyword when the purpose is to change the linkage behaviour.
2021-10-11 13:43:12 +02:00
static dns_adbnamehook_t *
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
new_adbnamehook(dns_adb_t *adb);
Remove use of the inline keyword used as suggestion to compiler Historically, the inline keyword was a strong suggestion to the compiler that it should inline the function marked inline. As compilers became better at optimising, this functionality has receded, and using inline as a suggestion to inline a function is obsolete. The compiler will happily ignore it and inline something else entirely if it finds that's a better optimisation. Therefore, remove all the occurences of the inline keyword with static functions inside single compilation unit and leave the decision whether to inline a function or not entirely on the compiler NOTE: We keep the usage the inline keyword when the purpose is to change the linkage behaviour.
2021-10-11 13:43:12 +02:00
static void
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
free_adbnamehook(dns_adb_t *adb, dns_adbnamehook_t **namehookp);
Remove use of the inline keyword used as suggestion to compiler Historically, the inline keyword was a strong suggestion to the compiler that it should inline the function marked inline. As compilers became better at optimising, this functionality has receded, and using inline as a suggestion to inline a function is obsolete. The compiler will happily ignore it and inline something else entirely if it finds that's a better optimisation. Therefore, remove all the occurences of the inline keyword with static functions inside single compilation unit and leave the decision whether to inline a function or not entirely on the compiler NOTE: We keep the usage the inline keyword when the purpose is to change the linkage behaviour.
2021-10-11 13:43:12 +02:00
static dns_adbentry_t *
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
new_adbentry(dns_adb_t *adb, const isc_sockaddr_t *addr);
Remove use of the inline keyword used as suggestion to compiler Historically, the inline keyword was a strong suggestion to the compiler that it should inline the function marked inline. As compilers became better at optimising, this functionality has receded, and using inline as a suggestion to inline a function is obsolete. The compiler will happily ignore it and inline something else entirely if it finds that's a better optimisation. Therefore, remove all the occurences of the inline keyword with static functions inside single compilation unit and leave the decision whether to inline a function or not entirely on the compiler NOTE: We keep the usage the inline keyword when the purpose is to change the linkage behaviour.
2021-10-11 13:43:12 +02:00
static void
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
destroy_adbentry(dns_adbentry_t *entry);
Refactor isc_hashmap to accept custom match function Refactor isc_hashmap to allow custom matching functions. This allows us to have better tailored keys that don't require fixed uint8_t arrays, but can be composed of more fields from the stored data structure.
2023-08-28 10:08:59 +02:00
static bool
match_adbentry(void *node, const void *key);
Remove use of the inline keyword used as suggestion to compiler Historically, the inline keyword was a strong suggestion to the compiler that it should inline the function marked inline. As compilers became better at optimising, this functionality has receded, and using inline as a suggestion to inline a function is obsolete. The compiler will happily ignore it and inline something else entirely if it finds that's a better optimisation. Therefore, remove all the occurences of the inline keyword with static functions inside single compilation unit and leave the decision whether to inline a function or not entirely on the compiler NOTE: We keep the usage the inline keyword when the purpose is to change the linkage behaviour.
2021-10-11 13:43:12 +02:00
static dns_adbfind_t *
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
new_adbfind(dns_adb_t *, in_port_t);
static void
free_adbfind(dns_adbfind_t **);
Remove use of the inline keyword used as suggestion to compiler Historically, the inline keyword was a strong suggestion to the compiler that it should inline the function marked inline. As compilers became better at optimising, this functionality has receded, and using inline as a suggestion to inline a function is obsolete. The compiler will happily ignore it and inline something else entirely if it finds that's a better optimisation. Therefore, remove all the occurences of the inline keyword with static functions inside single compilation unit and leave the decision whether to inline a function or not entirely on the compiler NOTE: We keep the usage the inline keyword when the purpose is to change the linkage behaviour.
2021-10-11 13:43:12 +02:00
static dns_adbaddrinfo_t *
set the default destination port in the view, and use it when returning addresses from the adb.
2000-06-01 00:30:58 +00:00
new_adbaddrinfo(dns_adb_t *, dns_adbentry_t *, in_port_t);
Remove use of the inline keyword used as suggestion to compiler Historically, the inline keyword was a strong suggestion to the compiler that it should inline the function marked inline. As compilers became better at optimising, this functionality has receded, and using inline as a suggestion to inline a function is obsolete. The compiler will happily ignore it and inline something else entirely if it finds that's a better optimisation. Therefore, remove all the occurences of the inline keyword with static functions inside single compilation unit and leave the decision whether to inline a function or not entirely on the compiler NOTE: We keep the usage the inline keyword when the purpose is to change the linkage behaviour.
2021-10-11 13:43:12 +02:00
static dns_adbfetch_t *
more v4/v6 splitting out
1999-10-29 01:26:45 +00:00
new_adbfetch(dns_adb_t *);
Remove use of the inline keyword used as suggestion to compiler Historically, the inline keyword was a strong suggestion to the compiler that it should inline the function marked inline. As compilers became better at optimising, this functionality has receded, and using inline as a suggestion to inline a function is obsolete. The compiler will happily ignore it and inline something else entirely if it finds that's a better optimisation. Therefore, remove all the occurences of the inline keyword with static functions inside single compilation unit and leave the decision whether to inline a function or not entirely on the compiler NOTE: We keep the usage the inline keyword when the purpose is to change the linkage behaviour.
2021-10-11 13:43:12 +02:00
static void
more v4/v6 splitting out
1999-10-29 01:26:45 +00:00
free_adbfetch(dns_adb_t *, dns_adbfetch_t **);
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
static void
Purge stale ADB names globaly, not per bucket Before the refactoring, there was only few buckets with many names in them, so cleaning up stale ADB names per-bucket made sense. After the refactoring, each bucket directly maps to ADB name, so purging has been effectively disabled. Create a global LRU list for ADB names (and ADB entries) and purge the stale ADB names globally.
2022-08-01 09:07:38 +02:00
purge_stale_names(dns_adb_t *adb, isc_stdtime_t now);
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
static dns_adbname_t *
Lock the adbname and adbentry prior to unlocking hash locks There was a datarace that could expire a freshly created ADB names and entries between the return from get_attached_{name,entry} and locking it again. Lock the ADB name and ADB entry inside the hash table lock, so they won't get expired until the full initialization has been complete.
2022-12-13 13:48:55 +01:00
get_attached_and_locked_name(dns_adb_t *, const dns_name_t *,
bool start_at_zone, isc_stdtime_t now);
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
static void
purge_stale_entries(dns_adb_t *adb, isc_stdtime_t now);
Remove use of the inline keyword used as suggestion to compiler Historically, the inline keyword was a strong suggestion to the compiler that it should inline the function marked inline. As compilers became better at optimising, this functionality has receded, and using inline as a suggestion to inline a function is obsolete. The compiler will happily ignore it and inline something else entirely if it finds that's a better optimisation. Therefore, remove all the occurences of the inline keyword with static functions inside single compilation unit and leave the decision whether to inline a function or not entirely on the compiler NOTE: We keep the usage the inline keyword when the purpose is to change the linkage behaviour.
2021-10-11 13:43:12 +02:00
static dns_adbentry_t *
Lock the adbname and adbentry prior to unlocking hash locks There was a datarace that could expire a freshly created ADB names and entries between the return from get_attached_{name,entry} and locking it again. Lock the ADB name and ADB entry inside the hash table lock, so they won't get expired until the full initialization has been complete.
2022-12-13 13:48:55 +01:00
get_attached_and_locked_entry(dns_adb_t *adb, isc_stdtime_t now,
const isc_sockaddr_t *addr);
Replace custom isc_boolean_t with C standard bool type
2018-04-17 08:29:14 -07:00
static void
dump_adb(dns_adb_t *, FILE *, bool debug, isc_stdtime_t);
4546. [func] Extend the use of const declarations. [RT #43379]
2016-12-30 15:45:08 +11:00
static void
dump the ADB in a more user-friendly format (unless a debug flag is set)
2000-12-18 20:03:33 +00:00
print_namehook_list(FILE *, const char *legend, dns_adb_t *adb,
update copyright notice
2008-02-07 23:46:54 +00:00
dns_adbnamehooklist_t *list, bool debug, isc_stdtime_t now);
replace all instances of 'handle' with 'find' since that's what they are now.
1999-10-29 18:30:48 +00:00
static void
print_find_list(FILE *, dns_adbname_t *);
snapshot; nearly all fetch infrastructure is in place.
1999-10-27 19:36:58 +00:00
static void
print_fetch_list(FILE *, dns_adbname_t *);
Remove use of the inline keyword used as suggestion to compiler Historically, the inline keyword was a strong suggestion to the compiler that it should inline the function marked inline. As compilers became better at optimising, this functionality has receded, and using inline as a suggestion to inline a function is obsolete. The compiler will happily ignore it and inline something else entirely if it finds that's a better optimisation. Therefore, remove all the occurences of the inline keyword with static functions inside single compilation unit and leave the decision whether to inline a function or not entirely on the compiler NOTE: We keep the usage the inline keyword when the purpose is to change the linkage behaviour.
2021-10-11 13:43:12 +02:00
static void
Don't expire an ADB entry without holding the entries lock The clean_namehooks() function does't hold the 'adb->entries_lock' lock, so calling maybe_expire_entry() is not thread-safe. Instead of adding a lock/unlock, leave the expiration to later, e.g. by the get_attached_and_locked_entry() function. Also fix a couple of comment typos.
2022-12-23 14:21:03 +00:00
clean_namehooks(dns_adb_t *, dns_adbnamehooklist_t *);
Lower cache minimum time to 10 seconds. Enforce the minimum. Preliminary support for the "target" attribute of the adbname, to be used for CNAME/DNAME chains later on. The resolver will now do appropriate findzonecut calls, so the ADB need not concern itself with them any more. Fix some bugs in NXDOMAIN/NXRRSET handling.
2000-01-19 01:43:58 +00:00
static void
clean_target(dns_adb_t *, dns_name_t *);
[master] refactor max-recursion-queries - the counters weren't set correctly when fetches timed out. instead we now pass down a counter object.
2014-11-19 18:21:02 -08:00
static void
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
clean_finds_at_name(dns_adbname_t *, dns_adbstatus_t, unsigned int);
CNAME/DNAME chaining (initial support; A6 still missing)
2000-01-21 02:50:27 +00:00
static void
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
maybe_expire_namehooks(dns_adbname_t *, isc_stdtime_t);
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
static bool
maybe_expire_name(dns_adbname_t *adbname, isc_stdtime_t now);
static void
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
expire_name(dns_adbname_t *adbname, dns_adbstatus_t astat);
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
static bool
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
entry_expired(dns_adbentry_t *adbentry, isc_stdtime_t now);
static bool
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
maybe_expire_entry(dns_adbentry_t *adbentry, isc_stdtime_t now);
static void
expire_entry(dns_adbentry_t *adbentry);
fix a few bugs, and start an AAAA db lookup/query if no A6 is found
1999-11-02 00:29:45 +00:00
static isc_result_t
dbfind_name(dns_adbname_t *, isc_stdtime_t, dns_rdatatype_t);
Replace custom isc_boolean_t with C standard bool type
2018-04-17 08:29:14 -07:00
static isc_result_t
fetch_name(dns_adbname_t *, bool, unsigned int, isc_counter_t *qc,
checkpoint
2018-09-28 16:51:55 +10:00
dns_rdatatype_t);
Remove use of the inline keyword used as suggestion to compiler Historically, the inline keyword was a strong suggestion to the compiler that it should inline the function marked inline. As compilers became better at optimising, this functionality has receded, and using inline as a suggestion to inline a function is obsolete. The compiler will happily ignore it and inline something else entirely if it finds that's a better optimisation. Therefore, remove all the occurences of the inline keyword with static functions inside single compilation unit and leave the decision whether to inline a function or not entirely on the compiler NOTE: We keep the usage the inline keyword when the purpose is to change the linkage behaviour.
2021-10-11 13:43:12 +02:00
static void
clean up warnings that shouldn't be there.
1999-10-25 23:18:19 +00:00
destroy(dns_adb_t *);
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
static void
Replace custom isc_boolean_t with C standard bool type
2018-04-17 08:29:14 -07:00
shutdown_names(dns_adb_t *);
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
static void
remove isc_task completely as there is no further use of isc_task in BIND, this commit removes it, along with isc_taskmgr, isc_event, and all other related types. functions that accepted taskmgr as a parameter have been cleaned up. as a result of this change, some functions can no longer fail, so they've been changed to type void, and their callers have been updated accordingly. the tasks table has been removed from the statistics channel and the stats version has been updated. dns_dyndbctx has been changed to reference the loopmgr instead of taskmgr, and DNS_DYNDB_VERSION has been udpated as well.
2022-10-29 14:22:56 -07:00
shutdown_entries(dns_adb_t *);
static void
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
dump_entry(FILE *, dns_adb_t *, dns_adbentry_t *, bool, isc_stdtime_t);
3855. [bug] Limit smoothed round trip time aging to no more than once a second. [RT #32909]
2014-05-21 10:08:52 +10:00
static void
adjustsrtt(dns_adbaddrinfo_t *addr, unsigned int rtt, unsigned int factor,
isc_stdtime_t now);
[rt36341] 3905. [bug] Address deadlock between view.c and adb.c. [RT #36341]
2014-07-31 11:38:11 +10:00
static void
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
log_quota(dns_adbentry_t *entry, const char *fmt, ...) ISC_FORMAT_PRINTF(2, 3);
more v4/v6 splitting out
1999-10-29 01:26:45 +00:00
Change dns_adbentry_overquota() to dns_adb_overquota() The dns_adbentry_overquota() was violating the layers accessing the adbentry struct members directly. Change it to dns_adb_overquota() to match the dns_adb API.
2023-03-30 21:19:17 +02:00
static bool
adbentry_overquota(dns_adbentry_t *entry);
dead/needs_poke are flags for adbnames, merge in Bob's changes
1999-10-30 01:58:02 +00:00
/*
clean up numbering of FETCHOPT and ADDRINFO flags in the past there was overlap between the fields used as resolver fetch options and ADB addrinfo flags. this has mostly been eliminated; now we can clean up the rest of it and remove some confusing comments.
2023-07-03 12:43:17 -07:00
* Private flag(s) for adbfind objects. These are used internally and
* are not meant to be seen or used by the caller; however, we use the
* same flags field as for DNS_ADBFIND_xxx flags, so we must be careful
* that there is no overlap between these values and those. To make it
* easier, we will number these starting from the most significant bit
* instead of the least significant.
dead/needs_poke are flags for adbnames, merge in Bob's changes
1999-10-30 01:58:02 +00:00
*/
clean up numbering of FETCHOPT and ADDRINFO flags in the past there was overlap between the fields used as resolver fetch options and ADB addrinfo flags. this has mostly been eliminated; now we can clean up the rest of it and remove some confusing comments.
2023-07-03 12:43:17 -07:00
enum {
FIND_EVENT_SENT = 1 << 31,
};
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
#define FIND_EVENTSENT(h) (((h)->flags & FIND_EVENT_SENT) != 0)
Use clang-format to reformat the source files
2020-02-12 13:59:18 +01:00
clean up numbering of FETCHOPT and ADDRINFO flags in the past there was overlap between the fields used as resolver fetch options and ADB addrinfo flags. this has mostly been eliminated; now we can clean up the rest of it and remove some confusing comments.
2023-07-03 12:43:17 -07:00
/*
* Private flag(s) for adbname objects.
*/
enum {
NAME_IS_DEAD = 1 << 31,
};
#define NAME_DEAD(n) (((n)->flags & NAME_IS_DEAD) != 0)
fix a few bugs, and start an AAAA db lookup/query if no A6 is found
1999-11-02 00:29:45 +00:00
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
/*
clean up numbering of FETCHOPT and ADDRINFO flags in the past there was overlap between the fields used as resolver fetch options and ADB addrinfo flags. this has mostly been eliminated; now we can clean up the rest of it and remove some confusing comments.
2023-07-03 12:43:17 -07:00
* Private flag(s) for adbentry objects. Note that these will also
* be used for addrinfo flags, and in resolver.c we'll use the same
* field for FCTX_ADDRINFO_xxx flags to store information about remote
* servers, so we must be careful that there is no overlap between
* these values and those. To make it easier, we will number these
* starting from the most significant bit instead of the least
* significant.
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
*/
clean up numbering of FETCHOPT and ADDRINFO flags in the past there was overlap between the fields used as resolver fetch options and ADB addrinfo flags. this has mostly been eliminated; now we can clean up the rest of it and remove some confusing comments.
2023-07-03 12:43:17 -07:00
enum {
ENTRY_IS_DEAD = 1 << 31,
};
Replace some ADB entry locking with atomics to reduce ADB contention Use atomics on couple of ADB entry members (.srtt, .flags, .expires, and .lastage) to remove ADB entry locking from couple of hot spots. The most prominent place is copy_namehook_lists() that gets called under ADB name lock and if the namehook list is long it acquires-releases quite a few ADB entry locks. Changing those ADB entry members to atomics allowed us to new_adbaddrinfo() not require locked ADB entry and since adbentry_overquota() already used atomics and handling lame information was dropped in the previous commit, we could not make the copy_namehook_lists() lockless. The other hotspot is dns_adb_adjustsrtt() and dns_adb_agesrtt() that can now use atomics because .srtt is already atomic_uint. And the last place that could now use atomics is dns_adb_changeflags().
2023-09-21 11:59:01 +02:00
#define ENTRY_DEAD(e) ((atomic_load(&(e)->flags) & ENTRY_IS_DEAD) != 0)
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
clean up macros to avoid HAVE_* names. First a6 is a fetch, not name, attribute.
1999-11-04 08:48:47 +00:00
/*
* To the name, address classes are all that really exist. If it has a
1402. [cleanup] A6 has been moved to experimental and is no longer fully supported. developer: jinmei reviewer: marka
2002-11-27 09:52:58 +00:00
* V6 address it doesn't care if it came from a AAAA query.
clean up macros to avoid HAVE_* names. First a6 is a fetch, not name, attribute.
1999-11-04 08:48:47 +00:00
*/
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
#define NAME_HAS_V4(n) (!ISC_LIST_EMPTY((n)->v4))
#define NAME_HAS_V6(n) (!ISC_LIST_EMPTY((n)->v6))
cleanup: kill_name(): names are killed by a single function. link_name(), unlink_name(): names (and bucket reference counts) are managed by these. cancel fetches at names when killing the name. if fetches are pending when killing, set a "dead" flag in the name instead and let it die when the last fetch comes in.
1999-10-26 23:44:36 +00:00
clean up macros to avoid HAVE_* names. First a6 is a fetch, not name, attribute.
1999-11-04 08:48:47 +00:00
/*
1402. [cleanup] A6 has been moved to experimental and is no longer fully supported. developer: jinmei reviewer: marka
2002-11-27 09:52:58 +00:00
* Fetches are broken out into A and AAAA types. In some cases,
clean up macros to avoid HAVE_* names. First a6 is a fetch, not name, attribute.
1999-11-04 08:48:47 +00:00
* however, it makes more sense to test for a particular class of fetches,
* like V4 or V6 above.
*/
commit lruttl to the mainline. A tag was set called skan_lruttl-mainline-base, and I will tag this as skan_lruttl-mainline-merge after this commit
2007-10-19 17:15:53 +00:00
#define NAME_FETCH_A(n) ((n)->fetch_a != NULL)
#define NAME_FETCH_AAAA(n) ((n)->fetch_aaaa != NULL)
incidental cleanup The NAME_FETCH_A and NAME_FETCH_AAAA macros were meant to be boolean, indicating whether the pointers were set or not, while the NAME_FETCH_V4 and NAME_FETCH_V6 macros were meant to return the pointer values. The latter were only used as booleans, so they've been removed in favor of the former. Also did some style cleanup and removed an unreachable code block.
2021-10-18 17:57:58 -07:00
#define NAME_FETCH(n) (NAME_FETCH_A(n) || NAME_FETCH_AAAA(n))
fetch change: A -> pointer, AAAA -> pointer, A6 -> list
1999-10-29 21:48:51 +00:00
clean up macros to avoid HAVE_* names. First a6 is a fetch, not name, attribute.
1999-11-04 08:48:47 +00:00
/*
* Find options and tests to see if there are addresses on the list.
*/
commit lruttl to the mainline. A tag was set called skan_lruttl-mainline-base, and I will tag this as skan_lruttl-mainline-merge after this commit
2007-10-19 17:15:53 +00:00
#define FIND_WANTEVENT(fn) (((fn)->options & DNS_ADBFIND_WANTEVENT) != 0)
#define FIND_WANTEMPTYEVENT(fn) (((fn)->options & DNS_ADBFIND_EMPTYEVENT) != 0)
#define FIND_AVOIDFETCHES(fn) (((fn)->options & DNS_ADBFIND_AVOIDFETCHES) != 0)
#define FIND_STARTATZONE(fn) (((fn)->options & DNS_ADBFIND_STARTATZONE) != 0)
#define FIND_HAS_ADDRS(fn) (!ISC_LIST_EMPTY((fn)->list))
Reduce the number of fetches we make when looking up addresses If there are more that 5 NS record for a zone only perform a maximum of 4 address lookups for all the name servers. This limits the amount of remote lookup performed for server addresses at each level for a given query.
2020-02-06 17:19:10 +11:00
#define FIND_NOFETCH(fn) (((fn)->options & DNS_ADBFIND_NOFETCH) != 0)
IPv6 query flood suppression
1999-12-14 00:10:06 +00:00
clean up macros to avoid HAVE_* names. First a6 is a fetch, not name, attribute.
1999-11-04 08:48:47 +00:00
/*
* These are currently used on simple unsigned ints, so they are
* not really associated with any particular type.
*/
Update the source code formatting using clang-format-17
2023-10-10 16:55:13 +02:00
#define WANT_INET(x) (((x) & DNS_ADBFIND_INET) != 0)
#define WANT_INET6(x) (((x) & DNS_ADBFIND_INET6) != 0)
clean up macros to avoid HAVE_* names. First a6 is a fetch, not name, attribute.
1999-11-04 08:48:47 +00:00
commit lruttl to the mainline. A tag was set called skan_lruttl-mainline-base, and I will tag this as skan_lruttl-mainline-merge after this commit
2007-10-19 17:15:53 +00:00
#define EXPIRE_OK(exp, now) ((exp == INT_MAX) || (exp < now))
snapshot; start noting negative cache ttls
1999-11-04 06:46:05 +00:00
implement DNS_ADBFIND_GLUEOK, DNS_ADBFIND_HINTOK, and DNS_ADBFIND_IGNORELAME
2000-02-02 23:24:04 +00:00
/*
* Find out if the flags on a name (nf) indicate if it is a hint or
* glue, and compare this to the appropriate bits set in o, to see if
* this is ok.
*/
clean up numbering of FETCHOPT and ADDRINFO flags in the past there was overlap between the fields used as resolver fetch options and ADB addrinfo flags. this has mostly been eliminated; now we can clean up the rest of it and remove some confusing comments.
2023-07-03 12:43:17 -07:00
#define STARTATZONE_MATCHES(nf, o) \
(((nf)->flags & DNS_ADBFIND_STARTATZONE) == \
Update the source code formatting using clang-format-17
2023-10-10 16:55:13 +02:00
((o) & DNS_ADBFIND_STARTATZONE))
Use clang-format to reformat the source files
2020-02-12 13:59:18 +01:00
commit lruttl to the mainline. A tag was set called skan_lruttl-mainline-base, and I will tag this as skan_lruttl-mainline-merge after this commit
2007-10-19 17:15:53 +00:00
#define ENTER_LEVEL ISC_LOG_DEBUG(50)
#define CLEAN_LEVEL ISC_LOG_DEBUG(100)
#define DEF_LEVEL ISC_LOG_DEBUG(5)
#define NCACHE_LEVEL ISC_LOG_DEBUG(20)
Use clang-format to reformat the source files
2020-02-12 13:59:18 +01:00
commit lruttl to the mainline. A tag was set called skan_lruttl-mainline-base, and I will tag this as skan_lruttl-mainline-merge after this commit
2007-10-19 17:15:53 +00:00
#define NCACHE_RESULT(r) \
((r) == DNS_R_NCACHENXDOMAIN || (r) == DNS_R_NCACHENXRRSET)
#define AUTH_NX(r) ((r) == DNS_R_NXDOMAIN || (r) == DNS_R_NXRRSET)
Fix a lot of negative caching issues. I believe the code will use ncache responses from fetches and database lookups correctly now, but there needs to be a little more cleanup here and there. Bugs exist.
1999-11-24 19:23:27 +00:00
add find->result_v[46]
2000-08-26 02:21:45 +00:00
/*
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
* Error states.
add find->result_v[46]
2000-08-26 02:21:45 +00:00
*/
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
enum {
FIND_ERR_SUCCESS = 0,
FIND_ERR_CANCELED,
FIND_ERR_FAILURE,
FIND_ERR_NXDOMAIN,
FIND_ERR_NXRRSET,
FIND_ERR_UNEXPECTED,
FIND_ERR_NOTFOUND,
};
Use clang-format to reformat the source files
2020-02-12 13:59:18 +01:00
further cleanup of ADB dumps
2001-01-27 02:44:35 +00:00
static const char *errnames[] = { "success", "canceled", "failure",
update copyright notice
2008-02-07 23:46:54 +00:00
"nxdomain", "nxrrset", "unexpected",
"not_found" };
further cleanup of ADB dumps
2001-01-27 02:44:35 +00:00
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
static isc_result_t find_err_map[] = {
update copyright notice
2008-02-07 23:46:54 +00:00
ISC_R_SUCCESS, ISC_R_CANCELED, ISC_R_FAILURE, DNS_R_NXDOMAIN,
DNS_R_NXRRSET, ISC_R_UNEXPECTED, ISC_R_NOTFOUND /* not YET found */
add find->result_v[46]
2000-08-26 02:21:45 +00:00
};
Fix a lot of negative caching issues. I believe the code will use ncache responses from fetches and database lookups correctly now, but there needs to be a little more cleanup here and there. Bugs exist.
1999-11-24 19:23:27 +00:00
format string bugs and improved format string checking [RT #1578]
2001-08-08 22:54:55 +00:00
static void
DP(int level, const char *format, ...) ISC_FORMAT_PRINTF(2, 3);
print v6 hooks too; use logging system; general cleanup
1999-10-29 22:28:57 +00:00
static void
Megacommit of many files. Mostly, several functions that take pointers as arguments, almost always char * pointers, had those pointers qualified with "const". Those that returned pointers to previously const-qualified arguments had their return values qualified as const. Some structure members were qualified as const to retain that attribute from the variables from which they were assigned. The macro DE_CONST is used to deal with a handful of very special places where something is qualified as const but really needs to have its const qualifier removed. rdata.c now defines macros for the prototypes of the basic rdata functions, and all of the lib/dns/rdata/**/*.c files now use them. Some minor integer-compatibility issues. (IE, ~0x03 is a signed int, so assigning it to an unsigned int should use a cast. The type of an enum member is int, so there are some conversion issues there, too.) A pointers-to-function should not be cast to a pointer-to-object. Variables should not be named for C reserved identifiers. One or two set-but-not-used variables removed. Minor other ISC style cleanups.
2000-06-01 18:26:56 +00:00
DP(int level, const char *format, ...) {
update copyright notice
2008-02-07 23:46:54 +00:00
va_list args;
print v6 hooks too; use logging system; general cleanup
1999-10-29 22:28:57 +00:00
update copyright notice
2008-02-07 23:46:54 +00:00
va_start(args, format);
isc_log_vwrite(dns_lctx, DNS_LOGCATEGORY_DATABASE, DNS_LOGMODULE_ADB,
level, format, args);
va_end(args);
print v6 hooks too; use logging system; general cleanup
1999-10-29 22:28:57 +00:00
}
2355. [func] Extend the number statistics counters available. [RT #17590]
2008-04-03 05:55:52 +00:00
/*%
* Increment resolver-related statistics counters.
*/
Remove use of the inline keyword used as suggestion to compiler Historically, the inline keyword was a strong suggestion to the compiler that it should inline the function marked inline. As compilers became better at optimising, this functionality has receded, and using inline as a suggestion to inline a function is obsolete. The compiler will happily ignore it and inline something else entirely if it finds that's a better optimisation. Therefore, remove all the occurences of the inline keyword with static functions inside single compilation unit and leave the decision whether to inline a function or not entirely on the compiler NOTE: We keep the usage the inline keyword when the purpose is to change the linkage behaviour.
2021-10-11 13:43:12 +02:00
static void
move ADB and resolver stats out of the view object for better object separation, ADB and resolver statistics counters are now stored in the ADB and resolver objects themsevles, rather than in the associated view.
2022-05-11 15:38:54 -07:00
inc_resstats(dns_adb_t *adb, isc_statscounter_t counter) {
attach the resolver to the ADB the ADB depends on the resolver, but previously only accessed it via the view. as view->resolver may now be detached before the ADB finishes, a shutdown race was possible. attaching to the resolver directly prevents this.
2022-05-12 15:51:10 -07:00
if (adb->res != NULL) {
dns_resolver_incstats(adb->res, counter);
remove resolver whenshutdown events weakly attaching and detaching when creating and destroying the resolver obviates the need to have a callback event to do the weak detach. remove the dns_resolver_whenshutdown() mechanism, as it is now unused.
2022-05-11 13:55:01 -07:00
}
2355. [func] Extend the number statistics counters available. [RT #17590]
2008-04-03 05:55:52 +00:00
}
Merge statistics code (ATT SoW, rt24117) This includes the following changes: 3326. [func] Added task list statistics: task model, worker threads, quantum, tasks running, tasks ready. [RT #27678] 3325. [func] Report cache statistics: memory use, number of nodes, number of hash buckets, hit and miss counts. [RT #27056] 3324. [test] Add better tests for ADB stats [RT #27057] 3323. [func] Report the number of buckets the resolver is using. [RT #27020] 3322. [func] Monitor the number of active TCP and UDP dispatches. [RT #27055] 3321. [func] Monitor the number of recursive fetches and the number of open sockets, and report these values in the statistics channel. [RT #27054] 3320. [func] Added support for monitoring of recursing client count. [RT #27009] 3319. [func] Added support for monitoring of ADB entry count and hash size. [RT #27057]
2012-05-14 10:06:05 -07:00
/*%
* Set adb-related statistics counters.
*/
Remove use of the inline keyword used as suggestion to compiler Historically, the inline keyword was a strong suggestion to the compiler that it should inline the function marked inline. As compilers became better at optimising, this functionality has receded, and using inline as a suggestion to inline a function is obsolete. The compiler will happily ignore it and inline something else entirely if it finds that's a better optimisation. Therefore, remove all the occurences of the inline keyword with static functions inside single compilation unit and leave the decision whether to inline a function or not entirely on the compiler NOTE: We keep the usage the inline keyword when the purpose is to change the linkage behaviour.
2021-10-11 13:43:12 +02:00
static void
Replace custom isc_u?intNN_t types with C99 u?intNN_t types
2018-03-28 14:19:37 +02:00
set_adbstat(dns_adb_t *adb, uint64_t val, isc_statscounter_t counter) {
move ADB and resolver stats out of the view object for better object separation, ADB and resolver statistics counters are now stored in the ADB and resolver objects themsevles, rather than in the associated view.
2022-05-11 15:38:54 -07:00
if (adb->stats != NULL) {
isc_stats_set(adb->stats, val, counter);
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
Merge statistics code (ATT SoW, rt24117) This includes the following changes: 3326. [func] Added task list statistics: task model, worker threads, quantum, tasks running, tasks ready. [RT #27678] 3325. [func] Report cache statistics: memory use, number of nodes, number of hash buckets, hit and miss counts. [RT #27056] 3324. [test] Add better tests for ADB stats [RT #27057] 3323. [func] Report the number of buckets the resolver is using. [RT #27020] 3322. [func] Monitor the number of active TCP and UDP dispatches. [RT #27055] 3321. [func] Monitor the number of recursive fetches and the number of open sockets, and report these values in the statistics channel. [RT #27054] 3320. [func] Added support for monitoring of recursing client count. [RT #27009] 3319. [func] Added support for monitoring of ADB entry count and hash size. [RT #27057]
2012-05-14 10:06:05 -07:00
}
Remove use of the inline keyword used as suggestion to compiler Historically, the inline keyword was a strong suggestion to the compiler that it should inline the function marked inline. As compilers became better at optimising, this functionality has receded, and using inline as a suggestion to inline a function is obsolete. The compiler will happily ignore it and inline something else entirely if it finds that's a better optimisation. Therefore, remove all the occurences of the inline keyword with static functions inside single compilation unit and leave the decision whether to inline a function or not entirely on the compiler NOTE: We keep the usage the inline keyword when the purpose is to change the linkage behaviour.
2021-10-11 13:43:12 +02:00
static void
Merge statistics code (ATT SoW, rt24117) This includes the following changes: 3326. [func] Added task list statistics: task model, worker threads, quantum, tasks running, tasks ready. [RT #27678] 3325. [func] Report cache statistics: memory use, number of nodes, number of hash buckets, hit and miss counts. [RT #27056] 3324. [test] Add better tests for ADB stats [RT #27057] 3323. [func] Report the number of buckets the resolver is using. [RT #27020] 3322. [func] Monitor the number of active TCP and UDP dispatches. [RT #27055] 3321. [func] Monitor the number of recursive fetches and the number of open sockets, and report these values in the statistics channel. [RT #27054] 3320. [func] Added support for monitoring of recursing client count. [RT #27009] 3319. [func] Added support for monitoring of ADB entry count and hash size. [RT #27057]
2012-05-14 10:06:05 -07:00
dec_adbstats(dns_adb_t *adb, isc_statscounter_t counter) {
move ADB and resolver stats out of the view object for better object separation, ADB and resolver statistics counters are now stored in the ADB and resolver objects themsevles, rather than in the associated view.
2022-05-11 15:38:54 -07:00
if (adb->stats != NULL) {
isc_stats_decrement(adb->stats, counter);
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
Merge statistics code (ATT SoW, rt24117) This includes the following changes: 3326. [func] Added task list statistics: task model, worker threads, quantum, tasks running, tasks ready. [RT #27678] 3325. [func] Report cache statistics: memory use, number of nodes, number of hash buckets, hit and miss counts. [RT #27056] 3324. [test] Add better tests for ADB stats [RT #27057] 3323. [func] Report the number of buckets the resolver is using. [RT #27020] 3322. [func] Monitor the number of active TCP and UDP dispatches. [RT #27055] 3321. [func] Monitor the number of recursive fetches and the number of open sockets, and report these values in the statistics channel. [RT #27054] 3320. [func] Added support for monitoring of recursing client count. [RT #27009] 3319. [func] Added support for monitoring of ADB entry count and hash size. [RT #27057]
2012-05-14 10:06:05 -07:00
}
Remove use of the inline keyword used as suggestion to compiler Historically, the inline keyword was a strong suggestion to the compiler that it should inline the function marked inline. As compilers became better at optimising, this functionality has receded, and using inline as a suggestion to inline a function is obsolete. The compiler will happily ignore it and inline something else entirely if it finds that's a better optimisation. Therefore, remove all the occurences of the inline keyword with static functions inside single compilation unit and leave the decision whether to inline a function or not entirely on the compiler NOTE: We keep the usage the inline keyword when the purpose is to change the linkage behaviour.
2021-10-11 13:43:12 +02:00
static void
Merge statistics code (ATT SoW, rt24117) This includes the following changes: 3326. [func] Added task list statistics: task model, worker threads, quantum, tasks running, tasks ready. [RT #27678] 3325. [func] Report cache statistics: memory use, number of nodes, number of hash buckets, hit and miss counts. [RT #27056] 3324. [test] Add better tests for ADB stats [RT #27057] 3323. [func] Report the number of buckets the resolver is using. [RT #27020] 3322. [func] Monitor the number of active TCP and UDP dispatches. [RT #27055] 3321. [func] Monitor the number of recursive fetches and the number of open sockets, and report these values in the statistics channel. [RT #27054] 3320. [func] Added support for monitoring of recursing client count. [RT #27009] 3319. [func] Added support for monitoring of ADB entry count and hash size. [RT #27057]
2012-05-14 10:06:05 -07:00
inc_adbstats(dns_adb_t *adb, isc_statscounter_t counter) {
move ADB and resolver stats out of the view object for better object separation, ADB and resolver statistics counters are now stored in the ADB and resolver objects themsevles, rather than in the associated view.
2022-05-11 15:38:54 -07:00
if (adb->stats != NULL) {
isc_stats_increment(adb->stats, counter);
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
Merge statistics code (ATT SoW, rt24117) This includes the following changes: 3326. [func] Added task list statistics: task model, worker threads, quantum, tasks running, tasks ready. [RT #27678] 3325. [func] Report cache statistics: memory use, number of nodes, number of hash buckets, hit and miss counts. [RT #27056] 3324. [test] Add better tests for ADB stats [RT #27057] 3323. [func] Report the number of buckets the resolver is using. [RT #27020] 3322. [func] Monitor the number of active TCP and UDP dispatches. [RT #27055] 3321. [func] Monitor the number of recursive fetches and the number of open sockets, and report these values in the statistics channel. [RT #27054] 3320. [func] Added support for monitoring of recursing client count. [RT #27009] 3319. [func] Added support for monitoring of ADB entry count and hash size. [RT #27057]
2012-05-14 10:06:05 -07:00
}
Remove use of the inline keyword used as suggestion to compiler Historically, the inline keyword was a strong suggestion to the compiler that it should inline the function marked inline. As compilers became better at optimising, this functionality has receded, and using inline as a suggestion to inline a function is obsolete. The compiler will happily ignore it and inline something else entirely if it finds that's a better optimisation. Therefore, remove all the occurences of the inline keyword with static functions inside single compilation unit and leave the decision whether to inline a function or not entirely on the compiler NOTE: We keep the usage the inline keyword when the purpose is to change the linkage behaviour.
2021-10-11 13:43:12 +02:00
static dns_ttl_t
clamp ttls in the adb to be 10 < ttl < 86400, that is, 10 seconds minimum and 24 hours max
2000-06-07 20:15:48 +00:00
ttlclamp(dns_ttl_t ttl) {
update copyright notice
2008-02-07 23:46:54 +00:00
if (ttl < ADB_CACHE_MINIMUM) {
ttl = ADB_CACHE_MINIMUM;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
if (ttl > ADB_CACHE_MAXIMUM) {
ttl = ADB_CACHE_MAXIMUM;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
clamp ttls in the adb to be 10 < ttl < 86400, that is, 10 seconds minimum and 24 hours max
2000-06-07 20:15:48 +00:00
update copyright notice
2008-02-07 23:46:54 +00:00
return (ttl);
clamp ttls in the adb to be 10 < ttl < 86400, that is, 10 seconds minimum and 24 hours max
2000-06-07 20:15:48 +00:00
}
adb does fetches for A records now. Has a memory leak still.
1999-10-27 22:24:40 +00:00
/*
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
* Requires the name to be locked and that no entries to be locked.
simplify importing rdatasets. One function handles A and AAAA types.
1999-10-29 19:42:06 +00:00
*
* This code handles A and AAAA rdatasets only.
adb does fetches for A records now. Has a memory leak still.
1999-10-27 22:24:40 +00:00
*/
static isc_result_t
simplify importing rdatasets. One function handles A and AAAA types.
1999-10-29 19:42:06 +00:00
import_rdataset(dns_adbname_t *adbname, dns_rdataset_t *rdataset,
update copyright notice
2008-02-07 23:46:54 +00:00
isc_stdtime_t now) {
isc_result_t result;
incidental cleanup The NAME_FETCH_A and NAME_FETCH_AAAA macros were meant to be boolean, indicating whether the pointers were set or not, while the NAME_FETCH_V4 and NAME_FETCH_V6 macros were meant to return the pointer values. The latter were only used as booleans, so they've been removed in favor of the former. Also did some style cleanup and removed an unreachable code block.
2021-10-18 17:57:58 -07:00
dns_adb_t *adb = NULL;
update copyright notice
2008-02-07 23:46:54 +00:00
dns_rdatatype_t rdtype;
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
REQUIRE(DNS_ADBNAME_VALID(adbname));
update copyright notice
2008-02-07 23:46:54 +00:00
adb = adbname->adb;
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
REQUIRE(DNS_ADB_VALID(adb));
update copyright notice
2008-02-07 23:46:54 +00:00
rdtype = rdataset->type;
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
REQUIRE(rdtype == dns_rdatatype_a || rdtype == dns_rdatatype_aaaa);
for (result = dns_rdataset_first(rdataset); result == ISC_R_SUCCESS;
result = dns_rdataset_next(rdataset))
{
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
/* FIXME: Move to a separate function */
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
dns_adbnamehooklist_t *hookhead = NULL;
dns_adbentry_t *entry = NULL;
dns_adbnamehook_t *nh = NULL;
dns_rdata_t rdata = DNS_RDATA_INIT;
isc_sockaddr_t sockaddr;
struct in_addr ina;
struct in6_addr in6a;
update copyright notice
2008-02-07 23:46:54 +00:00
dns_rdataset_current(rdataset, &rdata);
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
switch (rdtype) {
case dns_rdatatype_a:
update copyright notice
2008-02-07 23:46:54 +00:00
INSIST(rdata.length == 4);
[master] replace memcpy() with memmove(). 3698. [cleanup] Replaced all uses of memcpy() with memmove(). [RT #35120]
2014-01-08 16:27:10 -08:00
memmove(&ina.s_addr, rdata.data, 4);
update copyright notice
2008-02-07 23:46:54 +00:00
isc_sockaddr_fromin(&sockaddr, &ina, 0);
2343. [bug] (Seemingly) duplicate IPv6 entries could be created in ADB. [RT #17837] (plan to merge to 9.[345])
2008-03-20 19:23:42 +00:00
hookhead = &adbname->v4;
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
break;
case dns_rdatatype_aaaa:
update copyright notice
2008-02-07 23:46:54 +00:00
INSIST(rdata.length == 16);
[master] replace memcpy() with memmove(). 3698. [cleanup] Replaced all uses of memcpy() with memmove(). [RT #35120]
2014-01-08 16:27:10 -08:00
memmove(in6a.s6_addr, rdata.data, 16);
update copyright notice
2008-02-07 23:46:54 +00:00
isc_sockaddr_fromin6(&sockaddr, &in6a, 0);
2343. [bug] (Seemingly) duplicate IPv6 entries could be created in ADB. [RT #17837] (plan to merge to 9.[345])
2008-03-20 19:23:42 +00:00
hookhead = &adbname->v6;
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
break;
default:
UNREACHABLE();
update copyright notice
2008-02-07 23:46:54 +00:00
}
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
again:
Lock the adbname and adbentry prior to unlocking hash locks There was a datarace that could expire a freshly created ADB names and entries between the return from get_attached_{name,entry} and locking it again. Lock the ADB name and ADB entry inside the hash table lock, so they won't get expired until the full initialization has been complete.
2022-12-13 13:48:55 +01:00
entry = get_attached_and_locked_entry(adb, now, &sockaddr);
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
if (ENTRY_DEAD(entry)) {
UNLOCK(&entry->lock);
dns_adbentry_detach(&entry);
goto again;
}
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
dns_adbnamehook_t *anh = NULL;
for (anh = ISC_LIST_HEAD(*hookhead); anh != NULL;
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
anh = ISC_LIST_NEXT(anh, name_link))
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
{
if (anh->entry == entry) {
break;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
}
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
if (anh == NULL) {
nh = new_adbnamehook(adb);
dns_adbentry_attach(entry, &nh->entry);
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
ISC_LIST_APPEND(*hookhead, nh, name_link);
ISC_LIST_APPEND(entry->nhs, nh, entry_link);
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
UNLOCK(&entry->lock);
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
dns_adbentry_detach(&entry);
update copyright notice
2008-02-07 23:46:54 +00:00
}
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
if (result == ISC_R_NOMORE) {
result = ISC_R_SUCCESS;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
INSIST(result == ISC_R_SUCCESS);
update copyright notice
2008-02-07 23:46:54 +00:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
switch (rdataset->trust) {
case dns_trust_glue:
case dns_trust_additional:
update copyright notice
2008-02-07 23:46:54 +00:00
rdataset->ttl = ADB_CACHE_MINIMUM;
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
break;
case dns_trust_ultimate:
3074. [bug] Make the adb cache read through for zone data and glue learn for zone named is authoritative for. [RT #22842]
2011-03-13 02:49:28 +00:00
rdataset->ttl = 0;
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
break;
default:
update copyright notice
2008-02-07 23:46:54 +00:00
rdataset->ttl = ttlclamp(rdataset->ttl);
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
switch (rdtype) {
case dns_rdatatype_a:
DP(NCACHE_LEVEL,
"expire_v4 set to MIN(%u,%u,%u) import_rdataset",
adbname->expire_v4, now + ADB_ENTRY_WINDOW,
now + rdataset->ttl);
update copyright notice
2008-02-07 23:46:54 +00:00
adbname->expire_v4 = ISC_MIN(
adbname->expire_v4,
3593. [func] Update EDNS processing to better track remote server capabilities. [RT #30655]
2013-06-12 11:31:30 +10:00
ISC_MIN(now + ADB_ENTRY_WINDOW, now + rdataset->ttl));
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
break;
case dns_rdatatype_aaaa:
DP(NCACHE_LEVEL,
"expire_v6 set to MIN(%u,%u,%u) import_rdataset",
adbname->expire_v6, now + ADB_ENTRY_WINDOW,
now + rdataset->ttl);
update copyright notice
2008-02-07 23:46:54 +00:00
adbname->expire_v6 = ISC_MIN(
adbname->expire_v6,
3593. [func] Update EDNS processing to better track remote server capabilities. [RT #30655]
2013-06-12 11:31:30 +10:00
ISC_MIN(now + ADB_ENTRY_WINDOW, now + rdataset->ttl));
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
break;
default:
UNREACHABLE();
update copyright notice
2008-02-07 23:46:54 +00:00
}
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
return (ISC_R_SUCCESS);
adb does fetches for A records now. Has a memory leak still.
1999-10-27 22:24:40 +00:00
}
Refactor isc_hashmap to accept custom match function Refactor isc_hashmap to allow custom matching functions. This allows us to have better tailored keys that don't require fixed uint8_t arrays, but can be composed of more fields from the stored data structure.
2023-08-28 10:08:59 +02:00
static bool
match_ptr(void *node, const void *key) {
return (node == key);
}
cleanup: kill_name(): names are killed by a single function. link_name(), unlink_name(): names (and bucket reference counts) are managed by these. cancel fetches at names when killing the name. if fetches are pending when killing, set a "dead" flag in the name instead and let it die when the last fetch comes in.
1999-10-26 23:44:36 +00:00
/*
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
* Requires the name to be locked.
cleanup: kill_name(): names are killed by a single function. link_name(), unlink_name(): names (and bucket reference counts) are managed by these. cancel fetches at names when killing the name. if fetches are pending when killing, set a "dead" flag in the name instead and let it die when the last fetch comes in.
1999-10-26 23:44:36 +00:00
*/
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
static void
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
expire_name(dns_adbname_t *adbname, dns_adbstatus_t astat) {
refactor dns_resolver to use loop callbacks callback events from dns_resolver_createfetch() are now posted using isc_async_run. other modules which called the resolver and maintained task/taskmgr objects for this purpose have been cleaned up.
2022-10-28 01:33:40 -07:00
isc_result_t result;
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
REQUIRE(DNS_ADBNAME_VALID(adbname));
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
dns_adb_t *adb = adbname->adb;
update copyright notice
2008-02-07 23:46:54 +00:00
refactor dns_resolver to use loop callbacks callback events from dns_resolver_createfetch() are now posted using isc_async_run. other modules which called the resolver and maintained task/taskmgr objects for this purpose have been cleaned up.
2022-10-28 01:33:40 -07:00
REQUIRE(DNS_ADB_VALID(adb));
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
DP(DEF_LEVEL, "killing name %p", adbname);
update copyright notice
2008-02-07 23:46:54 +00:00
/*
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
* Clean up the name's various contents. These functions
* are destructive in that they will always empty the lists
* of finds and namehooks.
update copyright notice
2008-02-07 23:46:54 +00:00
*/
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
clean_finds_at_name(adbname, astat, DNS_ADBFIND_ADDRESSMASK);
Don't expire an ADB entry without holding the entries lock The clean_namehooks() function does't hold the 'adb->entries_lock' lock, so calling maybe_expire_entry() is not thread-safe. Instead of adding a lock/unlock, leave the expiration to later, e.g. by the get_attached_and_locked_entry() function. Also fix a couple of comment typos.
2022-12-23 14:21:03 +00:00
clean_namehooks(adb, &adbname->v4);
clean_namehooks(adb, &adbname->v6);
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
clean_target(adb, &adbname->target);
update copyright notice
2008-02-07 23:46:54 +00:00
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
if (NAME_FETCH_A(adbname)) {
dns_resolver_cancelfetch(adbname->fetch_a->fetch);
}
if (NAME_FETCH_AAAA(adbname)) {
dns_resolver_cancelfetch(adbname->fetch_aaaa->fetch);
update copyright notice
2008-02-07 23:46:54 +00:00
}
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
adbname->flags |= NAME_IS_DEAD;
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
/*
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
* Remove the adbname from the hashtable...
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
*/
Refactor isc_hashmap to accept custom match function Refactor isc_hashmap to allow custom matching functions. This allows us to have better tailored keys that don't require fixed uint8_t arrays, but can be composed of more fields from the stored data structure.
2023-08-28 10:08:59 +02:00
result = isc_hashmap_delete(adb->names, hash_adbname(adbname),
match_ptr, adbname);
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
RUNTIME_CHECK(result == ISC_R_SUCCESS);
/* ... and LRU list */
ISC_LIST_UNLINK(adb->names_lru, adbname, link);
refactor dns_resolver to use loop callbacks callback events from dns_resolver_createfetch() are now posted using isc_async_run. other modules which called the resolver and maintained task/taskmgr objects for this purpose have been cleaned up.
2022-10-28 01:33:40 -07:00
dns_adbname_unref(adbname);
cleanup: kill_name(): names are killed by a single function. link_name(), unlink_name(): names (and bucket reference counts) are managed by these. cancel fetches at names when killing the name. if fetches are pending when killing, set a "dead" flag in the name instead and let it die when the last fetch comes in.
1999-10-26 23:44:36 +00:00
}
more v4/v6 splitting out
1999-10-29 01:26:45 +00:00
/*
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
* Requires the name to be locked and no entries to be locked.
more v4/v6 splitting out
1999-10-29 01:26:45 +00:00
*/
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
static void
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
maybe_expire_namehooks(dns_adbname_t *adbname, isc_stdtime_t now) {
REQUIRE(DNS_ADBNAME_VALID(adbname));
REQUIRE(DNS_ADB_VALID(adbname->adb));
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
dns_adb_t *adb = adbname->adb;
update copyright notice
2008-02-07 23:46:54 +00:00
/*
* Check to see if we need to remove the v4 addresses
*/
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
if (!NAME_FETCH_A(adbname) && EXPIRE_OK(adbname->expire_v4, now)) {
if (NAME_HAS_V4(adbname)) {
DP(DEF_LEVEL, "expiring v4 for name %p", adbname);
Don't expire an ADB entry without holding the entries lock The clean_namehooks() function does't hold the 'adb->entries_lock' lock, so calling maybe_expire_entry() is not thread-safe. Instead of adding a lock/unlock, leave the expiration to later, e.g. by the get_attached_and_locked_entry() function. Also fix a couple of comment typos.
2022-12-23 14:21:03 +00:00
clean_namehooks(adb, &adbname->v4);
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
adbname->partial_result &= ~DNS_ADBFIND_INET;
update copyright notice
2008-02-07 23:46:54 +00:00
}
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
adbname->expire_v4 = INT_MAX;
adbname->fetch_err = FIND_ERR_UNEXPECTED;
update copyright notice
2008-02-07 23:46:54 +00:00
}
/*
* Check to see if we need to remove the v6 addresses
*/
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
if (!NAME_FETCH_AAAA(adbname) && EXPIRE_OK(adbname->expire_v6, now)) {
if (NAME_HAS_V6(adbname)) {
DP(DEF_LEVEL, "expiring v6 for name %p", adbname);
Don't expire an ADB entry without holding the entries lock The clean_namehooks() function does't hold the 'adb->entries_lock' lock, so calling maybe_expire_entry() is not thread-safe. Instead of adding a lock/unlock, leave the expiration to later, e.g. by the get_attached_and_locked_entry() function. Also fix a couple of comment typos.
2022-12-23 14:21:03 +00:00
clean_namehooks(adb, &adbname->v6);
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
adbname->partial_result &= ~DNS_ADBFIND_INET6;
update copyright notice
2008-02-07 23:46:54 +00:00
}
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
adbname->expire_v6 = INT_MAX;
adbname->fetch6_err = FIND_ERR_UNEXPECTED;
update copyright notice
2008-02-07 23:46:54 +00:00
}
/*
* Check to see if we need to remove the alias target.
*/
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
if (EXPIRE_OK(adbname->expire_target, now)) {
clean_target(adb, &adbname->target);
adbname->expire_target = INT_MAX;
update copyright notice
2008-02-07 23:46:54 +00:00
}
more v4/v6 splitting out
1999-10-29 01:26:45 +00:00
}
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
static void
Megacommit of dozens of files. Cleanup of redundant/useless header file inclusion. ISC style lint, primarily for function declarations and standalone comments -- ie, those that appear on a line without any code, which should be written as follows: /* * This is a comment. */
2000-05-08 14:38:29 +00:00
shutdown_names(dns_adb_t *adb) {
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
dns_adbname_t *next = NULL;
update copyright notice
2008-02-07 23:46:54 +00:00
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
RWLOCK(&adb->names_lock, isc_rwlocktype_write);
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
for (dns_adbname_t *name = ISC_LIST_HEAD(adb->names_lru); name != NULL;
name = next)
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
{
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
next = ISC_LIST_NEXT(name, link);
Expire names in shutdown_names() under the lock Since there could be fetches running, we need to run expire_name() under the lock when shutting down the names.
2022-12-14 15:07:58 +01:00
dns_adbname_ref(name);
LOCK(&name->lock);
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
/*
* Run through the list. For each name, clean up finds
* found there, and cancel any fetches running. When
* all the fetches are canceled, the name will destroy
* itself.
*/
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
expire_name(name, DNS_ADB_SHUTTINGDOWN);
Expire names in shutdown_names() under the lock Since there could be fetches running, we need to run expire_name() under the lock when shutting down the names.
2022-12-14 15:07:58 +01:00
UNLOCK(&name->lock);
dns_adbname_detach(&name);
update copyright notice
2008-02-07 23:46:54 +00:00
}
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
RWUNLOCK(&adb->names_lock, isc_rwlocktype_write);
add dns_adb_findaddrinfo() and dns_adb_freeaddrinfo()
2000-01-12 03:00:33 +00:00
}
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
static void
shutdown_entries(dns_adb_t *adb) {
dns_adbentry_t *next = NULL;
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
RWLOCK(&adb->entries_lock, isc_rwlocktype_write);
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
for (dns_adbentry_t *adbentry = ISC_LIST_HEAD(adb->entries_lru);
adbentry != NULL; adbentry = next)
{
next = ISC_LIST_NEXT(adbentry, link);
expire_entry(adbentry);
}
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
RWUNLOCK(&adb->entries_lock, isc_rwlocktype_write);
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
}
cleanup: kill_name(): names are killed by a single function. link_name(), unlink_name(): names (and bucket reference counts) are managed by these. cancel fetches at names when killing the name. if fetches are pending when killing, set a "dead" flag in the name instead and let it die when the last fetch comes in.
1999-10-26 23:44:36 +00:00
/*
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
* The name containing the 'namehooks' list must be locked.
cleanup: kill_name(): names are killed by a single function. link_name(), unlink_name(): names (and bucket reference counts) are managed by these. cancel fetches at names when killing the name. if fetches are pending when killing, set a "dead" flag in the name instead and let it die when the last fetch comes in.
1999-10-26 23:44:36 +00:00
*/
static void
Don't expire an ADB entry without holding the entries lock The clean_namehooks() function does't hold the 'adb->entries_lock' lock, so calling maybe_expire_entry() is not thread-safe. Instead of adding a lock/unlock, leave the expiration to later, e.g. by the get_attached_and_locked_entry() function. Also fix a couple of comment typos.
2022-12-23 14:21:03 +00:00
clean_namehooks(dns_adb_t *adb, dns_adbnamehooklist_t *namehooks) {
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
dns_adbnamehook_t *namehook = NULL;
update copyright notice
2008-02-07 23:46:54 +00:00
namehook = ISC_LIST_HEAD(*namehooks);
while (namehook != NULL) {
INSIST(DNS_ADBNAMEHOOK_VALID(namehook));
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
INSIST(DNS_ADBENTRY_VALID(namehook->entry));
update copyright notice
2008-02-07 23:46:54 +00:00
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
dns_adbentry_t *adbentry = namehook->entry;
namehook->entry = NULL;
update copyright notice
2008-02-07 23:46:54 +00:00
/*
* Free the namehook
*/
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
ISC_LIST_UNLINK(*namehooks, namehook, name_link);
LOCK(&adbentry->lock);
ISC_LIST_UNLINK(adbentry->nhs, namehook, entry_link);
UNLOCK(&adbentry->lock);
dns_adbentry_detach(&adbentry);
update copyright notice
2008-02-07 23:46:54 +00:00
free_adbnamehook(adb, &namehook);
namehook = ISC_LIST_HEAD(*namehooks);
}
snapshot ; external and internal reference counts are separated out, and everything but fetches work.
1999-10-19 01:37:22 +00:00
}
Lower cache minimum time to 10 seconds. Enforce the minimum. Preliminary support for the "target" attribute of the adbname, to be used for CNAME/DNAME chains later on. The resolver will now do appropriate findzonecut calls, so the ADB need not concern itself with them any more. Fix some bugs in NXDOMAIN/NXRRSET handling.
2000-01-19 01:43:58 +00:00
static void
clean_target(dns_adb_t *adb, dns_name_t *target) {
update copyright notice
2008-02-07 23:46:54 +00:00
if (dns_name_countlabels(target) > 0) {
dns_name_free(target, adb->mctx);
dns_name_init(target, NULL);
}
Lower cache minimum time to 10 seconds. Enforce the minimum. Preliminary support for the "target" attribute of the adbname, to be used for CNAME/DNAME chains later on. The resolver will now do appropriate findzonecut calls, so the ADB need not concern itself with them any more. Fix some bugs in NXDOMAIN/NXRRSET handling.
2000-01-19 01:43:58 +00:00
}
static isc_result_t
4546. [func] Extend the use of const declarations. [RT #43379]
2016-12-30 15:45:08 +11:00
set_target(dns_adb_t *adb, const dns_name_t *name, const dns_name_t *fname,
update copyright notice
2008-02-07 23:46:54 +00:00
dns_rdataset_t *rdataset, dns_name_t *target) {
isc_result_t result;
dns_rdata_t rdata = DNS_RDATA_INIT;
REQUIRE(dns_name_countlabels(target) == 0);
if (rdataset->type == dns_rdatatype_cname) {
dns_rdata_cname_t cname;
/*
* Copy the CNAME's target into the target name.
*/
result = dns_rdataset_first(rdataset);
if (result != ISC_R_SUCCESS) {
return (result);
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
dns_rdataset_current(rdataset, &rdata);
result = dns_rdata_tostruct(&rdata, &cname, NULL);
if (result != ISC_R_SUCCESS) {
return (result);
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
Refactor dns_name_dup() usage using the semantic patch
2019-11-01 08:31:13 -05:00
dns_name_dup(&cname.cname, adb->mctx, target);
update copyright notice
2008-02-07 23:46:54 +00:00
dns_rdata_freestruct(&cname);
} else {
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
dns_fixedname_t fixed1, fixed2;
dns_name_t *prefix = NULL, *new_target = NULL;
update copyright notice
2008-02-07 23:46:54 +00:00
dns_rdata_dname_t dname;
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
dns_namereln_t namereln;
unsigned int nlabels;
int order;
update copyright notice
2008-02-07 23:46:54 +00:00
INSIST(rdataset->type == dns_rdatatype_dname);
namereln = dns_name_fullcompare(name, fname, &order, &nlabels);
INSIST(namereln == dns_namereln_subdomain);
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
update copyright notice
2008-02-07 23:46:54 +00:00
/*
* Get the target name of the DNAME.
*/
result = dns_rdataset_first(rdataset);
if (result != ISC_R_SUCCESS) {
return (result);
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
dns_rdataset_current(rdataset, &rdata);
result = dns_rdata_tostruct(&rdata, &dname, NULL);
if (result != ISC_R_SUCCESS) {
return (result);
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
update copyright notice
2008-02-07 23:46:54 +00:00
/*
* Construct the new target name.
*/
Use dns_fixedname_initname() where possible Replace dns_fixedname_init() calls followed by dns_fixedname_name() calls with calls to dns_fixedname_initname() where it is possible without affecting current behavior and/or performance. This patch was mostly prepared using Coccinelle and the following semantic patch: @@ expression fixedname, name; @@ - dns_fixedname_init(&fixedname); ... - name = dns_fixedname_name(&fixedname); + name = dns_fixedname_initname(&fixedname); The resulting set of changes was then manually reviewed to exclude false positives and apply minor tweaks. It is likely that more occurrences of this pattern can be refactored in an identical way. This commit only takes care of the low-hanging fruit.
2018-03-28 14:38:09 +02:00
prefix = dns_fixedname_initname(&fixed1);
new_target = dns_fixedname_initname(&fixed2);
update copyright notice
2008-02-07 23:46:54 +00:00
dns_name_split(name, nlabels, prefix, NULL);
result = dns_name_concatenate(prefix, &dname.dname, new_target,
NULL);
dns_rdata_freestruct(&dname);
if (result != ISC_R_SUCCESS) {
return (result);
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
Refactor dns_name_dup() usage using the semantic patch
2019-11-01 08:31:13 -05:00
dns_name_dup(new_target, adb->mctx, target);
update copyright notice
2008-02-07 23:46:54 +00:00
}
return (ISC_R_SUCCESS);
Lower cache minimum time to 10 seconds. Enforce the minimum. Preliminary support for the "target" attribute of the adbname, to be used for CNAME/DNAME chains later on. The resolver will now do appropriate findzonecut calls, so the ADB need not concern itself with them any more. Fix some bugs in NXDOMAIN/NXRRSET handling.
2000-01-19 01:43:58 +00:00
}
snapshot
1999-10-20 23:26:06 +00:00
/*
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
* The name must be locked.
snapshot
1999-10-20 23:26:06 +00:00
*/
snapshot ; external and internal reference counts are separated out, and everything but fetches work.
1999-10-19 01:37:22 +00:00
static void
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
clean_finds_at_name(dns_adbname_t *name, dns_adbstatus_t astat,
[master] refactor max-recursion-queries - the counters weren't set correctly when fetches timed out. instead we now pass down a counter object.
2014-11-19 18:21:02 -08:00
unsigned int addrs) {
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
dns_adbfind_t *find = NULL, *next = NULL;
update copyright notice
2008-02-07 23:46:54 +00:00
DP(ENTER_LEVEL,
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
"ENTER clean_finds_at_name, name %p, astat %08x, addrs %08x", name,
astat, addrs);
update copyright notice
2008-02-07 23:46:54 +00:00
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
for (find = ISC_LIST_HEAD(name->finds); find != NULL; find = next) {
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
bool process = false;
unsigned int wanted, notify;
update copyright notice
2008-02-07 23:46:54 +00:00
LOCK(&find->lock);
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
next = ISC_LIST_NEXT(find, plink);
update copyright notice
2008-02-07 23:46:54 +00:00
wanted = find->flags & DNS_ADBFIND_ADDRESSMASK;
notify = wanted & addrs;
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
switch (astat) {
case DNS_ADB_MOREADDRESSES:
DP(ISC_LOG_DEBUG(3), "more addresses");
update copyright notice
2008-02-07 23:46:54 +00:00
if ((notify) != 0) {
find->flags &= ~addrs;
Replace custom isc_boolean_t with C standard bool type
2018-04-17 08:29:14 -07:00
process = true;
update copyright notice
2008-02-07 23:46:54 +00:00
}
break;
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
case DNS_ADB_NOMOREADDRESSES:
DP(ISC_LOG_DEBUG(3), "no more addresses");
update copyright notice
2008-02-07 23:46:54 +00:00
find->flags &= ~addrs;
wanted = find->flags & DNS_ADBFIND_ADDRESSMASK;
if (wanted == 0) {
Replace custom isc_boolean_t with C standard bool type
2018-04-17 08:29:14 -07:00
process = true;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
break;
default:
find->flags &= ~addrs;
Replace custom isc_boolean_t with C standard bool type
2018-04-17 08:29:14 -07:00
process = true;
update copyright notice
2008-02-07 23:46:54 +00:00
}
if (process) {
DP(DEF_LEVEL, "cfan: processing find %p", find);
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
update copyright notice
2008-02-07 23:46:54 +00:00
/*
* Unlink the find from the name, letting the caller
* call dns_adb_destroyfind() on it to clean it up
* later.
*/
ISC_LIST_UNLINK(name->finds, find, plink);
find->adbname = NULL;
INSIST(!FIND_EVENTSENT(find));
find->result_v4 = find_err_map[name->fetch_err];
find->result_v6 = find_err_map[name->fetch6_err];
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
find->status = astat;
update copyright notice
2008-02-07 23:46:54 +00:00
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
DP(DEF_LEVEL, "cfan: sending find %p to caller", find);
update copyright notice
2008-02-07 23:46:54 +00:00
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
isc_async_run(find->loop, find->cb, find);
Fix a crash due to use-after-free (#38495)
2015-03-18 06:42:54 +05:30
find->flags |= FIND_EVENT_SENT;
update copyright notice
2008-02-07 23:46:54 +00:00
} else {
DP(DEF_LEVEL, "cfan: skipping find %p", find);
}
UNLOCK(&find->lock);
}
DP(ENTER_LEVEL, "EXIT clean_finds_at_name, name %p", name);
snapshot ; external and internal reference counts are separated out, and everything but fetches work.
1999-10-19 01:37:22 +00:00
}
Remove use of the inline keyword used as suggestion to compiler Historically, the inline keyword was a strong suggestion to the compiler that it should inline the function marked inline. As compilers became better at optimising, this functionality has receded, and using inline as a suggestion to inline a function is obsolete. The compiler will happily ignore it and inline something else entirely if it finds that's a better optimisation. Therefore, remove all the occurences of the inline keyword with static functions inside single compilation unit and leave the decision whether to inline a function or not entirely on the compiler NOTE: We keep the usage the inline keyword when the purpose is to change the linkage behaviour.
2021-10-11 13:43:12 +02:00
static dns_adbname_t *
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
new_adbname(dns_adb_t *adb, const dns_name_t *dnsname, bool start_at_zone) {
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
dns_adbname_t *name = NULL;
update copyright notice
2008-02-07 23:46:54 +00:00
Replace locked mempools with memory contexts Current mempools are kind of hybrid structures - they serve two purposes: 1. mempool with a lock is basically static sized allocator with pre-allocated free items 2. mempool without a lock is a doubly-linked list of preallocated items The first kind of usage could be easily replaced with jemalloc small sized arena objects and thread-local caches. The second usage not-so-much and we need to keep this (in libdns:message.c) for performance reasons.
2021-05-12 21:16:17 +02:00
name = isc_mem_get(adb->mctx, sizeof(*name));
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
*name = (dns_adbname_t){
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
.adb = dns_adb_ref(adb),
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
.expire_v4 = INT_MAX,
.expire_v6 = INT_MAX,
.expire_target = INT_MAX,
.fetch_err = FIND_ERR_UNEXPECTED,
.fetch6_err = FIND_ERR_UNEXPECTED,
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
.v4 = ISC_LIST_INITIALIZER,
.v6 = ISC_LIST_INITIALIZER,
.finds = ISC_LIST_INITIALIZER,
.link = ISC_LINK_INITIALIZER,
.magic = DNS_ADBNAME_MAGIC,
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
};
update copyright notice
2008-02-07 23:46:54 +00:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
#if DNS_ADB_TRACE
fprintf(stderr, "dns_adbname__init:%s:%s:%d:%p->references = 1\n",
__func__, __FILE__, __LINE__ + 1, name);
#endif
isc_refcount_init(&name->references, 1);
isc_mutex_init(&name->lock);
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
Refactor isc_hashmap to accept custom match function Refactor isc_hashmap to allow custom matching functions. This allows us to have better tailored keys that don't require fixed uint8_t arrays, but can be composed of more fields from the stored data structure.
2023-08-28 10:08:59 +02:00
name->name = dns_fixedname_initname(&name->fname);
dns_name_copy(dnsname, name->name);
update copyright notice
2008-02-07 23:46:54 +00:00
dns_name_init(&name->target, NULL);
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
if (start_at_zone) {
clean up numbering of FETCHOPT and ADDRINFO flags in the past there was overlap between the fields used as resolver fetch options and ADB addrinfo flags. this has mostly been eliminated; now we can clean up the rest of it and remove some confusing comments.
2023-07-03 12:43:17 -07:00
name->flags |= DNS_ADBFIND_STARTATZONE;
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
}
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
Merge statistics code (ATT SoW, rt24117) This includes the following changes: 3326. [func] Added task list statistics: task model, worker threads, quantum, tasks running, tasks ready. [RT #27678] 3325. [func] Report cache statistics: memory use, number of nodes, number of hash buckets, hit and miss counts. [RT #27056] 3324. [test] Add better tests for ADB stats [RT #27057] 3323. [func] Report the number of buckets the resolver is using. [RT #27020] 3322. [func] Monitor the number of active TCP and UDP dispatches. [RT #27055] 3321. [func] Monitor the number of recursive fetches and the number of open sockets, and report these values in the statistics channel. [RT #27054] 3320. [func] Added support for monitoring of recursing client count. [RT #27009] 3319. [func] Added support for monitoring of ADB entry count and hash size. [RT #27057]
2012-05-14 10:06:05 -07:00
inc_adbstats(adb, dns_adbstats_namescnt);
update copyright notice
2008-02-07 23:46:54 +00:00
return (name);
Start flushing out the address.c stuff
1999-09-23 00:43:10 +00:00
}
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
#if DNS_ADB_TRACE
ISC_REFCOUNT_TRACE_IMPL(dns_adbname, destroy_adbname);
#else
ISC_REFCOUNT_IMPL(dns_adbname, destroy_adbname);
#endif
commit lruttl to the mainline. A tag was set called skan_lruttl-mainline-base, and I will tag this as skan_lruttl-mainline-merge after this commit
2007-10-19 17:15:53 +00:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
static void
destroy_adbname(dns_adbname_t *name) {
REQUIRE(DNS_ADBNAME_VALID(name));
commit lruttl to the mainline. A tag was set called skan_lruttl-mainline-base, and I will tag this as skan_lruttl-mainline-merge after this commit
2007-10-19 17:15:53 +00:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
dns_adb_t *adb = name->adb;
commit lruttl to the mainline. A tag was set called skan_lruttl-mainline-base, and I will tag this as skan_lruttl-mainline-merge after this commit
2007-10-19 17:15:53 +00:00
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
REQUIRE(!NAME_HAS_V4(name));
REQUIRE(!NAME_HAS_V6(name));
REQUIRE(!NAME_FETCH(name));
REQUIRE(ISC_LIST_EMPTY(name->finds));
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
REQUIRE(!ISC_LINK_LINKED(name, link));
Many changes. o added adb_test.c to test the functions already written. o added code to delete names and all associated addresses. o added a database dumper which will currently print the adb in a human (or at least developer) readable format. o found and fixed many twisted little locking issues.
1999-10-02 01:59:09 +00:00
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
name->magic = 0;
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
isc_mutex_destroy(&name->lock);
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
isc_mem_put(adb->mctx, name, sizeof(*name));
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
dec_adbstats(adb, dns_adbstats_namescnt);
dns_adb_detach(&adb);
Many changes. o added adb_test.c to test the functions already written. o added code to delete names and all associated addresses. o added a database dumper which will currently print the adb in a human (or at least developer) readable format. o found and fixed many twisted little locking issues.
1999-10-02 01:59:09 +00:00
}
Remove use of the inline keyword used as suggestion to compiler Historically, the inline keyword was a strong suggestion to the compiler that it should inline the function marked inline. As compilers became better at optimising, this functionality has receded, and using inline as a suggestion to inline a function is obsolete. The compiler will happily ignore it and inline something else entirely if it finds that's a better optimisation. Therefore, remove all the occurences of the inline keyword with static functions inside single compilation unit and leave the decision whether to inline a function or not entirely on the compiler NOTE: We keep the usage the inline keyword when the purpose is to change the linkage behaviour.
2021-10-11 13:43:12 +02:00
static dns_adbnamehook_t *
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
new_adbnamehook(dns_adb_t *adb) {
dns_adbnamehook_t *nh = isc_mem_get(adb->mctx, sizeof(*nh));
*nh = (dns_adbnamehook_t){
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
.name_link = ISC_LINK_INITIALIZER,
.entry_link = ISC_LINK_INITIALIZER,
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
.magic = DNS_ADBNAMEHOOK_MAGIC,
};
Start flushing out the address.c stuff
1999-09-23 00:43:10 +00:00
update copyright notice
2008-02-07 23:46:54 +00:00
return (nh);
Start flushing out the address.c stuff
1999-09-23 00:43:10 +00:00
}
Remove use of the inline keyword used as suggestion to compiler Historically, the inline keyword was a strong suggestion to the compiler that it should inline the function marked inline. As compilers became better at optimising, this functionality has receded, and using inline as a suggestion to inline a function is obsolete. The compiler will happily ignore it and inline something else entirely if it finds that's a better optimisation. Therefore, remove all the occurences of the inline keyword with static functions inside single compilation unit and leave the decision whether to inline a function or not entirely on the compiler NOTE: We keep the usage the inline keyword when the purpose is to change the linkage behaviour.
2021-10-11 13:43:12 +02:00
static void
Megacommit of dozens of files. Cleanup of redundant/useless header file inclusion. ISC style lint, primarily for function declarations and standalone comments -- ie, those that appear on a line without any code, which should be written as follows: /* * This is a comment. */
2000-05-08 14:38:29 +00:00
free_adbnamehook(dns_adb_t *adb, dns_adbnamehook_t **namehook) {
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
dns_adbnamehook_t *nh = NULL;
REQUIRE(namehook != NULL && DNS_ADBNAMEHOOK_VALID(*namehook));
Many changes. o added adb_test.c to test the functions already written. o added code to delete names and all associated addresses. o added a database dumper which will currently print the adb in a human (or at least developer) readable format. o found and fixed many twisted little locking issues.
1999-10-02 01:59:09 +00:00
update copyright notice
2008-02-07 23:46:54 +00:00
nh = *namehook;
*namehook = NULL;
Many changes. o added adb_test.c to test the functions already written. o added code to delete names and all associated addresses. o added a database dumper which will currently print the adb in a human (or at least developer) readable format. o found and fixed many twisted little locking issues.
1999-10-02 01:59:09 +00:00
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
REQUIRE(nh->entry == NULL);
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
REQUIRE(!ISC_LINK_LINKED(nh, name_link));
REQUIRE(!ISC_LINK_LINKED(nh, entry_link));
Many changes. o added adb_test.c to test the functions already written. o added code to delete names and all associated addresses. o added a database dumper which will currently print the adb in a human (or at least developer) readable format. o found and fixed many twisted little locking issues.
1999-10-02 01:59:09 +00:00
update copyright notice
2008-02-07 23:46:54 +00:00
nh->magic = 0;
Replace locked mempools with memory contexts Current mempools are kind of hybrid structures - they serve two purposes: 1. mempool with a lock is basically static sized allocator with pre-allocated free items 2. mempool without a lock is a doubly-linked list of preallocated items The first kind of usage could be easily replaced with jemalloc small sized arena objects and thread-local caches. The second usage not-so-much and we need to keep this (in libdns:message.c) for performance reasons.
2021-05-12 21:16:17 +02:00
isc_mem_put(adb->mctx, nh, sizeof(*nh));
Many changes. o added adb_test.c to test the functions already written. o added code to delete names and all associated addresses. o added a database dumper which will currently print the adb in a human (or at least developer) readable format. o found and fixed many twisted little locking issues.
1999-10-02 01:59:09 +00:00
}
Remove use of the inline keyword used as suggestion to compiler Historically, the inline keyword was a strong suggestion to the compiler that it should inline the function marked inline. As compilers became better at optimising, this functionality has receded, and using inline as a suggestion to inline a function is obsolete. The compiler will happily ignore it and inline something else entirely if it finds that's a better optimisation. Therefore, remove all the occurences of the inline keyword with static functions inside single compilation unit and leave the decision whether to inline a function or not entirely on the compiler NOTE: We keep the usage the inline keyword when the purpose is to change the linkage behaviour.
2021-10-11 13:43:12 +02:00
static dns_adbentry_t *
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
new_adbentry(dns_adb_t *adb, const isc_sockaddr_t *addr) {
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
dns_adbentry_t *entry = NULL;
update copyright notice
2008-02-07 23:46:54 +00:00
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
entry = isc_mem_get(adb->mctx, sizeof(*entry));
*entry = (dns_adbentry_t){
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
.srtt = isc_random_uniform(0x1f) + 1,
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
.sockaddr = *addr,
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
.link = ISC_LINK_INITIALIZER,
Replace some ADB entry locking with atomics to reduce ADB contention Use atomics on couple of ADB entry members (.srtt, .flags, .expires, and .lastage) to remove ADB entry locking from couple of hot spots. The most prominent place is copy_namehook_lists() that gets called under ADB name lock and if the namehook list is long it acquires-releases quite a few ADB entry locks. Changing those ADB entry members to atomics allowed us to new_adbaddrinfo() not require locked ADB entry and since adbentry_overquota() already used atomics and handling lame information was dropped in the previous commit, we could not make the copy_namehook_lists() lockless. The other hotspot is dns_adb_adjustsrtt() and dns_adb_agesrtt() that can now use atomics because .srtt is already atomic_uint. And the last place that could now use atomics is dns_adb_changeflags().
2023-09-21 11:59:01 +02:00
.quota = adb->quota,
.references = ISC_REFCOUNT_INITIALIZER(1),
.adb = dns_adb_ref(adb),
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
.magic = DNS_ADBENTRY_MAGIC,
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
};
update copyright notice
2008-02-07 23:46:54 +00:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
#if DNS_ADB_TRACE
fprintf(stderr, "dns_adbentry__init:%s:%s:%d:%p->references = 1\n",
__func__, __FILE__, __LINE__ + 1, entry);
#endif
isc_mutex_init(&entry->lock);
Merge statistics code (ATT SoW, rt24117) This includes the following changes: 3326. [func] Added task list statistics: task model, worker threads, quantum, tasks running, tasks ready. [RT #27678] 3325. [func] Report cache statistics: memory use, number of nodes, number of hash buckets, hit and miss counts. [RT #27056] 3324. [test] Add better tests for ADB stats [RT #27057] 3323. [func] Report the number of buckets the resolver is using. [RT #27020] 3322. [func] Monitor the number of active TCP and UDP dispatches. [RT #27055] 3321. [func] Monitor the number of recursive fetches and the number of open sockets, and report these values in the statistics channel. [RT #27054] 3320. [func] Added support for monitoring of recursing client count. [RT #27009] 3319. [func] Added support for monitoring of ADB entry count and hash size. [RT #27057]
2012-05-14 10:06:05 -07:00
inc_adbstats(adb, dns_adbstats_entriescnt);
commit lruttl to the mainline. A tag was set called skan_lruttl-mainline-base, and I will tag this as skan_lruttl-mainline-merge after this commit
2007-10-19 17:15:53 +00:00
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
return (entry);
Start flushing out the address.c stuff
1999-09-23 00:43:10 +00:00
}
Remove use of the inline keyword used as suggestion to compiler Historically, the inline keyword was a strong suggestion to the compiler that it should inline the function marked inline. As compilers became better at optimising, this functionality has receded, and using inline as a suggestion to inline a function is obsolete. The compiler will happily ignore it and inline something else entirely if it finds that's a better optimisation. Therefore, remove all the occurences of the inline keyword with static functions inside single compilation unit and leave the decision whether to inline a function or not entirely on the compiler NOTE: We keep the usage the inline keyword when the purpose is to change the linkage behaviour.
2021-10-11 13:43:12 +02:00
static void
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
destroy_adbentry(dns_adbentry_t *entry) {
REQUIRE(DNS_ADBENTRY_VALID(entry));
snapshot: dns_adb_done() partial implementation, handles mostly work, but no fetches are started yet.
1999-10-14 00:50:00 +00:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
dns_adb_t *adb = entry->adb;
INSIST that active quota is 0 in destroy_adbentry() This should catch ADB quota management errors in the resolver.
2022-12-22 13:43:47 +00:00
uint_fast32_t active;
Many changes. o added adb_test.c to test the functions already written. o added code to delete names and all associated addresses. o added a database dumper which will currently print the adb in a human (or at least developer) readable format. o found and fixed many twisted little locking issues.
1999-10-02 01:59:09 +00:00
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
entry->magic = 0;
Many changes. o added adb_test.c to test the functions already written. o added code to delete names and all associated addresses. o added a database dumper which will currently print the adb in a human (or at least developer) readable format. o found and fixed many twisted little locking issues.
1999-10-02 01:59:09 +00:00
Don't remove ADB entry from LRU before trying to expire it There was a code flow error that would remove the expired ADB entry from the LRU list and then a check in the expire_entry() would cause assertion error because it expect the ADB entry to be linked. Additionally, the expire mechanism would loop for cases when we would held only a read rwlock; in such case we need to upgrade the lock and try again, not just try again.
2023-02-16 12:26:01 +01:00
INSIST(!ISC_LINK_LINKED(entry, link));
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
INSIST(ISC_LIST_EMPTY(entry->nhs));
INSIST that active quota is 0 in destroy_adbentry() This should catch ADB quota management errors in the resolver.
2022-12-22 13:43:47 +00:00
active = atomic_load_acquire(&entry->active);
INSIST(active == 0);
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
if (entry->cookie != NULL) {
isc_mem_put(adb->mctx, entry->cookie, entry->cookielen);
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
3744. [experimental] SIT: send and process Source Identity Tokens (which are similar to DNS Cookies by Donald Eastlake) and are designed to help clients detect off path spoofed responses and for servers to detect legitimate clients. SIT use a experimental EDNS option code (65001). SIT can be enabled via --enable-developer or --enable-sit. It is on by default in Windows. RRL processing as been updated to know about SIT with legitimate clients not being rate limited. [RT #35389]
2014-02-19 12:53:42 +11:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
isc_mutex_destroy(&entry->lock);
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
isc_mem_put(adb->mctx, entry, sizeof(*entry));
Use isc_hashmap instead of isc_ht in the dns_adb API Replace the use of isc_ht API with isc_hashmap API in the dns_adb database implementation. This requires extending the dns_adbnamebucket_t and dns_adbentrybucket_t structures to include keysize and copy of the key because the isc_hashmap API needs the raw key in case of resizing the hashmap table.
2022-06-27 12:46:06 +02:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
dec_adbstats(adb, dns_adbstats_entriescnt);
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
dns_adb_detach(&adb);
Many changes. o added adb_test.c to test the functions already written. o added code to delete names and all associated addresses. o added a database dumper which will currently print the adb in a human (or at least developer) readable format. o found and fixed many twisted little locking issues.
1999-10-02 01:59:09 +00:00
}
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
#if DNS_ADB_TRACE
ISC_REFCOUNT_TRACE_IMPL(dns_adbentry, destroy_adbentry);
#else
ISC_REFCOUNT_IMPL(dns_adbentry, destroy_adbentry);
#endif
Remove use of the inline keyword used as suggestion to compiler Historically, the inline keyword was a strong suggestion to the compiler that it should inline the function marked inline. As compilers became better at optimising, this functionality has receded, and using inline as a suggestion to inline a function is obsolete. The compiler will happily ignore it and inline something else entirely if it finds that's a better optimisation. Therefore, remove all the occurences of the inline keyword with static functions inside single compilation unit and leave the decision whether to inline a function or not entirely on the compiler NOTE: We keep the usage the inline keyword when the purpose is to change the linkage behaviour.
2021-10-11 13:43:12 +02:00
static dns_adbfind_t *
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
new_adbfind(dns_adb_t *adb, in_port_t port) {
dns_adbfind_t *find = NULL;
update copyright notice
2008-02-07 23:46:54 +00:00
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
find = isc_mem_get(adb->mctx, sizeof(*find));
*find = (dns_adbfind_t){
.port = port,
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
.result_v4 = ISC_R_UNEXPECTED,
.result_v6 = ISC_R_UNEXPECTED,
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
.publink = ISC_LINK_INITIALIZER,
.plink = ISC_LINK_INITIALIZER,
.list = ISC_LIST_INITIALIZER,
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
};
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
dns_adb_attach(adb, &find->adb);
isc_mutex_init(&find->lock);
Start flushing out the address.c stuff
1999-09-23 00:43:10 +00:00
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
find->magic = DNS_ADBFIND_MAGIC;
start on fetch infrastructure
1999-10-25 21:02:54 +00:00
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
return (find);
start on fetch infrastructure
1999-10-25 21:02:54 +00:00
}
Remove use of the inline keyword used as suggestion to compiler Historically, the inline keyword was a strong suggestion to the compiler that it should inline the function marked inline. As compilers became better at optimising, this functionality has receded, and using inline as a suggestion to inline a function is obsolete. The compiler will happily ignore it and inline something else entirely if it finds that's a better optimisation. Therefore, remove all the occurences of the inline keyword with static functions inside single compilation unit and leave the decision whether to inline a function or not entirely on the compiler NOTE: We keep the usage the inline keyword when the purpose is to change the linkage behaviour.
2021-10-11 13:43:12 +02:00
static void
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
free_adbfind(dns_adbfind_t **findp) {
dns_adb_t *adb = NULL;
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
dns_adbfind_t *find = NULL;
REQUIRE(findp != NULL && DNS_ADBFIND_VALID(*findp));
snapshot
1999-10-08 23:12:21 +00:00
update copyright notice
2008-02-07 23:46:54 +00:00
find = *findp;
*findp = NULL;
snapshot
1999-10-08 23:12:21 +00:00
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
adb = find->adb;
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
REQUIRE(!FIND_HAS_ADDRS(find));
REQUIRE(!ISC_LINK_LINKED(find, publink));
REQUIRE(!ISC_LINK_LINKED(find, plink));
REQUIRE(find->adbname == NULL);
snapshot: dns_adb_done() partial implementation, handles mostly work, but no fetches are started yet.
1999-10-14 00:50:00 +00:00
update copyright notice
2008-02-07 23:46:54 +00:00
find->magic = 0;
snapshot
1999-10-08 23:12:21 +00:00
- Make isc_mutex_destroy return void - Make isc_mutexblock_init/destroy return void - Minor cleanups
2018-11-19 10:31:09 +00:00
isc_mutex_destroy(&find->lock);
Replace locked mempools with memory contexts Current mempools are kind of hybrid structures - they serve two purposes: 1. mempool with a lock is basically static sized allocator with pre-allocated free items 2. mempool without a lock is a doubly-linked list of preallocated items The first kind of usage could be easily replaced with jemalloc small sized arena objects and thread-local caches. The second usage not-so-much and we need to keep this (in libdns:message.c) for performance reasons.
2021-05-12 21:16:17 +02:00
isc_mem_put(adb->mctx, find, sizeof(*find));
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
dns_adb_detach(&adb);
}
static dns_adbfetch_t *
new_adbfetch(dns_adb_t *adb) {
dns_adbfetch_t *fetch = NULL;
fetch = isc_mem_get(adb->mctx, sizeof(*fetch));
*fetch = (dns_adbfetch_t){ 0 };
dns_rdataset_init(&fetch->rdataset);
fetch->magic = DNS_ADBFETCH_MAGIC;
return (fetch);
}
static void
free_adbfetch(dns_adb_t *adb, dns_adbfetch_t **fetchp) {
dns_adbfetch_t *fetch = NULL;
REQUIRE(fetchp != NULL && DNS_ADBFETCH_VALID(*fetchp));
fetch = *fetchp;
*fetchp = NULL;
fetch->magic = 0;
if (dns_rdataset_isassociated(&fetch->rdataset)) {
dns_rdataset_disassociate(&fetch->rdataset);
}
isc_mem_put(adb->mctx, fetch, sizeof(*fetch));
snapshot
1999-10-08 23:12:21 +00:00
}
Start flushing out the address.c stuff
1999-09-23 00:43:10 +00:00
/*
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
* Copy bits from an adbentry into a newly allocated adb_addrinfo structure.
* The entry must be locked, and its reference count must be incremented.
snapshot address.[ch]
1999-09-22 00:26:20 +00:00
*/
Remove use of the inline keyword used as suggestion to compiler Historically, the inline keyword was a strong suggestion to the compiler that it should inline the function marked inline. As compilers became better at optimising, this functionality has receded, and using inline as a suggestion to inline a function is obsolete. The compiler will happily ignore it and inline something else entirely if it finds that's a better optimisation. Therefore, remove all the occurences of the inline keyword with static functions inside single compilation unit and leave the decision whether to inline a function or not entirely on the compiler NOTE: We keep the usage the inline keyword when the purpose is to change the linkage behaviour.
2021-10-11 13:43:12 +02:00
static dns_adbaddrinfo_t *
set the default destination port in the view, and use it when returning addresses from the adb.
2000-06-01 00:30:58 +00:00
new_adbaddrinfo(dns_adb_t *adb, dns_adbentry_t *entry, in_port_t port) {
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
dns_adbaddrinfo_t *ai = NULL;
commit lruttl to the mainline. A tag was set called skan_lruttl-mainline-base, and I will tag this as skan_lruttl-mainline-merge after this commit
2007-10-19 17:15:53 +00:00
Replace locked mempools with memory contexts Current mempools are kind of hybrid structures - they serve two purposes: 1. mempool with a lock is basically static sized allocator with pre-allocated free items 2. mempool without a lock is a doubly-linked list of preallocated items The first kind of usage could be easily replaced with jemalloc small sized arena objects and thread-local caches. The second usage not-so-much and we need to keep this (in libdns:message.c) for performance reasons.
2021-05-12 21:16:17 +02:00
ai = isc_mem_get(adb->mctx, sizeof(*ai));
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
*ai = (dns_adbaddrinfo_t){
Replace some ADB entry locking with atomics to reduce ADB contention Use atomics on couple of ADB entry members (.srtt, .flags, .expires, and .lastage) to remove ADB entry locking from couple of hot spots. The most prominent place is copy_namehook_lists() that gets called under ADB name lock and if the namehook list is long it acquires-releases quite a few ADB entry locks. Changing those ADB entry members to atomics allowed us to new_adbaddrinfo() not require locked ADB entry and since adbentry_overquota() already used atomics and handling lame information was dropped in the previous commit, we could not make the copy_namehook_lists() lockless. The other hotspot is dns_adb_adjustsrtt() and dns_adb_agesrtt() that can now use atomics because .srtt is already atomic_uint. And the last place that could now use atomics is dns_adb_changeflags().
2023-09-21 11:59:01 +02:00
.srtt = atomic_load(&entry->srtt),
.flags = atomic_load(&entry->flags),
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
.publink = ISC_LINK_INITIALIZER,
.sockaddr = entry->sockaddr,
.entry = dns_adbentry_ref(entry),
.magic = DNS_ADBADDRINFO_MAGIC,
};
commit lruttl to the mainline. A tag was set called skan_lruttl-mainline-base, and I will tag this as skan_lruttl-mainline-merge after this commit
2007-10-19 17:15:53 +00:00
update copyright notice
2008-02-07 23:46:54 +00:00
isc_sockaddr_setport(&ai->sockaddr, port);
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
update copyright notice
2008-02-07 23:46:54 +00:00
return (ai);
Start flushing out the address.c stuff
1999-09-23 00:43:10 +00:00
}
Remove use of the inline keyword used as suggestion to compiler Historically, the inline keyword was a strong suggestion to the compiler that it should inline the function marked inline. As compilers became better at optimising, this functionality has receded, and using inline as a suggestion to inline a function is obsolete. The compiler will happily ignore it and inline something else entirely if it finds that's a better optimisation. Therefore, remove all the occurences of the inline keyword with static functions inside single compilation unit and leave the decision whether to inline a function or not entirely on the compiler NOTE: We keep the usage the inline keyword when the purpose is to change the linkage behaviour.
2021-10-11 13:43:12 +02:00
static void
Megacommit of dozens of files. Cleanup of redundant/useless header file inclusion. ISC style lint, primarily for function declarations and standalone comments -- ie, those that appear on a line without any code, which should be written as follows: /* * This is a comment. */
2000-05-08 14:38:29 +00:00
free_adbaddrinfo(dns_adb_t *adb, dns_adbaddrinfo_t **ainfo) {
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
dns_adbaddrinfo_t *ai = NULL;
REQUIRE(ainfo != NULL && DNS_ADBADDRINFO_VALID(*ainfo));
snapshot: dns_adb_done() partial implementation, handles mostly work, but no fetches are started yet.
1999-10-14 00:50:00 +00:00
update copyright notice
2008-02-07 23:46:54 +00:00
ai = *ainfo;
*ainfo = NULL;
snapshot: dns_adb_done() partial implementation, handles mostly work, but no fetches are started yet.
1999-10-14 00:50:00 +00:00
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
REQUIRE(!ISC_LINK_LINKED(ai, publink));
snapshot: dns_adb_done() partial implementation, handles mostly work, but no fetches are started yet.
1999-10-14 00:50:00 +00:00
update copyright notice
2008-02-07 23:46:54 +00:00
ai->magic = 0;
snapshot: dns_adb_done() partial implementation, handles mostly work, but no fetches are started yet.
1999-10-14 00:50:00 +00:00
Resolver query forwarding to DoT-enabled upstream servers Implement TLS transport usage in the resolver. Use the configured TLS transport for the forwarders in the resolver.
2022-12-08 14:18:22 +00:00
if (ai->transport != NULL) {
dns_transport_detach(&ai->transport);
}
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
dns_adbentry_detach(&ai->entry);
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
Replace locked mempools with memory contexts Current mempools are kind of hybrid structures - they serve two purposes: 1. mempool with a lock is basically static sized allocator with pre-allocated free items 2. mempool without a lock is a doubly-linked list of preallocated items The first kind of usage could be easily replaced with jemalloc small sized arena objects and thread-local caches. The second usage not-so-much and we need to keep this (in libdns:message.c) for performance reasons.
2021-05-12 21:16:17 +02:00
isc_mem_put(adb->mctx, ai, sizeof(*ai));
snapshot: dns_adb_done() partial implementation, handles mostly work, but no fetches are started yet.
1999-10-14 00:50:00 +00:00
}
Refactor isc_hashmap to accept custom match function Refactor isc_hashmap to allow custom matching functions. This allows us to have better tailored keys that don't require fixed uint8_t arrays, but can be composed of more fields from the stored data structure.
2023-08-28 10:08:59 +02:00
static bool
match_adbname(void *node, const void *key) {
const dns_adbname_t *adbname0 = node;
const dns_adbname_t *adbname1 = key;
if ((adbname0->flags & DNS_ADBFIND_STARTATZONE) !=
(adbname1->flags & DNS_ADBFIND_STARTATZONE))
{
return (false);
}
return (dns_name_equal(adbname0->name, adbname1->name));
}
static uint32_t
hash_adbname(const dns_adbname_t *adbname) {
isc_hash32_t hash;
bool start_at_zone = adbname->flags & DNS_ADBFIND_STARTATZONE;
isc_hash32_init(&hash);
isc_hash32_hash(&hash, adbname->name->ndata, adbname->name->length,
false);
isc_hash32_hash(&hash, &start_at_zone, sizeof(start_at_zone), true);
return (isc_hash32_finalize(&hash));
}
Start flushing out the address.c stuff
1999-09-23 00:43:10 +00:00
/*
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
* Search for the name in the hash table.
Start flushing out the address.c stuff
1999-09-23 00:43:10 +00:00
*/
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
static dns_adbname_t *
Lock the adbname and adbentry prior to unlocking hash locks There was a datarace that could expire a freshly created ADB names and entries between the return from get_attached_{name,entry} and locking it again. Lock the ADB name and ADB entry inside the hash table lock, so they won't get expired until the full initialization has been complete.
2022-12-13 13:48:55 +01:00
get_attached_and_locked_name(dns_adb_t *adb, const dns_name_t *name,
bool start_at_zone, isc_stdtime_t now) {
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
isc_result_t result;
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
dns_adbname_t *adbname = NULL;
isc_time_t timenow;
isc_stdtime_t last_update;
Refactor isc_hashmap to accept custom match function Refactor isc_hashmap to allow custom matching functions. This allows us to have better tailored keys that don't require fixed uint8_t arrays, but can be composed of more fields from the stored data structure.
2023-08-28 10:08:59 +02:00
dns_adbname_t key = {
.name = UNCONST(name),
.flags = (start_at_zone) ? DNS_ADBFIND_STARTATZONE : 0,
};
uint32_t hashval = hash_adbname(&key);
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
isc_rwlocktype_t locktype = isc_rwlocktype_read;
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
isc_time_set(&timenow, now, 0);
Start flushing out the address.c stuff
1999-09-23 00:43:10 +00:00
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
RWLOCK(&adb->names_lock, locktype);
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
last_update = adb->names_last_update;
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
if (last_update + ADB_STALE_MARGIN >= now ||
Refactor the handling of isc_mem overmem condition Previously, there were two methods of working with the overmem condition: 1. hi/lo water callback - when the overmem condition was reached for the first time, the water callback was called with HIWATER mark and .is_overmem boolean was set internally. Similarly, when the used memory went below the lo water mark, the water callback would be called with LOWATER mark and .is_overmem was reset. This check would be called **every** time memory was allocated or freed. 2. isc_mem_isovermem() - a simple getter for the internal .is_overmem flag This commit refactors removes the first method and move the hi/lo water checks to the isc_mem_isovermem() function, thus we now have only a single method of checking overmem condition and the check for hi/lo water is removed from the hot path for memory contexts that doesn't use overmem checks.
2023-11-29 09:01:56 +01:00
isc_mem_isovermem(adb->mctx))
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
{
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
last_update = now;
UPGRADELOCK(&adb->names_lock, locktype);
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
purge_stale_names(adb, now);
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
adb->names_last_update = last_update;
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
Refactor isc_hashmap to accept custom match function Refactor isc_hashmap to allow custom matching functions. This allows us to have better tailored keys that don't require fixed uint8_t arrays, but can be composed of more fields from the stored data structure.
2023-08-28 10:08:59 +02:00
result = isc_hashmap_find(adb->names, hashval, match_adbname,
(void *)&key, (void **)&adbname);
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
switch (result) {
case ISC_R_NOTFOUND:
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
UPGRADELOCK(&adb->names_lock, locktype);
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
/* Allocate a new name and add it to the hash table. */
adbname = new_adbname(adb, name, start_at_zone);
Fix reference counting in get_attached_entry When get_attached_entry() encounters entry that would be expired, it needs to get reference to the entry before calling maybe_expire_entry(), so the ADB entry doesn't get destroyed inside the its own lock.
2022-12-06 15:59:35 +01:00
Refactor isc_hashmap to accept custom match function Refactor isc_hashmap to allow custom matching functions. This allows us to have better tailored keys that don't require fixed uint8_t arrays, but can be composed of more fields from the stored data structure.
2023-08-28 10:08:59 +02:00
void *found = NULL;
result = isc_hashmap_add(adb->names, hashval, match_adbname,
(void *)&key, adbname, &found);
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
if (result == ISC_R_EXISTS) {
destroy_adbname(adbname);
Refactor isc_hashmap to accept custom match function Refactor isc_hashmap to allow custom matching functions. This allows us to have better tailored keys that don't require fixed uint8_t arrays, but can be composed of more fields from the stored data structure.
2023-08-28 10:08:59 +02:00
adbname = found;
result = ISC_R_SUCCESS;
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
ISC_LIST_UNLINK(adb->names_lru, adbname, link);
}
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
INSIST(result == ISC_R_SUCCESS);
break;
case ISC_R_SUCCESS:
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
if (locktype == isc_rwlocktype_write) {
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
ISC_LIST_UNLINK(adb->names_lru, adbname, link);
}
break;
default:
UNREACHABLE();
}
Lock the adbname and adbentry prior to unlocking hash locks There was a datarace that could expire a freshly created ADB names and entries between the return from get_attached_{name,entry} and locking it again. Lock the ADB name and ADB entry inside the hash table lock, so they won't get expired until the full initialization has been complete.
2022-12-13 13:48:55 +01:00
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
dns_adbname_ref(adbname);
LOCK(&adbname->lock); /* Must be unlocked by the caller */
if (adbname->last_used + ADB_CACHE_MINIMUM <= last_update) {
adbname->last_used = now;
}
if (locktype == isc_rwlocktype_write) {
ISC_LIST_PREPEND(adb->names_lru, adbname, link);
}
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
/*
* The refcount is now 2 and the final detach will happen in
* expire_name() - the unused adbname stored in the hashtable and lru
* has always refcount == 1
*/
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
RWUNLOCK(&adb->names_lock, locktype);
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
return (adbname);
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
}
Don't remove ADB entry from LRU before trying to expire it There was a code flow error that would remove the expired ADB entry from the LRU list and then a check in the expire_entry() would cause assertion error because it expect the ADB entry to be linked. Additionally, the expire mechanism would loop for cases when we would held only a read rwlock; in such case we need to upgrade the lock and try again, not just try again.
2023-02-16 12:26:01 +01:00
static void
upgrade_entries_lock(dns_adb_t *adb, isc_rwlocktype_t *locktypep,
isc_stdtime_t now) {
if (*locktypep == isc_rwlocktype_read) {
UPGRADELOCK(&adb->entries_lock, *locktypep);
purge_stale_entries(adb, now);
adb->entries_last_update = now;
}
}
Refactor isc_hashmap to accept custom match function Refactor isc_hashmap to allow custom matching functions. This allows us to have better tailored keys that don't require fixed uint8_t arrays, but can be composed of more fields from the stored data structure.
2023-08-28 10:08:59 +02:00
static bool
match_adbentry(void *node, const void *key) {
dns_adbentry_t *adbentry = node;
return (isc_sockaddr_equal(&adbentry->sockaddr, key));
}
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
/*
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
* Find the entry in the adb->entries hashtable.
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
*/
static dns_adbentry_t *
Lock the adbname and adbentry prior to unlocking hash locks There was a datarace that could expire a freshly created ADB names and entries between the return from get_attached_{name,entry} and locking it again. Lock the ADB name and ADB entry inside the hash table lock, so they won't get expired until the full initialization has been complete.
2022-12-13 13:48:55 +01:00
get_attached_and_locked_entry(dns_adb_t *adb, isc_stdtime_t now,
const isc_sockaddr_t *addr) {
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
isc_result_t result;
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
dns_adbentry_t *adbentry = NULL;
isc_time_t timenow;
isc_stdtime_t last_update;
Refactor isc_hashmap to accept custom match function Refactor isc_hashmap to allow custom matching functions. This allows us to have better tailored keys that don't require fixed uint8_t arrays, but can be composed of more fields from the stored data structure.
2023-08-28 10:08:59 +02:00
uint32_t hashval = isc_sockaddr_hash(addr, true);
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
isc_rwlocktype_t locktype = isc_rwlocktype_read;
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
isc_time_set(&timenow, now, 0);
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
RWLOCK(&adb->entries_lock, locktype);
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
last_update = adb->entries_last_update;
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
if (now - last_update > ADB_STALE_MARGIN ||
Refactor the handling of isc_mem overmem condition Previously, there were two methods of working with the overmem condition: 1. hi/lo water callback - when the overmem condition was reached for the first time, the water callback was called with HIWATER mark and .is_overmem boolean was set internally. Similarly, when the used memory went below the lo water mark, the water callback would be called with LOWATER mark and .is_overmem was reset. This check would be called **every** time memory was allocated or freed. 2. isc_mem_isovermem() - a simple getter for the internal .is_overmem flag This commit refactors removes the first method and move the hi/lo water checks to the isc_mem_isovermem() function, thus we now have only a single method of checking overmem condition and the check for hi/lo water is removed from the hot path for memory contexts that doesn't use overmem checks.
2023-11-29 09:01:56 +01:00
isc_mem_isovermem(adb->mctx))
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
{
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
last_update = now;
Don't remove ADB entry from LRU before trying to expire it There was a code flow error that would remove the expired ADB entry from the LRU list and then a check in the expire_entry() would cause assertion error because it expect the ADB entry to be linked. Additionally, the expire mechanism would loop for cases when we would held only a read rwlock; in such case we need to upgrade the lock and try again, not just try again.
2023-02-16 12:26:01 +01:00
upgrade_entries_lock(adb, &locktype, now);
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
}
Refactor isc_hashmap to accept custom match function Refactor isc_hashmap to allow custom matching functions. This allows us to have better tailored keys that don't require fixed uint8_t arrays, but can be composed of more fields from the stored data structure.
2023-08-28 10:08:59 +02:00
result = isc_hashmap_find(adb->entries, hashval, match_adbentry,
(const unsigned char *)addr,
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
(void **)&adbentry);
Don't remove ADB entry from LRU before trying to expire it There was a code flow error that would remove the expired ADB entry from the LRU list and then a check in the expire_entry() would cause assertion error because it expect the ADB entry to be linked. Additionally, the expire mechanism would loop for cases when we would held only a read rwlock; in such case we need to upgrade the lock and try again, not just try again.
2023-02-16 12:26:01 +01:00
if (result == ISC_R_NOTFOUND) {
upgrade_entries_lock(adb, &locktype, now);
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
create:
Don't remove ADB entry from LRU before trying to expire it There was a code flow error that would remove the expired ADB entry from the LRU list and then a check in the expire_entry() would cause assertion error because it expect the ADB entry to be linked. Additionally, the expire mechanism would loop for cases when we would held only a read rwlock; in such case we need to upgrade the lock and try again, not just try again.
2023-02-16 12:26:01 +01:00
INSIST(locktype == isc_rwlocktype_write);
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
/* Allocate a new entry and add it to the hash table. */
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
adbentry = new_adbentry(adb, addr);
Fix reference counting in get_attached_entry When get_attached_entry() encounters entry that would be expired, it needs to get reference to the entry before calling maybe_expire_entry(), so the ADB entry doesn't get destroyed inside the its own lock.
2022-12-06 15:59:35 +01:00
Refactor isc_hashmap to accept custom match function Refactor isc_hashmap to allow custom matching functions. This allows us to have better tailored keys that don't require fixed uint8_t arrays, but can be composed of more fields from the stored data structure.
2023-08-28 10:08:59 +02:00
void *found = NULL;
result = isc_hashmap_add(adb->entries, hashval, match_adbentry,
&adbentry->sockaddr, adbentry, &found);
Don't remove ADB entry from LRU before trying to expire it There was a code flow error that would remove the expired ADB entry from the LRU list and then a check in the expire_entry() would cause assertion error because it expect the ADB entry to be linked. Additionally, the expire mechanism would loop for cases when we would held only a read rwlock; in such case we need to upgrade the lock and try again, not just try again.
2023-02-16 12:26:01 +01:00
if (result == ISC_R_SUCCESS) {
ISC_LIST_PREPEND(adb->entries_lru, adbentry, link);
} else if (result == ISC_R_EXISTS) {
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
dns_adbentry_detach(&adbentry);
Refactor isc_hashmap to accept custom match function Refactor isc_hashmap to allow custom matching functions. This allows us to have better tailored keys that don't require fixed uint8_t arrays, but can be composed of more fields from the stored data structure.
2023-08-28 10:08:59 +02:00
adbentry = found;
result = ISC_R_SUCCESS;
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
}
}
Don't remove ADB entry from LRU before trying to expire it There was a code flow error that would remove the expired ADB entry from the LRU list and then a check in the expire_entry() would cause assertion error because it expect the ADB entry to be linked. Additionally, the expire mechanism would loop for cases when we would held only a read rwlock; in such case we need to upgrade the lock and try again, not just try again.
2023-02-16 12:26:01 +01:00
INSIST(result == ISC_R_SUCCESS);
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
/*
* The dns_adbentry_ref() must stay here before trying to expire
* the ADB entry, so it is not destroyed under the lock.
*/
dns_adbentry_ref(adbentry);
LOCK(&adbentry->lock); /* Must be unlocked by the caller */
switch (locktype) {
case isc_rwlocktype_read:
Don't remove ADB entry from LRU before trying to expire it There was a code flow error that would remove the expired ADB entry from the LRU list and then a check in the expire_entry() would cause assertion error because it expect the ADB entry to be linked. Additionally, the expire mechanism would loop for cases when we would held only a read rwlock; in such case we need to upgrade the lock and try again, not just try again.
2023-02-16 12:26:01 +01:00
if (!entry_expired(adbentry, now)) {
break;
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
}
Don't remove ADB entry from LRU before trying to expire it There was a code flow error that would remove the expired ADB entry from the LRU list and then a check in the expire_entry() would cause assertion error because it expect the ADB entry to be linked. Additionally, the expire mechanism would loop for cases when we would held only a read rwlock; in such case we need to upgrade the lock and try again, not just try again.
2023-02-16 12:26:01 +01:00
/* We need to upgrade the LRU lock */
UNLOCK(&adbentry->lock);
upgrade_entries_lock(adb, &locktype, now);
LOCK(&adbentry->lock);
FALLTHROUGH;
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
case isc_rwlocktype_write:
Don't remove ADB entry from LRU before trying to expire it There was a code flow error that would remove the expired ADB entry from the LRU list and then a check in the expire_entry() would cause assertion error because it expect the ADB entry to be linked. Additionally, the expire mechanism would loop for cases when we would held only a read rwlock; in such case we need to upgrade the lock and try again, not just try again.
2023-02-16 12:26:01 +01:00
if (ENTRY_DEAD(adbentry) || maybe_expire_entry(adbentry, now)) {
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
UNLOCK(&adbentry->lock);
dns_adbentry_detach(&adbentry);
goto create;
}
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
break;
default:
UNREACHABLE();
update copyright notice
2008-02-07 23:46:54 +00:00
}
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
Don't remove ADB entry from LRU before trying to expire it There was a code flow error that would remove the expired ADB entry from the LRU list and then a check in the expire_entry() would cause assertion error because it expect the ADB entry to be linked. Additionally, the expire mechanism would loop for cases when we would held only a read rwlock; in such case we need to upgrade the lock and try again, not just try again.
2023-02-16 12:26:01 +01:00
/* Did enough time pass to update the LRU? */
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
if (adbentry->last_used + ADB_CACHE_MINIMUM <= last_update) {
adbentry->last_used = now;
Don't remove ADB entry from LRU before trying to expire it There was a code flow error that would remove the expired ADB entry from the LRU list and then a check in the expire_entry() would cause assertion error because it expect the ADB entry to be linked. Additionally, the expire mechanism would loop for cases when we would held only a read rwlock; in such case we need to upgrade the lock and try again, not just try again.
2023-02-16 12:26:01 +01:00
if (locktype == isc_rwlocktype_write) {
ISC_LIST_UNLINK(adb->entries_lru, adbentry, link);
ISC_LIST_PREPEND(adb->entries_lru, adbentry, link);
}
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
}
RWUNLOCK(&adb->entries_lock, locktype);
Start flushing out the address.c stuff
1999-09-23 00:43:10 +00:00
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
return (adbentry);
Start flushing out the address.c stuff
1999-09-23 00:43:10 +00:00
}
snapshot address.[ch]
1999-09-22 00:26:20 +00:00
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
static void
log_quota(dns_adbentry_t *entry, const char *fmt, ...) {
va_list ap;
char msgbuf[2048];
char addrbuf[ISC_NETADDR_FORMATSIZE];
isc_netaddr_t netaddr;
va_start(ap, fmt);
vsnprintf(msgbuf, sizeof(msgbuf), fmt, ap);
va_end(ap);
isc_netaddr_fromsockaddr(&netaddr, &entry->sockaddr);
isc_netaddr_format(&netaddr, addrbuf, sizeof(addrbuf));
isc_log_write(dns_lctx, DNS_LOGCATEGORY_DATABASE, DNS_LOGMODULE_ADB,
lib/dns/adb.c: Use atomics for adb quota values and reference counting
2019-01-22 10:47:18 +01:00
ISC_LOG_INFO,
"adb: quota %s (%" PRIuFAST32 "/%" PRIuFAST32 "): %s",
addrbuf, atomic_load_relaxed(&entry->active),
atomic_load_relaxed(&entry->quota), msgbuf);
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
}
snapshot
1999-10-08 23:12:21 +00:00
static void
Remove dns_adblameinfo from dns_adb Keeping the information about lame server in the ADB was done in !322 to fix following security issue: [CVE-2021-25219] Disable "lame-ttl" cache The handling of the lame servers needs to be redesigned and it is not going to be enabled any time soon, and the current code is just dead code that takes up space, code and stands in the way of making ADB work faster. Remove all the internals needed for handling the lame servers in the ADB for now. It might get reintroduced later if and when we redesign ADB.
2023-09-21 15:20:58 +02:00
copy_namehook_lists(dns_adb_t *adb, dns_adbfind_t *find, dns_adbname_t *name) {
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
dns_adbnamehook_t *namehook = NULL;
dns_adbentry_t *entry = NULL;
update copyright notice
2008-02-07 23:46:54 +00:00
Turn (int & flag) into (int & flag) != 0 when implicitly typed to bool
2018-10-11 11:57:57 +02:00
if ((find->options & DNS_ADBFIND_INET) != 0) {
update copyright notice
2008-02-07 23:46:54 +00:00
namehook = ISC_LIST_HEAD(name->v4);
while (namehook != NULL) {
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
dns_adbaddrinfo_t *addrinfo = NULL;
update copyright notice
2008-02-07 23:46:54 +00:00
entry = namehook->entry;
Change dns_adbentry_overquota() to dns_adb_overquota() The dns_adbentry_overquota() was violating the layers accessing the adbentry struct members directly. Change it to dns_adb_overquota() to match the dns_adb API.
2023-03-30 21:19:17 +02:00
if (adbentry_overquota(entry)) {
Remove dns_adblameinfo from dns_adb Keeping the information about lame server in the ADB was done in !322 to fix following security issue: [CVE-2021-25219] Disable "lame-ttl" cache The handling of the lame servers needs to be redesigned and it is not going to be enabled any time soon, and the current code is just dead code that takes up space, code and stands in the way of making ADB work faster. Remove all the internals needed for handling the lame servers in the ADB for now. It might get reintroduced later if and when we redesign ADB.
2023-09-21 15:20:58 +02:00
find->options |= DNS_ADBFIND_OVERQUOTA;
update copyright notice
2008-02-07 23:46:54 +00:00
goto nextv4;
}
Rewrite dns_resolver and dns_request to use netmgr timeouts - The `timeout_action` parameter to dns_dispatch_addresponse() been replaced with a netmgr callback that is called when a dispatch read times out. this callback may optionally reset the read timer and resume reading. - Added a function to convert isc_interval to milliseconds; this is used to translate fctx->interval into a value that can be passed to dns_dispatch_addresponse() as the timeout. - Note that netmgr timeouts are accurate to the millisecond, so code to check whether a timeout has been reached cannot rely on microsecond accuracy. - If serve-stale is configured, then a timeout received by the resolver may trigger it to return stale data, and then resume waiting for the read timeout. this is no longer based on a separate stale timer. - The code for canceling requests in request.c has been altered so that it can run asynchronously. - TCP timeout events apply to the dispatch, which may be shared by multiple queries. since in the event of a timeout we have no query ID to use to identify the resp we wanted, we now just send the timeout to the oldest query that was pending. - There was some additional refactoring in the resolver: combining fctx_join() and fctx_try_events() into one function to reduce code duplication, and using fixednames in fetchctx and fetchevent. - Incidental fix: new_adbaddrinfo() can't return NULL anymore, so the code can be simplified.
2021-04-22 18:58:01 -07:00
update copyright notice
2008-02-07 23:46:54 +00:00
addrinfo = new_adbaddrinfo(adb, entry, find->port);
Rewrite dns_resolver and dns_request to use netmgr timeouts - The `timeout_action` parameter to dns_dispatch_addresponse() been replaced with a netmgr callback that is called when a dispatch read times out. this callback may optionally reset the read timer and resume reading. - Added a function to convert isc_interval to milliseconds; this is used to translate fctx->interval into a value that can be passed to dns_dispatch_addresponse() as the timeout. - Note that netmgr timeouts are accurate to the millisecond, so code to check whether a timeout has been reached cannot rely on microsecond accuracy. - If serve-stale is configured, then a timeout received by the resolver may trigger it to return stale data, and then resume waiting for the read timeout. this is no longer based on a separate stale timer. - The code for canceling requests in request.c has been altered so that it can run asynchronously. - TCP timeout events apply to the dispatch, which may be shared by multiple queries. since in the event of a timeout we have no query ID to use to identify the resp we wanted, we now just send the timeout to the oldest query that was pending. - There was some additional refactoring in the resolver: combining fctx_join() and fctx_try_events() into one function to reduce code duplication, and using fixednames in fetchctx and fetchevent. - Incidental fix: new_adbaddrinfo() can't return NULL anymore, so the code can be simplified.
2021-04-22 18:58:01 -07:00
update copyright notice
2008-02-07 23:46:54 +00:00
/*
* Found a valid entry. Add it to the find's list.
*/
ISC_LIST_APPEND(find->list, addrinfo, publink);
nextv4:
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
namehook = ISC_LIST_NEXT(namehook, name_link);
update copyright notice
2008-02-07 23:46:54 +00:00
}
}
Turn (int & flag) into (int & flag) != 0 when implicitly typed to bool
2018-10-11 11:57:57 +02:00
if ((find->options & DNS_ADBFIND_INET6) != 0) {
update copyright notice
2008-02-07 23:46:54 +00:00
namehook = ISC_LIST_HEAD(name->v6);
while (namehook != NULL) {
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
dns_adbaddrinfo_t *addrinfo = NULL;
update copyright notice
2008-02-07 23:46:54 +00:00
entry = namehook->entry;
Change dns_adbentry_overquota() to dns_adb_overquota() The dns_adbentry_overquota() was violating the layers accessing the adbentry struct members directly. Change it to dns_adb_overquota() to match the dns_adb API.
2023-03-30 21:19:17 +02:00
if (adbentry_overquota(entry)) {
Remove dns_adblameinfo from dns_adb Keeping the information about lame server in the ADB was done in !322 to fix following security issue: [CVE-2021-25219] Disable "lame-ttl" cache The handling of the lame servers needs to be redesigned and it is not going to be enabled any time soon, and the current code is just dead code that takes up space, code and stands in the way of making ADB work faster. Remove all the internals needed for handling the lame servers in the ADB for now. It might get reintroduced later if and when we redesign ADB.
2023-09-21 15:20:58 +02:00
find->options |= DNS_ADBFIND_OVERQUOTA;
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
goto nextv6;
}
update copyright notice
2008-02-07 23:46:54 +00:00
addrinfo = new_adbaddrinfo(adb, entry, find->port);
Rewrite dns_resolver and dns_request to use netmgr timeouts - The `timeout_action` parameter to dns_dispatch_addresponse() been replaced with a netmgr callback that is called when a dispatch read times out. this callback may optionally reset the read timer and resume reading. - Added a function to convert isc_interval to milliseconds; this is used to translate fctx->interval into a value that can be passed to dns_dispatch_addresponse() as the timeout. - Note that netmgr timeouts are accurate to the millisecond, so code to check whether a timeout has been reached cannot rely on microsecond accuracy. - If serve-stale is configured, then a timeout received by the resolver may trigger it to return stale data, and then resume waiting for the read timeout. this is no longer based on a separate stale timer. - The code for canceling requests in request.c has been altered so that it can run asynchronously. - TCP timeout events apply to the dispatch, which may be shared by multiple queries. since in the event of a timeout we have no query ID to use to identify the resp we wanted, we now just send the timeout to the oldest query that was pending. - There was some additional refactoring in the resolver: combining fctx_join() and fctx_try_events() into one function to reduce code duplication, and using fixednames in fetchctx and fetchevent. - Incidental fix: new_adbaddrinfo() can't return NULL anymore, so the code can be simplified.
2021-04-22 18:58:01 -07:00
update copyright notice
2008-02-07 23:46:54 +00:00
/*
* Found a valid entry. Add it to the find's list.
*/
ISC_LIST_APPEND(find->list, addrinfo, publink);
nextv6:
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
namehook = ISC_LIST_NEXT(namehook, name_link);
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
}
}
event-driven shutdown/destroy
1999-10-25 23:07:14 +00:00
}
start on cleanup timer/task
1999-10-25 22:53:15 +00:00
/*
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
* The name must be locked and write lock on adb->names_lock must be held.
start on cleanup timer/task
1999-10-25 22:53:15 +00:00
*/
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
static bool
maybe_expire_name(dns_adbname_t *adbname, isc_stdtime_t now) {
REQUIRE(DNS_ADBNAME_VALID(adbname));
update copyright notice
2008-02-07 23:46:54 +00:00
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
/* Leave this name alone if it still has active namehooks... */
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
if (NAME_HAS_V4(adbname) || NAME_HAS_V6(adbname)) {
return (false);
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
/* ...an active fetch in progres... */
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
if (NAME_FETCH(adbname)) {
return (false);
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
/* ... or is not yet expired. */
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
if (!EXPIRE_OK(adbname->expire_v4, now) ||
!EXPIRE_OK(adbname->expire_v6, now) ||
!EXPIRE_OK(adbname->expire_target, now))
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
{
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
return (false);
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
expire_name(adbname, DNS_ADB_EXPIRED);
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
return (true);
}
static void
expire_entry(dns_adbentry_t *adbentry) {
isc_result_t result;
dns_adb_t *adb = adbentry->adb;
Don't remove ADB entry from LRU before trying to expire it There was a code flow error that would remove the expired ADB entry from the LRU list and then a check in the expire_entry() would cause assertion error because it expect the ADB entry to be linked. Additionally, the expire mechanism would loop for cases when we would held only a read rwlock; in such case we need to upgrade the lock and try again, not just try again.
2023-02-16 12:26:01 +01:00
if (!ENTRY_DEAD(adbentry)) {
Replace some ADB entry locking with atomics to reduce ADB contention Use atomics on couple of ADB entry members (.srtt, .flags, .expires, and .lastage) to remove ADB entry locking from couple of hot spots. The most prominent place is copy_namehook_lists() that gets called under ADB name lock and if the namehook list is long it acquires-releases quite a few ADB entry locks. Changing those ADB entry members to atomics allowed us to new_adbaddrinfo() not require locked ADB entry and since adbentry_overquota() already used atomics and handling lame information was dropped in the previous commit, we could not make the copy_namehook_lists() lockless. The other hotspot is dns_adb_adjustsrtt() and dns_adb_agesrtt() that can now use atomics because .srtt is already atomic_uint. And the last place that could now use atomics is dns_adb_changeflags().
2023-09-21 11:59:01 +02:00
(void)atomic_fetch_or(&adbentry->flags, ENTRY_IS_DEAD);
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
Refactor isc_hashmap to accept custom match function Refactor isc_hashmap to allow custom matching functions. This allows us to have better tailored keys that don't require fixed uint8_t arrays, but can be composed of more fields from the stored data structure.
2023-08-28 10:08:59 +02:00
result = isc_hashmap_delete(
adb->entries,
isc_sockaddr_hash(&adbentry->sockaddr, true), match_ptr,
adbentry);
Don't remove ADB entry from LRU before trying to expire it There was a code flow error that would remove the expired ADB entry from the LRU list and then a check in the expire_entry() would cause assertion error because it expect the ADB entry to be linked. Additionally, the expire mechanism would loop for cases when we would held only a read rwlock; in such case we need to upgrade the lock and try again, not just try again.
2023-02-16 12:26:01 +01:00
RUNTIME_CHECK(result == ISC_R_SUCCESS);
ISC_LIST_UNLINK(adb->entries_lru, adbentry, link);
}
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
dns_adbentry_detach(&adbentry);
}
static bool
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
entry_expired(dns_adbentry_t *adbentry, isc_stdtime_t now) {
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
if (!ISC_LIST_EMPTY(adbentry->nhs)) {
return (false);
}
Replace some ADB entry locking with atomics to reduce ADB contention Use atomics on couple of ADB entry members (.srtt, .flags, .expires, and .lastage) to remove ADB entry locking from couple of hot spots. The most prominent place is copy_namehook_lists() that gets called under ADB name lock and if the namehook list is long it acquires-releases quite a few ADB entry locks. Changing those ADB entry members to atomics allowed us to new_adbaddrinfo() not require locked ADB entry and since adbentry_overquota() already used atomics and handling lame information was dropped in the previous commit, we could not make the copy_namehook_lists() lockless. The other hotspot is dns_adb_adjustsrtt() and dns_adb_agesrtt() that can now use atomics because .srtt is already atomic_uint. And the last place that could now use atomics is dns_adb_changeflags().
2023-09-21 11:59:01 +02:00
if (atomic_load(&adbentry->expires) == 0 ||
atomic_load(&adbentry->expires) > now)
{
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
return (false);
}
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
return (true);
commit lruttl to the mainline. A tag was set called skan_lruttl-mainline-base, and I will tag this as skan_lruttl-mainline-merge after this commit
2007-10-19 17:15:53 +00:00
}
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
static bool
maybe_expire_entry(dns_adbentry_t *adbentry, isc_stdtime_t now) {
REQUIRE(DNS_ADBENTRY_VALID(adbentry));
if (entry_expired(adbentry, now)) {
expire_entry(adbentry);
return (true);
}
return (false);
}
commit lruttl to the mainline. A tag was set called skan_lruttl-mainline-base, and I will tag this as skan_lruttl-mainline-merge after this commit
2007-10-19 17:15:53 +00:00
/*%
* Examine the tail entry of the LRU list to see if it expires or is stale
* (unused for some period); if so, the name entry will be freed. If the ADB
* is in the overmem condition, the tail and the next to tail entries
* will be unconditionally removed (unless they have an outstanding fetch).
* We don't care about a race on 'overmem' at the risk of causing some
Purge stale ADB names globaly, not per bucket Before the refactoring, there was only few buckets with many names in them, so cleaning up stale ADB names per-bucket made sense. After the refactoring, each bucket directly maps to ADB name, so purging has been effectively disabled. Create a global LRU list for ADB names (and ADB entries) and purge the stale ADB names globally.
2022-08-01 09:07:38 +02:00
* collateral damage or a small delay in starting cleanup.
commit lruttl to the mainline. A tag was set called skan_lruttl-mainline-base, and I will tag this as skan_lruttl-mainline-merge after this commit
2007-10-19 17:15:53 +00:00
*
Purge stale ADB names globaly, not per bucket Before the refactoring, there was only few buckets with many names in them, so cleaning up stale ADB names per-bucket made sense. After the refactoring, each bucket directly maps to ADB name, so purging has been effectively disabled. Create a global LRU list for ADB names (and ADB entries) and purge the stale ADB names globally.
2022-08-01 09:07:38 +02:00
* adb->names_lock MUST be write locked
commit lruttl to the mainline. A tag was set called skan_lruttl-mainline-base, and I will tag this as skan_lruttl-mainline-merge after this commit
2007-10-19 17:15:53 +00:00
*/
static void
Purge stale ADB names globaly, not per bucket Before the refactoring, there was only few buckets with many names in them, so cleaning up stale ADB names per-bucket made sense. After the refactoring, each bucket directly maps to ADB name, so purging has been effectively disabled. Create a global LRU list for ADB names (and ADB entries) and purge the stale ADB names globally.
2022-08-01 09:07:38 +02:00
purge_stale_names(dns_adb_t *adb, isc_stdtime_t now) {
Refactor the handling of isc_mem overmem condition Previously, there were two methods of working with the overmem condition: 1. hi/lo water callback - when the overmem condition was reached for the first time, the water callback was called with HIWATER mark and .is_overmem boolean was set internally. Similarly, when the used memory went below the lo water mark, the water callback would be called with LOWATER mark and .is_overmem was reset. This check would be called **every** time memory was allocated or freed. 2. isc_mem_isovermem() - a simple getter for the internal .is_overmem flag This commit refactors removes the first method and move the hi/lo water checks to the isc_mem_isovermem() function, thus we now have only a single method of checking overmem condition and the check for hi/lo water is removed from the hot path for memory contexts that doesn't use overmem checks.
2023-11-29 09:01:56 +01:00
bool overmem = isc_mem_isovermem(adb->mctx);
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
int max_removed = overmem ? 2 : 1;
int scans = 0, removed = 0;
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
dns_adbname_t *prev = NULL;
update copyright notice
2008-02-07 23:46:54 +00:00
/*
* We limit the number of scanned entries to 10 (arbitrary choice)
* in order to avoid examining too many entries when there are many
* tail entries that have fetches (this should be rare, but could
* happen).
*/
Purge stale ADB names globaly, not per bucket Before the refactoring, there was only few buckets with many names in them, so cleaning up stale ADB names per-bucket made sense. After the refactoring, each bucket directly maps to ADB name, so purging has been effectively disabled. Create a global LRU list for ADB names (and ADB entries) and purge the stale ADB names globally.
2022-08-01 09:07:38 +02:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
for (dns_adbname_t *adbname = ISC_LIST_TAIL(adb->names_lru);
adbname != NULL && removed < max_removed && scans < 10;
adbname = prev)
update copyright notice
2008-02-07 23:46:54 +00:00
{
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
prev = ISC_LIST_PREV(adbname, link);
dns_adbname_ref(adbname);
LOCK(&adbname->lock);
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
update copyright notice
2008-02-07 23:46:54 +00:00
scans++;
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
/*
* Remove the name if it's expired or unused,
* has no address data.
*/
maybe_expire_namehooks(adbname, now);
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
if (maybe_expire_name(adbname, now)) {
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
removed++;
goto next;
}
Don't expire fresh ADB names and entries The overmem cleaning in ADB could become overzealous and clean fresh ADB names and entries. Add a safety check to not clean any ADB names and entries that are below ADB_CACHE_MINIMUM threshold.
2022-12-13 14:32:19 +01:00
/*
* Make sure that we are not purging ADB names that has been
* just created.
*/
if (adbname->last_used + ADB_CACHE_MINIMUM >= now) {
prev = NULL;
goto next;
}
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
if (overmem) {
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
expire_name(adbname, DNS_ADB_CANCELED);
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
removed++;
goto next;
}
Purge stale ADB names globaly, not per bucket Before the refactoring, there was only few buckets with many names in them, so cleaning up stale ADB names per-bucket made sense. After the refactoring, each bucket directly maps to ADB name, so purging has been effectively disabled. Create a global LRU list for ADB names (and ADB entries) and purge the stale ADB names globally.
2022-08-01 09:07:38 +02:00
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
if (adbname->last_used + ADB_STALE_MARGIN < now) {
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
expire_name(adbname, DNS_ADB_CANCELED);
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
removed++;
Don't expire fresh ADB names and entries The overmem cleaning in ADB could become overzealous and clean fresh ADB names and entries. Add a safety check to not clean any ADB names and entries that are below ADB_CACHE_MINIMUM threshold.
2022-12-13 14:32:19 +01:00
goto next;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
/*
Don't expire fresh ADB names and entries The overmem cleaning in ADB could become overzealous and clean fresh ADB names and entries. Add a safety check to not clean any ADB names and entries that are below ADB_CACHE_MINIMUM threshold.
2022-12-13 14:32:19 +01:00
* We won't expire anything on the LRU list as the
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
* .last_used + ADB_STALE_MARGIN will always be bigger
* than `now` for all previous entries, so we just stop
Don't expire fresh ADB names and entries The overmem cleaning in ADB could become overzealous and clean fresh ADB names and entries. Add a safety check to not clean any ADB names and entries that are below ADB_CACHE_MINIMUM threshold.
2022-12-13 14:32:19 +01:00
* the scanning.
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
*/
prev = NULL;
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
next:
UNLOCK(&adbname->lock);
dns_adbname_detach(&adbname);
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
add dns_adb_findaddrinfo() and dns_adb_freeaddrinfo()
2000-01-12 03:00:33 +00:00
}
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
static void
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
cleanup_names(dns_adb_t *adb, isc_stdtime_t now) {
dns_adbname_t *next = NULL;
update copyright notice
2008-02-07 23:46:54 +00:00
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
RWLOCK(&adb->names_lock, isc_rwlocktype_write);
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
for (dns_adbname_t *adbname = ISC_LIST_HEAD(adb->names_lru);
adbname != NULL; adbname = next)
{
next = ISC_LIST_NEXT(adbname, link);
update copyright notice
2008-02-07 23:46:54 +00:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
dns_adbname_ref(adbname);
LOCK(&adbname->lock);
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
/*
* Name hooks expire after the address record's TTL
* or 30 minutes, whichever is shorter. If after cleaning
* those up there are no name hooks left, and no active
* fetches, we can remove this name from the bucket.
*/
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
maybe_expire_namehooks(adbname, now);
(void)maybe_expire_name(adbname, now);
UNLOCK(&adbname->lock);
dns_adbname_detach(&adbname);
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
}
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
RWUNLOCK(&adb->names_lock, isc_rwlocktype_write);
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
}
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
/*%
* Examine the tail entry of the LRU list to see if it expires or is stale
* (unused for some period); if so, the name entry will be freed. If the ADB
* is in the overmem condition, the tail and the next to tail entries
* will be unconditionally removed (unless they have an outstanding fetch).
* We don't care about a race on 'overmem' at the risk of causing some
* collateral damage or a small delay in starting cleanup.
*
Don't expire an ADB entry without holding the entries lock The clean_namehooks() function does't hold the 'adb->entries_lock' lock, so calling maybe_expire_entry() is not thread-safe. Instead of adding a lock/unlock, leave the expiration to later, e.g. by the get_attached_and_locked_entry() function. Also fix a couple of comment typos.
2022-12-23 14:21:03 +00:00
* adb->entries_lock MUST be write locked
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
*/
static void
purge_stale_entries(dns_adb_t *adb, isc_stdtime_t now) {
Refactor the handling of isc_mem overmem condition Previously, there were two methods of working with the overmem condition: 1. hi/lo water callback - when the overmem condition was reached for the first time, the water callback was called with HIWATER mark and .is_overmem boolean was set internally. Similarly, when the used memory went below the lo water mark, the water callback would be called with LOWATER mark and .is_overmem was reset. This check would be called **every** time memory was allocated or freed. 2. isc_mem_isovermem() - a simple getter for the internal .is_overmem flag This commit refactors removes the first method and move the hi/lo water checks to the isc_mem_isovermem() function, thus we now have only a single method of checking overmem condition and the check for hi/lo water is removed from the hot path for memory contexts that doesn't use overmem checks.
2023-11-29 09:01:56 +01:00
bool overmem = isc_mem_isovermem(adb->mctx);
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
int max_removed = overmem ? 2 : 1;
int scans = 0, removed = 0;
dns_adbentry_t *prev = NULL;
/*
* We limit the number of scanned entries to 10 (arbitrary choice)
* in order to avoid examining too many entries when there are many
* tail entries that have fetches (this should be rare, but could
* happen).
*/
for (dns_adbentry_t *adbentry = ISC_LIST_TAIL(adb->entries_lru);
adbentry != NULL && removed < max_removed && scans < 10;
adbentry = prev)
{
prev = ISC_LIST_PREV(adbentry, link);
dns_adbentry_ref(adbentry);
LOCK(&adbentry->lock);
scans++;
/*
* Remove the entry if it's expired and unused.
*/
if (maybe_expire_entry(adbentry, now)) {
removed++;
goto next;
}
Don't expire fresh ADB names and entries The overmem cleaning in ADB could become overzealous and clean fresh ADB names and entries. Add a safety check to not clean any ADB names and entries that are below ADB_CACHE_MINIMUM threshold.
2022-12-13 14:32:19 +01:00
/*
Don't expire an ADB entry without holding the entries lock The clean_namehooks() function does't hold the 'adb->entries_lock' lock, so calling maybe_expire_entry() is not thread-safe. Instead of adding a lock/unlock, leave the expiration to later, e.g. by the get_attached_and_locked_entry() function. Also fix a couple of comment typos.
2022-12-23 14:21:03 +00:00
* Make sure that we are not purging ADB entry that has been
Don't expire fresh ADB names and entries The overmem cleaning in ADB could become overzealous and clean fresh ADB names and entries. Add a safety check to not clean any ADB names and entries that are below ADB_CACHE_MINIMUM threshold.
2022-12-13 14:32:19 +01:00
* just created.
*/
if (adbentry->last_used + ADB_CACHE_MINIMUM >= now) {
prev = NULL;
goto next;
}
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
if (overmem) {
maybe_expire_entry(adbentry, INT_MAX);
removed++;
goto next;
}
if (adbentry->last_used + ADB_STALE_MARGIN < now) {
maybe_expire_entry(adbentry, INT_MAX);
removed++;
goto next;
}
/*
Don't expire fresh ADB names and entries The overmem cleaning in ADB could become overzealous and clean fresh ADB names and entries. Add a safety check to not clean any ADB names and entries that are below ADB_CACHE_MINIMUM threshold.
2022-12-13 14:32:19 +01:00
* We won't expire anything on the LRU list as the
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
* .last_used + ADB_STALE_MARGIN will always be bigger
* than `now` for all previous entries, so we just stop
* the scanning
*/
prev = NULL;
next:
UNLOCK(&adbentry->lock);
dns_adbentry_detach(&adbentry);
}
}
static void
cleanup_entries(dns_adb_t *adb, isc_stdtime_t now) {
dns_adbentry_t *next = NULL;
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
RWLOCK(&adb->entries_lock, isc_rwlocktype_write);
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
for (dns_adbentry_t *adbentry = ISC_LIST_HEAD(adb->entries_lru);
adbentry != NULL; adbentry = next)
{
next = ISC_LIST_NEXT(adbentry, link);
dns_adbentry_ref(adbentry);
LOCK(&adbentry->lock);
maybe_expire_entry(adbentry, now);
UNLOCK(&adbentry->lock);
dns_adbentry_detach(&adbentry);
}
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
RWUNLOCK(&adb->entries_lock, isc_rwlocktype_write);
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
}
use isc_mutexblock_init/destroy()
1999-09-24 23:54:42 +00:00
static void
Megacommit of dozens of files. Cleanup of redundant/useless header file inclusion. ISC style lint, primarily for function declarations and standalone comments -- ie, those that appear on a line without any code, which should be written as follows: /* * This is a comment. */
2000-05-08 14:38:29 +00:00
destroy(dns_adb_t *adb) {
Remove several superfluous newlines in log messages
2022-04-05 13:16:00 +01:00
DP(DEF_LEVEL, "destroying ADB %p", adb);
Add detailed ADB and entry attach/detach tracing To turn on detailed debug tracing of dns_adb and dns_adbentry reference counting, #define ADB_TRACE at the top of adb.c. This is off by default.
2022-03-24 15:58:26 -07:00
update copyright notice
2008-02-07 23:46:54 +00:00
adb->magic = 0;
commit lruttl to the mainline. A tag was set called skan_lruttl-mainline-base, and I will tag this as skan_lruttl-mainline-merge after this commit
2007-10-19 17:15:53 +00:00
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
RWLOCK(&adb->names_lock, isc_rwlocktype_write);
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
INSIST(isc_hashmap_count(adb->names) == 0);
isc_hashmap_destroy(&adb->names);
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
RWUNLOCK(&adb->names_lock, isc_rwlocktype_write);
isc_rwlock_destroy(&adb->names_lock);
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
RWLOCK(&adb->entries_lock, isc_rwlocktype_write);
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
/* There are no unassociated entries */
INSIST(isc_hashmap_count(adb->entries) == 0);
isc_hashmap_destroy(&adb->entries);
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
RWUNLOCK(&adb->entries_lock, isc_rwlocktype_write);
isc_rwlock_destroy(&adb->entries_lock);
remove ADB whenshutdown events weakly attaching and detaching the view when creating or destroying the ADB obviates the need for a whenshutdown callback event to do the detaching. remove the dns_adb_whenshutdown() mechanism, since it is no longer needed.
2022-05-11 13:55:01 -07:00
Refactor isc_stats_create() and its downstream users to return void The isc_stats_create() can no longer return anything else than ISC_R_SUCCESS. Refactor isc_stats_create() and its variants in libdns, libns and named to just return void.
2023-06-26 10:58:30 +02:00
isc_mem_detach(&adb->hmctx);
Exclude the ADB hashmaps from ADB overmem accounting The ADB overmem accounting would include the memory used by hashtables thus vastly reducing the space that can be used for ADB names and entries when the hashtables would grow. Create own memory context for the ADB names and entries hash tables.
2022-12-13 14:14:21 +01:00
- Make isc_mutex_destroy return void - Make isc_mutexblock_init/destroy return void - Minor cleanups
2018-11-19 10:31:09 +00:00
isc_mutex_destroy(&adb->lock);
use isc_mutexblock_init/destroy()
1999-09-24 23:54:42 +00:00
move ADB and resolver stats out of the view object for better object separation, ADB and resolver statistics counters are now stored in the ADB and resolver objects themsevles, rather than in the associated view.
2022-05-11 15:38:54 -07:00
isc_stats_detach(&adb->stats);
attach the resolver to the ADB the ADB depends on the resolver, but previously only accessed it via the view. as view->resolver may now be detached before the ADB finishes, a shutdown race was possible. attaching to the resolver directly prevents this.
2022-05-12 15:51:10 -07:00
dns_resolver_detach(&adb->res);
remove ADB whenshutdown events weakly attaching and detaching the view when creating or destroying the ADB obviates the need for a whenshutdown callback event to do the detaching. remove the dns_adb_whenshutdown() mechanism, since it is no longer needed.
2022-05-11 13:55:01 -07:00
dns_view_weakdetach(&adb->view);
update copyright notice
2008-02-07 23:46:54 +00:00
isc_mem_putanddetach(&adb->mctx, adb, sizeof(dns_adb_t));
use isc_mutexblock_init/destroy()
1999-09-24 23:54:42 +00:00
}
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
#if DNS_ADB_TRACE
ISC_REFCOUNT_TRACE_IMPL(dns_adb, destroy);
#else
ISC_REFCOUNT_IMPL(dns_adb, destroy);
#endif
snapshot address.[ch]
1999-09-22 00:26:20 +00:00
/*
* Public functions.
*/
Refactor dns_adb_create() to return void After isc_stats_create() change, the dns_adb_create() cannot fail, so refactor the function to return void and fix all its uses.
2023-06-26 11:04:33 +02:00
void
use a thread-local variable to get the current running loop if we had a method to get the running loop, similar to how isc_tid() gets the current thread ID, we can simplify loop and loopmgr initialization. remove most uses of isc_loop_current() in favor of isc_loop(). in some places where that was the only reason to pass loopmgr, remove loopmgr from the function parameters.
2024-03-26 00:13:45 -07:00
dns_adb_create(isc_mem_t *mem, dns_view_t *view, dns_adb_t **newadb) {
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
dns_adb_t *adb = NULL;
update copyright notice
2008-02-07 23:46:54 +00:00
REQUIRE(mem != NULL);
REQUIRE(view != NULL);
REQUIRE(newadb != NULL && *newadb == NULL);
adb = isc_mem_get(mem, sizeof(dns_adb_t));
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
*adb = (dns_adb_t){
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
.names_lru = ISC_LIST_INITIALIZER,
.entries_lru = ISC_LIST_INITIALIZER,
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
};
update copyright notice
2008-02-07 23:46:54 +00:00
/*
* Initialize things here that cannot fail, and especially things
* that must be NULL for the error return to work properly.
*/
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
#if DNS_ADB_TRACE
fprintf(stderr, "dns_adb__init:%s:%s:%d:%p->references = 1\n", __func__,
__FILE__, __LINE__ + 1, adb);
#endif
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
isc_refcount_init(&adb->references, 1);
remove ADB whenshutdown events weakly attaching and detaching the view when creating or destroying the ADB obviates the need for a whenshutdown callback event to do the detaching. remove the dns_adb_whenshutdown() mechanism, since it is no longer needed.
2022-05-11 13:55:01 -07:00
dns_view_weakattach(view, &adb->view);
attach the resolver to the ADB the ADB depends on the resolver, but previously only accessed it via the view. as view->resolver may now be detached before the ADB finishes, a shutdown race was possible. attaching to the resolver directly prevents this.
2022-05-12 15:51:10 -07:00
dns_resolver_attach(view->resolver, &adb->res);
update copyright notice
2008-02-07 23:46:54 +00:00
isc_mem_attach(mem, &adb->mctx);
Exclude the ADB hashmaps from ADB overmem accounting The ADB overmem accounting would include the memory used by hashtables thus vastly reducing the space that can be used for ADB names and entries when the hashtables would grow. Create own memory context for the ADB names and entries hash tables.
2022-12-13 14:14:21 +01:00
isc_mem_create(&adb->hmctx);
Properly name ADB hashmap and named log memory contexts The ADB hashmaps are stored in extra memory contexts, so the hash tables are excluded from the overmem accounting. The new memory context was unnamed, give it a proper name. Same thing has happened with extra memory context used for named global log context - give the extra memory context a proper name.
2023-01-30 09:49:35 +01:00
isc_mem_setname(adb->hmctx, "ADB_hashmaps");
Exclude the ADB hashmaps from ADB overmem accounting The ADB overmem accounting would include the memory used by hashtables thus vastly reducing the space that can be used for ADB names and entries when the hashtables would grow. Create own memory context for the ADB names and entries hash tables.
2022-12-13 14:14:21 +01:00
Refactor isc_hashmap to accept custom match function Refactor isc_hashmap to allow custom matching functions. This allows us to have better tailored keys that don't require fixed uint8_t arrays, but can be composed of more fields from the stored data structure.
2023-08-28 10:08:59 +02:00
isc_hashmap_create(adb->hmctx, ADB_HASH_BITS, &adb->names);
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
isc_rwlock_init(&adb->names_lock);
2993. [func] Dynamically grow adb hash tables. [RT #21186]
2010-12-21 03:11:42 +00:00
Refactor isc_hashmap to accept custom match function Refactor isc_hashmap to allow custom matching functions. This allows us to have better tailored keys that don't require fixed uint8_t arrays, but can be composed of more fields from the stored data structure.
2023-08-28 10:08:59 +02:00
isc_hashmap_create(adb->hmctx, ADB_HASH_BITS, &adb->entries);
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
isc_rwlock_init(&adb->entries_lock);
- Make isc_mutex_destroy return void - Make isc_mutexblock_init/destroy return void - Minor cleanups
2018-11-19 10:31:09 +00:00
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
isc_mutex_init(&adb->lock);
update copyright notice
2008-02-07 23:46:54 +00:00
Refactor isc_stats_create() and its downstream users to return void The isc_stats_create() can no longer return anything else than ISC_R_SUCCESS. Refactor isc_stats_create() and its variants in libdns, libns and named to just return void.
2023-06-26 10:58:30 +02:00
isc_stats_create(adb->mctx, &adb->stats, dns_adbstats_max);
3353. [bug] Use a single task for task exclusive operations. [RT #29872]
2012-07-19 23:00:21 +10:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
set_adbstat(adb, 0, dns_adbstats_nnames);
set_adbstat(adb, 0, dns_adbstats_nentries);
Merge statistics code (ATT SoW, rt24117) This includes the following changes: 3326. [func] Added task list statistics: task model, worker threads, quantum, tasks running, tasks ready. [RT #27678] 3325. [func] Report cache statistics: memory use, number of nodes, number of hash buckets, hit and miss counts. [RT #27056] 3324. [test] Add better tests for ADB stats [RT #27057] 3323. [func] Report the number of buckets the resolver is using. [RT #27020] 3322. [func] Monitor the number of active TCP and UDP dispatches. [RT #27055] 3321. [func] Monitor the number of recursive fetches and the number of open sockets, and report these values in the statistics channel. [RT #27054] 3320. [func] Added support for monitoring of recursing client count. [RT #27009] 3319. [func] Added support for monitoring of ADB entry count and hash size. [RT #27057]
2012-05-14 10:06:05 -07:00
update copyright notice
2008-02-07 23:46:54 +00:00
/*
* Normal return.
*/
adb->magic = DNS_ADB_MAGIC;
*newadb = adb;
Start on address.c
1999-09-21 22:46:42 +00:00
}
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
void
dns_adb_shutdown(dns_adb_t *adb) {
if (!atomic_compare_exchange_strong(&adb->exiting, &(bool){ false },
Update sources to Clang 15 formatting
2022-11-02 19:33:14 +01:00
true))
{
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
return;
}
[rt36341] 3905. [bug] Address deadlock between view.c and adb.c. [RT #36341]
2014-07-31 11:38:11 +10:00
Remove several superfluous newlines in log messages
2022-04-05 13:16:00 +01:00
DP(DEF_LEVEL, "shutting down ADB %p", adb);
Add detailed ADB and entry attach/detach tracing To turn on detailed debug tracing of dns_adb and dns_adbentry reference counting, #define ADB_TRACE at the top of adb.c. This is off by default.
2022-03-24 15:58:26 -07:00
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
isc_mem_clearwater(adb->mctx);
[rt36341] 3905. [bug] Address deadlock between view.c and adb.c. [RT #36341]
2014-07-31 11:38:11 +10:00
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
shutdown_names(adb);
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
shutdown_entries(adb);
Start on address.c
1999-09-21 22:46:42 +00:00
}
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
/*
* Look up the name in our internal database.
*
* There are three possibilities. Note that these are not always exclusive.
*
* - No name found. In this case, allocate a new name header and
* an initial namehook or two.
*
* - Name found, valid addresses present. Allocate one addrinfo
* structure for each found and append it to the linked list
* of addresses for this header.
*
refactor dns_resolver to use loop callbacks callback events from dns_resolver_createfetch() are now posted using isc_async_run. other modules which called the resolver and maintained task/taskmgr objects for this purpose have been cleaned up.
2022-10-28 01:33:40 -07:00
* - Name found, queries pending. In this case, if a loop was
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
* passed in, allocate a job id, attach it to the name's job
* list and remember to tell the caller that there will be
* more info coming later.
*/
Start on address.c
1999-09-21 22:46:42 +00:00
isc_result_t
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
dns_adb_createfind(dns_adb_t *adb, isc_loop_t *loop, isc_job_cb cb, void *cbarg,
const dns_name_t *name, const dns_name_t *qname,
Remove dns_adblameinfo from dns_adb Keeping the information about lame server in the ADB was done in !322 to fix following security issue: [CVE-2021-25219] Disable "lame-ttl" cache The handling of the lame servers needs to be redesigned and it is not going to be enabled any time soon, and the current code is just dead code that takes up space, code and stands in the way of making ADB work faster. Remove all the internals needed for handling the lame servers in the ADB for now. It might get reintroduced later if and when we redesign ADB.
2023-09-21 15:20:58 +02:00
dns_rdatatype_t qtype ISC_ATTR_UNUSED, unsigned int options,
update copyright notice
2008-02-07 23:46:54 +00:00
isc_stdtime_t now, dns_name_t *target, in_port_t port,
libdns refactoring: get rid of two versions of dns_adb_createfind and dns_adb_probesize
2018-04-03 13:10:03 +02:00
unsigned int depth, isc_counter_t *qc,
dns_adbfind_t **findp) {
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
isc_result_t result = ISC_R_UNEXPECTED;
incidental cleanup The NAME_FETCH_A and NAME_FETCH_AAAA macros were meant to be boolean, indicating whether the pointers were set or not, while the NAME_FETCH_V4 and NAME_FETCH_V6 macros were meant to return the pointer values. The latter were only used as booleans, so they've been removed in favor of the former. Also did some style cleanup and removed an unreachable code block.
2021-10-18 17:57:58 -07:00
dns_adbfind_t *find = NULL;
dns_adbname_t *adbname = NULL;
bool want_event = true;
bool start_at_zone = false;
bool alias = false;
bool have_address = false;
unsigned int wanted_addresses = (options & DNS_ADBFIND_ADDRESSMASK);
unsigned int wanted_fetches = 0;
unsigned int query_pending = 0;
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
char namebuf[DNS_NAME_FORMATSIZE] = { 0 };
update copyright notice
2008-02-07 23:46:54 +00:00
REQUIRE(DNS_ADB_VALID(adb));
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
if (loop != NULL) {
REQUIRE(cb != NULL);
update copyright notice
2008-02-07 23:46:54 +00:00
}
REQUIRE(name != NULL);
REQUIRE(qname != NULL);
REQUIRE(findp != NULL && *findp == NULL);
REQUIRE(target == NULL || dns_name_hasbuffer(target));
REQUIRE((options & DNS_ADBFIND_ADDRESSMASK) != 0);
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
if (atomic_load(&adb->exiting)) {
Fix the data race when shutting down dns_adb When dns_adb is shutting down, first the adb->shutting_down flag is set and then task is created that runs shutdown_stage2() that sets the shutdown flag on names and entries. However, when dns_adb_createfind() is called, only the individual shutdown flags are being checked, and the global adb->shutting_down flag was not checked. Because of that it was possible for a different thread to slip in and create new find between the dns_adb_shutdown() and dns_adb_detach(), but before the shutdown_stage2() task is complete. This was detected by ThreadSanitizer as data race because the zonetable might have been already detached by dns_view shutdown process and simultaneously accessed by dns_adb_createfind(). This commit converts the adb->shutting_down to atomic_bool to prevent the global adb lock when creating the find.
2021-11-15 12:13:22 +01:00
DP(DEF_LEVEL, "dns_adb_createfind: returning "
"ISC_R_SHUTTINGDOWN");
return (ISC_R_SHUTTINGDOWN);
}
update copyright notice
2008-02-07 23:46:54 +00:00
if (now == 0) {
Apply the semantic patch to remove isc_stdtime_get() This is a simple replacement using the semantic patch from the previous commit and as added bonus, one removal of previously undetected unused variable in named/server.c.
2023-03-30 21:13:41 +02:00
now = isc_stdtime_now();
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
/*
* Remember what types of addresses we are interested in.
*/
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
find = new_adbfind(adb, port);
update copyright notice
2008-02-07 23:46:54 +00:00
find->options = options;
find->flags |= wanted_addresses;
if (FIND_WANTEVENT(find)) {
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
REQUIRE(loop != NULL);
update copyright notice
2008-02-07 23:46:54 +00:00
}
3593. [func] Update EDNS processing to better track remote server capabilities. [RT #30655]
2013-06-12 11:31:30 +10:00
if (isc_log_wouldlog(dns_lctx, DEF_LEVEL)) {
dns_name_format(name, namebuf, sizeof(namebuf));
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
3593. [func] Update EDNS processing to better track remote server capabilities. [RT #30655]
2013-06-12 11:31:30 +10:00
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
again:
/* Try to see if we know anything about this name at all. */
Lock the adbname and adbentry prior to unlocking hash locks There was a datarace that could expire a freshly created ADB names and entries between the return from get_attached_{name,entry} and locking it again. Lock the ADB name and ADB entry inside the hash table lock, so they won't get expired until the full initialization has been complete.
2022-12-13 13:48:55 +01:00
adbname = get_attached_and_locked_name(adb, name,
FIND_STARTATZONE(find), now);
update copyright notice
2008-02-07 23:46:54 +00:00
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
if (NAME_DEAD(adbname)) {
UNLOCK(&adbname->lock);
dns_adbname_detach(&adbname);
goto again;
}
update copyright notice
2008-02-07 23:46:54 +00:00
/*
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
* Name hooks expire after the address record's TTL or 30 minutes,
* whichever is shorter. If there are expired name hooks, remove
* them so we'll send a new fetch.
update copyright notice
2008-02-07 23:46:54 +00:00
*/
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
maybe_expire_namehooks(adbname, now);
update copyright notice
2008-02-07 23:46:54 +00:00
/*
* Do we know that the name is an alias?
*/
if (!EXPIRE_OK(adbname->expire_target, now)) {
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
/* Yes, it is. */
update copyright notice
2008-02-07 23:46:54 +00:00
DP(DEF_LEVEL,
3593. [func] Update EDNS processing to better track remote server capabilities. [RT #30655]
2013-06-12 11:31:30 +10:00
"dns_adb_createfind: name %s (%p) is an alias (cached)",
namebuf, adbname);
Replace custom isc_boolean_t with C standard bool type
2018-04-17 08:29:14 -07:00
alias = true;
update copyright notice
2008-02-07 23:46:54 +00:00
goto post_copy;
}
/*
* Try to populate the name from the database and/or
* start fetches. First try looking for an A record
* in the database.
*/
if (!NAME_HAS_V4(adbname) && EXPIRE_OK(adbname->expire_v4, now) &&
WANT_INET(wanted_addresses))
{
result = dbfind_name(adbname, now, dns_rdatatype_a);
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
switch (result) {
case ISC_R_SUCCESS:
/* Found an A; now we proceed to check for AAAA */
update copyright notice
2008-02-07 23:46:54 +00:00
DP(DEF_LEVEL,
3593. [func] Update EDNS processing to better track remote server capabilities. [RT #30655]
2013-06-12 11:31:30 +10:00
"dns_adb_createfind: found A for name %s (%p) in db",
namebuf, adbname);
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
break;
update copyright notice
2008-02-07 23:46:54 +00:00
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
case DNS_R_ALIAS:
/* Got a CNAME or DNAME. */
update copyright notice
2008-02-07 23:46:54 +00:00
DP(DEF_LEVEL,
3593. [func] Update EDNS processing to better track remote server capabilities. [RT #30655]
2013-06-12 11:31:30 +10:00
"dns_adb_createfind: name %s (%p) is an alias",
namebuf, adbname);
Replace custom isc_boolean_t with C standard bool type
2018-04-17 08:29:14 -07:00
alias = true;
update copyright notice
2008-02-07 23:46:54 +00:00
goto post_copy;
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
case DNS_R_NXDOMAIN:
case DNS_R_NCACHENXDOMAIN:
/*
* If the name doesn't exist at all, don't bother with
* v6 queries; they won't work.
*/
update copyright notice
2008-02-07 23:46:54 +00:00
goto fetch;
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
case DNS_R_NXRRSET:
case DNS_R_NCACHENXRRSET:
case DNS_R_HINTNXRRSET:
/*
* The name does exist but we didn't get our data, go
* ahead and try AAAA.
*/
break;
default:
/*
* Any other result, start a fetch for A, then fall
* through to AAAA.
*/
if (!NAME_FETCH_A(adbname)) {
wanted_fetches |= DNS_ADBFIND_INET;
}
break;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
}
fetch contexts
1999-10-26 01:12:20 +00:00
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
/*
* Now look up or start fetches for AAAA.
*/
update copyright notice
2008-02-07 23:46:54 +00:00
if (!NAME_HAS_V6(adbname) && EXPIRE_OK(adbname->expire_v6, now) &&
WANT_INET6(wanted_addresses))
{
result = dbfind_name(adbname, now, dns_rdatatype_aaaa);
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
switch (result) {
case ISC_R_SUCCESS:
update copyright notice
2008-02-07 23:46:54 +00:00
DP(DEF_LEVEL,
3593. [func] Update EDNS processing to better track remote server capabilities. [RT #30655]
2013-06-12 11:31:30 +10:00
"dns_adb_createfind: found AAAA for name %s (%p)",
namebuf, adbname);
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
break;
update copyright notice
2008-02-07 23:46:54 +00:00
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
case DNS_R_ALIAS:
/* Got a CNAME or DNAME. */
update copyright notice
2008-02-07 23:46:54 +00:00
DP(DEF_LEVEL,
3593. [func] Update EDNS processing to better track remote server capabilities. [RT #30655]
2013-06-12 11:31:30 +10:00
"dns_adb_createfind: name %s (%p) is an alias",
namebuf, adbname);
Replace custom isc_boolean_t with C standard bool type
2018-04-17 08:29:14 -07:00
alias = true;
update copyright notice
2008-02-07 23:46:54 +00:00
goto post_copy;
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
case DNS_R_NXDOMAIN:
case DNS_R_NCACHENXDOMAIN:
case DNS_R_NXRRSET:
case DNS_R_NCACHENXRRSET:
/*
* Name doens't exist or was found in the negative
* cache to have no AAAA, don't bother fetching.
*/
break;
update copyright notice
2008-02-07 23:46:54 +00:00
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
default:
/*
* Any other result, start a fetch for AAAA.
*/
if (!NAME_FETCH_AAAA(adbname)) {
wanted_fetches |= DNS_ADBFIND_INET6;
}
break;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
}
Fix a lot of negative caching issues. I believe the code will use ncache responses from fetches and database lookups correctly now, but there needs to be a little more cleanup here and there. Bugs exist.
1999-11-24 19:23:27 +00:00
IPv6 query flood suppression
1999-12-14 00:10:06 +00:00
fetch:
update copyright notice
2008-02-07 23:46:54 +00:00
if ((WANT_INET(wanted_addresses) && NAME_HAS_V4(adbname)) ||
(WANT_INET6(wanted_addresses) && NAME_HAS_V6(adbname)))
{
Replace custom isc_boolean_t with C standard bool type
2018-04-17 08:29:14 -07:00
have_address = true;
update copyright notice
2008-02-07 23:46:54 +00:00
} else {
Replace custom isc_boolean_t with C standard bool type
2018-04-17 08:29:14 -07:00
have_address = false;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
Reduce the number of fetches we make when looking up addresses If there are more that 5 NS record for a zone only perform a maximum of 4 address lookups for all the name servers. This limits the amount of remote lookup performed for server addresses at each level for a given query.
2020-02-06 17:19:10 +11:00
if (wanted_fetches != 0 && !(FIND_AVOIDFETCHES(find) && have_address) &&
!FIND_NOFETCH(find))
{
update copyright notice
2008-02-07 23:46:54 +00:00
/*
* We're missing at least one address family. Either the
* caller hasn't instructed us to avoid fetches, or we don't
* know anything about any of the address families that would
* be acceptable so we have to launch fetches.
*/
if (FIND_STARTATZONE(find)) {
Replace custom isc_boolean_t with C standard bool type
2018-04-17 08:29:14 -07:00
start_at_zone = true;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
/*
* Start V4.
*/
if (WANT_INET(wanted_fetches) &&
[master] refactor max-recursion-queries - the counters weren't set correctly when fetches timed out. instead we now pass down a counter object.
2014-11-19 18:21:02 -08:00
fetch_name(adbname, start_at_zone, depth, qc,
checkpoint
2018-09-28 16:51:55 +10:00
dns_rdatatype_a) == ISC_R_SUCCESS)
{
3593. [func] Update EDNS processing to better track remote server capabilities. [RT #30655]
2013-06-12 11:31:30 +10:00
DP(DEF_LEVEL,
"dns_adb_createfind: "
"started A fetch for name %s (%p)",
namebuf, adbname);
update copyright notice
2008-02-07 23:46:54 +00:00
}
/*
* Start V6.
*/
if (WANT_INET6(wanted_fetches) &&
[master] refactor max-recursion-queries - the counters weren't set correctly when fetches timed out. instead we now pass down a counter object.
2014-11-19 18:21:02 -08:00
fetch_name(adbname, start_at_zone, depth, qc,
checkpoint
2018-09-28 16:51:55 +10:00
dns_rdatatype_aaaa) == ISC_R_SUCCESS)
{
3593. [func] Update EDNS processing to better track remote server capabilities. [RT #30655]
2013-06-12 11:31:30 +10:00
DP(DEF_LEVEL,
"dns_adb_createfind: "
"started AAAA fetch for name %s (%p)",
namebuf, adbname);
update copyright notice
2008-02-07 23:46:54 +00:00
}
}
/*
* Run through the name and copy out the bits we are
* interested in.
*/
Remove dns_adblameinfo from dns_adb Keeping the information about lame server in the ADB was done in !322 to fix following security issue: [CVE-2021-25219] Disable "lame-ttl" cache The handling of the lame servers needs to be redesigned and it is not going to be enabled any time soon, and the current code is just dead code that takes up space, code and stands in the way of making ADB work faster. Remove all the internals needed for handling the lame servers in the ADB for now. It might get reintroduced later if and when we redesign ADB.
2023-09-21 15:20:58 +02:00
copy_namehook_lists(adb, find, adbname);
clean up from Bob's quick code walk-through
1999-10-21 01:37:57 +00:00
always set find->query_pending correctly
2000-01-27 02:14:53 +00:00
post_copy:
incidental cleanup The NAME_FETCH_A and NAME_FETCH_AAAA macros were meant to be boolean, indicating whether the pointers were set or not, while the NAME_FETCH_V4 and NAME_FETCH_V6 macros were meant to return the pointer values. The latter were only used as booleans, so they've been removed in favor of the former. Also did some style cleanup and removed an unreachable code block.
2021-10-18 17:57:58 -07:00
if (NAME_FETCH_A(adbname)) {
update copyright notice
2008-02-07 23:46:54 +00:00
query_pending |= DNS_ADBFIND_INET;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
incidental cleanup The NAME_FETCH_A and NAME_FETCH_AAAA macros were meant to be boolean, indicating whether the pointers were set or not, while the NAME_FETCH_V4 and NAME_FETCH_V6 macros were meant to return the pointer values. The latter were only used as booleans, so they've been removed in favor of the former. Also did some style cleanup and removed an unreachable code block.
2021-10-18 17:57:58 -07:00
if (NAME_FETCH_AAAA(adbname)) {
update copyright notice
2008-02-07 23:46:54 +00:00
query_pending |= DNS_ADBFIND_INET6;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
add find->result_v[46]
2000-08-26 02:21:45 +00:00
update copyright notice
2008-02-07 23:46:54 +00:00
/*
* Attach to the name's query list if there are queries
* already running, and we have been asked to.
*/
if (!FIND_WANTEVENT(find)) {
Replace custom isc_boolean_t with C standard bool type
2018-04-17 08:29:14 -07:00
want_event = false;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
if (FIND_WANTEMPTYEVENT(find) && FIND_HAS_ADDRS(find)) {
Replace custom isc_boolean_t with C standard bool type
2018-04-17 08:29:14 -07:00
want_event = false;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
if ((wanted_addresses & query_pending) == 0) {
Replace custom isc_boolean_t with C standard bool type
2018-04-17 08:29:14 -07:00
want_event = false;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
if (alias) {
Replace custom isc_boolean_t with C standard bool type
2018-04-17 08:29:14 -07:00
want_event = false;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
if (want_event) {
incidental cleanup The NAME_FETCH_A and NAME_FETCH_AAAA macros were meant to be boolean, indicating whether the pointers were set or not, while the NAME_FETCH_V4 and NAME_FETCH_V6 macros were meant to return the pointer values. The latter were only used as booleans, so they've been removed in favor of the former. Also did some style cleanup and removed an unreachable code block.
2021-10-18 17:57:58 -07:00
bool empty;
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
update copyright notice
2008-02-07 23:46:54 +00:00
find->adbname = adbname;
incidental cleanup The NAME_FETCH_A and NAME_FETCH_AAAA macros were meant to be boolean, indicating whether the pointers were set or not, while the NAME_FETCH_V4 and NAME_FETCH_V6 macros were meant to return the pointer values. The latter were only used as booleans, so they've been removed in favor of the former. Also did some style cleanup and removed an unreachable code block.
2021-10-18 17:57:58 -07:00
empty = ISC_LIST_EMPTY(adbname->finds);
update copyright notice
2008-02-07 23:46:54 +00:00
ISC_LIST_APPEND(adbname->finds, find, plink);
find->query_pending = (query_pending & wanted_addresses);
find->flags &= ~DNS_ADBFIND_ADDRESSMASK;
find->flags |= (find->query_pending & DNS_ADBFIND_ADDRESSMASK);
incidental cleanup The NAME_FETCH_A and NAME_FETCH_AAAA macros were meant to be boolean, indicating whether the pointers were set or not, while the NAME_FETCH_V4 and NAME_FETCH_V6 macros were meant to return the pointer values. The latter were only used as booleans, so they've been removed in favor of the former. Also did some style cleanup and removed an unreachable code block.
2021-10-18 17:57:58 -07:00
DP(DEF_LEVEL, "createfind: attaching find %p to adbname %p %d",
checkpoint
2018-09-28 16:51:55 +10:00
find, adbname, empty);
update copyright notice
2008-02-07 23:46:54 +00:00
} else {
/*
* Remove the flag so the caller knows there will never
* be an event, and set internal flags to fake that
* the event was sent and freed, so dns_adb_destroyfind() will
* do the right thing.
*/
find->query_pending = (query_pending & wanted_addresses);
find->options &= ~DNS_ADBFIND_WANTEVENT;
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
find->flags |= FIND_EVENT_SENT;
update copyright notice
2008-02-07 23:46:54 +00:00
find->flags &= ~DNS_ADBFIND_ADDRESSMASK;
}
snapshot
1999-10-08 23:12:21 +00:00
update copyright notice
2008-02-07 23:46:54 +00:00
find->partial_result |= (adbname->partial_result & wanted_addresses);
if (alias) {
if (target != NULL) {
rename dns_name_copynf() to dns_name_copy() dns_name_copy() is now the standard name-copying function.
2021-05-21 17:20:44 -07:00
dns_name_copy(&adbname->target, target);
update copyright notice
2008-02-07 23:46:54 +00:00
}
result = DNS_R_ALIAS;
} else {
result = ISC_R_SUCCESS;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
snapshot; everything works other than fetches and database lookups.
1999-10-16 00:38:21 +00:00
update copyright notice
2008-02-07 23:46:54 +00:00
/*
* Copy out error flags from the name structure into the find.
*/
find->result_v4 = find_err_map[adbname->fetch_err];
find->result_v6 = find_err_map[adbname->fetch6_err];
snapshot; everything works other than fetches and database lookups.
1999-10-16 00:38:21 +00:00
Remove unnecessary NULL test leading to REVERSE_INULL false positive *** CID 351371: Null pointer dereferences (REVERSE_INULL) /lib/dns/adb.c: 2615 in dns_adb_createfind() 2609 /* 2610 * Copy out error flags from the name structure into the find. 2611 */ 2612 find->result_v4 = find_err_map[adbname->fetch_err]; 2613 find->result_v6 = find_err_map[adbname->fetch6_err]; 2614 >>> CID 351371: Null pointer dereferences (REVERSE_INULL) >>> Null-checking "find" suggests that it may be null, but it has already been dereferenced on all paths leading to the check. 2615 if (find != NULL) { 2616 if (want_event) { 2617 INSIST((find->flags & DNS_ADBFIND_ADDRESSMASK) != 0); 2618 isc_task_attach(task, &(isc_task_t *){ NULL }); 2619 find->event.ev_sender = task; 2620 find->event.ev_action = action;
2022-04-06 12:54:08 +10:00
if (want_event) {
INSIST((find->flags & DNS_ADBFIND_ADDRESSMASK) != 0);
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
find->loop = loop;
find->status = DNS_ADB_UNSET;
find->cb = cb;
find->cbarg = cbarg;
update copyright notice
2008-02-07 23:46:54 +00:00
}
Remove unnecessary NULL test leading to REVERSE_INULL false positive *** CID 351371: Null pointer dereferences (REVERSE_INULL) /lib/dns/adb.c: 2615 in dns_adb_createfind() 2609 /* 2610 * Copy out error flags from the name structure into the find. 2611 */ 2612 find->result_v4 = find_err_map[adbname->fetch_err]; 2613 find->result_v6 = find_err_map[adbname->fetch6_err]; 2614 >>> CID 351371: Null pointer dereferences (REVERSE_INULL) >>> Null-checking "find" suggests that it may be null, but it has already been dereferenced on all paths leading to the check. 2615 if (find != NULL) { 2616 if (want_event) { 2617 INSIST((find->flags & DNS_ADBFIND_ADDRESSMASK) != 0); 2618 isc_task_attach(task, &(isc_task_t *){ NULL }); 2619 find->event.ev_sender = task; 2620 find->event.ev_action = action;
2022-04-06 12:54:08 +10:00
*findp = find;
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
UNLOCK(&adbname->lock);
dns_adbname_detach(&adbname);
update copyright notice
2008-02-07 23:46:54 +00:00
return (result);
add dns_adb_refresh()
1999-09-25 01:56:10 +00:00
}
Start on address.c
1999-09-21 22:46:42 +00:00
void
Megacommit of dozens of files. Cleanup of redundant/useless header file inclusion. ISC style lint, primarily for function declarations and standalone comments -- ie, those that appear on a line without any code, which should be written as follows: /* * This is a comment. */
2000-05-08 14:38:29 +00:00
dns_adb_destroyfind(dns_adbfind_t **findp) {
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
dns_adbfind_t *find = NULL;
dns_adbaddrinfo_t *ai = NULL;
dns_adb_t *adb = NULL;
update copyright notice
2008-02-07 23:46:54 +00:00
REQUIRE(findp != NULL && DNS_ADBFIND_VALID(*findp));
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
update copyright notice
2008-02-07 23:46:54 +00:00
find = *findp;
*findp = NULL;
DP(DEF_LEVEL, "dns_adb_destroyfind on find %p", find);
adb = find->adb;
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
LOCK(&find->lock);
update copyright notice
2008-02-07 23:46:54 +00:00
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
REQUIRE(find->adbname == NULL);
update copyright notice
2008-02-07 23:46:54 +00:00
/*
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
* Free the addrinfo objects on the find's list. Note that
* we also need to decrement the reference counter in the
* associated adbentry every time we remove one from the list.
update copyright notice
2008-02-07 23:46:54 +00:00
*/
ai = ISC_LIST_HEAD(find->list);
while (ai != NULL) {
ISC_LIST_UNLINK(find->list, ai, publink);
free_adbaddrinfo(adb, &ai);
ai = ISC_LIST_HEAD(find->list);
}
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
UNLOCK(&find->lock);
update copyright notice
2008-02-07 23:46:54 +00:00
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
free_adbfind(&find);
Start on address.c
1999-09-21 22:46:42 +00:00
}
Many changes. o added adb_test.c to test the functions already written. o added code to delete names and all associated addresses. o added a database dumper which will currently print the adb in a human (or at least developer) readable format. o found and fixed many twisted little locking issues.
1999-10-02 01:59:09 +00:00
lock find when unlinking adbname->finds in dns_adb_cancelfind() In dns_adb_cancelfind(), we need to release the find lock and then acquire the bucket and find locks in that order, for consistency with locking hierarchy elsehwere. Previously we were only acquiring the bucket lock. Also rewrote the function for better readability.
2022-04-19 19:14:49 -07:00
/*
* Caller must hold find lock.
*/
static void
find_sendevent(dns_adbfind_t *find) {
if (!FIND_EVENTSENT(find)) {
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
find->status = DNS_ADB_CANCELED;
lock find when unlinking adbname->finds in dns_adb_cancelfind() In dns_adb_cancelfind(), we need to release the find lock and then acquire the bucket and find locks in that order, for consistency with locking hierarchy elsehwere. Previously we were only acquiring the bucket lock. Also rewrote the function for better readability.
2022-04-19 19:14:49 -07:00
find->result_v4 = ISC_R_CANCELED;
find->result_v6 = ISC_R_CANCELED;
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
DP(DEF_LEVEL, "sending find %p to caller", find);
lock find when unlinking adbname->finds in dns_adb_cancelfind() In dns_adb_cancelfind(), we need to release the find lock and then acquire the bucket and find locks in that order, for consistency with locking hierarchy elsehwere. Previously we were only acquiring the bucket lock. Also rewrote the function for better readability.
2022-04-19 19:14:49 -07:00
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
isc_async_run(find->loop, find->cb, find);
lock find when unlinking adbname->finds in dns_adb_cancelfind() In dns_adb_cancelfind(), we need to release the find lock and then acquire the bucket and find locks in that order, for consistency with locking hierarchy elsehwere. Previously we were only acquiring the bucket lock. Also rewrote the function for better readability.
2022-04-19 19:14:49 -07:00
}
}
implement dns_adb_cancelfind(), rename _done() to _destroyfind(), and _lookup to _createfind()
1999-10-29 18:00:31 +00:00
void
Megacommit of dozens of files. Cleanup of redundant/useless header file inclusion. ISC style lint, primarily for function declarations and standalone comments -- ie, those that appear on a line without any code, which should be written as follows: /* * This is a comment. */
2000-05-08 14:38:29 +00:00
dns_adb_cancelfind(dns_adbfind_t *find) {
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
dns_adbname_t *adbname = NULL;
update copyright notice
2008-02-07 23:46:54 +00:00
DP(DEF_LEVEL, "dns_adb_cancelfind on find %p", find);
lock find when unlinking adbname->finds in dns_adb_cancelfind() In dns_adb_cancelfind(), we need to release the find lock and then acquire the bucket and find locks in that order, for consistency with locking hierarchy elsehwere. Previously we were only acquiring the bucket lock. Also rewrote the function for better readability.
2022-04-19 19:14:49 -07:00
REQUIRE(DNS_ADBFIND_VALID(find));
REQUIRE(DNS_ADB_VALID(find->adb));
update copyright notice
2008-02-07 23:46:54 +00:00
lock find when unlinking adbname->finds in dns_adb_cancelfind() In dns_adb_cancelfind(), we need to release the find lock and then acquire the bucket and find locks in that order, for consistency with locking hierarchy elsehwere. Previously we were only acquiring the bucket lock. Also rewrote the function for better readability.
2022-04-19 19:14:49 -07:00
LOCK(&find->lock);
update copyright notice
2008-02-07 23:46:54 +00:00
REQUIRE(FIND_WANTEVENT(find));
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
adbname = find->adbname;
update copyright notice
2008-02-07 23:46:54 +00:00
lock find when unlinking adbname->finds in dns_adb_cancelfind() In dns_adb_cancelfind(), we need to release the find lock and then acquire the bucket and find locks in that order, for consistency with locking hierarchy elsehwere. Previously we were only acquiring the bucket lock. Also rewrote the function for better readability.
2022-04-19 19:14:49 -07:00
if (adbname == NULL) {
find_sendevent(find);
UNLOCK(&find->lock);
} else {
/*
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
* Release the find lock, then acquire the name and find
lock find when unlinking adbname->finds in dns_adb_cancelfind() In dns_adb_cancelfind(), we need to release the find lock and then acquire the bucket and find locks in that order, for consistency with locking hierarchy elsehwere. Previously we were only acquiring the bucket lock. Also rewrote the function for better readability.
2022-04-19 19:14:49 -07:00
* locks in that order, to match locking hierarchy
* elsewhere.
*/
UNLOCK(&find->lock);
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
LOCK(&adbname->lock);
lock find when unlinking adbname->finds in dns_adb_cancelfind() In dns_adb_cancelfind(), we need to release the find lock and then acquire the bucket and find locks in that order, for consistency with locking hierarchy elsehwere. Previously we were only acquiring the bucket lock. Also rewrote the function for better readability.
2022-04-19 19:14:49 -07:00
LOCK(&find->lock);
implement dns_adb_cancelfind(), rename _done() to _destroyfind(), and _lookup to _createfind()
1999-10-29 18:00:31 +00:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
ISC_LIST_UNLINK(adbname->finds, find, plink);
find->adbname = NULL;
lock find when unlinking adbname->finds in dns_adb_cancelfind() In dns_adb_cancelfind(), we need to release the find lock and then acquire the bucket and find locks in that order, for consistency with locking hierarchy elsehwere. Previously we were only acquiring the bucket lock. Also rewrote the function for better readability.
2022-04-19 19:14:49 -07:00
find_sendevent(find);
implement dns_adb_cancelfind(), rename _done() to _destroyfind(), and _lookup to _createfind()
1999-10-29 18:00:31 +00:00
lock find when unlinking adbname->finds in dns_adb_cancelfind() In dns_adb_cancelfind(), we need to release the find lock and then acquire the bucket and find locks in that order, for consistency with locking hierarchy elsehwere. Previously we were only acquiring the bucket lock. Also rewrote the function for better readability.
2022-04-19 19:14:49 -07:00
UNLOCK(&find->lock);
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
UNLOCK(&adbname->lock);
update copyright notice
2008-02-07 23:46:54 +00:00
}
implement dns_adb_cancelfind(), rename _done() to _destroyfind(), and _lookup to _createfind()
1999-10-29 18:00:31 +00:00
}
Many changes. o added adb_test.c to test the functions already written. o added code to delete names and all associated addresses. o added a database dumper which will currently print the adb in a human (or at least developer) readable format. o found and fixed many twisted little locking issues.
1999-10-02 01:59:09 +00:00
void
Megacommit of dozens of files. Cleanup of redundant/useless header file inclusion. ISC style lint, primarily for function declarations and standalone comments -- ie, those that appear on a line without any code, which should be written as follows: /* * This is a comment. */
2000-05-08 14:38:29 +00:00
dns_adb_dump(dns_adb_t *adb, FILE *f) {
Apply the semantic patch to remove isc_stdtime_get() This is a simple replacement using the semantic patch from the previous commit and as added bonus, one removal of previously undetected unused variable in named/server.c.
2023-03-30 21:13:41 +02:00
isc_stdtime_t now = isc_stdtime_now();
1761. [bug] 'rndc dumpdb' didn't report unassociated entries. [RT #12971]
2004-11-10 22:33:18 +00:00
update copyright notice
2008-02-07 23:46:54 +00:00
REQUIRE(DNS_ADB_VALID(adb));
REQUIRE(f != NULL);
Many changes. o added adb_test.c to test the functions already written. o added code to delete names and all associated addresses. o added a database dumper which will currently print the adb in a human (or at least developer) readable format. o found and fixed many twisted little locking issues.
1999-10-02 01:59:09 +00:00
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
if (atomic_load(&adb->exiting)) {
return;
}
Many changes. o added adb_test.c to test the functions already written. o added code to delete names and all associated addresses. o added a database dumper which will currently print the adb in a human (or at least developer) readable format. o found and fixed many twisted little locking issues.
1999-10-02 01:59:09 +00:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
cleanup_names(adb, now);
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
cleanup_entries(adb, now);
Replace custom isc_boolean_t with C standard bool type
2018-04-17 08:29:14 -07:00
dump_adb(adb, f, false, now);
snapshot; nearly all fetch infrastructure is in place.
1999-10-27 19:36:58 +00:00
}
static void
dump the ADB in a more user-friendly format (unless a debug flag is set)
2000-12-18 20:03:33 +00:00
dump_ttl(FILE *f, const char *legend, isc_stdtime_t value, isc_stdtime_t now) {
update copyright notice
2008-02-07 23:46:54 +00:00
if (value == INT_MAX) {
return;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
formally cast to int
2018-02-14 19:13:56 +11:00
fprintf(f, " [%s TTL %d]", legend, (int)(value - now));
dump the ADB in a more user-friendly format (unless a debug flag is set)
2000-12-18 20:03:33 +00:00
}
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
/*
* Both rwlocks for the hash tables need to be held by the caller.
*/
dump the ADB in a more user-friendly format (unless a debug flag is set)
2000-12-18 20:03:33 +00:00
static void
Replace custom isc_boolean_t with C standard bool type
2018-04-17 08:29:14 -07:00
dump_adb(dns_adb_t *adb, FILE *f, bool debug, isc_stdtime_t now) {
update copyright notice
2008-02-07 23:46:54 +00:00
fprintf(f, ";\n; Address database dump\n;\n");
Simplify the EDNS buffer size logic for DNS Flag Day 2020 The DNS Flag Day 2020 aims to remove the IP fragmentation problem from the UDP DNS communication. In this commit, we implement the required changes and simplify the logic for picking the EDNS Buffer Size. 1. The defaults for `edns-udp-size`, `max-udp-size` and `nocookie-udp-size` have been changed to `1232` (the value picked by DNS Flag Day 2020). 2. The probing heuristics that would try 512->4096->1432->1232 buffer sizes has been removed and the resolver will always use just the `edns-udp-size` value. 3. Instead of just disabling the PMTUD mechanism on the UDP sockets, we now set IP_DONTFRAG (IPV6_DONTFRAG) flag. That means that the UDP packets won't get ever fragmented. If the ICMP packets are lost the UDP will just timeout and eventually be retried over TCP.
2020-09-23 14:47:26 +02:00
fprintf(f, "; [edns success/timeout]\n");
add partial legend for Address database dump
2013-11-13 12:10:43 +11:00
fprintf(f, "; [plain success/timeout]\n;\n");
Fix missing adb->{e,i}refcnt locking
2019-11-19 13:49:28 +08:00
if (debug) {
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
fprintf(f, "; addr %p, references %" PRIuFAST32 "\n", adb,
isc_refcount_current(&adb->references));
Fix missing adb->{e,i}refcnt locking
2019-11-19 13:49:28 +08:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
/*
* Ensure this operation is applied to both hash tables at once.
*/
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
RWLOCK(&adb->names_lock, isc_rwlocktype_write);
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
for (dns_adbname_t *name = ISC_LIST_HEAD(adb->names_lru); name != NULL;
name = ISC_LIST_NEXT(name, link))
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
{
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
LOCK(&name->lock);
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
/*
* Dump the names
*/
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
if (debug) {
fprintf(f, "; name %p (flags %08x)\n", name,
name->flags);
}
fprintf(f, "; ");
Refactor isc_hashmap to accept custom match function Refactor isc_hashmap to allow custom matching functions. This allows us to have better tailored keys that don't require fixed uint8_t arrays, but can be composed of more fields from the stored data structure.
2023-08-28 10:08:59 +02:00
dns_name_print(name->name, f);
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
if (dns_name_countlabels(&name->target) > 0) {
fprintf(f, " alias ");
dns_name_print(&name->target, f);
update copyright notice
2008-02-07 23:46:54 +00:00
}
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
dump_ttl(f, "v4", name->expire_v4, now);
dump_ttl(f, "v6", name->expire_v6, now);
dump_ttl(f, "target", name->expire_target, now);
update copyright notice
2008-02-07 23:46:54 +00:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
fprintf(f, " [v4 %s] [v6 %s]", errnames[name->fetch_err],
errnames[name->fetch6_err]);
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
fprintf(f, "\n");
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
print_namehook_list(f, "v4", adb, &name->v4, debug, now);
print_namehook_list(f, "v6", adb, &name->v6, debug, now);
if (debug) {
print_fetch_list(f, name);
print_find_list(f, name);
update copyright notice
2008-02-07 23:46:54 +00:00
}
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
UNLOCK(&name->lock);
lib/dns/adb.c: don't use more than 64 lock simultaneously when run under TSAN - TSAN can't handle more than 64 locks in one thread, lock ADB bucket-by-bucket in TSAN mode. This means that the dump won't be consistent but it's good enough for testing - Use proper order when unlocking adb->namelocks and adb->entrylocks when dumping ADB.
2019-01-22 10:47:18 +01:00
}
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
RWLOCK(&adb->entries_lock, isc_rwlocktype_write);
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
fprintf(f, ";\n; Unassociated entries\n;\n");
for (dns_adbentry_t *adbentry = ISC_LIST_HEAD(adb->entries_lru);
adbentry != NULL; adbentry = ISC_LIST_NEXT(adbentry, link))
{
LOCK(&adbentry->lock);
if (ISC_LIST_EMPTY(adbentry->nhs)) {
dump_entry(f, adb, adbentry, debug, now);
}
UNLOCK(&adbentry->lock);
}
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
RWUNLOCK(&adb->entries_lock, isc_rwlocktype_write);
RWUNLOCK(&adb->names_lock, isc_rwlocktype_write);
Many changes. o added adb_test.c to test the functions already written. o added code to delete names and all associated addresses. o added a database dumper which will currently print the adb in a human (or at least developer) readable format. o found and fixed many twisted little locking issues.
1999-10-02 01:59:09 +00:00
}
dump the ADB in a more user-friendly format (unless a debug flag is set)
2000-12-18 20:03:33 +00:00
static void
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
dump_entry(FILE *f, dns_adb_t *adb, dns_adbentry_t *entry, bool debug,
Replace custom isc_boolean_t with C standard bool type
2018-04-17 08:29:14 -07:00
isc_stdtime_t now) {
update copyright notice
2008-02-07 23:46:54 +00:00
char addrbuf[ISC_NETADDR_FORMATSIZE];
isc_netaddr_t netaddr;
isc_netaddr_fromsockaddr(&netaddr, &entry->sockaddr);
isc_netaddr_format(&netaddr, addrbuf, sizeof(addrbuf));
if (debug) {
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
fprintf(f, ";\t%p: refcnt %" PRIuFAST32 "\n", entry,
isc_refcount_current(&entry->references));
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
3593. [func] Update EDNS processing to better track remote server capabilities. [RT #30655]
2013-06-12 11:31:30 +10:00
fprintf(f,
Simplify the EDNS buffer size logic for DNS Flag Day 2020 The DNS Flag Day 2020 aims to remove the IP fragmentation problem from the UDP DNS communication. In this commit, we implement the required changes and simplify the logic for picking the EDNS Buffer Size. 1. The defaults for `edns-udp-size`, `max-udp-size` and `nocookie-udp-size` have been changed to `1232` (the value picked by DNS Flag Day 2020). 2. The probing heuristics that would try 512->4096->1432->1232 buffer sizes has been removed and the resolver will always use just the `edns-udp-size` value. 3. Instead of just disabling the PMTUD mechanism on the UDP sockets, we now set IP_DONTFRAG (IPV6_DONTFRAG) flag. That means that the UDP packets won't get ever fragmented. If the ICMP packets are lost the UDP will just timeout and eventually be retried over TCP.
2020-09-23 14:47:26 +02:00
";\t%s [srtt %u] [flags %08x] [edns %u/%u] "
3593. [func] Update EDNS processing to better track remote server capabilities. [RT #30655]
2013-06-12 11:31:30 +10:00
"[plain %u/%u]",
Replace some ADB entry locking with atomics to reduce ADB contention Use atomics on couple of ADB entry members (.srtt, .flags, .expires, and .lastage) to remove ADB entry locking from couple of hot spots. The most prominent place is copy_namehook_lists() that gets called under ADB name lock and if the namehook list is long it acquires-releases quite a few ADB entry locks. Changing those ADB entry members to atomics allowed us to new_adbaddrinfo() not require locked ADB entry and since adbentry_overquota() already used atomics and handling lame information was dropped in the previous commit, we could not make the copy_namehook_lists() lockless. The other hotspot is dns_adb_adjustsrtt() and dns_adb_agesrtt() that can now use atomics because .srtt is already atomic_uint. And the last place that could now use atomics is dns_adb_changeflags().
2023-09-21 11:59:01 +02:00
addrbuf, atomic_load(&entry->srtt), atomic_load(&entry->flags),
entry->edns, entry->ednsto, entry->plain, entry->plainto);
3593. [func] Update EDNS processing to better track remote server capabilities. [RT #30655]
2013-06-12 11:31:30 +10:00
if (entry->udpsize != 0U) {
fprintf(f, " [udpsize %u]", entry->udpsize);
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
4152. [func] Implement DNS COOKIE option. This replaces the experimental SIT option of BIND 9.10. The following named.conf directives are avaliable: send-cookie, cookie-secret, cookie-algorithm and nocookie-udp-size. The following dig options are available: +[no]cookie[=value] and +[no]badcookie. [RT #39928]
2015-07-06 09:44:24 +10:00
if (entry->cookie != NULL) {
3744. [experimental] SIT: send and process Source Identity Tokens (which are similar to DNS Cookies by Donald Eastlake) and are designed to help clients detect off path spoofed responses and for servers to detect legitimate clients. SIT use a experimental EDNS option code (65001). SIT can be enabled via --enable-developer or --enable-sit. It is on by default in Windows. RRL processing as been updated to know about SIT with legitimate clients not being rate limited. [RT #35389]
2014-02-19 12:53:42 +11:00
unsigned int i;
4152. [func] Implement DNS COOKIE option. This replaces the experimental SIT option of BIND 9.10. The following named.conf directives are avaliable: send-cookie, cookie-secret, cookie-algorithm and nocookie-udp-size. The following dig options are available: +[no]cookie[=value] and +[no]badcookie. [RT #39928]
2015-07-06 09:44:24 +10:00
fprintf(f, " [cookie=");
for (i = 0; i < entry->cookielen; i++) {
fprintf(f, "%02x", entry->cookie[i]);
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
3744. [experimental] SIT: send and process Source Identity Tokens (which are similar to DNS Cookies by Donald Eastlake) and are designed to help clients detect off path spoofed responses and for servers to detect legitimate clients. SIT use a experimental EDNS option code (65001). SIT can be enabled via --enable-developer or --enable-sit. It is on by default in Windows. RRL processing as been updated to know about SIT with legitimate clients not being rate limited. [RT #35389]
2014-02-19 12:53:42 +11:00
fprintf(f, "]");
}
Replace some ADB entry locking with atomics to reduce ADB contention Use atomics on couple of ADB entry members (.srtt, .flags, .expires, and .lastage) to remove ADB entry locking from couple of hot spots. The most prominent place is copy_namehook_lists() that gets called under ADB name lock and if the namehook list is long it acquires-releases quite a few ADB entry locks. Changing those ADB entry members to atomics allowed us to new_adbaddrinfo() not require locked ADB entry and since adbentry_overquota() already used atomics and handling lame information was dropped in the previous commit, we could not make the copy_namehook_lists() lockless. The other hotspot is dns_adb_adjustsrtt() and dns_adb_agesrtt() that can now use atomics because .srtt is already atomic_uint. And the last place that could now use atomics is dns_adb_changeflags().
2023-09-21 11:59:01 +02:00
if (atomic_load(&entry->expires) != 0) {
fprintf(f, " [ttl %d]",
(int)(atomic_load(&entry->expires) - now));
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
if (adb != NULL && adb->quota != 0 && adb->atr_freq != 0) {
Workaround MSVC warning C4477 Due to a way the stdatomic.h shim is implemented on Windows, the MSVC always things that the outside type is the largest - atomic_(u)int_fast64_t. This can lead to false positives as this one: lib\dns\adb.c(3678): warning C4477: 'fprintf' : format string '%u' requires an argument of type 'unsigned int', but variadic argument 2 has type 'unsigned __int64' We workaround the issue by loading the value in a scoped local variable with correct type first.
2020-04-15 12:36:12 +02:00
uint_fast32_t quota = atomic_load_relaxed(&entry->quota);
lib/dns/adb.c: Use atomics for adb quota values and reference counting
2019-01-22 10:47:18 +01:00
fprintf(f, " [atr %0.2f] [quota %" PRIuFAST32 "]", entry->atr,
Workaround MSVC warning C4477 Due to a way the stdatomic.h shim is implemented on Windows, the MSVC always things that the outside type is the largest - atomic_(u)int_fast64_t. This can lead to false positives as this one: lib\dns\adb.c(3678): warning C4477: 'fprintf' : format string '%u' requires an argument of type 'unsigned int', but variadic argument 2 has type 'unsigned __int64' We workaround the issue by loading the value in a scoped local variable with correct type first.
2020-04-15 12:36:12 +02:00
quota);
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
fprintf(f, "\n");
dump the ADB in a more user-friendly format (unless a debug flag is set)
2000-12-18 20:03:33 +00:00
}
clean up unused API the dns_adb_dumpfind() function was only used inside adb.c and can be static. dns_view_dumpdbtostream() was not used anywhere.
2022-05-25 12:02:18 -07:00
static void
dumpfind(dns_adbfind_t *find, FILE *f) {
update copyright notice
2008-02-07 23:46:54 +00:00
char tmp[512];
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
const char *tmpp = NULL;
dns_adbaddrinfo_t *ai = NULL;
isc_sockaddr_t *sa = NULL;
update copyright notice
2008-02-07 23:46:54 +00:00
/*
* Not used currently, in the API Just In Case we
* want to dump out the name and/or entries too.
*/
LOCK(&find->lock);
fprintf(f, ";Find %p\n", find);
fprintf(f, ";\tqpending %08x partial %08x options %08x flags %08x\n",
find->query_pending, find->partial_result, find->options,
find->flags);
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
fprintf(f, ";\tname %p\n", find->adbname);
update copyright notice
2008-02-07 23:46:54 +00:00
ai = ISC_LIST_HEAD(find->list);
if (ai != NULL) {
fprintf(f, "\tAddresses:\n");
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
while (ai != NULL) {
sa = &ai->sockaddr;
switch (sa->type.sa.sa_family) {
case AF_INET:
tmpp = inet_ntop(AF_INET, &sa->type.sin.sin_addr, tmp,
sizeof(tmp));
break;
case AF_INET6:
tmpp = inet_ntop(AF_INET6, &sa->type.sin6.sin6_addr,
tmp, sizeof(tmp));
break;
default:
tmpp = "UnkFamily";
}
if (tmpp == NULL) {
tmpp = "BadAddress";
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
fprintf(f,
"\t\tentry %p, flags %08x"
" srtt %u addr %s\n",
ai->entry, ai->flags, ai->srtt, tmpp);
ai = ISC_LIST_NEXT(ai, publink);
}
UNLOCK(&find->lock);
snapshot; everything works other than fetches and database lookups.
1999-10-16 00:38:21 +00:00
}
Many changes. o added adb_test.c to test the functions already written. o added code to delete names and all associated addresses. o added a database dumper which will currently print the adb in a human (or at least developer) readable format. o found and fixed many twisted little locking issues.
1999-10-02 01:59:09 +00:00
static void
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
print_namehook_list(FILE *f, const char *legend, dns_adb_t *adb,
dns_adbnamehooklist_t *list, bool debug,
isc_stdtime_t now) {
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
dns_adbnamehook_t *nh = NULL;
update copyright notice
2008-02-07 23:46:54 +00:00
for (nh = ISC_LIST_HEAD(*list); nh != NULL;
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
nh = ISC_LIST_NEXT(nh, name_link))
Update sources to Clang 15 formatting
2022-11-02 19:33:14 +01:00
{
update copyright notice
2008-02-07 23:46:54 +00:00
if (debug) {
fprintf(f, ";\tHook(%s) %p\n", legend, nh);
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
LOCK(&nh->entry->lock);
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
dump_entry(f, adb, nh->entry, debug, now);
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
UNLOCK(&nh->entry->lock);
update copyright notice
2008-02-07 23:46:54 +00:00
}
Many changes. o added adb_test.c to test the functions already written. o added code to delete names and all associated addresses. o added a database dumper which will currently print the adb in a human (or at least developer) readable format. o found and fixed many twisted little locking issues.
1999-10-02 01:59:09 +00:00
}
snapshot; everything works other than fetches and database lookups.
1999-10-16 00:38:21 +00:00
Remove use of the inline keyword used as suggestion to compiler Historically, the inline keyword was a strong suggestion to the compiler that it should inline the function marked inline. As compilers became better at optimising, this functionality has receded, and using inline as a suggestion to inline a function is obsolete. The compiler will happily ignore it and inline something else entirely if it finds that's a better optimisation. Therefore, remove all the occurences of the inline keyword with static functions inside single compilation unit and leave the decision whether to inline a function or not entirely on the compiler NOTE: We keep the usage the inline keyword when the purpose is to change the linkage behaviour.
2021-10-11 13:43:12 +02:00
static void
Megacommit of many files. Mostly, several functions that take pointers as arguments, almost always char * pointers, had those pointers qualified with "const". Those that returned pointers to previously const-qualified arguments had their return values qualified as const. Some structure members were qualified as const to retain that attribute from the variables from which they were assigned. The macro DE_CONST is used to deal with a handful of very special places where something is qualified as const but really needs to have its const qualifier removed. rdata.c now defines macros for the prototypes of the basic rdata functions, and all of the lib/dns/rdata/**/*.c files now use them. Some minor integer-compatibility issues. (IE, ~0x03 is a signed int, so assigning it to an unsigned int should use a cast. The type of an enum member is int, so there are some conversion issues there, too.) A pointers-to-function should not be cast to a pointer-to-object. Variables should not be named for C reserved identifiers. One or two set-but-not-used variables removed. Minor other ISC style cleanups.
2000-06-01 18:26:56 +00:00
print_fetch(FILE *f, dns_adbfetch_t *ft, const char *type) {
2544. [cleanup] Removed unused structure members in adb.c. [RT #19225]
2009-02-03 22:33:13 +00:00
fprintf(f, "\t\tFetch(%s): %p -> { fetch %p }\n", type, ft, ft->fetch);
kill name->query_pending, since that bitmask is really represented more accurately by other means. Reverse the sense of fetch checking from NO_FETCH_A() and NO_FETCH_AAAA(), etc to NAME_FETCH_A(), NAME_FETCH_AAAA(), and add address family checks like NO_FETCH_V4() and _V6(). Fix a database dump bug where the A fetches were printed twice rather than printing the AAAA fetches. Add printing of A6 fetches.
1999-11-02 18:36:54 +00:00
}
fetch change: A -> pointer, AAAA -> pointer, A6 -> list
1999-10-29 21:48:51 +00:00
static void
Megacommit of dozens of files. Cleanup of redundant/useless header file inclusion. ISC style lint, primarily for function declarations and standalone comments -- ie, those that appear on a line without any code, which should be written as follows: /* * This is a comment. */
2000-05-08 14:38:29 +00:00
print_fetch_list(FILE *f, dns_adbname_t *n) {
update copyright notice
2008-02-07 23:46:54 +00:00
if (NAME_FETCH_A(n)) {
print_fetch(f, n->fetch_a, "A");
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
if (NAME_FETCH_AAAA(n)) {
print_fetch(f, n->fetch_aaaa, "AAAA");
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
snapshot; nearly all fetch infrastructure is in place.
1999-10-27 19:36:58 +00:00
}
snapshot; everything works other than fetches and database lookups.
1999-10-16 00:38:21 +00:00
static void
Megacommit of dozens of files. Cleanup of redundant/useless header file inclusion. ISC style lint, primarily for function declarations and standalone comments -- ie, those that appear on a line without any code, which should be written as follows: /* * This is a comment. */
2000-05-08 14:38:29 +00:00
print_find_list(FILE *f, dns_adbname_t *name) {
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
dns_adbfind_t *find = NULL;
snapshot; everything works other than fetches and database lookups.
1999-10-16 00:38:21 +00:00
update copyright notice
2008-02-07 23:46:54 +00:00
find = ISC_LIST_HEAD(name->finds);
while (find != NULL) {
clean up unused API the dns_adb_dumpfind() function was only used inside adb.c and can be static. dns_view_dumpdbtostream() was not used anywhere.
2022-05-25 12:02:18 -07:00
dumpfind(find, f);
update copyright notice
2008-02-07 23:46:54 +00:00
find = ISC_LIST_NEXT(find, plink);
}
snapshot; everything works other than fetches and database lookups.
1999-10-16 00:38:21 +00:00
}
Add code to check for a lame zone, and to expire lame entries as needed.
1999-10-19 20:55:04 +00:00
add "rndc fetchlimit" to show fetchlimited servers this command runs dns_adb_dumpquota() to display all servers in the ADB that are being actively fetchlimited by the fetches-per-server controls (i.e, servers with a nonzero average timeout ratio or with the quota having been reduced from the default value). the "fetchlimit" system test has been updated to use the new command to check quota values instead of "rndc dumpdb".
2022-05-26 02:33:52 -07:00
static isc_result_t
putstr(isc_buffer_t **b, const char *str) {
isc_result_t result;
Change the isc_buffer_reserve() to take just buffer pointer The isc_buffer_reserve() would be passed a reference to the buffer pointer, which was unnecessary as the pointer would never be changed in the current implementation. Remove the extra dereference.
2022-12-15 11:54:51 +01:00
result = isc_buffer_reserve(*b, strlen(str));
add "rndc fetchlimit" to show fetchlimited servers this command runs dns_adb_dumpquota() to display all servers in the ADB that are being actively fetchlimited by the fetches-per-server controls (i.e, servers with a nonzero average timeout ratio or with the quota having been reduced from the default value). the "fetchlimit" system test has been updated to use the new command to check quota values instead of "rndc dumpdb".
2022-05-26 02:33:52 -07:00
if (result != ISC_R_SUCCESS) {
return (result);
}
isc_buffer_putstr(*b, str);
return (ISC_R_SUCCESS);
}
isc_result_t
dns_adb_dumpquota(dns_adb_t *adb, isc_buffer_t **buf) {
REQUIRE(DNS_ADB_VALID(adb));
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
isc_hashmap_iter_t *it = NULL;
isc_result_t result;
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
RWLOCK(&adb->entries_lock, isc_rwlocktype_read);
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
isc_hashmap_iter_create(adb->entries, &it);
for (result = isc_hashmap_iter_first(it); result == ISC_R_SUCCESS;
result = isc_hashmap_iter_next(it))
add "rndc fetchlimit" to show fetchlimited servers this command runs dns_adb_dumpquota() to display all servers in the ADB that are being actively fetchlimited by the fetches-per-server controls (i.e, servers with a nonzero average timeout ratio or with the quota having been reduced from the default value). the "fetchlimit" system test has been updated to use the new command to check quota values instead of "rndc dumpdb".
2022-05-26 02:33:52 -07:00
{
dns_adbentry_t *entry = NULL;
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
isc_hashmap_iter_current(it, (void **)&entry);
add "rndc fetchlimit" to show fetchlimited servers this command runs dns_adb_dumpquota() to display all servers in the ADB that are being actively fetchlimited by the fetches-per-server controls (i.e, servers with a nonzero average timeout ratio or with the quota having been reduced from the default value). the "fetchlimit" system test has been updated to use the new command to check quota values instead of "rndc dumpdb".
2022-05-26 02:33:52 -07:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
LOCK(&entry->lock);
char addrbuf[ISC_NETADDR_FORMATSIZE];
char text[ISC_NETADDR_FORMATSIZE + BUFSIZ];
isc_netaddr_t netaddr;
add "rndc fetchlimit" to show fetchlimited servers this command runs dns_adb_dumpquota() to display all servers in the ADB that are being actively fetchlimited by the fetches-per-server controls (i.e, servers with a nonzero average timeout ratio or with the quota having been reduced from the default value). the "fetchlimit" system test has been updated to use the new command to check quota values instead of "rndc dumpdb".
2022-05-26 02:33:52 -07:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
if (entry->atr == 0.0 && entry->quota == adb->quota) {
goto unlock;
}
add "rndc fetchlimit" to show fetchlimited servers this command runs dns_adb_dumpquota() to display all servers in the ADB that are being actively fetchlimited by the fetches-per-server controls (i.e, servers with a nonzero average timeout ratio or with the quota having been reduced from the default value). the "fetchlimit" system test has been updated to use the new command to check quota values instead of "rndc dumpdb".
2022-05-26 02:33:52 -07:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
isc_netaddr_fromsockaddr(&netaddr, &entry->sockaddr);
isc_netaddr_format(&netaddr, addrbuf, sizeof(addrbuf));
add "rndc fetchlimit" to show fetchlimited servers this command runs dns_adb_dumpquota() to display all servers in the ADB that are being actively fetchlimited by the fetches-per-server controls (i.e, servers with a nonzero average timeout ratio or with the quota having been reduced from the default value). the "fetchlimit" system test has been updated to use the new command to check quota values instead of "rndc dumpdb".
2022-05-26 02:33:52 -07:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
snprintf(text, sizeof(text),
"\n- quota %s (%" PRIuFAST32 "/%d) atr %0.2f", addrbuf,
atomic_load_relaxed(&entry->quota), adb->quota,
entry->atr);
putstr(buf, text);
unlock:
UNLOCK(&entry->lock);
add "rndc fetchlimit" to show fetchlimited servers this command runs dns_adb_dumpquota() to display all servers in the ADB that are being actively fetchlimited by the fetches-per-server controls (i.e, servers with a nonzero average timeout ratio or with the quota having been reduced from the default value). the "fetchlimit" system test has been updated to use the new command to check quota values instead of "rndc dumpdb".
2022-05-26 02:33:52 -07:00
}
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
isc_hashmap_iter_destroy(&it);
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
RWUNLOCK(&adb->entries_lock, isc_rwlocktype_read);
add "rndc fetchlimit" to show fetchlimited servers this command runs dns_adb_dumpquota() to display all servers in the ADB that are being actively fetchlimited by the fetches-per-server controls (i.e, servers with a nonzero average timeout ratio or with the quota having been reduced from the default value). the "fetchlimit" system test has been updated to use the new command to check quota values instead of "rndc dumpdb".
2022-05-26 02:33:52 -07:00
Purge stale ADB names globaly, not per bucket Before the refactoring, there was only few buckets with many names in them, so cleaning up stale ADB names per-bucket made sense. After the refactoring, each bucket directly maps to ADB name, so purging has been effectively disabled. Create a global LRU list for ADB names (and ADB entries) and purge the stale ADB names globally.
2022-08-01 09:07:38 +02:00
return (ISC_R_SUCCESS);
add "rndc fetchlimit" to show fetchlimited servers this command runs dns_adb_dumpquota() to display all servers in the ADB that are being actively fetchlimited by the fetches-per-server controls (i.e, servers with a nonzero average timeout ratio or with the quota having been reduced from the default value). the "fetchlimit" system test has been updated to use the new command to check quota values instead of "rndc dumpdb".
2022-05-26 02:33:52 -07:00
}
Add code to check for a lame zone, and to expire lame entries as needed.
1999-10-19 20:55:04 +00:00
static isc_result_t
Megacommit of dozens of files. Cleanup of redundant/useless header file inclusion. ISC style lint, primarily for function declarations and standalone comments -- ie, those that appear on a line without any code, which should be written as follows: /* * This is a comment. */
2000-05-08 14:38:29 +00:00
dbfind_name(dns_adbname_t *adbname, isc_stdtime_t now, dns_rdatatype_t rdtype) {
update copyright notice
2008-02-07 23:46:54 +00:00
isc_result_t result;
dns_rdataset_t rdataset;
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
dns_adb_t *adb = NULL;
update copyright notice
2008-02-07 23:46:54 +00:00
dns_fixedname_t foundname;
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
dns_name_t *fname = NULL;
REQUIRE(DNS_ADBNAME_VALID(adbname));
update copyright notice
2008-02-07 23:46:54 +00:00
adb = adbname->adb;
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
REQUIRE(DNS_ADB_VALID(adb));
REQUIRE(rdtype == dns_rdatatype_a || rdtype == dns_rdatatype_aaaa);
update copyright notice
2008-02-07 23:46:54 +00:00
Use dns_fixedname_initname() where possible Replace dns_fixedname_init() calls followed by dns_fixedname_name() calls with calls to dns_fixedname_initname() where it is possible without affecting current behavior and/or performance. This patch was mostly prepared using Coccinelle and the following semantic patch: @@ expression fixedname, name; @@ - dns_fixedname_init(&fixedname); ... - name = dns_fixedname_name(&fixedname); + name = dns_fixedname_initname(&fixedname); The resulting set of changes was then manually reviewed to exclude false positives and apply minor tweaks. It is likely that more occurrences of this pattern can be refactored in an identical way. This commit only takes care of the low-hanging fruit.
2018-03-28 14:38:09 +02:00
fname = dns_fixedname_initname(&foundname);
update copyright notice
2008-02-07 23:46:54 +00:00
dns_rdataset_init(&rdataset);
if (rdtype == dns_rdatatype_a) {
adbname->fetch_err = FIND_ERR_UNEXPECTED;
} else {
adbname->fetch6_err = FIND_ERR_UNEXPECTED;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
2947. [func] Add new zone type "static-stub". It's like a stub zone, but the nameserver names and/or their IP addresses are statically configured. [RT #21474] (for 9.8.0)
2010-12-16 09:51:30 +00:00
/*
* We need to specify whether to search static-stub zones (if
* configured) depending on whether this is a "start at zone" lookup,
* i.e., whether it's a "bailiwick" glue. If it's bailiwick (in which
clean up numbering of FETCHOPT and ADDRINFO flags in the past there was overlap between the fields used as resolver fetch options and ADB addrinfo flags. this has mostly been eliminated; now we can clean up the rest of it and remove some confusing comments.
2023-07-03 12:43:17 -07:00
* case DNS_ADBFIND_STARTATZONE is set) we need to stop the search at
* any matching static-stub zone without looking into the cache to honor
2947. [func] Add new zone type "static-stub". It's like a stub zone, but the nameserver names and/or their IP addresses are statically configured. [RT #21474] (for 9.8.0)
2010-12-16 09:51:30 +00:00
* the configuration on which server we should send queries to.
*/
Also look for additional records in dns_adb_find If a child zone is served by the same servers as a parent zone and a NS query is made for the zone name then the addresses of the nameservers are returned in the additional section are tagged as trust additional.
2023-08-14 10:26:18 +10:00
result =
dns_view_find(adb->view, adbname->name, rdtype, now,
DNS_DBFIND_GLUEOK | DNS_DBFIND_ADDITIONALOK, true,
((adbname->flags & DNS_ADBFIND_STARTATZONE) != 0),
NULL, NULL, fname, &rdataset, NULL);
update copyright notice
2008-02-07 23:46:54 +00:00
switch (result) {
case DNS_R_GLUE:
case DNS_R_HINT:
case ISC_R_SUCCESS:
/*
* Found in the database. Even if we can't copy out
* any information, return success, or else a fetch
* will be made, which will only make things worse.
*/
if (rdtype == dns_rdatatype_a) {
adbname->fetch_err = FIND_ERR_SUCCESS;
} else {
adbname->fetch6_err = FIND_ERR_SUCCESS;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
result = import_rdataset(adbname, &rdataset, now);
break;
case DNS_R_NXDOMAIN:
case DNS_R_NXRRSET:
/*
* We're authoritative and the data doesn't exist.
* Make up a negative cache entry so we don't ask again
* for a while.
*
* XXXRTH What time should we use? I'm putting in 30 seconds
* for now.
*/
if (rdtype == dns_rdatatype_a) {
adbname->expire_v4 = now + 30;
DP(NCACHE_LEVEL,
"adb name %p: Caching auth negative entry for A",
adbname);
if (result == DNS_R_NXDOMAIN) {
adbname->fetch_err = FIND_ERR_NXDOMAIN;
} else {
adbname->fetch_err = FIND_ERR_NXRRSET;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
} else {
DP(NCACHE_LEVEL,
"adb name %p: Caching auth negative entry for AAAA",
adbname);
adbname->expire_v6 = now + 30;
if (result == DNS_R_NXDOMAIN) {
adbname->fetch6_err = FIND_ERR_NXDOMAIN;
} else {
adbname->fetch6_err = FIND_ERR_NXRRSET;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
}
break;
case DNS_R_NCACHENXDOMAIN:
case DNS_R_NCACHENXRRSET:
/*
* We found a negative cache entry. Pull the TTL from it
* so we won't ask again for a while.
*/
rdataset.ttl = ttlclamp(rdataset.ttl);
if (rdtype == dns_rdatatype_a) {
adbname->expire_v4 = rdataset.ttl + now;
if (result == DNS_R_NCACHENXDOMAIN) {
adbname->fetch_err = FIND_ERR_NXDOMAIN;
} else {
adbname->fetch_err = FIND_ERR_NXRRSET;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
DP(NCACHE_LEVEL,
"adb name %p: Caching negative entry for A (ttl %u)",
adbname, rdataset.ttl);
} else {
DP(NCACHE_LEVEL,
"adb name %p: Caching negative entry for AAAA (ttl "
"%u)",
adbname, rdataset.ttl);
adbname->expire_v6 = rdataset.ttl + now;
if (result == DNS_R_NCACHENXDOMAIN) {
adbname->fetch6_err = FIND_ERR_NXDOMAIN;
} else {
adbname->fetch6_err = FIND_ERR_NXRRSET;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
}
break;
case DNS_R_CNAME:
case DNS_R_DNAME:
rdataset.ttl = ttlclamp(rdataset.ttl);
clean_target(adb, &adbname->target);
adbname->expire_target = INT_MAX;
Refactor isc_hashmap to accept custom match function Refactor isc_hashmap to allow custom matching functions. This allows us to have better tailored keys that don't require fixed uint8_t arrays, but can be composed of more fields from the stored data structure.
2023-08-28 10:08:59 +02:00
result = set_target(adb, adbname->name, fname, &rdataset,
update copyright notice
2008-02-07 23:46:54 +00:00
&adbname->target);
if (result == ISC_R_SUCCESS) {
result = DNS_R_ALIAS;
DP(NCACHE_LEVEL, "adb name %p: caching alias target",
adbname);
adbname->expire_target = rdataset.ttl + now;
}
if (rdtype == dns_rdatatype_a) {
adbname->fetch_err = FIND_ERR_SUCCESS;
} else {
adbname->fetch6_err = FIND_ERR_SUCCESS;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
break;
Make isc_result a static enum Remove the dynamic registration of result codes. Convert isc_result_t from unsigned + #defines into 32-bit enum type in grand unified <isc/result.h> header. Keep the existing values of the result codes even at the expense of the description and identifier tables being unnecessary large. Additionally, add couple of: switch (result) { [...] default: break; } statements where compiler now complains about missing enum values in the switch statement.
2021-10-04 17:14:53 +02:00
default:
break;
update copyright notice
2008-02-07 23:46:54 +00:00
}
if (dns_rdataset_isassociated(&rdataset)) {
dns_rdataset_disassociate(&rdataset);
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
return (result);
ipv6 AAAA queries work now.
1999-10-29 19:20:36 +00:00
}
snapshot; nearly all fetch infrastructure is in place.
1999-10-27 19:36:58 +00:00
static void
refactor dns_resolver to use loop callbacks callback events from dns_resolver_createfetch() are now posted using isc_async_run. other modules which called the resolver and maintained task/taskmgr objects for this purpose have been cleaned up.
2022-10-28 01:33:40 -07:00
fetch_callback(void *arg) {
dns_fetchresponse_t *resp = (dns_fetchresponse_t *)arg;
dns_adbname_t *name = resp->arg;
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
dns_adb_t *adb = NULL;
dns_adbfetch_t *fetch = NULL;
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
dns_adbstatus_t astat = DNS_ADB_NOMOREADDRESSES;
update copyright notice
2008-02-07 23:46:54 +00:00
isc_stdtime_t now;
isc_result_t result;
unsigned int address_type;
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
REQUIRE(DNS_ADBNAME_VALID(name));
remove ADB whenshutdown events weakly attaching and detaching the view when creating or destroying the ADB obviates the need for a whenshutdown callback event to do the detaching. remove the dns_adb_whenshutdown() mechanism, since it is no longer needed.
2022-05-11 13:55:01 -07:00
dns_adb_attach(name->adb, &adb);
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
REQUIRE(DNS_ADB_VALID(adb));
update copyright notice
2008-02-07 23:46:54 +00:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
LOCK(&name->lock);
update copyright notice
2008-02-07 23:46:54 +00:00
INSIST(NAME_FETCH_A(name) || NAME_FETCH_AAAA(name));
address_type = 0;
refactor dns_resolver to use loop callbacks callback events from dns_resolver_createfetch() are now posted using isc_async_run. other modules which called the resolver and maintained task/taskmgr objects for this purpose have been cleaned up.
2022-10-28 01:33:40 -07:00
if (NAME_FETCH_A(name) && (name->fetch_a->fetch == resp->fetch)) {
update copyright notice
2008-02-07 23:46:54 +00:00
address_type = DNS_ADBFIND_INET;
fetch = name->fetch_a;
name->fetch_a = NULL;
} else if (NAME_FETCH_AAAA(name) &&
refactor dns_resolver to use loop callbacks callback events from dns_resolver_createfetch() are now posted using isc_async_run. other modules which called the resolver and maintained task/taskmgr objects for this purpose have been cleaned up.
2022-10-28 01:33:40 -07:00
(name->fetch_aaaa->fetch == resp->fetch))
Update sources to Clang 15 formatting
2022-11-02 19:33:14 +01:00
{
update copyright notice
2008-02-07 23:46:54 +00:00
address_type = DNS_ADBFIND_INET6;
fetch = name->fetch_aaaa;
name->fetch_aaaa = NULL;
silence coverity false positive
2008-10-15 04:22:30 +00:00
} else {
fetch = NULL;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
silence coverity false positive
2008-10-15 04:22:30 +00:00
INSIST(address_type != 0 && fetch != NULL);
update copyright notice
2008-02-07 23:46:54 +00:00
/*
* Cleanup things we don't care about.
*/
refactor dns_resolver to use loop callbacks callback events from dns_resolver_createfetch() are now posted using isc_async_run. other modules which called the resolver and maintained task/taskmgr objects for this purpose have been cleaned up.
2022-10-28 01:33:40 -07:00
if (resp->node != NULL) {
dns_db_detachnode(resp->db, &resp->node);
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
refactor dns_resolver to use loop callbacks callback events from dns_resolver_createfetch() are now posted using isc_async_run. other modules which called the resolver and maintained task/taskmgr objects for this purpose have been cleaned up.
2022-10-28 01:33:40 -07:00
if (resp->db != NULL) {
dns_db_detach(&resp->db);
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
/*
* If this name is marked as dead, clean up, throwing away
* potentially good data.
*/
if (NAME_DEAD(name)) {
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
astat = DNS_ADB_CANCELED;
remove ADB whenshutdown events weakly attaching and detaching the view when creating or destroying the ADB obviates the need for a whenshutdown callback event to do the detaching. remove the dns_adb_whenshutdown() mechanism, since it is no longer needed.
2022-05-11 13:55:01 -07:00
goto out;
update copyright notice
2008-02-07 23:46:54 +00:00
}
Apply the semantic patch to remove isc_stdtime_get() This is a simple replacement using the semantic patch from the previous commit and as added bonus, one removal of previously undetected unused variable in named/server.c.
2023-03-30 21:13:41 +02:00
now = isc_stdtime_now();
update copyright notice
2008-02-07 23:46:54 +00:00
/*
* If we got a negative cache response, remember it.
*/
refactor dns_resolver to use loop callbacks callback events from dns_resolver_createfetch() are now posted using isc_async_run. other modules which called the resolver and maintained task/taskmgr objects for this purpose have been cleaned up.
2022-10-28 01:33:40 -07:00
if (NCACHE_RESULT(resp->result)) {
resp->rdataset->ttl = ttlclamp(resp->rdataset->ttl);
update copyright notice
2008-02-07 23:46:54 +00:00
if (address_type == DNS_ADBFIND_INET) {
DP(NCACHE_LEVEL,
"adb fetch name %p: "
"caching negative entry for A (ttl %u)",
refactor dns_resolver to use loop callbacks callback events from dns_resolver_createfetch() are now posted using isc_async_run. other modules which called the resolver and maintained task/taskmgr objects for this purpose have been cleaned up.
2022-10-28 01:33:40 -07:00
name, resp->rdataset->ttl);
update copyright notice
2008-02-07 23:46:54 +00:00
name->expire_v4 = ISC_MIN(name->expire_v4,
refactor dns_resolver to use loop callbacks callback events from dns_resolver_createfetch() are now posted using isc_async_run. other modules which called the resolver and maintained task/taskmgr objects for this purpose have been cleaned up.
2022-10-28 01:33:40 -07:00
resp->rdataset->ttl + now);
if (resp->result == DNS_R_NCACHENXDOMAIN) {
update copyright notice
2008-02-07 23:46:54 +00:00
name->fetch_err = FIND_ERR_NXDOMAIN;
} else {
name->fetch_err = FIND_ERR_NXRRSET;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
move ADB and resolver stats out of the view object for better object separation, ADB and resolver statistics counters are now stored in the ADB and resolver objects themsevles, rather than in the associated view.
2022-05-11 15:38:54 -07:00
inc_resstats(adb, dns_resstatscounter_gluefetchv4fail);
update copyright notice
2008-02-07 23:46:54 +00:00
} else {
DP(NCACHE_LEVEL,
"adb fetch name %p: "
"caching negative entry for AAAA (ttl %u)",
refactor dns_resolver to use loop callbacks callback events from dns_resolver_createfetch() are now posted using isc_async_run. other modules which called the resolver and maintained task/taskmgr objects for this purpose have been cleaned up.
2022-10-28 01:33:40 -07:00
name, resp->rdataset->ttl);
update copyright notice
2008-02-07 23:46:54 +00:00
name->expire_v6 = ISC_MIN(name->expire_v6,
refactor dns_resolver to use loop callbacks callback events from dns_resolver_createfetch() are now posted using isc_async_run. other modules which called the resolver and maintained task/taskmgr objects for this purpose have been cleaned up.
2022-10-28 01:33:40 -07:00
resp->rdataset->ttl + now);
if (resp->result == DNS_R_NCACHENXDOMAIN) {
update copyright notice
2008-02-07 23:46:54 +00:00
name->fetch6_err = FIND_ERR_NXDOMAIN;
} else {
name->fetch6_err = FIND_ERR_NXRRSET;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
move ADB and resolver stats out of the view object for better object separation, ADB and resolver statistics counters are now stored in the ADB and resolver objects themsevles, rather than in the associated view.
2022-05-11 15:38:54 -07:00
inc_resstats(adb, dns_resstatscounter_gluefetchv6fail);
update copyright notice
2008-02-07 23:46:54 +00:00
}
goto out;
}
/*
* Handle CNAME/DNAME.
*/
refactor dns_resolver to use loop callbacks callback events from dns_resolver_createfetch() are now posted using isc_async_run. other modules which called the resolver and maintained task/taskmgr objects for this purpose have been cleaned up.
2022-10-28 01:33:40 -07:00
if (resp->result == DNS_R_CNAME || resp->result == DNS_R_DNAME) {
resp->rdataset->ttl = ttlclamp(resp->rdataset->ttl);
update copyright notice
2008-02-07 23:46:54 +00:00
clean_target(adb, &name->target);
name->expire_target = INT_MAX;
Refactor isc_hashmap to accept custom match function Refactor isc_hashmap to allow custom matching functions. This allows us to have better tailored keys that don't require fixed uint8_t arrays, but can be composed of more fields from the stored data structure.
2023-08-28 10:08:59 +02:00
result = set_target(adb, name->name, resp->foundname,
refactor dns_resolver to use loop callbacks callback events from dns_resolver_createfetch() are now posted using isc_async_run. other modules which called the resolver and maintained task/taskmgr objects for this purpose have been cleaned up.
2022-10-28 01:33:40 -07:00
resp->rdataset, &name->target);
update copyright notice
2008-02-07 23:46:54 +00:00
if (result == ISC_R_SUCCESS) {
DP(NCACHE_LEVEL,
"adb fetch name %p: caching alias target", name);
refactor dns_resolver to use loop callbacks callback events from dns_resolver_createfetch() are now posted using isc_async_run. other modules which called the resolver and maintained task/taskmgr objects for this purpose have been cleaned up.
2022-10-28 01:33:40 -07:00
name->expire_target = resp->rdataset->ttl + now;
update copyright notice
2008-02-07 23:46:54 +00:00
}
goto check_result;
}
/*
* Did we get back junk? If so, and there are no more fetches
* sitting out there, tell all the finds about it.
*/
refactor dns_resolver to use loop callbacks callback events from dns_resolver_createfetch() are now posted using isc_async_run. other modules which called the resolver and maintained task/taskmgr objects for this purpose have been cleaned up.
2022-10-28 01:33:40 -07:00
if (resp->result != ISC_R_SUCCESS) {
update copyright notice
2008-02-07 23:46:54 +00:00
char buf[DNS_NAME_FORMATSIZE];
Refactor isc_hashmap to accept custom match function Refactor isc_hashmap to allow custom matching functions. This allows us to have better tailored keys that don't require fixed uint8_t arrays, but can be composed of more fields from the stored data structure.
2023-08-28 10:08:59 +02:00
dns_name_format(name->name, buf, sizeof(buf));
update copyright notice
2008-02-07 23:46:54 +00:00
DP(DEF_LEVEL, "adb: fetch of '%s' %s failed: %s", buf,
address_type == DNS_ADBFIND_INET ? "A" : "AAAA",
refactor dns_resolver to use loop callbacks callback events from dns_resolver_createfetch() are now posted using isc_async_run. other modules which called the resolver and maintained task/taskmgr objects for this purpose have been cleaned up.
2022-10-28 01:33:40 -07:00
isc_result_totext(resp->result));
[master] limit recursion depth and iterative queries 4006. [security] A flaw in delegation handling could be exploited to put named into an infinite loop. This has been addressed by placing limits on the number of levels of recursion named will allow (default 7), and the number of iterative queries that it will send (default 50) before terminating a recursive query (CVE-2014-8500). The recursion depth limit is configured via the "max-recursion-depth" option. [RT #35780]
2014-11-17 23:24:44 -08:00
/*
* Don't record a failure unless this is the initial
* fetch of a chain.
*/
if (fetch->depth > 1) {
goto out;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
/* XXXMLG Don't pound on bad servers. */
if (address_type == DNS_ADBFIND_INET) {
[master] adjust max-recursion-queries 4021. [bug] Adjust max-recursion-queries to accommodate the need for more queries when the cache is empty. [RT #38104]
2014-12-15 22:28:06 -08:00
name->expire_v4 = ISC_MIN(name->expire_v4, now + 10);
update copyright notice
2008-02-07 23:46:54 +00:00
name->fetch_err = FIND_ERR_FAILURE;
move ADB and resolver stats out of the view object for better object separation, ADB and resolver statistics counters are now stored in the ADB and resolver objects themsevles, rather than in the associated view.
2022-05-11 15:38:54 -07:00
inc_resstats(adb, dns_resstatscounter_gluefetchv4fail);
update copyright notice
2008-02-07 23:46:54 +00:00
} else {
[master] adjust max-recursion-queries 4021. [bug] Adjust max-recursion-queries to accommodate the need for more queries when the cache is empty. [RT #38104]
2014-12-15 22:28:06 -08:00
name->expire_v6 = ISC_MIN(name->expire_v6, now + 10);
update copyright notice
2008-02-07 23:46:54 +00:00
name->fetch6_err = FIND_ERR_FAILURE;
move ADB and resolver stats out of the view object for better object separation, ADB and resolver statistics counters are now stored in the ADB and resolver objects themsevles, rather than in the associated view.
2022-05-11 15:38:54 -07:00
inc_resstats(adb, dns_resstatscounter_gluefetchv6fail);
update copyright notice
2008-02-07 23:46:54 +00:00
}
goto out;
}
/*
* We got something potentially useful.
*/
result = import_rdataset(name, &fetch->rdataset, now);
CNAME/DNAME chaining (initial support; A6 still missing)
2000-01-21 02:50:27 +00:00
check_result:
update copyright notice
2008-02-07 23:46:54 +00:00
if (result == ISC_R_SUCCESS) {
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
astat = DNS_ADB_MOREADDRESSES;
update copyright notice
2008-02-07 23:46:54 +00:00
if (address_type == DNS_ADBFIND_INET) {
name->fetch_err = FIND_ERR_SUCCESS;
} else {
name->fetch6_err = FIND_ERR_SUCCESS;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
}
ipv6 AAAA queries work now.
1999-10-29 19:20:36 +00:00
out:
remove ADB whenshutdown events weakly attaching and detaching the view when creating or destroying the ADB obviates the need for a whenshutdown callback event to do the detaching. remove the dns_adb_whenshutdown() mechanism, since it is no longer needed.
2022-05-11 13:55:01 -07:00
dns_resolver_destroyfetch(&fetch->fetch);
update copyright notice
2008-02-07 23:46:54 +00:00
free_adbfetch(adb, &fetch);
refactor dns_resolver to use loop callbacks callback events from dns_resolver_createfetch() are now posted using isc_async_run. other modules which called the resolver and maintained task/taskmgr objects for this purpose have been cleaned up.
2022-10-28 01:33:40 -07:00
isc_mem_putanddetach(&resp->mctx, resp, sizeof(*resp));
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
if (astat != DNS_ADB_CANCELED) {
clean_finds_at_name(name, astat, address_type);
remove ADB whenshutdown events weakly attaching and detaching the view when creating or destroying the ADB obviates the need for a whenshutdown callback event to do the detaching. remove the dns_adb_whenshutdown() mechanism, since it is no longer needed.
2022-05-11 13:55:01 -07:00
}
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
UNLOCK(&name->lock);
dns_adbname_detach(&name);
remove ADB whenshutdown events weakly attaching and detaching the view when creating or destroying the ADB obviates the need for a whenshutdown callback event to do the detaching. remove the dns_adb_whenshutdown() mechanism, since it is no longer needed.
2022-05-11 13:55:01 -07:00
dns_adb_detach(&adb);
snapshot; nearly all fetch infrastructure is in place.
1999-10-27 19:36:58 +00:00
}
fetch contexts
1999-10-26 01:12:20 +00:00
static isc_result_t
Replace custom isc_boolean_t with C standard bool type
2018-04-17 08:29:14 -07:00
fetch_name(dns_adbname_t *adbname, bool start_at_zone, unsigned int depth,
checkpoint
2018-09-28 16:51:55 +10:00
isc_counter_t *qc, dns_rdatatype_t type) {
update copyright notice
2008-02-07 23:46:54 +00:00
isc_result_t result;
dns_adbfetch_t *fetch = NULL;
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
dns_adb_t *adb = NULL;
update copyright notice
2008-02-07 23:46:54 +00:00
dns_fixedname_t fixed;
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
dns_name_t *name = NULL;
update copyright notice
2008-02-07 23:46:54 +00:00
dns_rdataset_t rdataset;
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
dns_rdataset_t *nameservers = NULL;
update copyright notice
2008-02-07 23:46:54 +00:00
unsigned int options;
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
REQUIRE(DNS_ADBNAME_VALID(adbname));
update copyright notice
2008-02-07 23:46:54 +00:00
adb = adbname->adb;
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
REQUIRE(DNS_ADB_VALID(adb));
REQUIRE((type == dns_rdatatype_a && !NAME_FETCH_A(adbname)) ||
(type == dns_rdatatype_aaaa && !NAME_FETCH_AAAA(adbname)));
update copyright notice
2008-02-07 23:46:54 +00:00
adbname->fetch_err = FIND_ERR_NOTFOUND;
dns_rdataset_init(&rdataset);
options = DNS_FETCHOPT_NOVALIDATE;
Turn on QNAME minimisation when fetching nameserver addresses
2023-07-18 12:18:05 +10:00
update copyright notice
2008-02-07 23:46:54 +00:00
if (start_at_zone) {
DP(ENTER_LEVEL, "fetch_name: starting at zone for name %p",
adbname);
Use dns_fixedname_initname() where possible Replace dns_fixedname_init() calls followed by dns_fixedname_name() calls with calls to dns_fixedname_initname() where it is possible without affecting current behavior and/or performance. This patch was mostly prepared using Coccinelle and the following semantic patch: @@ expression fixedname, name; @@ - dns_fixedname_init(&fixedname); ... - name = dns_fixedname_name(&fixedname); + name = dns_fixedname_initname(&fixedname); The resulting set of changes was then manually reviewed to exclude false positives and apply minor tweaks. It is likely that more occurrences of this pattern can be refactored in an identical way. This commit only takes care of the low-hanging fruit.
2018-03-28 14:38:09 +02:00
name = dns_fixedname_initname(&fixed);
Refactor isc_hashmap to accept custom match function Refactor isc_hashmap to allow custom matching functions. This allows us to have better tailored keys that don't require fixed uint8_t arrays, but can be composed of more fields from the stored data structure.
2023-08-28 10:08:59 +02:00
result = dns_view_findzonecut(adb->view, adbname->name, name,
QNAME miminimization should create a separate fetch context for each fetch - this makes the cache more efficient and eliminates duplicates queries.
2018-06-13 09:29:40 +02:00
NULL, 0, 0, true, false,
libdns refactoring: get rid of multiple versions of dns_view_find, dns_view_findzonecut and dns_view_flushcache
2018-04-04 09:51:42 +02:00
&rdataset, NULL);
update copyright notice
2008-02-07 23:46:54 +00:00
if (result != ISC_R_SUCCESS && result != DNS_R_HINT) {
goto cleanup;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
nameservers = &rdataset;
options |= DNS_FETCHOPT_UNSHARED;
Turn on QNAME minimisation when fetching nameserver addresses
2023-07-18 12:18:05 +10:00
} else if (adb->view->qminimization) {
options |= DNS_FETCHOPT_QMINIMIZE | DNS_FETCHOPT_QMIN_SKIP_IP6A;
if (adb->view->qmin_strict) {
options |= DNS_FETCHOPT_QMIN_STRICT;
}
update copyright notice
2008-02-07 23:46:54 +00:00
}
rename test to qmin; add it to conf.sh.in and Makefile.in; fix copyrights
2018-05-18 13:07:31 -07:00
update copyright notice
2008-02-07 23:46:54 +00:00
fetch = new_adbfetch(adb);
[master] limit recursion depth and iterative queries 4006. [security] A flaw in delegation handling could be exploited to put named into an infinite loop. This has been addressed by placing limits on the number of levels of recursion named will allow (default 7), and the number of iterative queries that it will send (default 50) before terminating a recursive query (CVE-2014-8500). The recursion depth limit is configured via the "max-recursion-depth" option. [RT #35780]
2014-11-17 23:24:44 -08:00
fetch->depth = depth;
update copyright notice
2008-02-07 23:46:54 +00:00
QNAME miminimization should create a separate fetch context for each fetch - this makes the cache more efficient and eliminates duplicates queries.
2018-06-13 09:29:40 +02:00
/*
* We're not minimizing this query, as nothing user-related should
* be leaked here.
* However, if we'd ever want to change it we'd have to modify
* createfetch to find deepest cached name when we're providing
* domain and nameservers.
*/
libdns refactoring: get rid of 3 versions of dns_resolver_createfetch
2018-04-03 16:05:11 +02:00
result = dns_resolver_createfetch(
Refactor isc_hashmap to accept custom match function Refactor isc_hashmap to allow custom matching functions. This allows us to have better tailored keys that don't require fixed uint8_t arrays, but can be composed of more fields from the stored data structure.
2023-08-28 10:08:59 +02:00
adb->res, adbname->name, type, name, nameservers, NULL, NULL, 0,
use a thread-local variable to get the current running loop if we had a method to get the running loop, similar to how isc_tid() gets the current thread ID, we can simplify loop and loopmgr initialization. remove most uses of isc_loop_current() in favor of isc_loop(). in some places where that was the only reason to pass loopmgr, remove loopmgr from the function parameters.
2024-03-26 00:13:45 -07:00
options, depth, qc, isc_loop(), fetch_callback, adbname,
&fetch->rdataset, NULL, &fetch->fetch);
Fix looping issues
2018-07-14 20:11:03 +02:00
if (result != ISC_R_SUCCESS) {
DP(ENTER_LEVEL, "fetch_name: createfetch failed with %s",
isc_result_totext(result));
update copyright notice
2008-02-07 23:46:54 +00:00
goto cleanup;
Fix looping issues
2018-07-14 20:11:03 +02:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
dns_adbname_ref(adbname);
update copyright notice
2008-04-03 06:09:05 +00:00
if (type == dns_rdatatype_a) {
update copyright notice
2008-02-07 23:46:54 +00:00
adbname->fetch_a = fetch;
move ADB and resolver stats out of the view object for better object separation, ADB and resolver statistics counters are now stored in the ADB and resolver objects themsevles, rather than in the associated view.
2022-05-11 15:38:54 -07:00
inc_resstats(adb, dns_resstatscounter_gluefetchv4);
update copyright notice
2008-04-03 06:09:05 +00:00
} else {
update copyright notice
2008-02-07 23:46:54 +00:00
adbname->fetch_aaaa = fetch;
move ADB and resolver stats out of the view object for better object separation, ADB and resolver statistics counters are now stored in the ADB and resolver objects themsevles, rather than in the associated view.
2022-05-11 15:38:54 -07:00
inc_resstats(adb, dns_resstatscounter_gluefetchv6);
2355. [func] Extend the number statistics counters available. [RT #17590]
2008-04-03 05:55:52 +00:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
fetch = NULL; /* Keep us from cleaning this up below. */
snapshot; nearly all fetch infrastructure is in place.
1999-10-27 19:36:58 +00:00
cleanup:
update copyright notice
2008-02-07 23:46:54 +00:00
if (fetch != NULL) {
free_adbfetch(adb, &fetch);
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
if (dns_rdataset_isassociated(&rdataset)) {
dns_rdataset_disassociate(&rdataset);
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
snapshot; a6 seems to work?
1999-10-30 02:26:03 +00:00
update copyright notice
2008-02-07 23:46:54 +00:00
return (result);
Add the API to allow marking an address as lame for a zone
1999-10-19 21:10:29 +00:00
}
Add a way to adjust goodness.
1999-10-19 22:06:08 +00:00
Allow the srtt value to be adjusted, and update the goodness and srtt when they are modified
1999-10-19 22:21:27 +00:00
void
dns_adb_adjustsrtt(dns_adb_t *adb, dns_adbaddrinfo_t *addr, unsigned int rtt,
update copyright notice
2008-02-07 23:46:54 +00:00
unsigned int factor) {
REQUIRE(DNS_ADB_VALID(adb));
REQUIRE(DNS_ADBADDRINFO_VALID(addr));
REQUIRE(factor <= 10);
Allow the srtt value to be adjusted, and update the goodness and srtt when they are modified
1999-10-19 22:21:27 +00:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
dns_adbentry_t *entry = addr->entry;
Allow the srtt value to be adjusted, and update the goodness and srtt when they are modified
1999-10-19 22:21:27 +00:00
Replace some ADB entry locking with atomics to reduce ADB contention Use atomics on couple of ADB entry members (.srtt, .flags, .expires, and .lastage) to remove ADB entry locking from couple of hot spots. The most prominent place is copy_namehook_lists() that gets called under ADB name lock and if the namehook list is long it acquires-releases quite a few ADB entry locks. Changing those ADB entry members to atomics allowed us to new_adbaddrinfo() not require locked ADB entry and since adbentry_overquota() already used atomics and handling lame information was dropped in the previous commit, we could not make the copy_namehook_lists() lockless. The other hotspot is dns_adb_adjustsrtt() and dns_adb_agesrtt() that can now use atomics because .srtt is already atomic_uint. And the last place that could now use atomics is dns_adb_changeflags().
2023-09-21 11:59:01 +02:00
isc_stdtime_t now = 0;
if (atomic_load(&entry->expires) == 0 || factor == DNS_ADB_RTTADJAGE) {
Apply the semantic patch to remove isc_stdtime_get() This is a simple replacement using the semantic patch from the previous commit and as added bonus, one removal of previously undetected unused variable in named/server.c.
2023-03-30 21:13:41 +02:00
now = isc_stdtime_now();
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
3855. [bug] Limit smoothed round trip time aging to no more than once a second. [RT #32909]
2014-05-21 10:08:52 +10:00
Replace some ADB entry locking with atomics to reduce ADB contention Use atomics on couple of ADB entry members (.srtt, .flags, .expires, and .lastage) to remove ADB entry locking from couple of hot spots. The most prominent place is copy_namehook_lists() that gets called under ADB name lock and if the namehook list is long it acquires-releases quite a few ADB entry locks. Changing those ADB entry members to atomics allowed us to new_adbaddrinfo() not require locked ADB entry and since adbentry_overquota() already used atomics and handling lame information was dropped in the previous commit, we could not make the copy_namehook_lists() lockless. The other hotspot is dns_adb_adjustsrtt() and dns_adb_agesrtt() that can now use atomics because .srtt is already atomic_uint. And the last place that could now use atomics is dns_adb_changeflags().
2023-09-21 11:59:01 +02:00
adjustsrtt(addr, rtt, factor, now);
3855. [bug] Limit smoothed round trip time aging to no more than once a second. [RT #32909]
2014-05-21 10:08:52 +10:00
}
void
dns_adb_agesrtt(dns_adb_t *adb, dns_adbaddrinfo_t *addr, isc_stdtime_t now) {
REQUIRE(DNS_ADB_VALID(adb));
REQUIRE(DNS_ADBADDRINFO_VALID(addr));
adjustsrtt(addr, 0, DNS_ADB_RTTADJAGE, now);
}
static void
adjustsrtt(dns_adbaddrinfo_t *addr, unsigned int rtt, unsigned int factor,
isc_stdtime_t now) {
Replace some ADB entry locking with atomics to reduce ADB contention Use atomics on couple of ADB entry members (.srtt, .flags, .expires, and .lastage) to remove ADB entry locking from couple of hot spots. The most prominent place is copy_namehook_lists() that gets called under ADB name lock and if the namehook list is long it acquires-releases quite a few ADB entry locks. Changing those ADB entry members to atomics allowed us to new_adbaddrinfo() not require locked ADB entry and since adbentry_overquota() already used atomics and handling lame information was dropped in the previous commit, we could not make the copy_namehook_lists() lockless. The other hotspot is dns_adb_adjustsrtt() and dns_adb_agesrtt() that can now use atomics because .srtt is already atomic_uint. And the last place that could now use atomics is dns_adb_changeflags().
2023-09-21 11:59:01 +02:00
unsigned int new_srtt;
3593. [func] Update EDNS processing to better track remote server capabilities. [RT #30655]
2013-06-12 11:31:30 +10:00
if (factor == DNS_ADB_RTTADJAGE) {
Replace some ADB entry locking with atomics to reduce ADB contention Use atomics on couple of ADB entry members (.srtt, .flags, .expires, and .lastage) to remove ADB entry locking from couple of hot spots. The most prominent place is copy_namehook_lists() that gets called under ADB name lock and if the namehook list is long it acquires-releases quite a few ADB entry locks. Changing those ADB entry members to atomics allowed us to new_adbaddrinfo() not require locked ADB entry and since adbentry_overquota() already used atomics and handling lame information was dropped in the previous commit, we could not make the copy_namehook_lists() lockless. The other hotspot is dns_adb_adjustsrtt() and dns_adb_agesrtt() that can now use atomics because .srtt is already atomic_uint. And the last place that could now use atomics is dns_adb_changeflags().
2023-09-21 11:59:01 +02:00
if (atomic_load(&addr->entry->lastage) != now) {
Use mul and div instead of bitshifts to calculate srtt There was a microoptimization for smoothing srtt with bitshifts. Revert the code to use * 98 / 100, it doesn't really make that difference on modern CPUs, for comparison here: muldiv: imul eax, edi, 98 imul rax, rax, 1374389535 shr rax, 37 ret shift: mov eax, edi sal eax, 9 sub eax, edi shr eax, 9 ret
2023-10-12 09:20:42 +02:00
new_srtt = (uint64_t)atomic_load(&addr->entry->srtt) *
98 / 100;
Replace some ADB entry locking with atomics to reduce ADB contention Use atomics on couple of ADB entry members (.srtt, .flags, .expires, and .lastage) to remove ADB entry locking from couple of hot spots. The most prominent place is copy_namehook_lists() that gets called under ADB name lock and if the namehook list is long it acquires-releases quite a few ADB entry locks. Changing those ADB entry members to atomics allowed us to new_adbaddrinfo() not require locked ADB entry and since adbentry_overquota() already used atomics and handling lame information was dropped in the previous commit, we could not make the copy_namehook_lists() lockless. The other hotspot is dns_adb_adjustsrtt() and dns_adb_agesrtt() that can now use atomics because .srtt is already atomic_uint. And the last place that could now use atomics is dns_adb_changeflags().
2023-09-21 11:59:01 +02:00
atomic_store(&addr->entry->lastage, now);
Skip the no-op code in adjustsrtt() If factor == DNS_ADB_RTTADJAGE and addr->entry->lastage == now we would load value into new_srtt and then immediatelly store it back which triggers the synchronization between threads using .srtt values.
2023-10-12 09:17:40 +02:00
atomic_store(&addr->entry->srtt, new_srtt);
addr->srtt = new_srtt;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
3593. [func] Update EDNS processing to better track remote server capabilities. [RT #30655]
2013-06-12 11:31:30 +10:00
} else {
Replace some ADB entry locking with atomics to reduce ADB contention Use atomics on couple of ADB entry members (.srtt, .flags, .expires, and .lastage) to remove ADB entry locking from couple of hot spots. The most prominent place is copy_namehook_lists() that gets called under ADB name lock and if the namehook list is long it acquires-releases quite a few ADB entry locks. Changing those ADB entry members to atomics allowed us to new_adbaddrinfo() not require locked ADB entry and since adbentry_overquota() already used atomics and handling lame information was dropped in the previous commit, we could not make the copy_namehook_lists() lockless. The other hotspot is dns_adb_adjustsrtt() and dns_adb_agesrtt() that can now use atomics because .srtt is already atomic_uint. And the last place that could now use atomics is dns_adb_changeflags().
2023-09-21 11:59:01 +02:00
new_srtt = ((uint64_t)atomic_load(&addr->entry->srtt) / 10 *
factor) +
Replace custom isc_u?intNN_t types with C99 u?intNN_t types
2018-03-28 14:19:37 +02:00
((uint64_t)rtt / 10 * (10 - factor));
Skip the no-op code in adjustsrtt() If factor == DNS_ADB_RTTADJAGE and addr->entry->lastage == now we would load value into new_srtt and then immediatelly store it back which triggers the synchronization between threads using .srtt values.
2023-10-12 09:17:40 +02:00
atomic_store(&addr->entry->srtt, new_srtt);
addr->srtt = new_srtt;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
Minor API change. The 'factor' now ranges from 0..10. See adb.h for details.
1999-11-19 00:30:13 +00:00
Replace some ADB entry locking with atomics to reduce ADB contention Use atomics on couple of ADB entry members (.srtt, .flags, .expires, and .lastage) to remove ADB entry locking from couple of hot spots. The most prominent place is copy_namehook_lists() that gets called under ADB name lock and if the namehook list is long it acquires-releases quite a few ADB entry locks. Changing those ADB entry members to atomics allowed us to new_adbaddrinfo() not require locked ADB entry and since adbentry_overquota() already used atomics and handling lame information was dropped in the previous commit, we could not make the copy_namehook_lists() lockless. The other hotspot is dns_adb_adjustsrtt() and dns_adb_agesrtt() that can now use atomics because .srtt is already atomic_uint. And the last place that could now use atomics is dns_adb_changeflags().
2023-09-21 11:59:01 +02:00
(void)atomic_compare_exchange_strong(&addr->entry->expires,
&(isc_stdtime_t){ 0 },
now + ADB_ENTRY_WINDOW);
Allow the srtt value to be adjusted, and update the goodness and srtt when they are modified
1999-10-19 22:21:27 +00:00
}
add dns_adb_changeflags()
1999-11-16 06:53:29 +00:00
void
dns_adb_changeflags(dns_adb_t *adb, dns_adbaddrinfo_t *addr, unsigned int bits,
update copyright notice
2008-02-07 23:46:54 +00:00
unsigned int mask) {
REQUIRE(DNS_ADB_VALID(adb));
REQUIRE(DNS_ADBADDRINFO_VALID(addr));
add dns_adb_changeflags()
1999-11-16 06:53:29 +00:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
isc_stdtime_t now;
dns_adbentry_t *entry = addr->entry;
Replace some ADB entry locking with atomics to reduce ADB contention Use atomics on couple of ADB entry members (.srtt, .flags, .expires, and .lastage) to remove ADB entry locking from couple of hot spots. The most prominent place is copy_namehook_lists() that gets called under ADB name lock and if the namehook list is long it acquires-releases quite a few ADB entry locks. Changing those ADB entry members to atomics allowed us to new_adbaddrinfo() not require locked ADB entry and since adbentry_overquota() already used atomics and handling lame information was dropped in the previous commit, we could not make the copy_namehook_lists() lockless. The other hotspot is dns_adb_adjustsrtt() and dns_adb_agesrtt() that can now use atomics because .srtt is already atomic_uint. And the last place that could now use atomics is dns_adb_changeflags().
2023-09-21 11:59:01 +02:00
unsigned int flags = atomic_load(&entry->flags);
while (!atomic_compare_exchange_strong(&entry->flags, &flags,
(flags & ~mask) | (bits & mask)))
{
/* repeat */
}
add dns_adb_changeflags()
1999-11-16 06:53:29 +00:00
Replace some ADB entry locking with atomics to reduce ADB contention Use atomics on couple of ADB entry members (.srtt, .flags, .expires, and .lastage) to remove ADB entry locking from couple of hot spots. The most prominent place is copy_namehook_lists() that gets called under ADB name lock and if the namehook list is long it acquires-releases quite a few ADB entry locks. Changing those ADB entry members to atomics allowed us to new_adbaddrinfo() not require locked ADB entry and since adbentry_overquota() already used atomics and handling lame information was dropped in the previous commit, we could not make the copy_namehook_lists() lockless. The other hotspot is dns_adb_adjustsrtt() and dns_adb_agesrtt() that can now use atomics because .srtt is already atomic_uint. And the last place that could now use atomics is dns_adb_changeflags().
2023-09-21 11:59:01 +02:00
if (atomic_load(&entry->expires) == 0) {
Apply the semantic patch to remove isc_stdtime_get() This is a simple replacement using the semantic patch from the previous commit and as added bonus, one removal of previously undetected unused variable in named/server.c.
2023-03-30 21:13:41 +02:00
now = isc_stdtime_now();
Replace some ADB entry locking with atomics to reduce ADB contention Use atomics on couple of ADB entry members (.srtt, .flags, .expires, and .lastage) to remove ADB entry locking from couple of hot spots. The most prominent place is copy_namehook_lists() that gets called under ADB name lock and if the namehook list is long it acquires-releases quite a few ADB entry locks. Changing those ADB entry members to atomics allowed us to new_adbaddrinfo() not require locked ADB entry and since adbentry_overquota() already used atomics and handling lame information was dropped in the previous commit, we could not make the copy_namehook_lists() lockless. The other hotspot is dns_adb_adjustsrtt() and dns_adb_agesrtt() that can now use atomics because .srtt is already atomic_uint. And the last place that could now use atomics is dns_adb_changeflags().
2023-09-21 11:59:01 +02:00
atomic_store(&entry->expires, now + ADB_ENTRY_WINDOW);
3352. [bug] Ensure that learned server attributes timeout of the adb cache. [RT #29856]
2012-07-18 14:16:37 +10:00
}
update copyright notice
2008-02-07 23:46:54 +00:00
/*
* Note that we do not update the other bits in addr->flags with
* the most recent values from addr->entry->flags.
*/
addr->flags = (addr->flags & ~mask) | (bits & mask);
add dns_adb_changeflags()
1999-11-16 06:53:29 +00:00
}
add dns_adb_findaddrinfo() and dns_adb_freeaddrinfo()
2000-01-12 03:00:33 +00:00
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
/*
Describe the polynomial backoff curve used in the quota adjustment
2019-11-01 12:22:56 -05:00
* The polynomial backoff curve (10000 / ((10 + n) / 10)^(3/2)) <0..99> drops
* fairly aggressively at first, then slows down and tails off at around 2-3%.
*
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
* These will be used to make quota adjustments.
*/
static int quota_adj[] = {
10000, 8668, 7607, 6747, 6037, 5443, 4941, 4512, 4141, 3818, 3536,
3286, 3065, 2867, 2690, 2530, 2385, 2254, 2134, 2025, 1925, 1832,
1747, 1668, 1595, 1527, 1464, 1405, 1350, 1298, 1250, 1205, 1162,
1121, 1083, 1048, 1014, 981, 922, 894, 868, 843, 820, 797,
775, 755, 735, 716, 698, 680, 664, 648, 632, 618, 603,
590, 577, 564, 552, 540, 529, 518, 507, 497, 487, 477,
468, 459, 450, 442, 434, 426, 418, 411, 404, 397, 390,
383, 377, 370, 364, 358, 353, 347, 342, 336, 331, 326,
321, 316, 312, 307, 303, 298, 294, 290, 286, 282, 278
};
4719. [bug] Address PVS static analyzer warnings. [RT #45946]
2017-09-13 09:50:51 +10:00
#define QUOTA_ADJ_SIZE (sizeof(quota_adj) / sizeof(quota_adj[0]))
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
/*
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
* The adb entry associated with 'addr' must be locked.
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
*/
static void
maybe_adjust_quota(dns_adb_t *adb, dns_adbaddrinfo_t *addr, bool timeout) {
double tr;
UNUSED(adb);
if (adb->quota == 0 || adb->atr_freq == 0) {
return;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
if (timeout) {
addr->entry->timeouts++;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
if (addr->entry->completed++ <= adb->atr_freq) {
return;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
/*
* Calculate an exponential rolling average of the timeout ratio
*
* XXX: Integer arithmetic might be better than floating point
*/
tr = (double)addr->entry->timeouts / addr->entry->completed;
addr->entry->timeouts = addr->entry->completed = 0;
INSIST(addr->entry->atr >= 0.0);
INSIST(addr->entry->atr <= 1.0);
INSIST(adb->atr_discount >= 0.0);
INSIST(adb->atr_discount <= 1.0);
addr->entry->atr *= 1.0 - adb->atr_discount;
addr->entry->atr += tr * adb->atr_discount;
addr->entry->atr = ISC_CLAMP(addr->entry->atr, 0.0, 1.0);
if (addr->entry->atr < adb->atr_low && addr->entry->mode > 0) {
lib/dns/adb.c: Use atomics for adb quota values and reference counting
2019-01-22 10:47:18 +01:00
uint_fast32_t new_quota =
adb->quota * quota_adj[--addr->entry->mode] / 10000;
atomic_store_release(&addr->entry->quota,
Fix the fetches-per-server quota calculation Since commit bad5a523c2e, when the fetches-per-server quota was increased or decreased, instead of the value being set to the newly calculated quota, it was set to the *minimum* of the new quota or 1 - which effectively meant it was always set to 1. it should instead have been the maximum, to prevent the value from ever dropping to zero.
2022-05-04 17:27:56 -07:00
ISC_MAX(1, new_quota));
lib/dns/adb.c: Use atomics for adb quota values and reference counting
2019-01-22 10:47:18 +01:00
log_quota(addr->entry,
"atr %0.2f, quota increased to %" PRIuFAST32,
addr->entry->atr, new_quota);
4719. [bug] Address PVS static analyzer warnings. [RT #45946]
2017-09-13 09:50:51 +10:00
} else if (addr->entry->atr > adb->atr_high &&
addr->entry->mode < (QUOTA_ADJ_SIZE - 1))
{
lib/dns/adb.c: Use atomics for adb quota values and reference counting
2019-01-22 10:47:18 +01:00
uint_fast32_t new_quota =
adb->quota * quota_adj[++addr->entry->mode] / 10000;
atomic_store_release(&addr->entry->quota,
Fix the fetches-per-server quota calculation Since commit bad5a523c2e, when the fetches-per-server quota was increased or decreased, instead of the value being set to the newly calculated quota, it was set to the *minimum* of the new quota or 1 - which effectively meant it was always set to 1. it should instead have been the maximum, to prevent the value from ever dropping to zero.
2022-05-04 17:27:56 -07:00
ISC_MAX(1, new_quota));
lib/dns/adb.c: Use atomics for adb quota values and reference counting
2019-01-22 10:47:18 +01:00
log_quota(addr->entry,
"atr %0.2f, quota decreased to %" PRIuFAST32,
addr->entry->atr, new_quota);
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
}
}
3593. [func] Update EDNS processing to better track remote server capabilities. [RT #30655]
2013-06-12 11:31:30 +10:00
#define EDNSTOS 3U
void
dns_adb_plainresponse(dns_adb_t *adb, dns_adbaddrinfo_t *addr) {
REQUIRE(DNS_ADB_VALID(adb));
REQUIRE(DNS_ADBADDRINFO_VALID(addr));
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
dns_adbentry_t *entry = addr->entry;
LOCK(&entry->lock);
3744. [experimental] SIT: send and process Source Identity Tokens (which are similar to DNS Cookies by Donald Eastlake) and are designed to help clients detect off path spoofed responses and for servers to detect legitimate clients. SIT use a experimental EDNS option code (65001). SIT can be enabled via --enable-developer or --enable-sit. It is on by default in Windows. RRL processing as been updated to know about SIT with legitimate clients not being rate limited. [RT #35389]
2014-02-19 12:53:42 +11:00
Replace custom isc_boolean_t with C standard bool type
2018-04-17 08:29:14 -07:00
maybe_adjust_quota(adb, addr, false);
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
entry->plain++;
if (entry->plain == 0xff) {
entry->edns >>= 1;
entry->ednsto >>= 1;
entry->plain >>= 1;
entry->plainto >>= 1;
3593. [func] Update EDNS processing to better track remote server capabilities. [RT #30655]
2013-06-12 11:31:30 +10:00
}
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
UNLOCK(&entry->lock);
3593. [func] Update EDNS processing to better track remote server capabilities. [RT #30655]
2013-06-12 11:31:30 +10:00
}
void
dns_adb_timeout(dns_adb_t *adb, dns_adbaddrinfo_t *addr) {
REQUIRE(DNS_ADB_VALID(adb));
REQUIRE(DNS_ADBADDRINFO_VALID(addr));
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
dns_adbentry_t *entry = addr->entry;
LOCK(&entry->lock);
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
Replace custom isc_boolean_t with C standard bool type
2018-04-17 08:29:14 -07:00
maybe_adjust_quota(adb, addr, true);
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
3593. [func] Update EDNS processing to better track remote server capabilities. [RT #30655]
2013-06-12 11:31:30 +10:00
addr->entry->plainto++;
if (addr->entry->plainto == 0xff) {
addr->entry->edns >>= 1;
Simplify the EDNS buffer size logic for DNS Flag Day 2020 The DNS Flag Day 2020 aims to remove the IP fragmentation problem from the UDP DNS communication. In this commit, we implement the required changes and simplify the logic for picking the EDNS Buffer Size. 1. The defaults for `edns-udp-size`, `max-udp-size` and `nocookie-udp-size` have been changed to `1232` (the value picked by DNS Flag Day 2020). 2. The probing heuristics that would try 512->4096->1432->1232 buffer sizes has been removed and the resolver will always use just the `edns-udp-size` value. 3. Instead of just disabling the PMTUD mechanism on the UDP sockets, we now set IP_DONTFRAG (IPV6_DONTFRAG) flag. That means that the UDP packets won't get ever fragmented. If the ICMP packets are lost the UDP will just timeout and eventually be retried over TCP.
2020-09-23 14:47:26 +02:00
addr->entry->ednsto >>= 1;
3593. [func] Update EDNS processing to better track remote server capabilities. [RT #30655]
2013-06-12 11:31:30 +10:00
addr->entry->plain >>= 1;
addr->entry->plainto >>= 1;
}
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
UNLOCK(&entry->lock);
3593. [func] Update EDNS processing to better track remote server capabilities. [RT #30655]
2013-06-12 11:31:30 +10:00
}
void
Simplify the EDNS buffer size logic for DNS Flag Day 2020 The DNS Flag Day 2020 aims to remove the IP fragmentation problem from the UDP DNS communication. In this commit, we implement the required changes and simplify the logic for picking the EDNS Buffer Size. 1. The defaults for `edns-udp-size`, `max-udp-size` and `nocookie-udp-size` have been changed to `1232` (the value picked by DNS Flag Day 2020). 2. The probing heuristics that would try 512->4096->1432->1232 buffer sizes has been removed and the resolver will always use just the `edns-udp-size` value. 3. Instead of just disabling the PMTUD mechanism on the UDP sockets, we now set IP_DONTFRAG (IPV6_DONTFRAG) flag. That means that the UDP packets won't get ever fragmented. If the ICMP packets are lost the UDP will just timeout and eventually be retried over TCP.
2020-09-23 14:47:26 +02:00
dns_adb_ednsto(dns_adb_t *adb, dns_adbaddrinfo_t *addr) {
3593. [func] Update EDNS processing to better track remote server capabilities. [RT #30655]
2013-06-12 11:31:30 +10:00
REQUIRE(DNS_ADB_VALID(adb));
REQUIRE(DNS_ADBADDRINFO_VALID(addr));
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
dns_adbentry_t *entry = addr->entry;
LOCK(&entry->lock);
3593. [func] Update EDNS processing to better track remote server capabilities. [RT #30655]
2013-06-12 11:31:30 +10:00
Replace custom isc_boolean_t with C standard bool type
2018-04-17 08:29:14 -07:00
maybe_adjust_quota(adb, addr, true);
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
entry->ednsto++;
Simplify the EDNS buffer size logic for DNS Flag Day 2020 The DNS Flag Day 2020 aims to remove the IP fragmentation problem from the UDP DNS communication. In this commit, we implement the required changes and simplify the logic for picking the EDNS Buffer Size. 1. The defaults for `edns-udp-size`, `max-udp-size` and `nocookie-udp-size` have been changed to `1232` (the value picked by DNS Flag Day 2020). 2. The probing heuristics that would try 512->4096->1432->1232 buffer sizes has been removed and the resolver will always use just the `edns-udp-size` value. 3. Instead of just disabling the PMTUD mechanism on the UDP sockets, we now set IP_DONTFRAG (IPV6_DONTFRAG) flag. That means that the UDP packets won't get ever fragmented. If the ICMP packets are lost the UDP will just timeout and eventually be retried over TCP.
2020-09-23 14:47:26 +02:00
if (addr->entry->ednsto == 0xff) {
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
entry->edns >>= 1;
entry->ednsto >>= 1;
entry->plain >>= 1;
entry->plainto >>= 1;
3593. [func] Update EDNS processing to better track remote server capabilities. [RT #30655]
2013-06-12 11:31:30 +10:00
}
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
UNLOCK(&entry->lock);
3593. [func] Update EDNS processing to better track remote server capabilities. [RT #30655]
2013-06-12 11:31:30 +10:00
}
void
dns_adb_setudpsize(dns_adb_t *adb, dns_adbaddrinfo_t *addr, unsigned int size) {
REQUIRE(DNS_ADB_VALID(adb));
REQUIRE(DNS_ADBADDRINFO_VALID(addr));
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
dns_adbentry_t *entry = addr->entry;
LOCK(&entry->lock);
3593. [func] Update EDNS processing to better track remote server capabilities. [RT #30655]
2013-06-12 11:31:30 +10:00
if (size < 512U) {
size = 512U;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
3593. [func] Update EDNS processing to better track remote server capabilities. [RT #30655]
2013-06-12 11:31:30 +10:00
if (size > addr->entry->udpsize) {
addr->entry->udpsize = size;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
Replace custom isc_boolean_t with C standard bool type
2018-04-17 08:29:14 -07:00
maybe_adjust_quota(adb, addr, false);
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
entry->edns++;
if (entry->edns == 0xff) {
entry->edns >>= 1;
entry->ednsto >>= 1;
entry->plain >>= 1;
entry->plainto >>= 1;
3593. [func] Update EDNS processing to better track remote server capabilities. [RT #30655]
2013-06-12 11:31:30 +10:00
}
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
UNLOCK(&entry->lock);
3593. [func] Update EDNS processing to better track remote server capabilities. [RT #30655]
2013-06-12 11:31:30 +10:00
}
unsigned int
dns_adb_getudpsize(dns_adb_t *adb, dns_adbaddrinfo_t *addr) {
REQUIRE(DNS_ADB_VALID(adb));
REQUIRE(DNS_ADBADDRINFO_VALID(addr));
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
unsigned int size;
dns_adbentry_t *entry = addr->entry;
LOCK(&entry->lock);
size = entry->udpsize;
UNLOCK(&entry->lock);
3593. [func] Update EDNS processing to better track remote server capabilities. [RT #30655]
2013-06-12 11:31:30 +10:00
return (size);
}
3744. [experimental] SIT: send and process Source Identity Tokens (which are similar to DNS Cookies by Donald Eastlake) and are designed to help clients detect off path spoofed responses and for servers to detect legitimate clients. SIT use a experimental EDNS option code (65001). SIT can be enabled via --enable-developer or --enable-sit. It is on by default in Windows. RRL processing as been updated to know about SIT with legitimate clients not being rate limited. [RT #35389]
2014-02-19 12:53:42 +11:00
void
4152. [func] Implement DNS COOKIE option. This replaces the experimental SIT option of BIND 9.10. The following named.conf directives are avaliable: send-cookie, cookie-secret, cookie-algorithm and nocookie-udp-size. The following dig options are available: +[no]cookie[=value] and +[no]badcookie. [RT #39928]
2015-07-06 09:44:24 +10:00
dns_adb_setcookie(dns_adb_t *adb, dns_adbaddrinfo_t *addr,
const unsigned char *cookie, size_t len) {
3744. [experimental] SIT: send and process Source Identity Tokens (which are similar to DNS Cookies by Donald Eastlake) and are designed to help clients detect off path spoofed responses and for servers to detect legitimate clients. SIT use a experimental EDNS option code (65001). SIT can be enabled via --enable-developer or --enable-sit. It is on by default in Windows. RRL processing as been updated to know about SIT with legitimate clients not being rate limited. [RT #35389]
2014-02-19 12:53:42 +11:00
REQUIRE(DNS_ADB_VALID(adb));
REQUIRE(DNS_ADBADDRINFO_VALID(addr));
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
dns_adbentry_t *entry = addr->entry;
LOCK(&entry->lock);
3744. [experimental] SIT: send and process Source Identity Tokens (which are similar to DNS Cookies by Donald Eastlake) and are designed to help clients detect off path spoofed responses and for servers to detect legitimate clients. SIT use a experimental EDNS option code (65001). SIT can be enabled via --enable-developer or --enable-sit. It is on by default in Windows. RRL processing as been updated to know about SIT with legitimate clients not being rate limited. [RT #35389]
2014-02-19 12:53:42 +11:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
if (entry->cookie != NULL &&
(cookie == NULL || len != entry->cookielen))
4152. [func] Implement DNS COOKIE option. This replaces the experimental SIT option of BIND 9.10. The following named.conf directives are avaliable: send-cookie, cookie-secret, cookie-algorithm and nocookie-udp-size. The following dig options are available: +[no]cookie[=value] and +[no]badcookie. [RT #39928]
2015-07-06 09:44:24 +10:00
{
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
isc_mem_put(adb->mctx, entry->cookie, entry->cookielen);
entry->cookie = NULL;
entry->cookielen = 0;
3744. [experimental] SIT: send and process Source Identity Tokens (which are similar to DNS Cookies by Donald Eastlake) and are designed to help clients detect off path spoofed responses and for servers to detect legitimate clients. SIT use a experimental EDNS option code (65001). SIT can be enabled via --enable-developer or --enable-sit. It is on by default in Windows. RRL processing as been updated to know about SIT with legitimate clients not being rate limited. [RT #35389]
2014-02-19 12:53:42 +11:00
}
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
if (entry->cookie == NULL && cookie != NULL && len != 0U) {
entry->cookie = isc_mem_get(adb->mctx, len);
entry->cookielen = (uint16_t)len;
3744. [experimental] SIT: send and process Source Identity Tokens (which are similar to DNS Cookies by Donald Eastlake) and are designed to help clients detect off path spoofed responses and for servers to detect legitimate clients. SIT use a experimental EDNS option code (65001). SIT can be enabled via --enable-developer or --enable-sit. It is on by default in Windows. RRL processing as been updated to know about SIT with legitimate clients not being rate limited. [RT #35389]
2014-02-19 12:53:42 +11:00
}
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
if (entry->cookie != NULL) {
memmove(entry->cookie, cookie, len);
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
UNLOCK(&entry->lock);
3744. [experimental] SIT: send and process Source Identity Tokens (which are similar to DNS Cookies by Donald Eastlake) and are designed to help clients detect off path spoofed responses and for servers to detect legitimate clients. SIT use a experimental EDNS option code (65001). SIT can be enabled via --enable-developer or --enable-sit. It is on by default in Windows. RRL processing as been updated to know about SIT with legitimate clients not being rate limited. [RT #35389]
2014-02-19 12:53:42 +11:00
}
size_t
Clean up and refactor dns_adb_getcookie() The dns_adb_getcookie() doesn't use the 'adb' parameter, remove it. Refactor the dns_adb_getcookie() function to just return the size of the cookie when the caller passes 'NULL' as the 'cookie' argument.
2022-12-09 15:09:12 +00:00
dns_adb_getcookie(dns_adbaddrinfo_t *addr, unsigned char *cookie, size_t len) {
3744. [experimental] SIT: send and process Source Identity Tokens (which are similar to DNS Cookies by Donald Eastlake) and are designed to help clients detect off path spoofed responses and for servers to detect legitimate clients. SIT use a experimental EDNS option code (65001). SIT can be enabled via --enable-developer or --enable-sit. It is on by default in Windows. RRL processing as been updated to know about SIT with legitimate clients not being rate limited. [RT #35389]
2014-02-19 12:53:42 +11:00
REQUIRE(DNS_ADBADDRINFO_VALID(addr));
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
dns_adbentry_t *entry = addr->entry;
LOCK(&entry->lock);
Clean up and refactor dns_adb_getcookie() The dns_adb_getcookie() doesn't use the 'adb' parameter, remove it. Refactor the dns_adb_getcookie() function to just return the size of the cookie when the caller passes 'NULL' as the 'cookie' argument.
2022-12-09 15:09:12 +00:00
if (entry->cookie == NULL) {
3744. [experimental] SIT: send and process Source Identity Tokens (which are similar to DNS Cookies by Donald Eastlake) and are designed to help clients detect off path spoofed responses and for servers to detect legitimate clients. SIT use a experimental EDNS option code (65001). SIT can be enabled via --enable-developer or --enable-sit. It is on by default in Windows. RRL processing as been updated to know about SIT with legitimate clients not being rate limited. [RT #35389]
2014-02-19 12:53:42 +11:00
len = 0;
Clean up and refactor dns_adb_getcookie() The dns_adb_getcookie() doesn't use the 'adb' parameter, remove it. Refactor the dns_adb_getcookie() function to just return the size of the cookie when the caller passes 'NULL' as the 'cookie' argument.
2022-12-09 15:09:12 +00:00
goto unlock;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
Clean up and refactor dns_adb_getcookie() The dns_adb_getcookie() doesn't use the 'adb' parameter, remove it. Refactor the dns_adb_getcookie() function to just return the size of the cookie when the caller passes 'NULL' as the 'cookie' argument.
2022-12-09 15:09:12 +00:00
if (cookie != NULL) {
if (len < entry->cookielen) {
len = 0;
goto unlock;
}
memmove(cookie, entry->cookie, entry->cookielen);
}
len = entry->cookielen;
unlock:
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
UNLOCK(&entry->lock);
3744. [experimental] SIT: send and process Source Identity Tokens (which are similar to DNS Cookies by Donald Eastlake) and are designed to help clients detect off path spoofed responses and for servers to detect legitimate clients. SIT use a experimental EDNS option code (65001). SIT can be enabled via --enable-developer or --enable-sit. It is on by default in Windows. RRL processing as been updated to know about SIT with legitimate clients not being rate limited. [RT #35389]
2014-02-19 12:53:42 +11:00
return (len);
}
add dns_adb_findaddrinfo() and dns_adb_freeaddrinfo()
2000-01-12 03:00:33 +00:00
isc_result_t
4546. [func] Extend the use of const declarations. [RT #43379]
2016-12-30 15:45:08 +11:00
dns_adb_findaddrinfo(dns_adb_t *adb, const isc_sockaddr_t *sa,
update copyright notice
2008-02-07 23:46:54 +00:00
dns_adbaddrinfo_t **addrp, isc_stdtime_t now) {
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
REQUIRE(DNS_ADB_VALID(adb));
REQUIRE(addrp != NULL && *addrp == NULL);
UNUSED(now);
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
isc_result_t result = ISC_R_SUCCESS;
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
dns_adbentry_t *entry = NULL;
dns_adbaddrinfo_t *addr = NULL;
update copyright notice
2008-02-07 23:46:54 +00:00
in_port_t port;
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
if (atomic_load(&adb->exiting)) {
return (ISC_R_SHUTTINGDOWN);
update copyright notice
2008-02-07 23:46:54 +00:00
}
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
Lock the adbname and adbentry prior to unlocking hash locks There was a datarace that could expire a freshly created ADB names and entries between the return from get_attached_{name,entry} and locking it again. Lock the ADB name and ADB entry inside the hash table lock, so they won't get expired until the full initialization has been complete.
2022-12-13 13:48:55 +01:00
entry = get_attached_and_locked_entry(adb, now, sa);
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
INSIST(entry != NULL);
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
Replace some ADB entry locking with atomics to reduce ADB contention Use atomics on couple of ADB entry members (.srtt, .flags, .expires, and .lastage) to remove ADB entry locking from couple of hot spots. The most prominent place is copy_namehook_lists() that gets called under ADB name lock and if the namehook list is long it acquires-releases quite a few ADB entry locks. Changing those ADB entry members to atomics allowed us to new_adbaddrinfo() not require locked ADB entry and since adbentry_overquota() already used atomics and handling lame information was dropped in the previous commit, we could not make the copy_namehook_lists() lockless. The other hotspot is dns_adb_adjustsrtt() and dns_adb_agesrtt() that can now use atomics because .srtt is already atomic_uint. And the last place that could now use atomics is dns_adb_changeflags().
2023-09-21 11:59:01 +02:00
UNLOCK(&entry->lock);
update copyright notice
2008-02-07 23:46:54 +00:00
port = isc_sockaddr_getport(sa);
addr = new_adbaddrinfo(adb, entry, port);
Rewrite dns_resolver and dns_request to use netmgr timeouts - The `timeout_action` parameter to dns_dispatch_addresponse() been replaced with a netmgr callback that is called when a dispatch read times out. this callback may optionally reset the read timer and resume reading. - Added a function to convert isc_interval to milliseconds; this is used to translate fctx->interval into a value that can be passed to dns_dispatch_addresponse() as the timeout. - Note that netmgr timeouts are accurate to the millisecond, so code to check whether a timeout has been reached cannot rely on microsecond accuracy. - If serve-stale is configured, then a timeout received by the resolver may trigger it to return stale data, and then resume waiting for the read timeout. this is no longer based on a separate stale timer. - The code for canceling requests in request.c has been altered so that it can run asynchronously. - TCP timeout events apply to the dispatch, which may be shared by multiple queries. since in the event of a timeout we have no query ID to use to identify the resp we wanted, we now just send the timeout to the oldest query that was pending. - There was some additional refactoring in the resolver: combining fctx_join() and fctx_try_events() into one function to reduce code duplication, and using fixednames in fetchctx and fetchevent. - Incidental fix: new_adbaddrinfo() can't return NULL anymore, so the code can be simplified.
2021-04-22 18:58:01 -07:00
*addrp = addr;
add dns_adb_findaddrinfo() and dns_adb_freeaddrinfo()
2000-01-12 03:00:33 +00:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
dns_adbentry_detach(&entry);
update copyright notice
2008-02-07 23:46:54 +00:00
return (result);
add dns_adb_findaddrinfo() and dns_adb_freeaddrinfo()
2000-01-12 03:00:33 +00:00
}
void
Megacommit of dozens of files. Cleanup of redundant/useless header file inclusion. ISC style lint, primarily for function declarations and standalone comments -- ie, those that appear on a line without any code, which should be written as follows: /* * This is a comment. */
2000-05-08 14:38:29 +00:00
dns_adb_freeaddrinfo(dns_adb_t *adb, dns_adbaddrinfo_t **addrp) {
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
dns_adbaddrinfo_t *addr = NULL;
dns_adbentry_t *entry = NULL;
update copyright notice
2008-02-07 23:46:54 +00:00
isc_stdtime_t now;
add dns_adb_findaddrinfo() and dns_adb_freeaddrinfo()
2000-01-12 03:00:33 +00:00
update copyright notice
2008-02-07 23:46:54 +00:00
REQUIRE(DNS_ADB_VALID(adb));
REQUIRE(addrp != NULL);
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
update copyright notice
2008-02-07 23:46:54 +00:00
addr = *addrp;
Clear the pointer to destroyed object early using the semantic patch Also disable the semantic patch as the code needs tweaks here and there because some destroy functions might not destroy the object and return early if the object is still in use.
2020-02-08 04:37:54 -08:00
*addrp = NULL;
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
update copyright notice
2008-02-07 23:46:54 +00:00
REQUIRE(DNS_ADBADDRINFO_VALID(addr));
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
update copyright notice
2008-02-07 23:46:54 +00:00
entry = addr->entry;
add dns_adb_findaddrinfo() and dns_adb_freeaddrinfo()
2000-01-12 03:00:33 +00:00
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
REQUIRE(DNS_ADBENTRY_VALID(entry));
add dns_adb_findaddrinfo() and dns_adb_freeaddrinfo()
2000-01-12 03:00:33 +00:00
Replace some ADB entry locking with atomics to reduce ADB contention Use atomics on couple of ADB entry members (.srtt, .flags, .expires, and .lastage) to remove ADB entry locking from couple of hot spots. The most prominent place is copy_namehook_lists() that gets called under ADB name lock and if the namehook list is long it acquires-releases quite a few ADB entry locks. Changing those ADB entry members to atomics allowed us to new_adbaddrinfo() not require locked ADB entry and since adbentry_overquota() already used atomics and handling lame information was dropped in the previous commit, we could not make the copy_namehook_lists() lockless. The other hotspot is dns_adb_adjustsrtt() and dns_adb_agesrtt() that can now use atomics because .srtt is already atomic_uint. And the last place that could now use atomics is dns_adb_changeflags().
2023-09-21 11:59:01 +02:00
now = isc_stdtime_now();
(void)atomic_compare_exchange_strong(
&entry->expires, &(isc_stdtime_t){ 0 }, now + ADB_ENTRY_WINDOW);
add dns_adb_findaddrinfo() and dns_adb_freeaddrinfo()
2000-01-12 03:00:33 +00:00
update copyright notice
2008-02-07 23:46:54 +00:00
free_adbaddrinfo(adb, &addr);
add dns_adb_findaddrinfo() and dns_adb_freeaddrinfo()
2000-01-12 03:00:33 +00:00
}
808. [func] Add 'rndc flush' to flush the server's cache.
2001-04-11 20:37:50 +00:00
void
dns_adb_flush(dns_adb_t *adb) {
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
REQUIRE(DNS_ADB_VALID(adb));
808. [func] Add 'rndc flush' to flush the server's cache.
2001-04-11 20:37:50 +00:00
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
if (atomic_load(&adb->exiting)) {
return;
}
808. [func] Add 'rndc flush' to flush the server's cache.
2001-04-11 20:37:50 +00:00
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
cleanup_names(adb, INT_MAX);
Keep the unlink adb entries until expiration Currently, the ADB uses TTL of 0 for ADB names that the server is authoritative for and TTL of 10 seconds for HINT and GLUE ADB names. This requires the unlinked ADB entries to be kept around, because they would disappear too quickly. This especially affect the root zone as the trust level is "ultimate" for the root zone nameservers. This commit restores the ability to keep the unlinked ADB entries in the database for later reuse, restores printing the unlinked entries and adds some extra cleaning of the unlinked ADB entries on the tail of the LRU list (similar to what we are doing for the ADB names).
2022-11-23 09:56:19 +01:00
cleanup_entries(adb, INT_MAX);
808. [func] Add 'rndc flush' to flush the server's cache.
2001-04-11 20:37:50 +00:00
#ifdef DUMP_ADB_AFTER_CLEANING
Replace custom isc_boolean_t with C standard bool type
2018-04-17 08:29:14 -07:00
dump_adb(adb, stdout, true, INT_MAX);
808. [func] Add 'rndc flush' to flush the server's cache.
2001-04-11 20:37:50 +00:00
#endif /* ifdef DUMP_ADB_AFTER_CLEANING */
}
1073. [bug] The adb cache should also be space driven, max-adb-size. [RT #1915, #1938]
2001-10-25 04:57:46 +00:00
1137. [func] It is now possible to flush given names from the adb cache dns_adb_flushname().
2001-11-27 03:00:50 +00:00
void
4546. [func] Extend the use of const declarations. [RT #43379]
2016-12-30 15:45:08 +11:00
dns_adb_flushname(dns_adb_t *adb, const dns_name_t *name) {
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
dns_adbname_t *adbname = NULL;
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
isc_result_t result;
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
bool start_at_zone = false;
Refactor isc_hashmap to accept custom match function Refactor isc_hashmap to allow custom matching functions. This allows us to have better tailored keys that don't require fixed uint8_t arrays, but can be composed of more fields from the stored data structure.
2023-08-28 10:08:59 +02:00
dns_adbname_t key = { .name = UNCONST(name) };
update copyright notice
2008-02-07 23:46:54 +00:00
[master] "rndc flushtree -all <name>" 3606. [func] "rndc flushtree -all" flushes matching records in the ADB and bad cache as well as the DNS cache. (Without the "-all" option, flushtree will still only flush records from the DNS cache.) [RT #33970]
2013-06-26 14:59:32 -07:00
REQUIRE(DNS_ADB_VALID(adb));
REQUIRE(name != NULL);
update copyright notice
2008-02-07 23:46:54 +00:00
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
if (atomic_load(&adb->exiting)) {
return;
}
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
RWLOCK(&adb->names_lock, isc_rwlocktype_write);
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
again:
/*
clean up numbering of FETCHOPT and ADDRINFO flags in the past there was overlap between the fields used as resolver fetch options and ADB addrinfo flags. this has mostly been eliminated; now we can clean up the rest of it and remove some confusing comments.
2023-07-03 12:43:17 -07:00
* Delete both entries - without and with DNS_ADBFIND_STARTATZONE set.
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
*/
Refactor isc_hashmap to accept custom match function Refactor isc_hashmap to allow custom matching functions. This allows us to have better tailored keys that don't require fixed uint8_t arrays, but can be composed of more fields from the stored data structure.
2023-08-28 10:08:59 +02:00
key.flags = (start_at_zone) ? DNS_ADBFIND_STARTATZONE : 0;
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
Refactor isc_hashmap to accept custom match function Refactor isc_hashmap to allow custom matching functions. This allows us to have better tailored keys that don't require fixed uint8_t arrays, but can be composed of more fields from the stored data structure.
2023-08-28 10:08:59 +02:00
result = isc_hashmap_find(adb->names, hash_adbname(&key), match_adbname,
(void *)&key, (void **)&adbname);
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
if (result == ISC_R_SUCCESS) {
dns_adbname_ref(adbname);
LOCK(&adbname->lock);
Refactor isc_hashmap to accept custom match function Refactor isc_hashmap to allow custom matching functions. This allows us to have better tailored keys that don't require fixed uint8_t arrays, but can be composed of more fields from the stored data structure.
2023-08-28 10:08:59 +02:00
if (dns_name_equal(name, adbname->name)) {
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
expire_name(adbname, DNS_ADB_CANCELED);
update copyright notice
2008-02-07 23:46:54 +00:00
}
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
UNLOCK(&adbname->lock);
dns_adbname_detach(&adbname);
}
if (!start_at_zone) {
start_at_zone = true;
goto again;
update copyright notice
2008-02-07 23:46:54 +00:00
}
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
RWUNLOCK(&adb->names_lock, isc_rwlocktype_write);
1137. [func] It is now possible to flush given names from the adb cache dns_adb_flushname().
2001-11-27 03:00:50 +00:00
}
[master] "rndc flushtree -all <name>" 3606. [func] "rndc flushtree -all" flushes matching records in the ADB and bad cache as well as the DNS cache. (Without the "-all" option, flushtree will still only flush records from the DNS cache.) [RT #33970]
2013-06-26 14:59:32 -07:00
void
4546. [func] Extend the use of const declarations. [RT #43379]
2016-12-30 15:45:08 +11:00
dns_adb_flushnames(dns_adb_t *adb, const dns_name_t *name) {
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
dns_adbname_t *next = NULL;
[master] "rndc flushtree -all <name>" 3606. [func] "rndc flushtree -all" flushes matching records in the ADB and bad cache as well as the DNS cache. (Without the "-all" option, flushtree will still only flush records from the DNS cache.) [RT #33970]
2013-06-26 14:59:32 -07:00
REQUIRE(DNS_ADB_VALID(adb));
REQUIRE(name != NULL);
Refactor ADB reference counting, shutdown and locking The ADB previously used separate reference counters for internal and external references, plus additional counters for ABD find and namehook objects, and used all these counters to coordinate its shutdown process, which was a multi-stage affair involving a sequence of control events. It also used a complex interlocking set of static functions for referencing, deferencing, linking, unlinking, and cleaning up various internal objects; these functions returned boolean values to their callers to indicate what additional processing was needed. The changes in the previous two commits destabilized this fragile system in a way that was difficult to recover from, so in this commit we refactor all of it. The dns_adb and dns_adbentry objects now use conventional attach and detach functions for reference counting, and the shutdown process is much more straightforward. Instead of handling shutdown asynchronously, we can just destroy the ADB when references reach zero In addition, ADB locking has been simplified. Instead of a single `find_{name,entry}_and_lock()` function which searches for a name or entry's hash bucket, locks it, and then searches for the name or entry in the bucket, we now use one function to find the bucket (leaving it to the caller to do the locking) and another find the name or entry. Instead of locking the entire ADB when modifying hash tables, we now use read-write locks around the specific hash table. The only remaining need for adb->lock is when modifying the `whenshutdown` list. Comments throughout the module have been improved.
2022-03-21 12:48:52 -07:00
if (atomic_load(&adb->exiting)) {
return;
}
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
RWLOCK(&adb->names_lock, isc_rwlocktype_write);
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
for (dns_adbname_t *adbname = ISC_LIST_HEAD(adb->names_lru);
adbname != NULL; adbname = next)
Refactor how ADB names and entries are stored in the dns_adb Replace adb->{names,entries} and related arrays (indexed by hashed bucket) with a isc_ht hash tables storing the new struct adb{name,entry}bucket_t that wraps all the variables that were originally stored in arrays indexed by "bucket" number stored directly in the struct dns_adb. Previously, the task exclusive mode has been used to grow the internal arrays used to store the named and entries objects. The isc_ht hash tables are now protected by the isc_rwlock instead and thus the usage of the task exclusive mode has been removed from the dns_adb. Co-authored-by: Ondřej Surý <ondrej@isc.org>
2022-03-16 21:58:55 +01:00
{
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
next = ISC_LIST_NEXT(adbname, link);
dns_adbname_ref(adbname);
LOCK(&adbname->lock);
Refactor isc_hashmap to accept custom match function Refactor isc_hashmap to allow custom matching functions. This allows us to have better tailored keys that don't require fixed uint8_t arrays, but can be composed of more fields from the stored data structure.
2023-08-28 10:08:59 +02:00
if (dns_name_issubdomain(adbname->name, name)) {
refactor dns_adb to use loop callbacks The callbacks from dns_abd_createfind() are now posted using isc_async_run() instead of isc_task_send(). ADB event types have been replaced with a new dns_adbstatus_t type which is included as find->status. (The ADB still uses a task for dns_resolver_createfetch().)
2022-10-27 22:55:47 -07:00
expire_name(adbname, DNS_ADB_CANCELED);
[master] "rndc flushtree -all <name>" 3606. [func] "rndc flushtree -all" flushes matching records in the ADB and bad cache as well as the DNS cache. (Without the "-all" option, flushtree will still only flush records from the DNS cache.) [RT #33970]
2013-06-26 14:59:32 -07:00
}
Refactor the dns_adb unit The dns_adb unit has been refactored to be much simpler. Following changes have been made: 1. Simplify the ADB to always allow GLUE and hints There were only two places where dns_adb_createfind() was used - in the dns_resolver unit where hints and GLUE addresses were ok, and in the dns_zone where dns_adb_createfind() would be called without DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set. Simplify the logic by allowing hint and GLUE addresses when looking up the nameserver addresses to notify. The difference is negligible and would cause a difference in the notified addresses only when there's mismatch between the parent and child addresses and we haven't cached the child addresses yet. 2. Drop the namebuckets and entrybuckets Formerly, the namebuckets and entrybuckets were used to reduced the lock contention when accessing the double-linked lists stored in each bucket. In the previous refactoring, the custom hashtable for the buckets has been replaced with isc_ht/isc_hashmap, so only a single item (mostly, see below) would end up in each bucket. Removing the entrybuckets has been straightforward, the only matching was done on the isc_sockaddr_t member of the dns_adbentry. Removing the zonebuckets required GLUEOK and HINTOK bits to be removed because the find could match entries with-or-without the bits set, and creating a custom key that stores the DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a straightforward lookup into the hashtable without traversing a list that contains items with different flags. 3. Remove unassociated entries from ADB database Previously, the adbentries could live in the ADB database even after unlinking them from dns_adbnames. Such entries would show up as "Unassociated entries" in the ADB dump. The benefit of keeping such entries is little - the chance that we link such entry to a adbname is small, and it's simpler to evict unlinked entries from the ADB cache (and the hashtable) than create second LRU cleaning mechanism. Unlinked ADB entries are now directly deleted from the hash table (hashmap) upon destruction. 4. Cleanup expired entries from the hash table When buckets were still in place, the code would keep the buckets always allocated and never shrink the hash table (hashmap). With proper reference counting in place, we can delete the adbnames from the hash table and the LRU list. 5. Stop purging the names early when we hit the time limit Because the LRU list is now time ordered, we can stop purging the names when we find a first entry that doesn't fullfil our time-based eviction criteria because no further entry on the LRU list will meet the criteria. Future work: 1. Lock contention In this commit, the focus was on correctness of the data structure, but in the future, the lock contention in the ADB database needs to be addressed. Currently, we use simple mutex to lock the hash tables, because we almost always need to use a write lock for properly purging the hashtables. The ADB database needs to be sharded (similar to the effect that buckets had in the past). Each shard would contain own hashmap and own LRU list. 2. Time-based purging The ADB names and entries stay intact when there are no lookups. When we add separate shards, a timer needs to be added for time-based cleaning in case there's no traffic hashing to the inactive shard. 3. Revisit the 30 minutes limit The ADB cache is capped at 30 minutes. This needs to be revisited, and at least the limit should be configurable (in both directions).
2022-10-13 12:00:54 +02:00
UNLOCK(&adbname->lock);
dns_adbname_detach(&adbname);
[master] "rndc flushtree -all <name>" 3606. [func] "rndc flushtree -all" flushes matching records in the ADB and bad cache as well as the DNS cache. (Without the "-all" option, flushtree will still only flush records from the DNS cache.) [RT #33970]
2013-06-26 14:59:32 -07:00
}
Use C-RW-WP lock in the dns_adb unit Replace the isc_mutex in the dns_adb unit with isc_rwlock for better performance. Both ADB names and ADB entries hashtables and LRU are now using isc_rwlock.
2023-02-13 15:52:51 +01:00
RWUNLOCK(&adb->names_lock, isc_rwlocktype_write);
[master] "rndc flushtree -all <name>" 3606. [func] "rndc flushtree -all" flushes matching records in the ADB and bad cache as well as the DNS cache. (Without the "-all" option, flushtree will still only flush records from the DNS cache.) [RT #33970]
2013-06-26 14:59:32 -07:00
}
1073. [bug] The adb cache should also be space driven, max-adb-size. [RT #1915, #1938]
2001-10-25 04:57:46 +00:00
void
[master] accept >4g max-{,a}cache-size 3506. [func] When setting "max-cache-size" and "max-acache-size", the keyword "unlimited" is no longer defined as equal to 4 gigabytes (except on 32-bit platforms); it means literally unlimited. [RT #32358] 3505. [bug] When setting "max-cache-size" and "max-acache-size", larger values than 4 gigabytes could not be set explicitly, though larger sizes were available when setting cache size to 0. This has been corrected; the full range is now available. [RT #32358]
2013-02-28 09:29:12 -08:00
dns_adb_setadbsize(dns_adb_t *adb, size_t size) {
size_t hiwater, lowater;
1073. [bug] The adb cache should also be space driven, max-adb-size. [RT #1915, #1938]
2001-10-25 04:57:46 +00:00
minor pre-refactoring cleanups the use of "result" as a variable name for a boolean return value was confusing; all 'result' variables that are not isc_result_t have been renamed to 'ret'. The static function print_dns_name() was a duplicate of dns_name_print(), so it has been replaced with that. Changed INSIST to REQUIRE where appropriate, and added NULL initialization for pointer variables.
2022-03-24 16:06:48 -07:00
REQUIRE(DNS_ADB_VALID(adb));
1073. [bug] The adb cache should also be space driven, max-adb-size. [RT #1915, #1938]
2001-10-25 04:57:46 +00:00
silence compiler warnings
2013-03-05 23:41:22 +11:00
if (size != 0U && size < DNS_ADB_MINADBSIZE) {
update copyright notice
2008-02-07 23:46:54 +00:00
size = DNS_ADB_MINADBSIZE;
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
1073. [bug] The adb cache should also be space driven, max-adb-size. [RT #1915, #1938]
2001-10-25 04:57:46 +00:00
update copyright notice
2008-02-07 23:46:54 +00:00
hiwater = size - (size >> 3); /* Approximately 7/8ths. */
lowater = size - (size >> 2); /* Approximately 3/4ths. */
1073. [bug] The adb cache should also be space driven, max-adb-size. [RT #1915, #1938]
2001-10-25 04:57:46 +00:00
silence compiler warnings
2013-03-05 23:41:22 +11:00
if (size == 0U || hiwater == 0U || lowater == 0U) {
Use static storage for isc_mem water_t On the isc_mem water change the old water_t structure could be used after free. Instead of introducing reference counting on the hot-path we are going to introduce additional constraints on the isc_mem_setwater. Once it's set for the first time, the additional calls have to be made with the same water and water_arg arguments.
2021-07-22 06:14:32 +02:00
isc_mem_clearwater(adb->mctx);
update copyright notice
2008-02-07 23:46:54 +00:00
} else {
Refactor the handling of isc_mem overmem condition Previously, there were two methods of working with the overmem condition: 1. hi/lo water callback - when the overmem condition was reached for the first time, the water callback was called with HIWATER mark and .is_overmem boolean was set internally. Similarly, when the used memory went below the lo water mark, the water callback would be called with LOWATER mark and .is_overmem was reset. This check would be called **every** time memory was allocated or freed. 2. isc_mem_isovermem() - a simple getter for the internal .is_overmem flag This commit refactors removes the first method and move the hi/lo water checks to the isc_mem_isovermem() function, thus we now have only a single method of checking overmem condition and the check for hi/lo water is removed from the hot path for memory contexts that doesn't use overmem checks.
2023-11-29 09:01:56 +01:00
isc_mem_setwater(adb->mctx, hiwater, lowater);
Use clang-tidy to add curly braces around one-line statements The command used to reformat the files in this commit was: ./util/run-clang-tidy \ -clang-tidy-binary clang-tidy-11 -clang-apply-replacements-binary clang-apply-replacements-11 \ -checks=-*,readability-braces-around-statements \ -j 9 \ -fix \ -format \ -style=file \ -quiet clang-format -i --style=format $(git ls-files '*.c' '*.h') uncrustify -c .uncrustify.cfg --replace --no-backup $(git ls-files '*.c' '*.h') clang-format -i --style=format $(git ls-files '*.c' '*.h')
2020-02-13 21:48:23 +01:00
}
1073. [bug] The adb cache should also be space driven, max-adb-size. [RT #1915, #1938]
2001-10-25 04:57:46 +00:00
}
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
void
Replace custom isc_u?intNN_t types with C99 u?intNN_t types
2018-03-28 14:19:37 +02:00
dns_adb_setquota(dns_adb_t *adb, uint32_t quota, uint32_t freq, double low,
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
double high, double discount) {
REQUIRE(DNS_ADB_VALID(adb));
adb->quota = quota;
adb->atr_freq = freq;
adb->atr_low = low;
adb->atr_high = high;
adb->atr_discount = discount;
}
add "rndc fetchlimit" to show fetchlimited servers this command runs dns_adb_dumpquota() to display all servers in the ADB that are being actively fetchlimited by the fetches-per-server controls (i.e, servers with a nonzero average timeout ratio or with the quota having been reduced from the default value). the "fetchlimit" system test has been updated to use the new command to check quota values instead of "rndc dumpdb".
2022-05-26 02:33:52 -07:00
void
dns_adb_getquota(dns_adb_t *adb, uint32_t *quotap, uint32_t *freqp,
double *lowp, double *highp, double *discountp) {
REQUIRE(DNS_ADB_VALID(adb));
Apply the SET_IF_NOT_NULL() semantic patch spatch --sp-file cocci/set_if_not_null.spatch --use-gitgrep --dir "." --include-headers --in-place
2023-04-06 16:32:16 +01:00
SET_IF_NOT_NULL(quotap, adb->quota);
add "rndc fetchlimit" to show fetchlimited servers this command runs dns_adb_dumpquota() to display all servers in the ADB that are being actively fetchlimited by the fetches-per-server controls (i.e, servers with a nonzero average timeout ratio or with the quota having been reduced from the default value). the "fetchlimit" system test has been updated to use the new command to check quota values instead of "rndc dumpdb".
2022-05-26 02:33:52 -07:00
Apply the SET_IF_NOT_NULL() semantic patch spatch --sp-file cocci/set_if_not_null.spatch --use-gitgrep --dir "." --include-headers --in-place
2023-04-06 16:32:16 +01:00
SET_IF_NOT_NULL(freqp, adb->atr_freq);
add "rndc fetchlimit" to show fetchlimited servers this command runs dns_adb_dumpquota() to display all servers in the ADB that are being actively fetchlimited by the fetches-per-server controls (i.e, servers with a nonzero average timeout ratio or with the quota having been reduced from the default value). the "fetchlimit" system test has been updated to use the new command to check quota values instead of "rndc dumpdb".
2022-05-26 02:33:52 -07:00
Apply the SET_IF_NOT_NULL() semantic patch spatch --sp-file cocci/set_if_not_null.spatch --use-gitgrep --dir "." --include-headers --in-place
2023-04-06 16:32:16 +01:00
SET_IF_NOT_NULL(lowp, adb->atr_low);
add "rndc fetchlimit" to show fetchlimited servers this command runs dns_adb_dumpquota() to display all servers in the ADB that are being actively fetchlimited by the fetches-per-server controls (i.e, servers with a nonzero average timeout ratio or with the quota having been reduced from the default value). the "fetchlimit" system test has been updated to use the new command to check quota values instead of "rndc dumpdb".
2022-05-26 02:33:52 -07:00
Apply the SET_IF_NOT_NULL() semantic patch spatch --sp-file cocci/set_if_not_null.spatch --use-gitgrep --dir "." --include-headers --in-place
2023-04-06 16:32:16 +01:00
SET_IF_NOT_NULL(highp, adb->atr_high);
add "rndc fetchlimit" to show fetchlimited servers this command runs dns_adb_dumpquota() to display all servers in the ADB that are being actively fetchlimited by the fetches-per-server controls (i.e, servers with a nonzero average timeout ratio or with the quota having been reduced from the default value). the "fetchlimit" system test has been updated to use the new command to check quota values instead of "rndc dumpdb".
2022-05-26 02:33:52 -07:00
Apply the SET_IF_NOT_NULL() semantic patch spatch --sp-file cocci/set_if_not_null.spatch --use-gitgrep --dir "." --include-headers --in-place
2023-04-06 16:32:16 +01:00
SET_IF_NOT_NULL(discountp, adb->atr_discount);
add "rndc fetchlimit" to show fetchlimited servers this command runs dns_adb_dumpquota() to display all servers in the ADB that are being actively fetchlimited by the fetches-per-server controls (i.e, servers with a nonzero average timeout ratio or with the quota having been reduced from the default value). the "fetchlimit" system test has been updated to use the new command to check quota values instead of "rndc dumpdb".
2022-05-26 02:33:52 -07:00
}
Change dns_adbentry_overquota() to dns_adb_overquota() The dns_adbentry_overquota() was violating the layers accessing the adbentry struct members directly. Change it to dns_adb_overquota() to match the dns_adb API.
2023-03-30 21:19:17 +02:00
static bool
adbentry_overquota(dns_adbentry_t *entry) {
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
REQUIRE(DNS_ADBENTRY_VALID(entry));
lib/dns/adb.c: Use atomics for adb quota values and reference counting
2019-01-22 10:47:18 +01:00
Change dns_adbentry_overquota() to dns_adb_overquota() The dns_adbentry_overquota() was violating the layers accessing the adbentry struct members directly. Change it to dns_adb_overquota() to match the dns_adb API.
2023-03-30 21:19:17 +02:00
uint_fast32_t quota = atomic_load_relaxed(&entry->quota);
uint_fast32_t active = atomic_load_acquire(&entry->active);
lib/dns/adb.c: Use atomics for adb quota values and reference counting
2019-01-22 10:47:18 +01:00
return (quota != 0 && active >= quota);
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
}
Change dns_adbentry_overquota() to dns_adb_overquota() The dns_adbentry_overquota() was violating the layers accessing the adbentry struct members directly. Change it to dns_adb_overquota() to match the dns_adb API.
2023-03-30 21:19:17 +02:00
bool
dns_adb_overquota(dns_adb_t *adb ISC_ATTR_UNUSED, dns_adbaddrinfo_t *addrinfo) {
REQUIRE(DNS_ADBADDRINFO_VALID(addrinfo));
return (adbentry_overquota(addrinfo->entry));
}
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
void
dns_adb_beginudpfetch(dns_adb_t *adb, dns_adbaddrinfo_t *addr) {
REQUIRE should not have side effects it's a style violation to have REQUIRE or INSIST contain code that must run for the server to work. this was being done with some atomic_compare_exchange calls. these have been cleaned up. uses of atomic_compare_exchange in assertions have been replaced with a new macro atomic_compare_exchange_enforced, which uses RUNTIME_CHECK to ensure that the exchange was successful.
2021-10-21 02:28:48 -07:00
uint_fast32_t active;
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
REQUIRE(DNS_ADB_VALID(adb));
REQUIRE(DNS_ADBADDRINFO_VALID(addr));
REQUIRE should not have side effects it's a style violation to have REQUIRE or INSIST contain code that must run for the server to work. this was being done with some atomic_compare_exchange calls. these have been cleaned up. uses of atomic_compare_exchange in assertions have been replaced with a new macro atomic_compare_exchange_enforced, which uses RUNTIME_CHECK to ensure that the exchange was successful.
2021-10-21 02:28:48 -07:00
active = atomic_fetch_add_relaxed(&addr->entry->active, 1);
INSIST(active != UINT32_MAX);
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
}
void
dns_adb_endudpfetch(dns_adb_t *adb, dns_adbaddrinfo_t *addr) {
REQUIRE should not have side effects it's a style violation to have REQUIRE or INSIST contain code that must run for the server to work. this was being done with some atomic_compare_exchange calls. these have been cleaned up. uses of atomic_compare_exchange in assertions have been replaced with a new macro atomic_compare_exchange_enforced, which uses RUNTIME_CHECK to ensure that the exchange was successful.
2021-10-21 02:28:48 -07:00
uint_fast32_t active;
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
REQUIRE(DNS_ADB_VALID(adb));
REQUIRE(DNS_ADBADDRINFO_VALID(addr));
REQUIRE should not have side effects it's a style violation to have REQUIRE or INSIST contain code that must run for the server to work. this was being done with some atomic_compare_exchange calls. these have been cleaned up. uses of atomic_compare_exchange in assertions have been replaced with a new macro atomic_compare_exchange_enforced, which uses RUNTIME_CHECK to ensure that the exchange was successful.
2021-10-21 02:28:48 -07:00
active = atomic_fetch_sub_release(&addr->entry->active, 1);
INSIST(active != 0);
[master] DDoS mitigation features 3938. [func] Added quotas to be used in recursive resolvers that are under high query load for names in zones whose authoritative servers are nonresponsive or are experiencing a denial of service attack. - "fetches-per-server" limits the number of simultaneous queries that can be sent to any single authoritative server. The configured value is a starting point; it is automatically adjusted downward if the server is partially or completely non-responsive. The algorithm used to adjust the quota can be configured via the "fetch-quota-params" option. - "fetches-per-zone" limits the number of simultaneous queries that can be sent for names within a single domain. (Note: Unlike "fetches-per-server", this value is not self-tuning.) - New stats counters have been added to count queries spilled due to these quotas. See the ARM for details of these options. [RT #37125]
2015-07-08 22:53:39 -07:00
}
move ADB and resolver stats out of the view object for better object separation, ADB and resolver statistics counters are now stored in the ADB and resolver objects themsevles, rather than in the associated view.
2022-05-11 15:38:54 -07:00
isc_stats_t *
dns_adb_getstats(dns_adb_t *adb) {
REQUIRE(DNS_ADB_VALID(adb));
return (adb->stats);
}
Reference in New Issue Copy Permalink