OpenSSL 3.0 has completely revamed the way applications are supposed to
interact with libcrypto with the old way gaining a hefty perfomance
penalty.
Almost every cryptographic functionality will have a pre-3.0 and
post-3.0 counterpart split by the preprocessor after the migration is
complete.
Instead of having a macros mess everywhere, start the long term plan of
having a cryptographic functionality layer with OpenSSL >=3.0 and
OpenSSL <3.0 as two separate backends.
Start this process by splitting the initialization function.
The default stack sizes varies between operating systems and between
different system libc libraries from 128kB (Alpine Linux with MUSL) to
8M (Linux with glibc). Document the different values used to justify
the value of THREAD_MINSTACKSIZE (currently set to 1MB).
When running the isc_quota unit test with less than usual amount of
RAM (e.g. in a CI for architectures with 32 bits of address space),
the pthread_create() function fails with the "Resource temporarily
unavailable (11):" error code.
Add functions to get and set the thread stack size (if requested),
and use these to set the thread stack size to smaller value in the
isc_quota unit test.
There is a data race when QP is reclaiming chunks on the call_rcu
threads and it tries to log the number of reclaimed chunks while the
server is shuttingdown. Workaround this by adding rcu_barrier() before
shuttingdown the global log context.
This required couple of internal changes to the isc_mem_debugging.
The isc_mem_debugging is now internal to isc_mem unit and there are
three new functions:
1. isc_mem_setdebugging() can change the debugging setting for an
individual memory context. This is need for the memory contexts used
for OpenSSL, libxml and libuv accounting as recording and tracing
memory is broken there.
2. isc_mem_debugon() / isc_mem_debugoff() can be used to change default
memory debugging flags as well as debugging flags for isc_g_mctx.
Additionally, the memory debugging is inconsistent across the code-base.
For now, we are keeping the existing flags, but three new environment
variables have been added 'ISC_MEM_DEBUGRECORD', 'ISC_MEM_DEBUGTRACE'
and 'ISC_MEM_DEBUGUSAGE' to set the global debugging flags at any
program using the memory contexts.
Instead of having individual memory contexts scattered across different
files and called different names, add a single memory context called
isc_g_mctx that replaces named_g_mctx and various other global memory
contexts in various utilities and tests.
There is only a single network manager running on top of the loop
manager (except for tests). Refactor the network manager to be a
singleton (a single instance) and change the unit tests, so that the
shorter read timeouts apply only to a specific handle, not the whole
extra 'connect_nm' network manager instance.
All the applications built on top of the loop manager were required to
create just a single instance of the loop manager. Refactor the loop
manager to not expose this instance to the callers and keep the loop
manager object internal to the isc_loop compilation unit.
This significantly simplifies a number of data structures and calls to
the isc_loop API.
Instead of having hand crafted attach/detach/destroy functions, replace
them with the standard ISC_REFCOUNT macro. This also have advantage
that delayed netmgr detach (from dns_dispatch) now doesn't cause
assertion failure. This can happen with delayed (call_rcu) shutdown of
dns_adb.
The dns_adb cleaning is little bit muddled as it mixes the "TTL"
based cleaning (.expire_v4 and .expire_v6 for adbname, .expires for
adbentry) with overmem cleaning.
Rewrite the LRU based cleaning to use SIEVE algorithm and to be overmem
cleaning only with a requirement to always cleanup at least 2-times the
size of the newly added entry.
The .inuse member was causing a lot of contention between threads using
the same memory context. Scather the .inuse and .overmem members of
isc_mem_t structure to be an per-tid array of variables to reduce the
contention as the writes are now independent of each other.
The array uses one tad bit nasty trick, as ISC_TID_UNKNOWN is now -1,
the array has been sized to fit the unknown tid with [-1] index into the
array accomplished with `ctx->stat = &ctx->stat_s[1];`. It will not win
a beauty contest, but it works seamlessly by just passing `isc_tid()` as
an index into the array.
The caveat here is that gathering the real inuse value requires walking
the whole array for all registered tid values (isc_tid_count()). The
gather part happens only when statistics are being gathered or when
isc_mem_isovermem() is called. As the isc_mem_isovermem() call happens
only when new data is being added to cache or ADB, it doesn't happen on
the hottest (read-only) path and according to the measurements, it
doesn't slow down neither the cold cache nor the hot cache latency.
As POSIX guarantees only that the type ssize_t shall be capable of
storing values at least in the range [-1, {SSIZE_MAX}], it can't be used
to calculate the difference between two memory sizes. Change the logic
for junk filling to test whether the new size is larger than old size
and then use size_t as the result will be always positive.
The jemalloc arena in isc_mem was added to solve runaway memory problem
for outgoing TCP connections. In the end, this was a red herring and
the jemalloc arena code is now unused (via e28266bf). Remove the
support for jemalloc memory arenas as we can restore this at any time if
we need it ever again, but right now it's just a dead code.
In jemalloc_shim.h, we relied on including <isc/overflow.h> implicitly
instead of explicitly and same was happening inside isc/overflow.h - the
stdbool.h (for bool type) was being included implicitly instead of
explicitly.
Change the internal type used for isc_tid unit to isc_tid_t to hide the
specific integer type being used for the 'tid'. Internally, the signed
integer type is being used. This allows us to have negatively indexed
arrays that works both for threads with assigned tid and the threads
with unassigned tid. This should be used only in specific situations.
Meson is a modern build system that has seen a rise in adoption and some
version of it is available in almost every platform supported.
Compared to automake, meson has the following advantages:
* Meson provides a significant boost to the build and configuration time
by better exploiting parallelism.
* Meson is subjectively considered to be better in readability.
These merits alone justify experimenting with meson as a way of
improving development time and ergonomics. However, there are some
compromises to ensure the transition goes relatively smooth:
* The system tests currently rely on various files within the source
directory. Changing this requirement is a non-trivial task that can't
be currently justified. Currently the last compiled build directory
writes into the source tree which is in turn used by pytest.
* The minimum version supported has been fixed at 0.61. Increasing this
value will require choosing a baseline of distributions that can
package with meson. On the contrary, there will likely be an attempt
to decrease this value to ensure almost universal support for building
BIND 9 with meson.
commandline.c failed to compile on Solaris because NAME_MAX was
undefined. Include 'isc/dir.h' which defines NAME_MAX for platforms
that don't define it.
In file included from commandline.c:54:
./include/isc/commandline.h:31:38: error: 'NAME_MAX' undeclared here (not in a function)
31 | extern char isc_commandline_progname[NAME_MAX];
| ^~~~~~~~
Using C23 attributes for `counted_by` is broken with clang.
`__has_attribute` is used since `__has_c_attribute` only works with C23
attributes, (`gnu::counted_by`/`clang::counted_by`)
There were several methods how we used 'argv[0]'. Some programs had a
static value, some programs did use isc_file_progname(), some programs
stripped 'lt-' from the beginning of the name. And some used argv[0]
directly.
Unify the handling and all the variables into isc_commandline_progname
that gets populated by the new isc_commandline_init(argc, argv) call.
Instead of giving the memory pools names with an explicit call to
isc_mempool_setname(), add the name to isc_mempool_create() call to have
all the memory pools an unconditional name.
Instead of giving the memory context names with an explicit call to
isc_mem_setname(), add the name to isc_mem_create() call to have all the
memory contexts an unconditional name.
The memory context for isc_managers and dst_api units had no name and
that was causing trouble with the statistics channel output. Set the
name for the two memory context that were missing a proper name.
previously, ISC_LIST_FOREACH and ISC_LIST_FOREACH_SAFE were
two separate macros, with the _SAFE version allowing entries
to be unlinked during the loop. ISC_LIST_FOREACH is now also
safe, and the separate _SAFE macro has been removed.
similarly, the ISC_LIST_FOREACH_REV macro is now safe, and
ISC_LIST_FOREACH_REV_SAFE has also been removed.
qp-tries allocate their nodes (twigs) in chunks to reduce allocator
pressure and improve memory locality. The choice of chunk size presents
a tradeoff: larger chunks benefit qp-tries with many values (as seen
in large zones and resolvers) but waste memory in smaller use cases.
Previously, our fixed chunk size of 2^10 twigs meant that even an
empty qp-trie would consume 12KB of memory, while reducing this size
would negatively impact resolver performance.
This commit implements an adaptive chunking strategy that:
- Tracks the size of the most recently allocated chunk.
- Doubles the chunk size for each new allocation until reaching a
predefined maximum.
This approach effectively balances memory efficiency for small tries
while maintaining the performance benefits of larger chunk sizes for
bigger data structures.
This commit also splits the callback freeing qpmultis into two
phases, one that frees the underlying qptree, and one that reclaims
the qpmulti memory. In order to prevent races between the qpmulti
destructor and chunk garbage collection jobs, the second phase is
protected by reference counting.
If a call_rcu thread is running, there is a possible race condition
where the destructors run before all call_rcu callbacks have finished
running. This can happen, for example, if the call_rcu callback tries to
log something after the logging context has been torn down.
In !10394, we tried to counter this by explicitely creating a call_rcu
thread an shutting it down before running the destructors, but it is
possible for things to "slip" and end up on the default call_rcu thread.
As a quickfix, this commit moves an rcu_barrier() that was in the mem
context destructor earlier, so that it "protects" all libisc
destructors.
Previously all kinds of TCP timeouts had a single getter and setter
functions. Separate each timeout to its own getter/setter functions,
because in majority of cases only one is required at a time, and it's
not optimal expanding those functions every time a new timeout value
is implemented.
The new 'tcp-primaries-timeout' configuration option works the same way
as the existing 'tcp-initial-timeout' option, but applies only to the
TCP connections made to the primary servers, so that the timeout value
can be set separately for them. The default is 15 seconds.
Also, while accommodating zone.c's code to support the new option, make
a light refactoring with the way UDP timeouts are calculated by using
definitions instead of hardcoded values.
The custom allocation API for libxml2 is deprecated starting in macOS
Sequoia 15.4, iOS 18.4, tvOS 18.4, visionOS 2.4, and tvOS 18.4.
Disable the memory function override for libxml2 when
LIBXML_HAS_DEPRECATED_MEMORY_ALLOCATION_FUNCTIONS is defined as Apple
broke the system-wide libxml2 starting with macOS Sequoia 15.4.
When isc__thread_initialize() is called from a library constructor, it
could be called before we fork the main process. This happens with
named, and then we have the call_rcu_thread attached to the pre-fork
process and not the post-fork process, which means that the initial
process will never shutdown, because there's noone to tell it so.
Move the isc__thread_initialize() and isc__thread_shutdown() to the
isc_loop unit where we call it before creating the extra thread and
after joining all the extra threads respectively.
use the ISC_LIST_FOREACH pattern in places where lists had
been iterated using a different pattern from the typical
`for` loop: for example, `while (!ISC_LIST_EMPTY(...))` or
`while ((e = ISC_LIST_HEAD(...)) != NULL)`.
the pattern `for (x = ISC_LIST_HEAD(...); x != NULL; ISC_LIST_NEXT(...)`
has been changed to `ISC_LIST_FOREACH` throughout BIND, except in a few
cases where the change would be excessively complex.
in most cases this was a straightforward change. in some places,
however, the list element variable was referenced after the loop
ended, and the code was refactored to avoid this necessity.
also, because `ISC_LIST_FOREACH` uses typeof(list.head) to declare
the list elements, compilation failures can occur if the list object
has a `const` qualifier. some `const` qualifiers have been removed
from function parameters to avoid this problem, and where that was not
possible, `UNCONST` was used.
ISC_LIST_FOREACH and related macros now use 'typeof(list.head)' to
declare the list elements automatically; the caller no longer needs
to do so.
ISC_LIST_FOREACH_SAFE also now implicitly declares its own 'next'
pointer, so it only needs three parameters instead of four.
Add a function that checks if a 'hostname' is not a valid IPv4 or IPv6
address. Returns 'true' if the hostname is likely a domain name, and
'false' if it represents an IP address.
When allocating memory under -m trace|record, the __FILE__ pointer is
stored, so it can be printed out later in order to figure out in which
file an allocation leaked. (among others, like the line number).
However named crashes when called with -m record and using a plugin
leaking memory. The reason is that plugins are unloaded earlier than
when the leaked allocations are dumped (obviously, as it's done as late
as possible). In such circumstances, __FILE__ is dangling because the
dynamically loaded library (the plugin) is not in memory anymore.
Fix the crash by systematically copying the __FILE__ string
instead of copying the pointer. Of course, this make each allocation to
consume a bit more memory (and longer, as it needs to calculate the
length of __FILE__) but this occurs only under -m trace|record debugging
flags.
In term of unit test, because grepping in C is not fun, and because the
whole "syntax" of the dump output is tested in other tests, this simply
search for a substring in the whole buffer to make sure the expected
allocations are found.
This is the core implementation of the SIEVE algorithm described in the
following paper:
Zhang, Yazhuo, Juncheng Yang, Yao Yue, Ymir Vigfusson, and K V
Rashmi. “SIEVE Is Simpler than LRU: An Efficient Turn-Key Eviction
Algorithm for Web Caches,” n.d.. available online from
https://junchengyang.com/publication/nsdi24-SIEVE.pdf
This can be set at the option, view and server levels and causes
named to add an EDNS ZONEVERSION option to requests. Replies are
logged to the 'zoneversion' category.
This merge request resolves some performance regressions introduced
with the change from isc_symtab_t to isc_hashmap_t.
The key improvements are:
1. Using a faster hash function than both isc_hashmap_t and
isc_symtab_t. The previous implementation used SipHash, but the
hashflood resistance properties of SipHash are unneeded for config
parsing.
2. Shrinking the initial size of the isc_hashmap_t used inside
isc_symtab_t. Symtab is mainly used for config parsing, and the
when used that way it will have between 1 and ~50 keys, but the
previous implementation initialized a map with 128 slots.
By initializing a smaller map, we speed up mallocs and optimize for
the typical case of few config keys.
3. Slight optimization of the string matching in the hashmap, so that
the tail is handled in a single load + comparison, instead of byte
by byte.
Of the three improvements, this is the least important.
The isc_mem API is one of the most commonly used APIs that didn't
used ISC_REFCOUNT_DECL and ISC_REFCOUNT_IMPL macros. Replace the
implementation of isc_mem_attach(), isc_mem_detach() and
isc_mem_destroy() with the respective macros.
This also removes the legacy isc_mem_destroy() functionality that would
check whether all references had been detached from the memory context
as it doesn't work reliably when using the call_rcu() API. Instead of
doing this individually, call isc_mem_checkdestroyed(stderr) from the
isc_mem_destroy() macro to keep the extra check that all contexts were
freed when the program is exiting.
