mir/bind
2
0
Fork 0
mirror of https://gitlab.isc.org/isc-projects/bind9 synced 2025-08-24 02:58:38 +00:00
Code Issues Releases Wiki Activity
bind/lib/isc/tests/netmgr_test.c

2891 lines
74 KiB
C
Raw Normal View History

Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
/*
* Copyright (C) Internet Systems Consortium, Inc. ("ISC")
*
Update the copyright information in all files in the repository This commit converts the license handling to adhere to the REUSE specification. It specifically: 1. Adds used licnses to LICENSES/ directory 2. Add "isc" template for adding the copyright boilerplate 3. Changes all source files to include copyright and SPDX license header, this includes all the C sources, documentation, zone files, configuration files. There are notes in the doc/dev/copyrights file on how to add correct headers to the new files. 4. Handle the rest that can't be modified via .reuse/dep5 file. The binary (or otherwise unmodifiable) files could have license places next to them in <foo>.license file, but this would lead to cluttered repository and most of the files handled in the .reuse/dep5 file are system test files.
2021-06-03 08:37:05 +02:00
* SPDX-License-Identifier: MPL-2.0
*
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
* This Source Code Form is subject to the terms of the Mozilla Public
* License, v. 2.0. If a copy of the MPL was not distributed with this
* file, you can obtain one at https://mozilla.org/MPL/2.0/.
*
* See the COPYRIGHT file distributed with this work for additional
* information regarding copyright ownership.
*/
#include <sched.h> /* IWYU pragma: keep */
#include <setjmp.h>
#include <signal.h>
#include <stdarg.h>
#include <stdlib.h>
#include <unistd.h>
#define UNIT_TESTING
#include <cmocka.h>
#include <isc/nonce.h>
#include <isc/os.h>
#include <isc/quota.h>
#include <isc/refcount.h>
#include <isc/sockaddr.h>
#include <isc/thread.h>
#include <isc/util.h>
Move netmgr/uv-compat.h to <isc/uv.h> As we are going to use libuv outside of the netmgr, we need the shims to be readily available for the rest of the codebase. Move the "netmgr/uv-compat.h" to <isc/uv.h> and netmgr/uv-compat.c to uv.c, and as a rule of thumb, the users of libuv should include <isc/uv.h> instead of <uv.h> directly. Additionally, merge netmgr/uverr2result.c into uv.c and rename the single function from isc__nm_uverr2result() to isc_uverr2result().
2022-04-27 17:41:47 +02:00
#include <isc/uv.h>
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
#include "uv_wrap.h"
#define KEEP_BEFORE
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wshadow"
#include "../netmgr/netmgr.c"
Move socket related functions to netmgr/socket.c Move the netmgr socket related functions from netmgr/netmgr.c and netmgr/uv-compat.c to netmgr/socket.c, so they are all present all in the same place. Adjust the names of couple interal functions accordingly.
2022-04-27 17:41:47 +02:00
#include "../netmgr/socket.c"
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
#include "../netmgr/udp.c"
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
#include "../uv.c"
#pragma GCC diagnostic pop
#include <isc/test.h>
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
typedef void (*stream_connect_function)(isc_nm_t *nm);
allow client read callback to be assignable allow netmgr client tests to choose the function that will be used as a read callback, without having to write a different connect callback handler.
2021-04-20 17:13:15 -07:00
static void
connect_connect_cb(isc_nmhandle_t *handle, isc_result_t eresult, void *cbarg);
static void
connect_read_cb(isc_nmhandle_t *handle, isc_result_t eresult,
isc_region_t *region, void *cbarg);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_nm_t *listen_nm = NULL;
isc_nm_t *connect_nm = NULL;
static isc_sockaddr_t udp_listen_addr;
static isc_sockaddr_t udp_connect_addr;
static isc_sockaddr_t tcp_listen_addr;
static isc_sockaddr_t tcp_connect_addr;
static isc_tlsctx_t *tcp_listen_tlsctx = NULL;
static isc_tlsctx_t *tcp_connect_tlsctx = NULL;
TLS stream/DoH: implement TLS client session resumption This commit extends TLS stream code and DoH code with TLS client session resumption support implemented on top of the TLS client session cache.
2022-04-22 15:59:11 +03:00
static isc_tlsctx_client_session_cache_t *tcp_tlsctx_client_sess_cache = NULL;
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
static uint64_t send_magic = 0;
static uint64_t stop_magic = 0;
Fix failures in isc netmgr_test on big endian machines Typing from libuv structure to isc_region_t is not possible, because their sizes differ on 64 bit architectures. Little endian machines seems to be lucky and still result in test passed. But big endian machine such as s390x fails the test reliably. Fix by directly creating the buffer as isc_region_t and skipping the type conversion. More readable and still more correct.
2022-05-24 19:42:41 +02:00
static isc_region_t send_msg = { .base = (unsigned char *)&send_magic,
.length = sizeof(send_magic) };
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
Fix failures in isc netmgr_test on big endian machines Typing from libuv structure to isc_region_t is not possible, because their sizes differ on 64 bit architectures. Little endian machines seems to be lucky and still result in test passed. But big endian machine such as s390x fails the test reliably. Fix by directly creating the buffer as isc_region_t and skipping the type conversion. More readable and still more correct.
2022-05-24 19:42:41 +02:00
static isc_region_t stop_msg = { .base = (unsigned char *)&stop_magic,
.length = sizeof(stop_magic) };
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
Remove usage of deprecated ATOMIC_VAR_INIT() macro The C17 standard deprecated ATOMIC_VAR_INIT() macro (see [1]). Follow the suite and remove the ATOMIC_VAR_INIT() usage in favor of simple assignment of the value as this is what all supported stdatomic.h implementations do anyway: * MacOSX.plaform: #define ATOMIC_VAR_INIT(__v) {__v} * Gcc stdatomic.h: #define ATOMIC_VAR_INIT(VALUE) (VALUE) 1. http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p1138r0.pdf
2022-03-08 23:55:10 +01:00
static atomic_bool do_send = false;
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
static atomic_int_fast64_t nsends;
static int_fast64_t esends; /* expected sends */
Remove usage of deprecated ATOMIC_VAR_INIT() macro The C17 standard deprecated ATOMIC_VAR_INIT() macro (see [1]). Follow the suite and remove the ATOMIC_VAR_INIT() usage in favor of simple assignment of the value as this is what all supported stdatomic.h implementations do anyway: * MacOSX.plaform: #define ATOMIC_VAR_INIT(__v) {__v} * Gcc stdatomic.h: #define ATOMIC_VAR_INIT(VALUE) (VALUE) 1. http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p1138r0.pdf
2022-03-08 23:55:10 +01:00
static atomic_int_fast64_t ssends = 0;
static atomic_int_fast64_t sreads = 0;
static atomic_int_fast64_t saccepts = 0;
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
Remove usage of deprecated ATOMIC_VAR_INIT() macro The C17 standard deprecated ATOMIC_VAR_INIT() macro (see [1]). Follow the suite and remove the ATOMIC_VAR_INIT() usage in favor of simple assignment of the value as this is what all supported stdatomic.h implementations do anyway: * MacOSX.plaform: #define ATOMIC_VAR_INIT(__v) {__v} * Gcc stdatomic.h: #define ATOMIC_VAR_INIT(VALUE) (VALUE) 1. http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p1138r0.pdf
2022-03-08 23:55:10 +01:00
static atomic_int_fast64_t cconnects = 0;
static atomic_int_fast64_t csends = 0;
static atomic_int_fast64_t creads = 0;
static atomic_int_fast64_t ctimeouts = 0;
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
static isc_refcount_t active_cconnects;
static isc_refcount_t active_csends;
static isc_refcount_t active_creads;
static isc_refcount_t active_ssends;
static isc_refcount_t active_sreads;
static isc_quota_t listener_quota;
static atomic_bool check_listener_quota;
static bool skip_long_tests = false;
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
static bool allow_send_back = false;
add TCPDNS and TLSDNS timeout recovery tests this is similar in structure to the UDP timeout recovery test. this commit adds a new mechanism to the netmgr test allowing the listen socket to accept incoming TCP connections but never send a response. this forces the client to time out on read.
2021-04-21 01:09:30 -07:00
static bool noanswer = false;
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
static bool stream_use_TLS = false;
allow client read callback to be assignable allow netmgr client tests to choose the function that will be used as a read callback, without having to write a different connect callback handler.
2021-04-20 17:13:15 -07:00
static isc_nm_recv_cb_t connect_readcb = NULL;
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
#define SKIP_IN_CI \
if (skip_long_tests) { \
skip(); \
return; \
}
#define NSENDS 100
Use a constant for timeouts in soft-timeout tests It makes it easier to change the value should the need arise.
2021-04-22 13:29:34 +03:00
/* Timeout for soft-timeout tests (0.05 seconds) */
#define T_SOFT 50
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
/* Timeouts in miliseconds */
#define T_INIT 120 * 1000
#define T_IDLE 120 * 1000
#define T_KEEPALIVE 120 * 1000
#define T_ADVERTISED 120 * 1000
#define T_CONNECT 30 * 1000
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
/* Wait for 1 second (1000 milliseconds) */
Refactor async callbacks and fix the double tlsdnsconnect callback The isc_nm_tlsdnsconnect() call could end up with two connect callbacks called when the timeout fired and the TCP connection was aborted, but the TLS handshake was not complete yet. isc__nm_connecttimeout_cb() forgot to clean up sock->tls.pending_req when the connect callback was called with ISC_R_TIMEDOUT, leading to a second callback running later. A new argument has been added to the isc__nm_*_failed_connect_cb and isc__nm_*_failed_read_cb functions, to indicate whether the callback needs to run asynchronously or not.
2021-04-06 18:27:38 +02:00
#define WAIT_REPEATS 1000
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
#define T_WAIT 1 /* 1 millisecond */
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
#define WAIT_FOR(v, op, val) \
{ \
X(v); \
int_fast64_t __r = WAIT_REPEATS; \
int_fast64_t __o = 0; \
do { \
int_fast64_t __l = atomic_load(&v); \
if (__l op val) { \
break; \
}; \
if (__o == __l) { \
__r--; \
} else { \
__r = WAIT_REPEATS; \
} \
__o = __l; \
Add nanosleep and usleep Windows shims This commit adds POSIX nanosleep() and usleep() shim implementation for Windows to help implementors use less #ifdef _WIN32 in the code.
2021-04-27 10:16:01 +02:00
isc_test_nap(T_WAIT); \
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
} while (__r > 0); \
X(v); \
P(__r); \
assert_true(atomic_load(&v) op val); \
}
#define WAIT_FOR_EQ(v, val) WAIT_FOR(v, ==, val)
#define WAIT_FOR_NE(v, val) WAIT_FOR(v, !=, val)
#define WAIT_FOR_LE(v, val) WAIT_FOR(v, <=, val)
#define WAIT_FOR_LT(v, val) WAIT_FOR(v, <, val)
#define WAIT_FOR_GE(v, val) WAIT_FOR(v, >=, val)
#define WAIT_FOR_GT(v, val) WAIT_FOR(v, >, val)
#define DONE() atomic_store(&do_send, false);
#define CHECK_RANGE_FULL(v) \
{ \
int __v = atomic_load(&v); \
assert_true(__v > 1); \
}
#define CHECK_RANGE_HALF(v) \
{ \
int __v = atomic_load(&v); \
assert_true(__v > 1); \
}
/* Enable this to print values while running tests */
#undef PRINT_DEBUG
#ifdef PRINT_DEBUG
#define X(v) \
fprintf(stderr, "%s:%s:%d:%s = %" PRId64 "\n", __func__, __FILE__, \
__LINE__, #v, atomic_load(&v))
#define P(v) fprintf(stderr, #v " = %" PRId64 "\n", v)
#define F() \
fprintf(stderr, "%s(%p, %s, %p)\n", __func__, handle, \
isc_result_totext(eresult), cbarg)
#else
#define X(v)
#define P(v)
#define F()
#endif
#define atomic_assert_int_eq(val, exp) assert_int_equal(atomic_load(&val), exp)
#define atomic_assert_int_ne(val, exp) \
assert_int_not_equal(atomic_load(&val), exp)
#define atomic_assert_int_le(val, exp) assert_true(atomic_load(&val) <= exp)
#define atomic_assert_int_lt(val, exp) assert_true(atomic_load(&val) > exp)
#define atomic_assert_int_ge(val, exp) assert_true(atomic_load(&val) >= exp)
#define atomic_assert_int_gt(val, exp) assert_true(atomic_load(&val) > exp)
static int
setup_ephemeral_port(isc_sockaddr_t *addr, sa_family_t family) {
socklen_t addrlen = sizeof(*addr);
Move netmgr/uv-compat.h to <isc/uv.h> As we are going to use libuv outside of the netmgr, we need the shims to be readily available for the rest of the codebase. Move the "netmgr/uv-compat.h" to <isc/uv.h> and netmgr/uv-compat.c to uv.c, and as a rule of thumb, the users of libuv should include <isc/uv.h> instead of <uv.h> directly. Additionally, merge netmgr/uverr2result.c into uv.c and rename the single function from isc__nm_uverr2result() to isc_uverr2result().
2022-04-27 17:41:47 +02:00
uv_os_sock_t fd = -1;
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
int r;
isc_sockaddr_fromin6(addr, &in6addr_loopback, 0);
fd = socket(AF_INET6, family, 0);
if (fd < 0) {
perror("setup_ephemeral_port: socket()");
return (-1);
}
r = bind(fd, (const struct sockaddr *)&addr->type.sa,
sizeof(addr->type.sin6));
if (r != 0) {
perror("setup_ephemeral_port: bind()");
isc__nm_closesocket(fd);
return (r);
}
r = getsockname(fd, (struct sockaddr *)&addr->type.sa, &addrlen);
if (r != 0) {
perror("setup_ephemeral_port: getsockname()");
isc__nm_closesocket(fd);
return (r);
}
#if IPV6_RECVERR
#define setsockopt_on(socket, level, name) \
setsockopt(socket, level, name, &(int){ 1 }, sizeof(int))
r = setsockopt_on(fd, IPPROTO_IPV6, IPV6_RECVERR);
if (r != 0) {
perror("setup_ephemeral_port");
isc__nm_closesocket(fd);
return (r);
}
#endif
return (fd);
}
static int
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
setup_test(void **state __attribute__((unused))) {
char *env_workers = getenv("ISC_TASK_WORKERS");
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
uv_os_sock_t tcp_listen_sock = -1;
uv_os_sock_t udp_listen_sock = -1;
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
size_t nworkers;
if (env_workers != NULL) {
workers = atoi(env_workers);
} else {
workers = isc_os_ncpus();
}
INSIST(workers > 0);
nworkers = ISC_MAX(ISC_MIN(workers, 32), 1);
if (getenv("CI") != NULL && getenv("CI_ENABLE_ALL_TESTS") == NULL) {
skip_long_tests = true;
esends = nworkers;
} else {
esends = NSENDS * nworkers;
}
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
udp_connect_addr = (isc_sockaddr_t){ .length = 0 };
isc_sockaddr_fromin6(&udp_connect_addr, &in6addr_loopback, 0);
udp_listen_addr = (isc_sockaddr_t){ .length = 0 };
udp_listen_sock = setup_ephemeral_port(&udp_listen_addr, SOCK_DGRAM);
if (udp_listen_sock < 0) {
return (-1);
}
isc__nm_closesocket(udp_listen_sock);
udp_listen_sock = -1;
tcp_connect_addr = (isc_sockaddr_t){ .length = 0 };
isc_sockaddr_fromin6(&tcp_connect_addr, &in6addr_loopback, 0);
tcp_listen_addr = (isc_sockaddr_t){ .length = 0 };
tcp_listen_sock = setup_ephemeral_port(&tcp_listen_addr, SOCK_STREAM);
if (tcp_listen_sock < 0) {
return (-1);
}
isc__nm_closesocket(tcp_listen_sock);
tcp_listen_sock = -1;
atomic_store(&do_send, true);
atomic_store(&nsends, esends);
atomic_store(&saccepts, 0);
atomic_store(&sreads, 0);
atomic_store(&ssends, 0);
atomic_store(&cconnects, 0);
atomic_store(&csends, 0);
atomic_store(&creads, 0);
add a UDP timeout recovery test this test sets up a server socket that listens for UDP connections but never responds. the client will always time out; it should retry five times before giving up.
2021-04-20 21:08:10 -07:00
atomic_store(&ctimeouts, 0);
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
allow_send_back = false;
stream_use_TLS = false;
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_refcount_init(&active_cconnects, 0);
isc_refcount_init(&active_csends, 0);
isc_refcount_init(&active_creads, 0);
isc_refcount_init(&active_ssends, 0);
isc_refcount_init(&active_sreads, 0);
isc_nonce_buf(&send_magic, sizeof(send_magic));
isc_nonce_buf(&stop_magic, sizeof(stop_magic));
if (send_magic == stop_magic) {
return (-1);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
isc__netmgr_create(mctx, nworkers, &listen_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_non_null(listen_nm);
isc_nm_settimeouts(listen_nm, T_INIT, T_IDLE, T_KEEPALIVE,
T_ADVERTISED);
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
isc__netmgr_create(mctx, nworkers, &connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_non_null(connect_nm);
isc_nm_settimeouts(connect_nm, T_INIT, T_IDLE, T_KEEPALIVE,
T_ADVERTISED);
isc_quota_init(&listener_quota, 0);
atomic_store(&check_listener_quota, false);
allow client read callback to be assignable allow netmgr client tests to choose the function that will be used as a read callback, without having to write a different connect callback handler.
2021-04-20 17:13:15 -07:00
connect_readcb = connect_read_cb;
add TCPDNS and TLSDNS timeout recovery tests this is similar in structure to the UDP timeout recovery test. this commit adds a new mechanism to the netmgr test allowing the listen socket to accept incoming TCP connections but never send a response. this forces the client to time out on read.
2021-04-21 01:09:30 -07:00
noanswer = false;
allow client read callback to be assignable allow netmgr client tests to choose the function that will be used as a read callback, without having to write a different connect callback handler.
2021-04-20 17:13:15 -07:00
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
if (isc_tlsctx_createserver(NULL, NULL, &tcp_listen_tlsctx) !=
ISC_R_SUCCESS) {
return (-1);
}
if (isc_tlsctx_createclient(&tcp_connect_tlsctx) != ISC_R_SUCCESS) {
return (-1);
}
isc_tlsctx_enable_dot_client_alpn(tcp_connect_tlsctx);
tcp_tlsctx_client_sess_cache = isc_tlsctx_client_session_cache_new(
mctx, tcp_connect_tlsctx,
ISC_TLSCTX_CLIENT_SESSION_CACHE_DEFAULT_SIZE);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
return (0);
}
static int
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
teardown_test(void **state __attribute__((unused))) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
UNUSED(state);
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
isc_tlsctx_free(&tcp_connect_tlsctx);
isc_tlsctx_free(&tcp_listen_tlsctx);
Add new isc_managers API to simplify <*>mgr create/destroy Previously, netmgr, taskmgr, timermgr and socketmgr all had their own isc_<*>mgr_create() and isc_<*>mgr_destroy() functions. The new isc_managers_create() and isc_managers_destroy() fold all four into a single function and makes sure the objects are created and destroy in correct order. Especially now, when taskmgr runs on top of netmgr, the correct order is important and when the code was duplicated at many places it's easy to make mistake. The former isc_<*>mgr_create() and isc_<*>mgr_destroy() functions were made private and a single call to isc_managers_create() and isc_managers_destroy() is required at the program startup / shutdown.
2021-04-27 00:07:43 +02:00
isc__netmgr_destroy(&connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_null(connect_nm);
Add new isc_managers API to simplify <*>mgr create/destroy Previously, netmgr, taskmgr, timermgr and socketmgr all had their own isc_<*>mgr_create() and isc_<*>mgr_destroy() functions. The new isc_managers_create() and isc_managers_destroy() fold all four into a single function and makes sure the objects are created and destroy in correct order. Especially now, when taskmgr runs on top of netmgr, the correct order is important and when the code was duplicated at many places it's easy to make mistake. The former isc_<*>mgr_create() and isc_<*>mgr_destroy() functions were made private and a single call to isc_managers_create() and isc_managers_destroy() is required at the program startup / shutdown.
2021-04-27 00:07:43 +02:00
isc__netmgr_destroy(&listen_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_null(listen_nm);
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
isc_tlsctx_client_session_cache_detach(&tcp_tlsctx_client_sess_cache);
rearrange nm_teardown() to check correctness after shutting down if a test failed at the beginning of nm_teardown(), the function would abort before isc_nm_destroy() or isc_tlsctx_free() were reached; we would then abort when nm_setup() was run for the next test case. rearranging the teardown function prevents this problem.
2021-04-02 22:34:18 -07:00
WAIT_FOR_EQ(active_cconnects, 0);
WAIT_FOR_EQ(active_csends, 0);
WAIT_FOR_EQ(active_csends, 0);
WAIT_FOR_EQ(active_ssends, 0);
WAIT_FOR_EQ(active_sreads, 0);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_refcount_destroy(&active_cconnects);
isc_refcount_destroy(&active_csends);
isc_refcount_destroy(&active_creads);
isc_refcount_destroy(&active_ssends);
isc_refcount_destroy(&active_sreads);
return (0);
}
/* Callbacks */
static void
noop_recv_cb(isc_nmhandle_t *handle, isc_result_t eresult, isc_region_t *region,
void *cbarg) {
UNUSED(handle);
UNUSED(eresult);
UNUSED(region);
UNUSED(cbarg);
}
static unsigned int
noop_accept_cb(isc_nmhandle_t *handle, unsigned int result, void *cbarg) {
UNUSED(handle);
UNUSED(result);
UNUSED(cbarg);
return (0);
}
static void
connect_send_cb(isc_nmhandle_t *handle, isc_result_t eresult, void *cbarg);
static void
connect_send(isc_nmhandle_t *handle);
static void
connect_send_cb(isc_nmhandle_t *handle, isc_result_t eresult, void *cbarg) {
isc_nmhandle_t *sendhandle = handle;
assert_non_null(sendhandle);
UNUSED(cbarg);
F();
if (eresult != ISC_R_SUCCESS) {
/* Send failed, we need to stop reading too */
isc_nm_cancelread(handle);
goto unref;
}
atomic_fetch_add(&csends, 1);
unref:
isc_refcount_decrement(&active_csends);
isc_nmhandle_detach(&sendhandle);
}
static void
connect_send(isc_nmhandle_t *handle) {
isc_nmhandle_t *sendhandle = NULL;
isc_refcount_increment0(&active_csends);
isc_nmhandle_attach(handle, &sendhandle);
Update writetimeout to be T_IDLE in netmgr_test.c Use the isc_nmhandle_setwritetimeout() function in the netmgr unit test to allow more time for writing and reading the responses because some of the intervals that are used in the unit tests are really small leaving a little room for any delays.
2022-02-10 08:42:22 +01:00
isc_nmhandle_setwritetimeout(handle, T_IDLE);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
if (atomic_fetch_sub(&nsends, 1) > 1) {
Fix failures in isc netmgr_test on big endian machines Typing from libuv structure to isc_region_t is not possible, because their sizes differ on 64 bit architectures. Little endian machines seems to be lucky and still result in test passed. But big endian machine such as s390x fails the test reliably. Fix by directly creating the buffer as isc_region_t and skipping the type conversion. More readable and still more correct.
2022-05-24 19:42:41 +02:00
isc_nm_send(sendhandle, &send_msg, connect_send_cb, NULL);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
} else {
Fix failures in isc netmgr_test on big endian machines Typing from libuv structure to isc_region_t is not possible, because their sizes differ on 64 bit architectures. Little endian machines seems to be lucky and still result in test passed. But big endian machine such as s390x fails the test reliably. Fix by directly creating the buffer as isc_region_t and skipping the type conversion. More readable and still more correct.
2022-05-24 19:42:41 +02:00
isc_nm_send(sendhandle, &stop_msg, connect_send_cb, NULL);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
}
}
static void
connect_read_cb(isc_nmhandle_t *handle, isc_result_t eresult,
isc_region_t *region, void *cbarg) {
uint64_t magic = 0;
UNUSED(cbarg);
assert_non_null(handle);
F();
if (eresult != ISC_R_SUCCESS) {
goto unref;
}
add TCP and TLS timeout recovery tests NOTE: currently these tests fail
2021-04-21 10:27:09 -07:00
assert_true(region->length >= sizeof(magic));
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
atomic_fetch_add(&creads, 1);
memmove(&magic, region->base, sizeof(magic));
assert_true(magic == stop_magic || magic == send_magic);
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
if (magic == send_magic && allow_send_back) {
connect_send(handle);
return;
}
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
unref:
add a UDP timeout recovery test this test sets up a server socket that listens for UDP connections but never responds. the client will always time out; it should retry five times before giving up.
2021-04-20 21:08:10 -07:00
isc_refcount_decrement(&active_creads);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_nmhandle_detach(&handle);
}
static void
connect_connect_cb(isc_nmhandle_t *handle, isc_result_t eresult, void *cbarg) {
isc_nmhandle_t *readhandle = NULL;
isc_nm_*connect() always return via callback The isc_nm_*connect() functions were refactored to always return the connection status via the connect callback instead of sometimes returning the hard failure directly (for example, when the socket could not be created, or when the network manager was shutting down). This commit changes the connect functions in all the network manager modules, and also makes the necessary refactoring changes in places where the connect functions are called.
2021-03-31 18:32:32 +02:00
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
UNUSED(cbarg);
F();
isc_nm_*connect() always return via callback The isc_nm_*connect() functions were refactored to always return the connection status via the connect callback instead of sometimes returning the hard failure directly (for example, when the socket could not be created, or when the network manager was shutting down). This commit changes the connect functions in all the network manager modules, and also makes the necessary refactoring changes in places where the connect functions are called.
2021-03-31 18:32:32 +02:00
isc_refcount_decrement(&active_cconnects);
allow client read callback to be assignable allow netmgr client tests to choose the function that will be used as a read callback, without having to write a different connect callback handler.
2021-04-20 17:13:15 -07:00
if (eresult != ISC_R_SUCCESS || connect_readcb == NULL) {
isc_nm_*connect() always return via callback The isc_nm_*connect() functions were refactored to always return the connection status via the connect callback instead of sometimes returning the hard failure directly (for example, when the socket could not be created, or when the network manager was shutting down). This commit changes the connect functions in all the network manager modules, and also makes the necessary refactoring changes in places where the connect functions are called.
2021-03-31 18:32:32 +02:00
return;
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
}
atomic_fetch_add(&cconnects, 1);
isc_refcount_increment0(&active_creads);
isc_nmhandle_attach(handle, &readhandle);
allow client read callback to be assignable allow netmgr client tests to choose the function that will be used as a read callback, without having to write a different connect callback handler.
2021-04-20 17:13:15 -07:00
isc_nm_read(handle, connect_readcb, NULL);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
connect_send(handle);
}
static void
listen_send_cb(isc_nmhandle_t *handle, isc_result_t eresult, void *cbarg) {
isc_nmhandle_t *sendhandle = handle;
UNUSED(cbarg);
UNUSED(eresult);
assert_non_null(sendhandle);
F();
if (eresult != ISC_R_SUCCESS) {
goto unref;
}
atomic_fetch_add(&ssends, 1);
unref:
isc_nmhandle_detach(&sendhandle);
isc_refcount_decrement(&active_ssends);
}
static void
listen_read_cb(isc_nmhandle_t *handle, isc_result_t eresult,
isc_region_t *region, void *cbarg) {
uint64_t magic = 0;
assert_non_null(handle);
F();
if (eresult != ISC_R_SUCCESS) {
goto unref;
}
atomic_fetch_add(&sreads, 1);
add TCP and TLS timeout recovery tests NOTE: currently these tests fail
2021-04-21 10:27:09 -07:00
assert_true(region->length >= sizeof(magic));
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
memmove(&magic, region->base, sizeof(magic));
assert_true(magic == stop_magic || magic == send_magic);
add TCP and TLS timeout recovery tests NOTE: currently these tests fail
2021-04-21 10:27:09 -07:00
if (magic == send_magic) {
if (!noanswer) {
isc_nmhandle_t *sendhandle = NULL;
isc_nmhandle_attach(handle, &sendhandle);
isc_refcount_increment0(&active_ssends);
Update writetimeout to be T_IDLE in netmgr_test.c Use the isc_nmhandle_setwritetimeout() function in the netmgr unit test to allow more time for writing and reading the responses because some of the intervals that are used in the unit tests are really small leaving a little room for any delays.
2022-02-10 08:42:22 +01:00
isc_nmhandle_setwritetimeout(sendhandle, T_IDLE);
Fix failures in isc netmgr_test on big endian machines Typing from libuv structure to isc_region_t is not possible, because their sizes differ on 64 bit architectures. Little endian machines seems to be lucky and still result in test passed. But big endian machine such as s390x fails the test reliably. Fix by directly creating the buffer as isc_region_t and skipping the type conversion. More readable and still more correct.
2022-05-24 19:42:41 +02:00
isc_nm_send(sendhandle, &send_msg, listen_send_cb,
cbarg);
add TCP and TLS timeout recovery tests NOTE: currently these tests fail
2021-04-21 10:27:09 -07:00
}
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
return;
}
allow client read callback to be assignable allow netmgr client tests to choose the function that will be used as a read callback, without having to write a different connect callback handler.
2021-04-20 17:13:15 -07:00
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
unref:
if (handle == cbarg) {
isc_refcount_decrement(&active_sreads);
isc_nmhandle_detach(&handle);
}
}
static isc_result_t
listen_accept_cb(isc_nmhandle_t *handle, isc_result_t eresult, void *cbarg) {
UNUSED(handle);
UNUSED(cbarg);
F();
return (eresult);
}
static isc_result_t
stream_accept_cb(isc_nmhandle_t *handle, isc_result_t eresult, void *cbarg) {
isc_nmhandle_t *readhandle = NULL;
UNUSED(cbarg);
F();
if (eresult != ISC_R_SUCCESS) {
return (eresult);
}
atomic_fetch_add(&saccepts, 1);
isc_refcount_increment0(&active_sreads);
isc_nmhandle_attach(handle, &readhandle);
isc_nm_read(handle, listen_read_cb, readhandle);
return (ISC_R_SUCCESS);
}
isc_nm_*connect() always return via callback The isc_nm_*connect() functions were refactored to always return the connection status via the connect callback instead of sometimes returning the hard failure directly (for example, when the socket could not be created, or when the network manager was shutting down). This commit changes the connect functions in all the network manager modules, and also makes the necessary refactoring changes in places where the connect functions are called.
2021-03-31 18:32:32 +02:00
typedef void (*connect_func)(isc_nm_t *);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
static isc_threadresult_t
connect_thread(isc_threadarg_t arg) {
connect_func connect = (connect_func)arg;
isc_sockaddr_t connect_addr;
connect_addr = (isc_sockaddr_t){ .length = 0 };
isc_sockaddr_fromin6(&connect_addr, &in6addr_loopback, 0);
while (atomic_load(&do_send)) {
Refactor async callbacks and fix the double tlsdnsconnect callback The isc_nm_tlsdnsconnect() call could end up with two connect callbacks called when the timeout fired and the TCP connection was aborted, but the TLS handshake was not complete yet. isc__nm_connecttimeout_cb() forgot to clean up sock->tls.pending_req when the connect callback was called with ISC_R_TIMEDOUT, leading to a second callback running later. A new argument has been added to the isc__nm_*_failed_connect_cb and isc__nm_*_failed_read_cb functions, to indicate whether the callback needs to run asynchronously or not.
2021-04-06 18:27:38 +02:00
uint_fast32_t active =
isc_refcount_increment0(&active_cconnects);
isc_nm_*connect() always return via callback The isc_nm_*connect() functions were refactored to always return the connection status via the connect callback instead of sometimes returning the hard failure directly (for example, when the socket could not be created, or when the network manager was shutting down). This commit changes the connect functions in all the network manager modules, and also makes the necessary refactoring changes in places where the connect functions are called.
2021-03-31 18:32:32 +02:00
if (active > workers) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
/*
isc_nm_*connect() always return via callback The isc_nm_*connect() functions were refactored to always return the connection status via the connect callback instead of sometimes returning the hard failure directly (for example, when the socket could not be created, or when the network manager was shutting down). This commit changes the connect functions in all the network manager modules, and also makes the necessary refactoring changes in places where the connect functions are called.
2021-03-31 18:32:32 +02:00
* If we have more active connections than workers,
* start slowing down the connections to prevent the
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
* thundering herd problem.
*/
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
isc_test_nap(active - workers);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
}
isc_nm_*connect() always return via callback The isc_nm_*connect() functions were refactored to always return the connection status via the connect callback instead of sometimes returning the hard failure directly (for example, when the socket could not be created, or when the network manager was shutting down). This commit changes the connect functions in all the network manager modules, and also makes the necessary refactoring changes in places where the connect functions are called.
2021-03-31 18:32:32 +02:00
connect(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
}
return ((isc_threadresult_t)0);
}
/* UDP */
isc_nm_*connect() always return via callback The isc_nm_*connect() functions were refactored to always return the connection status via the connect callback instead of sometimes returning the hard failure directly (for example, when the socket could not be created, or when the network manager was shutting down). This commit changes the connect functions in all the network manager modules, and also makes the necessary refactoring changes in places where the connect functions are called.
2021-03-31 18:32:32 +02:00
static void
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
udp_connect(isc_nm_t *nm) {
Refactor the interface handling in the netmgr The isc_nmiface_t type was holding just a single isc_sockaddr_t, so we got rid of the datatype and use plain isc_sockaddr_t in place where isc_nmiface_t was used before. This means less type-casting and shorter path to access isc_sockaddr_t members. At the same time, instead of keeping the reference to the isc_sockaddr_t that was passed to us when we start listening, we will keep a local copy. This prevents the data race on destruction of the ns_interface_t objects where pending nmsockets could reference the sockaddr of already destroyed ns_interface_t object.
2021-05-26 08:15:34 +02:00
isc_nm_udpconnect(nm, &udp_connect_addr, &udp_listen_addr,
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
connect_connect_cb, NULL, T_CONNECT);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(mock_listenudp_uv_udp_open) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
WILL_RETURN(uv_udp_open, UV_ENOMEM);
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listenudp(listen_nm, ISC_NM_LISTEN_ALL,
&udp_listen_addr, noop_recv_cb, NULL,
&listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_not_equal(result, ISC_R_SUCCESS);
assert_null(listen_sock);
RESET_RETURN;
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(mock_listenudp_uv_udp_bind) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
WILL_RETURN(uv_udp_bind, UV_EADDRINUSE);
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listenudp(listen_nm, ISC_NM_LISTEN_ALL,
&udp_listen_addr, noop_recv_cb, NULL,
&listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_not_equal(result, ISC_R_SUCCESS);
assert_null(listen_sock);
RESET_RETURN;
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(mock_listenudp_uv_udp_recv_start) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
WILL_RETURN(uv_udp_recv_start, UV_EADDRINUSE);
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listenudp(listen_nm, ISC_NM_LISTEN_ALL,
&udp_listen_addr, noop_recv_cb, NULL,
&listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_not_equal(result, ISC_R_SUCCESS);
assert_null(listen_sock);
RESET_RETURN;
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(mock_udpconnect_uv_udp_open) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
WILL_RETURN(uv_udp_open, UV_ENOMEM);
allow client read callback to be assignable allow netmgr client tests to choose the function that will be used as a read callback, without having to write a different connect callback handler.
2021-04-20 17:13:15 -07:00
connect_readcb = NULL;
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_refcount_increment0(&active_cconnects);
Refactor the interface handling in the netmgr The isc_nmiface_t type was holding just a single isc_sockaddr_t, so we got rid of the datatype and use plain isc_sockaddr_t in place where isc_nmiface_t was used before. This means less type-casting and shorter path to access isc_sockaddr_t members. At the same time, instead of keeping the reference to the isc_sockaddr_t that was passed to us when we start listening, we will keep a local copy. This prevents the data race on destruction of the ns_interface_t objects where pending nmsockets could reference the sockaddr of already destroyed ns_interface_t object.
2021-05-26 08:15:34 +02:00
isc_nm_udpconnect(connect_nm, &udp_connect_addr, &udp_listen_addr,
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
connect_connect_cb, NULL, T_CONNECT);
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
RESET_RETURN;
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(mock_udpconnect_uv_udp_bind) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
WILL_RETURN(uv_udp_bind, UV_ENOMEM);
allow client read callback to be assignable allow netmgr client tests to choose the function that will be used as a read callback, without having to write a different connect callback handler.
2021-04-20 17:13:15 -07:00
connect_readcb = NULL;
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_refcount_increment0(&active_cconnects);
Refactor the interface handling in the netmgr The isc_nmiface_t type was holding just a single isc_sockaddr_t, so we got rid of the datatype and use plain isc_sockaddr_t in place where isc_nmiface_t was used before. This means less type-casting and shorter path to access isc_sockaddr_t members. At the same time, instead of keeping the reference to the isc_sockaddr_t that was passed to us when we start listening, we will keep a local copy. This prevents the data race on destruction of the ns_interface_t objects where pending nmsockets could reference the sockaddr of already destroyed ns_interface_t object.
2021-05-26 08:15:34 +02:00
isc_nm_udpconnect(connect_nm, &udp_connect_addr, &udp_listen_addr,
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
connect_connect_cb, NULL, T_CONNECT);
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
RESET_RETURN;
}
Cleanup the remaining of HAVE_UV_<func> macros While cleaning up the usage of HAVE_UV_<func> macros, we forgot to cleanup the HAVE_UV_UDP_CONNECT in the actual code and HAVE_UV_TRANSLATE_SYS_ERROR and this was causing Windows build to fail on uv_udp_send() because the socket was already connected and we were falsely assuming that it was not. The platforms with autoconf support were not affected, because we were still checking for the functions from the configure.
2021-06-02 11:23:36 +02:00
#if UV_VERSION_HEX >= UV_VERSION(1, 27, 0)
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(mock_udpconnect_uv_udp_connect) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
WILL_RETURN(uv_udp_connect, UV_ENOMEM);
allow client read callback to be assignable allow netmgr client tests to choose the function that will be used as a read callback, without having to write a different connect callback handler.
2021-04-20 17:13:15 -07:00
connect_readcb = NULL;
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_refcount_increment0(&active_cconnects);
Refactor the interface handling in the netmgr The isc_nmiface_t type was holding just a single isc_sockaddr_t, so we got rid of the datatype and use plain isc_sockaddr_t in place where isc_nmiface_t was used before. This means less type-casting and shorter path to access isc_sockaddr_t members. At the same time, instead of keeping the reference to the isc_sockaddr_t that was passed to us when we start listening, we will keep a local copy. This prevents the data race on destruction of the ns_interface_t objects where pending nmsockets could reference the sockaddr of already destroyed ns_interface_t object.
2021-05-26 08:15:34 +02:00
isc_nm_udpconnect(connect_nm, &udp_connect_addr, &udp_listen_addr,
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
connect_connect_cb, NULL, T_CONNECT);
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
RESET_RETURN;
}
#endif
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(mock_udpconnect_uv_recv_buffer_size) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
WILL_RETURN(uv_recv_buffer_size, UV_ENOMEM);
allow client read callback to be assignable allow netmgr client tests to choose the function that will be used as a read callback, without having to write a different connect callback handler.
2021-04-20 17:13:15 -07:00
connect_readcb = NULL;
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_refcount_increment0(&active_cconnects);
Refactor the interface handling in the netmgr The isc_nmiface_t type was holding just a single isc_sockaddr_t, so we got rid of the datatype and use plain isc_sockaddr_t in place where isc_nmiface_t was used before. This means less type-casting and shorter path to access isc_sockaddr_t members. At the same time, instead of keeping the reference to the isc_sockaddr_t that was passed to us when we start listening, we will keep a local copy. This prevents the data race on destruction of the ns_interface_t objects where pending nmsockets could reference the sockaddr of already destroyed ns_interface_t object.
2021-05-26 08:15:34 +02:00
isc_nm_udpconnect(connect_nm, &udp_connect_addr, &udp_listen_addr,
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
connect_connect_cb, NULL, T_CONNECT);
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
RESET_RETURN;
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(mock_udpconnect_uv_send_buffer_size) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
WILL_RETURN(uv_send_buffer_size, UV_ENOMEM);
allow client read callback to be assignable allow netmgr client tests to choose the function that will be used as a read callback, without having to write a different connect callback handler.
2021-04-20 17:13:15 -07:00
connect_readcb = NULL;
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_refcount_increment0(&active_cconnects);
Refactor the interface handling in the netmgr The isc_nmiface_t type was holding just a single isc_sockaddr_t, so we got rid of the datatype and use plain isc_sockaddr_t in place where isc_nmiface_t was used before. This means less type-casting and shorter path to access isc_sockaddr_t members. At the same time, instead of keeping the reference to the isc_sockaddr_t that was passed to us when we start listening, we will keep a local copy. This prevents the data race on destruction of the ns_interface_t objects where pending nmsockets could reference the sockaddr of already destroyed ns_interface_t object.
2021-05-26 08:15:34 +02:00
isc_nm_udpconnect(connect_nm, &udp_connect_addr, &udp_listen_addr,
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
connect_connect_cb, NULL, T_CONNECT);
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
RESET_RETURN;
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(udp_noop) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listenudp(listen_nm, ISC_NM_LISTEN_ALL,
&udp_listen_addr, noop_recv_cb, NULL,
&listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_equal(result, ISC_R_SUCCESS);
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
allow client read callback to be assignable allow netmgr client tests to choose the function that will be used as a read callback, without having to write a different connect callback handler.
2021-04-20 17:13:15 -07:00
connect_readcb = NULL;
isc_nm_*connect() always return via callback The isc_nm_*connect() functions were refactored to always return the connection status via the connect callback instead of sometimes returning the hard failure directly (for example, when the socket could not be created, or when the network manager was shutting down). This commit changes the connect functions in all the network manager modules, and also makes the necessary refactoring changes in places where the connect functions are called.
2021-03-31 18:32:32 +02:00
isc_refcount_increment0(&active_cconnects);
Refactor the interface handling in the netmgr The isc_nmiface_t type was holding just a single isc_sockaddr_t, so we got rid of the datatype and use plain isc_sockaddr_t in place where isc_nmiface_t was used before. This means less type-casting and shorter path to access isc_sockaddr_t members. At the same time, instead of keeping the reference to the isc_sockaddr_t that was passed to us when we start listening, we will keep a local copy. This prevents the data race on destruction of the ns_interface_t objects where pending nmsockets could reference the sockaddr of already destroyed ns_interface_t object.
2021-05-26 08:15:34 +02:00
isc_nm_udpconnect(connect_nm, &udp_connect_addr, &udp_listen_addr,
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
connect_connect_cb, NULL, T_CONNECT);
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
atomic_assert_int_eq(cconnects, 0);
atomic_assert_int_eq(csends, 0);
atomic_assert_int_eq(creads, 0);
atomic_assert_int_eq(sreads, 0);
atomic_assert_int_eq(ssends, 0);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(udp_noresponse) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listenudp(listen_nm, ISC_NM_LISTEN_ALL,
&udp_listen_addr, noop_recv_cb, NULL,
&listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_equal(result, ISC_R_SUCCESS);
isc_nm_*connect() always return via callback The isc_nm_*connect() functions were refactored to always return the connection status via the connect callback instead of sometimes returning the hard failure directly (for example, when the socket could not be created, or when the network manager was shutting down). This commit changes the connect functions in all the network manager modules, and also makes the necessary refactoring changes in places where the connect functions are called.
2021-03-31 18:32:32 +02:00
isc_refcount_increment0(&active_cconnects);
Refactor the interface handling in the netmgr The isc_nmiface_t type was holding just a single isc_sockaddr_t, so we got rid of the datatype and use plain isc_sockaddr_t in place where isc_nmiface_t was used before. This means less type-casting and shorter path to access isc_sockaddr_t members. At the same time, instead of keeping the reference to the isc_sockaddr_t that was passed to us when we start listening, we will keep a local copy. This prevents the data race on destruction of the ns_interface_t objects where pending nmsockets could reference the sockaddr of already destroyed ns_interface_t object.
2021-05-26 08:15:34 +02:00
isc_nm_udpconnect(connect_nm, &udp_connect_addr, &udp_listen_addr,
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
connect_connect_cb, NULL, T_CONNECT);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
WAIT_FOR_EQ(cconnects, 1);
WAIT_FOR_EQ(csends, 1);
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
X(cconnects);
X(csends);
X(creads);
X(sreads);
X(ssends);
atomic_assert_int_eq(cconnects, 1);
atomic_assert_int_eq(csends, 1);
atomic_assert_int_eq(creads, 0);
atomic_assert_int_eq(sreads, 0);
atomic_assert_int_eq(ssends, 0);
}
add a UDP timeout recovery test this test sets up a server socket that listens for UDP connections but never responds. the client will always time out; it should retry five times before giving up.
2021-04-20 21:08:10 -07:00
static void
timeout_retry_cb(isc_nmhandle_t *handle, isc_result_t eresult,
isc_region_t *region, void *cbarg) {
UNUSED(region);
UNUSED(cbarg);
assert_non_null(handle);
F();
if (eresult == ISC_R_TIMEDOUT && atomic_load(&csends) < 5) {
Use a constant for timeouts in soft-timeout tests It makes it easier to change the value should the need arise.
2021-04-22 13:29:34 +03:00
isc_nmhandle_settimeout(handle, T_SOFT);
add a UDP timeout recovery test this test sets up a server socket that listens for UDP connections but never responds. the client will always time out; it should retry five times before giving up.
2021-04-20 21:08:10 -07:00
connect_send(handle);
return;
}
atomic_fetch_add(&ctimeouts, 1);
isc_refcount_decrement(&active_creads);
isc_nmhandle_detach(&handle);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(udp_timeout_recovery) {
add a UDP timeout recovery test this test sets up a server socket that listens for UDP connections but never responds. the client will always time out; it should retry five times before giving up.
2021-04-20 21:08:10 -07:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
SKIP_IN_CI;
/*
* Listen using the noop callback so that client reads will time out.
*/
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listenudp(listen_nm, ISC_NM_LISTEN_ALL,
&udp_listen_addr, noop_recv_cb, NULL,
&listen_sock);
add a UDP timeout recovery test this test sets up a server socket that listens for UDP connections but never responds. the client will always time out; it should retry five times before giving up.
2021-04-20 21:08:10 -07:00
assert_int_equal(result, ISC_R_SUCCESS);
/*
* Connect with client timeout set to 0.05 seconds, then sleep for at
* least a second for each 'tick'. timeout_retry_cb() will give up
* after five timeouts.
*/
connect_readcb = timeout_retry_cb;
isc_refcount_increment0(&active_cconnects);
Refactor the interface handling in the netmgr The isc_nmiface_t type was holding just a single isc_sockaddr_t, so we got rid of the datatype and use plain isc_sockaddr_t in place where isc_nmiface_t was used before. This means less type-casting and shorter path to access isc_sockaddr_t members. At the same time, instead of keeping the reference to the isc_sockaddr_t that was passed to us when we start listening, we will keep a local copy. This prevents the data race on destruction of the ns_interface_t objects where pending nmsockets could reference the sockaddr of already destroyed ns_interface_t object.
2021-05-26 08:15:34 +02:00
isc_nm_udpconnect(connect_nm, &udp_connect_addr, &udp_listen_addr,
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
connect_connect_cb, NULL, T_SOFT);
add a UDP timeout recovery test this test sets up a server socket that listens for UDP connections but never responds. the client will always time out; it should retry five times before giving up.
2021-04-20 21:08:10 -07:00
WAIT_FOR_EQ(cconnects, 1);
WAIT_FOR_GE(csends, 1);
WAIT_FOR_GE(csends, 2);
WAIT_FOR_GE(csends, 3);
WAIT_FOR_GE(csends, 4);
WAIT_FOR_EQ(csends, 5);
WAIT_FOR_EQ(ctimeouts, 1);
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
add a UDP timeout recovery test this test sets up a server socket that listens for UDP connections but never responds. the client will always time out; it should retry five times before giving up.
2021-04-20 21:08:10 -07:00
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(udp_recv_one) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
atomic_store(&nsends, 1);
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listenudp(listen_nm, ISC_NM_LISTEN_ALL,
&udp_listen_addr, listen_read_cb, NULL,
&listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_equal(result, ISC_R_SUCCESS);
isc_nm_*connect() always return via callback The isc_nm_*connect() functions were refactored to always return the connection status via the connect callback instead of sometimes returning the hard failure directly (for example, when the socket could not be created, or when the network manager was shutting down). This commit changes the connect functions in all the network manager modules, and also makes the necessary refactoring changes in places where the connect functions are called.
2021-03-31 18:32:32 +02:00
isc_refcount_increment0(&active_cconnects);
Refactor the interface handling in the netmgr The isc_nmiface_t type was holding just a single isc_sockaddr_t, so we got rid of the datatype and use plain isc_sockaddr_t in place where isc_nmiface_t was used before. This means less type-casting and shorter path to access isc_sockaddr_t members. At the same time, instead of keeping the reference to the isc_sockaddr_t that was passed to us when we start listening, we will keep a local copy. This prevents the data race on destruction of the ns_interface_t objects where pending nmsockets could reference the sockaddr of already destroyed ns_interface_t object.
2021-05-26 08:15:34 +02:00
isc_nm_udpconnect(connect_nm, &udp_connect_addr, &udp_listen_addr,
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
connect_connect_cb, NULL, T_CONNECT);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
WAIT_FOR_EQ(cconnects, 1);
WAIT_FOR_LE(nsends, 0);
WAIT_FOR_EQ(csends, 1);
WAIT_FOR_EQ(sreads, 1);
WAIT_FOR_EQ(ssends, 0);
WAIT_FOR_EQ(creads, 0);
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
X(cconnects);
X(csends);
X(creads);
X(sreads);
X(ssends);
atomic_assert_int_eq(cconnects, 1);
atomic_assert_int_eq(csends, 1);
atomic_assert_int_eq(creads, 0);
atomic_assert_int_eq(sreads, 1);
atomic_assert_int_eq(ssends, 0);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(udp_recv_two) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
atomic_store(&nsends, 2);
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listenudp(listen_nm, ISC_NM_LISTEN_ALL,
&udp_listen_addr, listen_read_cb, NULL,
&listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_equal(result, ISC_R_SUCCESS);
isc_nm_*connect() always return via callback The isc_nm_*connect() functions were refactored to always return the connection status via the connect callback instead of sometimes returning the hard failure directly (for example, when the socket could not be created, or when the network manager was shutting down). This commit changes the connect functions in all the network manager modules, and also makes the necessary refactoring changes in places where the connect functions are called.
2021-03-31 18:32:32 +02:00
isc_refcount_increment0(&active_cconnects);
Refactor the interface handling in the netmgr The isc_nmiface_t type was holding just a single isc_sockaddr_t, so we got rid of the datatype and use plain isc_sockaddr_t in place where isc_nmiface_t was used before. This means less type-casting and shorter path to access isc_sockaddr_t members. At the same time, instead of keeping the reference to the isc_sockaddr_t that was passed to us when we start listening, we will keep a local copy. This prevents the data race on destruction of the ns_interface_t objects where pending nmsockets could reference the sockaddr of already destroyed ns_interface_t object.
2021-05-26 08:15:34 +02:00
isc_nm_udpconnect(connect_nm, &udp_connect_addr, &udp_listen_addr,
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
connect_connect_cb, NULL, T_CONNECT);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
WAIT_FOR_EQ(cconnects, 1);
isc_nm_*connect() always return via callback The isc_nm_*connect() functions were refactored to always return the connection status via the connect callback instead of sometimes returning the hard failure directly (for example, when the socket could not be created, or when the network manager was shutting down). This commit changes the connect functions in all the network manager modules, and also makes the necessary refactoring changes in places where the connect functions are called.
2021-03-31 18:32:32 +02:00
isc_refcount_increment0(&active_cconnects);
Refactor the interface handling in the netmgr The isc_nmiface_t type was holding just a single isc_sockaddr_t, so we got rid of the datatype and use plain isc_sockaddr_t in place where isc_nmiface_t was used before. This means less type-casting and shorter path to access isc_sockaddr_t members. At the same time, instead of keeping the reference to the isc_sockaddr_t that was passed to us when we start listening, we will keep a local copy. This prevents the data race on destruction of the ns_interface_t objects where pending nmsockets could reference the sockaddr of already destroyed ns_interface_t object.
2021-05-26 08:15:34 +02:00
isc_nm_udpconnect(connect_nm, &udp_connect_addr, &udp_listen_addr,
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
connect_connect_cb, NULL, T_CONNECT);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
WAIT_FOR_EQ(cconnects, 2);
WAIT_FOR_LE(nsends, 0);
WAIT_FOR_EQ(csends, 2);
WAIT_FOR_EQ(sreads, 2);
WAIT_FOR_EQ(ssends, 1);
WAIT_FOR_EQ(creads, 1);
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
X(cconnects);
X(csends);
X(creads);
X(sreads);
X(ssends);
atomic_assert_int_eq(cconnects, 2);
atomic_assert_int_eq(csends, 2);
atomic_assert_int_eq(creads, 1);
atomic_assert_int_eq(sreads, 2);
atomic_assert_int_eq(ssends, 1);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(udp_recv_send) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
isc_thread_t threads[workers];
SKIP_IN_CI;
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listenudp(listen_nm, ISC_NM_LISTEN_ALL,
&udp_listen_addr, listen_read_cb, NULL,
&listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_equal(result, ISC_R_SUCCESS);
memset(threads, 0, sizeof(threads));
for (size_t i = 0; i < workers; i++) {
isc_thread_create(connect_thread, udp_connect, &threads[i]);
}
WAIT_FOR_GE(cconnects, esends);
WAIT_FOR_GE(csends, esends);
WAIT_FOR_GE(sreads, esends);
WAIT_FOR_GE(ssends, esends / 2);
WAIT_FOR_GE(creads, esends / 2);
DONE();
for (size_t i = 0; i < workers; i++) {
isc_thread_join(threads[i], NULL);
}
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
X(cconnects);
X(csends);
X(creads);
X(sreads);
X(ssends);
CHECK_RANGE_FULL(csends);
CHECK_RANGE_FULL(creads);
CHECK_RANGE_FULL(sreads);
CHECK_RANGE_FULL(ssends);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(udp_recv_half_send) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
isc_thread_t threads[workers];
SKIP_IN_CI;
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listenudp(listen_nm, ISC_NM_LISTEN_ALL,
&udp_listen_addr, listen_read_cb, NULL,
&listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_equal(result, ISC_R_SUCCESS);
memset(threads, 0, sizeof(threads));
for (size_t i = 0; i < workers; i++) {
isc_thread_create(connect_thread, udp_connect, &threads[i]);
}
WAIT_FOR_GE(cconnects, esends / 2);
WAIT_FOR_GE(csends, esends / 2);
WAIT_FOR_GE(sreads, esends / 2);
WAIT_FOR_GE(ssends, esends / 2);
WAIT_FOR_GE(creads, esends / 2);
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
DONE();
for (size_t i = 0; i < workers; i++) {
isc_thread_join(threads[i], NULL);
}
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
X(cconnects);
X(csends);
X(creads);
X(sreads);
X(ssends);
CHECK_RANGE_FULL(csends);
CHECK_RANGE_HALF(creads);
CHECK_RANGE_HALF(sreads);
CHECK_RANGE_HALF(ssends);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(udp_half_recv_send) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
isc_thread_t threads[workers];
SKIP_IN_CI;
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listenudp(listen_nm, ISC_NM_LISTEN_ALL,
&udp_listen_addr, listen_read_cb, NULL,
&listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_equal(result, ISC_R_SUCCESS);
memset(threads, 0, sizeof(threads));
for (size_t i = 0; i < workers; i++) {
isc_thread_create(connect_thread, udp_connect, &threads[i]);
}
WAIT_FOR_GE(cconnects, esends / 2);
WAIT_FOR_GE(csends, esends / 2);
WAIT_FOR_GE(sreads, esends / 2);
WAIT_FOR_GE(ssends, esends / 2);
WAIT_FOR_GE(creads, esends / 2);
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
/* Try to send a little while longer */
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
isc_test_nap((esends / 2) * 10);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
DONE();
for (size_t i = 0; i < workers; i++) {
isc_thread_join(threads[i], NULL);
}
X(cconnects);
X(csends);
X(creads);
X(sreads);
X(ssends);
CHECK_RANGE_FULL(csends);
CHECK_RANGE_HALF(creads);
CHECK_RANGE_HALF(sreads);
CHECK_RANGE_HALF(ssends);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(udp_half_recv_half_send) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
isc_thread_t threads[workers];
SKIP_IN_CI;
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listenudp(listen_nm, ISC_NM_LISTEN_ALL,
&udp_listen_addr, listen_read_cb, NULL,
&listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_equal(result, ISC_R_SUCCESS);
memset(threads, 0, sizeof(threads));
for (size_t i = 0; i < workers; i++) {
isc_thread_create(connect_thread, udp_connect, &threads[i]);
}
WAIT_FOR_GE(cconnects, esends / 2);
WAIT_FOR_GE(csends, esends / 2);
WAIT_FOR_GE(sreads, esends / 2);
WAIT_FOR_GE(ssends, esends / 2);
WAIT_FOR_GE(creads, esends / 2);
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
DONE();
for (size_t i = 0; i < workers; i++) {
isc_thread_join(threads[i], NULL);
}
X(cconnects);
X(csends);
X(creads);
X(sreads);
X(ssends);
CHECK_RANGE_FULL(csends);
CHECK_RANGE_HALF(creads);
CHECK_RANGE_HALF(sreads);
CHECK_RANGE_HALF(ssends);
}
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
/* Common stream protocols code */
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
static isc_quota_t *
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
tcp_listener_init_quota(size_t nthreads) {
isc_quota_t *quotap = NULL;
if (atomic_load(&check_listener_quota)) {
unsigned max_quota = ISC_MAX(nthreads / 2, 1);
isc_quota_max(&listener_quota, max_quota);
quotap = &listener_quota;
}
return (quotap);
}
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_nm_*connect() always return via callback The isc_nm_*connect() functions were refactored to always return the connection status via the connect callback instead of sometimes returning the hard failure directly (for example, when the socket could not be created, or when the network manager was shutting down). This commit changes the connect functions in all the network manager modules, and also makes the necessary refactoring changes in places where the connect functions are called.
2021-03-31 18:32:32 +02:00
static void
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
tcp_connect(isc_nm_t *nm) {
Refactor the interface handling in the netmgr The isc_nmiface_t type was holding just a single isc_sockaddr_t, so we got rid of the datatype and use plain isc_sockaddr_t in place where isc_nmiface_t was used before. This means less type-casting and shorter path to access isc_sockaddr_t members. At the same time, instead of keeping the reference to the isc_sockaddr_t that was passed to us when we start listening, we will keep a local copy. This prevents the data race on destruction of the ns_interface_t objects where pending nmsockets could reference the sockaddr of already destroyed ns_interface_t object.
2021-05-26 08:15:34 +02:00
isc_nm_tcpconnect(nm, &tcp_connect_addr, &tcp_listen_addr,
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
connect_connect_cb, NULL, T_CONNECT);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
}
Make the DNS over HTTPS support optional This commit adds two new autoconf options `--enable-doh` (enabled by default) and `--with-libnghttp2` (mandatory when DoH is enabled). When DoH support is disabled the library is not linked-in and support for http(s) protocol is disabled in the netmgr, named and dig.
2021-04-21 13:52:15 +02:00
#if HAVE_LIBNGHTTP2
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
static void
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
tls_connect(isc_nm_t *nm);
Make the DNS over HTTPS support optional This commit adds two new autoconf options `--enable-doh` (enabled by default) and `--with-libnghttp2` (mandatory when DoH is enabled). When DoH support is disabled the library is not linked-in and support for http(s) protocol is disabled in the netmgr, named and dig.
2021-04-21 13:52:15 +02:00
#endif
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
static stream_connect_function
get_stream_connect_function(void) {
Make the DNS over HTTPS support optional This commit adds two new autoconf options `--enable-doh` (enabled by default) and `--with-libnghttp2` (mandatory when DoH is enabled). When DoH support is disabled the library is not linked-in and support for http(s) protocol is disabled in the netmgr, named and dig.
2021-04-21 13:52:15 +02:00
#if HAVE_LIBNGHTTP2
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
if (stream_use_TLS) {
return (tls_connect);
}
Make the DNS over HTTPS support optional This commit adds two new autoconf options `--enable-doh` (enabled by default) and `--with-libnghttp2` (mandatory when DoH is enabled). When DoH support is disabled the library is not linked-in and support for http(s) protocol is disabled in the netmgr, named and dig.
2021-04-21 13:52:15 +02:00
#endif
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
return (tcp_connect);
}
static isc_result_t
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
stream_listen(isc_nm_accept_cb_t accept_cb, void *accept_cbarg, int backlog,
isc_quota_t *quota, isc_nmsocket_t **sockp) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
isc_result_t result = ISC_R_SUCCESS;
Make the DNS over HTTPS support optional This commit adds two new autoconf options `--enable-doh` (enabled by default) and `--with-libnghttp2` (mandatory when DoH is enabled). When DoH support is disabled the library is not linked-in and support for http(s) protocol is disabled in the netmgr, named and dig.
2021-04-21 13:52:15 +02:00
#if HAVE_LIBNGHTTP2
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
if (stream_use_TLS) {
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listentls(listen_nm, ISC_NM_LISTEN_ALL,
&tcp_listen_addr, accept_cb,
accept_cbarg, backlog, quota,
tcp_listen_tlsctx, sockp);
Make the DNS over HTTPS support optional This commit adds two new autoconf options `--enable-doh` (enabled by default) and `--with-libnghttp2` (mandatory when DoH is enabled). When DoH support is disabled the library is not linked-in and support for http(s) protocol is disabled in the netmgr, named and dig.
2021-04-21 13:52:15 +02:00
return (result);
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
}
Make the DNS over HTTPS support optional This commit adds two new autoconf options `--enable-doh` (enabled by default) and `--with-libnghttp2` (mandatory when DoH is enabled). When DoH support is disabled the library is not linked-in and support for http(s) protocol is disabled in the netmgr, named and dig.
2021-04-21 13:52:15 +02:00
#endif
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listentcp(listen_nm, ISC_NM_LISTEN_ALL,
&tcp_listen_addr, accept_cb, accept_cbarg,
backlog, quota, sockp);
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
return (result);
}
static void
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
stream_connect(isc_nm_cb_t cb, void *cbarg, unsigned int timeout) {
Make the DNS over HTTPS support optional This commit adds two new autoconf options `--enable-doh` (enabled by default) and `--with-libnghttp2` (mandatory when DoH is enabled). When DoH support is disabled the library is not linked-in and support for http(s) protocol is disabled in the netmgr, named and dig.
2021-04-21 13:52:15 +02:00
#if HAVE_LIBNGHTTP2
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
if (stream_use_TLS) {
Refactor the interface handling in the netmgr The isc_nmiface_t type was holding just a single isc_sockaddr_t, so we got rid of the datatype and use plain isc_sockaddr_t in place where isc_nmiface_t was used before. This means less type-casting and shorter path to access isc_sockaddr_t members. At the same time, instead of keeping the reference to the isc_sockaddr_t that was passed to us when we start listening, we will keep a local copy. This prevents the data race on destruction of the ns_interface_t objects where pending nmsockets could reference the sockaddr of already destroyed ns_interface_t object.
2021-05-26 08:15:34 +02:00
isc_nm_tlsconnect(connect_nm, &tcp_connect_addr,
&tcp_listen_addr, cb, cbarg,
TLS stream/DoH: implement TLS client session resumption This commit extends TLS stream code and DoH code with TLS client session resumption support implemented on top of the TLS client session cache.
2022-04-22 15:59:11 +03:00
tcp_connect_tlsctx,
tcp_tlsctx_client_sess_cache, timeout);
Make the DNS over HTTPS support optional This commit adds two new autoconf options `--enable-doh` (enabled by default) and `--with-libnghttp2` (mandatory when DoH is enabled). When DoH support is disabled the library is not linked-in and support for http(s) protocol is disabled in the netmgr, named and dig.
2021-04-21 13:52:15 +02:00
return;
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
}
Make the DNS over HTTPS support optional This commit adds two new autoconf options `--enable-doh` (enabled by default) and `--with-libnghttp2` (mandatory when DoH is enabled). When DoH support is disabled the library is not linked-in and support for http(s) protocol is disabled in the netmgr, named and dig.
2021-04-21 13:52:15 +02:00
#endif
isc_nm_tcpconnect(connect_nm, &tcp_connect_addr, &tcp_listen_addr, cb,
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
cbarg, timeout);
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
}
static void
stream_noop(void **state __attribute__((unused))) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
result = stream_listen(noop_accept_cb, NULL, 0, NULL, &listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_equal(result, ISC_R_SUCCESS);
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
allow client read callback to be assignable allow netmgr client tests to choose the function that will be used as a read callback, without having to write a different connect callback handler.
2021-04-20 17:13:15 -07:00
connect_readcb = NULL;
isc_nm_*connect() always return via callback The isc_nm_*connect() functions were refactored to always return the connection status via the connect callback instead of sometimes returning the hard failure directly (for example, when the socket could not be created, or when the network manager was shutting down). This commit changes the connect functions in all the network manager modules, and also makes the necessary refactoring changes in places where the connect functions are called.
2021-03-31 18:32:32 +02:00
isc_refcount_increment0(&active_cconnects);
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
stream_connect(connect_connect_cb, NULL, T_CONNECT);
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
atomic_assert_int_eq(cconnects, 0);
atomic_assert_int_eq(csends, 0);
atomic_assert_int_eq(creads, 0);
atomic_assert_int_eq(sreads, 0);
atomic_assert_int_eq(ssends, 0);
}
static void
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
stream_noresponse(void **state __attribute__((unused))) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
result = stream_listen(noop_accept_cb, NULL, 0, NULL, &listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_equal(result, ISC_R_SUCCESS);
isc_nm_*connect() always return via callback The isc_nm_*connect() functions were refactored to always return the connection status via the connect callback instead of sometimes returning the hard failure directly (for example, when the socket could not be created, or when the network manager was shutting down). This commit changes the connect functions in all the network manager modules, and also makes the necessary refactoring changes in places where the connect functions are called.
2021-03-31 18:32:32 +02:00
isc_refcount_increment0(&active_cconnects);
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
stream_connect(connect_connect_cb, NULL, T_CONNECT);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
WAIT_FOR_EQ(cconnects, 1);
WAIT_FOR_EQ(csends, 1);
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
X(cconnects);
X(csends);
X(creads);
X(sreads);
X(ssends);
atomic_assert_int_eq(cconnects, 1);
atomic_assert_int_eq(csends, 1);
atomic_assert_int_eq(creads, 0);
atomic_assert_int_eq(sreads, 0);
atomic_assert_int_eq(ssends, 0);
}
add TCP and TLS timeout recovery tests NOTE: currently these tests fail
2021-04-21 10:27:09 -07:00
static void
stream_timeout_recovery(void **state __attribute__((unused))) {
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
SKIP_IN_CI;
/*
* Accept connections but don't send responses, forcing client
* reads to time out.
*/
noanswer = true;
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
result = stream_listen(stream_accept_cb, NULL, 0, NULL, &listen_sock);
add TCP and TLS timeout recovery tests NOTE: currently these tests fail
2021-04-21 10:27:09 -07:00
assert_int_equal(result, ISC_R_SUCCESS);
/*
* Shorten all the client timeouts to 0.05 seconds.
*/
Use a constant for timeouts in soft-timeout tests It makes it easier to change the value should the need arise.
2021-04-22 13:29:34 +03:00
isc_nm_settimeouts(connect_nm, T_SOFT, T_SOFT, T_SOFT, T_SOFT);
add TCP and TLS timeout recovery tests NOTE: currently these tests fail
2021-04-21 10:27:09 -07:00
connect_readcb = timeout_retry_cb;
isc_refcount_increment0(&active_cconnects);
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
stream_connect(connect_connect_cb, NULL, T_SOFT);
add TCP and TLS timeout recovery tests NOTE: currently these tests fail
2021-04-21 10:27:09 -07:00
WAIT_FOR_EQ(cconnects, 1);
WAIT_FOR_GE(csends, 1);
WAIT_FOR_GE(csends, 2);
WAIT_FOR_GE(csends, 3);
WAIT_FOR_GE(csends, 4);
WAIT_FOR_EQ(csends, 5);
WAIT_FOR_EQ(ctimeouts, 1);
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
add TCP and TLS timeout recovery tests NOTE: currently these tests fail
2021-04-21 10:27:09 -07:00
}
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
static void
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
stream_recv_one(void **state __attribute__((unused))) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
isc_quota_t *quotap = tcp_listener_init_quota(1);
atomic_store(&nsends, 1);
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
result = stream_listen(stream_accept_cb, NULL, 0, quotap, &listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_equal(result, ISC_R_SUCCESS);
isc_nm_*connect() always return via callback The isc_nm_*connect() functions were refactored to always return the connection status via the connect callback instead of sometimes returning the hard failure directly (for example, when the socket could not be created, or when the network manager was shutting down). This commit changes the connect functions in all the network manager modules, and also makes the necessary refactoring changes in places where the connect functions are called.
2021-03-31 18:32:32 +02:00
isc_refcount_increment0(&active_cconnects);
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
stream_connect(connect_connect_cb, NULL, T_CONNECT);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
WAIT_FOR_EQ(cconnects, 1);
WAIT_FOR_LE(nsends, 0);
WAIT_FOR_EQ(csends, 1);
WAIT_FOR_EQ(sreads, 1);
WAIT_FOR_EQ(ssends, 0);
WAIT_FOR_EQ(creads, 0);
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
X(cconnects);
X(csends);
X(creads);
X(sreads);
X(ssends);
atomic_assert_int_eq(cconnects, 1);
atomic_assert_int_eq(csends, 1);
atomic_assert_int_eq(creads, 0);
atomic_assert_int_eq(sreads, 1);
atomic_assert_int_eq(ssends, 0);
}
static void
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
stream_recv_two(void **state __attribute__((unused))) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
isc_quota_t *quotap = tcp_listener_init_quota(1);
atomic_store(&nsends, 2);
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
result = stream_listen(stream_accept_cb, NULL, 0, quotap, &listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_equal(result, ISC_R_SUCCESS);
isc_nm_*connect() always return via callback The isc_nm_*connect() functions were refactored to always return the connection status via the connect callback instead of sometimes returning the hard failure directly (for example, when the socket could not be created, or when the network manager was shutting down). This commit changes the connect functions in all the network manager modules, and also makes the necessary refactoring changes in places where the connect functions are called.
2021-03-31 18:32:32 +02:00
isc_refcount_increment0(&active_cconnects);
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
stream_connect(connect_connect_cb, NULL, T_CONNECT);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
WAIT_FOR_EQ(cconnects, 1);
isc_nm_*connect() always return via callback The isc_nm_*connect() functions were refactored to always return the connection status via the connect callback instead of sometimes returning the hard failure directly (for example, when the socket could not be created, or when the network manager was shutting down). This commit changes the connect functions in all the network manager modules, and also makes the necessary refactoring changes in places where the connect functions are called.
2021-03-31 18:32:32 +02:00
isc_refcount_increment0(&active_cconnects);
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
stream_connect(connect_connect_cb, NULL, T_CONNECT);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
WAIT_FOR_EQ(cconnects, 2);
WAIT_FOR_LE(nsends, 0);
WAIT_FOR_EQ(csends, 2);
WAIT_FOR_EQ(sreads, 2);
WAIT_FOR_EQ(ssends, 1);
WAIT_FOR_EQ(creads, 1);
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
X(cconnects);
X(csends);
X(creads);
X(sreads);
X(ssends);
atomic_assert_int_eq(cconnects, 2);
atomic_assert_int_eq(csends, 2);
atomic_assert_int_eq(creads, 1);
atomic_assert_int_eq(sreads, 2);
atomic_assert_int_eq(ssends, 1);
}
static void
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
stream_recv_send(void **state __attribute__((unused))) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
isc_thread_t threads[workers];
isc_quota_t *quotap = tcp_listener_init_quota(workers);
SKIP_IN_CI;
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
result = stream_listen(stream_accept_cb, NULL, 0, quotap, &listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_equal(result, ISC_R_SUCCESS);
memset(threads, 0, sizeof(threads));
for (size_t i = 0; i < workers; i++) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
isc_thread_create(connect_thread, get_stream_connect_function(),
&threads[i]);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
}
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
if (allow_send_back) {
WAIT_FOR_GE(cconnects, 1);
} else {
WAIT_FOR_GE(cconnects, esends);
}
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
WAIT_FOR_GE(csends, esends);
WAIT_FOR_GE(sreads, esends);
WAIT_FOR_GE(ssends, esends / 2);
WAIT_FOR_GE(creads, esends / 2);
DONE();
for (size_t i = 0; i < workers; i++) {
isc_thread_join(threads[i], NULL);
}
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
X(cconnects);
X(csends);
X(creads);
X(sreads);
X(ssends);
CHECK_RANGE_FULL(csends);
CHECK_RANGE_FULL(creads);
CHECK_RANGE_FULL(sreads);
CHECK_RANGE_FULL(ssends);
}
static void
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
stream_recv_half_send(void **state __attribute__((unused))) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
isc_thread_t threads[workers];
isc_quota_t *quotap = tcp_listener_init_quota(workers);
SKIP_IN_CI;
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
result = stream_listen(stream_accept_cb, NULL, 0, quotap, &listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_equal(result, ISC_R_SUCCESS);
memset(threads, 0, sizeof(threads));
for (size_t i = 0; i < workers; i++) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
isc_thread_create(connect_thread, get_stream_connect_function(),
&threads[i]);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
}
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
if (allow_send_back) {
WAIT_FOR_GE(cconnects, 1);
} else {
WAIT_FOR_GE(cconnects, esends / 2);
}
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
WAIT_FOR_GE(csends, esends / 2);
WAIT_FOR_GE(sreads, esends / 2);
WAIT_FOR_GE(ssends, esends / 2);
WAIT_FOR_GE(creads, esends / 2);
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
DONE();
for (size_t i = 0; i < workers; i++) {
isc_thread_join(threads[i], NULL);
}
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
X(cconnects);
X(csends);
X(creads);
X(sreads);
X(ssends);
CHECK_RANGE_HALF(csends);
CHECK_RANGE_HALF(creads);
CHECK_RANGE_HALF(sreads);
CHECK_RANGE_HALF(ssends);
}
static void
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
stream_half_recv_send(void **state __attribute__((unused))) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
isc_thread_t threads[workers];
isc_quota_t *quotap = tcp_listener_init_quota(workers);
SKIP_IN_CI;
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
result = stream_listen(stream_accept_cb, NULL, 0, quotap, &listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_equal(result, ISC_R_SUCCESS);
memset(threads, 0, sizeof(threads));
for (size_t i = 0; i < workers; i++) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
isc_thread_create(connect_thread, get_stream_connect_function(),
&threads[i]);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
}
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
if (allow_send_back) {
WAIT_FOR_GE(cconnects, 1);
} else {
WAIT_FOR_GE(cconnects, esends / 2);
}
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
WAIT_FOR_GE(csends, esends / 2);
WAIT_FOR_GE(sreads, esends / 2);
WAIT_FOR_GE(ssends, esends / 2);
WAIT_FOR_GE(creads, esends / 2);
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
/* Try to send a little while longer */
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
isc_test_nap((esends / 2) * 10);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
DONE();
for (size_t i = 0; i < workers; i++) {
isc_thread_join(threads[i], NULL);
}
X(cconnects);
X(csends);
X(creads);
X(sreads);
X(ssends);
CHECK_RANGE_HALF(csends);
CHECK_RANGE_HALF(creads);
CHECK_RANGE_HALF(sreads);
CHECK_RANGE_HALF(ssends);
}
static void
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
stream_half_recv_half_send(void **state __attribute__((unused))) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
isc_thread_t threads[workers];
isc_quota_t *quotap = tcp_listener_init_quota(workers);
SKIP_IN_CI;
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
result = stream_listen(stream_accept_cb, NULL, 0, quotap, &listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_equal(result, ISC_R_SUCCESS);
memset(threads, 0, sizeof(threads));
for (size_t i = 0; i < workers; i++) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
isc_thread_create(connect_thread, get_stream_connect_function(),
&threads[i]);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
}
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
if (allow_send_back) {
WAIT_FOR_GE(cconnects, 1);
} else {
WAIT_FOR_GE(cconnects, esends / 2);
}
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
WAIT_FOR_GE(csends, esends / 2);
WAIT_FOR_GE(sreads, esends / 2);
WAIT_FOR_GE(ssends, esends / 2);
WAIT_FOR_GE(creads, esends / 2);
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
DONE();
for (size_t i = 0; i < workers; i++) {
isc_thread_join(threads[i], NULL);
}
X(cconnects);
X(csends);
X(creads);
X(sreads);
X(ssends);
CHECK_RANGE_HALF(csends);
CHECK_RANGE_HALF(creads);
CHECK_RANGE_HALF(sreads);
CHECK_RANGE_HALF(ssends);
}
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
/* TCP */
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tcp_noop) { stream_noop(state); }
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tcp_noresponse) { stream_noresponse(state); }
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tcp_timeout_recovery) { stream_timeout_recovery(state); }
add TCP and TLS timeout recovery tests NOTE: currently these tests fail
2021-04-21 10:27:09 -07:00
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tcp_recv_one) { stream_recv_one(state); }
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tcp_recv_two) { stream_recv_two(state); }
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tcp_recv_send) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
SKIP_IN_CI;
stream_recv_send(state);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tcp_recv_half_send) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
SKIP_IN_CI;
stream_recv_half_send(state);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tcp_half_recv_send) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
SKIP_IN_CI;
stream_half_recv_send(state);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tcp_half_recv_half_send) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
SKIP_IN_CI;
stream_half_recv_half_send(state);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tcp_recv_send_sendback) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
SKIP_IN_CI;
stream_recv_send(state);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tcp_recv_half_send_sendback) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
SKIP_IN_CI;
stream_recv_half_send(state);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tcp_half_recv_send_sendback) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
SKIP_IN_CI;
stream_half_recv_send(state);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tcp_half_recv_half_send_sendback) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
SKIP_IN_CI;
stream_half_recv_half_send(state);
}
/* TCP Quota */
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tcp_recv_one_quota) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
atomic_store(&check_listener_quota, true);
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
stream_recv_one(state);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tcp_recv_two_quota) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
atomic_store(&check_listener_quota, true);
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
stream_recv_two(state);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tcp_recv_send_quota) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
SKIP_IN_CI;
atomic_store(&check_listener_quota, true);
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
stream_recv_send(state);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tcp_recv_half_send_quota) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
SKIP_IN_CI;
atomic_store(&check_listener_quota, true);
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
stream_recv_half_send(state);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tcp_half_recv_send_quota) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
SKIP_IN_CI;
atomic_store(&check_listener_quota, true);
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
stream_half_recv_send(state);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tcp_half_recv_half_send_quota) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
SKIP_IN_CI;
atomic_store(&check_listener_quota, true);
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
stream_half_recv_half_send(state);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tcp_recv_send_quota_sendback) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
SKIP_IN_CI;
atomic_store(&check_listener_quota, true);
allow_send_back = true;
stream_recv_send(state);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tcp_recv_half_send_quota_sendback) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
SKIP_IN_CI;
atomic_store(&check_listener_quota, true);
allow_send_back = true;
stream_recv_half_send(state);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tcp_half_recv_send_quota_sendback) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
SKIP_IN_CI;
atomic_store(&check_listener_quota, true);
allow_send_back = true;
stream_half_recv_send(state);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tcp_half_recv_half_send_quota_sendback) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
SKIP_IN_CI;
atomic_store(&check_listener_quota, true);
allow_send_back = true;
stream_half_recv_half_send(state);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
}
/* TCPDNS */
isc_nm_*connect() always return via callback The isc_nm_*connect() functions were refactored to always return the connection status via the connect callback instead of sometimes returning the hard failure directly (for example, when the socket could not be created, or when the network manager was shutting down). This commit changes the connect functions in all the network manager modules, and also makes the necessary refactoring changes in places where the connect functions are called.
2021-03-31 18:32:32 +02:00
static void
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
tcpdns_connect(isc_nm_t *nm) {
Refactor the interface handling in the netmgr The isc_nmiface_t type was holding just a single isc_sockaddr_t, so we got rid of the datatype and use plain isc_sockaddr_t in place where isc_nmiface_t was used before. This means less type-casting and shorter path to access isc_sockaddr_t members. At the same time, instead of keeping the reference to the isc_sockaddr_t that was passed to us when we start listening, we will keep a local copy. This prevents the data race on destruction of the ns_interface_t objects where pending nmsockets could reference the sockaddr of already destroyed ns_interface_t object.
2021-05-26 08:15:34 +02:00
isc_nm_tcpdnsconnect(nm, &tcp_connect_addr, &tcp_listen_addr,
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
connect_connect_cb, NULL, T_CONNECT);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tcpdns_noop) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listentcpdns(
listen_nm, ISC_NM_LISTEN_ALL, &tcp_listen_addr, noop_recv_cb,
NULL, noop_accept_cb, NULL, 0, NULL, &listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_equal(result, ISC_R_SUCCESS);
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
allow client read callback to be assignable allow netmgr client tests to choose the function that will be used as a read callback, without having to write a different connect callback handler.
2021-04-20 17:13:15 -07:00
connect_readcb = NULL;
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_refcount_increment0(&active_cconnects);
Refactor the interface handling in the netmgr The isc_nmiface_t type was holding just a single isc_sockaddr_t, so we got rid of the datatype and use plain isc_sockaddr_t in place where isc_nmiface_t was used before. This means less type-casting and shorter path to access isc_sockaddr_t members. At the same time, instead of keeping the reference to the isc_sockaddr_t that was passed to us when we start listening, we will keep a local copy. This prevents the data race on destruction of the ns_interface_t objects where pending nmsockets could reference the sockaddr of already destroyed ns_interface_t object.
2021-05-26 08:15:34 +02:00
isc_nm_tcpdnsconnect(connect_nm, &tcp_connect_addr, &tcp_listen_addr,
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
connect_connect_cb, NULL, T_CONNECT);
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
atomic_assert_int_eq(cconnects, 0);
atomic_assert_int_eq(csends, 0);
atomic_assert_int_eq(creads, 0);
atomic_assert_int_eq(sreads, 0);
atomic_assert_int_eq(ssends, 0);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tcpdns_noresponse) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
isc_refcount_increment0(&active_cconnects);
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listentcpdns(
listen_nm, ISC_NM_LISTEN_ALL, &tcp_listen_addr, noop_recv_cb,
NULL, noop_accept_cb, NULL, 0, NULL, &listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
if (result != ISC_R_SUCCESS) {
isc_refcount_decrement(&active_cconnects);
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
isc_test_nap(1);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
}
assert_int_equal(result, ISC_R_SUCCESS);
Refactor the interface handling in the netmgr The isc_nmiface_t type was holding just a single isc_sockaddr_t, so we got rid of the datatype and use plain isc_sockaddr_t in place where isc_nmiface_t was used before. This means less type-casting and shorter path to access isc_sockaddr_t members. At the same time, instead of keeping the reference to the isc_sockaddr_t that was passed to us when we start listening, we will keep a local copy. This prevents the data race on destruction of the ns_interface_t objects where pending nmsockets could reference the sockaddr of already destroyed ns_interface_t object.
2021-05-26 08:15:34 +02:00
isc_nm_tcpdnsconnect(connect_nm, &tcp_connect_addr, &tcp_listen_addr,
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
connect_connect_cb, NULL, T_CONNECT);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
WAIT_FOR_EQ(cconnects, 1);
WAIT_FOR_EQ(csends, 1);
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
X(cconnects);
X(csends);
X(creads);
X(sreads);
X(ssends);
atomic_assert_int_eq(cconnects, 1);
atomic_assert_int_eq(csends, 1);
atomic_assert_int_eq(creads, 0);
atomic_assert_int_eq(sreads, 0);
atomic_assert_int_eq(ssends, 0);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tcpdns_timeout_recovery) {
add TCPDNS and TLSDNS timeout recovery tests this is similar in structure to the UDP timeout recovery test. this commit adds a new mechanism to the netmgr test allowing the listen socket to accept incoming TCP connections but never send a response. this forces the client to time out on read.
2021-04-21 01:09:30 -07:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
SKIP_IN_CI;
/*
* Accept connections but don't send responses, forcing client
* reads to time out.
*/
noanswer = true;
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listentcpdns(
listen_nm, ISC_NM_LISTEN_ALL, &tcp_listen_addr, listen_read_cb,
NULL, listen_accept_cb, NULL, 0, NULL, &listen_sock);
add TCPDNS and TLSDNS timeout recovery tests this is similar in structure to the UDP timeout recovery test. this commit adds a new mechanism to the netmgr test allowing the listen socket to accept incoming TCP connections but never send a response. this forces the client to time out on read.
2021-04-21 01:09:30 -07:00
assert_int_equal(result, ISC_R_SUCCESS);
/*
* Shorten all the TCP client timeouts to 0.05 seconds, connect,
* then sleep for at least a second for each 'tick'.
* timeout_retry_cb() will give up after five timeouts.
*/
connect_readcb = timeout_retry_cb;
Use a constant for timeouts in soft-timeout tests It makes it easier to change the value should the need arise.
2021-04-22 13:29:34 +03:00
isc_nm_settimeouts(connect_nm, T_SOFT, T_SOFT, T_SOFT, T_SOFT);
add TCPDNS and TLSDNS timeout recovery tests this is similar in structure to the UDP timeout recovery test. this commit adds a new mechanism to the netmgr test allowing the listen socket to accept incoming TCP connections but never send a response. this forces the client to time out on read.
2021-04-21 01:09:30 -07:00
isc_refcount_increment0(&active_cconnects);
Refactor the interface handling in the netmgr The isc_nmiface_t type was holding just a single isc_sockaddr_t, so we got rid of the datatype and use plain isc_sockaddr_t in place where isc_nmiface_t was used before. This means less type-casting and shorter path to access isc_sockaddr_t members. At the same time, instead of keeping the reference to the isc_sockaddr_t that was passed to us when we start listening, we will keep a local copy. This prevents the data race on destruction of the ns_interface_t objects where pending nmsockets could reference the sockaddr of already destroyed ns_interface_t object.
2021-05-26 08:15:34 +02:00
isc_nm_tcpdnsconnect(connect_nm, &tcp_connect_addr, &tcp_listen_addr,
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
connect_connect_cb, NULL, T_SOFT);
add TCPDNS and TLSDNS timeout recovery tests this is similar in structure to the UDP timeout recovery test. this commit adds a new mechanism to the netmgr test allowing the listen socket to accept incoming TCP connections but never send a response. this forces the client to time out on read.
2021-04-21 01:09:30 -07:00
WAIT_FOR_EQ(cconnects, 1);
WAIT_FOR_GE(csends, 1);
WAIT_FOR_GE(csends, 2);
WAIT_FOR_GE(csends, 3);
WAIT_FOR_GE(csends, 4);
WAIT_FOR_EQ(csends, 5);
WAIT_FOR_EQ(ctimeouts, 1);
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
add TCPDNS and TLSDNS timeout recovery tests this is similar in structure to the UDP timeout recovery test. this commit adds a new mechanism to the netmgr test allowing the listen socket to accept incoming TCP connections but never send a response. this forces the client to time out on read.
2021-04-21 01:09:30 -07:00
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tcpdns_recv_one) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
atomic_store(&nsends, 1);
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listentcpdns(
listen_nm, ISC_NM_LISTEN_ALL, &tcp_listen_addr, listen_read_cb,
NULL, listen_accept_cb, NULL, 0, NULL, &listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_equal(result, ISC_R_SUCCESS);
isc_refcount_increment0(&active_cconnects);
Refactor the interface handling in the netmgr The isc_nmiface_t type was holding just a single isc_sockaddr_t, so we got rid of the datatype and use plain isc_sockaddr_t in place where isc_nmiface_t was used before. This means less type-casting and shorter path to access isc_sockaddr_t members. At the same time, instead of keeping the reference to the isc_sockaddr_t that was passed to us when we start listening, we will keep a local copy. This prevents the data race on destruction of the ns_interface_t objects where pending nmsockets could reference the sockaddr of already destroyed ns_interface_t object.
2021-05-26 08:15:34 +02:00
isc_nm_tcpdnsconnect(connect_nm, &tcp_connect_addr, &tcp_listen_addr,
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
connect_connect_cb, NULL, T_CONNECT);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
WAIT_FOR_EQ(cconnects, 1);
WAIT_FOR_LE(nsends, 0);
WAIT_FOR_EQ(csends, 1);
WAIT_FOR_EQ(sreads, 1);
WAIT_FOR_EQ(ssends, 0);
WAIT_FOR_EQ(creads, 0);
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
X(cconnects);
X(csends);
X(creads);
X(sreads);
X(ssends);
atomic_assert_int_eq(cconnects, 1);
atomic_assert_int_eq(csends, 1);
atomic_assert_int_eq(creads, 0);
atomic_assert_int_eq(sreads, 1);
atomic_assert_int_eq(ssends, 0);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tcpdns_recv_two) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
atomic_store(&nsends, 2);
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listentcpdns(
listen_nm, ISC_NM_LISTEN_ALL, &tcp_listen_addr, listen_read_cb,
NULL, listen_accept_cb, NULL, 0, NULL, &listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_equal(result, ISC_R_SUCCESS);
isc_nm_*connect() always return via callback The isc_nm_*connect() functions were refactored to always return the connection status via the connect callback instead of sometimes returning the hard failure directly (for example, when the socket could not be created, or when the network manager was shutting down). This commit changes the connect functions in all the network manager modules, and also makes the necessary refactoring changes in places where the connect functions are called.
2021-03-31 18:32:32 +02:00
isc_refcount_increment0(&active_cconnects);
Refactor the interface handling in the netmgr The isc_nmiface_t type was holding just a single isc_sockaddr_t, so we got rid of the datatype and use plain isc_sockaddr_t in place where isc_nmiface_t was used before. This means less type-casting and shorter path to access isc_sockaddr_t members. At the same time, instead of keeping the reference to the isc_sockaddr_t that was passed to us when we start listening, we will keep a local copy. This prevents the data race on destruction of the ns_interface_t objects where pending nmsockets could reference the sockaddr of already destroyed ns_interface_t object.
2021-05-26 08:15:34 +02:00
isc_nm_tcpdnsconnect(connect_nm, &tcp_connect_addr, &tcp_listen_addr,
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
connect_connect_cb, NULL, T_CONNECT);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
WAIT_FOR_EQ(cconnects, 1);
isc_nm_*connect() always return via callback The isc_nm_*connect() functions were refactored to always return the connection status via the connect callback instead of sometimes returning the hard failure directly (for example, when the socket could not be created, or when the network manager was shutting down). This commit changes the connect functions in all the network manager modules, and also makes the necessary refactoring changes in places where the connect functions are called.
2021-03-31 18:32:32 +02:00
isc_refcount_increment0(&active_cconnects);
Refactor the interface handling in the netmgr The isc_nmiface_t type was holding just a single isc_sockaddr_t, so we got rid of the datatype and use plain isc_sockaddr_t in place where isc_nmiface_t was used before. This means less type-casting and shorter path to access isc_sockaddr_t members. At the same time, instead of keeping the reference to the isc_sockaddr_t that was passed to us when we start listening, we will keep a local copy. This prevents the data race on destruction of the ns_interface_t objects where pending nmsockets could reference the sockaddr of already destroyed ns_interface_t object.
2021-05-26 08:15:34 +02:00
isc_nm_tcpdnsconnect(connect_nm, &tcp_connect_addr, &tcp_listen_addr,
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
connect_connect_cb, NULL, T_CONNECT);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
WAIT_FOR_EQ(cconnects, 2);
WAIT_FOR_LE(nsends, 0);
WAIT_FOR_EQ(csends, 2);
WAIT_FOR_EQ(sreads, 2);
WAIT_FOR_EQ(ssends, 1);
WAIT_FOR_EQ(creads, 1);
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
X(cconnects);
X(csends);
X(creads);
X(sreads);
X(ssends);
atomic_assert_int_eq(cconnects, 2);
atomic_assert_int_eq(csends, 2);
atomic_assert_int_eq(creads, 1);
atomic_assert_int_eq(sreads, 2);
atomic_assert_int_eq(ssends, 1);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tcpdns_recv_send) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
isc_thread_t threads[workers];
SKIP_IN_CI;
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listentcpdns(
listen_nm, ISC_NM_LISTEN_ALL, &tcp_listen_addr, listen_read_cb,
NULL, listen_accept_cb, NULL, 0, NULL, &listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_equal(result, ISC_R_SUCCESS);
memset(threads, 0, sizeof(threads));
for (size_t i = 0; i < workers; i++) {
isc_thread_create(connect_thread, tcpdns_connect, &threads[i]);
}
WAIT_FOR_GE(cconnects, esends);
WAIT_FOR_GE(csends, esends);
WAIT_FOR_GE(sreads, esends);
WAIT_FOR_GE(ssends, esends / 2);
WAIT_FOR_GE(creads, esends / 2);
DONE();
for (size_t i = 0; i < workers; i++) {
isc_thread_join(threads[i], NULL);
}
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
X(cconnects);
X(csends);
X(creads);
X(sreads);
X(ssends);
CHECK_RANGE_FULL(csends);
CHECK_RANGE_FULL(creads);
CHECK_RANGE_FULL(sreads);
CHECK_RANGE_FULL(ssends);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tcpdns_recv_half_send) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
isc_thread_t threads[workers];
SKIP_IN_CI;
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listentcpdns(
listen_nm, ISC_NM_LISTEN_ALL, &tcp_listen_addr, listen_read_cb,
NULL, listen_accept_cb, NULL, 0, NULL, &listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_equal(result, ISC_R_SUCCESS);
memset(threads, 0, sizeof(threads));
for (size_t i = 0; i < workers; i++) {
isc_thread_create(connect_thread, tcpdns_connect, &threads[i]);
}
WAIT_FOR_GE(cconnects, esends / 2);
WAIT_FOR_GE(csends, esends / 2);
WAIT_FOR_GE(sreads, esends / 2);
WAIT_FOR_GE(ssends, esends / 2);
WAIT_FOR_GE(creads, esends / 2);
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
DONE();
for (size_t i = 0; i < workers; i++) {
isc_thread_join(threads[i], NULL);
}
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
X(cconnects);
X(csends);
X(creads);
X(sreads);
X(ssends);
CHECK_RANGE_HALF(csends);
CHECK_RANGE_HALF(creads);
CHECK_RANGE_HALF(sreads);
CHECK_RANGE_HALF(ssends);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tcpdns_half_recv_send) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
isc_thread_t threads[workers];
SKIP_IN_CI;
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listentcpdns(
listen_nm, ISC_NM_LISTEN_ALL, &tcp_listen_addr, listen_read_cb,
NULL, listen_accept_cb, NULL, 0, NULL, &listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_equal(result, ISC_R_SUCCESS);
memset(threads, 0, sizeof(threads));
for (size_t i = 0; i < workers; i++) {
isc_thread_create(connect_thread, tcpdns_connect, &threads[i]);
}
WAIT_FOR_GE(cconnects, esends / 2);
WAIT_FOR_GE(csends, esends / 2);
WAIT_FOR_GE(sreads, esends / 2);
WAIT_FOR_GE(ssends, esends / 2);
WAIT_FOR_GE(creads, esends / 2);
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
/* Try to send a little while longer */
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
isc_test_nap((esends / 2) * 10);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
DONE();
for (size_t i = 0; i < workers; i++) {
isc_thread_join(threads[i], NULL);
}
X(cconnects);
X(csends);
X(creads);
X(sreads);
X(ssends);
CHECK_RANGE_HALF(csends);
CHECK_RANGE_HALF(creads);
CHECK_RANGE_HALF(sreads);
CHECK_RANGE_HALF(ssends);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tcpdns_half_recv_half_send) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
isc_thread_t threads[workers];
SKIP_IN_CI;
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listentcpdns(
listen_nm, ISC_NM_LISTEN_ALL, &tcp_listen_addr, listen_read_cb,
NULL, listen_accept_cb, NULL, 0, NULL, &listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_equal(result, ISC_R_SUCCESS);
memset(threads, 0, sizeof(threads));
for (size_t i = 0; i < workers; i++) {
isc_thread_create(connect_thread, tcpdns_connect, &threads[i]);
}
WAIT_FOR_GE(cconnects, esends / 2);
WAIT_FOR_GE(csends, esends / 2);
WAIT_FOR_GE(sreads, esends / 2);
WAIT_FOR_GE(ssends, esends / 2);
WAIT_FOR_GE(creads, esends / 2);
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
DONE();
for (size_t i = 0; i < workers; i++) {
isc_thread_join(threads[i], NULL);
}
X(cconnects);
X(csends);
X(creads);
X(sreads);
X(ssends);
CHECK_RANGE_HALF(csends);
CHECK_RANGE_HALF(creads);
CHECK_RANGE_HALF(sreads);
CHECK_RANGE_HALF(ssends);
}
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
/* TLS */
Make the DNS over HTTPS support optional This commit adds two new autoconf options `--enable-doh` (enabled by default) and `--with-libnghttp2` (mandatory when DoH is enabled). When DoH support is disabled the library is not linked-in and support for http(s) protocol is disabled in the netmgr, named and dig.
2021-04-21 13:52:15 +02:00
#if HAVE_LIBNGHTTP2
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
static void
tls_connect(isc_nm_t *nm) {
Refactor the interface handling in the netmgr The isc_nmiface_t type was holding just a single isc_sockaddr_t, so we got rid of the datatype and use plain isc_sockaddr_t in place where isc_nmiface_t was used before. This means less type-casting and shorter path to access isc_sockaddr_t members. At the same time, instead of keeping the reference to the isc_sockaddr_t that was passed to us when we start listening, we will keep a local copy. This prevents the data race on destruction of the ns_interface_t objects where pending nmsockets could reference the sockaddr of already destroyed ns_interface_t object.
2021-05-26 08:15:34 +02:00
isc_nm_tlsconnect(nm, &tcp_connect_addr, &tcp_listen_addr,
connect_connect_cb, NULL, tcp_connect_tlsctx,
TLS stream/DoH: implement TLS client session resumption This commit extends TLS stream code and DoH code with TLS client session resumption support implemented on top of the TLS client session cache.
2022-04-22 15:59:11 +03:00
tcp_tlsctx_client_sess_cache, T_CONNECT);
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tls_noop) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
stream_use_TLS = true;
stream_noop(state);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tls_noresponse) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
stream_use_TLS = true;
stream_noresponse(state);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tls_timeout_recovery) {
add TCP and TLS timeout recovery tests NOTE: currently these tests fail
2021-04-21 10:27:09 -07:00
stream_use_TLS = true;
stream_timeout_recovery(state);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tls_recv_one) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
stream_use_TLS = true;
stream_recv_one(state);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tls_recv_two) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
stream_use_TLS = true;
stream_recv_two(state);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tls_recv_send) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
SKIP_IN_CI;
stream_use_TLS = true;
stream_recv_send(state);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tls_recv_half_send) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
SKIP_IN_CI;
stream_use_TLS = true;
stream_recv_half_send(state);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tls_half_recv_send) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
SKIP_IN_CI;
stream_use_TLS = true;
stream_half_recv_send(state);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tls_half_recv_half_send) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
SKIP_IN_CI;
stream_use_TLS = true;
stream_half_recv_half_send(state);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tls_recv_send_sendback) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
SKIP_IN_CI;
stream_use_TLS = true;
allow_send_back = true;
stream_recv_send(state);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tls_recv_half_send_sendback) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
SKIP_IN_CI;
stream_use_TLS = true;
allow_send_back = true;
stream_recv_half_send(state);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tls_half_recv_send_sendback) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
SKIP_IN_CI;
stream_use_TLS = true;
allow_send_back = true;
stream_half_recv_send(state);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tls_half_recv_half_send_sendback) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
SKIP_IN_CI;
stream_use_TLS = true;
allow_send_back = true;
stream_half_recv_half_send(state);
}
/* TLS quota */
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tls_recv_one_quota) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
stream_use_TLS = true;
atomic_store(&check_listener_quota, true);
stream_recv_one(state);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tls_recv_two_quota) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
stream_use_TLS = true;
atomic_store(&check_listener_quota, true);
stream_recv_two(state);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tls_recv_send_quota) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
SKIP_IN_CI;
stream_use_TLS = true;
atomic_store(&check_listener_quota, true);
stream_recv_send(state);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tls_recv_half_send_quota) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
SKIP_IN_CI;
stream_use_TLS = true;
atomic_store(&check_listener_quota, true);
stream_recv_half_send(state);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tls_half_recv_send_quota) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
SKIP_IN_CI;
stream_use_TLS = true;
Make some TLS tests actually use quota A directive to check quota was missing from some of the TLS tests which were supposed to test TLS code with quotas.
2021-04-30 16:06:09 +03:00
atomic_store(&check_listener_quota, true);
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
stream_half_recv_send(state);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tls_half_recv_half_send_quota) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
SKIP_IN_CI;
stream_use_TLS = true;
Make some TLS tests actually use quota A directive to check quota was missing from some of the TLS tests which were supposed to test TLS code with quotas.
2021-04-30 16:06:09 +03:00
atomic_store(&check_listener_quota, true);
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
stream_half_recv_half_send(state);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tls_recv_send_quota_sendback) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
SKIP_IN_CI;
stream_use_TLS = true;
allow_send_back = true;
atomic_store(&check_listener_quota, true);
stream_recv_send(state);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tls_recv_half_send_quota_sendback) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
SKIP_IN_CI;
stream_use_TLS = true;
allow_send_back = true;
atomic_store(&check_listener_quota, true);
stream_recv_half_send(state);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tls_half_recv_send_quota_sendback) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
SKIP_IN_CI;
stream_use_TLS = true;
allow_send_back = true;
atomic_store(&check_listener_quota, true);
stream_half_recv_send(state);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tls_half_recv_half_send_quota_sendback) {
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
SKIP_IN_CI;
stream_use_TLS = true;
allow_send_back = true;
atomic_store(&check_listener_quota, true);
stream_half_recv_half_send(state);
}
Make the DNS over HTTPS support optional This commit adds two new autoconf options `--enable-doh` (enabled by default) and `--with-libnghttp2` (mandatory when DoH is enabled). When DoH support is disabled the library is not linked-in and support for http(s) protocol is disabled in the netmgr, named and dig.
2021-04-21 13:52:15 +02:00
#endif
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
/* TLSDNS */
isc_nm_*connect() always return via callback The isc_nm_*connect() functions were refactored to always return the connection status via the connect callback instead of sometimes returning the hard failure directly (for example, when the socket could not be created, or when the network manager was shutting down). This commit changes the connect functions in all the network manager modules, and also makes the necessary refactoring changes in places where the connect functions are called.
2021-03-31 18:32:32 +02:00
static void
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
tlsdns_connect(isc_nm_t *nm) {
Refactor the interface handling in the netmgr The isc_nmiface_t type was holding just a single isc_sockaddr_t, so we got rid of the datatype and use plain isc_sockaddr_t in place where isc_nmiface_t was used before. This means less type-casting and shorter path to access isc_sockaddr_t members. At the same time, instead of keeping the reference to the isc_sockaddr_t that was passed to us when we start listening, we will keep a local copy. This prevents the data race on destruction of the ns_interface_t objects where pending nmsockets could reference the sockaddr of already destroyed ns_interface_t object.
2021-05-26 08:15:34 +02:00
isc_nm_tlsdnsconnect(nm, &tcp_connect_addr, &tcp_listen_addr,
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
connect_connect_cb, NULL, T_CONNECT,
DoT: implement TLS client session resumption This commit extends DoT code with TLS client session resumption support implemented on top of the TLS client session cache.
2022-04-25 16:47:06 +03:00
tcp_connect_tlsctx, tcp_tlsctx_client_sess_cache);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tlsdns_noop) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listentlsdns(listen_nm, ISC_NM_LISTEN_ALL,
&tcp_listen_addr, noop_recv_cb, NULL,
noop_accept_cb, NULL, 0, NULL,
Refactor the interface handling in the netmgr The isc_nmiface_t type was holding just a single isc_sockaddr_t, so we got rid of the datatype and use plain isc_sockaddr_t in place where isc_nmiface_t was used before. This means less type-casting and shorter path to access isc_sockaddr_t members. At the same time, instead of keeping the reference to the isc_sockaddr_t that was passed to us when we start listening, we will keep a local copy. This prevents the data race on destruction of the ns_interface_t objects where pending nmsockets could reference the sockaddr of already destroyed ns_interface_t object.
2021-05-26 08:15:34 +02:00
tcp_listen_tlsctx, &listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_equal(result, ISC_R_SUCCESS);
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
allow client read callback to be assignable allow netmgr client tests to choose the function that will be used as a read callback, without having to write a different connect callback handler.
2021-04-20 17:13:15 -07:00
connect_readcb = NULL;
isc_nm_*connect() always return via callback The isc_nm_*connect() functions were refactored to always return the connection status via the connect callback instead of sometimes returning the hard failure directly (for example, when the socket could not be created, or when the network manager was shutting down). This commit changes the connect functions in all the network manager modules, and also makes the necessary refactoring changes in places where the connect functions are called.
2021-03-31 18:32:32 +02:00
isc_refcount_increment0(&active_cconnects);
Refactor the interface handling in the netmgr The isc_nmiface_t type was holding just a single isc_sockaddr_t, so we got rid of the datatype and use plain isc_sockaddr_t in place where isc_nmiface_t was used before. This means less type-casting and shorter path to access isc_sockaddr_t members. At the same time, instead of keeping the reference to the isc_sockaddr_t that was passed to us when we start listening, we will keep a local copy. This prevents the data race on destruction of the ns_interface_t objects where pending nmsockets could reference the sockaddr of already destroyed ns_interface_t object.
2021-05-26 08:15:34 +02:00
isc_nm_tlsdnsconnect(connect_nm, &tcp_connect_addr, &tcp_listen_addr,
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
connect_connect_cb, NULL, T_CONNECT,
DoT: implement TLS client session resumption This commit extends DoT code with TLS client session resumption support implemented on top of the TLS client session cache.
2022-04-25 16:47:06 +03:00
tcp_connect_tlsctx, tcp_tlsctx_client_sess_cache);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
atomic_assert_int_eq(cconnects, 0);
atomic_assert_int_eq(csends, 0);
atomic_assert_int_eq(creads, 0);
atomic_assert_int_eq(sreads, 0);
atomic_assert_int_eq(ssends, 0);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tlsdns_noresponse) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
isc_sockaddr_t connect_addr;
connect_addr = (isc_sockaddr_t){ .length = 0 };
isc_sockaddr_fromin6(&connect_addr, &in6addr_loopback, 0);
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listentlsdns(listen_nm, ISC_NM_LISTEN_ALL,
&tcp_listen_addr, noop_recv_cb, NULL,
noop_accept_cb, NULL, 0, NULL,
Refactor the interface handling in the netmgr The isc_nmiface_t type was holding just a single isc_sockaddr_t, so we got rid of the datatype and use plain isc_sockaddr_t in place where isc_nmiface_t was used before. This means less type-casting and shorter path to access isc_sockaddr_t members. At the same time, instead of keeping the reference to the isc_sockaddr_t that was passed to us when we start listening, we will keep a local copy. This prevents the data race on destruction of the ns_interface_t objects where pending nmsockets could reference the sockaddr of already destroyed ns_interface_t object.
2021-05-26 08:15:34 +02:00
tcp_listen_tlsctx, &listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_equal(result, ISC_R_SUCCESS);
isc_nm_*connect() always return via callback The isc_nm_*connect() functions were refactored to always return the connection status via the connect callback instead of sometimes returning the hard failure directly (for example, when the socket could not be created, or when the network manager was shutting down). This commit changes the connect functions in all the network manager modules, and also makes the necessary refactoring changes in places where the connect functions are called.
2021-03-31 18:32:32 +02:00
isc_refcount_increment0(&active_cconnects);
Refactor the interface handling in the netmgr The isc_nmiface_t type was holding just a single isc_sockaddr_t, so we got rid of the datatype and use plain isc_sockaddr_t in place where isc_nmiface_t was used before. This means less type-casting and shorter path to access isc_sockaddr_t members. At the same time, instead of keeping the reference to the isc_sockaddr_t that was passed to us when we start listening, we will keep a local copy. This prevents the data race on destruction of the ns_interface_t objects where pending nmsockets could reference the sockaddr of already destroyed ns_interface_t object.
2021-05-26 08:15:34 +02:00
isc_nm_tlsdnsconnect(connect_nm, &connect_addr, &tcp_listen_addr,
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
connect_connect_cb, NULL, T_CONNECT,
DoT: implement TLS client session resumption This commit extends DoT code with TLS client session resumption support implemented on top of the TLS client session cache.
2022-04-25 16:47:06 +03:00
tcp_connect_tlsctx, tcp_tlsctx_client_sess_cache);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
WAIT_FOR_EQ(cconnects, 1);
WAIT_FOR_EQ(csends, 1);
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
X(cconnects);
X(csends);
X(creads);
X(sreads);
X(ssends);
atomic_assert_int_eq(cconnects, 1);
atomic_assert_int_eq(csends, 1);
atomic_assert_int_eq(creads, 0);
atomic_assert_int_eq(sreads, 0);
atomic_assert_int_eq(ssends, 0);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tlsdns_timeout_recovery) {
add TCPDNS and TLSDNS timeout recovery tests this is similar in structure to the UDP timeout recovery test. this commit adds a new mechanism to the netmgr test allowing the listen socket to accept incoming TCP connections but never send a response. this forces the client to time out on read.
2021-04-21 01:09:30 -07:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
isc_sockaddr_t connect_addr;
SKIP_IN_CI;
connect_addr = (isc_sockaddr_t){ .length = 0 };
isc_sockaddr_fromin6(&connect_addr, &in6addr_loopback, 0);
/*
* Accept connections but don't send responses, forcing client
* reads to time out.
*/
noanswer = true;
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listentlsdns(listen_nm, ISC_NM_LISTEN_ALL,
&tcp_listen_addr, listen_read_cb, NULL,
listen_accept_cb, NULL, 0, NULL,
tcp_listen_tlsctx, &listen_sock);
add TCPDNS and TLSDNS timeout recovery tests this is similar in structure to the UDP timeout recovery test. this commit adds a new mechanism to the netmgr test allowing the listen socket to accept incoming TCP connections but never send a response. this forces the client to time out on read.
2021-04-21 01:09:30 -07:00
assert_int_equal(result, ISC_R_SUCCESS);
/*
* Shorten all the TCP client timeouts to 0.05 seconds, connect,
* then sleep for at least a second for each 'tick'.
* timeout_retry_cb() will give up after five timeouts.
*/
connect_readcb = timeout_retry_cb;
Use a constant for timeouts in soft-timeout tests It makes it easier to change the value should the need arise.
2021-04-22 13:29:34 +03:00
isc_nm_settimeouts(connect_nm, T_SOFT, T_SOFT, T_SOFT, T_SOFT);
add TCPDNS and TLSDNS timeout recovery tests this is similar in structure to the UDP timeout recovery test. this commit adds a new mechanism to the netmgr test allowing the listen socket to accept incoming TCP connections but never send a response. this forces the client to time out on read.
2021-04-21 01:09:30 -07:00
isc_refcount_increment0(&active_cconnects);
Refactor the interface handling in the netmgr The isc_nmiface_t type was holding just a single isc_sockaddr_t, so we got rid of the datatype and use plain isc_sockaddr_t in place where isc_nmiface_t was used before. This means less type-casting and shorter path to access isc_sockaddr_t members. At the same time, instead of keeping the reference to the isc_sockaddr_t that was passed to us when we start listening, we will keep a local copy. This prevents the data race on destruction of the ns_interface_t objects where pending nmsockets could reference the sockaddr of already destroyed ns_interface_t object.
2021-05-26 08:15:34 +02:00
isc_nm_tlsdnsconnect(connect_nm, &tcp_connect_addr, &tcp_listen_addr,
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
connect_connect_cb, NULL, T_SOFT,
DoT: implement TLS client session resumption This commit extends DoT code with TLS client session resumption support implemented on top of the TLS client session cache.
2022-04-25 16:47:06 +03:00
tcp_connect_tlsctx, tcp_tlsctx_client_sess_cache);
add TCPDNS and TLSDNS timeout recovery tests this is similar in structure to the UDP timeout recovery test. this commit adds a new mechanism to the netmgr test allowing the listen socket to accept incoming TCP connections but never send a response. this forces the client to time out on read.
2021-04-21 01:09:30 -07:00
WAIT_FOR_EQ(cconnects, 1);
WAIT_FOR_GE(csends, 1);
WAIT_FOR_GE(csends, 2);
WAIT_FOR_GE(csends, 3);
WAIT_FOR_GE(csends, 4);
WAIT_FOR_EQ(csends, 5);
WAIT_FOR_EQ(ctimeouts, 1);
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
add TCPDNS and TLSDNS timeout recovery tests this is similar in structure to the UDP timeout recovery test. this commit adds a new mechanism to the netmgr test allowing the listen socket to accept incoming TCP connections but never send a response. this forces the client to time out on read.
2021-04-21 01:09:30 -07:00
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tlsdns_recv_one) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
atomic_store(&nsends, 1);
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listentlsdns(listen_nm, ISC_NM_LISTEN_ALL,
&tcp_listen_addr, listen_read_cb, NULL,
listen_accept_cb, NULL, 0, NULL,
tcp_listen_tlsctx, &listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_equal(result, ISC_R_SUCCESS);
isc_refcount_increment0(&active_cconnects);
Refactor the interface handling in the netmgr The isc_nmiface_t type was holding just a single isc_sockaddr_t, so we got rid of the datatype and use plain isc_sockaddr_t in place where isc_nmiface_t was used before. This means less type-casting and shorter path to access isc_sockaddr_t members. At the same time, instead of keeping the reference to the isc_sockaddr_t that was passed to us when we start listening, we will keep a local copy. This prevents the data race on destruction of the ns_interface_t objects where pending nmsockets could reference the sockaddr of already destroyed ns_interface_t object.
2021-05-26 08:15:34 +02:00
isc_nm_tlsdnsconnect(connect_nm, &tcp_connect_addr, &tcp_listen_addr,
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
connect_connect_cb, NULL, T_CONNECT,
DoT: implement TLS client session resumption This commit extends DoT code with TLS client session resumption support implemented on top of the TLS client session cache.
2022-04-25 16:47:06 +03:00
tcp_connect_tlsctx, tcp_tlsctx_client_sess_cache);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
WAIT_FOR_EQ(cconnects, 1);
WAIT_FOR_LE(nsends, 0);
WAIT_FOR_EQ(csends, 1);
WAIT_FOR_EQ(sreads, 1);
WAIT_FOR_EQ(ssends, 0);
WAIT_FOR_EQ(creads, 0);
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
X(cconnects);
X(csends);
X(creads);
X(sreads);
X(ssends);
atomic_assert_int_eq(cconnects, 1);
atomic_assert_int_eq(csends, 1);
atomic_assert_int_eq(creads, 0);
atomic_assert_int_eq(sreads, 1);
atomic_assert_int_eq(ssends, 0);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tlsdns_recv_two) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
atomic_store(&nsends, 2);
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listentlsdns(listen_nm, ISC_NM_LISTEN_ALL,
&tcp_listen_addr, listen_read_cb, NULL,
listen_accept_cb, NULL, 0, NULL,
tcp_listen_tlsctx, &listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_equal(result, ISC_R_SUCCESS);
isc_refcount_increment0(&active_cconnects);
Refactor the interface handling in the netmgr The isc_nmiface_t type was holding just a single isc_sockaddr_t, so we got rid of the datatype and use plain isc_sockaddr_t in place where isc_nmiface_t was used before. This means less type-casting and shorter path to access isc_sockaddr_t members. At the same time, instead of keeping the reference to the isc_sockaddr_t that was passed to us when we start listening, we will keep a local copy. This prevents the data race on destruction of the ns_interface_t objects where pending nmsockets could reference the sockaddr of already destroyed ns_interface_t object.
2021-05-26 08:15:34 +02:00
isc_nm_tlsdnsconnect(connect_nm, &tcp_connect_addr, &tcp_listen_addr,
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
connect_connect_cb, NULL, T_CONNECT,
DoT: implement TLS client session resumption This commit extends DoT code with TLS client session resumption support implemented on top of the TLS client session cache.
2022-04-25 16:47:06 +03:00
tcp_connect_tlsctx, tcp_tlsctx_client_sess_cache);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
WAIT_FOR_EQ(cconnects, 1);
isc_refcount_increment0(&active_cconnects);
Refactor the interface handling in the netmgr The isc_nmiface_t type was holding just a single isc_sockaddr_t, so we got rid of the datatype and use plain isc_sockaddr_t in place where isc_nmiface_t was used before. This means less type-casting and shorter path to access isc_sockaddr_t members. At the same time, instead of keeping the reference to the isc_sockaddr_t that was passed to us when we start listening, we will keep a local copy. This prevents the data race on destruction of the ns_interface_t objects where pending nmsockets could reference the sockaddr of already destroyed ns_interface_t object.
2021-05-26 08:15:34 +02:00
isc_nm_tlsdnsconnect(connect_nm, &tcp_connect_addr, &tcp_listen_addr,
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
connect_connect_cb, NULL, T_CONNECT,
DoT: implement TLS client session resumption This commit extends DoT code with TLS client session resumption support implemented on top of the TLS client session cache.
2022-04-25 16:47:06 +03:00
tcp_connect_tlsctx, tcp_tlsctx_client_sess_cache);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
WAIT_FOR_EQ(cconnects, 2);
WAIT_FOR_LE(nsends, 0);
WAIT_FOR_EQ(csends, 2);
WAIT_FOR_EQ(sreads, 2);
WAIT_FOR_EQ(ssends, 1);
WAIT_FOR_EQ(creads, 1);
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
X(cconnects);
X(csends);
X(creads);
X(sreads);
X(ssends);
atomic_assert_int_eq(cconnects, 2);
atomic_assert_int_eq(csends, 2);
atomic_assert_int_eq(creads, 1);
atomic_assert_int_eq(sreads, 2);
atomic_assert_int_eq(ssends, 1);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tlsdns_recv_send) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
isc_thread_t threads[workers];
SKIP_IN_CI;
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listentlsdns(listen_nm, ISC_NM_LISTEN_ALL,
&tcp_listen_addr, listen_read_cb, NULL,
listen_accept_cb, NULL, 0, NULL,
tcp_listen_tlsctx, &listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_equal(result, ISC_R_SUCCESS);
memset(threads, 0, sizeof(threads));
for (size_t i = 0; i < workers; i++) {
isc_thread_create(connect_thread, tlsdns_connect, &threads[i]);
}
WAIT_FOR_GE(cconnects, esends);
WAIT_FOR_GE(csends, esends);
WAIT_FOR_GE(sreads, esends);
WAIT_FOR_GE(ssends, esends / 2);
WAIT_FOR_GE(creads, esends / 2);
DONE();
for (size_t i = 0; i < workers; i++) {
isc_thread_join(threads[i], NULL);
}
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
X(cconnects);
X(csends);
X(creads);
X(sreads);
X(ssends);
CHECK_RANGE_FULL(csends);
CHECK_RANGE_FULL(creads);
CHECK_RANGE_FULL(sreads);
CHECK_RANGE_FULL(ssends);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tlsdns_recv_half_send) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
isc_thread_t threads[workers];
SKIP_IN_CI;
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listentlsdns(listen_nm, ISC_NM_LISTEN_ALL,
&tcp_listen_addr, listen_read_cb, NULL,
listen_accept_cb, NULL, 0, NULL,
tcp_listen_tlsctx, &listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_equal(result, ISC_R_SUCCESS);
memset(threads, 0, sizeof(threads));
for (size_t i = 0; i < workers; i++) {
isc_thread_create(connect_thread, tlsdns_connect, &threads[i]);
}
WAIT_FOR_GE(cconnects, esends / 2);
WAIT_FOR_GE(csends, esends / 2);
WAIT_FOR_GE(sreads, esends / 2);
WAIT_FOR_GE(ssends, esends / 2);
WAIT_FOR_GE(creads, esends / 2);
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
DONE();
for (size_t i = 0; i < workers; i++) {
isc_thread_join(threads[i], NULL);
}
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
X(cconnects);
X(csends);
X(creads);
X(sreads);
X(ssends);
CHECK_RANGE_HALF(csends);
CHECK_RANGE_HALF(creads);
CHECK_RANGE_HALF(sreads);
CHECK_RANGE_HALF(ssends);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tlsdns_half_recv_send) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
isc_thread_t threads[workers];
SKIP_IN_CI;
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listentlsdns(listen_nm, ISC_NM_LISTEN_ALL,
&tcp_listen_addr, listen_read_cb, NULL,
listen_accept_cb, NULL, 0, NULL,
tcp_listen_tlsctx, &listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_equal(result, ISC_R_SUCCESS);
memset(threads, 0, sizeof(threads));
for (size_t i = 0; i < workers; i++) {
isc_thread_create(connect_thread, tlsdns_connect, &threads[i]);
}
WAIT_FOR_GE(cconnects, esends / 2);
WAIT_FOR_GE(csends, esends / 2);
WAIT_FOR_GE(sreads, esends / 2);
WAIT_FOR_GE(ssends, esends / 2);
WAIT_FOR_GE(creads, esends / 2);
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
/* Try to send a little while longer */
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
isc_test_nap((esends / 2) * 10);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
DONE();
for (size_t i = 0; i < workers; i++) {
isc_thread_join(threads[i], NULL);
}
X(cconnects);
X(csends);
X(creads);
X(sreads);
X(ssends);
CHECK_RANGE_HALF(csends);
CHECK_RANGE_HALF(creads);
CHECK_RANGE_HALF(sreads);
CHECK_RANGE_HALF(ssends);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tlsdns_half_recv_half_send) {
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
isc_thread_t threads[workers];
SKIP_IN_CI;
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listentlsdns(listen_nm, ISC_NM_LISTEN_ALL,
&tcp_listen_addr, listen_read_cb, NULL,
listen_accept_cb, NULL, 0, NULL,
tcp_listen_tlsctx, &listen_sock);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
assert_int_equal(result, ISC_R_SUCCESS);
memset(threads, 0, sizeof(threads));
for (size_t i = 0; i < workers; i++) {
isc_thread_create(connect_thread, tlsdns_connect, &threads[i]);
}
WAIT_FOR_GE(cconnects, esends / 2);
WAIT_FOR_GE(csends, esends / 2);
WAIT_FOR_GE(sreads, esends / 2);
WAIT_FOR_GE(ssends, esends / 2);
WAIT_FOR_GE(creads, esends / 2);
Destroy netmgr before destroying taskmgr With taskmgr running on top of netmgr, the ordering of how the tasks and netmgr shutdown interacts was wrong as previously isc_taskmgr_destroy() was waiting until all tasks were properly shutdown and detached. This responsibility was moved to netmgr, so we now need to do the following: 1. shutdown all the tasks - this schedules all shutdown events onto the netmgr queue 2. shutdown the netmgr - this also makes sure all the tasks and events are properly executed 3. Shutdown the taskmgr - this now waits for all the tasks to finish running before returning 4. Shutdown the netmgr - this call waits for all the netmgr netievents to finish before returning This solves the race when the taskmgr object would be destroyed before all the tasks were finished running in the netmgr loops.
2021-04-27 10:28:40 +02:00
isc__netmgr_shutdown(connect_nm);
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
DONE();
for (size_t i = 0; i < workers; i++) {
isc_thread_join(threads[i], NULL);
}
X(cconnects);
X(csends);
X(creads);
X(sreads);
X(ssends);
CHECK_RANGE_HALF(csends);
CHECK_RANGE_HALF(creads);
CHECK_RANGE_HALF(sreads);
CHECK_RANGE_HALF(ssends);
}
Add ALPN negotiation tests to TLS DNS test set This commits adds a set of unit tests to ensure that ALPN happens over the connections and that the result of the negotiation can be checked.
2021-08-31 16:41:59 +03:00
static void
tlsdns_connect_connect_noalpn(isc_nmhandle_t *handle, isc_result_t eresult,
void *cbarg) {
isc_nmhandle_t *readhandle = NULL;
UNUSED(cbarg);
F();
isc_refcount_decrement(&active_cconnects);
if (eresult != ISC_R_SUCCESS || connect_readcb == NULL ||
!isc_nm_xfr_allowed(handle))
{
return;
}
atomic_fetch_add(&cconnects, 1);
isc_refcount_increment0(&active_creads);
isc_nmhandle_attach(handle, &readhandle);
isc_nm_read(handle, connect_readcb, NULL);
connect_send(handle);
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tlsdns_connect_noalpn) {
Add ALPN negotiation tests to TLS DNS test set This commits adds a set of unit tests to ensure that ALPN happens over the connections and that the result of the negotiation can be checked.
2021-08-31 16:41:59 +03:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
isc_sockaddr_t connect_addr;
isc_tlsctx_t *connect_tlsctx_noalpn = NULL;
result = isc_tlsctx_createclient(&connect_tlsctx_noalpn);
assert_true(result == ISC_R_SUCCESS);
connect_addr = (isc_sockaddr_t){ .length = 0 };
isc_sockaddr_fromin6(&connect_addr, &in6addr_loopback, 0);
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listentlsdns(listen_nm, ISC_NM_LISTEN_ALL,
&tcp_listen_addr, noop_recv_cb, NULL,
noop_accept_cb, NULL, 0, NULL,
Add ALPN negotiation tests to TLS DNS test set This commits adds a set of unit tests to ensure that ALPN happens over the connections and that the result of the negotiation can be checked.
2021-08-31 16:41:59 +03:00
tcp_listen_tlsctx, &listen_sock);
assert_int_equal(result, ISC_R_SUCCESS);
isc_refcount_increment0(&active_cconnects);
isc_nm_tlsdnsconnect(connect_nm, &connect_addr, &tcp_listen_addr,
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
tlsdns_connect_connect_noalpn, NULL, T_CONNECT,
DoT: implement TLS client session resumption This commit extends DoT code with TLS client session resumption support implemented on top of the TLS client session cache.
2022-04-25 16:47:06 +03:00
connect_tlsctx_noalpn, NULL);
Add ALPN negotiation tests to TLS DNS test set This commits adds a set of unit tests to ensure that ALPN happens over the connections and that the result of the negotiation can be checked.
2021-08-31 16:41:59 +03:00
WAIT_FOR_EQ(active_cconnects, 0);
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
isc__netmgr_shutdown(connect_nm);
X(cconnects);
X(csends);
X(creads);
X(sreads);
X(ssends);
atomic_assert_int_eq(cconnects, 0);
atomic_assert_int_eq(csends, 0);
atomic_assert_int_eq(creads, 0);
atomic_assert_int_eq(sreads, 0);
atomic_assert_int_eq(ssends, 0);
isc_tlsctx_free(&connect_tlsctx_noalpn);
}
#ifdef HAVE_LIBNGHTTP2
static isc_result_t
tls_accept_cb_noalpn(isc_nmhandle_t *handle, isc_result_t eresult,
void *cbarg) {
F();
if (eresult != ISC_R_SUCCESS) {
return (eresult);
}
atomic_fetch_add(&saccepts, 1);
if (!isc_nm_xfr_allowed(handle)) {
return (ISC_R_FAILURE);
}
return (stream_accept_cb(handle, eresult, cbarg));
}
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_RUN_TEST_IMPL(tlsdns_listen_noalpn) {
Add ALPN negotiation tests to TLS DNS test set This commits adds a set of unit tests to ensure that ALPN happens over the connections and that the result of the negotiation can be checked.
2021-08-31 16:41:59 +03:00
isc_result_t result = ISC_R_SUCCESS;
isc_nmsocket_t *listen_sock = NULL;
isc_sockaddr_t connect_addr;
isc_tlsctx_t *server_tlsctx_noalpn = NULL;
result = isc_tlsctx_createserver(NULL, NULL, &server_tlsctx_noalpn);
assert_true(result == ISC_R_SUCCESS);
connect_addr = (isc_sockaddr_t){ .length = 0 };
isc_sockaddr_fromin6(&connect_addr, &in6addr_loopback, 0);
/* We use TLS stream listener here intentionally, as it does not
* try to do ALPN. */
Allow listening on less than nworkers threads For some applications, it's useful to not listen on full battery of threads. Add workers argument to all isc_nm_listen*() functions and convenience ISC_NM_LISTEN_ONE and ISC_NM_LISTEN_ALL macros.
2022-03-25 16:44:21 +01:00
result = isc_nm_listentls(listen_nm, ISC_NM_LISTEN_ALL,
&tcp_listen_addr, tls_accept_cb_noalpn, NULL,
0, NULL, server_tlsctx_noalpn, &listen_sock);
Add ALPN negotiation tests to TLS DNS test set This commits adds a set of unit tests to ensure that ALPN happens over the connections and that the result of the negotiation can be checked.
2021-08-31 16:41:59 +03:00
assert_int_equal(result, ISC_R_SUCCESS);
isc_refcount_increment0(&active_cconnects);
isc_nm_tlsdnsconnect(connect_nm, &connect_addr, &tcp_listen_addr,
Remove extrahandle size from netmgr Previously, it was possible to assign a bit of memory space in the nmhandle to store the client data. This was complicated and prevents further refactoring of isc_nmhandle_t caching (future work). Instead of caching the data in the nmhandle, allocate the hot-path ns_client_t objects from per-thread clientmgr memory context and just assign it to the isc_nmhandle_t via isc_nmhandle_set().
2022-03-23 13:57:15 +01:00
connect_connect_cb, NULL, T_CONNECT,
DoT: implement TLS client session resumption This commit extends DoT code with TLS client session resumption support implemented on top of the TLS client session cache.
2022-04-25 16:47:06 +03:00
tcp_connect_tlsctx, tcp_tlsctx_client_sess_cache);
Add ALPN negotiation tests to TLS DNS test set This commits adds a set of unit tests to ensure that ALPN happens over the connections and that the result of the negotiation can be checked.
2021-08-31 16:41:59 +03:00
WAIT_FOR_EQ(saccepts, 1);
WAIT_FOR_EQ(cconnects, 1);
WAIT_FOR_EQ(csends, 1);
isc_nm_stoplistening(listen_sock);
isc_nmsocket_close(&listen_sock);
assert_null(listen_sock);
isc__netmgr_shutdown(connect_nm);
X(cconnects);
X(csends);
X(creads);
X(sreads);
X(ssends);
atomic_assert_int_eq(saccepts, 1);
atomic_assert_int_eq(cconnects, 1);
atomic_assert_int_eq(csends, 1);
atomic_assert_int_eq(creads, 0);
atomic_assert_int_eq(sreads, 0);
atomic_assert_int_eq(ssends, 0);
isc_tlsctx_free(&server_tlsctx_noalpn);
}
#endif /* HAVE_LIBNGHTTP2 */
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_TEST_LIST_START
ISC_TEST_ENTRY_CUSTOM(mock_listenudp_uv_udp_open, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(mock_listenudp_uv_udp_bind, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(mock_listenudp_uv_udp_recv_start, setup_test,
teardown_test)
ISC_TEST_ENTRY_CUSTOM(mock_udpconnect_uv_udp_open, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(mock_udpconnect_uv_udp_bind, setup_test, teardown_test)
Cleanup the remaining of HAVE_UV_<func> macros While cleaning up the usage of HAVE_UV_<func> macros, we forgot to cleanup the HAVE_UV_UDP_CONNECT in the actual code and HAVE_UV_TRANSLATE_SYS_ERROR and this was causing Windows build to fail on uv_udp_send() because the socket was already connected and we were falsely assuming that it was not. The platforms with autoconf support were not affected, because we were still checking for the functions from the configure.
2021-06-02 11:23:36 +02:00
#if UV_VERSION_HEX >= UV_VERSION(1, 27, 0)
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_TEST_ENTRY_CUSTOM(mock_udpconnect_uv_udp_connect, setup_test, teardown_test)
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
#endif
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_TEST_ENTRY_CUSTOM(mock_udpconnect_uv_recv_buffer_size, setup_test,
teardown_test)
ISC_TEST_ENTRY_CUSTOM(mock_udpconnect_uv_send_buffer_size, setup_test,
teardown_test)
ISC_TEST_ENTRY_CUSTOM(udp_noop, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(udp_noresponse, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(udp_timeout_recovery, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(udp_recv_one, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(udp_recv_two, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(udp_recv_send, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(udp_recv_half_send, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(udp_half_recv_send, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(udp_half_recv_half_send, setup_test, teardown_test)
/* TCP */
ISC_TEST_ENTRY_CUSTOM(tcp_noop, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tcp_noresponse, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tcp_timeout_recovery, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tcp_recv_one, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tcp_recv_two, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tcp_recv_send, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tcp_recv_half_send, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tcp_half_recv_send, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tcp_half_recv_half_send, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tcp_recv_send_sendback, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tcp_recv_half_send_sendback, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tcp_half_recv_send_sendback, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tcp_half_recv_half_send_sendback, setup_test,
teardown_test)
/* TCP Quota */
ISC_TEST_ENTRY_CUSTOM(tcp_recv_one_quota, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tcp_recv_two_quota, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tcp_recv_send_quota, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tcp_recv_half_send_quota, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tcp_half_recv_send_quota, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tcp_half_recv_half_send_quota, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tcp_recv_send_quota_sendback, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tcp_recv_half_send_quota_sendback, setup_test,
teardown_test)
ISC_TEST_ENTRY_CUSTOM(tcp_half_recv_send_quota_sendback, setup_test,
teardown_test)
ISC_TEST_ENTRY_CUSTOM(tcp_half_recv_half_send_quota_sendback, setup_test,
teardown_test)
/* TCPDNS */
ISC_TEST_ENTRY_CUSTOM(tcpdns_recv_one, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tcpdns_recv_two, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tcpdns_noop, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tcpdns_noresponse, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tcpdns_timeout_recovery, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tcpdns_recv_send, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tcpdns_recv_half_send, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tcpdns_half_recv_send, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tcpdns_half_recv_half_send, setup_test, teardown_test)
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
Make the DNS over HTTPS support optional This commit adds two new autoconf options `--enable-doh` (enabled by default) and `--with-libnghttp2` (mandatory when DoH is enabled). When DoH support is disabled the library is not linked-in and support for http(s) protocol is disabled in the netmgr, named and dig.
2021-04-21 13:52:15 +02:00
#if HAVE_LIBNGHTTP2
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
/* TLS */
ISC_TEST_ENTRY_CUSTOM(tls_noop, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tls_noresponse, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tls_timeout_recovery, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tls_recv_one, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tls_recv_two, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tls_recv_send, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tls_recv_half_send, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tls_half_recv_send, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tls_half_recv_half_send, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tls_recv_send_sendback, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tls_recv_half_send_sendback, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tls_half_recv_send_sendback, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tls_half_recv_half_send_sendback, setup_test,
teardown_test)
/* TLS quota */
ISC_TEST_ENTRY_CUSTOM(tls_recv_one_quota, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tls_recv_two_quota, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tls_recv_send_quota, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tls_recv_half_send_quota, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tls_half_recv_send_quota, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tls_half_recv_half_send_quota, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tls_recv_send_quota_sendback, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tls_recv_half_send_quota_sendback, setup_test,
teardown_test)
ISC_TEST_ENTRY_CUSTOM(tls_half_recv_send_quota_sendback, setup_test,
teardown_test)
ISC_TEST_ENTRY_CUSTOM(tls_half_recv_half_send_quota_sendback, setup_test,
teardown_test)
Make the DNS over HTTPS support optional This commit adds two new autoconf options `--enable-doh` (enabled by default) and `--with-libnghttp2` (mandatory when DoH is enabled). When DoH support is disabled the library is not linked-in and support for http(s) protocol is disabled in the netmgr, named and dig.
2021-04-21 13:52:15 +02:00
#endif
Merge the tls_test.c into netmgr_test.c and extend the tests suite This commit merges TLS tests into the common Network Manager unit tests suite and extends the unit test framework to include support for additional "ping-pong" style tests where all data could be sent via lesser number of connections (the behaviour of the old test suite). The tests for TCP and TLS were extended to make use of the new mode, as this mode better translates to how the code is used in DoH. Both TLS and TCP tests now share most of the unit tests' code, as they are expected to function similarly from a users's perspective anyway. Additionally to the above, the TLS test suite was extended to include TLS tests using the connections quota facility.
2021-04-01 09:15:43 +02:00
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
/* TLSDNS */
ISC_TEST_ENTRY_CUSTOM(tlsdns_recv_one, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tlsdns_recv_two, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tlsdns_noop, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tlsdns_noresponse, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tlsdns_timeout_recovery, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tlsdns_recv_send, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tlsdns_recv_half_send, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tlsdns_half_recv_send, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tlsdns_half_recv_half_send, setup_test, teardown_test)
ISC_TEST_ENTRY_CUSTOM(tlsdns_connect_noalpn, setup_test, teardown_test)
Add ALPN negotiation tests to TLS DNS test set This commits adds a set of unit tests to ensure that ALPN happens over the connections and that the result of the negotiation can be checked.
2021-08-31 16:41:59 +03:00
#ifdef HAVE_LIBNGHTTP2
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_TEST_ENTRY_CUSTOM(tlsdns_listen_noalpn, setup_test, teardown_test)
Add ALPN negotiation tests to TLS DNS test set This commits adds a set of unit tests to ensure that ALPN happens over the connections and that the result of the negotiation can be checked.
2021-08-31 16:41:59 +03:00
#endif
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_TEST_LIST_END
Improve reliability of the netmgr unit tests The netmgr unit tests were designed to push the system limits to maximum by sending as many queries as possible in the busy loop from multiple threads. This mostly works with UDP, but in the stateful protocol where establishing the connection takes more time, it failed quite often in the CI. On FreeBSD, this happened more often, because the socket() call would fail spuriosly making the problem even worse. This commit does several things to improve reliability: * return value of isc_nm_<proto>connect() is always checked and retried when scheduling the connection fails * The busy while loop has been slowed down with usleep(1000); so the netmgr threads could schedule the work and get executed. * The isc_thread_yield() was replaced with usleep(1000); also to allow the other threads to do any work. * Instead of waiting on just one variable, we wait for multiple variables to reach the final value * We are wrapping the netmgr operations (connects, reads, writes, accepts) with reference counting and waiting for all the callbacks to be accounted for. This has two effects: a) the isc_nm_t is always clean of active sockets and handles when destroyed, so it will prevent the spurious INSIST(references == 1) from isc_nm_destroy() b) the unit test now ensures that all the callbacks are always called when they should be called, so any stuck test means that there was a missing callback call and it is always a real bug These changes allows us to remove the workaround that would not run certain tests on systems without port load-balancing.
2021-01-29 13:00:46 +01:00
Give the unit tests a big overhaul The unit tests contain a lot of duplicated code and here's an attempt to reduce code duplication. This commit does several things: 1. Remove #ifdef HAVE_CMOCKA - we already solve this with automake conditionals. 2. Create a set of ISC_TEST_* and ISC_*_TEST_ macros to wrap the test implementations, test lists, and the main test routine, so we don't have to repeat this all over again. The macros were modeled after libuv test suite but adapted to cmocka as the test driver. A simple example of a unit test would be: ISC_RUN_TEST_IMPL(test1) { assert_true(true); } ISC_TEST_LIST_START ISC_TEST_ENTRY(test1) ISC_TEST_LIST_END ISC_TEST_MAIN (Discussion: Should this be ISC_TEST_RUN ?) For more complicated examples including group setup and teardown functions, and per-test setup and teardown functions. 3. The macros prefix the test functions and cmocka entries, so the name of the test can now match the tested function name, and we don't have to append `_test` because `run_test_` is automatically prepended to the main test function, and `setup_test_` and `teardown_test_` is prepended to setup and teardown function. 4. Update all the unit tests to use the new syntax and fix a few bits here and there. 5. In the future, we can separate the test declarations and test implementations which are going to greatly help with uncluttering the bigger unit tests like doh_test and netmgr_test, because the test implementations are not declared static (see `ISC_RUN_TEST_DECLARE` and `ISC_RUN_TEST_IMPL` for more details. NOTE: This heavily relies on preprocessor macros, but the result greatly outweighs all the negatives of using the macros. There's less duplicated code, the tests are more uniform and the implementation can be more flexible.
2022-05-02 10:56:42 +02:00
ISC_TEST_MAIN
Reference in New Issue Copy Permalink