[#3019] retry db open on startup

doc/examples/kea4/all-keys.json

@@ -408,6 +408,10 @@
                 // serve-retry-continue
                 "on-fail": "stop-retry-exit",
 
+                // Flag which indicates if the DB recovery should be attempted
+                // at server startup and on reconfiguration events.
+                "retry-on-startup": false,
+
                 // Connection connect timeout in seconds.
                 "connect-timeout": 100,
 

doc/examples/kea4/backends.json

@@ -42,6 +42,7 @@
 //      "reconnect-wait-time": 3000, // expressed in ms
 //      "max-reconnect-tries": 3,
 //      "on-fail": "stop-retry-exit",
+//      "retry-on-startup": false,
 //      "connect-timeout": 3
 //  },
 
@@ -62,6 +63,7 @@
 //      "reconnect-wait-time": 3000, // expressed in ms
 //      "max-reconnect-tries": 3,
 //      "on-fail": "stop-retry-exit",
+//      "retry-on-startup": false,
 //      "connect-timeout": 3
 //  },
 

doc/examples/kea6/all-keys.json

@@ -350,6 +350,10 @@
                 // serve-retry-continue
                 "on-fail": "stop-retry-exit",
 
+                // Flag which indicates if the DB recovery should be attempted
+                // at server startup and on reconfiguration events.
+                "retry-on-startup": false,
+
                 // Connection connect timeout in seconds.
                 "connect-timeout": 100,
 

doc/examples/kea6/backends.json

@@ -42,6 +42,7 @@
 //      "reconnect-wait-time": 3000, // expressed in ms
 //      "max-reconnect-tries": 3,
 //      "on-fail": "stop-retry-exit",
+//      "retry-on-startup": false,
 //      "connect-timeout": 3
 //  },
 
@@ -62,6 +63,7 @@
 //      "reconnect-wait-time": 3000, // expressed in ms
 //      "max-reconnect-tries": 3,
 //      "on-fail": "stop-retry-exit",
+//      "retry-on-startup": false,
 //      "connect-timeout": 3
 //  },
 

doc/sphinx/arm/database-connectivity.rst

@@ -6,9 +6,9 @@ Database Connectivity
 The Kea servers (:iscman:`kea-dhcp4` and :iscman:`kea-dhcp6`) can be configured to use a variety of
 database backends for leases, hosts, and configuration. They can be
 configured to support automatic recovery when connectivity is lost, via
-the ``on-fail`` parameter. (The ``reconnect-wait-time`` and
+the ``on-fail`` and ``retry-on-startup`` parameters.
-``max-reconnect-tries`` parameters are described
+(The ``reconnect-wait-time`` and ``max-reconnect-tries`` parameters are
-in :ref:`database-configuration4` and :ref:`database-configuration6`.)
+described in :ref:`database-configuration4` and :ref:`database-configuration6`.)
 
 It is important to understand how and when automatic recovery comes into play.
 Automatic recovery, when configured, only operates after a successful startup
@@ -16,10 +16,13 @@ or reconfiguration during which connectivity to all backends has been
 successfully established.
 
 During server startup, the inability to connect to any of the configured
-backends is always considered fatal. A fatal error is logged and the server
+backends is considered fatal only if ``retry-on-startup`` is set to ``false``.
-exits, based on the idea that the configuration should be valid
+A fatal error is logged and the server exits, based on the idea that the
-at startup. Exiting to the operating system allows nanny scripts to detect
+configuration should be valid at startup. Exiting to the operating system allows
-the problem.
+nanny scripts to detect the problem.
+If ``retry-on-startup`` is set to ``true``, the server will start reconnection
+attempts even at server startup or on reconfigure events, and will honor the
+action specified in ``on-fail`` parameter.
 
 During dynamic reconfiguration, all backends are disconnected and then
 reconnected using the new configuration. If connectivity to any of the

doc/sphinx/arm/dhcp4-srv.rst

@@ -576,6 +576,19 @@ The possible values are:
    active while processing DHCP traffic. Change this only if the server is used
    exclusively as a configuration tool.
 
+::
+
+   "Dhcp4": { "lease-database": { "retry-on-startup" : true, ... }, ... }
+
+During server startup, the inability to connect to any of the configured
+backends is considered fatal only if ``retry-on-startup`` is set to ``false``.
+A fatal error is logged and the server exits, based on the idea that the
+configuration should be valid at startup. Exiting to the operating system allows
+nanny scripts to detect the problem.
+If ``retry-on-startup`` is set to ``true``, the server will start reconnection
+attempts even at server startup or on reconfigure events, and will honor the
+action specified in ``on-fail`` parameter.
+
 The host parameter is used by the MySQL and PostgreSQL backends.
 
 Finally, the credentials of the account under which the server will
@@ -798,6 +811,19 @@ The possible values are:
    the lease database backend and the hosts database backend are connected to
    the same database instance.
 
+::
+
+   "Dhcp4": { "hosts-database": { "retry-on-startup" : true, ... }, ... }
+
+During server startup, the inability to connect to any of the configured
+backends is considered fatal only if ``retry-on-startup`` is set to ``false``.
+A fatal error is logged and the server exits, based on the idea that the
+configuration should be valid at startup. Exiting to the operating system allows
+nanny scripts to detect the problem.
+If ``retry-on-startup`` is set to ``true``, the server will start reconnection
+attempts even at server startup or on reconfigure events, and will honor the
+action specified in ``on-fail`` parameter.
+
 Finally, the credentials of the account under which the server will
 access the database should be set:
 

doc/sphinx/arm/dhcp6-srv.rst

@@ -532,6 +532,19 @@ The possible values are:
    active while processing DHCP traffic. Change this only if the server is used
    exclusively as a configuration tool.
 
+::
+
+   "Dhcp6": { "lease-database": { "retry-on-startup" : true, ... }, ... }
+
+During server startup, the inability to connect to any of the configured
+backends is considered fatal only if ``retry-on-startup`` is set to ``false``.
+A fatal error is logged and the server exits, based on the idea that the
+configuration should be valid at startup. Exiting to the operating system allows
+nanny scripts to detect the problem.
+If ``retry-on-startup`` is set to ``true``, the server will start reconnection
+attempts even at server startup or on reconfigure events, and will honor the
+action specified in ``on-fail`` parameter.
+
 The host parameter is used by the MySQL and PostgreSQL backends.
 
 Finally, the credentials of the account under which the server will
@@ -754,6 +767,19 @@ The possible values are:
    the lease database backend and the hosts database backend are connected to
    the same database instance.
 
+::
+
+   "Dhcp6": { "hosts-database": { "retry-on-startup" : true, ... }, ... }
+
+During server startup, the inability to connect to any of the configured
+backends is considered fatal only if ``retry-on-startup`` is set to ``false``.
+A fatal error is logged and the server exits, based on the idea that the
+configuration should be valid at startup. Exiting to the operating system allows
+nanny scripts to detect the problem.
+If ``retry-on-startup`` is set to ``true``, the server will start reconnection
+attempts even at server startup or on reconfigure events, and will honor the
+action specified in ``on-fail`` parameter.
+
 Finally, the credentials of the account under which the server will
 access the database should be set:
 

doc/sphinx/arm/hooks-legal-log.rst

@@ -1083,3 +1083,12 @@ If ``on-fail`` is set to ``serve-retry-exit``, the server will shut down if
 the connection to the database backend is not restored according to the
 ``max-reconnect-tries`` and ``reconnect-wait-time`` parameters, but it
 continues serving clients while this mechanism is activated.
+
+During server startup, the inability to connect to any of the configured
+backends is considered fatal only if ``retry-on-startup`` is set to ``false``.
+A fatal error is logged and the server exits, based on the idea that the
+configuration should be valid at startup. Exiting to the operating system allows
+nanny scripts to detect the problem.
+If ``retry-on-startup`` is set to ``true``, the server will start reconnection
+attempts even at server startup or on reconfigure events, and will honor the
+action specified in ``on-fail`` parameter.

src/bin/dhcp4/ctrl_dhcp4_srv.cc

@@ -911,6 +911,10 @@ isc::data::ConstElementPtr
 ControlledDhcpv4Srv::processConfig(isc::data::ConstElementPtr config) {
     ControlledDhcpv4Srv* srv = ControlledDhcpv4Srv::getInstance();
 
+    // Allow DB reconnect on startup. The database connection parameters specify
+    // respective details.
+    DbConnectionInitWithRetry retry;
+
     // Single stream instance used in all error clauses
     std::ostringstream err;
 
@@ -1155,11 +1159,8 @@ ControlledDhcpv4Srv::ControlledDhcpv4Srv(uint16_t server_port /*= DHCP4_SERVER_P
     // CommandMgr uses IO service to run asynchronous socket operations.
     CommandMgr::instance().setIOService(getIOService());
 
-    // LeaseMgr uses IO service to run asynchronous timers.
+    // DatabaseConnection uses IO service to run asynchronous timers.
-    LeaseMgr::setIOService(getIOService());
+    DatabaseConnection::setIOService(getIOService());
-
-    // HostMgr uses IO service to run asynchronous timers.
-    HostMgr::setIOService(getIOService());
 
     // These are the commands always supported by the DHCPv4 server.
     // Please keep the list in alphabetic order.
@@ -1295,11 +1296,8 @@ ControlledDhcpv4Srv::~ControlledDhcpv4Srv() {
         CommandMgr::instance().deregisterCommand("status-get");
         CommandMgr::instance().deregisterCommand("version-get");
 
-        // LeaseMgr uses IO service to run asynchronous timers.
+        // Reset DatabaseConnection IO service.
-        LeaseMgr::setIOService(IOServicePtr());
+        DatabaseConnection::setIOService(IOServicePtr());
-
-        // HostMgr uses IO service to run asynchronous timers.
-        HostMgr::setIOService(IOServicePtr());
     } catch (...) {
         // Don't want to throw exceptions from the destructor. The server
         // is shutting down anyway.

src/bin/dhcp4/dhcp4_lexer.ll

@@ -593,6 +593,17 @@ ControlCharacterFill            [^"\\]|\\["\\/bfnrtu]
     }
 }
 
+\"retry-on-startup\" {
+    switch(driver.ctx_) {
+    case isc::dhcp::Parser4Context::LEASE_DATABASE:
+    case isc::dhcp::Parser4Context::HOSTS_DATABASE:
+    case isc::dhcp::Parser4Context::CONFIG_DATABASE:
+        return isc::dhcp::Dhcp4Parser::make_RETRY_ON_STARTUP(driver.loc_);
+    default:
+        return isc::dhcp::Dhcp4Parser::make_STRING("retry-on-startup", driver.loc_);
+    }
+}
+
 \"max-reconnect-tries\" {
     switch(driver.ctx_) {
     case isc::dhcp::Parser4Context::LEASE_DATABASE:

src/bin/dhcp4/dhcp4_parser.yy

@@ -109,6 +109,7 @@ using namespace std;
   STOP_RETRY_EXIT "stop-retry-exit"
   SERVE_RETRY_EXIT "serve-retry-exit"
   SERVE_RETRY_CONTINUE "serve-retry-continue"
+  RETRY_ON_STARTUP "retry-on-startup"
   MAX_ROW_ERRORS "max-row-errors"
   TRUST_ANCHOR "trust-anchor"
   CERT_FILE "cert-file"
@@ -1093,6 +1094,7 @@ database_map_param: database_type
                   | max_reconnect_tries
                   | reconnect_wait_time
                   | on_fail
+                  | retry_on_startup
                   | max_row_errors
                   | trust_anchor
                   | cert_file
@@ -1223,6 +1225,12 @@ on_fail_mode: STOP_RETRY_EXIT { $$ = ElementPtr(new StringElement("stop-retry-ex
             | SERVE_RETRY_CONTINUE { $$ = ElementPtr(new StringElement("serve-retry-continue", ctx.loc2pos(@1))); }
             ;
 
+retry_on_startup: RETRY_ON_STARTUP COLON BOOLEAN {
+    ctx.unique("retry-on-startup", ctx.loc2pos(@1));
+    ElementPtr n(new BoolElement($3, ctx.loc2pos(@3)));
+    ctx.stack_.back()->set("retry-on-startup", n);
+};
+
 max_row_errors: MAX_ROW_ERRORS COLON INTEGER {
     ctx.unique("max-row-errors", ctx.loc2pos(@1));
     ElementPtr n(new IntElement($3, ctx.loc2pos(@3)));

src/bin/dhcp4/tests/dhcp4_srv_unittest.cc

@@ -53,15 +53,16 @@
 #include <arpa/inet.h>
 #include <dirent.h>
 
-using namespace std;
 using namespace isc;
-using namespace isc::dhcp;
-using namespace isc::data;
 using namespace isc::asiolink;
 using namespace isc::cb;
 using namespace isc::config;
+using namespace isc::data;
+using namespace isc::db;
+using namespace isc::dhcp;
 using namespace isc::dhcp::test;
 using namespace isc::util;
+using namespace std;
 
 namespace {
 
@@ -2879,11 +2880,8 @@ Dhcpv4SrvTest::loadConfigFile(const string& path) {
     // CommandMgr uses IO service to run asynchronous socket operations.
     CommandMgr::instance().setIOService(srv.getIOService());
 
-    // LeaseMgr uses IO service to run asynchronous timers.
+    // DatabaseConnection uses IO service to run asynchronous timers.
-    LeaseMgr::setIOService(srv.getIOService());
+    DatabaseConnection::setIOService(srv.getIOService());
-
-    // HostMgr uses IO service to run asynchronous timers.
-    HostMgr::setIOService(srv.getIOService());
 
     Parser4Context parser;
     ConstElementPtr json;
@@ -2917,11 +2915,8 @@ Dhcpv4SrvTest::loadConfigFile(const string& path) {
     // Reset CommandMgr IO service.
     CommandMgr::instance().setIOService(IOServicePtr());
 
-    // Reset LeaseMgr IO service.
+    // Reset DatabaseConnection IO service.
-    LeaseMgr::setIOService(IOServicePtr());
+    DatabaseConnection::setIOService(IOServicePtr());
-
-    // Reset HostMgr IO service.
-    HostMgr::setIOService(IOServicePtr());
 }
 
 /// @brief Class which handles initialization of database

src/bin/dhcp4/tests/dhcp4_test_utils.h

@@ -125,7 +125,7 @@ public:
         // Create fixed server id.
         server_id_.reset(new Option4AddrLst(DHO_DHCP_SERVER_IDENTIFIER,
                                             asiolink::IOAddress("192.0.3.1")));
-        LeaseMgr::setIOService(getIOService());
+        db::DatabaseConnection::setIOService(getIOService());
     }
 
     /// @brief Returns fixed server identifier assigned to the naked server

src/bin/dhcp6/ctrl_dhcp6_srv.cc

@@ -916,6 +916,10 @@ isc::data::ConstElementPtr
 ControlledDhcpv6Srv::processConfig(isc::data::ConstElementPtr config) {
     ControlledDhcpv6Srv* srv = ControlledDhcpv6Srv::getInstance();
 
+    // Allow DB reconnect on startup. The database connection parameters specify
+    // respective details.
+    DbConnectionInitWithRetry retry;
+
     // Single stream instance used in all error clauses
     std::ostringstream err;
 
@@ -1174,11 +1178,8 @@ ControlledDhcpv6Srv::ControlledDhcpv6Srv(uint16_t server_port /*= DHCP6_SERVER_P
     // CommandMgr uses IO service to run asynchronous socket operations.
     CommandMgr::instance().setIOService(getIOService());
 
-    // LeaseMgr uses IO service to run asynchronous timers.
+    // DatabaseConnection uses IO service to run asynchronous timers.
-    LeaseMgr::setIOService(getIOService());
+    DatabaseConnection::setIOService(getIOService());
-
-    // HostMgr uses IO service to run asynchronous timers.
-    HostMgr::setIOService(getIOService());
 
     // These are the commands always supported by the DHCPv6 server.
     // Please keep the list in alphabetic order.
@@ -1314,11 +1315,8 @@ ControlledDhcpv6Srv::~ControlledDhcpv6Srv() {
         CommandMgr::instance().deregisterCommand("status-get");
         CommandMgr::instance().deregisterCommand("version-get");
 
-        // LeaseMgr uses IO service to run asynchronous timers.
+        // Reset DatabaseConnection IO service.
-        LeaseMgr::setIOService(IOServicePtr());
+        DatabaseConnection::setIOService(IOServicePtr());
-
-        // HostMgr uses IO service to run asynchronous timers.
-        HostMgr::setIOService(IOServicePtr());
     } catch (...) {
         // Don't want to throw exceptions from the destructor. The server
         // is shutting down anyway.

src/bin/dhcp6/dhcp6_lexer.ll

@@ -791,6 +791,17 @@ ControlCharacterFill            [^"\\]|\\["\\/bfnrtu]
     }
 }
 
+\"retry-on-startup\" {
+    switch(driver.ctx_) {
+    case isc::dhcp::Parser6Context::LEASE_DATABASE:
+    case isc::dhcp::Parser6Context::HOSTS_DATABASE:
+    case isc::dhcp::Parser6Context::CONFIG_DATABASE:
+        return isc::dhcp::Dhcp6Parser::make_RETRY_ON_STARTUP(driver.loc_);
+    default:
+        return isc::dhcp::Dhcp6Parser::make_STRING("retry-on-startup", driver.loc_);
+    }
+}
+
 \"max-reconnect-tries\" {
     switch(driver.ctx_) {
     case isc::dhcp::Parser6Context::LEASE_DATABASE:

src/bin/dhcp6/dhcp6_parser.yy

@@ -92,6 +92,7 @@ using namespace std;
   STOP_RETRY_EXIT "stop-retry-exit"
   SERVE_RETRY_EXIT "serve-retry-exit"
   SERVE_RETRY_CONTINUE "serve-retry-continue"
+  RETRY_ON_STARTUP "retry-on-startup"
   MAX_ROW_ERRORS "max-row-errors"
   TRUST_ANCHOR "trust-anchor"
   CERT_FILE "cert-file"
@@ -1024,6 +1025,7 @@ database_map_param: database_type
                   | max_reconnect_tries
                   | reconnect_wait_time
                   | on_fail
+                  | retry_on_startup
                   | max_row_errors
                   | trust_anchor
                   | cert_file
@@ -1149,6 +1151,12 @@ on_fail_mode: STOP_RETRY_EXIT { $$ = ElementPtr(new StringElement("stop-retry-ex
             | SERVE_RETRY_CONTINUE { $$ = ElementPtr(new StringElement("serve-retry-continue", ctx.loc2pos(@1))); }
             ;
 
+retry_on_startup: RETRY_ON_STARTUP COLON BOOLEAN {
+    ctx.unique("retry-on-startup", ctx.loc2pos(@1));
+    ElementPtr n(new BoolElement($3, ctx.loc2pos(@3)));
+    ctx.stack_.back()->set("retry-on-startup", n);
+};
+
 max_row_errors: MAX_ROW_ERRORS COLON INTEGER {
     ctx.unique("max-row-errors", ctx.loc2pos(@1));
     ElementPtr n(new IntElement($3, ctx.loc2pos(@3)));

src/bin/dhcp6/tests/dhcp6_srv_unittest.cc

@@ -60,6 +60,7 @@ using namespace isc::asiolink;
 using namespace isc::cb;
 using namespace isc::config;
 using namespace isc::data;
+using namespace isc::db;
 using namespace isc::dhcp;
 using namespace isc::dhcp::test;
 using namespace isc::util;
@@ -248,11 +249,8 @@ Dhcpv6SrvTest::loadConfigFile(const string& path) {
     // CommandMgr uses IO service to run asynchronous socket operations.
     CommandMgr::instance().setIOService(srv.getIOService());
 
-    // LeaseMgr uses IO service to run asynchronous timers.
+    // DatabaseConnection uses IO service to run asynchronous timers.
-    LeaseMgr::setIOService(srv.getIOService());
+    DatabaseConnection::setIOService(srv.getIOService());
-
-    // HostMgr uses IO service to run asynchronous timers.
-    HostMgr::setIOService(srv.getIOService());
 
     Parser6Context parser;
     ConstElementPtr json;
@@ -286,11 +284,8 @@ Dhcpv6SrvTest::loadConfigFile(const string& path) {
     // Reset CommandMgr IO service.
     CommandMgr::instance().setIOService(IOServicePtr());
 
-    // Reset LeaseMgr IO service.
+    // Reset DatabaseConnection IO service.
-    LeaseMgr::setIOService(IOServicePtr());
+    DatabaseConnection::setIOService(IOServicePtr());
-
-    // Reset HostMgr IO service.
-    HostMgr::setIOService(IOServicePtr());
 }
 
 /// @brief Class which handles initialization of database

src/bin/dhcp6/tests/dhcp6_test_utils.h

@@ -135,7 +135,7 @@ public:
         // Open the "memfile" database for leases
         std::string memfile = "type=memfile universe=6 persist=false";
         isc::dhcp::LeaseMgrFactory::create(memfile);
-        LeaseMgr::setIOService(getIOService());
+        db::DatabaseConnection::setIOService(getIOService());
     }
 
     /// @brief fakes packet reception

src/hooks/dhcp/mysql_cb/mysql_cb_dhcp4.cc

@@ -27,6 +27,7 @@
 #include <dhcpsrv/parsers/client_class_def_parser.h>
 #include <util/buffer.h>
 #include <util/boost_time_utils.h>
+#include <util/dhcp_space.h>
 #include <util/multi_threading_mgr.h>
 #include <mysql/mysql_connection.h>
 #include <boost/date_time/posix_time/posix_time.hpp>
@@ -3687,7 +3688,8 @@ TaggedStatementArray tagged_statements = { {
 } // end anonymous namespace
 
 MySqlConfigBackendDHCPv4Impl::MySqlConfigBackendDHCPv4Impl(const DatabaseConnection::ParameterMap& parameters)
-    : MySqlConfigBackendImpl(parameters, &MySqlConfigBackendDHCPv4Impl::dbReconnect) {
+    : MySqlConfigBackendImpl(std::string(cStringDhcpSpace<DHCPv4>()), parameters,
+                             &MySqlConfigBackendDHCPv4Impl::dbReconnect) {
     // Prepare query statements. Those are will be only used to retrieve
     // information from the database, so they can be used even if the
     // database is read only for the current user.
@@ -3696,14 +3698,6 @@ MySqlConfigBackendDHCPv4Impl::MySqlConfigBackendDHCPv4Impl(const DatabaseConnect
 // @todo As part of enabling read-only CB access, statements need to
 // be limited:
 //                            tagged_statements.begin() + WRITE_STMTS_BEGIN);
-
-    // Create unique timer name per instance.
-    timer_name_ = "MySqlConfigBackend4[";
-    timer_name_ += boost::lexical_cast<std::string>(reinterpret_cast<uint64_t>(this));
-    timer_name_ += "]DbReconnectTimer";
-
-    // Create ReconnectCtl for this connection.
-    conn_.makeReconnectCtl(timer_name_);
 }
 
 MySqlConfigBackendDHCPv4Impl::~MySqlConfigBackendDHCPv4Impl() {

src/hooks/dhcp/mysql_cb/mysql_cb_dhcp6.cc

@@ -28,6 +28,7 @@
 #include <dhcpsrv/parsers/client_class_def_parser.h>
 #include <util/buffer.h>
 #include <util/boost_time_utils.h>
+#include <util/dhcp_space.h>
 #include <util/multi_threading_mgr.h>
 #include <util/triplet.h>
 #include <mysql/mysql_connection.h>
@@ -4143,7 +4144,8 @@ TaggedStatementArray tagged_statements = { {
 } // end anonymous namespace
 
 MySqlConfigBackendDHCPv6Impl::MySqlConfigBackendDHCPv6Impl(const DatabaseConnection::ParameterMap& parameters)
-    : MySqlConfigBackendImpl(parameters, &MySqlConfigBackendDHCPv6Impl::dbReconnect) {
+    : MySqlConfigBackendImpl(std::string(cStringDhcpSpace<DHCPv6>()), parameters,
+                             &MySqlConfigBackendDHCPv6Impl::dbReconnect) {
     // Prepare query statements. Those are will be only used to retrieve
     // information from the database, so they can be used even if the
     // database is read only for the current user.
@@ -4152,14 +4154,6 @@ MySqlConfigBackendDHCPv6Impl::MySqlConfigBackendDHCPv6Impl(const DatabaseConnect
 // @todo As part of enabling read-only CB access, statements need to
 // be limited:
 //                            tagged_statements.begin() + WRITE_STMTS_BEGIN);
-
-    // Create unique timer name per instance.
-    timer_name_ = "MySqlConfigBackend6[";
-    timer_name_ += boost::lexical_cast<std::string>(reinterpret_cast<uint64_t>(this));
-    timer_name_ += "]DbReconnectTimer";
-
-    // Create ReconnectCtl for this connection.
-    conn_.makeReconnectCtl(timer_name_);
 }
 
 MySqlConfigBackendDHCPv6Impl::~MySqlConfigBackendDHCPv6Impl() {

src/hooks/dhcp/mysql_cb/mysql_cb_impl.cc

@@ -5,11 +5,13 @@
 // file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 #include <config.h>
+
 #include <mysql_cb_impl.h>
 #include <mysql_cb_log.h>
 #include <asiolink/io_address.h>
 #include <config_backend/constants.h>
 #include <dhcp/option_space.h>
+#include <dhcpsrv/timer_mgr.h>
 #include <util/buffer.h>
 
 #include <mysql.h>
@@ -48,17 +50,43 @@ ScopedAuditRevision::~ScopedAuditRevision() {
 }
 
 MySqlConfigBackendImpl::
-MySqlConfigBackendImpl(const DatabaseConnection::ParameterMap& parameters,
+MySqlConfigBackendImpl(const std::string& space,
+                       const DatabaseConnection::ParameterMap& parameters,
                        const DbCallback db_reconnect_callback)
     : conn_(parameters,
-            IOServiceAccessorPtr(new IOServiceAccessor(MySqlConfigBackendImpl::getIOService)),
+            IOServiceAccessorPtr(new IOServiceAccessor(&MySqlConfigBackendImpl::getIOService)),
             db_reconnect_callback), timer_name_(""),
       audit_revision_ref_count_(0), parameters_(parameters) {
+
+    // Create unique timer name per instance.
+    timer_name_ = "MySqlConfigBackend";
+    timer_name_ += space;
+    timer_name_ += "[";
+    timer_name_ += boost::lexical_cast<std::string>(reinterpret_cast<uint64_t>(this));
+    timer_name_ += "]DbReconnectTimer";
+
+    // Create ReconnectCtl for this connection.
+    conn_.makeReconnectCtl(timer_name_);
+
     // Test schema version first.
     std::pair<uint32_t, uint32_t> code_version(MYSQL_SCHEMA_VERSION_MAJOR,
                                                MYSQL_SCHEMA_VERSION_MINOR);
+
+    std::string timer_name = "";
+    bool retry = false;
+    if (parameters.count("retry-on-startup")) {
+        if (parameters.at("retry-on-startup") == "true") {
+            retry = true;
+        }
+    }
+    if (retry) {
+        timer_name = timer_name_;
+    }
+
+    IOServiceAccessorPtr ac(new IOServiceAccessor(&DatabaseConnection::getIOService));
+
     std::pair<uint32_t, uint32_t> db_version =
-        MySqlConnection::getVersion(parameters);
+        MySqlConnection::getVersion(parameters, ac, db_reconnect_callback, timer_name);
     if (code_version != db_version) {
         isc_throw(DbOpenError, "MySQL schema version mismatch: need version: "
                   << code_version.first << "." << code_version.second
@@ -829,7 +857,6 @@ MySqlConfigBackendImpl::processOptionRow(const Option::Universe& universe,
         code = (*(first_binding + 1))->getInteger<uint16_t>();
     }
 
-
     // Get formatted value if available.
     std::string formatted_value = (*(first_binding + 3))->getStringOrDefault("");
 

src/hooks/dhcp/mysql_cb/mysql_cb_impl.h

@@ -106,10 +106,12 @@ public:
 
     /// @brief Constructor.
     ///
+    /// @param space The DHCP space (v4 or v6).
     /// @param parameters A data structure relating keywords and values
     /// concerned with the database.
     /// @param db_reconnect_callback The connection recovery callback.
-    explicit MySqlConfigBackendImpl(const db::DatabaseConnection::ParameterMap& parameters,
+    explicit MySqlConfigBackendImpl(const std::string& space,
+                                    const db::DatabaseConnection::ParameterMap& parameters,
                                     const db::DbCallback db_reconnect_callback);
 
     /// @brief Destructor.

src/hooks/dhcp/mysql_cb/tests/mysql_cb_dhcp4_unittest.cc

@@ -462,6 +462,46 @@ public:
     }
 };
 
+TEST_F(MySqlConfigBackendDHCPv4DbLostCallbackTest, testRetryOpenDbLostAndRecoveredCallback) {
+    MultiThreadingTest mt(false);
+    testRetryOpenDbLostAndRecoveredCallback();
+}
+
+TEST_F(MySqlConfigBackendDHCPv4DbLostCallbackTest, testRetryOpenDbLostAndRecoveredCallbackMultiThreading) {
+    MultiThreadingTest mt(true);
+    testRetryOpenDbLostAndRecoveredCallback();
+}
+
+TEST_F(MySqlConfigBackendDHCPv4DbLostCallbackTest, testRetryOpenDbLostAndFailedCallback) {
+    MultiThreadingTest mt(false);
+    testRetryOpenDbLostAndFailedCallback();
+}
+
+TEST_F(MySqlConfigBackendDHCPv4DbLostCallbackTest, testRetryOpenDbLostAndFailedCallbackMultiThreading) {
+    MultiThreadingTest mt(true);
+    testRetryOpenDbLostAndFailedCallback();
+}
+
+TEST_F(MySqlConfigBackendDHCPv4DbLostCallbackTest, testRetryOpenDbLostAndRecoveredAfterTimeoutCallback) {
+    MultiThreadingTest mt(false);
+    testRetryOpenDbLostAndRecoveredAfterTimeoutCallback();
+}
+
+TEST_F(MySqlConfigBackendDHCPv4DbLostCallbackTest, testRetryOpenDbLostAndRecoveredAfterTimeoutCallbackMultiThreading) {
+    MultiThreadingTest mt(true);
+    testRetryOpenDbLostAndRecoveredAfterTimeoutCallback();
+}
+
+TEST_F(MySqlConfigBackendDHCPv4DbLostCallbackTest, testRetryOpenDbLostAndFailedAfterTimeoutCallback) {
+    MultiThreadingTest mt(false);
+    testRetryOpenDbLostAndFailedAfterTimeoutCallback();
+}
+
+TEST_F(MySqlConfigBackendDHCPv4DbLostCallbackTest, testRetryOpenDbLostAndFailedAfterTimeoutCallbackMultiThreading) {
+    MultiThreadingTest mt(true);
+    testRetryOpenDbLostAndFailedAfterTimeoutCallback();
+}
+
 TEST_F(MySqlConfigBackendDHCPv4DbLostCallbackTest, testNoCallbackOnOpenFailure) {
     MultiThreadingTest mt(false);
     testNoCallbackOnOpenFailure();

src/hooks/dhcp/mysql_cb/tests/mysql_cb_dhcp6_unittest.cc

@@ -466,6 +466,46 @@ public:
     }
 };
 
+TEST_F(MySqlConfigBackendDHCPv6DbLostCallbackTest, testRetryOpenDbLostAndRecoveredCallback) {
+    MultiThreadingTest mt(false);
+    testRetryOpenDbLostAndRecoveredCallback();
+}
+
+TEST_F(MySqlConfigBackendDHCPv6DbLostCallbackTest, testRetryOpenDbLostAndRecoveredCallbackMultiThreading) {
+    MultiThreadingTest mt(true);
+    testRetryOpenDbLostAndRecoveredCallback();
+}
+
+TEST_F(MySqlConfigBackendDHCPv6DbLostCallbackTest, testRetryOpenDbLostAndFailedCallback) {
+    MultiThreadingTest mt(false);
+    testRetryOpenDbLostAndFailedCallback();
+}
+
+TEST_F(MySqlConfigBackendDHCPv6DbLostCallbackTest, testRetryOpenDbLostAndFailedCallbackMultiThreading) {
+    MultiThreadingTest mt(true);
+    testRetryOpenDbLostAndFailedCallback();
+}
+
+TEST_F(MySqlConfigBackendDHCPv6DbLostCallbackTest, testRetryOpenDbLostAndRecoveredAfterTimeoutCallback) {
+    MultiThreadingTest mt(false);
+    testRetryOpenDbLostAndRecoveredAfterTimeoutCallback();
+}
+
+TEST_F(MySqlConfigBackendDHCPv6DbLostCallbackTest, testRetryOpenDbLostAndRecoveredAfterTimeoutCallbackMultiThreading) {
+    MultiThreadingTest mt(true);
+    testRetryOpenDbLostAndRecoveredAfterTimeoutCallback();
+}
+
+TEST_F(MySqlConfigBackendDHCPv6DbLostCallbackTest, testRetryOpenDbLostAndFailedAfterTimeoutCallback) {
+    MultiThreadingTest mt(false);
+    testRetryOpenDbLostAndFailedAfterTimeoutCallback();
+}
+
+TEST_F(MySqlConfigBackendDHCPv6DbLostCallbackTest, testRetryOpenDbLostAndFailedAfterTimeoutCallbackMultiThreading) {
+    MultiThreadingTest mt(true);
+    testRetryOpenDbLostAndFailedAfterTimeoutCallback();
+}
+
 TEST_F(MySqlConfigBackendDHCPv6DbLostCallbackTest, testNoCallbackOnOpenFailure) {
     MultiThreadingTest mt(false);
     testNoCallbackOnOpenFailure();

src/hooks/dhcp/pgsql_cb/pgsql_cb_dhcp4.cc

@@ -27,6 +27,7 @@
 #include <dhcpsrv/parsers/client_class_def_parser.h>
 #include <util/buffer.h>
 #include <util/boost_time_utils.h>
+#include <util/dhcp_space.h>
 #include <util/multi_threading_mgr.h>
 #include <pgsql/pgsql_connection.h>
 #include <boost/date_time/posix_time/posix_time.hpp>
@@ -4555,7 +4556,8 @@ TaggedStatementArray tagged_statements = { {
 } // end anonymous namespace
 
 PgSqlConfigBackendDHCPv4Impl::PgSqlConfigBackendDHCPv4Impl(const DatabaseConnection::ParameterMap& parameters)
-    : PgSqlConfigBackendImpl(parameters, &PgSqlConfigBackendDHCPv4Impl::dbReconnect,
+    : PgSqlConfigBackendImpl(std::string(cStringDhcpSpace<DHCPv4>()), parameters,
+                             &PgSqlConfigBackendDHCPv4Impl::dbReconnect,
       PgSqlConfigBackendDHCPv4Impl::GET_LAST_INSERT_ID4) {
     // Prepare query statements. Those are will be only used to retrieve
     // information from the database, so they can be used even if the
@@ -4565,14 +4567,6 @@ PgSqlConfigBackendDHCPv4Impl::PgSqlConfigBackendDHCPv4Impl(const DatabaseConnect
 // @todo As part of enabling read-only CB access, statements need to
 // be limited:
 //                            tagged_statements.begin() + WRITE_STMTS_BEGIN);
-
-    // Create unique timer name per instance.
-    timer_name_ = "PgSqlConfigBackend4[";
-    timer_name_ += boost::lexical_cast<std::string>(reinterpret_cast<uint64_t>(this));
-    timer_name_ += "]DbReconnectTimer";
-
-    // Create ReconnectCtl for this connection.
-    conn_.makeReconnectCtl(timer_name_);
 }
 
 PgSqlConfigBackendDHCPv4Impl::~PgSqlConfigBackendDHCPv4Impl() {

src/hooks/dhcp/pgsql_cb/pgsql_cb_dhcp6.cc

@@ -28,6 +28,7 @@
 #include <dhcpsrv/parsers/client_class_def_parser.h>
 #include <util/buffer.h>
 #include <util/boost_time_utils.h>
+#include <util/dhcp_space.h>
 #include <util/multi_threading_mgr.h>
 #include <util/triplet.h>
 #include <pgsql/pgsql_connection.h>
@@ -5016,7 +5017,8 @@ TaggedStatementArray tagged_statements = { {
 } // end anonymous namespace
 
 PgSqlConfigBackendDHCPv6Impl::PgSqlConfigBackendDHCPv6Impl(const DatabaseConnection::ParameterMap& parameters)
-    : PgSqlConfigBackendImpl(parameters, &PgSqlConfigBackendDHCPv6Impl::dbReconnect,
+    : PgSqlConfigBackendImpl(std::string(cStringDhcpSpace<DHCPv6>()), parameters,
+                             &PgSqlConfigBackendDHCPv6Impl::dbReconnect,
       PgSqlConfigBackendDHCPv6Impl::GET_LAST_INSERT_ID6) {
     // Prepare query statements. Those are will be only used to retrieve
     // information from the database, so they can be used even if the
@@ -5026,14 +5028,6 @@ PgSqlConfigBackendDHCPv6Impl::PgSqlConfigBackendDHCPv6Impl(const DatabaseConnect
 // @todo As part of enabling read-only CB access, statements need to
 // be limited:
 //                            tagged_statements.begin() + WRITE_STMTS_BEGIN);
-
-    // Create unique timer name per instance.
-    timer_name_ = "PgSqlConfigBackend6[";
-    timer_name_ += boost::lexical_cast<std::string>(reinterpret_cast<uint64_t>(this));
-    timer_name_ += "]DbReconnectTimer";
-
-    // Create ReconnectCtl for this connection.
-    conn_.makeReconnectCtl(timer_name_);
 }
 
 PgSqlConfigBackendDHCPv6Impl::~PgSqlConfigBackendDHCPv6Impl() {

src/hooks/dhcp/pgsql_cb/pgsql_cb_impl.cc

@@ -8,8 +8,9 @@
 
 #include <asiolink/io_address.h>
 #include <config_backend/constants.h>
-#include <dhcp/option_space.h>
 #include <database/db_exceptions.h>
+#include <dhcp/option_space.h>
+#include <dhcpsrv/timer_mgr.h>
 #include <pgsql/pgsql_exchange.h>
 #include <util/buffer.h>
 
@@ -70,15 +71,26 @@ PgSqlConfigBackendImpl::ScopedAuditRevision::~ScopedAuditRevision() {
     impl_->clearAuditRevision();
 }
 
-PgSqlConfigBackendImpl::PgSqlConfigBackendImpl(const DatabaseConnection::ParameterMap& parameters,
+PgSqlConfigBackendImpl::PgSqlConfigBackendImpl(const std::string& space,
+                                               const DatabaseConnection::ParameterMap& parameters,
                                                const DbCallback db_reconnect_callback,
                                                size_t last_insert_id_index)
     : conn_(parameters,
-            IOServiceAccessorPtr(new IOServiceAccessor(PgSqlConfigBackendImpl::getIOService)),
+            IOServiceAccessorPtr(new IOServiceAccessor(&PgSqlConfigBackendImpl::getIOService)),
             db_reconnect_callback), timer_name_(""),
             audit_revision_ref_count_(0), parameters_(parameters),
             last_insert_id_index_(last_insert_id_index) {
 
+    // Create unique timer name per instance.
+    timer_name_ = "PgSqlConfigBackend";
+    timer_name_ += space;
+    timer_name_ += "[";
+    timer_name_ += boost::lexical_cast<std::string>(reinterpret_cast<uint64_t>(this));
+    timer_name_ += "]DbReconnectTimer";
+
+    // Create ReconnectCtl for this connection.
+    conn_.makeReconnectCtl(timer_name_);
+
     // Check TLS support.
     size_t tls(0);
     tls += parameters.count("trust-anchor");
@@ -102,8 +114,24 @@ PgSqlConfigBackendImpl::PgSqlConfigBackendImpl(const DatabaseConnection::Paramet
 #endif
 
     // Test schema version first.
-    std::pair<uint32_t, uint32_t> code_version(PGSQL_SCHEMA_VERSION_MAJOR, PGSQL_SCHEMA_VERSION_MINOR);
+    std::pair<uint32_t, uint32_t> code_version(PGSQL_SCHEMA_VERSION_MAJOR,
-    std::pair<uint32_t, uint32_t> db_version = PgSqlConnection::getVersion(parameters);
+                                               PGSQL_SCHEMA_VERSION_MINOR);
+
+    std::string timer_name = "";
+    bool retry = false;
+    if (parameters.count("retry-on-startup")) {
+        if (parameters.at("retry-on-startup") == "true") {
+            retry = true;
+        }
+    }
+    if (retry) {
+        timer_name = timer_name_;
+    }
+
+    IOServiceAccessorPtr ac(new IOServiceAccessor(&DatabaseConnection::getIOService));
+
+    std::pair<uint32_t, uint32_t> db_version =
+        PgSqlConnection::getVersion(parameters, ac, db_reconnect_callback, timer_name);
     if (code_version != db_version) {
         isc_throw(DbOpenError, "PostgreSQL schema version mismatch: need version: "
                   << code_version.first << "." << code_version.second

src/hooks/dhcp/pgsql_cb/pgsql_cb_impl.h

@@ -106,12 +106,14 @@ public:
 
     /// @brief Constructor.
     ///
+    /// @param space The DHCP space (v4 or v6).
     /// @param parameters A data structure relating keywords and values
     /// concerned with the database.
     /// @param db_reconnect_callback The connection recovery callback.
     /// @param last_insert_id_index statement index of the SQL statement to
     /// use when fetching the last insert id for a given table.
-    explicit PgSqlConfigBackendImpl(const db::DatabaseConnection::ParameterMap& parameters,
+    explicit PgSqlConfigBackendImpl(const std::string& space,
+                                    const db::DatabaseConnection::ParameterMap& parameters,
                                     const db::DbCallback db_reconnect_callback,
                                     const size_t last_insert_id_index);
 

src/hooks/dhcp/pgsql_cb/tests/pgsql_cb_dhcp4_unittest.cc

@@ -460,6 +460,46 @@ public:
     }
 };
 
+TEST_F(PgSqlConfigBackendDHCPv4DbLostCallbackTest, testRetryOpenDbLostAndRecoveredCallback) {
+    MultiThreadingTest mt(false);
+    testRetryOpenDbLostAndRecoveredCallback();
+}
+
+TEST_F(PgSqlConfigBackendDHCPv4DbLostCallbackTest, testRetryOpenDbLostAndRecoveredCallbackMultiThreading) {
+    MultiThreadingTest mt(true);
+    testRetryOpenDbLostAndRecoveredCallback();
+}
+
+TEST_F(PgSqlConfigBackendDHCPv4DbLostCallbackTest, testRetryOpenDbLostAndFailedCallback) {
+    MultiThreadingTest mt(false);
+    testRetryOpenDbLostAndFailedCallback();
+}
+
+TEST_F(PgSqlConfigBackendDHCPv4DbLostCallbackTest, testRetryOpenDbLostAndFailedCallbackMultiThreading) {
+    MultiThreadingTest mt(true);
+    testRetryOpenDbLostAndFailedCallback();
+}
+
+TEST_F(PgSqlConfigBackendDHCPv4DbLostCallbackTest, testRetryOpenDbLostAndRecoveredAfterTimeoutCallback) {
+    MultiThreadingTest mt(false);
+    testRetryOpenDbLostAndRecoveredAfterTimeoutCallback();
+}
+
+TEST_F(PgSqlConfigBackendDHCPv4DbLostCallbackTest, testRetryOpenDbLostAndRecoveredAfterTimeoutCallbackMultiThreading) {
+    MultiThreadingTest mt(true);
+    testRetryOpenDbLostAndRecoveredAfterTimeoutCallback();
+}
+
+TEST_F(PgSqlConfigBackendDHCPv4DbLostCallbackTest, testRetryOpenDbLostAndFailedAfterTimeoutCallback) {
+    MultiThreadingTest mt(false);
+    testRetryOpenDbLostAndFailedAfterTimeoutCallback();
+}
+
+TEST_F(PgSqlConfigBackendDHCPv4DbLostCallbackTest, testRetryOpenDbLostAndFailedAfterTimeoutCallbackMultiThreading) {
+    MultiThreadingTest mt(true);
+    testRetryOpenDbLostAndFailedAfterTimeoutCallback();
+}
+
 TEST_F(PgSqlConfigBackendDHCPv4DbLostCallbackTest, testNoCallbackOnOpenFailure) {
     MultiThreadingTest mt(false);
     testNoCallbackOnOpenFailure();

src/hooks/dhcp/pgsql_cb/tests/pgsql_cb_dhcp6_unittest.cc

@@ -464,6 +464,46 @@ public:
     }
 };
 
+TEST_F(PgSqlConfigBackendDHCPv6DbLostCallbackTest, testRetryOpenDbLostAndRecoveredCallback) {
+    MultiThreadingTest mt(false);
+    testRetryOpenDbLostAndRecoveredCallback();
+}
+
+TEST_F(PgSqlConfigBackendDHCPv6DbLostCallbackTest, testRetryOpenDbLostAndRecoveredCallbackMultiThreading) {
+    MultiThreadingTest mt(true);
+    testRetryOpenDbLostAndRecoveredCallback();
+}
+
+TEST_F(PgSqlConfigBackendDHCPv6DbLostCallbackTest, testRetryOpenDbLostAndFailedCallback) {
+    MultiThreadingTest mt(false);
+    testRetryOpenDbLostAndFailedCallback();
+}
+
+TEST_F(PgSqlConfigBackendDHCPv6DbLostCallbackTest, testRetryOpenDbLostAndFailedCallbackMultiThreading) {
+    MultiThreadingTest mt(true);
+    testRetryOpenDbLostAndFailedCallback();
+}
+
+TEST_F(PgSqlConfigBackendDHCPv6DbLostCallbackTest, testRetryOpenDbLostAndRecoveredAfterTimeoutCallback) {
+    MultiThreadingTest mt(false);
+    testRetryOpenDbLostAndRecoveredAfterTimeoutCallback();
+}
+
+TEST_F(PgSqlConfigBackendDHCPv6DbLostCallbackTest, testRetryOpenDbLostAndRecoveredAfterTimeoutCallbackMultiThreading) {
+    MultiThreadingTest mt(true);
+    testRetryOpenDbLostAndRecoveredAfterTimeoutCallback();
+}
+
+TEST_F(PgSqlConfigBackendDHCPv6DbLostCallbackTest, testRetryOpenDbLostAndFailedAfterTimeoutCallback) {
+    MultiThreadingTest mt(false);
+    testRetryOpenDbLostAndFailedAfterTimeoutCallback();
+}
+
+TEST_F(PgSqlConfigBackendDHCPv6DbLostCallbackTest, testRetryOpenDbLostAndFailedAfterTimeoutCallbackMultiThreading) {
+    MultiThreadingTest mt(true);
+    testRetryOpenDbLostAndFailedAfterTimeoutCallback();
+}
+
 TEST_F(PgSqlConfigBackendDHCPv6DbLostCallbackTest, testNoCallbackOnOpenFailure) {
     MultiThreadingTest mt(false);
     testNoCallbackOnOpenFailure();

src/hooks/dhcp/pgsql_cb/tests/pgsql_cb_impl_unittest.cc

@@ -35,7 +35,7 @@ public:
         params["name"] = "keatest";
         params["password"] = "keatest";
         params["user"] = "keatest";
-        ASSERT_NO_THROW_LOG(cbptr_.reset(new PgSqlConfigBackendImpl(params, 0, 0)));
+        ASSERT_NO_THROW_LOG(cbptr_.reset(new PgSqlConfigBackendImpl("", params, 0, 0)));
     }
 
     /// @brief Cleans up after each test.

src/lib/database/database_connection.cc

@@ -18,6 +18,7 @@
 #include <boost/foreach.hpp>
 #include <vector>
 
+using namespace isc::asiolink;
 using namespace isc::util;
 using namespace std;
 
@@ -257,6 +258,7 @@ DatabaseConnection::toElement(const ParameterMap& params) {
                    (keyword == "host") ||
                    (keyword == "name") ||
                    (keyword == "on-fail") ||
+                   (keyword == "retry-on-startup") ||
                    (keyword == "trust-anchor") ||
                    (keyword == "cert-file") ||
                    (keyword == "key-file") ||
@@ -280,6 +282,8 @@ DatabaseConnection::toElementDbAccessString(const std::string& dbaccess) {
 DbCallback DatabaseConnection::db_lost_callback_ = 0;
 DbCallback DatabaseConnection::db_recovered_callback_ = 0;
 DbCallback DatabaseConnection::db_failed_callback_ = 0;
+bool DatabaseConnection::retry_ = false;
+IOServicePtr DatabaseConnection::io_service_ = IOServicePtr();
 
 }  // namespace db
 }  // namespace isc

src/lib/database/database_connection.h

@@ -34,6 +34,13 @@ public:
         isc::Exception(file, line, what) {}
 };
 
+/// @brief Exception thrown on failure to open database but permit retries
+class DbOpenErrorWithRetry : public Exception {
+public:
+    DbOpenErrorWithRetry(const char* file, size_t line, const char* what) :
+        isc::Exception(file, line, what) {}
+};
+
 /// @brief Exception thrown on failure to execute a database function
 class DbOperationError : public Exception {
 public:
@@ -215,6 +222,18 @@ public:
     /// @return a pointer to configuration
     static isc::data::ElementPtr toElementDbAccessString(const std::string& dbaccess);
 
+    /// @brief Sets IO service to be used by the Lease Manager.
+    ///
+    /// @param io_service IOService object, used for all ASIO operations.
+    static void setIOService(const isc::asiolink::IOServicePtr& io_service) {
+        io_service_ = io_service;
+    }
+
+    /// @brief Returns pointer to the IO service.
+    static isc::asiolink::IOServicePtr& getIOService() {
+        return (io_service_);
+    }
+
     /// @brief Optional callback function to invoke if an opened connection is
     /// lost
     static DbCallback db_lost_callback_;
@@ -227,6 +246,14 @@ public:
     /// recovery failed
     static DbCallback db_failed_callback_;
 
+    /// @brief Flag which indicated if retry database connection on fail should
+    /// be attempted.
+    ///
+    /// Allow only the first database connection attempt to fail and start
+    /// recovery. Sequential tries invoked by the dbReconnect callback should
+    /// not start yet another database connection attempt.
+    static bool retry_;
+
     /// @brief Throws an exception if the connection is not usable.
     /// @throw DbConnectionUnusable
     void checkUnusable() {
@@ -243,6 +270,7 @@ public:
     }
 
 protected:
+
     /// @brief Sets the unusable flag to true.
     void markUnusable() { unusable_ = true; }
 
@@ -272,6 +300,27 @@ private:
 
     /// @brief Reconnect settings.
     util::ReconnectCtlPtr reconnect_ctl_;
+
+    /// The IOService object, used for all ASIO operations.
+    static isc::asiolink::IOServicePtr io_service_;
+};
+
+/// @brief RAII class to enable DB reconnect retries on server startup.
+class DbConnectionInitWithRetry {
+public:
+    /// @brief Constructor.
+    ///
+    /// Enable DB reconnect retries on server startup.
+    DbConnectionInitWithRetry() {
+        DatabaseConnection::retry_ = true;
+    }
+
+    /// @brief Destructor.
+    ///
+    /// Disable DB reconnect retries.
+    ~DbConnectionInitWithRetry() {
+        DatabaseConnection::retry_ = false;
+    }
 };
 
 }  // namespace db

src/lib/database/dbaccess_parser.cc

@@ -61,7 +61,8 @@ DbAccessParser::parse(std::string& access_string,
     for (std::pair<std::string, ConstElementPtr> param : database_config->mapValue()) {
         try {
             if ((param.first == "persist") ||
-                (param.first == "readonly")) {
+                (param.first == "readonly") ||
+                (param.first == "retry-on-startup")) {
                 values_copy[param.first] = (param.second->boolValue() ?
                                             "true" : "false");
 

src/lib/dhcpsrv/cfg_db_access.cc

@@ -5,8 +5,10 @@
 // file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 #include <config.h>
+#include <database/database_connection.h>
 #include <dhcpsrv/cfg_db_access.h>
 #include <dhcpsrv/db_type.h>
+#include <dhcpsrv/dhcpsrv_log.h>
 #include <dhcpsrv/host_data_source_factory.h>
 #include <dhcpsrv/host_mgr.h>
 #include <dhcpsrv/lease_mgr_factory.h>
@@ -17,6 +19,7 @@
 #include <vector>
 
 using namespace isc::data;
+using namespace isc::db;
 
 namespace isc {
 namespace dhcp {
@@ -55,8 +58,19 @@ CfgDbAccess::getHostDbAccessStringList() const {
 
 void
 CfgDbAccess::createManagers() const {
+    try {
         // Recreate lease manager without preserving the registered callbacks.
         LeaseMgrFactory::recreate(getLeaseDbAccessString(), false);
+    } catch (const isc::db::DbOpenErrorWithRetry& err) {
+        std::string redacted;
+        try {
+            DatabaseConnection::ParameterMap parameters = DatabaseConnection::parse(getLeaseDbAccessString());
+            redacted = DatabaseConnection::redactedAccessString(parameters);
+        } catch (...) {
+        }
+        LOG_INFO(dhcpsrv_logger, DHCPSRV_LEASE_MGR_DB_OPEN_CONNECTION_WITH_RETRY_FAILED)
+                .arg(redacted).arg(err.what());
+    }
 
     // Recreate host data source.
     HostMgr::create();
@@ -69,7 +83,18 @@ CfgDbAccess::createManagers() const {
     // Add database backends.
     std::list<std::string> host_db_access_list = getHostDbAccessStringList();
     for (std::string& hds : host_db_access_list) {
+        try {
             HostMgr::addBackend(hds);
+        } catch (const isc::db::DbOpenErrorWithRetry& err) {
+            std::string redacted;
+            try {
+                DatabaseConnection::ParameterMap parameters = DatabaseConnection::parse(hds);
+                redacted = DatabaseConnection::redactedAccessString(parameters);
+            } catch (...) {
+            }
+            LOG_INFO(dhcpsrv_logger, DHCPSRV_HOST_MGR_DB_OPEN_CONNECTION_WITH_RETRY_FAILED)
+                    .arg(redacted).arg(err.what());
+        }
     }
 
     // Check for a host cache.

src/lib/dhcpsrv/dhcpsrv_messages.cc

@@ -71,6 +71,7 @@ extern const isc::log::MessageID DHCPSRV_HOOK_LEASE4_SELECT_SKIP = "DHCPSRV_HOOK
 extern const isc::log::MessageID DHCPSRV_HOOK_LEASE6_EXTEND_SKIP = "DHCPSRV_HOOK_LEASE6_EXTEND_SKIP";
 extern const isc::log::MessageID DHCPSRV_HOOK_LEASE6_RECOVER_SKIP = "DHCPSRV_HOOK_LEASE6_RECOVER_SKIP";
 extern const isc::log::MessageID DHCPSRV_HOOK_LEASE6_SELECT_SKIP = "DHCPSRV_HOOK_LEASE6_SELECT_SKIP";
+extern const isc::log::MessageID DHCPSRV_HOST_MGR_DB_OPEN_CONNECTION_WITH_RETRY_FAILED = "DHCPSRV_HOST_MGR_DB_OPEN_CONNECTION_WITH_RETRY_FAILED";
 extern const isc::log::MessageID DHCPSRV_INVALID_ACCESS = "DHCPSRV_INVALID_ACCESS";
 extern const isc::log::MessageID DHCPSRV_LEASE4_EXTENDED_INFO_SANITY_FAIL = "DHCPSRV_LEASE4_EXTENDED_INFO_SANITY_FAIL";
 extern const isc::log::MessageID DHCPSRV_LEASE4_EXTENDED_INFO_UPGRADED = "DHCPSRV_LEASE4_EXTENDED_INFO_UPGRADED";
@@ -78,6 +79,7 @@ extern const isc::log::MessageID DHCPSRV_LEASE6_EXTENDED_INFO_SANITY_FAIL = "DHC
 extern const isc::log::MessageID DHCPSRV_LEASE6_EXTENDED_INFO_UPGRADED = "DHCPSRV_LEASE6_EXTENDED_INFO_UPGRADED";
 extern const isc::log::MessageID DHCPSRV_LEASE_MGR_CALLBACK_EXCEPTION = "DHCPSRV_LEASE_MGR_CALLBACK_EXCEPTION";
 extern const isc::log::MessageID DHCPSRV_LEASE_MGR_CALLBACK_UNKNOWN_EXCEPTION = "DHCPSRV_LEASE_MGR_CALLBACK_UNKNOWN_EXCEPTION";
+extern const isc::log::MessageID DHCPSRV_LEASE_MGR_DB_OPEN_CONNECTION_WITH_RETRY_FAILED = "DHCPSRV_LEASE_MGR_DB_OPEN_CONNECTION_WITH_RETRY_FAILED";
 extern const isc::log::MessageID DHCPSRV_LEASE_SANITY_FAIL = "DHCPSRV_LEASE_SANITY_FAIL";
 extern const isc::log::MessageID DHCPSRV_LEASE_SANITY_FAIL_DISCARD = "DHCPSRV_LEASE_SANITY_FAIL_DISCARD";
 extern const isc::log::MessageID DHCPSRV_LEASE_SANITY_FIXED = "DHCPSRV_LEASE_SANITY_FIXED";
@@ -355,6 +357,7 @@ const char* values[] = {
     "DHCPSRV_HOOK_LEASE6_EXTEND_SKIP", "DHCPv6 lease lifetime was not extended because a callout set the skip flag for message %1",
     "DHCPSRV_HOOK_LEASE6_RECOVER_SKIP", "DHCPv6 lease %1 was not recovered from declined state because a callout set the skip status.",
     "DHCPSRV_HOOK_LEASE6_SELECT_SKIP", "Lease6 (non-temporary) creation was skipped, because of callout skip flag.",
+    "DHCPSRV_HOST_MGR_DB_OPEN_CONNECTION_WITH_RETRY_FAILED", "Failed to connect to database: %1 with error: %2",
     "DHCPSRV_INVALID_ACCESS", "invalid database access string: %1",
     "DHCPSRV_LEASE4_EXTENDED_INFO_SANITY_FAIL", "extended info for lease %1 failed checks (%2)",
     "DHCPSRV_LEASE4_EXTENDED_INFO_UPGRADED", "extended info for lease %1 was upgraded",
@@ -362,6 +365,7 @@ const char* values[] = {
     "DHCPSRV_LEASE6_EXTENDED_INFO_UPGRADED", "extended info for lease %1 was upgraded",
     "DHCPSRV_LEASE_MGR_CALLBACK_EXCEPTION", "exception occurred in a lease manager callback for callback type %1, subnet id %2, and lease %3: %4",
     "DHCPSRV_LEASE_MGR_CALLBACK_UNKNOWN_EXCEPTION", "unknown exception occurred in a lease manager callback for callback type %1, subnet id %2, and lease %3",
+    "DHCPSRV_LEASE_MGR_DB_OPEN_CONNECTION_WITH_RETRY_FAILED", "Failed to connect to database: %1 with error: %2",
     "DHCPSRV_LEASE_SANITY_FAIL", "The lease %1 with subnet-id %2 failed subnet-id checks (%3).",
     "DHCPSRV_LEASE_SANITY_FAIL_DISCARD", "The lease %1 with subnet-id %2 failed subnet-id checks (%3) and was dropped.",
     "DHCPSRV_LEASE_SANITY_FIXED", "The lease %1 with subnet-id %2 failed subnet-id checks, but was corrected to subnet-id %3.",

src/lib/dhcpsrv/dhcpsrv_messages.h

@@ -72,6 +72,7 @@ extern const isc::log::MessageID DHCPSRV_HOOK_LEASE4_SELECT_SKIP;
 extern const isc::log::MessageID DHCPSRV_HOOK_LEASE6_EXTEND_SKIP;
 extern const isc::log::MessageID DHCPSRV_HOOK_LEASE6_RECOVER_SKIP;
 extern const isc::log::MessageID DHCPSRV_HOOK_LEASE6_SELECT_SKIP;
+extern const isc::log::MessageID DHCPSRV_HOST_MGR_DB_OPEN_CONNECTION_WITH_RETRY_FAILED;
 extern const isc::log::MessageID DHCPSRV_INVALID_ACCESS;
 extern const isc::log::MessageID DHCPSRV_LEASE4_EXTENDED_INFO_SANITY_FAIL;
 extern const isc::log::MessageID DHCPSRV_LEASE4_EXTENDED_INFO_UPGRADED;
@@ -79,6 +80,7 @@ extern const isc::log::MessageID DHCPSRV_LEASE6_EXTENDED_INFO_SANITY_FAIL;
 extern const isc::log::MessageID DHCPSRV_LEASE6_EXTENDED_INFO_UPGRADED;
 extern const isc::log::MessageID DHCPSRV_LEASE_MGR_CALLBACK_EXCEPTION;
 extern const isc::log::MessageID DHCPSRV_LEASE_MGR_CALLBACK_UNKNOWN_EXCEPTION;
+extern const isc::log::MessageID DHCPSRV_LEASE_MGR_DB_OPEN_CONNECTION_WITH_RETRY_FAILED;
 extern const isc::log::MessageID DHCPSRV_LEASE_SANITY_FAIL;
 extern const isc::log::MessageID DHCPSRV_LEASE_SANITY_FAIL_DISCARD;
 extern const isc::log::MessageID DHCPSRV_LEASE_SANITY_FIXED;

src/lib/dhcpsrv/dhcpsrv_messages.mes

@@ -341,6 +341,12 @@ have been experienced. Any such errors should have preceding entries in the
 log with details. No further attempts to communicate with kea-dhcp-ddns will
 be made without intervention.
 
+% DHCPSRV_HOST_MGR_DB_OPEN_CONNECTION_WITH_RETRY_FAILED Failed to connect to database: %1 with error: %2
+This is an informational message issued when the the server failed to connect to
+the host database. The operation started a retry to connect procedure.
+The database access string with password redacted is logged, along with the
+error and details for the reconnect procedure.
+
 % DHCPSRV_HOOK_LEASE4_RECOVER_SKIP DHCPv4 lease %1 was not recovered from the declined state because a callout set the skip status.
 This debug message is printed when a callout installed on lease4_recover
 hook point set the next step status to SKIP. For this particular hook point, this
@@ -424,6 +430,12 @@ The third argument prints the lease for which the error has occurred.
 This log message variant contains no error text because it is triggered
 by an unknown exception.
 
+% DHCPSRV_LEASE_MGR_DB_OPEN_CONNECTION_WITH_RETRY_FAILED Failed to connect to database: %1 with error: %2
+This is an informational message issued when the the server failed to connect to
+the lease database. The operation started a retry to connect procedure.
+The database access string with password redacted is logged, along with the
+error and details for the reconnect procedure.
+
 % DHCPSRV_LEASE_SANITY_FAIL The lease %1 with subnet-id %2 failed subnet-id checks (%3).
 This warning message is printed when the lease being loaded does not match the
 configuration. Due to lease-checks value, the lease will be loaded, but

src/lib/dhcpsrv/host_mgr.cc

@@ -42,8 +42,6 @@ namespace dhcp {
 using namespace isc::asiolink;
 using namespace isc::db;
 
-IOServicePtr HostMgr::io_service_ = IOServicePtr();
-
 boost::scoped_ptr<HostMgr>&
 HostMgr::getHostMgrPtr() {
     static boost::scoped_ptr<HostMgr> host_mgr_ptr;

src/lib/dhcpsrv/host_mgr.h

@@ -832,18 +832,6 @@ public:
         return (ip_reservations_unique_);
     }
 
-    /// @brief Sets IO service to be used by the Host Manager.
-    ///
-    /// @param io_service IOService object, used for all ASIO operations.
-    static void setIOService(const isc::asiolink::IOServicePtr& io_service) {
-        io_service_ = io_service;
-    }
-
-    /// @brief Returns pointer to the IO service.
-    static isc::asiolink::IOServicePtr& getIOService() {
-        return (io_service_);
-    }
-
 protected:
 
     /// @brief The negative caching flag.
@@ -901,9 +889,6 @@ private:
     /// @brief Returns a pointer to the currently used instance of the
     /// @c HostMgr.
     static boost::scoped_ptr<HostMgr>& getHostMgrPtr();
-
-    /// The IOService object, used for all ASIO operations.
-    static isc::asiolink::IOServicePtr io_service_;
 };
 
 }  // namespace dhcp

src/lib/dhcpsrv/lease_mgr.cc

@@ -38,8 +38,6 @@ using namespace std;
 namespace isc {
 namespace dhcp {
 
-IOServicePtr LeaseMgr::io_service_ = IOServicePtr();
-
 LeasePageSize::LeasePageSize(const size_t page_size)
     : page_size_(page_size) {
 

src/lib/dhcpsrv/lease_mgr.h

@@ -805,6 +805,7 @@ public:
 
     /// @brief Returns backend version.
     ///
+    /// @param timer_name The DB reconnect timer name.
     /// @return Version number as a pair of unsigned integers.  "first" is the
     ///         major version number, "second" the minor number.
     ///
@@ -817,7 +818,7 @@ public:
     /// B>=A and B=C (it is ok to have newer backend, as it should be backward
     /// compatible)
     /// Also if B>C, some database upgrade procedure may be triggered
-    virtual VersionPair getVersion() const = 0;
+    virtual VersionPair getVersion(const std::string& timer_name = "") const = 0;
 
     /// @brief Commit Transactions
     ///
@@ -831,18 +832,6 @@ public:
     /// support transactions, this is a no-op.
     virtual void rollback() = 0;
 
-    /// @brief Sets IO service to be used by the Lease Manager.
-    ///
-    /// @param io_service IOService object, used for all ASIO operations.
-    static void setIOService(const isc::asiolink::IOServicePtr& io_service) {
-        io_service_ = io_service;
-    }
-
-    /// @brief Returns pointer to the IO service.
-    static isc::asiolink::IOServicePtr& getIOService() {
-        return (io_service_);
-    }
-
     // -- The following are memfile only, but defined in the base LeaseMgr for convenience. --
 
     /// @brief Returns the class lease count for a given class and lease type.
@@ -1101,8 +1090,6 @@ protected:
                               const std::vector<uint8_t>& remote_id) = 0;
 
 private:
-    /// The IOService object, used for all ASIO operations.
-    static isc::asiolink::IOServicePtr io_service_;
 
     /// @brief Holds the setting whether the lease extended info tables
     /// are enabled or disabled. The default is disabled.

src/lib/dhcpsrv/memfile_lease_mgr.cc

@@ -209,7 +209,7 @@ LFCSetup::setup(const uint32_t lfc_interval,
     args.push_back("ignored-path");
 
     // Create the process (do not start it yet).
-    process_.reset(new ProcessSpawn(LeaseMgr::getIOService(), executable, args));
+    process_.reset(new ProcessSpawn(DatabaseConnection::getIOService(), executable, args));
 
     // If we've been told to run it once now, invoke the callback directly.
     if (run_once_now) {
@@ -2113,7 +2113,7 @@ Memfile_LeaseMgr::getDescription() const {
 }
 
 std::pair<uint32_t, uint32_t>
-Memfile_LeaseMgr::getVersion() const {
+Memfile_LeaseMgr::getVersion(const std::string& /* timer_name */) const {
     std::string const& universe(conn_.getParameter("universe"));
     if (universe == "4") {
         return std::make_pair(MAJOR_VERSION_V4, MINOR_VERSION_V4);

src/lib/dhcpsrv/memfile_lease_mgr.h

@@ -898,9 +898,10 @@ public:
 
     /// @brief Returns backend version.
     ///
+    /// @param timer_name The DB reconnect timer name.
     /// @return Version number as a pair of unsigned integers.  "first" is the
     ///         major version number, "second" the minor number.
-    virtual std::pair<uint32_t, uint32_t> getVersion() const override;
+    virtual std::pair<uint32_t, uint32_t> getVersion(const std::string& /* timer_name */ = "") const override;
 
     /// @brief Commit Transactions
     ///

src/lib/dhcpsrv/mysql_host_data_source.cc

@@ -2171,13 +2171,14 @@ public:
     /// The method is called by the constructor before opening the database
     /// to verify that the schema version is correct.
     ///
+    /// @param timer_name The DB reconnect timer name.
     /// @return Version number stored in the database, as a pair of unsigned
     ///         integers. "first" is the major version number, "second" the
     ///         minor number.
     ///
     /// @throw isc::dhcp::DbOperationError An operation on the open database
     ///        has failed.
-    std::pair<uint32_t, uint32_t> getVersion() const;
+    std::pair<uint32_t, uint32_t> getVersion(const std::string& timer_name = "") const;
 
     /// @brief Executes statements which inserts a row into one of the tables.
     ///
@@ -2882,7 +2883,19 @@ MySqlHostDataSourceImpl::MySqlHostDataSourceImpl(const DatabaseConnection::Param
     // Validate the schema version first.
     std::pair<uint32_t, uint32_t> code_version(MYSQL_SCHEMA_VERSION_MAJOR,
                                                MYSQL_SCHEMA_VERSION_MINOR);
-    std::pair<uint32_t, uint32_t> db_version = getVersion();
+
+    std::string timer_name = "";
+    bool retry = false;
+    if (parameters.count("retry-on-startup")) {
+        if (parameters.at("retry-on-startup") == "true") {
+            retry = true;
+        }
+    }
+    if (retry) {
+        timer_name = timer_name_;
+    }
+
+    std::pair<uint32_t, uint32_t> db_version = getVersion(timer_name);
     if (code_version != db_version) {
         isc_throw(DbOpenError,
                   "MySQL schema version mismatch: need version: "
@@ -2901,7 +2914,7 @@ MySqlHostDataSourceImpl::MySqlHostDataSourceImpl(const DatabaseConnection::Param
 MySqlHostContextPtr
 MySqlHostDataSourceImpl::createContext() const {
     MySqlHostContextPtr ctx(new MySqlHostContext(parameters_,
-        IOServiceAccessorPtr(new IOServiceAccessor(&HostMgr::getIOService)),
+        IOServiceAccessorPtr(new IOServiceAccessor(&DatabaseConnection::getIOService)),
         &MySqlHostDataSourceImpl::dbReconnect));
 
     // Open the database.
@@ -3030,11 +3043,14 @@ MySqlHostDataSourceImpl::dbReconnect(ReconnectCtlPtr db_reconnect_ctl) {
 }
 
 std::pair<uint32_t, uint32_t>
-MySqlHostDataSourceImpl::getVersion() const {
+MySqlHostDataSourceImpl::getVersion(const std::string& timer_name) const {
     LOG_DEBUG(dhcpsrv_logger, DHCPSRV_DBG_TRACE_DETAIL,
               DHCPSRV_MYSQL_HOST_DB_GET_VERSION);
 
-    return (MySqlConnection::getVersion(parameters_));
+    IOServiceAccessorPtr ac(new IOServiceAccessor(&DatabaseConnection::getIOService));
+    DbCallback cb(&MySqlHostDataSourceImpl::dbReconnect);
+
+    return (MySqlConnection::getVersion(parameters_, ac, cb, timer_name));
 }
 
 void
@@ -4096,8 +4112,8 @@ MySqlHostDataSource::getDescription() const {
 }
 
 std::pair<uint32_t, uint32_t>
-MySqlHostDataSource::getVersion() const {
+MySqlHostDataSource::getVersion(const std::string& timer_name) const {
-    return(impl_->getVersion());
+    return(impl_->getVersion(timer_name));
 }
 
 void

src/lib/dhcpsrv/mysql_host_data_source.h

@@ -458,13 +458,14 @@ public:
 
     /// @brief Returns backend version.
     ///
+    /// @param timer_name The DB reconnect timer name.
     /// @return Version number stored in the database, as a pair of unsigned
     ///         integers. "first" is the major version number, "second" the
     ///         minor number.
     ///
     /// @throw isc::db::DbOperationError An operation on the open database
     ///        has failed.
-    virtual std::pair<uint32_t, uint32_t> getVersion() const;
+    virtual std::pair<uint32_t, uint32_t> getVersion(const std::string& timer_name = "") const;
 
     /// @brief Commit Transactions
     ///

src/lib/dhcpsrv/mysql_lease_mgr.cc

@@ -2198,7 +2198,19 @@ MySqlLeaseMgr::MySqlLeaseMgr(const DatabaseConnection::ParameterMap& parameters)
     // Validate schema version first.
     std::pair<uint32_t, uint32_t> code_version(MYSQL_SCHEMA_VERSION_MAJOR,
                                                MYSQL_SCHEMA_VERSION_MINOR);
-    std::pair<uint32_t, uint32_t> db_version = getVersion();
+
+    std::string timer_name = "";
+    bool retry = false;
+    if (parameters.count("retry-on-startup")) {
+        if (parameters.at("retry-on-startup") == "true") {
+            retry = true;
+        }
+    }
+    if (retry) {
+        timer_name = timer_name_;
+    }
+
+    std::pair<uint32_t, uint32_t> db_version = getVersion(timer_name);
     if (code_version != db_version) {
         isc_throw(DbOpenError,
                   "MySQL schema version mismatch: need version: "
@@ -2287,9 +2299,12 @@ MySqlLeaseMgr::dbReconnect(ReconnectCtlPtr db_reconnect_ctl) {
 MySqlLeaseContextPtr
 MySqlLeaseMgr::createContext() const {
     MySqlLeaseContextPtr ctx(new MySqlLeaseContext(parameters_,
-        IOServiceAccessorPtr(new IOServiceAccessor(&LeaseMgr::getIOService)),
+        IOServiceAccessorPtr(new IOServiceAccessor(&DatabaseConnection::getIOService)),
         &MySqlLeaseMgr::dbReconnect));
 
+    // Create ReconnectCtl for this connection.
+    ctx->conn_.makeReconnectCtl(timer_name_);
+
     // Open the database.
     ctx->conn_.openDatabase();
 
@@ -2314,9 +2329,6 @@ MySqlLeaseMgr::createContext() const {
     ctx->exchange4_.reset(new MySqlLease4Exchange());
     ctx->exchange6_.reset(new MySqlLease6Exchange());
 
-    // Create ReconnectCtl for this connection.
-    ctx->conn_.makeReconnectCtl(timer_name_);
-
     return (ctx);
 }
 
@@ -3824,10 +3836,13 @@ MySqlLeaseMgr::getDescription() const {
 }
 
 std::pair<uint32_t, uint32_t>
-MySqlLeaseMgr::getVersion() const {
+MySqlLeaseMgr::getVersion(const string& timer_name) const {
     LOG_DEBUG(dhcpsrv_logger, DHCPSRV_DBG_TRACE_DETAIL, DHCPSRV_MYSQL_GET_VERSION);
 
-    return (MySqlConnection::getVersion(parameters_));
+    IOServiceAccessorPtr ac(new IOServiceAccessor(&DatabaseConnection::getIOService));
+    DbCallback cb(&MySqlLeaseMgr::dbReconnect);
+
+    return (MySqlConnection::getVersion(parameters_, ac, cb, timer_name));
 }
 
 void

src/lib/dhcpsrv/mysql_lease_mgr.h

@@ -689,12 +689,13 @@ public:
 
     /// @brief Returns backend version.
     ///
+    /// @param timer_name The DB reconnect timer name.
     /// @return Version number as a pair of unsigned integers.  "first" is the
     ///         major version number, "second" the minor number.
     ///
     /// @throw isc::db::DbOperationError An operation on the open database has
     ///        failed.
-    virtual std::pair<uint32_t, uint32_t> getVersion() const override;
+    virtual std::pair<uint32_t, uint32_t> getVersion(const std::string& timer_name = "") const override;
 
     /// @brief Commit Transactions
     ///

src/lib/dhcpsrv/pgsql_host_data_source.cc

@@ -1613,13 +1613,14 @@ public:
 
     /// @brief Returns PostgreSQL schema version of the open database
     ///
+    /// @param timer_name The DB reconnect timer name.
     /// @return Version number stored in the database, as a pair of unsigned
     ///         integers. "first" is the major version number, "second" the
     ///         minor number.
     ///
     /// @throw isc::db::DbOperationError An operation on the open database
     ///        has failed.
-    std::pair<uint32_t, uint32_t> getVersion() const;
+    std::pair<uint32_t, uint32_t> getVersion(const std::string& timer_name = "") const;
 
     /// @brief The parameters
     DatabaseConnection::ParameterMap parameters_;
@@ -2299,7 +2300,19 @@ PgSqlHostDataSourceImpl::PgSqlHostDataSourceImpl(const DatabaseConnection::Param
     // Validate the schema version first.
     std::pair<uint32_t, uint32_t> code_version(PGSQL_SCHEMA_VERSION_MAJOR,
                                                PGSQL_SCHEMA_VERSION_MINOR);
-    std::pair<uint32_t, uint32_t> db_version = getVersion();
+
+    std::string timer_name = "";
+    bool retry = false;
+    if (parameters.count("retry-on-startup")) {
+        if (parameters.at("retry-on-startup") == "true") {
+            retry = true;
+        }
+    }
+    if (retry) {
+        timer_name = timer_name_;
+    }
+
+    std::pair<uint32_t, uint32_t> db_version = getVersion(timer_name);
     if (code_version != db_version) {
         isc_throw(DbOpenError,
                   "PostgreSQL schema version mismatch: need version: "
@@ -2318,7 +2331,7 @@ PgSqlHostDataSourceImpl::PgSqlHostDataSourceImpl(const DatabaseConnection::Param
 PgSqlHostContextPtr
 PgSqlHostDataSourceImpl::createContext() const {
     PgSqlHostContextPtr ctx(new PgSqlHostContext(parameters_,
-        IOServiceAccessorPtr(new IOServiceAccessor(&HostMgr::getIOService)),
+        IOServiceAccessorPtr(new IOServiceAccessor(&DatabaseConnection::getIOService)),
         &PgSqlHostDataSourceImpl::dbReconnect));
 
     // Open the database.
@@ -2616,10 +2629,14 @@ PgSqlHostDataSourceImpl::getHost(PgSqlHostContextPtr& ctx,
 }
 
 std::pair<uint32_t, uint32_t>
-PgSqlHostDataSourceImpl::getVersion() const {
+PgSqlHostDataSourceImpl::getVersion(const std::string& timer_name) const {
     LOG_DEBUG(dhcpsrv_logger, DHCPSRV_DBG_TRACE_DETAIL,
               DHCPSRV_PGSQL_HOST_DB_GET_VERSION);
-    return (PgSqlConnection::getVersion(parameters_));
+
+    IOServiceAccessorPtr ac(new IOServiceAccessor(&DatabaseConnection::getIOService));
+    DbCallback cb(&PgSqlHostDataSourceImpl::dbReconnect);
+
+    return (PgSqlConnection::getVersion(parameters_, ac, cb, timer_name));
 }
 
 void
@@ -3298,8 +3315,8 @@ PgSqlHostDataSource::getDescription() const {
 }
 
 std::pair<uint32_t, uint32_t>
-PgSqlHostDataSource::getVersion() const {
+PgSqlHostDataSource::getVersion(const std::string& timer_name) const {
-    return(impl_->getVersion());
+    return(impl_->getVersion(timer_name));
 }
 
 void

src/lib/dhcpsrv/pgsql_host_data_source.h

@@ -510,13 +510,14 @@ public:
     /// is correct. Thus it must not rely on a pre-prepared statement or
     /// formal statement execution error checking.
     ///
+    /// @param timer_name The DB reconnect timer name.
     /// @return Version number stored in the database, as a pair of unsigned
     ///         integers. "first" is the major version number, "second" the
     ///         minor number.
     ///
     /// @throw isc::db::DbOperationError An operation on the open database
     ///        has failed.
-    virtual std::pair<uint32_t, uint32_t> getVersion() const;
+    virtual std::pair<uint32_t, uint32_t> getVersion(const std::string& timer_name = "") const;
 
     /// @brief Commit Transactions
     ///

src/lib/dhcpsrv/pgsql_lease_mgr.cc

@@ -1647,7 +1647,19 @@ PgSqlLeaseMgr::PgSqlLeaseMgr(const DatabaseConnection::ParameterMap& parameters)
     // Validate schema version first.
     std::pair<uint32_t, uint32_t> code_version(PGSQL_SCHEMA_VERSION_MAJOR,
                                                PGSQL_SCHEMA_VERSION_MINOR);
-    std::pair<uint32_t, uint32_t> db_version = getVersion();
+
+    std::string timer_name = "";
+    bool retry = false;
+    if (parameters.count("retry-on-startup")) {
+        if (parameters.at("retry-on-startup") == "true") {
+            retry = true;
+        }
+    }
+    if (retry) {
+        timer_name = timer_name_;
+    }
+
+    std::pair<uint32_t, uint32_t> db_version = getVersion(timer_name);
     if (code_version != db_version) {
         isc_throw(DbOpenError,
                   "PostgreSQL schema version mismatch: need version: "
@@ -1736,9 +1748,12 @@ PgSqlLeaseMgr::dbReconnect(ReconnectCtlPtr db_reconnect_ctl) {
 PgSqlLeaseContextPtr
 PgSqlLeaseMgr::createContext() const {
     PgSqlLeaseContextPtr ctx(new PgSqlLeaseContext(parameters_,
-        IOServiceAccessorPtr(new IOServiceAccessor(&LeaseMgr::getIOService)),
+        IOServiceAccessorPtr(new IOServiceAccessor(&DatabaseConnection::getIOService)),
         &PgSqlLeaseMgr::dbReconnect));
 
+    // Create ReconnectCtl for this connection.
+    ctx->conn_.makeReconnectCtl(timer_name_);
+
     // Open the database.
     ctx->conn_.openDatabase();
 
@@ -1759,9 +1774,6 @@ PgSqlLeaseMgr::createContext() const {
     ctx->exchange4_.reset(new PgSqlLease4Exchange());
     ctx->exchange6_.reset(new PgSqlLease6Exchange());
 
-    // Create ReconnectCtl for this connection.
-    ctx->conn_.makeReconnectCtl(timer_name_);
-
     return (ctx);
 }
 
@@ -2989,10 +3001,13 @@ PgSqlLeaseMgr::getDescription() const {
 }
 
 std::pair<uint32_t, uint32_t>
-PgSqlLeaseMgr::getVersion() const {
+PgSqlLeaseMgr::getVersion(const string& timer_name) const {
     LOG_DEBUG(dhcpsrv_logger, DHCPSRV_DBG_TRACE_DETAIL, DHCPSRV_PGSQL_GET_VERSION);
 
-    return (PgSqlConnection::getVersion(parameters_));
+    IOServiceAccessorPtr ac(new IOServiceAccessor(&DatabaseConnection::getIOService));
+    DbCallback cb(&PgSqlLeaseMgr::dbReconnect);
+
+    return (PgSqlConnection::getVersion(parameters_, ac, cb, timer_name));
 }
 
 void

src/lib/dhcpsrv/pgsql_lease_mgr.h

@@ -665,12 +665,13 @@ public:
 
     /// @brief Returns backend version.
     ///
+    /// @param timer_name The DB reconnect timer name.
     /// @return Version number as a pair of unsigned integers.  "first" is the
     ///         major version number, "second" the minor number.
     ///
     /// @throw isc::db::DbOperationError An operation on the open database has
     ///        failed.
-    virtual std::pair<uint32_t, uint32_t> getVersion() const override;
+    virtual std::pair<uint32_t, uint32_t> getVersion(const std::string& timer_name = "") const override;
 
     /// @brief Commit Transactions
     ///

src/lib/dhcpsrv/tests/Makefile.am

@@ -107,7 +107,6 @@ libdhcpsrv_unittests_SOURCES += lease_file_loader_unittest.cc
 libdhcpsrv_unittests_SOURCES += lease_unittest.cc
 libdhcpsrv_unittests_SOURCES += lease_mgr_factory_unittest.cc
 libdhcpsrv_unittests_SOURCES += lease_mgr_unittest.cc
-libdhcpsrv_unittests_SOURCES += generic_lease_mgr_unittest.cc generic_lease_mgr_unittest.h
 libdhcpsrv_unittests_SOURCES += memfile_lease_extended_info_unittest.cc
 libdhcpsrv_unittests_SOURCES += memfile_lease_limits_unittest.cc
 libdhcpsrv_unittests_SOURCES += memfile_lease_mgr_unittest.cc

src/lib/dhcpsrv/tests/lease_mgr_unittest.cc

@@ -11,7 +11,7 @@
 #include <dhcpsrv/memfile_lease_mgr.h>
 #include <dhcpsrv/testutils/test_utils.h>
 #include <dhcpsrv/testutils/concrete_lease_mgr.h>
-#include <dhcpsrv/tests/generic_lease_mgr_unittest.h>
+#include <dhcpsrv/testutils/generic_lease_mgr_unittest.h>
 #include <testutils/gtest_utils.h>
 
 #include <iostream>

src/lib/dhcpsrv/tests/memfile_lease_mgr_unittest.cc

@@ -20,7 +20,7 @@
 #include <dhcpsrv/timer_mgr.h>
 #include <dhcpsrv/testutils/lease_file_io.h>
 #include <dhcpsrv/testutils/test_utils.h>
-#include <dhcpsrv/tests/generic_lease_mgr_unittest.h>
+#include <dhcpsrv/testutils/generic_lease_mgr_unittest.h>
 #include <testutils/gtest_utils.h>
 #include <util/multi_threading_mgr.h>
 #include <util/pid_file.h>
@@ -117,7 +117,7 @@ public:
         extra_files_() {
 
         timer_mgr_->setIOService(io_service_);
-        LeaseMgr::setIOService(io_service_);
+        DatabaseConnection::setIOService(io_service_);
 
         std::ostringstream s;
         s << KEA_LFC_BUILD_DIR << "/kea-lfc";

src/lib/dhcpsrv/tests/mysql_host_data_source_unittest.cc

@@ -1690,6 +1690,54 @@ TEST_F(MySQLHostMgrTest, setIPReservationsUnique) {
     EXPECT_TRUE(HostMgr::instance().setIPReservationsUnique(false));
 }
 
+/// @brief Verifies that loss of connectivity to MySQL is handled correctly.
+TEST_F(MySQLHostMgrDbLostCallbackTest, testRetryOpenDbLostAndRecoveredCallback) {
+    MultiThreadingTest mt(false);
+    testRetryOpenDbLostAndRecoveredCallback();
+}
+
+/// @brief Verifies that loss of connectivity to MySQL is handled correctly.
+TEST_F(MySQLHostMgrDbLostCallbackTest, testRetryOpenDbLostAndRecoveredCallbackMultiThreading) {
+    MultiThreadingTest mt(true);
+    testRetryOpenDbLostAndRecoveredCallback();
+}
+
+/// @brief Verifies that loss of connectivity to MySQL is handled correctly.
+TEST_F(MySQLHostMgrDbLostCallbackTest, testRetryOpenDbLostAndFailedCallback) {
+    MultiThreadingTest mt(false);
+    testRetryOpenDbLostAndFailedCallback();
+}
+
+/// @brief Verifies that loss of connectivity to MySQL is handled correctly.
+TEST_F(MySQLHostMgrDbLostCallbackTest, testRetryOpenDbLostAndFailedCallbackMultiThreading) {
+    MultiThreadingTest mt(true);
+    testRetryOpenDbLostAndFailedCallback();
+}
+
+/// @brief Verifies that loss of connectivity to MySQL is handled correctly.
+TEST_F(MySQLHostMgrDbLostCallbackTest, testRetryOpenDbLostAndRecoveredAfterTimeoutCallback) {
+    MultiThreadingTest mt(false);
+    testRetryOpenDbLostAndRecoveredAfterTimeoutCallback();
+}
+
+/// @brief Verifies that loss of connectivity to MySQL is handled correctly.
+TEST_F(MySQLHostMgrDbLostCallbackTest, testRetryOpenDbLostAndRecoveredAfterTimeoutCallbackMultiThreading) {
+    MultiThreadingTest mt(true);
+    testRetryOpenDbLostAndRecoveredAfterTimeoutCallback();
+}
+
+/// @brief Verifies that loss of connectivity to MySQL is handled correctly.
+TEST_F(MySQLHostMgrDbLostCallbackTest, testRetryOpenDbLostAndFailedAfterTimeoutCallback) {
+    MultiThreadingTest mt(false);
+    testRetryOpenDbLostAndFailedAfterTimeoutCallback();
+}
+
+/// @brief Verifies that loss of connectivity to MySQL is handled correctly.
+TEST_F(MySQLHostMgrDbLostCallbackTest, testRetryOpenDbLostAndFailedAfterTimeoutCallbackMultiThreading) {
+    MultiThreadingTest mt(true);
+    testRetryOpenDbLostAndFailedAfterTimeoutCallback();
+}
+
 /// @brief Verifies that db lost callback is not invoked on an open failure
 TEST_F(MySQLHostMgrDbLostCallbackTest, testNoCallbackOnOpenFailure) {
     MultiThreadingTest mt(false);

src/lib/dhcpsrv/tests/mysql_lease_mgr_unittest.cc

@@ -10,8 +10,8 @@
 #include <dhcpsrv/lease_mgr_factory.h>
 #include <dhcpsrv/mysql_lease_mgr.h>
 #include <dhcpsrv/testutils/test_utils.h>
+#include <dhcpsrv/testutils/generic_lease_mgr_unittest.h>
 #include <dhcpsrv/testutils/mysql_generic_backend_unittest.h>
-#include <dhcpsrv/tests/generic_lease_mgr_unittest.h>
 #include <exceptions/exceptions.h>
 #include <mysql/mysql_connection.h>
 #include <mysql/testutils/mysql_schema.h>
@@ -976,6 +976,54 @@ public:
     }
 };
 
+/// @brief Verifies that loss of connectivity to MySQL is handled correctly.
+TEST_F(MySqlLeaseMgrDbLostCallbackTest, testRetryOpenDbLostAndRecoveredCallback) {
+    MultiThreadingTest mt(false);
+    testRetryOpenDbLostAndRecoveredCallback();
+}
+
+/// @brief Verifies that loss of connectivity to MySQL is handled correctly.
+TEST_F(MySqlLeaseMgrDbLostCallbackTest, testRetryOpenDbLostAndRecoveredCallbackMultiThreading) {
+    MultiThreadingTest mt(true);
+    testRetryOpenDbLostAndRecoveredCallback();
+}
+
+/// @brief Verifies that loss of connectivity to MySQL is handled correctly.
+TEST_F(MySqlLeaseMgrDbLostCallbackTest, testRetryOpenDbLostAndFailedCallback) {
+    MultiThreadingTest mt(false);
+    testRetryOpenDbLostAndFailedCallback();
+}
+
+/// @brief Verifies that loss of connectivity to MySQL is handled correctly.
+TEST_F(MySqlLeaseMgrDbLostCallbackTest, testRetryOpenDbLostAndFailedCallbackMultiThreading) {
+    MultiThreadingTest mt(true);
+    testRetryOpenDbLostAndFailedCallback();
+}
+
+/// @brief Verifies that loss of connectivity to MySQL is handled correctly.
+TEST_F(MySqlLeaseMgrDbLostCallbackTest, testRetryOpenDbLostAndRecoveredAfterTimeoutCallback) {
+    MultiThreadingTest mt(false);
+    testRetryOpenDbLostAndRecoveredAfterTimeoutCallback();
+}
+
+/// @brief Verifies that loss of connectivity to MySQL is handled correctly.
+TEST_F(MySqlLeaseMgrDbLostCallbackTest, testRetryOpenDbLostAndRecoveredAfterTimeoutCallbackMultiThreading) {
+    MultiThreadingTest mt(true);
+    testRetryOpenDbLostAndRecoveredAfterTimeoutCallback();
+}
+
+/// @brief Verifies that loss of connectivity to MySQL is handled correctly.
+TEST_F(MySqlLeaseMgrDbLostCallbackTest, testRetryOpenDbLostAndFailedAfterTimeoutCallback) {
+    MultiThreadingTest mt(false);
+    testRetryOpenDbLostAndFailedAfterTimeoutCallback();
+}
+
+/// @brief Verifies that loss of connectivity to MySQL is handled correctly.
+TEST_F(MySqlLeaseMgrDbLostCallbackTest, testRetryOpenDbLostAndFailedAfterTimeoutCallbackMultiThreading) {
+    MultiThreadingTest mt(true);
+    testRetryOpenDbLostAndFailedAfterTimeoutCallback();
+}
+
 /// @brief Verifies that db lost callback is not invoked on an open failure
 TEST_F(MySqlLeaseMgrDbLostCallbackTest, testNoCallbackOnOpenFailure) {
     MultiThreadingTest mt(false);

src/lib/dhcpsrv/tests/pgsql_host_data_source_unittest.cc

@@ -1658,6 +1658,54 @@ TEST_F(PgSQLHostMgrTest, setIPReservationsUnique) {
     EXPECT_TRUE(HostMgr::instance().setIPReservationsUnique(false));
 }
 
+/// @brief Verifies that loss of connectivity to PostgreSQL is handled correctly.
+TEST_F(PgSQLHostMgrDbLostCallbackTest, testRetryOpenDbLostAndRecoveredCallback) {
+    MultiThreadingTest mt(false);
+    testRetryOpenDbLostAndRecoveredCallback();
+}
+
+/// @brief Verifies that loss of connectivity to PostgreSQL is handled correctly.
+TEST_F(PgSQLHostMgrDbLostCallbackTest, testRetryOpenDbLostAndRecoveredCallbackMultiThreading) {
+    MultiThreadingTest mt(true);
+    testRetryOpenDbLostAndRecoveredCallback();
+}
+
+/// @brief Verifies that loss of connectivity to PostgreSQL is handled correctly.
+TEST_F(PgSQLHostMgrDbLostCallbackTest, testRetryOpenDbLostAndFailedCallback) {
+    MultiThreadingTest mt(false);
+    testRetryOpenDbLostAndFailedCallback();
+}
+
+/// @brief Verifies that loss of connectivity to PostgreSQL is handled correctly.
+TEST_F(PgSQLHostMgrDbLostCallbackTest, testRetryOpenDbLostAndFailedCallbackMultiThreading) {
+    MultiThreadingTest mt(true);
+    testRetryOpenDbLostAndFailedCallback();
+}
+
+/// @brief Verifies that loss of connectivity to PostgreSQL is handled correctly.
+TEST_F(PgSQLHostMgrDbLostCallbackTest, testRetryOpenDbLostAndRecoveredAfterTimeoutCallback) {
+    MultiThreadingTest mt(false);
+    testRetryOpenDbLostAndRecoveredAfterTimeoutCallback();
+}
+
+/// @brief Verifies that loss of connectivity to PostgreSQL is handled correctly.
+TEST_F(PgSQLHostMgrDbLostCallbackTest, testRetryOpenDbLostAndRecoveredAfterTimeoutCallbackMultiThreading) {
+    MultiThreadingTest mt(true);
+    testRetryOpenDbLostAndRecoveredAfterTimeoutCallback();
+}
+
+/// @brief Verifies that loss of connectivity to PostgreSQL is handled correctly.
+TEST_F(PgSQLHostMgrDbLostCallbackTest, testRetryOpenDbLostAndFailedAfterTimeoutCallback) {
+    MultiThreadingTest mt(false);
+    testRetryOpenDbLostAndFailedAfterTimeoutCallback();
+}
+
+/// @brief Verifies that loss of connectivity to PostgreSQL is handled correctly.
+TEST_F(PgSQLHostMgrDbLostCallbackTest, testRetryOpenDbLostAndFailedAfterTimeoutCallbackMultiThreading) {
+    MultiThreadingTest mt(true);
+    testRetryOpenDbLostAndFailedAfterTimeoutCallback();
+}
+
 /// @brief Verifies that db lost callback is not invoked on an open failure
 TEST_F(PgSQLHostMgrDbLostCallbackTest, testNoCallbackOnOpenFailure) {
     MultiThreadingTest mt(false);

src/lib/dhcpsrv/tests/pgsql_lease_mgr_unittest.cc

@@ -10,8 +10,8 @@
 #include <dhcpsrv/lease_mgr_factory.h>
 #include <dhcpsrv/pgsql_lease_mgr.h>
 #include <dhcpsrv/testutils/test_utils.h>
+#include <dhcpsrv/testutils/generic_lease_mgr_unittest.h>
 #include <dhcpsrv/testutils/pgsql_generic_backend_unittest.h>
-#include <dhcpsrv/tests/generic_lease_mgr_unittest.h>
 #include <exceptions/exceptions.h>
 #include <pgsql/pgsql_connection.h>
 #include <pgsql/testutils/pgsql_schema.h>
@@ -943,6 +943,54 @@ public:
     }
 };
 
+/// @brief Verifies that loss of connectivity to PostgreSQL is handled correctly.
+TEST_F(PgSqlLeaseMgrDbLostCallbackTest, testRetryOpenDbLostAndRecoveredCallback) {
+    MultiThreadingTest mt(false);
+    testRetryOpenDbLostAndRecoveredCallback();
+}
+
+/// @brief Verifies that loss of connectivity to PostgreSQL is handled correctly.
+TEST_F(PgSqlLeaseMgrDbLostCallbackTest, testRetryOpenDbLostAndRecoveredCallbackMultiThreading) {
+    MultiThreadingTest mt(true);
+    testRetryOpenDbLostAndRecoveredCallback();
+}
+
+/// @brief Verifies that loss of connectivity to PostgreSQL is handled correctly.
+TEST_F(PgSqlLeaseMgrDbLostCallbackTest, testRetryOpenDbLostAndFailedCallback) {
+    MultiThreadingTest mt(false);
+    testRetryOpenDbLostAndFailedCallback();
+}
+
+/// @brief Verifies that loss of connectivity to PostgreSQL is handled correctly.
+TEST_F(PgSqlLeaseMgrDbLostCallbackTest, testRetryOpenDbLostAndFailedCallbackMultiThreading) {
+    MultiThreadingTest mt(true);
+    testRetryOpenDbLostAndFailedCallback();
+}
+
+/// @brief Verifies that loss of connectivity to PostgreSQL is handled correctly.
+TEST_F(PgSqlLeaseMgrDbLostCallbackTest, testRetryOpenDbLostAndRecoveredAfterTimeoutCallback) {
+    MultiThreadingTest mt(false);
+    testRetryOpenDbLostAndRecoveredAfterTimeoutCallback();
+}
+
+/// @brief Verifies that loss of connectivity to PostgreSQL is handled correctly.
+TEST_F(PgSqlLeaseMgrDbLostCallbackTest, testRetryOpenDbLostAndRecoveredAfterTimeoutCallbackMultiThreading) {
+    MultiThreadingTest mt(true);
+    testRetryOpenDbLostAndRecoveredAfterTimeoutCallback();
+}
+
+/// @brief Verifies that loss of connectivity to PostgreSQL is handled correctly.
+TEST_F(PgSqlLeaseMgrDbLostCallbackTest, testRetryOpenDbLostAndFailedAfterTimeoutCallback) {
+    MultiThreadingTest mt(false);
+    testRetryOpenDbLostAndFailedAfterTimeoutCallback();
+}
+
+/// @brief Verifies that loss of connectivity to PostgreSQL is handled correctly.
+TEST_F(PgSqlLeaseMgrDbLostCallbackTest, testRetryOpenDbLostAndFailedAfterTimeoutCallbackMultiThreading) {
+    MultiThreadingTest mt(true);
+    testRetryOpenDbLostAndFailedAfterTimeoutCallback();
+}
+
 /// @brief Verifies that db lost callback is not invoked on an open failure
 TEST_F(PgSqlLeaseMgrDbLostCallbackTest, testNoCallbackOnOpenFailure) {
     MultiThreadingTest mt(false);

src/lib/dhcpsrv/tests/tracking_lease_mgr_unittest.cc

@@ -8,7 +8,7 @@
 
 #include <dhcpsrv/subnet_id.h>
 #include <dhcpsrv/testutils/concrete_lease_mgr.h>
-#include <dhcpsrv/tests/generic_lease_mgr_unittest.h>
+#include <dhcpsrv/testutils/generic_lease_mgr_unittest.h>
 #include <exceptions/exceptions.h>
 #include <functional>
 #include <gtest/gtest.h>

src/lib/dhcpsrv/testutils/Makefile.am

@@ -19,6 +19,7 @@ libdhcpsrvtest_la_SOURCES += host_data_source_utils.cc host_data_source_utils.h
 libdhcpsrvtest_la_SOURCES += memory_host_data_source.cc memory_host_data_source.h
 libdhcpsrvtest_la_SOURCES += test_utils.cc test_utils.h
 libdhcpsrvtest_la_SOURCES += generic_backend_unittest.cc generic_backend_unittest.h
+libdhcpsrvtest_la_SOURCES += generic_lease_mgr_unittest.cc generic_lease_mgr_unittest.h
 libdhcpsrvtest_la_SOURCES += generic_host_data_source_unittest.cc generic_host_data_source_unittest.h
 libdhcpsrvtest_la_SOURCES += generic_cb_dhcp4_unittest.h generic_cb_dhcp4_unittest.cc
 libdhcpsrvtest_la_SOURCES += generic_cb_dhcp6_unittest.h generic_cb_dhcp6_unittest.cc

src/lib/dhcpsrv/testutils/concrete_lease_mgr.cc

@@ -328,7 +328,7 @@ ConcreteLeaseMgr::getDescription() const {
 }
 
 std::pair<uint32_t, uint32_t>
-ConcreteLeaseMgr::getVersion() const {
+ConcreteLeaseMgr::getVersion(const std::string& /* timer_name */) const {
     return (make_pair(uint32_t(0), uint32_t(0)));
 }
 

src/lib/dhcpsrv/testutils/concrete_lease_mgr.h

@@ -414,7 +414,9 @@ public:
     virtual std::string getDescription() const override;
 
     /// @brief Returns backend version.
-    virtual std::pair<uint32_t, uint32_t> getVersion() const override;
+    ///
+    /// @param timer_name The DB reconnect timer name.
+    virtual std::pair<uint32_t, uint32_t> getVersion(const std::string& /* timer_name */ = "") const override;
 
     /// @brief Commit transactions
     virtual void commit() override;

src/lib/dhcpsrv/testutils/generic_cb_recovery_unittest.cc

@@ -41,6 +41,7 @@ GenericConfigBackendDbLostCallbackTest::SetUp() {
     DatabaseConnection::db_lost_callback_ = 0;
     DatabaseConnection::db_recovered_callback_ = 0;
     DatabaseConnection::db_failed_callback_ = 0;
+    isc::db::DatabaseConnection::setIOService(io_service_);
     setConfigBackendImplIOService(io_service_);
     isc::dhcp::TimerMgr::instance()->setIOService(io_service_);
     isc::dhcp::CfgMgr::instance().clear();
@@ -61,11 +62,256 @@ GenericConfigBackendDbLostCallbackTest::TearDown() {
     DatabaseConnection::db_lost_callback_ = 0;
     DatabaseConnection::db_recovered_callback_ = 0;
     DatabaseConnection::db_failed_callback_ = 0;
+    isc::db::DatabaseConnection::setIOService(IOServicePtr());
     setConfigBackendImplIOService(IOServicePtr());
     isc::dhcp::TimerMgr::instance()->unregisterTimers();
     isc::dhcp::CfgMgr::instance().clear();
 }
 
+void
+GenericConfigBackendDbLostCallbackTest::testRetryOpenDbLostAndRecoveredCallback() {
+    // Set the connectivity lost callback.
+    DatabaseConnection::db_lost_callback_ =
+        std::bind(&GenericConfigBackendDbLostCallbackTest::db_lost_callback, this, ph::_1);
+
+    // Set the connectivity recovered callback.
+    DatabaseConnection::db_recovered_callback_ =
+        std::bind(&GenericConfigBackendDbLostCallbackTest::db_recovered_callback, this, ph::_1);
+
+    // Set the connectivity failed callback.
+    DatabaseConnection::db_failed_callback_ =
+        std::bind(&GenericConfigBackendDbLostCallbackTest::db_failed_callback, this, ph::_1);
+
+    std::string access = invalidConnectionString();
+    access += " retry-on-startup=true";
+
+    ConfigControlInfoPtr config_ctl_info(new ConfigControlInfo());
+    config_ctl_info->addConfigDatabase(access);
+    CfgMgr::instance().getCurrentCfg()->setConfigControlInfo(config_ctl_info);
+
+    std::shared_ptr<DbConnectionInitWithRetry> dbr(new DbConnectionInitWithRetry);
+
+    // Connect to the CB backend.
+    ASSERT_THROW(addBackend(access), DbOpenErrorWithRetry);
+
+    dbr.reset();
+
+    // Verify we can execute a query.  We don't care about the answer.
+    ServerCollection servers;
+
+    // A query should fail with NoSuchDatabase,.
+    ASSERT_THROW(servers = getAllServers(), NoSuchDatabase);
+
+    access = validConnectionString();
+    CfgMgr::instance().clear();
+    // by adding an invalid access will cause the manager factory to throw
+    // resulting in failure to recreate the manager
+    config_ctl_info.reset(new ConfigControlInfo());
+    config_ctl_info->addConfigDatabase(access);
+    CfgMgr::instance().getCurrentCfg()->setConfigControlInfo(config_ctl_info);
+    const ConfigDbInfoList& cfg = CfgMgr::instance().getCurrentCfg()->getConfigControlInfo()->getConfigDatabases();
+    (const_cast<ConfigDbInfoList&>(cfg))[0].setAccessString(access, true);
+
+    io_service_->poll();
+
+    // Our lost and recovered connectivity callback should have been invoked.
+    EXPECT_EQ(1, db_lost_callback_called_);
+    EXPECT_EQ(1, db_recovered_callback_called_);
+    EXPECT_EQ(0, db_failed_callback_called_);
+
+    // Verify we can execute a query.  We don't care about the answer.
+    ASSERT_NO_THROW_LOG(servers = getAllServers());
+}
+
+void
+GenericConfigBackendDbLostCallbackTest::testRetryOpenDbLostAndFailedCallback() {
+    // Set the connectivity lost callback.
+    DatabaseConnection::db_lost_callback_ =
+        std::bind(&GenericConfigBackendDbLostCallbackTest::db_lost_callback, this, ph::_1);
+
+    // Set the connectivity recovered callback.
+    DatabaseConnection::db_recovered_callback_ =
+        std::bind(&GenericConfigBackendDbLostCallbackTest::db_recovered_callback, this, ph::_1);
+
+    // Set the connectivity failed callback.
+    DatabaseConnection::db_failed_callback_ =
+        std::bind(&GenericConfigBackendDbLostCallbackTest::db_failed_callback, this, ph::_1);
+
+    std::string access = invalidConnectionString();
+    access += " retry-on-startup=true";
+
+    ConfigControlInfoPtr config_ctl_info(new ConfigControlInfo());
+    config_ctl_info->addConfigDatabase(access);
+    CfgMgr::instance().getCurrentCfg()->setConfigControlInfo(config_ctl_info);
+
+    std::shared_ptr<DbConnectionInitWithRetry> dbr(new DbConnectionInitWithRetry);
+
+    // Connect to the CB backend.
+    ASSERT_THROW(addBackend(access), DbOpenErrorWithRetry);
+
+    dbr.reset();
+
+    // Verify we can execute a query.  We don't care about the answer.
+    ServerCollection servers;
+
+    // A query should fail with NoSuchDatabase,.
+    ASSERT_THROW(servers = getAllServers(), NoSuchDatabase);
+
+    io_service_->poll();
+
+    // Our lost and failed connectivity callback should have been invoked.
+    EXPECT_EQ(1, db_lost_callback_called_);
+    EXPECT_EQ(0, db_recovered_callback_called_);
+    EXPECT_EQ(1, db_failed_callback_called_);
+
+    // A query should fail with NoSuchDatabase,.
+    ASSERT_THROW(servers = getAllServers(), NoSuchDatabase);
+}
+
+void
+GenericConfigBackendDbLostCallbackTest::testRetryOpenDbLostAndRecoveredAfterTimeoutCallback() {
+    // Set the connectivity lost callback.
+    DatabaseConnection::db_lost_callback_ =
+        std::bind(&GenericConfigBackendDbLostCallbackTest::db_lost_callback, this, ph::_1);
+
+    // Set the connectivity recovered callback.
+    DatabaseConnection::db_recovered_callback_ =
+        std::bind(&GenericConfigBackendDbLostCallbackTest::db_recovered_callback, this, ph::_1);
+
+    // Set the connectivity failed callback.
+    DatabaseConnection::db_failed_callback_ =
+        std::bind(&GenericConfigBackendDbLostCallbackTest::db_failed_callback, this, ph::_1);
+
+    std::string access = invalidConnectionString();
+    std::string extra = " max-reconnect-tries=3 reconnect-wait-time=1 retry-on-startup=true";
+    access += extra;
+
+    ConfigControlInfoPtr config_ctl_info(new ConfigControlInfo());
+    config_ctl_info->addConfigDatabase(access);
+    CfgMgr::instance().getCurrentCfg()->setConfigControlInfo(config_ctl_info);
+
+    std::shared_ptr<DbConnectionInitWithRetry> dbr(new DbConnectionInitWithRetry);
+
+    // Connect to the CB backend.
+    ASSERT_THROW(addBackend(access), DbOpenErrorWithRetry);
+
+    dbr.reset();
+
+    // Verify we can execute a query.  We don't care about the answer.
+    ServerCollection servers;
+
+    // A query should fail with NoSuchDatabase,.
+    ASSERT_THROW(servers = getAllServers(), NoSuchDatabase);
+
+    io_service_->poll();
+
+    // Our lost connectivity callback should have been invoked.
+    EXPECT_EQ(1, db_lost_callback_called_);
+    EXPECT_EQ(0, db_recovered_callback_called_);
+    EXPECT_EQ(0, db_failed_callback_called_);
+
+    access = validConnectionString();
+    access += extra;
+    CfgMgr::instance().clear();
+    config_ctl_info.reset(new ConfigControlInfo());
+    config_ctl_info->addConfigDatabase(access);
+    CfgMgr::instance().getCurrentCfg()->setConfigControlInfo(config_ctl_info);
+
+    sleep(1);
+
+    io_service_->poll();
+
+    // Our lost and recovered connectivity callback should have been invoked.
+    EXPECT_EQ(2, db_lost_callback_called_);
+    EXPECT_EQ(1, db_recovered_callback_called_);
+    EXPECT_EQ(0, db_failed_callback_called_);
+
+    // Verify we can execute a query.  We don't care about the answer.
+    ASSERT_NO_THROW_LOG(servers = getAllServers());
+
+    sleep(1);
+
+    io_service_->poll();
+
+    // No callback should have been invoked.
+    EXPECT_EQ(2, db_lost_callback_called_);
+    EXPECT_EQ(1, db_recovered_callback_called_);
+    EXPECT_EQ(0, db_failed_callback_called_);
+
+    // Verify we can execute a query.  We don't care about the answer.
+    ASSERT_NO_THROW_LOG(servers = getAllServers());
+}
+
+void
+GenericConfigBackendDbLostCallbackTest::testRetryOpenDbLostAndFailedAfterTimeoutCallback() {
+    // Set the connectivity lost callback.
+    DatabaseConnection::db_lost_callback_ =
+        std::bind(&GenericConfigBackendDbLostCallbackTest::db_lost_callback, this, ph::_1);
+
+    // Set the connectivity recovered callback.
+    DatabaseConnection::db_recovered_callback_ =
+        std::bind(&GenericConfigBackendDbLostCallbackTest::db_recovered_callback, this, ph::_1);
+
+    // Set the connectivity failed callback.
+    DatabaseConnection::db_failed_callback_ =
+        std::bind(&GenericConfigBackendDbLostCallbackTest::db_failed_callback, this, ph::_1);
+
+    std::string access = invalidConnectionString();
+    std::string extra = " max-reconnect-tries=3 reconnect-wait-time=1 retry-on-startup=true";
+    access += extra;
+
+    ConfigControlInfoPtr config_ctl_info(new ConfigControlInfo());
+    config_ctl_info->addConfigDatabase(access);
+    CfgMgr::instance().getCurrentCfg()->setConfigControlInfo(config_ctl_info);
+
+    std::shared_ptr<DbConnectionInitWithRetry> dbr(new DbConnectionInitWithRetry);
+
+    // Connect to the CB backend.
+    ASSERT_THROW(addBackend(access), DbOpenErrorWithRetry);
+
+    dbr.reset();
+
+    // Verify we can execute a query.  We don't care about the answer.
+    ServerCollection servers;
+
+    // A query should fail with NoSuchDatabase,.
+    ASSERT_THROW(servers = getAllServers(), NoSuchDatabase);
+
+    io_service_->poll();
+
+    // Our lost connectivity callback should have been invoked.
+    EXPECT_EQ(1, db_lost_callback_called_);
+    EXPECT_EQ(0, db_recovered_callback_called_);
+    EXPECT_EQ(0, db_failed_callback_called_);
+
+    // A query should fail with NoSuchDatabase,.
+    ASSERT_THROW(servers = getAllServers(), NoSuchDatabase);
+
+    sleep(1);
+
+    io_service_->poll();
+
+    // Our lost connectivity callback should have been invoked.
+    EXPECT_EQ(2, db_lost_callback_called_);
+    EXPECT_EQ(0, db_recovered_callback_called_);
+    EXPECT_EQ(0, db_failed_callback_called_);
+
+    // A query should fail with NoSuchDatabase,.
+    ASSERT_THROW(servers = getAllServers(), NoSuchDatabase);
+
+    sleep(1);
+
+    io_service_->poll();
+
+    // Our lost and failed connectivity callback should have been invoked.
+    EXPECT_EQ(3, db_lost_callback_called_);
+    EXPECT_EQ(0, db_recovered_callback_called_);
+    EXPECT_EQ(1, db_failed_callback_called_);
+
+    // A query should fail with NoSuchDatabase,.
+    ASSERT_THROW(servers = getAllServers(), NoSuchDatabase);
+}
+
 void
 GenericConfigBackendDbLostCallbackTest::testNoCallbackOnOpenFailure() {
     DatabaseConnection::db_lost_callback_ =
@@ -158,6 +404,7 @@ GenericConfigBackendDbLostCallbackTest::testDbLostAndFailedCallback() {
         std::bind(&GenericConfigBackendDbLostCallbackTest::db_failed_callback, this, ph::_1);
 
     std::string access = validConnectionString();
+
     ConfigControlInfoPtr config_ctl_info(new ConfigControlInfo());
     config_ctl_info->addConfigDatabase(access);
     CfgMgr::instance().getCurrentCfg()->setConfigControlInfo(config_ctl_info);
@@ -221,6 +468,7 @@ GenericConfigBackendDbLostCallbackTest::testDbLostAndRecoveredAfterTimeoutCallba
     std::string access = validConnectionString();
     std::string extra = " max-reconnect-tries=3 reconnect-wait-time=1";
     access += extra;
+
     ConfigControlInfoPtr config_ctl_info(new ConfigControlInfo());
     config_ctl_info->addConfigDatabase(access);
     CfgMgr::instance().getCurrentCfg()->setConfigControlInfo(config_ctl_info);
@@ -310,6 +558,7 @@ GenericConfigBackendDbLostCallbackTest::testDbLostAndFailedAfterTimeoutCallback(
     std::string access = validConnectionString();
     std::string extra = " max-reconnect-tries=3 reconnect-wait-time=1";
     access += extra;
+
     ConfigControlInfoPtr config_ctl_info(new ConfigControlInfo());
     config_ctl_info->addConfigDatabase(access);
     CfgMgr::instance().getCurrentCfg()->setConfigControlInfo(config_ctl_info);

src/lib/dhcpsrv/testutils/generic_cb_recovery_unittest.h

@@ -72,6 +72,48 @@ public:
     /// we created.
     virtual void TearDown();
 
+    /// @brief Verifies the CB manager's behavior if DB connection can not be
+    /// established but succeeds on retry
+    ///
+    /// This function creates a CB manager with a back end that supports
+    /// connectivity lost callback. It verifies that connectivity is unavailable
+    /// and then recovered on retry:
+    /// -# The registered DbLostCallback was invoked
+    /// -# The registered DbRecoveredCallback was invoked
+    void testRetryOpenDbLostAndRecoveredCallback();
+
+    /// @brief Verifies the CB manager's behavior if DB connection can not be
+    /// established but fails on retry
+    ///
+    /// This function creates a CB manager with a back end that supports
+    /// connectivity lost callback. It verifies that connectivity is unavailable
+    /// and then fails again on retry:
+    /// -# The registered DbLostCallback was invoked
+    /// -# The registered DbFailedCallback was invoked
+    void testRetryOpenDbLostAndFailedCallback();
+
+    /// @brief Verifies the CB manager's behavior if DB connection can not be
+    /// established but succeeds on retry
+    ///
+    /// This function creates a CB manager with a back end that supports
+    /// connectivity lost callback. It verifies that connectivity is unavailable
+    /// and then recovered on retry:
+    /// -# The registered DbLostCallback was invoked
+    /// -# The registered DbRecoveredCallback was invoked after two reconnect
+    /// attempts (once failing and second triggered by timer)
+    void testRetryOpenDbLostAndRecoveredAfterTimeoutCallback();
+
+    /// @brief Verifies the CB manager's behavior if DB connection can not be
+    /// established but fails on retry
+    ///
+    /// This function creates a CB manager with a back end that supports
+    /// connectivity lost callback. It verifies that connectivity is unavailable
+    /// and then fails again on retry:
+    /// -# The registered DbLostCallback was invoked
+    /// -# The registered DbFailedCallback was invoked after two reconnect
+    /// attempts (once failing and second triggered by timer)
+    void testRetryOpenDbLostAndFailedAfterTimeoutCallback();
+
     /// @brief Verifies open failures do NOT invoke db lost callback
     ///
     /// The db lost callback should only be invoked after successfully

src/lib/dhcpsrv/testutils/generic_host_data_source_unittest.cc

@@ -2674,6 +2674,216 @@ GenericHostDataSourceTest::testMultipleHosts6() {
     ASSERT_NO_THROW(hdsptr_->add(host2));
 }
 
+void
+HostMgrDbLostCallbackTest::testRetryOpenDbLostAndRecoveredCallback() {
+    // Set the connectivity lost callback.
+    isc::db::DatabaseConnection::db_lost_callback_ =
+        std::bind(&HostMgrDbLostCallbackTest::db_lost_callback, this, ph::_1);
+
+    // Set the connectivity recovered callback.
+    isc::db::DatabaseConnection::db_recovered_callback_ =
+        std::bind(&HostMgrDbLostCallbackTest::db_recovered_callback, this, ph::_1);
+
+    // Set the connectivity failed callback.
+    isc::db::DatabaseConnection::db_failed_callback_ =
+        std::bind(&HostMgrDbLostCallbackTest::db_failed_callback, this, ph::_1);
+
+    std::string access = invalidConnectString();
+    access += " retry-on-startup=true";
+    CfgMgr::instance().getCurrentCfg()->getCfgDbAccess()->setHostDbAccessString(access);
+
+    std::shared_ptr<DbConnectionInitWithRetry> dbr(new DbConnectionInitWithRetry);
+
+    // Create the HostMgr.
+    HostMgr::create();
+
+    ASSERT_THROW(HostMgr::addBackend(access), DbOpenErrorWithRetry);
+
+    dbr.reset();
+
+    ASSERT_FALSE(HostMgr::instance().getHostDataSource());
+
+    access = validConnectString();
+    access += " retry-on-startup=true";
+    CfgMgr::instance().clear();
+    CfgMgr::instance().getCurrentCfg()->getCfgDbAccess()->setHostDbAccessString(access);
+
+    io_service_->poll();
+
+    // Our lost and recovered connectivity callback should have been invoked.
+    EXPECT_EQ(1, db_lost_callback_called_);
+    EXPECT_EQ(1, db_recovered_callback_called_);
+    EXPECT_EQ(0, db_failed_callback_called_);
+
+    ASSERT_TRUE(HostMgr::instance().getHostDataSource());
+}
+
+void
+HostMgrDbLostCallbackTest::testRetryOpenDbLostAndFailedCallback() {
+    // Set the connectivity lost callback.
+    isc::db::DatabaseConnection::db_lost_callback_ =
+        std::bind(&HostMgrDbLostCallbackTest::db_lost_callback, this, ph::_1);
+
+    // Set the connectivity recovered callback.
+    isc::db::DatabaseConnection::db_recovered_callback_ =
+        std::bind(&HostMgrDbLostCallbackTest::db_recovered_callback, this, ph::_1);
+
+    // Set the connectivity failed callback.
+    isc::db::DatabaseConnection::db_failed_callback_ =
+        std::bind(&HostMgrDbLostCallbackTest::db_failed_callback, this, ph::_1);
+
+    std::string access = invalidConnectString();
+    access += " retry-on-startup=true";
+    CfgMgr::instance().getCurrentCfg()->getCfgDbAccess()->setHostDbAccessString(access);
+
+    std::shared_ptr<DbConnectionInitWithRetry> dbr(new DbConnectionInitWithRetry);
+
+    // Create the HostMgr.
+    HostMgr::create();
+
+    ASSERT_THROW(HostMgr::addBackend(access), DbOpenErrorWithRetry);
+
+    dbr.reset();
+
+    ASSERT_FALSE(HostMgr::instance().getHostDataSource());
+
+    io_service_->poll();
+
+    // Our lost and failed connectivity callback should have been invoked.
+    EXPECT_EQ(1, db_lost_callback_called_);
+    EXPECT_EQ(0, db_recovered_callback_called_);
+    EXPECT_EQ(1, db_failed_callback_called_);
+
+    ASSERT_FALSE(HostMgr::instance().getHostDataSource());
+}
+
+void
+HostMgrDbLostCallbackTest::testRetryOpenDbLostAndRecoveredAfterTimeoutCallback() {
+    // Set the connectivity lost callback.
+    isc::db::DatabaseConnection::db_lost_callback_ =
+        std::bind(&HostMgrDbLostCallbackTest::db_lost_callback, this, ph::_1);
+
+    // Set the connectivity recovered callback.
+    isc::db::DatabaseConnection::db_recovered_callback_ =
+        std::bind(&HostMgrDbLostCallbackTest::db_recovered_callback, this, ph::_1);
+
+    // Set the connectivity failed callback.
+    isc::db::DatabaseConnection::db_failed_callback_ =
+        std::bind(&HostMgrDbLostCallbackTest::db_failed_callback, this, ph::_1);
+
+    std::string access = invalidConnectString();
+    std::string extra = " max-reconnect-tries=3 reconnect-wait-time=1 retry-on-startup=true";
+    access += extra;
+    CfgMgr::instance().getCurrentCfg()->getCfgDbAccess()->setHostDbAccessString(access);
+
+    std::shared_ptr<DbConnectionInitWithRetry> dbr(new DbConnectionInitWithRetry);
+
+    // Create the HostMgr.
+    HostMgr::create();
+
+    ASSERT_THROW(HostMgr::addBackend(access), DbOpenErrorWithRetry);
+
+    dbr.reset();
+
+    ASSERT_FALSE(HostMgr::instance().getHostDataSource());
+
+    io_service_->poll();
+
+    // Our lost connectivity callback should have been invoked.
+    EXPECT_EQ(1, db_lost_callback_called_);
+    EXPECT_EQ(0, db_recovered_callback_called_);
+    EXPECT_EQ(0, db_failed_callback_called_);
+
+    access = validConnectString();
+    access += extra;
+    CfgMgr::instance().clear();
+    CfgMgr::instance().getCurrentCfg()->getCfgDbAccess()->setHostDbAccessString(access);
+
+    sleep(1);
+
+    io_service_->poll();
+
+    // Our lost and recovered connectivity callback should have been invoked.
+    EXPECT_EQ(2, db_lost_callback_called_);
+    EXPECT_EQ(1, db_recovered_callback_called_);
+    EXPECT_EQ(0, db_failed_callback_called_);
+
+    ASSERT_TRUE(HostMgr::instance().getHostDataSource());
+
+    sleep(1);
+
+    io_service_->poll();
+
+    // No callback should have been invoked.
+    EXPECT_EQ(2, db_lost_callback_called_);
+    EXPECT_EQ(1, db_recovered_callback_called_);
+    EXPECT_EQ(0, db_failed_callback_called_);
+
+    ASSERT_TRUE(HostMgr::instance().getHostDataSource());
+}
+
+void
+HostMgrDbLostCallbackTest::testRetryOpenDbLostAndFailedAfterTimeoutCallback() {
+    // Set the connectivity lost callback.
+    isc::db::DatabaseConnection::db_lost_callback_ =
+        std::bind(&HostMgrDbLostCallbackTest::db_lost_callback, this, ph::_1);
+
+    // Set the connectivity recovered callback.
+    isc::db::DatabaseConnection::db_recovered_callback_ =
+        std::bind(&HostMgrDbLostCallbackTest::db_recovered_callback, this, ph::_1);
+
+    // Set the connectivity failed callback.
+    isc::db::DatabaseConnection::db_failed_callback_ =
+        std::bind(&HostMgrDbLostCallbackTest::db_failed_callback, this, ph::_1);
+
+    std::string access = invalidConnectString();
+    std::string extra = " max-reconnect-tries=3 reconnect-wait-time=1 retry-on-startup=true";
+    access += extra;
+    CfgMgr::instance().getCurrentCfg()->getCfgDbAccess()->setHostDbAccessString(access);
+
+    std::shared_ptr<DbConnectionInitWithRetry> dbr(new DbConnectionInitWithRetry);
+
+    // Create the HostMgr.
+    HostMgr::create();
+
+    ASSERT_THROW(HostMgr::addBackend(access), DbOpenErrorWithRetry);
+
+    dbr.reset();
+
+    ASSERT_FALSE(HostMgr::instance().getHostDataSource());
+
+    io_service_->poll();
+
+    // Our lost connectivity callback should have been invoked.
+    EXPECT_EQ(1, db_lost_callback_called_);
+    EXPECT_EQ(0, db_recovered_callback_called_);
+    EXPECT_EQ(0, db_failed_callback_called_);
+
+    ASSERT_FALSE(HostMgr::instance().getHostDataSource());
+
+    sleep(1);
+
+    io_service_->poll();
+
+    // Our lost connectivity callback should have been invoked.
+    EXPECT_EQ(2, db_lost_callback_called_);
+    EXPECT_EQ(0, db_recovered_callback_called_);
+    EXPECT_EQ(0, db_failed_callback_called_);
+
+    ASSERT_FALSE(HostMgr::instance().getHostDataSource());
+
+    sleep(1);
+
+    io_service_->poll();
+
+    // Our lost and failed connectivity callback should have been invoked.
+    EXPECT_EQ(3, db_lost_callback_called_);
+    EXPECT_EQ(0, db_recovered_callback_called_);
+    EXPECT_EQ(1, db_failed_callback_called_);
+
+    ASSERT_FALSE(HostMgr::instance().getHostDataSource());
+}
+
 void
 HostMgrDbLostCallbackTest::testNoCallbackOnOpenFailure() {
     isc::db::DatabaseConnection::db_lost_callback_ =

src/lib/dhcpsrv/testutils/generic_host_data_source_unittest.h

@@ -555,7 +555,7 @@ public:
         isc::db::DatabaseConnection::db_lost_callback_ = 0;
         isc::db::DatabaseConnection::db_recovered_callback_ = 0;
         isc::db::DatabaseConnection::db_failed_callback_ = 0;
-        isc::dhcp::HostMgr::setIOService(io_service_);
+        isc::db::DatabaseConnection::setIOService(io_service_);
         isc::dhcp::TimerMgr::instance()->setIOService(io_service_);
         isc::dhcp::CfgMgr::instance().clear();
     }
@@ -564,7 +564,7 @@ public:
         isc::db::DatabaseConnection::db_lost_callback_ = 0;
         isc::db::DatabaseConnection::db_recovered_callback_ = 0;
         isc::db::DatabaseConnection::db_failed_callback_ = 0;
-        isc::dhcp::HostMgr::setIOService(isc::asiolink::IOServicePtr());
+        isc::db::DatabaseConnection::setIOService(isc::asiolink::IOServicePtr());
         isc::dhcp::TimerMgr::instance()->unregisterTimers();
         isc::dhcp::CfgMgr::instance().clear();
     }
@@ -606,6 +606,50 @@ public:
     /// string
     virtual std::string invalidConnectString() = 0;
 
+    /// @brief Verifies the host manager's behavior if DB connection can not be
+    /// established but succeeds on retry
+    ///
+    /// This function creates a host manager with a back end that supports
+    /// connectivity lost callback (currently only MySQL and PostgreSQL). It
+    /// verifies that connectivity is unavailable and then recovered on retry:
+    /// -# The registered DbLostCallback was invoked
+    /// -# The registered DbRecoveredCallback was invoked
+    void testRetryOpenDbLostAndRecoveredCallback();
+
+    /// @brief Verifies the host manager's behavior if DB connection can not be
+    /// established but fails on retry
+    ///
+    /// This function creates a host manager with a back end that supports
+    /// connectivity lost callback (currently only MySQL and PostgreSQL). It
+    /// It verifies that connectivity is unavailable and then fails again on
+    /// retry:
+    /// -# The registered DbLostCallback was invoked
+    /// -# The registered DbFailedCallback was invoked
+    void testRetryOpenDbLostAndFailedCallback();
+
+    /// @brief Verifies the host manager's behavior if DB connection can not be
+    /// established but succeeds on retry
+    ///
+    /// This function creates a host manager with a back end that supports
+    /// connectivity lost callback (currently only MySQL and PostgreSQL). It
+    /// verifies that connectivity is unavailable and then recovered on retry:
+    /// -# The registered DbLostCallback was invoked
+    /// -# The registered DbRecoveredCallback was invoked after two reconnect
+    /// attempts (once failing and second triggered by timer)
+    void testRetryOpenDbLostAndRecoveredAfterTimeoutCallback();
+
+    /// @brief Verifies the host manager's behavior if DB connection can not be
+    /// established but fails on retry
+    ///
+    /// This function creates a host manager with a back end that supports
+    /// connectivity lost callback (currently only MySQL and PostgreSQL). It
+    /// It verifies that connectivity is unavailable and then fails again on
+    /// retry:
+    /// -# The registered DbLostCallback was invoked
+    /// -# The registered DbFailedCallback was invoked after two reconnect
+    /// attempts (once failing and second triggered by timer)
+    void testRetryOpenDbLostAndFailedAfterTimeoutCallback();
+
     /// @brief Verifies open failures do NOT invoke db lost callback
     ///
     /// The db lost callback should only be invoked after successfully

src/lib/dhcpsrv/testutils/generic_lease_mgr_unittest.cc

@@ -12,7 +12,7 @@
 #include <dhcpsrv/cfgmgr.h>
 #include <dhcpsrv/dhcpsrv_exceptions.h>
 #include <dhcpsrv/lease_mgr_factory.h>
-#include <dhcpsrv/tests/generic_lease_mgr_unittest.h>
+#include <dhcpsrv/testutils/generic_lease_mgr_unittest.h>
 #include <dhcpsrv/testutils/test_utils.h>
 #include <exceptions/exceptions.h>
 #include <stats/stats_mgr.h>
@@ -3350,6 +3350,204 @@ LeaseMgrDbLostCallbackTest::TearDown() {
     isc::dhcp::LeaseMgrFactory::destroy();
 }
 
+void
+LeaseMgrDbLostCallbackTest::testRetryOpenDbLostAndRecoveredCallback() {
+    // Set the connectivity lost callback.
+    DatabaseConnection::db_lost_callback_ =
+        std::bind(&LeaseMgrDbLostCallbackTest::db_lost_callback, this, ph::_1);
+
+    // Set the connectivity recovered callback.
+    DatabaseConnection::db_recovered_callback_ =
+        std::bind(&LeaseMgrDbLostCallbackTest::db_recovered_callback, this, ph::_1);
+
+    // Set the connectivity failed callback.
+    DatabaseConnection::db_failed_callback_ =
+        std::bind(&LeaseMgrDbLostCallbackTest::db_failed_callback, this, ph::_1);
+
+    std::string access = invalidConnectString();
+    access += " retry-on-startup=true";
+    CfgMgr::instance().getCurrentCfg()->getCfgDbAccess()->setLeaseDbAccessString(access);
+
+    std::shared_ptr<DbConnectionInitWithRetry> dbr(new DbConnectionInitWithRetry);
+
+    // Connect to the lease backend.
+    ASSERT_THROW(LeaseMgrFactory::create(access), DbOpenErrorWithRetry);
+
+    dbr.reset();
+
+    ASSERT_FALSE(LeaseMgrFactory::haveInstance());
+
+    access = validConnectString();
+    access += " retry-on-startup=true";
+    CfgMgr::instance().getCurrentCfg()->getCfgDbAccess()->setLeaseDbAccessString(access);
+
+    io_service_->poll();
+
+    // Our lost and recovered connectivity callback should have been invoked.
+    EXPECT_EQ(1, db_lost_callback_called_);
+    EXPECT_EQ(1, db_recovered_callback_called_);
+    EXPECT_EQ(0, db_failed_callback_called_);
+
+    ASSERT_TRUE(LeaseMgrFactory::haveInstance());
+}
+
+void
+LeaseMgrDbLostCallbackTest::testRetryOpenDbLostAndFailedCallback() {
+    // Set the connectivity lost callback.
+    DatabaseConnection::db_lost_callback_ =
+        std::bind(&LeaseMgrDbLostCallbackTest::db_lost_callback, this, ph::_1);
+
+    // Set the connectivity recovered callback.
+    DatabaseConnection::db_recovered_callback_ =
+        std::bind(&LeaseMgrDbLostCallbackTest::db_recovered_callback, this, ph::_1);
+
+    // Set the connectivity failed callback.
+    DatabaseConnection::db_failed_callback_ =
+        std::bind(&LeaseMgrDbLostCallbackTest::db_failed_callback, this, ph::_1);
+
+    std::string access = invalidConnectString();
+    access += " retry-on-startup=true";
+    CfgMgr::instance().getCurrentCfg()->getCfgDbAccess()->setLeaseDbAccessString(access);
+
+    std::shared_ptr<DbConnectionInitWithRetry> dbr(new DbConnectionInitWithRetry);
+
+    // Connect to the lease backend.
+    ASSERT_THROW(LeaseMgrFactory::create(access), DbOpenErrorWithRetry);
+
+    dbr.reset();
+
+    ASSERT_FALSE(LeaseMgrFactory::haveInstance());
+
+    io_service_->poll();
+
+    // Our lost and failed connectivity callback should have been invoked.
+    EXPECT_EQ(1, db_lost_callback_called_);
+    EXPECT_EQ(0, db_recovered_callback_called_);
+    EXPECT_EQ(1, db_failed_callback_called_);
+
+    ASSERT_FALSE(LeaseMgrFactory::haveInstance());
+}
+
+void
+LeaseMgrDbLostCallbackTest::testRetryOpenDbLostAndRecoveredAfterTimeoutCallback() {
+    // Set the connectivity lost callback.
+    DatabaseConnection::db_lost_callback_ =
+        std::bind(&LeaseMgrDbLostCallbackTest::db_lost_callback, this, ph::_1);
+
+    // Set the connectivity recovered callback.
+    DatabaseConnection::db_recovered_callback_ =
+        std::bind(&LeaseMgrDbLostCallbackTest::db_recovered_callback, this, ph::_1);
+
+    // Set the connectivity failed callback.
+    DatabaseConnection::db_failed_callback_ =
+        std::bind(&LeaseMgrDbLostCallbackTest::db_failed_callback, this, ph::_1);
+
+    std::string access = invalidConnectString();
+    std::string extra = " max-reconnect-tries=3 reconnect-wait-time=1 retry-on-startup=true";
+    access += extra;
+    CfgMgr::instance().getCurrentCfg()->getCfgDbAccess()->setLeaseDbAccessString(access);
+
+    std::shared_ptr<DbConnectionInitWithRetry> dbr(new DbConnectionInitWithRetry);
+
+    // Connect to the lease backend.
+    ASSERT_THROW(LeaseMgrFactory::create(access), DbOpenErrorWithRetry);
+
+    dbr.reset();
+
+    ASSERT_FALSE(LeaseMgrFactory::haveInstance());
+
+    io_service_->poll();
+
+    // Our lost connectivity callback should have been invoked.
+    EXPECT_EQ(1, db_lost_callback_called_);
+    EXPECT_EQ(0, db_recovered_callback_called_);
+    EXPECT_EQ(0, db_failed_callback_called_);
+
+    access = validConnectString();
+    access += extra;
+    CfgMgr::instance().getCurrentCfg()->getCfgDbAccess()->setLeaseDbAccessString(access);
+
+    sleep(1);
+
+    io_service_->poll();
+
+    // Our lost and recovered connectivity callback should have been invoked.
+    EXPECT_EQ(2, db_lost_callback_called_);
+    EXPECT_EQ(1, db_recovered_callback_called_);
+    EXPECT_EQ(0, db_failed_callback_called_);
+
+    ASSERT_TRUE(LeaseMgrFactory::haveInstance());
+
+    sleep(1);
+
+    io_service_->poll();
+
+    // No callback should have been invoked.
+    EXPECT_EQ(2, db_lost_callback_called_);
+    EXPECT_EQ(1, db_recovered_callback_called_);
+    EXPECT_EQ(0, db_failed_callback_called_);
+
+    ASSERT_TRUE(LeaseMgrFactory::haveInstance());
+}
+
+void
+LeaseMgrDbLostCallbackTest::testRetryOpenDbLostAndFailedAfterTimeoutCallback() {
+    // Set the connectivity lost callback.
+    DatabaseConnection::db_lost_callback_ =
+        std::bind(&LeaseMgrDbLostCallbackTest::db_lost_callback, this, ph::_1);
+
+    // Set the connectivity recovered callback.
+    DatabaseConnection::db_recovered_callback_ =
+        std::bind(&LeaseMgrDbLostCallbackTest::db_recovered_callback, this, ph::_1);
+
+    // Set the connectivity failed callback.
+    DatabaseConnection::db_failed_callback_ =
+        std::bind(&LeaseMgrDbLostCallbackTest::db_failed_callback, this, ph::_1);
+
+    std::string access = invalidConnectString();
+    std::string extra = " max-reconnect-tries=3 reconnect-wait-time=1 retry-on-startup=true";
+    access += extra;
+    CfgMgr::instance().getCurrentCfg()->getCfgDbAccess()->setLeaseDbAccessString(access);
+
+    std::shared_ptr<DbConnectionInitWithRetry> dbr(new DbConnectionInitWithRetry);
+
+    // Connect to the lease backend.
+    ASSERT_THROW(LeaseMgrFactory::create(access), DbOpenErrorWithRetry);
+
+    dbr.reset();
+
+    ASSERT_FALSE(LeaseMgrFactory::haveInstance());
+
+    io_service_->poll();
+
+    // Our lost connectivity callback should have been invoked.
+    EXPECT_EQ(1, db_lost_callback_called_);
+    EXPECT_EQ(0, db_recovered_callback_called_);
+    EXPECT_EQ(0, db_failed_callback_called_);
+
+    sleep(1);
+
+    io_service_->poll();
+
+    // Our lost connectivity callback should have been invoked.
+    EXPECT_EQ(2, db_lost_callback_called_);
+    EXPECT_EQ(0, db_recovered_callback_called_);
+    EXPECT_EQ(0, db_failed_callback_called_);
+
+    ASSERT_FALSE(LeaseMgrFactory::haveInstance());
+
+    sleep(1);
+
+    io_service_->poll();
+
+    // Our lost and failed connectivity callback should have been invoked.
+    EXPECT_EQ(3, db_lost_callback_called_);
+    EXPECT_EQ(0, db_recovered_callback_called_);
+    EXPECT_EQ(1, db_failed_callback_called_);
+
+    ASSERT_FALSE(LeaseMgrFactory::haveInstance());
+}
+
 void
 LeaseMgrDbLostCallbackTest::testNoCallbackOnOpenFailure() {
     DatabaseConnection::db_lost_callback_ =

src/lib/dhcpsrv/testutils/generic_lease_mgr_unittest.h

@@ -8,6 +8,7 @@
 #define GENERIC_LEASE_MGR_UNITTEST_H
 
 #include <asiolink/io_service.h>
+#include <database/database_connection.h>
 #include <dhcpsrv/tracking_lease_mgr.h>
 #include <dhcpsrv/timer_mgr.h>
 
@@ -682,7 +683,7 @@ public:
         db::DatabaseConnection::db_lost_callback_ = 0;
         db::DatabaseConnection::db_recovered_callback_ = 0;
         db::DatabaseConnection::db_failed_callback_ = 0;
-        LeaseMgr::setIOService(io_service_);
+        db::DatabaseConnection::setIOService(io_service_);
         TimerMgr::instance()->setIOService(io_service_);
     }
 
@@ -690,7 +691,7 @@ public:
         db::DatabaseConnection::db_lost_callback_ = 0;
         db::DatabaseConnection::db_recovered_callback_ = 0;
         db::DatabaseConnection::db_failed_callback_ = 0;
-        LeaseMgr::setIOService(isc::asiolink::IOServicePtr());
+        db::DatabaseConnection::setIOService(isc::asiolink::IOServicePtr());
         TimerMgr::instance()->unregisterTimers();
     }
 
@@ -720,6 +721,50 @@ public:
     /// string
     virtual std::string invalidConnectString() = 0;
 
+    /// @brief Verifies the lease manager's behavior if DB connection can not be
+    /// established but succeeds on retry
+    ///
+    /// This function creates a lease manager with a back end that supports
+    /// connectivity lost callback (currently only MySQL and PostgreSQL). It
+    /// verifies that connectivity is unavailable and then recovered on retry:
+    /// -# The registered DbLostCallback was invoked
+    /// -# The registered DbRecoveredCallback was invoked
+    void testRetryOpenDbLostAndRecoveredCallback();
+
+    /// @brief Verifies the lease manager's behavior if DB connection can not be
+    /// established but fails on retry
+    ///
+    /// This function creates a lease manager with a back end that supports
+    /// connectivity lost callback (currently only MySQL and PostgreSQL). It
+    /// It verifies that connectivity is unavailable and then fails again on
+    /// retry:
+    /// -# The registered DbLostCallback was invoked
+    /// -# The registered DbFailedCallback was invoked
+    void testRetryOpenDbLostAndFailedCallback();
+
+    /// @brief Verifies the lease manager's behavior if DB connection can not be
+    /// established but succeeds on retry
+    ///
+    /// This function creates a lease manager with a back end that supports
+    /// connectivity lost callback (currently only MySQL and PostgreSQL). It
+    /// verifies that connectivity is unavailable and then recovered on retry:
+    /// -# The registered DbLostCallback was invoked
+    /// -# The registered DbRecoveredCallback was invoked after two reconnect
+    /// attempts (once failing and second triggered by timer)
+    void testRetryOpenDbLostAndRecoveredAfterTimeoutCallback();
+
+    /// @brief Verifies the lease manager's behavior if DB connection can not be
+    /// established but fails on retry
+    ///
+    /// This function creates a lease manager with a back end that supports
+    /// connectivity lost callback (currently only MySQL and PostgreSQL). It
+    /// It verifies that connectivity is unavailable and then fails again on
+    /// retry:
+    /// -# The registered DbLostCallback was invoked
+    /// -# The registered DbFailedCallback was invoked after two reconnect
+    /// attempts (once failing and second triggered by timer)
+    void testRetryOpenDbLostAndFailedAfterTimeoutCallback();
+
     /// @brief Verifies open failures do NOT invoke db lost callback
     ///
     /// The db lost callback should only be invoked after successfully

src/lib/dhcpsrv/timer_mgr.cc

@@ -88,7 +88,7 @@ public:
 
     /// @brief Sets IO service to be used by the Timer Manager.
     ///
-    /// @param io_service Pointer to the new IO service.
+    /// @param io_service Pointer to the IO service.
     void setIOService(const IOServicePtr& io_service);
 
     /// @brief Registers new timer in the @c TimerMgr.
@@ -107,7 +107,6 @@ public:
                        const long interval,
                        const asiolink::IntervalTimer::Mode& scheduling_mode);
 
-
     /// @brief Unregisters specified timer.
     ///
     /// This method cancels the timer if it is setup and removes the timer
@@ -176,7 +175,6 @@ private:
                                const long interval,
                                const asiolink::IntervalTimer::Mode& scheduling_mode);
 
-
     /// @brief Unregisters specified timer.
     ///
     /// This method cancels the timer if it is setup and removes the timer

src/lib/dhcpsrv/timer_mgr.h

@@ -73,7 +73,7 @@ public:
 
     /// @brief Sets IO service to be used by the Timer Manager.
     ///
-    /// @param io_service Pointer to the new IO service.
+    /// @param io_service Pointer to the IO service.
     void setIOService(const asiolink::IOServicePtr& io_service);
 
     /// @name Registering, unregistering and scheduling the timers.

src/lib/mysql/mysql_connection.cc

@@ -6,6 +6,7 @@
 
 #include <config.h>
 
+#include <database/database_connection.h>
 #include <database/db_log.h>
 #include <exceptions/exceptions.h>
 #include <mysql/mysql_connection.h>
@@ -243,7 +244,23 @@ MySqlConnection::openDatabase() {
     MYSQL* status = mysql_real_connect(mysql_, host, user, password, name,
                                        port, NULL, CLIENT_FOUND_ROWS);
     if (status != mysql_) {
-        isc_throw(DbOpenError, mysql_error(mysql_));
+        std::string error_message = mysql_error(mysql_);
+
+        auto const& rec = reconnectCtl();
+        if (rec && DatabaseConnection::retry_) {
+            // Start the connection recovery.
+            startRecoverDbConnection();
+
+            std::ostringstream s;
+
+            s << " (scheduling retry " << rec->retriesLeft() << " in " << rec->retryInterval() << " milliseconds)";
+
+            error_message += s.str();
+
+            isc_throw(DbOpenErrorWithRetry, error_message);
+        }
+
+        isc_throw(DbOpenError, error_message);
     }
 
     // Enable autocommit. In case transaction is explicitly used, this
@@ -268,9 +285,16 @@ MySqlConnection::openDatabase() {
 // Get schema version.
 
 std::pair<uint32_t, uint32_t>
-MySqlConnection::getVersion(const ParameterMap& parameters) {
+MySqlConnection::getVersion(const ParameterMap& parameters,
+                            const IOServiceAccessorPtr& ac,
+                            const DbCallback& cb,
+                            const string& timer_name) {
     // Get a connection.
-    MySqlConnection conn(parameters);
+    MySqlConnection conn(parameters, ac, cb);
+
+    if (!timer_name.empty()) {
+        conn.makeReconnectCtl(timer_name);
+    }
 
     // Open the database.
     conn.openDatabase();

src/lib/mysql/mysql_connection.h

@@ -260,6 +260,9 @@ public:
     ///
     /// @param parameters A data structure relating keywords and values
     ///        concerned with the database.
+    /// @param ac An IOServiceAccessor object.
+    /// @param cb The dbReconnect callback.
+    /// @param timer_name The DB reconnect timer name.
     ///
     /// @return Version number as a pair of unsigned integers.  "first" is the
     ///         major version number, "second" the minor number.
@@ -267,7 +270,10 @@ public:
     /// @throw isc::db::DbOperationError An operation on the open database has
     ///        failed.
     static std::pair<uint32_t, uint32_t>
-    getVersion(const ParameterMap& parameters);
+    getVersion(const ParameterMap& parameters,
+               const IOServiceAccessorPtr& ac = IOServiceAccessorPtr(),
+               const DbCallback& cb = DbCallback(),
+               const std::string& timer_name = "");
 
     /// @brief Prepare Single Statement
     ///

src/lib/pgsql/pgsql_connection.cc

@@ -6,6 +6,7 @@
 
 #include <config.h>
 
+#include <database/database_connection.h>
 #include <database/db_exceptions.h>
 #include <database/db_log.h>
 #include <pgsql/pgsql_connection.h>
@@ -134,9 +135,16 @@ PgSqlConnection::~PgSqlConnection() {
 }
 
 std::pair<uint32_t, uint32_t>
-PgSqlConnection::getVersion(const ParameterMap& parameters) {
+PgSqlConnection::getVersion(const ParameterMap& parameters,
+                            const IOServiceAccessorPtr& ac,
+                            const DbCallback& cb,
+                            const string& timer_name) {
     // Get a connection.
-    PgSqlConnection conn(parameters);
+    PgSqlConnection conn(parameters, ac, cb);
+
+    if (!timer_name.empty()) {
+        conn.makeReconnectCtl(timer_name);
+    }
 
     // Open the database.
     conn.openDatabaseInternal(false);
@@ -294,6 +302,21 @@ PgSqlConnection::openDatabaseInternal(bool logging) {
         // to release it, but grab the error message first.
         std::string error_message = PQerrorMessage(new_conn);
         PQfinish(new_conn);
+
+        auto const& rec = reconnectCtl();
+        if (rec && DatabaseConnection::retry_) {
+            // Start the connection recovery.
+            startRecoverDbConnection();
+
+            std::ostringstream s;
+
+            s << " (scheduling retry " << rec->retriesLeft() << " in " << rec->retryInterval() << " milliseconds)";
+
+            error_message += s.str();
+
+            isc_throw(DbOpenErrorWithRetry, error_message);
+        }
+
         isc_throw(DbOpenError, error_message);
     }
 

src/lib/pgsql/pgsql_connection.h

@@ -230,6 +230,9 @@ public:
     ///
     /// @param parameters A data structure relating keywords and values
     ///        concerned with the database.
+    /// @param ac An IOServiceAccessor object.
+    /// @param cb The dbReconnect callback.
+    /// @param timer_name The DB reconnect timer name.
     ///
     /// @return Version number as a pair of unsigned integers.  "first" is the
     ///         major version number, "second" the minor number.
@@ -237,7 +240,10 @@ public:
     /// @throw isc::db::DbOperationError An operation on the open database has
     ///        failed.
     static std::pair<uint32_t, uint32_t>
-    getVersion(const ParameterMap& parameters);
+    getVersion(const ParameterMap& parameters,
+               const IOServiceAccessorPtr& ac = IOServiceAccessorPtr(),
+               const DbCallback& cb = DbCallback(),
+               const std::string& timer_name = "");
 
     /// @brief Prepare Single Statement
     ///

src/lib/process/cb_ctl_base.h

@@ -9,6 +9,7 @@
 
 #include <database/audit_entry.h>
 #include <database/backend_selector.h>
+#include <database/database_connection.h>
 #include <database/server_selector.h>
 #include <process/config_base.h>
 #include <process/config_ctl_info.h>
@@ -142,9 +143,15 @@ public:
 
         // Iterate over the configured DBs and instantiate them.
         for (auto db : config_ctl->getConfigDatabases()) {
+            const std::string& redacted = db.redactedAccessString();
             LOG_INFO(dctl_logger, DCTL_OPEN_CONFIG_DB)
-                .arg(db.redactedAccessString());
+                .arg(redacted);
+            try {
                 getMgr().addBackend(db.getAccessString());
+            } catch (const isc::db::DbOpenErrorWithRetry& err) {
+                LOG_INFO(dctl_logger, DCTL_DB_OPEN_CONNECTION_WITH_RETRY_FAILED)
+                        .arg(redacted).arg(err.what());
+            }
         }
 
         // Let the caller know we have opened DBs.

src/lib/process/process_messages.cc

@@ -21,6 +21,7 @@ extern const isc::log::MessageID DCTL_CONFIG_LOAD_FAIL = "DCTL_CONFIG_LOAD_FAIL"
 extern const isc::log::MessageID DCTL_CONFIG_START = "DCTL_CONFIG_START";
 extern const isc::log::MessageID DCTL_CONFIG_STUB = "DCTL_CONFIG_STUB";
 extern const isc::log::MessageID DCTL_CONFIG_UPDATE = "DCTL_CONFIG_UPDATE";
+extern const isc::log::MessageID DCTL_DB_OPEN_CONNECTION_WITH_RETRY_FAILED = "DCTL_DB_OPEN_CONNECTION_WITH_RETRY_FAILED";
 extern const isc::log::MessageID DCTL_DEVELOPMENT_VERSION = "DCTL_DEVELOPMENT_VERSION";
 extern const isc::log::MessageID DCTL_INIT_PROCESS = "DCTL_INIT_PROCESS";
 extern const isc::log::MessageID DCTL_INIT_PROCESS_FAIL = "DCTL_INIT_PROCESS_FAIL";
@@ -58,12 +59,13 @@ const char* values[] = {
     "DCTL_CONFIG_START", "parsing new configuration: %1",
     "DCTL_CONFIG_STUB", "%1 configuration stub handler called",
     "DCTL_CONFIG_UPDATE", "%1 updated configuration received: %2",
+    "DCTL_DB_OPEN_CONNECTION_WITH_RETRY_FAILED", "Failed to connect to database: %1 with error: %2",
     "DCTL_DEVELOPMENT_VERSION", "This software is a development branch of Kea. It is not recommended for production use.",
     "DCTL_INIT_PROCESS", "%1 initializing the application",
     "DCTL_INIT_PROCESS_FAIL", "%1 application initialization failed: %2",
     "DCTL_NOT_RUNNING", "%1 application instance is not running",
     "DCTL_OPEN_CONFIG_DB", "Opening configuration database: %1",
-    "DCTL_PARSER_FAIL", ": %1",
+    "DCTL_PARSER_FAIL", "Parser error: %1",
     "DCTL_PID_FILE_ERROR", "%1 could not create a PID file: %2",
     "DCTL_PROCESS_FAILED", "%1 application execution failed: %2",
     "DCTL_RUN_PROCESS", "%1 starting application event loop",

src/lib/process/process_messages.h

@@ -22,6 +22,7 @@ extern const isc::log::MessageID DCTL_CONFIG_LOAD_FAIL;
 extern const isc::log::MessageID DCTL_CONFIG_START;
 extern const isc::log::MessageID DCTL_CONFIG_STUB;
 extern const isc::log::MessageID DCTL_CONFIG_UPDATE;
+extern const isc::log::MessageID DCTL_DB_OPEN_CONNECTION_WITH_RETRY_FAILED;
 extern const isc::log::MessageID DCTL_DEVELOPMENT_VERSION;
 extern const isc::log::MessageID DCTL_INIT_PROCESS;
 extern const isc::log::MessageID DCTL_INIT_PROCESS_FAIL;

src/lib/process/process_messages.mes

@@ -96,7 +96,13 @@ This message is printed when the Kea server is attempting to open a
 configuration database.  The database access string with password redacted
 is logged.
 
-% DCTL_PARSER_FAIL : %1
+% DCTL_DB_OPEN_CONNECTION_WITH_RETRY_FAILED Failed to connect to database: %1 with error: %2
+This is an informational message issued when the the server failed to connect to
+the configuration database. The operation started a retry to connect procedure.
+The database access string with password redacted is logged, along with the
+error and details for the reconnect procedure.
+
+% DCTL_PARSER_FAIL Parser error: %1
 On receipt of a new configuration, the server failed to create a parser to
 decode the contents of the named configuration element, or the creation
 succeeded but the parsing actions and committal of changes failed.