[master] Database reconnect now supported for MySQL lease/host back ends

Merges branch 'trac5556a'

doc/guide/dhcp4-srv.xml

@@ -480,7 +480,26 @@ be followed by a comma and another object definition.</para>
 The default value of five seconds should be more than adequate for local connections.
 If a timeout is given though, it should be an integer greater than zero.
   </para>
-
+  <para>
+The maxiumum number of times the server will automatically attempt to reconnect to
+the lease database after connectivity has been lost may be specified:
+<screen>
+"Dhcp4": { "lease-database": { <userinput>"max-reconnect-tries" : <replaceable>number-of-tries</replaceable></userinput>, ... }, ... }
+</screen>
+If the server is unable to reconnect to the database after making the maximum number
+of attempts the server will exit. A value of zero (the default) disables automatic
+recovery and the server will exit immediately upon detecting a loss of connectivity
+(MySQL and Postgres only).
+  </para>
+  <para>
+The number of seconds the server will wait in between attempts to reconnect to the
+lease database after connectivity has been lost may also be specified:
+<screen>
+"Dhcp4": { "lease-database": { <userinput>"reconnect-wait-time" : <replaceable>number-of-seconds</replaceable></userinput>, ... }, ... }
+</screen>
+A value of zero (the default) disables automatic recovery and the server will exit
+immediately upon detecting a loss of connectivity (MySQL and Postgres only).
+    </para>
   <para>Finally, the credentials of the account under which the server will
   access the database should be set:
 <screen>
@@ -627,6 +646,26 @@ Similar parameters can be specified for hosts database.
 <screen>
 "Dhcp4": { "hosts-database": { <userinput>"port" : 12345</userinput>, ... }, ... }
 </screen>
+  <para>
+The maxiumum number of times the server will automatically attempt to reconnect to
+the host database after connectivity has been lost may be specified:
+<screen>
+"Dhcp4": { "hosts-database": { <userinput>"max-reconnect-tries" : <replaceable>number-of-tries</replaceable></userinput>, ... }, ... }
+</screen>
+If the server is unable to reconnect to the database after making the maximum number
+of attempts the server will exit. A value of zero (the default) disables automatic
+recovery and the server will exit immediately upon detecting a loss of connectivity
+(MySQL and Postgres only).
+  </para>
+  <para>
+The number of seconds the server will wait in between attempts to reconnect to the
+host database after connectivity has been lost may also be specified:
+<screen>
+"Dhcp4": { "hosts-database": { <userinput>"reconnect-wait-time" : <replaceable>number-of-seconds</replaceable></userinput>, ... }, ... }
+</screen>
+A value of zero (the default) disables automatic recovery and the server will exit
+immediately upon detecting a loss of connectivity (MySQL and Postgres only).
+    </para>
 
   </para>
   <para>Finally, the credentials of the account under which the server will

doc/guide/dhcp6-srv.xml

@@ -471,7 +471,7 @@ be followed by a comma and another object definition.</para>
   Should the database use a port different than default, it may be
   specified as well:
 <screen>
-"Dhcp4": { "lease-database": { <userinput>"port" : 12345</userinput>, ... }, ... }
+"Dhcp6": { "lease-database": { <userinput>"port" : 12345</userinput>, ... }, ... }
 </screen>
   Should the database be located on a different system, you may need to specify a longer interval
   for the connection timeout:
@@ -481,14 +481,33 @@ be followed by a comma and another object definition.</para>
 The default value of five seconds should be more than adequate for local connections.
 If a timeout is given though, it should be an integer greater than zero.
   </para>
-
+  <para>
+The maxiumum number of times the server will automatically attempt to reconnect to
+the lease database after connectivity has been lost may be specified:
+<screen>
+"Dhcp6": { "lease-database": { <userinput>"max-reconnect-tries" : <replaceable>number-of-tries</replaceable></userinput>, ... }, ... }
+</screen>
+If the server is unable to reconnect to the database after making the maximum number
+of attempts the server will exit. A value of zero (the default) disables automatic
+recovery and the server will exit immediately upon detecting a loss of connectivity
+(MySQL and Postgres only).
+  </para>
+  <para>
+The number of seconds the server will wait in between attempts to reconnect to the
+lease database after connectivity has been lost may also be specified:
+<screen>
+"Dhcp6": { "lease-database": { <userinput>"reconnect-wait-time" : <replaceable>number-of-seconds</replaceable></userinput>, ... }, ... }
+</screen>
+A value of zero (the default) disables automatic recovery and the server will exit
+immediately upon detecting a loss of connectivity (MySQL and Postgres only).
+  </para>
   <para>
     Note that host parameter is used by MySQL and PostgreSQL
     backends. Cassandra has a concept of contact points that could be
     used to contact the cluster, instead of a single IP or
     hostname. It takes a list of comma separated IP addresses. This may be specified as:
 <screen>
-"Dhcp4": { "lease-database": { <userinput>"contact-points" : "192.0.2.1,192.0.2.2"</userinput>, ... }, ... }
+"Dhcp6": { "lease-database": { <userinput>"contact-points" : "192.0.2.1,192.0.2.2"</userinput>, ... }, ... }
 </screen>
   </para>
 
@@ -565,8 +584,28 @@ linkend="cassandra-database-configuration4"/> for details.</para>
 "Dhcp6": { "hosts-database": { <userinput>"host" : ""</userinput>, ... }, ... }
 </screen>
 <screen>
-"Dhcp4": { "hosts-database": { <userinput>"port" : 12345</userinput>, ... }, ... }
+"Dhcp6": { "hosts-database": { <userinput>"port" : 12345</userinput>, ... }, ... }
 </screen>
+  </para>
+  <para>
+The maxiumum number of times the server will automatically attempt to reconnect to
+the host database after connectivity has been lost may be specified:
+<screen>
+"Dhcp6": { "host-database": { <userinput>"max-reconnect-tries" : <replaceable>number-of-tries</replaceable></userinput>, ... }, ... }
+</screen>
+If the server is unable to reconnect to the database after making the maximum number
+of attempts the server will exit. A value of zero (the default) disables automatic
+recovery and the server will exit immediately upon detecting a loss of connectivity
+(MySQL and Postgres only).
+  </para>
+  <para>
+The number of seconds the server will wait in between attempts to reconnect to the
+host database after connectivity has been lost may also be specified:
+<screen>
+"Dhcp6": { "hosts-database": { <userinput>"reconnect-wait-time" : <replaceable>number-of-seconds</replaceable></userinput>, ... }, ... }
+</screen>
+A value of zero (the default) disables automatic recovery and the server will exit
+immediately upon detecting a loss of connectivity (MySQL and Postgres only).
   </para>
   <para>Finally, the credentials of the account under which the server will
   access the database should be set:

doc/guide/kea-guide.xml

@@ -31,7 +31,7 @@
     <releaseinfo>This is the reference guide for Kea version &keaversion;.</releaseinfo>
 
     <copyright>
-      <year>2010-2017</year>
+      <year>2010-2018</year>
       <holder>Internet Systems Consortium, Inc. ("ISC")</holder>
     </copyright>
 

src/bin/dhcp4/dhcp4_messages.mes

@@ -157,7 +157,7 @@ occur. It will prohibit the server from attempting to reconnect to its
 databases if connectivy is lost, and the server will exit. This error
 should be reported.
 
-% DHCP4_DB_RECONNECT_DISABLED database reconnect is disabled: max-reconnect-tries %1, reconnect-wait-time %1
+% DHCP4_DB_RECONNECT_DISABLED database reconnect is disabled: max-reconnect-tries %1, reconnect-wait-time %2
 This is an informational message indicating that connectivity to either the
 lease or host database or both and that automatic reconnect is not enabled.
 

src/bin/dhcp6/dhcp6_messages.mes

@@ -117,7 +117,7 @@ occur. It will prohibit the server from attempting to reconnect to its
 databases if connectivy is lost, and the server will exit. This error
 should be reported.
 
-% DHCP6_DB_RECONNECT_DISABLED database reconnect is disabled: max-reconnect-tries %1, reconnect-wait-time %1
+% DHCP6_DB_RECONNECT_DISABLED database reconnect is disabled: max-reconnect-tries %1, reconnect-wait-time %2
 This is an informational message indicating that connectivity to either the
 lease or host database or both and that automatic reconnect is not enabled.
 

src/lib/dhcpsrv/dhcpsrv_messages.mes

@@ -704,10 +704,12 @@ leases which have expired longer than a specified period of time.
 The argument is the amount of time Kea waits after a reclaimed
 lease expires before considering its removal.
 
-% DHCPSRV_MYSQL_FATAL_ERROR Unrecoverable MySQL error occurred: %1 for <%2>, reason: %3 (error code: %4). Server exiting now!
+% DHCPSRV_MYSQL_FATAL_ERROR Unrecoverable MySQL error occurred: %1 for <%2>, reason: %3 (error code: %4).
 An error message indicating that communication with the MySQL database server
-has been lost.  When this occurs the server exits immediately with a non-zero
+has been lost.  If automatic recovery has been enabled,  then the server will
-exit code.  This is most likely due to a network issue.
+attempt to recover connectivity.  If not the server wil exit with a
+non-zero exit code.  The cause of such an error is most likely a network issue
+or the MySQL server has gone down.
 
 % DHCPSRV_MYSQL_GET4 obtaining all IPv4 leases
 A debug message issued when the server is attempting to obtain all IPv4
@@ -854,7 +856,7 @@ connection including database name and username needed to access it
 % DHCPSRV_PGSQL_DEALLOC_ERROR An error occurred deallocating SQL statements while closing the PostgreSQL lease database: %1
 This is an error message issued when a DHCP server (either V4 or V6) experienced
 and error freeing database SQL resources as part of closing its connection to
-the Postgresql database.  The connection is closed as part of normal server
+the PostgreSQL database.  The connection is closed as part of normal server
 shutdown.  This error is most likely a programmatic issue that is highly
 unlikely to occur or negatively impact server operation.
 
@@ -874,10 +876,12 @@ leases which have expired longer than a specified period of time.
 The argument is the amount of time Kea waits after a reclaimed
 lease expires before considering its removal.
 
-% DHCPSRV_PGSQL_FATAL_ERROR Unrecoverable PostgreSQL error occurred: Statement: <%1>, reason: %2 (error code: %3). Server exiting now!
+% DHCPSRV_PGSQL_FATAL_ERROR Unrecoverable PostgreSQL error occurred: Statement: <%1>, reason: %2 (error code: %3).
-An error message indicating that communication with the MySQL database server
+An error message indicating that communication with the PostgreSQL database server
-has been lost.  When this occurs the server exits immediately with a non-zero
+has been lost.  If automatic recovery has been enabled,  then the server will
-exit code.  This is most likely due to a network issue.
+attempt to recover the connectivity.  If not the server wil exit with a
+non-zero exit code.  The cause of such an error is most likely a network issue
+or the PostgreSQL server has gone down.
 
 % DHCPSRV_PGSQL_GET4 obtaining all IPv4 leases
 A debug message issued when the server is attempting to obtain all IPv4

src/lib/dhcpsrv/mysql_connection.h

@@ -355,15 +355,16 @@ public:
     /// in the event of failures, decide whether or not those failures are
     /// recoverable.
     ///
-    /// If the error is recoverable, the method will throw a DbOperationError.
+    /// If the error is recoverable, the function will throw a DbOperationError.
-    /// In the error is deemed unrecoverable, such as a loss of connectivity
+    /// If the error is deemed unrecoverable, such as a loss of connectivity
-    /// with the server, this method will log the error and call exit(-1);
+    /// with the server, the function will call invokeDbLostCallback(). If the
+    /// invocation returns false then either there is no callback registered
+    /// or the callback has elected not to attempt to reconnect, and exit(-1)
+    /// is called;
     ///
-    /// @todo Calling exit() is viewed as a short term solution for Kea 1.0.
+    /// If the invocation returns true, this indicates the calling layer will
-    /// Two tickets are likely to alter this behavior, first is #3639, which
+    /// attempt recovery, and the function throws a DbOperationError to allow
-    /// calls for the ability to attempt to reconnect to the database. The
+    /// the caller to error handle the failed db access attempt.
-    /// second ticket, #4087 which calls for the implementation of a generic,
-    /// FatalException class which will propagate outward.
     ///
     /// @param status Status code: non-zero implies an error
     /// @param index Index of statement that caused the error
@@ -389,14 +390,22 @@ public:
             case CR_SERVER_LOST:
             case CR_OUT_OF_MEMORY:
             case CR_CONNECTION_ERROR:
-                // We're exiting on fatal
                 DB_LOG_ERROR(MYSQL_FATAL_ERROR)
                     .arg(what)
                     .arg(text_statements_[static_cast<int>(index)])
                     .arg(mysql_error(mysql_))
                     .arg(mysql_errno(mysql_));
-                exit (-1);
 
+                // If there's no lost db callback or it returns false,
+                // then we're not attempting to recover so we're done
+                if (!invokeDbLostCallback()) {
+                    exit (-1);
+                }
+
+                // We still need to throw so caller can error out of the current
+                // processing.
+                isc_throw(DbOperationError,
+                          "fatal database errror or connectivity lost");
             default:
                 // Connection is ok, so it must be an SQL error
                 isc_throw(DbOperationError, what << " for <"

src/lib/dhcpsrv/mysql_lease_mgr.cc

@@ -2240,11 +2240,7 @@ MySqlLeaseMgr::getVersion() const {
 
     // Execute the prepared statement
     int status = mysql_stmt_execute(conn_.statements_[stindex]);
-    if (status != 0) {
+    checkError(status, stindex, "unable to execute statement");
-        isc_throw(DbOperationError, "unable to execute <"
-                  << conn_.text_statements_[stindex] << "> - reason: " <<
-                  mysql_error(conn_.mysql_));
-    }
 
     // Bind the output of the statement to the appropriate variables.
     MYSQL_BIND bind[2];
@@ -2261,19 +2257,13 @@ MySqlLeaseMgr::getVersion() const {
     bind[1].buffer_length = sizeof(minor);
 
     status = mysql_stmt_bind_result(conn_.statements_[stindex], bind);
-    if (status != 0) {
+    checkError(status, stindex, "unable to bind result set");
-        isc_throw(DbOperationError, "unable to bind result set: " <<
-                  mysql_error(conn_.mysql_));
-    }
 
     // Fetch the data and set up the "release" object to release associated
     // resources when this method exits then retrieve the data.
     MySqlFreeResult fetch_release(conn_.statements_[stindex]);
     status = mysql_stmt_fetch(conn_.statements_[stindex]);
-    if (status != 0) {
+    checkError(status, stindex, "unable to fetch result set");
-        isc_throw(DbOperationError, "unable to obtain result set: " <<
-                  mysql_error(conn_.mysql_));
-    }
 
     return (std::make_pair(major, minor));
 }

src/lib/dhcpsrv/pgsql_connection.cc

@@ -40,11 +40,16 @@ const char PgSqlConnection::DUPLICATE_KEY[] = ERRCODE_UNIQUE_VIOLATION;
 PgSqlResult::PgSqlResult(PGresult *result)
     : result_(result), rows_(0), cols_(0) {
     if (!result) {
-        isc_throw (BadValue, "PgSqlResult result pointer cannot be null");
+        // Certain failures, like a loss of connectivity, can return a
-    }
+        // null PGresult and we still need to be able to create a PgSqlResult.
-
+        // We'll set row and col counts to -1 to prevent anyone going off the
+        // rails.
+        rows_ = -1;
+        cols_ = -1;
+    } else {
         rows_ = PQntuples(result);
         cols_ = PQnfields(result);
+    }
 }
 
 void
@@ -302,7 +307,9 @@ PgSqlConnection::checkStatementError(const PgSqlResult& r,
                 .arg(statement.name)
                 .arg(PQerrorMessage(conn_))
                 .arg(sqlstate ? sqlstate : "<sqlstate null>");
-            // If there's no lost db callback, then exit
+
+            // If there's no lost db callback or it returns false,
+            // then we're not attempting to recover so we're done
             if (!invokeDbLostCallback()) {
                 exit (-1);
             }

src/lib/dhcpsrv/pgsql_connection.h

@@ -89,6 +89,10 @@ public:
     /// Store the pointer to the result set to being fetched.  Set row
     /// and column counts for convenience.
     ///
+    /// @param result - pointer to the Postgresql client layer result
+    /// If the value of is NULL, row and col values will be set to -1.
+    /// This allows PgSqlResult to be passed into statement error
+    /// checking.
     PgSqlResult(PGresult *result);
 
     /// @brief Destructor
@@ -378,15 +382,16 @@ public:
     /// execution succeeded, and in the event of failures, decide whether or
     /// not the failures are recoverable.
     ///
-    /// If the error is recoverable, the method will throw a DbOperationError.
+    /// If the error is recoverable, the function will throw a DbOperationError.
-    /// In the error is deemed unrecoverable, such as a loss of connectivity
+    /// If the error is deemed unrecoverable, such as a loss of connectivity
-    /// with the server, this method will log the error and call exit(-1);
+    /// with the server, the function will call invokeDbLostCallback(). If the
+    /// invocation returns false then either there is no callback registered
+    /// or the callback has elected not to attempt to reconnect, and exit(-1)
+    /// is called;
     ///
-    /// @todo Calling exit() is viewed as a short term solution for Kea 1.0.
+    /// If the invocation returns true, this indicates the calling layer will
-    /// Two tickets are likely to alter this behavior, first is #3639, which
+    /// attempt recovery, and the function throws a DbOperationError to allow
-    /// calls for the ability to attempt to reconnect to the database. The
+    /// the caller to error handle the failed db access attempt.
-    /// second ticket, #4087 which calls for the implementation of a generic,
-    /// FatalException class which will propagate outward.
     ///
     /// @param r result of the last PostgreSQL operation
     /// @param statement - tagged statement that was executed

src/lib/dhcpsrv/tests/generic_lease_mgr_unittest.cc

@@ -9,6 +9,7 @@
 #include <asiolink/io_address.h>
 #include <dhcpsrv/cfgmgr.h>
 #include <dhcpsrv/database_connection.h>
+#include <dhcpsrv/lease_mgr_factory.h>
 #include <dhcpsrv/tests/generic_lease_mgr_unittest.h>
 #include <dhcpsrv/tests/test_utils.h>
 #include <stats/stats_mgr.h>
@@ -2800,6 +2801,62 @@ GenericLeaseMgrTest::testWipeLeases4() {
     EXPECT_EQ(0, lmptr_->wipeLeases4(333));
 }
 
+void
+LeaseMgrDbLostCallbackTest::SetUp() {
+    destroySchema();
+    createSchema();
+    isc::dhcp::LeaseMgrFactory::destroy();
+}
+
+void
+LeaseMgrDbLostCallbackTest::TearDown() {
+    destroySchema();
+    isc::dhcp::LeaseMgrFactory::destroy();
+}
+
+void
+LeaseMgrDbLostCallbackTest::testNoCallbackOnOpenFailure() {
+    DatabaseConnection::db_lost_callback =
+        boost::bind(&LeaseMgrDbLostCallbackTest::db_lost_callback, this, _1);
+
+    callback_called_ = false;
+    ASSERT_THROW(LeaseMgrFactory::create(invalidConnectString()),
+                 DbOpenError);
+
+    EXPECT_FALSE(callback_called_);
+}
+
+void
+LeaseMgrDbLostCallbackTest::testDbLostCallback() {
+    // Set the connectivity lost callback.
+    DatabaseConnection::db_lost_callback =
+        boost::bind(&LeaseMgrDbLostCallbackTest::db_lost_callback, this, _1);
+
+    // Connect to the lease backend.
+    ASSERT_NO_THROW(LeaseMgrFactory::create(validConnectString()));
+
+    // The most recently opened socket should be for our SQL client.
+    int sql_socket = test::findLastSocketFd();
+    ASSERT_TRUE(sql_socket > -1);
+
+    // Clear the callback invocation marker.
+    callback_called_ = false;
+
+    // Verify we can execute a query.
+    LeaseMgr& lm = LeaseMgrFactory::instance();
+    pair<uint32_t, uint32_t> version;
+    ASSERT_NO_THROW(version = lm.getVersion());
+
+    // Now close the sql socket out from under backend client
+    ASSERT_EQ(0, close(sql_socket));
+
+    // A query should fail with DbOperationError.
+    ASSERT_THROW(version = lm.getVersion(), DbOperationError);
+
+    // Our lost connectivity callback should have been invoked.
+    EXPECT_TRUE(callback_called_);
+}
+
 }; // namespace test
 }; // namespace dhcp
 }; // namespace isc

src/lib/dhcpsrv/tests/generic_lease_mgr_unittest.h

@@ -413,6 +413,71 @@ public:
     LeaseMgr* lmptr_;
 };
 
+class LeaseMgrDbLostCallbackTest : public ::testing::Test {
+public:
+    LeaseMgrDbLostCallbackTest() {
+        DatabaseConnection::db_lost_callback = 0;
+    }
+
+    virtual ~LeaseMgrDbLostCallbackTest() {
+        DatabaseConnection::db_lost_callback = 0;
+    }
+
+    /// @brief Prepares the class for a test.
+    ///
+    /// Invoked by gtest prior test entry, we create the
+    /// appropriate schema and wipe out any residual lease manager
+    virtual void SetUp();
+
+    /// @brief Pre-text exit clean up
+    ///
+    /// Invoked by gtest upon test exit, we destroy the schema
+    /// we created and toss our lease manager.
+    virtual void TearDown();
+
+    /// @brief Abstract method for destroying the back end specific shcema
+    virtual void destroySchema() = 0;
+
+    /// @brief Abstract method for creating the back end specific shcema
+    virtual void createSchema() = 0;
+
+    /// @brief Abstract method which returns the back end specific connection
+    /// string
+    virtual std::string validConnectString() = 0;
+
+    /// @brief Abstract method which returns invalid back end specific connection
+    /// string
+    virtual std::string invalidConnectString() = 0;
+
+    /// @brief Verifies open failures do NOT invoke db lost callback
+    ///
+    /// The db lost callback should only be invoked after succesfully
+    /// opening the DB and then subsequently losing it. Failing to
+    /// open should be handled directly by the application layer.
+    void testNoCallbackOnOpenFailure();
+
+    /// @brief Verifies the host manager's behavior if DB connection is lost
+    ///
+    /// This function creates a lease manager with an back end that
+    /// supports connectivity lost callback (currently only MySQL and
+    /// PostgreSQL currently).  It verifies connectivity by issuing a known
+    /// valid query.  Next it simulates connectivity lost by identifying and
+    /// closing the socket connection to the host backend.  It then reissues
+    /// the query and verifies that:
+    /// -# The Query throws  DbOperationError (rather than exiting)
+    /// -# The registered DbLostCallback was invoked
+    void testDbLostCallback();
+
+    /// @brief Callback function registered with the host manager
+    bool db_lost_callback(ReconnectCtlPtr /* not_used */) {
+        return (callback_called_ = true);
+    }
+
+    /// @brief Flag used to detect calls to db_lost_callback function
+    bool callback_called_;
+
+};
+
 }; // namespace test
 }; // namespace dhcp
 }; // namespace isc

src/lib/dhcpsrv/tests/host_mgr_unittest.cc

@@ -11,6 +11,7 @@
 #include <dhcpsrv/host.h>
 #include <dhcpsrv/host_data_source_factory.h>
 #include <dhcpsrv/host_mgr.h>
+#include <dhcpsrv/tests/test_utils.h>
 
 #if defined HAVE_MYSQL
 #include <dhcpsrv/testutils/mysql_schema.h>
@@ -537,6 +538,102 @@ TEST_F(HostMgrTest, addNoDataSource) {
     EXPECT_THROW(HostMgr::instance().add(host), NoHostDataSourceManager);
 }
 
+class HostMgrDbLostCallbackTest : public ::testing::Test {
+public:
+    HostMgrDbLostCallbackTest() : callback_called_(false) {};
+
+    /// @brief Prepares the class for a test.
+    ///
+    /// Invoked by gtest prior test entry, we create the
+    /// appropriate schema and create a basic host manager to
+    /// wipe out any prior instance
+    virtual void SetUp() {
+        DatabaseConnection::db_lost_callback = 0;  
+        destroySchema();
+        createSchema();
+        // Wipe out any pre-existing mgr
+        HostMgr::create();
+    }
+
+    /// @brief Pre-text exit clean up
+    ///
+    /// Invoked by gtest upon test exit, we destroy the schema
+    /// we created.
+    virtual void TearDown() {
+        DatabaseConnection::db_lost_callback = 0;  
+        destroySchema();
+    }
+
+    /// @brief Abstract method for destroying the back end specific shcema
+    virtual void destroySchema() = 0;
+
+    /// @brief Abstract method for creating the back end specific shcema
+    virtual void createSchema() = 0;
+
+    /// @brief Abstract method which returns the back end specific connection
+    /// string
+    virtual std::string validConnectString() = 0;
+
+    /// @brief Verifies the host manager's behavior if DB connection is lost
+    ///
+    /// This function creates a host manager with an alternate data source
+    /// that supports connectivity lost callback (currently only MySQL and
+    /// PostgreSQL currently).  It verifies connectivity by issuing a known
+    /// valid query.  Next it simulates connectivity lost by identifying and
+    /// closing the socket connection to the host backend.  It then reissues
+    /// the query and verifies that:
+    /// -# The Query throws  DbOperationError (rather than exiting)
+    /// -# The registered DbLostCallback was invoked
+    void testDbLostCallback();
+
+    /// @brief Callback function registered with the host manager
+    bool db_lost_callback(ReconnectCtlPtr /* not_used */) {
+        return (callback_called_ = true);
+    }
+
+    /// @brief Flag used to detect calls to db_lost_callback function
+    bool callback_called_;
+};
+
+
+void
+HostMgrDbLostCallbackTest::testDbLostCallback() {
+    // Create the HostMgr.
+    HostMgr::create();
+
+    // Set the connectivity lost callback.
+    DatabaseConnection::db_lost_callback =
+        boost::bind(&HostMgrDbLostCallbackTest::db_lost_callback, this, _1);
+
+    // Find the most recently opened socket. Our SQL client's socket should
+    // be the next one.
+    int last_open_socket = test::findLastSocketFd();
+
+    // Connect to the host backend.
+    ASSERT_NO_THROW(HostMgr::addBackend(validConnectString()));
+
+    // Find the SQL client socket.
+    int sql_socket = test::findLastSocketFd();
+    ASSERT_TRUE(sql_socket > last_open_socket);
+
+    // Clear the callback invocation marker.
+    callback_called_ = false;
+
+    // Verify we can execute a query.  We don't care about the answer.
+    ConstHostCollection hosts;
+    ASSERT_NO_THROW(hosts = HostMgr::instance().getAll4(IOAddress("192.0.2.5")));
+
+    // Now close the sql socket out from under backend client
+    ASSERT_FALSE(close(sql_socket)) << "failed to close socket";
+
+    // A query should fail with DbOperationError.
+    ASSERT_THROW(hosts = HostMgr::instance().getAll4(IOAddress("192.0.2.5")),
+                 DbOperationError);
+
+    // Our lost connectivity callback should have been invoked.
+    EXPECT_TRUE(callback_called_);
+}
+
 // The following tests require MySQL enabled.
 #if defined HAVE_MYSQL
 
@@ -580,6 +677,23 @@ MySQLHostMgrTest::TearDown() {
     test::destroyMySQLSchema();
 }
 
+/// @brief Test fixture class for validating @c HostMgr using
+/// MySQL as alternate host data source and MySQL connectivity loss.
+class MySQLHostMgrDbLostCallbackTest : public HostMgrDbLostCallbackTest {
+public:
+    virtual void destroySchema() {
+        test::destroyMySQLSchema();
+    }
+
+    virtual void createSchema() {
+        test::createMySQLSchema();
+    }
+
+    virtual std::string validConnectString() {
+        return (test::validMySQLConnectionString());
+    }
+};
+
 // This test verifies that reservations for a particular client can
 // be retrieved from the configuration file and a database simultaneously.
 TEST_F(MySQLHostMgrTest, getAll) {
@@ -610,6 +724,10 @@ TEST_F(MySQLHostMgrTest, get6ByPrefix) {
     testGet6ByPrefix(*getCfgHosts(), HostMgr::instance());
 }
 
+// Verifies that loss of connectivity to MySQL is handled correctly.
+TEST_F(MySQLHostMgrDbLostCallbackTest, testDbLostCallback) {
+    testDbLostCallback();
+}
 #endif
 
 
@@ -656,7 +774,23 @@ PostgreSQLHostMgrTest::TearDown() {
     test::destroyPgSQLSchema();
 }
 
-// This test verifies that reservations for a particular client can
+/// @brief Test fixture class for validating @c HostMgr using
+/// PostgreSQL as alternate host data source and PostgreSQL connectivity loss.
+class PostgreSQLHostMgrDbLostCallbackTest : public HostMgrDbLostCallbackTest {
+public:
+    virtual void destroySchema() {
+        test::destroyPgSQLSchema();
+    }
+
+    virtual void createSchema() {
+        test::createPgSQLSchema();
+    }
+
+    virtual std::string validConnectString() {
+        return (test::validPgSQLConnectionString());
+    }
+};
+
 // be retrieved from the configuration file and a database simultaneously.
 TEST_F(PostgreSQLHostMgrTest, getAll) {
     testGetAll(*getCfgHosts(), HostMgr::instance());
@@ -686,9 +820,12 @@ TEST_F(PostgreSQLHostMgrTest, get6ByPrefix) {
     testGet6ByPrefix(*getCfgHosts(), HostMgr::instance());
 }
 
+// Verifies that loss of connectivity to PostgreSQL is handled correctly.
+TEST_F(PostgreSQLHostMgrDbLostCallbackTest, testDbLostCallback) {
+    testDbLostCallback();
+}
 #endif
 
-
 // The following tests require Cassandra enabled.
 #if defined HAVE_CQL
 

src/lib/dhcpsrv/tests/mysql_lease_mgr_unittest.cc

@@ -536,4 +536,36 @@ TEST_F(MySqlLeaseMgrTest, DISABLED_wipeLeases6) {
     testWipeLeases6();
 }
 
+/// @brief Test fixture class for validating @c LeaseMgr using
+/// MySQL as back end and MySQL connectivity loss.
+class MySQLLeaseMgrDbLostCallbackTest : public LeaseMgrDbLostCallbackTest {
+public:
+    virtual void destroySchema() {
+        test::destroyMySQLSchema();
+    }
+
+    virtual void createSchema() {
+        test::createMySQLSchema();
+    }
+
+    virtual std::string validConnectString() {
+        return (test::validMySQLConnectionString());
+    }
+
+    virtual std::string invalidConnectString() {
+        return (connectionString(MYSQL_VALID_TYPE, VALID_NAME, INVALID_HOST,
+                        VALID_USER, VALID_PASSWORD));
+    }
+};
+
+// Verifies that db lost callback is not invoked on an open failure
+TEST_F(MySQLLeaseMgrDbLostCallbackTest, testNoCallbackOnOpenFailure) {
+    testDbLostCallback();
+}
+
+// Verifies that loss of connectivity to MySQL is handled correctly.
+TEST_F(MySQLLeaseMgrDbLostCallbackTest, testDbLostCallback) {
+    testDbLostCallback();
+}
+
 }  // namespace

src/lib/dhcpsrv/tests/pgsql_lease_mgr_unittest.cc

@@ -194,35 +194,36 @@ TEST(PgSqlOpenTest, OpenDatabase) {
     destroyPgSQLSchema(true);
 }
 
-/// @brief Flag used to detect calls to db_lost_callback function
+/// @brief Test fixture class for validating @c LeaseMgr using
-bool callback_called = false;
+/// PostgreSQL as back end and PostgreSQL connectivity loss.
+class PgSqlLeaseMgrDbLostCallbackTest : public LeaseMgrDbLostCallbackTest {
+public:
+    virtual void destroySchema() {
+        test::destroyPgSQLSchema();
+    }
 
-/// @brief Callback function used in open database testing
+    virtual void createSchema() {
-bool db_lost_callback(ReconnectCtlPtr /* db_conn_retry */) {
+        test::createPgSQLSchema();
-    return (callback_called = true);
+    }
+
+    virtual std::string validConnectString() {
+        return (test::validPgSQLConnectionString());
+    }
+
+    virtual std::string invalidConnectString() {
+        return (connectionString(PGSQL_VALID_TYPE, VALID_NAME, INVALID_HOST,
+                        VALID_USER, VALID_PASSWORD));
+    }
+};
+
+// Verifies that db lost callback is not invoked on an open failure
+TEST_F(PgSqlLeaseMgrDbLostCallbackTest, testNoCallbackOnOpenFailure) {
+    testDbLostCallback();
 }
 
-/// @brief Make sure open failures do NOT invoke db lost callback
+// Verifies that loss of connectivity to PostgreSQL is handled correctly.
-/// The db lost callback should only be invoked after succesfully
+TEST_F(PgSqlLeaseMgrDbLostCallbackTest, testDbLostCallback) {
-/// opening the DB and then subsequently losing it. Failing to
+    testDbLostCallback();
-/// open should be handled directly by the application layer.
-/// There is simply no good way to break the connection in a
-/// unit test environment.  So testing the callback invocation
-/// in a unit test is next to impossible. That has to be done
-/// as a system test.
-TEST(PgSqlOpenTest, NoCallbackOnOpenFail) {
-    // Schema needs to be created for the test to work.
-    destroyPgSQLSchema();
-    createPgSQLSchema();
-
-    callback_called = false;
-    DatabaseConnection::db_lost_callback = db_lost_callback;
-    EXPECT_THROW(LeaseMgrFactory::create(connectionString(
-        PGSQL_VALID_TYPE, VALID_NAME, INVALID_HOST, VALID_USER, VALID_PASSWORD)),
-        DbOpenError);
-    EXPECT_FALSE(callback_called);
-
-    destroyPgSQLSchema();
 }
 
 /// @brief Check the getType() method

src/lib/dhcpsrv/tests/test_utils.cc

@@ -1,4 +1,4 @@
-// Copyright (C) 2012-2015 Internet Systems Consortium, Inc. ("ISC")
+// Copyright (C) 2012-2018 Internet Systems Consortium, Inc. ("ISC")
 //
 // This Source Code Form is subject to the terms of the Mozilla Public
 // License, v. 2.0. If a copy of the MPL was not distributed with this
@@ -9,6 +9,9 @@
 #include <asiolink/io_address.h>
 #include <gtest/gtest.h>
 #include <sstream>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <unistd.h>
 
 using namespace std;
 using namespace isc::asiolink;
@@ -76,6 +79,30 @@ detailCompareLease(const Lease6Ptr& first, const Lease6Ptr& second) {
     EXPECT_EQ(first->hostname_, second->hostname_);
 }
 
+int findLastSocketFd() {
+    int max_fd_number = getdtablesize();
+    int last_socket = -1;
+    struct stat stats;
+
+    // Iterate over the open fds
+    for (int fd = 0; fd <= max_fd_number; fd++ ) {
+        errno = 0;
+        fstat(fd, &stats);
+
+        if (errno == EBADF ) {
+            // Skip any that aren't open
+            continue;
+        }
+
+        // it's a socket, remember it
+        if (S_ISSOCK(stats.st_mode)) {
+            last_socket = fd;
+        }
+    }
+
+    return (last_socket);
+}
+
 }; // namespace test
 }; // namespace dhcp
 }; // namespace isc

src/lib/dhcpsrv/tests/test_utils.h

@@ -1,4 +1,4 @@
-// Copyright (C) 2012-2015 Internet Systems Consortium, Inc. ("ISC")
+// Copyright (C) 2012-2018 Internet Systems Consortium, Inc. ("ISC")
 //
 // This Source Code Form is subject to the terms of the Mozilla Public
 // License, v. 2.0. If a copy of the MPL was not distributed with this
@@ -34,6 +34,21 @@ detailCompareLease(const Lease6Ptr& first, const Lease6Ptr& second);
 void
 detailCompareLease(const Lease4Ptr& first, const Lease4Ptr& second);
 
+/// @brief Function that finds the last open socket descriptor
+///
+/// This function is used to attempt lost connectivity
+/// with backends, notably MySQL and Postgresql.
+///
+/// The theory being, that in a confined test environment the last 
+/// such descriptor is the SQL client socket descriptor.  This allows
+/// us to the close that descriptor and simulate a loss of server
+/// connectivity.
+///
+/// @return the descriptor of the last open socket or -1 if there
+/// are none.
+int findLastSocketFd();
+
+
 }; // namespace test
 }; // namespace dhcp
 }; // namespace isc