postfix-2.11-20130319

postfix/HISTORY

@@ -18288,3 +18288,13 @@ Apologies for any names omitted.
 20130318
 
 	Portability: botched #ifdef. File: util/dict_lmdb.c.
+
+20130319
+
+	Postfix support for LMDB databases is withdrawn due to the
+	existence of a hard limit (an "out of storage" failure mode
+	that cannot be resolved by increasing the database size).
+
+	Postfix may support LMDB again when without exceptions all
+	"out of storage" failure modes are resolved by increasing
+	the database size.

postfix/README_FILES/DATABASE_README

@@ -210,12 +210,11 @@ To find out what database types your Postfix system supports, use the "ppooss
     iinntteerrnnaall
         A non-shared, in-memory hash table. Its content are lost when a process
         terminates.
-    llmmddbb
+    llmmddbb (unsupported)
         The OpenLDAP LMDB database (a memory-mapped, persistent file). Database
         files are created with the postmap(1) or postalias(1) command. The
         database name as used in "lmdb:table" is the database file name without
-        the ".lmdb" suffix. This database type has unexpected limitations and
+        the ".lmdb" suffix.
-        is therefore not part of the stable Postfix release.
     llddaapp (read-only)
         Perform lookups using the LDAP protocol. Configuration details are
         given in the ldap_table(5).

postfix/README_FILES/LMDB_README

@@ -4,6 +4,14 @@ PPoossttffiixx OOppeennLLDDAAPP LLMMDDBB HHoowwttoo
 
 IInnttrroodduuccttiioonn
 
+Note:
+    Postfix support for LMDB databases is withdrawn due to the existence of a
+    hard limit (an "out of storage" failure mode that cannot be resolved by
+    increasing the database size).
+
+    Postfix may support LMDB again when without exception all "out of storage"
+    failure modes are resolved by increasing the database size.
+
 Postfix uses databases of various kinds to store and look up information.
 Postfix databases are specified as "type:name". OpenLDAP LMDB implements the
 Postfix database type "lmdb". The name of a Postfix OpenLDAP LMDB database is
@@ -70,6 +78,20 @@ that don't exist with other Postfix databases.
 
 UUnneexxppeecctteedd ppoossttmmaapp((11))//ppoossttaalliiaass((11)) ""ddaattaabbaassee ffuullll"" eerrrroorrss..
 
+Problem:
+    The "postmap lmdb:filename" command fails with an MDB_TXN_FULL error. This
+    problem does not exist with other Postfix databases.
+
+Background:
+    The LMDB implementation has a hard limit on the total transaction size.
+    This limit is independent of the LMDB database size. Therefore, the problem
+    cannot be resolved by increasing the lmdb_map_size value.
+
+    This symptom is indicative of a flawed design. All LMDB data structures
+    should share the same storage pool so that they can scale with the database
+    size, and so that all "out of storage" errors are resolved by increasing
+    the database size.
+
 Problem:
     The "postmap lmdb:filename" command fails with an MDB_MAP_FULL error. This
     problem does not exist with other Postfix databases.

postfix/RELEASE_NOTES

@@ -14,6 +14,16 @@ specifies the release date of a stable release or snapshot release.
 If you upgrade from Postfix 2.9 or earlier, read RELEASE_NOTES-2.10
 before proceeding.
 
+Major changes with snapshot 20130319
+====================================
+
+Postfix support for LMDB databases is withdrawn due to the existence
+of a hard limit (an "out of storage" failure mode that cannot be
+resolved by increasing the database size).
+
+Postfix may support LMDB again when without exception all "out of
+storage" failure modes are resolved by increasing the database size.
+
 Major changes with snapshot 20130315
 ====================================
 

postfix/html/DATABASE_README.html

@@ -311,14 +311,12 @@ name as used in "hash:table" is the database file name without the
 <dd> A non-shared, in-memory hash table. Its content are lost when
 a process terminates. </dd>
 
-<dt> <b>lmdb</b> </dt>
+<dt> <b>lmdb</b> (unsupported) </dt>
 
 <dd> The OpenLDAP LMDB database (a memory-mapped, persistent file).
 Database files are created with the <a href="postmap.1.html">postmap(1)</a> or <a href="postalias.1.html">postalias(1)</a>
 command. The database name as used in "<a href="LMDB_README.html">lmdb</a>:table" is the database
-file name without the ".lmdb" suffix. This database type has
+file name without the ".lmdb" suffix.
-unexpected limitations and is therefore not part of the stable
-Postfix release.
 
 <dt> <b>ldap</b> (read-only) </dt>
 

postfix/html/LMDB_README.html

@@ -19,6 +19,13 @@
 
 <h2>Introduction</h2>
 
+<dl> <dt> Note: </dt> <dd> <p> Postfix support for LMDB databases
+is withdrawn due to the existence of a hard limit (an "out of
+storage" failure mode that cannot be resolved by increasing the
+database size). </p> <p> Postfix may support LMDB again when without
+exception all "out of storage" failure modes are resolved by
+increasing the database size.  </p> </dd> </dl>
+
 <p> Postfix uses databases of various kinds to store and look up
 information. Postfix databases are specified as "type:name".
 OpenLDAP LMDB implements the Postfix database type "lmdb".
@@ -120,6 +127,24 @@ errors.  </strong></p>
 
 <dl>
 
+<dt> Problem: </dt> <dd> <p> The "postmap <a href="LMDB_README.html">lmdb</a>:filename" command
+fails with an MDB_TXN_FULL error.  This problem does not exist with
+other Postfix databases. </p> </dd>
+
+<dt> Background: </dt>
+
+<dd>
+
+<p> The LMDB implementation has a hard limit on the total transaction
+size. This limit is independent of the LMDB database size. Therefore,
+the problem cannot be resolved by increasing the <a href="postconf.5.html#lmdb_map_size">lmdb_map_size</a>
+value. </p>
+
+<p> This symptom is indicative of a flawed design.  All LMDB data
+structures should share the same storage pool so that they can scale
+with the database size, and so that all "out of storage" errors are
+resolved by increasing the database size. </p> </dd>
+
 <dt> Problem: </dt> <dd> <p> The "postmap <a href="LMDB_README.html">lmdb</a>:filename" command
 fails with an MDB_MAP_FULL error.  This problem does not exist with
 other Postfix databases. </p> </dd>

postfix/html/postconf.1.html

@@ -195,6 +195,10 @@ POSTCONF(1)                                                        POSTCONF(1)
                      A non-shared, in-memory hash table. Its con-
                      tent are lost when a process terminates.
 
+              <b>lmdb</b> (unsupported)
+                     The OpenLDAP LMDB database (a memory-mapped,
+                     persistent file).
+
               <b>ldap</b> (read-only)
                      Perform  lookups  using  the  LDAP protocol.
                      This is described in <a href="ldap_table.5.html"><b>ldap_table</b>(5)</a>.

postfix/man/man1/postconf.1

@@ -183,6 +183,9 @@ databases.
 .IP \fBinternal\fR
 A non-shared, in-memory hash table. Its content are lost
 when a process terminates.
+.IP "\fBlmdb\fR (unsupported)"
+The OpenLDAP LMDB database (a memory-mapped, persistent
+file).
 .IP "\fBldap\fR (read-only)"
 Perform lookups using the LDAP protocol. This is described
 in \fBldap_table\fR(5).

postfix/proto/DATABASE_README.html

@@ -311,14 +311,12 @@ name as used in "hash:table" is the database file name without the
 <dd> A non-shared, in-memory hash table. Its content are lost when
 a process terminates. </dd>
 
-<dt> <b>lmdb</b> </dt>
+<dt> <b>lmdb</b> (unsupported) </dt>
 
 <dd> The OpenLDAP LMDB database (a memory-mapped, persistent file).
 Database files are created with the postmap(1) or postalias(1)
 command. The database name as used in "lmdb:table" is the database
-file name without the ".lmdb" suffix. This database type has
+file name without the ".lmdb" suffix.
-unexpected limitations and is therefore not part of the stable
-Postfix release.
 
 <dt> <b>ldap</b> (read-only) </dt>
 

postfix/proto/LMDB_README.html

@@ -19,6 +19,13 @@
 
 <h2>Introduction</h2>
 
+<dl> <dt> Note: </dt> <dd> <p> Postfix support for LMDB databases
+is withdrawn due to the existence of a hard limit (an "out of
+storage" failure mode that cannot be resolved by increasing the
+database size). </p> <p> Postfix may support LMDB again when without
+exception all "out of storage" failure modes are resolved by
+increasing the database size.  </p> </dd> </dl>
+
 <p> Postfix uses databases of various kinds to store and look up
 information. Postfix databases are specified as "type:name".
 OpenLDAP LMDB implements the Postfix database type "lmdb".
@@ -120,6 +127,24 @@ errors.  </strong></p>
 
 <dl>
 
+<dt> Problem: </dt> <dd> <p> The "postmap lmdb:filename" command
+fails with an MDB_TXN_FULL error.  This problem does not exist with
+other Postfix databases. </p> </dd>
+
+<dt> Background: </dt>
+
+<dd>
+
+<p> The LMDB implementation has a hard limit on the total transaction
+size. This limit is independent of the LMDB database size. Therefore,
+the problem cannot be resolved by increasing the lmdb_map_size
+value. </p>
+
+<p> This symptom is indicative of a flawed design.  All LMDB data
+structures should share the same storage pool so that they can scale
+with the database size, and so that all "out of storage" errors are
+resolved by increasing the database size. </p> </dd>
+
 <dt> Problem: </dt> <dd> <p> The "postmap lmdb:filename" command
 fails with an MDB_MAP_FULL error.  This problem does not exist with
 other Postfix databases. </p> </dd>

postfix/src/postconf/postconf.c

@@ -177,6 +177,9 @@
 /* .IP \fBinternal\fR
 /*	A non-shared, in-memory hash table. Its content are lost
 /*	when a process terminates.
+/* .IP "\fBlmdb\fR (unsupported)"
+/*	The OpenLDAP LMDB database (a memory-mapped, persistent
+/*	file).
 /* .IP "\fBldap\fR (read-only)"
 /*	Perform lookups using the LDAP protocol. This is described
 /*	in \fBldap_table\fR(5).

postfix/src/util/dict_lmdb.c

@@ -22,11 +22,11 @@
 /*	size the table can grow to, so it must be set large enough
 /*	to accomodate the largest tables in use.
 /*
-/*	As a safety measure, when Postfix opens an LMDB database it
+/*	As a safety measure, when Postfix opens an LMDB database
-/*	will set the memory size limit to at least 3x the
+/*	it will set the memory map size to at least 3x the ".lmdb"
-/*	".lmdb" file size, so that there is room for the file to
+/*	file size, so that there is room for the file to grow. This
-/*	grow. This ensures continued availability of Postfix daemon
+/*	ensures that a process can recover from a "table full" error
-/*	processes.
+/*	with a simple terminate-and-restart.
 /* DIAGNOSTICS
 /*	Fatal errors: cannot open file, file write error, out of memory.
 /* SEE ALSO
@@ -472,11 +472,11 @@ DICT   *dict_lmdb_open(const char *path, int open_flags, int dict_flags)
 
     /*
      * Try to ensure that the LMDB size limit is at least 3x the current LMDB
-     * file size. This should be sufficient to ensure that short-lived
+     * file size. This ensures that Postfix daemon processes can recover from
-     * Postfix daemon processes can recover from a "table full" error.
+     * a "table full" error with a simple terminate-and-restart.
      * 
-     * Note: readers must increase their LMDB size limit, too, otherwise they
+     * Note: read-only applications must increase their LMDB size limit, too,
-     * won't be able to continue reading a table after grows.
+     * otherwise they won't be able to read a table after it grows.
      */
 #ifndef SIZE_T_MAX
 #define SIZE_T_MAX __MAXINT__(size_t)