mirror of
https://github.com/vdukhovni/postfix
synced 2025-08-30 21:55:20 +00:00
259 lines
9.3 KiB
HTML
259 lines
9.3 KiB
HTML
<!doctype html public "-//W3C//DTD HTML 4.01 Transitional//EN"
|
|
"http://www.w3.org/TR/html4/loose.dtd">
|
|
|
|
<html>
|
|
|
|
<head>
|
|
|
|
<title>Postfix OpenLDAP LMDB Howto</title>
|
|
|
|
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
|
|
|
|
</head>
|
|
|
|
<body>
|
|
|
|
<h1><img src="postfix-logo.jpg" width="203" height="98" ALT="">Postfix OpenLDAP LMDB Howto</h1>
|
|
|
|
<hr>
|
|
|
|
<h2>Introduction</h2>
|
|
|
|
<p> Postfix uses databases of various kinds to store and look up
|
|
information. Postfix databases are specified as "type:name".
|
|
OpenLDAP LMDB implements the Postfix database type "lmdb".
|
|
The name of a Postfix OpenLDAP LMDB database is the name
|
|
of the database file without the ".lmdb" suffix. OpenLDAP LMDB databases
|
|
are maintained with the <a href="postmap.1.html">postmap(1)</a> and <a href="postalias.1.html">postalias(1)</a> commands. </p>
|
|
|
|
<p> This document describes: </p>
|
|
|
|
<ol>
|
|
|
|
<li> <p> How to build Postfix <a href="#with_lmdb">with OpenLDAP
|
|
LMDB support</a>. </p>
|
|
|
|
<li> <p> How to <a href="#configure">configure</a> LMDB settings. </p>
|
|
|
|
<li> <p> Missing <a href="#pthread">pthread</a> library trouble. </p>
|
|
|
|
<li> <p> Unexpected <a href="#limitations">failure modes</a> that
|
|
don't exist with other Postfix databases. </p>
|
|
|
|
</ol>
|
|
|
|
<h2><a name="with_lmdb">Building Postfix with OpenLDAP LMDB support</a></h2>
|
|
|
|
<p> Postfix normally does not enable OpenLDAP LMDB support. To
|
|
build Postfix with OpenLDAP LMDB support, use something like: </p>
|
|
|
|
<blockquote>
|
|
<pre>
|
|
% make makefiles CCARGS="-DHAS_LMDB -I/usr/local/include" \
|
|
AUXLIBS="-L/usr/local/lib -llmdb"
|
|
% make
|
|
</pre>
|
|
</blockquote>
|
|
|
|
<p> Solaris may need this: </p>
|
|
|
|
<blockquote>
|
|
<pre>
|
|
% make makefiles CCARGS="-DHAS_LMDB -I/usr/local/include" \
|
|
AUXLIBS="-R/usr/local/lib -L/usr/local/lib -llmdb"
|
|
% make
|
|
</pre>
|
|
</blockquote>
|
|
|
|
<p> The exact pathnames depend on how OpenLDAP LMDB was installed. </p>
|
|
|
|
<h2><a name="configure">Configure LMDB settings</a></h2>
|
|
|
|
<p> Postfix provides one configuration parameter that controls
|
|
OpenLDAP LMDB database behavior. </p>
|
|
|
|
<ul>
|
|
|
|
<li> <p> <a href="postconf.5.html#lmdb_map_size">lmdb_map_size</a> (default: 16777216). This setting specifies
|
|
the initial OpenLDAP LMDB database size limit in bytes. Each time
|
|
a database becomes full, its size limit is doubled. </p>
|
|
|
|
</ul>
|
|
|
|
<h2><a name="pthread">Missing pthread library trouble</a></h2>
|
|
|
|
<p> When building Postfix fails with: </p>
|
|
|
|
<blockquote>
|
|
<pre>
|
|
undefined reference to `pthread_mutexattr_destroy'
|
|
undefined reference to `pthread_mutexattr_init'
|
|
undefined reference to `pthread_mutex_lock'
|
|
</pre>
|
|
</blockquote>
|
|
|
|
<p> Add the "-lpthread" library to the "make makefiles" command. </p>
|
|
|
|
<blockquote>
|
|
<pre>
|
|
% make makefiles .... AUXLIBS="... -lpthread"
|
|
</pre>
|
|
</blockquote>
|
|
|
|
<p> Source code for OpenLDAP LMDB is available at
|
|
<a href="http://www.openldap.org">http://www.openldap.org</a>.
|
|
More information is available at
|
|
<a href="http://highlandsun.com/hyc/mdb/">http://highlandsun.com/hyc/mdb/</a>. </p>
|
|
|
|
<h2><a name="limitations">Unexpected failure modes of Postfix LMDB
|
|
databases. </a> </h2>
|
|
|
|
<p> As documented below, conversion to LMDB introduces a number of
|
|
failure modes that don't exist with other Postfix databases. Some
|
|
failure modes have been eliminated on the course of time.
|
|
The writeup below reflects the status as of of LMDB 0.9.8. </p>
|
|
|
|
<!--
|
|
|
|
<p> <strong>Unexpected <a href="postmap.1.html">postmap(1)</a>/<a href="postalias.1.html">postalias(1)</a> "database full"
|
|
errors. </strong></p>
|
|
|
|
<dl>
|
|
|
|
<dt> Problem: </dt> <dd> <p> The "postmap <a href="LMDB_README.html">lmdb</a>:filename" command
|
|
fails with an MDB_TXN_FULL error. This problem does not exist with
|
|
other Postfix databases. </p> </dd>
|
|
|
|
<dt> Background: </dt>
|
|
|
|
<dd>
|
|
|
|
<p> The LMDB implementation has a hard limit on the total transaction
|
|
size. This limit is independent of the LMDB database size. Therefore,
|
|
the problem cannot be resolved by increasing the <a href="postconf.5.html#lmdb_map_size">lmdb_map_size</a>
|
|
value. </p>
|
|
|
|
<p> This symptom is indicative of a flawed design. All LMDB data
|
|
structures should share the same storage pool so that they can scale
|
|
with the database size, and so that all "out of storage" errors are
|
|
resolved by increasing the database size. </p> </dd>
|
|
|
|
<dt> Problem: </dt> <dd> <p> The "postmap <a href="LMDB_README.html">lmdb</a>:filename" command
|
|
fails with an MDB_MAP_FULL error. This problem does not exist with
|
|
other Postfix databases. </p> </dd>
|
|
|
|
<dt> Background: </dt>
|
|
|
|
<dd>
|
|
|
|
<p> LMDB databases have a hard size limit (configured with the
|
|
<a href="postconf.5.html#lmdb_map_size">lmdb_map_size</a> configuration parameter). </p>
|
|
|
|
<p> When executing "postmap <a href="LMDB_README.html">lmdb</a>:filename", the Postfix LMDB database
|
|
client stores the new data in a transaction which takes up space
|
|
in addition to the existing data, and commits the transaction when
|
|
it closes the database. Only then can the space for old data be
|
|
reused. </p>
|
|
|
|
</dd>
|
|
|
|
<dt> Impact: </dt> <dd> <p> This failure does not affect Postfix
|
|
availability, because the old data still exists in the database.
|
|
</p> </dd>
|
|
|
|
<dt> Mitigation: </dt> <dd>
|
|
|
|
<p> When the <a href="postmap.1.html">postmap(1)</a> or <a href="postalias.1.html">postalias(1)</a> command fails with an
|
|
MDB_MAP_FULL error, it expands the database file size to the current
|
|
LMDB map size limit before terminating. </p>
|
|
|
|
<p> Next, when you re-run the <a href="postmap.1.html">postmap(1)</a> or <a href="postalias.1.html">postalias(1)</a> command,
|
|
it discovers that the LMDB file is larger than <a href="postconf.5.html#lmdb_map_size">lmdb_map_size</a>/3,
|
|
logs a warning, and uses a larger LMDB map size limit instead: </p>
|
|
|
|
<p> <tt> warning: <i>filename</i>.<a href="LMDB_README.html">lmdb</a>: file size 15024128 ≥
|
|
(lmdb map size limit 16777216)/3<br> warning: <i>filename</i>.<a href="LMDB_README.html">lmdb</a>:
|
|
using map size limit 45072384</tt> </p>
|
|
|
|
<p> By repeating the two steps above you can automate recovery and
|
|
avoid the need for human intervention. Just repeat "postmap
|
|
<a href="LMDB_README.html">lmdb</a>:filename" (up to some limit). After each failure it will use
|
|
a 3x larger size limit, and eventually the "database full" error
|
|
should disappear. This fails only when the disk is full or when
|
|
the LMDB map size limit would exceed the memory address space size
|
|
limit. </p>
|
|
|
|
<dt> Prevention: </dt> <dd> <p> Monitor your LMDB files and make
|
|
sure that in <a href="postconf.5.html">main.cf</a>, <a href="postconf.5.html#lmdb_map_size">lmdb_map_size</a> > 3x the largest LMDB file
|
|
size. </p> </dd> </dl>
|
|
|
|
<p> <strong>Unexpected Postfix daemon "database full" errors.
|
|
</strong></p>
|
|
|
|
<dl>
|
|
|
|
<dt> Problem: </dt> <dd> <p> Postfix daemon programs fail with
|
|
"database full" errors, such as <a href="postscreen.8.html">postscreen(8)</a>, <a href="tlsmgr.8.html">tlsmgr(8)</a> or <a href="verify.8.html">verify(8)</a>.
|
|
This problem does not exist with other Postfix databases. </p>
|
|
</dd>
|
|
|
|
<dt> Impact: </dt> <dd> <p> This failure temporarily affects Postfix
|
|
availability. The daemon restarts automatically and tries to open
|
|
the database again as described next. </p> </dd>
|
|
|
|
<dt> Mitigation: </dt> <dd> <p> When a Postfix daemon opens an LMDB
|
|
file larger than <a href="postconf.5.html#lmdb_map_size">lmdb_map_size</a>/3, it logs a warning and uses a
|
|
larger size limit instead: </p>
|
|
|
|
<p> <tt> warning: <i>filename</i>.<a href="LMDB_README.html">lmdb</a>: file size 15024128 ≥
|
|
(lmdb map size limit 16777216)/3 <br>warning: <i>filename</i>.<a href="LMDB_README.html">lmdb</a>:
|
|
using map size limit 45072384</tt> </p>
|
|
|
|
<p> This can be used to automate recovery and avoid the need for
|
|
human intervention. Each time the daemon runs into a "database full"
|
|
error, it restarts and uses a 3x larger size limit. The "database
|
|
full" error will disappear, at least for a while. </p>
|
|
|
|
<dt> Prevention: </dt> <dd> <p> Monitor your LMDB files and make
|
|
sure that <a href="postconf.5.html#lmdb_map_size">lmdb_map_size</a> > 3x the largest LMDB file size. </p>
|
|
</dd> </dl>
|
|
|
|
-->
|
|
|
|
<p> <strong>Non-obvious recovery with <a href="postmap.1.html">postmap(1)</a>/<a href="postalias.1.html">postalias(1)</a>/<a href="tlsmgr.8.html">tlsmgr(8)</a>
|
|
from a corrupted database. </strong></p>
|
|
|
|
<dl>
|
|
|
|
<dt> Problem: </dt> <dd> <p> You cannot rebuild a corrupted LMDB
|
|
database simply by re-running <a href="postmap.1.html">postmap(1)</a> or <a href="postalias.1.html">postalias(1)</a>, or by
|
|
waiting until the <a href="tlsmgr.8.html">tlsmgr(8)</a> daemon restarts. This problem
|
|
does not exist with other Postfix databases. </p> </dd>
|
|
|
|
<dt> Background: </dt> <dd> <p> The Postfix LMDB database client
|
|
does not truncate the database file. Instead it attempts to create
|
|
a transaction for a "drop" request plus subsequent "store" requests.
|
|
That is obviously not possible with a corrupted database file. </p>
|
|
</dd>
|
|
|
|
<dt> Impact: </dt> <dd> <p> Postfix does not process mail until
|
|
someone fixes the problem. </p> </dd>
|
|
|
|
<dt> Recovery: </dt> <dd> <p> First delete the ".lmdb" file by hand.
|
|
Then, rebuild the file with the <a href="postmap.1.html">postmap(1)</a> or <a href="postalias.1.html">postalias(1)</a> command,
|
|
or wait until the <a href="tlsmgr.8.html">tlsmgr(8)</a> daemon restarts. </p>
|
|
</dd>
|
|
|
|
<dt> Prevention: </dt> <dd>
|
|
|
|
<p> Arrange your file systems such that they never run out of free
|
|
space. </p>
|
|
|
|
<p> Use ECC memory to detect and correct silent corruption of
|
|
in-memory file system data and metadata. </p>
|
|
|
|
<p> Use a file system such as ZFS to detect and correct silent
|
|
corruption of on-disk file system data and metadata. </p>
|
|
|
|
</dd> </dl>
|