mirror of
https://github.com/vdukhovni/postfix
synced 2025-08-30 05:38:06 +00:00
postfix-2.11-20130928
This commit is contained in:
Wietse Venema
Viktor Dukhovni
parent
2bf7cc9990
commit
2ffd296c6a
@ -18943,3 +18943,14 @@ Apologies for any names omitted.
|
||||
util/dict_open.c, util/dict_alloc.c, util/dict_lmdb.c,
|
||||
postmap/postmap.c, postalias/postalias.c, proto/LMDB_README.html,
|
||||
proto/postconf.proto.
|
||||
|
||||
20130928
|
||||
|
||||
Cleanup: the lmdb_max_readers property is now configurable.
|
||||
This is a hard limit built into the OpenLDAP library that
|
||||
causes requests to fail when the number of open read
|
||||
transactions exceeds the limit. When this happens the LMDB
|
||||
client logs a MDB_READERS_FULL warning and continues with
|
||||
reduced performance. Files: util/dict_lmdb.c, util/dict_lmdb.h,
|
||||
global/mail_params.h, global/mail_params.c, proto/postconf.proto,
|
||||
proto/LMDB_README.html.
|
||||
|
||||
@ -39,13 +39,19 @@ The exact pathnames depend on how OpenLDAP LMDB was installed.
|
||||
|
||||
CCoonnffiigguurree LLMMDDBB sseettttiinnggss
|
||||
|
||||
Postfix provides one configuration parameter that controls OpenLDAP LMDB
|
||||
database behavior.
|
||||
Postfix provides configuration parameters that control OpenLDAP LMDB database
|
||||
behavior.
|
||||
|
||||
* lmdb_map_size (default: 16777216). This setting specifies the initial
|
||||
OpenLDAP LMDB database size limit in bytes. Each time a database becomes
|
||||
full, its size limit is doubled.
|
||||
|
||||
* lmdb_max_readers (default: $default_process_limit). This specifies a hard
|
||||
limit on the number of read transactions that may be open at the same time
|
||||
for the same OpenLDAP LMDB database. When this number is too small, the
|
||||
Postfix LMDB client will log MDB_READERS_FULL warnings, and will run with
|
||||
reduced performance.
|
||||
|
||||
MMiissssiinngg pptthhrreeaadd lliibbrraarryy ttrroouubbllee
|
||||
|
||||
When building Postfix fails with:
|
||||
@ -68,6 +74,26 @@ that don't exist with other Postfix databases. Some failure modes have been
|
||||
eliminated on the course of time. The writeup below reflects the status as of
|
||||
of LMDB 0.9.8.
|
||||
|
||||
UUnneexxppeecctteedd ""rreeaaddeerrss ffuullll"" eerrrroorrss..
|
||||
|
||||
Problem:
|
||||
Under heavy load, database read operations fail with MDB_READERS_FULL
|
||||
errors. This problem does not exist with other Postfix databases.
|
||||
|
||||
Background:
|
||||
The LMDB implementation enforces a hard limit on the number of simultaneous
|
||||
read requests for the same database environment. This limit must be
|
||||
specified with the lmdb_max_readers configuration parameter.
|
||||
|
||||
Mitigation:
|
||||
Postfix logs a warning suggesting that the lmdb_max_readers parameter value
|
||||
be increased, and retries the failed operation for a limited number of
|
||||
times while running with reduced performance.
|
||||
|
||||
Prevention:
|
||||
Monitor your LMDB files for MDB_READERS_FULL errors and make the necessary
|
||||
adjustments. Consider using CDB for read-mostly databases.
|
||||
|
||||
NNoonn--oobbvviioouuss rreeccoovveerryy wwiitthh ppoossttmmaapp((11))//ppoossttaalliiaass((11))//ttllssmmggrr((88)) ffrroomm aa ccoorrrruupptteedd
|
||||
ddaattaabbaassee..
|
||||
|
||||
|
||||
@ -69,7 +69,7 @@ build Postfix with OpenLDAP LMDB support, use something like: </p>
|
||||
|
||||
<h2><a name="configure">Configure LMDB settings</a></h2>
|
||||
|
||||
<p> Postfix provides one configuration parameter that controls
|
||||
<p> Postfix provides configuration parameters that control
|
||||
OpenLDAP LMDB database behavior. </p>
|
||||
|
||||
<ul>
|
||||
@ -78,6 +78,13 @@ OpenLDAP LMDB database behavior. </p>
|
||||
the initial OpenLDAP LMDB database size limit in bytes. Each time
|
||||
a database becomes full, its size limit is doubled. </p>
|
||||
|
||||
<li> <p> <a href="postconf.5.html#lmdb_max_readers">lmdb_max_readers</a> (default: $<a href="postconf.5.html#default_process_limit">default_process_limit</a>). This
|
||||
specifies a hard limit on the number of read transactions that may
|
||||
be open at the same time for the same OpenLDAP LMDB database. When
|
||||
this number is too small, the Postfix LMDB client will log
|
||||
MDB_READERS_FULL warnings, and will run with reduced performance.
|
||||
</p>
|
||||
|
||||
</ul>
|
||||
|
||||
<h2><a name="pthread">Missing pthread library trouble</a></h2>
|
||||
@ -113,6 +120,28 @@ failure modes that don't exist with other Postfix databases. Some
|
||||
failure modes have been eliminated on the course of time.
|
||||
The writeup below reflects the status as of of LMDB 0.9.8. </p>
|
||||
|
||||
<p> <strong>Unexpected "readers full" errors. </strong></p>
|
||||
|
||||
<dl>
|
||||
|
||||
<dt> Problem: </dt> <dd> <p> Under heavy load, database read
|
||||
operations fail with MDB_READERS_FULL errors. This problem does not
|
||||
exist with other Postfix databases. </p> </dd>
|
||||
|
||||
<dt> Background: </dt> <dd> <p> The LMDB implementation enforces a
|
||||
hard limit on the number of simultaneous read requests for the same
|
||||
database environment. This limit must be specified with the
|
||||
<a href="postconf.5.html#lmdb_max_readers">lmdb_max_readers</a> configuration parameter. </p> </dd>
|
||||
|
||||
<dt> Mitigation: </dt> <dd> <p> Postfix logs a warning suggesting
|
||||
that the <a href="postconf.5.html#lmdb_max_readers">lmdb_max_readers</a> parameter value be increased, and retries
|
||||
the failed operation for a limited number of times while running
|
||||
with reduced performance. </p> </dd>
|
||||
|
||||
<dt> Prevention: </dt> <dd> <p> Monitor your LMDB files for
|
||||
MDB_READERS_FULL errors and make the necessary adjustments.
|
||||
Consider using CDB for read-mostly databases. </p> </dd> </dl>
|
||||
|
||||
<!--
|
||||
|
||||
<p> <strong>Unexpected <a href="postmap.1.html">postmap(1)</a>/<a href="postalias.1.html">postalias(1)</a> "database full"
|
||||
|
||||
@ -292,8 +292,9 @@ OQMGR(8) OQMGR(8)
|
||||
deferred message.
|
||||
|
||||
<b><a href="postconf.5.html#maximal_queue_lifetime">maximal_queue_lifetime</a> (5d)</b>
|
||||
The maximal time a message is queued before it is
|
||||
sent back as undeliverable.
|
||||
Consider a message as undeliverable, when delivery
|
||||
fails with a temporary error, and the time in the
|
||||
queue has reached the <a href="postconf.5.html#maximal_queue_lifetime">maximal_queue_lifetime</a> limit.
|
||||
|
||||
<b><a href="postconf.5.html#queue_run_delay">queue_run_delay</a> (300s)</b>
|
||||
The time between <a href="QSHAPE_README.html#deferred_queue">deferred queue</a> scans by the queue
|
||||
@ -308,8 +309,10 @@ OQMGR(8) OQMGR(8)
|
||||
Available in Postfix version 2.1 and later:
|
||||
|
||||
<b><a href="postconf.5.html#bounce_queue_lifetime">bounce_queue_lifetime</a> (5d)</b>
|
||||
The maximal time a bounce message is queued before
|
||||
it is considered undeliverable.
|
||||
Consider a bounce message as undeliverable, when
|
||||
delivery fails with a temporary error, and the time
|
||||
in the queue has reached the <a href="postconf.5.html#bounce_queue_lifetime">bounce_queue_lifetime</a>
|
||||
limit.
|
||||
|
||||
Available in Postfix version 2.5 and later:
|
||||
|
||||
|
||||
@ -3794,6 +3794,18 @@ This feature is available in Postfix 2.11 and later.
|
||||
</p>
|
||||
|
||||
|
||||
</DD>
|
||||
|
||||
<DT><b><a name="lmdb_max_readers">lmdb_max_readers</a>
|
||||
(default: $<a href="postconf.5.html#default_process_limit">default_process_limit</a>)</b></DT><DD>
|
||||
|
||||
<p> The hard limit on the number of read transactions that may be
|
||||
open at the same time for the same OpenLDAP LMDB database. When
|
||||
this number is too small, the Postfix LMDB client will log
|
||||
MDB_READERS_FULL errors, and will run with reduced performance.
|
||||
</p>
|
||||
|
||||
|
||||
</DD>
|
||||
|
||||
<DT><b><a name="lmtp_address_preference">lmtp_address_preference</a>
|
||||
|
||||
@ -364,8 +364,9 @@ QMGR(8) QMGR(8)
|
||||
deferred message.
|
||||
|
||||
<b><a href="postconf.5.html#maximal_queue_lifetime">maximal_queue_lifetime</a> (5d)</b>
|
||||
The maximal time a message is queued before it is
|
||||
sent back as undeliverable.
|
||||
Consider a message as undeliverable, when delivery
|
||||
fails with a temporary error, and the time in the
|
||||
queue has reached the <a href="postconf.5.html#maximal_queue_lifetime">maximal_queue_lifetime</a> limit.
|
||||
|
||||
<b><a href="postconf.5.html#queue_run_delay">queue_run_delay</a> (300s)</b>
|
||||
The time between <a href="QSHAPE_README.html#deferred_queue">deferred queue</a> scans by the queue
|
||||
@ -380,8 +381,10 @@ QMGR(8) QMGR(8)
|
||||
Available in Postfix version 2.1 and later:
|
||||
|
||||
<b><a href="postconf.5.html#bounce_queue_lifetime">bounce_queue_lifetime</a> (5d)</b>
|
||||
The maximal time a bounce message is queued before
|
||||
it is considered undeliverable.
|
||||
Consider a bounce message as undeliverable, when
|
||||
delivery fails with a temporary error, and the time
|
||||
in the queue has reached the <a href="postconf.5.html#bounce_queue_lifetime">bounce_queue_lifetime</a>
|
||||
limit.
|
||||
|
||||
Available in Postfix version 2.5 and later:
|
||||
|
||||
|
||||
@ -2247,6 +2247,11 @@ The initial OpenLDAP LMDB database size limit in bytes. Each time
|
||||
a database becomes full, its size limit is doubled.
|
||||
.PP
|
||||
This feature is available in Postfix 2.11 and later.
|
||||
.SH lmdb_max_readers (default: $default_process_limit)
|
||||
The hard limit on the number of read transactions that may be
|
||||
open at the same time for the same OpenLDAP LMDB database. When
|
||||
this number is too small, the Postfix LMDB client will log
|
||||
MDB_READERS_FULL errors, and will run with reduced performance.
|
||||
.SH lmtp_address_preference (default: ipv6)
|
||||
The LMTP-specific version of the smtp_address_preference
|
||||
configuration parameter. See there for details.
|
||||
|
||||
@ -266,8 +266,9 @@ prior to Postfix 2.4 the default value was 1000s.
|
||||
.IP "\fBmaximal_backoff_time (4000s)\fR"
|
||||
The maximal time between attempts to deliver a deferred message.
|
||||
.IP "\fBmaximal_queue_lifetime (5d)\fR"
|
||||
The maximal time a message is queued before it is sent back as
|
||||
undeliverable.
|
||||
Consider a message as undeliverable, when delivery fails with a
|
||||
temporary error, and the time in the queue has reached the
|
||||
maximal_queue_lifetime limit.
|
||||
.IP "\fBqueue_run_delay (300s)\fR"
|
||||
The time between deferred queue scans by the queue manager;
|
||||
prior to Postfix 2.4 the default value was 1000s.
|
||||
@ -277,8 +278,9 @@ a malfunctioning message delivery transport.
|
||||
.PP
|
||||
Available in Postfix version 2.1 and later:
|
||||
.IP "\fBbounce_queue_lifetime (5d)\fR"
|
||||
The maximal time a bounce message is queued before it is considered
|
||||
undeliverable.
|
||||
Consider a bounce message as undeliverable, when delivery fails
|
||||
with a temporary error, and the time in the queue has reached the
|
||||
bounce_queue_lifetime limit.
|
||||
.PP
|
||||
Available in Postfix version 2.5 and later:
|
||||
.IP "\fBdefault_destination_rate_delay (0s)\fR"
|
||||
|
||||
@ -314,8 +314,9 @@ prior to Postfix 2.4 the default value was 1000s.
|
||||
.IP "\fBmaximal_backoff_time (4000s)\fR"
|
||||
The maximal time between attempts to deliver a deferred message.
|
||||
.IP "\fBmaximal_queue_lifetime (5d)\fR"
|
||||
The maximal time a message is queued before it is sent back as
|
||||
undeliverable.
|
||||
Consider a message as undeliverable, when delivery fails with a
|
||||
temporary error, and the time in the queue has reached the
|
||||
maximal_queue_lifetime limit.
|
||||
.IP "\fBqueue_run_delay (300s)\fR"
|
||||
The time between deferred queue scans by the queue manager;
|
||||
prior to Postfix 2.4 the default value was 1000s.
|
||||
@ -325,8 +326,9 @@ a malfunctioning message delivery transport.
|
||||
.PP
|
||||
Available in Postfix version 2.1 and later:
|
||||
.IP "\fBbounce_queue_lifetime (5d)\fR"
|
||||
The maximal time a bounce message is queued before it is considered
|
||||
undeliverable.
|
||||
Consider a bounce message as undeliverable, when delivery fails
|
||||
with a temporary error, and the time in the queue has reached the
|
||||
bounce_queue_lifetime limit.
|
||||
.PP
|
||||
Available in Postfix version 2.5 and later:
|
||||
.IP "\fBdefault_destination_rate_delay (0s)\fR"
|
||||
|
||||
@ -209,6 +209,7 @@ while (<>) {
|
||||
s;\bipc_ttl\b;<a href="postconf.5.html#ipc_ttl">$&</a>;g;
|
||||
s;\bline_length_limit\b;<a href="postconf.5.html#line_length_limit">$&</a>;g;
|
||||
s;\blmdb_map_size\b;<a href="postconf.5.html#lmdb_map_size">$&</a>;g;
|
||||
s;\blmdb_max_readers\b;<a href="postconf.5.html#lmdb_max_readers">$&</a>;g;
|
||||
s;\blmtp_address_preference\b;<a href="postconf.5.html#lmtp_address_preference">$&</a>;g;
|
||||
s;\blmtp_body_checks\b;<a href="postconf.5.html#lmtp_body_checks">$&</a>;g;
|
||||
s;\blmtp_cname_overrides_servername\b;<a href="postconf.5.html#lmtp_cname_overrides_servername">$&</a>;g;
|
||||
|
||||
@ -69,7 +69,7 @@ build Postfix with OpenLDAP LMDB support, use something like: </p>
|
||||
|
||||
<h2><a name="configure">Configure LMDB settings</a></h2>
|
||||
|
||||
<p> Postfix provides one configuration parameter that controls
|
||||
<p> Postfix provides configuration parameters that control
|
||||
OpenLDAP LMDB database behavior. </p>
|
||||
|
||||
<ul>
|
||||
@ -78,6 +78,13 @@ OpenLDAP LMDB database behavior. </p>
|
||||
the initial OpenLDAP LMDB database size limit in bytes. Each time
|
||||
a database becomes full, its size limit is doubled. </p>
|
||||
|
||||
<li> <p> lmdb_max_readers (default: $default_process_limit). This
|
||||
specifies a hard limit on the number of read transactions that may
|
||||
be open at the same time for the same OpenLDAP LMDB database. When
|
||||
this number is too small, the Postfix LMDB client will log
|
||||
MDB_READERS_FULL warnings, and will run with reduced performance.
|
||||
</p>
|
||||
|
||||
</ul>
|
||||
|
||||
<h2><a name="pthread">Missing pthread library trouble</a></h2>
|
||||
@ -113,6 +120,28 @@ failure modes that don't exist with other Postfix databases. Some
|
||||
failure modes have been eliminated on the course of time.
|
||||
The writeup below reflects the status as of of LMDB 0.9.8. </p>
|
||||
|
||||
<p> <strong>Unexpected "readers full" errors. </strong></p>
|
||||
|
||||
<dl>
|
||||
|
||||
<dt> Problem: </dt> <dd> <p> Under heavy load, database read
|
||||
operations fail with MDB_READERS_FULL errors. This problem does not
|
||||
exist with other Postfix databases. </p> </dd>
|
||||
|
||||
<dt> Background: </dt> <dd> <p> The LMDB implementation enforces a
|
||||
hard limit on the number of simultaneous read requests for the same
|
||||
database environment. This limit must be specified with the
|
||||
lmdb_max_readers configuration parameter. </p> </dd>
|
||||
|
||||
<dt> Mitigation: </dt> <dd> <p> Postfix logs a warning suggesting
|
||||
that the lmdb_max_readers parameter value be increased, and retries
|
||||
the failed operation for a limited number of times while running
|
||||
with reduced performance. </p> </dd>
|
||||
|
||||
<dt> Prevention: </dt> <dd> <p> Monitor your LMDB files for
|
||||
MDB_READERS_FULL errors and make the necessary adjustments.
|
||||
Consider using CDB for read-mostly databases. </p> </dd> </dl>
|
||||
|
||||
<!--
|
||||
|
||||
<p> <strong>Unexpected postmap(1)/postalias(1) "database full"
|
||||
|
||||
@ -2848,6 +2848,15 @@ a database becomes full, its size limit is doubled.
|
||||
This feature is available in Postfix 2.11 and later.
|
||||
</p>
|
||||
|
||||
%PARAM lmdb_max_readers $default_process_limit
|
||||
|
||||
<p> The hard limit on the number of read transactions that may be
|
||||
open at the same time for the same OpenLDAP LMDB database. When
|
||||
this number is too small, the Postfix LMDB client will log
|
||||
MDB_READERS_FULL errors, and will run with reduced performance.
|
||||
</p>
|
||||
|
||||
|
||||
%PARAM message_size_limit 10240000
|
||||
|
||||
<p>
|
||||
|
||||
@ -97,6 +97,8 @@
|
||||
/* int var_db_create_buf;
|
||||
/* int var_db_read_buf;
|
||||
/* long var_lmdb_map_size;
|
||||
/* int var_proc_limit;
|
||||
/* int var_lmdb_max_readers;
|
||||
/* int var_mime_maxdepth;
|
||||
/* int var_mime_bound_len;
|
||||
/* int var_header_limit;
|
||||
@ -289,6 +291,8 @@ char *var_proxywrite_service;
|
||||
int var_db_create_buf;
|
||||
int var_db_read_buf;
|
||||
long var_lmdb_map_size;
|
||||
int var_lmdb_max_readers;
|
||||
int var_proc_limit;
|
||||
int var_mime_maxdepth;
|
||||
int var_mime_bound_len;
|
||||
int var_header_limit;
|
||||
@ -591,6 +595,7 @@ void mail_params_init()
|
||||
0,
|
||||
};
|
||||
static const CONFIG_INT_TABLE other_int_defaults[] = {
|
||||
VAR_PROC_LIMIT, DEF_PROC_LIMIT, &var_proc_limit, 1, 0,
|
||||
VAR_MAX_USE, DEF_MAX_USE, &var_use_limit, 1, 0,
|
||||
VAR_DONT_REMOVE, DEF_DONT_REMOVE, &var_dont_remove, 0, 0,
|
||||
VAR_LINE_LIMIT, DEF_LINE_LIMIT, &var_line_limit, 512, 0,
|
||||
@ -609,9 +614,13 @@ void mail_params_init()
|
||||
VAR_INET_WINDOW, DEF_INET_WINDOW, &var_inet_windowsize, 0, 0,
|
||||
0,
|
||||
};
|
||||
static const CONFIG_NINT_TABLE nint_defaults[] = {
|
||||
VAR_LMDB_MAX_READERS, DEF_LMDB_MAX_READERS, &var_lmdb_max_readers, 1, 0,
|
||||
0,
|
||||
};
|
||||
static const CONFIG_LONG_TABLE long_defaults[] = {
|
||||
VAR_MESSAGE_LIMIT, DEF_MESSAGE_LIMIT, &var_message_limit, 0, 0,
|
||||
VAR_LMDB_MAP_SIZE, DEF_LMDB_MAP_SIZE, &var_lmdb_map_size, 1, 0,
|
||||
VAR_LMDB_MAP_SIZE, DEF_LMDB_MAP_SIZE, &var_lmdb_map_size, 8192, 0,
|
||||
0,
|
||||
};
|
||||
static const CONFIG_TIME_TABLE time_defaults[] = {
|
||||
@ -709,6 +718,7 @@ void mail_params_init()
|
||||
}
|
||||
#endif
|
||||
get_mail_conf_int_table(other_int_defaults);
|
||||
get_mail_conf_nint_table(nint_defaults);
|
||||
get_mail_conf_long_table(long_defaults);
|
||||
get_mail_conf_bool_table(bool_defaults);
|
||||
get_mail_conf_time_table(time_defaults);
|
||||
@ -721,6 +731,7 @@ void mail_params_init()
|
||||
#endif
|
||||
#ifdef HAS_LMDB
|
||||
dict_lmdb_map_size = var_lmdb_map_size;
|
||||
dict_lmdb_max_readers = var_lmdb_max_readers;
|
||||
#endif
|
||||
inet_windowsize = var_inet_windowsize;
|
||||
|
||||
|
||||
@ -2769,12 +2769,16 @@ extern int var_db_create_buf;
|
||||
extern int var_db_read_buf;
|
||||
|
||||
/*
|
||||
* OpenLDAP LMDB memory map size.
|
||||
* OpenLDAP LMDB settings.
|
||||
*/
|
||||
#define VAR_LMDB_MAP_SIZE "lmdb_map_size"
|
||||
#define DEF_LMDB_MAP_SIZE (16 * 1024 *1024)
|
||||
extern long var_lmdb_map_size;
|
||||
|
||||
#define VAR_LMDB_MAX_READERS "lmdb_max_readers"
|
||||
#define DEF_LMDB_MAX_READERS "$" VAR_PROC_LIMIT
|
||||
extern int var_lmdb_max_readers;
|
||||
|
||||
/*
|
||||
* Named queue file attributes.
|
||||
*/
|
||||
|
||||
@ -20,7 +20,7 @@
|
||||
* Patches change both the patchlevel and the release date. Snapshots have no
|
||||
* patchlevel; they change the release date only.
|
||||
*/
|
||||
#define MAIL_RELEASE_DATE "20130927"
|
||||
#define MAIL_RELEASE_DATE "20130928"
|
||||
#define MAIL_VERSION_NUMBER "2.11"
|
||||
|
||||
#ifdef SNAPSHOT
|
||||
|
||||
@ -61,53 +61,11 @@
|
||||
|
||||
#include "mkmap.h"
|
||||
|
||||
int var_proc_limit;
|
||||
|
||||
/* mkmap_lmdb_open */
|
||||
|
||||
MKMAP *mkmap_lmdb_open(const char *path)
|
||||
{
|
||||
MKMAP *mkmap = (MKMAP *) mymalloc(sizeof(*mkmap));
|
||||
static const CONFIG_INT_TABLE int_table[] = {
|
||||
VAR_PROC_LIMIT, DEF_PROC_LIMIT, &var_proc_limit, 1, 0,
|
||||
0,
|
||||
};
|
||||
|
||||
get_mail_conf_int_table(int_table);
|
||||
|
||||
/*
|
||||
* XXX Why is this not set in mail_params.c (with proper #ifdefs)?
|
||||
*
|
||||
* Override the default per-table map size for map (re)builds.
|
||||
*
|
||||
* lmdb_map_size is defined in util/dict_lmdb.c and defaults to 10MB. It
|
||||
* needs to be large enough to contain the largest tables in use.
|
||||
*
|
||||
* XXX This should be specified via the DICT interface so that the buffer
|
||||
* size becomes an object property, instead of being specified by poking
|
||||
* a global variable so that it becomes a class property.
|
||||
*
|
||||
* XXX Wietse disagrees: storage infrastructure that requires up-front
|
||||
* max-size information is evil. This unlike Postfix (e.g. line length or
|
||||
* process count) limits which are a defense against out-of-control or
|
||||
* malicious external actors.
|
||||
*
|
||||
* XXX Need to check that an existing table can be rebuilt with a larger
|
||||
* size limit than was used for the initial build.
|
||||
*/
|
||||
dict_lmdb_map_size = var_lmdb_map_size;
|
||||
|
||||
/*
|
||||
* XXX Why is this not set in mail_params.c (with proper #ifdefs)?
|
||||
*
|
||||
* Set the max number of concurrent readers per table. This is the
|
||||
* maximum number of postfix processes, plus some extra for CLI users.
|
||||
*
|
||||
* XXX Postfix uses asynchronous or blocking I/O with single-threaded
|
||||
* processes so this limit will never be reached, assuming that the limit
|
||||
* is a per-client property, not a shared database property.
|
||||
*/
|
||||
dict_lmdb_max_readers = var_proc_limit * 2 + 16;
|
||||
|
||||
/*
|
||||
* Fill in the generic members.
|
||||
|
||||
@ -47,7 +47,6 @@
|
||||
* Tunable parameters.
|
||||
*/
|
||||
char *var_inet_protocols;
|
||||
int var_proc_limit;
|
||||
int var_throttle_time;
|
||||
char *var_master_disable;
|
||||
|
||||
@ -60,10 +59,6 @@ void master_vars_init(void)
|
||||
VAR_MASTER_DISABLE, DEF_MASTER_DISABLE, &var_master_disable, 0, 0,
|
||||
0,
|
||||
};
|
||||
static const CONFIG_INT_TABLE int_table[] = {
|
||||
VAR_PROC_LIMIT, DEF_PROC_LIMIT, &var_proc_limit, 1, 0,
|
||||
0,
|
||||
};
|
||||
static const CONFIG_TIME_TABLE time_table[] = {
|
||||
VAR_THROTTLE_TIME, DEF_THROTTLE_TIME, &var_throttle_time, 1, 0,
|
||||
0,
|
||||
@ -87,7 +82,6 @@ void master_vars_init(void)
|
||||
set_mail_conf_str(VAR_PROCNAME, var_procname);
|
||||
mail_conf_read();
|
||||
get_mail_conf_str_table(str_table);
|
||||
get_mail_conf_int_table(int_table);
|
||||
get_mail_conf_time_table(time_table);
|
||||
path = concatenate(var_config_dir, "/", MASTER_CONF_FILE, (char *) 0);
|
||||
fset_master_ent(path);
|
||||
|
||||
@ -232,8 +232,9 @@
|
||||
/* .IP "\fBmaximal_backoff_time (4000s)\fR"
|
||||
/* The maximal time between attempts to deliver a deferred message.
|
||||
/* .IP "\fBmaximal_queue_lifetime (5d)\fR"
|
||||
/* The maximal time a message is queued before it is sent back as
|
||||
/* undeliverable.
|
||||
/* Consider a message as undeliverable, when delivery fails with a
|
||||
/* temporary error, and the time in the queue has reached the
|
||||
/* maximal_queue_lifetime limit.
|
||||
/* .IP "\fBqueue_run_delay (300s)\fR"
|
||||
/* The time between deferred queue scans by the queue manager;
|
||||
/* prior to Postfix 2.4 the default value was 1000s.
|
||||
@ -243,8 +244,9 @@
|
||||
/* .PP
|
||||
/* Available in Postfix version 2.1 and later:
|
||||
/* .IP "\fBbounce_queue_lifetime (5d)\fR"
|
||||
/* The maximal time a bounce message is queued before it is considered
|
||||
/* undeliverable.
|
||||
/* Consider a bounce message as undeliverable, when delivery fails
|
||||
/* with a temporary error, and the time in the queue has reached the
|
||||
/* bounce_queue_lifetime limit.
|
||||
/* .PP
|
||||
/* Available in Postfix version 2.5 and later:
|
||||
/* .IP "\fBdefault_destination_rate_delay (0s)\fR"
|
||||
@ -373,7 +375,6 @@ char *var_defer_xports;
|
||||
int var_qmgr_fudge;
|
||||
int var_local_rcpt_lim; /* XXX */
|
||||
int var_local_con_lim; /* XXX */
|
||||
int var_proc_limit;
|
||||
bool var_verp_bounce_off;
|
||||
int var_qmgr_clog_warn_time;
|
||||
char *var_conc_pos_feedback;
|
||||
@ -641,7 +642,6 @@ int main(int argc, char **argv)
|
||||
VAR_QMGR_FUDGE, DEF_QMGR_FUDGE, &var_qmgr_fudge, 10, 100,
|
||||
VAR_LOCAL_RCPT_LIMIT, DEF_LOCAL_RCPT_LIMIT, &var_local_rcpt_lim, 0, 0,
|
||||
VAR_LOCAL_CON_LIMIT, DEF_LOCAL_CON_LIMIT, &var_local_con_lim, 0, 0,
|
||||
VAR_PROC_LIMIT, DEF_PROC_LIMIT, &var_proc_limit, 1, 0,
|
||||
VAR_CONC_COHORT_LIM, DEF_CONC_COHORT_LIM, &var_conc_cohort_limit, 0, 0,
|
||||
0,
|
||||
};
|
||||
|
||||
@ -424,7 +424,6 @@
|
||||
/*
|
||||
* Configuration parameters.
|
||||
*/
|
||||
int var_proc_limit;
|
||||
char *var_smtpd_service;
|
||||
char *var_smtpd_banner;
|
||||
bool var_disable_vrfy_cmd;
|
||||
@ -1100,7 +1099,6 @@ int main(int argc, char **argv)
|
||||
0,
|
||||
};
|
||||
static const CONFIG_INT_TABLE int_table[] = {
|
||||
VAR_PROC_LIMIT, DEF_PROC_LIMIT, &var_proc_limit, 1, 0,
|
||||
VAR_PSC_DNSBL_THRESH, DEF_PSC_DNSBL_THRESH, &var_psc_dnsbl_thresh, 0, 0,
|
||||
VAR_PSC_DNSBL_WTHRESH, DEF_PSC_DNSBL_WTHRESH, &var_psc_dnsbl_wthresh, 0, 0,
|
||||
VAR_PSC_CMD_COUNT, DEF_PSC_CMD_COUNT, &var_psc_cmd_count, 1, 0,
|
||||
|
||||
@ -278,8 +278,9 @@
|
||||
/* .IP "\fBmaximal_backoff_time (4000s)\fR"
|
||||
/* The maximal time between attempts to deliver a deferred message.
|
||||
/* .IP "\fBmaximal_queue_lifetime (5d)\fR"
|
||||
/* The maximal time a message is queued before it is sent back as
|
||||
/* undeliverable.
|
||||
/* Consider a message as undeliverable, when delivery fails with a
|
||||
/* temporary error, and the time in the queue has reached the
|
||||
/* maximal_queue_lifetime limit.
|
||||
/* .IP "\fBqueue_run_delay (300s)\fR"
|
||||
/* The time between deferred queue scans by the queue manager;
|
||||
/* prior to Postfix 2.4 the default value was 1000s.
|
||||
@ -289,8 +290,9 @@
|
||||
/* .PP
|
||||
/* Available in Postfix version 2.1 and later:
|
||||
/* .IP "\fBbounce_queue_lifetime (5d)\fR"
|
||||
/* The maximal time a bounce message is queued before it is considered
|
||||
/* undeliverable.
|
||||
/* Consider a bounce message as undeliverable, when delivery fails
|
||||
/* with a temporary error, and the time in the queue has reached the
|
||||
/* bounce_queue_lifetime limit.
|
||||
/* .PP
|
||||
/* Available in Postfix version 2.5 and later:
|
||||
/* .IP "\fBdefault_destination_rate_delay (0s)\fR"
|
||||
@ -433,7 +435,6 @@ int var_dest_rcpt_limit;
|
||||
char *var_defer_xports;
|
||||
int var_local_con_lim;
|
||||
int var_local_rcpt_lim;
|
||||
int var_proc_limit;
|
||||
bool var_verp_bounce_off;
|
||||
int var_qmgr_clog_warn_time;
|
||||
char *var_conc_pos_feedback;
|
||||
@ -716,7 +717,6 @@ int main(int argc, char **argv)
|
||||
VAR_DEST_RCPT_LIMIT, DEF_DEST_RCPT_LIMIT, &var_dest_rcpt_limit, 0, 0,
|
||||
VAR_LOCAL_RCPT_LIMIT, DEF_LOCAL_RCPT_LIMIT, &var_local_rcpt_lim, 0, 0,
|
||||
VAR_LOCAL_CON_LIMIT, DEF_LOCAL_CON_LIMIT, &var_local_con_lim, 0, 0,
|
||||
VAR_PROC_LIMIT, DEF_PROC_LIMIT, &var_proc_limit, 1, 0,
|
||||
VAR_CONC_COHORT_LIM, DEF_CONC_COHORT_LIM, &var_conc_cohort_limit, 0, 0,
|
||||
0,
|
||||
};
|
||||
|
||||
@ -7,6 +7,7 @@
|
||||
/* #include <dict_lmdb.h>
|
||||
/*
|
||||
/* size_t dict_lmdb_map_size;
|
||||
/* unsigned int dict_lmdb_max_readers;
|
||||
/*
|
||||
/* DICT *dict_lmdb_open(path, open_flags, dict_flags)
|
||||
/* const char *name;
|
||||
@ -21,6 +22,11 @@
|
||||
/* The dict_lmdb_map_size variable specifies the initial
|
||||
/* database memory map size. When a map becomes full its size
|
||||
/* is doubled, and other programs pick up the size change.
|
||||
/*
|
||||
/* The dict_lmdb_max_readers variable specifies the hard (ugh)
|
||||
/* limit on the number of read transactions that may be open
|
||||
/* at the same time. This should be propertional to the number
|
||||
/* of processes that read the table.
|
||||
/* DIAGNOSTICS
|
||||
/* Fatal errors: cannot open file, file write error, out of
|
||||
/* memory.
|
||||
@ -97,8 +103,9 @@ typedef struct {
|
||||
/* The following facilitate LMDB quirk workarounds. */
|
||||
int dict_api_retries; /* workarounds per dict(3) call */
|
||||
int bulk_mode_retries; /* workarounds per bulk transaction */
|
||||
int open_flags; /* dict(3) open flags */
|
||||
int env_flags; /* LMDB open flags */
|
||||
int dict_open_flags; /* dict(3) open flags */
|
||||
int mdb_open_flags; /* LMDB open flags */
|
||||
int readers_full; /* MDB_READERS_FULL errors */
|
||||
} DICT_LMDB;
|
||||
|
||||
/*
|
||||
@ -130,7 +137,7 @@ typedef struct {
|
||||
#define DICT_LMDB_SIZE_INCR 2 /* Increase size by 1 bit on retry */
|
||||
#define DICT_LMDB_SIZE_MAX SSIZE_T_MAX
|
||||
|
||||
#define DICT_LMDB_API_RETRY_LIMIT 2 /* Retries per dict(3) API call */
|
||||
#define DICT_LMDB_API_RETRY_LIMIT 100 /* Retries per dict(3) API call */
|
||||
#define DICT_LMDB_BULK_RETRY_LIMIT \
|
||||
(2 * sizeof(size_t) * CHAR_BIT) /* Retries per bulk-mode transaction */
|
||||
|
||||
@ -146,8 +153,9 @@ unsigned int dict_lmdb_max_readers = 216; /* 200 postfix processes,
|
||||
|
||||
/*
|
||||
* The purpose of the error-recovering functions below is to hide LMDB
|
||||
* quirks (MAP_FULL, MAP_CHANGED), so that the dict(3) API routines can
|
||||
* pretend that those quirks don't exist, and focus on their own job.
|
||||
* quirks (MAP_FULL, MAP_CHANGED, READERS_FULL), so that the dict(3) API
|
||||
* routines can pretend that those quirks don't exist, and focus on their
|
||||
* own job.
|
||||
*
|
||||
* - To recover from a single-transaction LMDB error, each wrapper function
|
||||
* uses tail recursion instead of goto. Since LMDB errors are rare, code
|
||||
@ -175,7 +183,7 @@ static void dict_lmdb_prepare(DICT_LMDB *dict_lmdb)
|
||||
* - With DICT_FLAG_BULK_UPDATE we commit a bulk-mode transaction when the
|
||||
* database is closed.
|
||||
*/
|
||||
if (dict_lmdb->open_flags & O_TRUNC) {
|
||||
if (dict_lmdb->dict_open_flags & O_TRUNC) {
|
||||
if ((status = mdb_drop(dict_lmdb->txn, dict_lmdb->dbi, 0)) != 0)
|
||||
msg_fatal("truncate %s:%s: %s",
|
||||
dict_lmdb->dict.type, dict_lmdb->dict.name,
|
||||
@ -187,7 +195,7 @@ static void dict_lmdb_prepare(DICT_LMDB *dict_lmdb)
|
||||
mdb_strerror(status));
|
||||
dict_lmdb->txn = NULL;
|
||||
}
|
||||
} else if ((dict_lmdb->env_flags & MDB_RDONLY) != 0
|
||||
} else if ((dict_lmdb->mdb_open_flags & MDB_RDONLY) != 0
|
||||
|| (dict_lmdb->dict.flags & DICT_FLAG_BULK_UPDATE) == 0) {
|
||||
mdb_txn_abort(dict_lmdb->txn);
|
||||
dict_lmdb->txn = NULL;
|
||||
@ -277,6 +285,20 @@ static int dict_lmdb_recover(DICT_LMDB *dict_lmdb, int status)
|
||||
(unsigned long) dict_lmdb->map_size);
|
||||
break;
|
||||
|
||||
/*
|
||||
* What is it with these built-in hard limits that cause systems to
|
||||
* fail when resources are needed most? When the system is under
|
||||
* stress it should slow down, not stop working.
|
||||
*/
|
||||
case MDB_READERS_FULL:
|
||||
if (dict_lmdb->readers_full++ == 0)
|
||||
msg_warn("database %s:%s: %s - increase lmdb_max_readers",
|
||||
dict_lmdb->dict.type, dict_lmdb->dict.name,
|
||||
mdb_strerror(status));
|
||||
rand_sleep(1000000, 1000000);
|
||||
status = 0;
|
||||
break;
|
||||
|
||||
/*
|
||||
* We can't solve this problem. The application should terminate with
|
||||
* a fatal run-time error and the program should be re-run later.
|
||||
@ -293,7 +315,7 @@ static int dict_lmdb_recover(DICT_LMDB *dict_lmdb, int status)
|
||||
if (dict_lmdb->txn != 0 && status == 0
|
||||
&& (dict_lmdb->bulk_mode_retries += 1) <= DICT_LMDB_BULK_RETRY_LIMIT) {
|
||||
status = mdb_txn_begin(dict_lmdb->env, NULL,
|
||||
dict_lmdb->env_flags & MDB_RDONLY,
|
||||
dict_lmdb->mdb_open_flags & MDB_RDONLY,
|
||||
&dict_lmdb->txn);
|
||||
if (status != 0)
|
||||
msg_fatal("txn_begin %s:%s: %s",
|
||||
@ -466,9 +488,21 @@ static int dict_lmdb_cursor_get(DICT_LMDB *dict_lmdb, MDB_val *mdb_key,
|
||||
* Database lookup.
|
||||
*/
|
||||
status = mdb_cursor_get(dict_lmdb->cursor, mdb_key, mdb_value, op);
|
||||
if (status != 0 && status != MDB_NOTFOUND)
|
||||
|
||||
/*
|
||||
* Handle end-of-database or other error.
|
||||
*/
|
||||
if (status != 0) {
|
||||
if (status == MDB_NOTFOUND) {
|
||||
txn = mdb_cursor_txn(dict_lmdb->cursor);
|
||||
mdb_cursor_close(dict_lmdb->cursor);
|
||||
mdb_txn_abort(txn);
|
||||
dict_lmdb->cursor = 0;
|
||||
} else {
|
||||
if ((status = dict_lmdb_recover(dict_lmdb, status)) == 0)
|
||||
return (dict_lmdb_cursor_get(dict_lmdb, mdb_key, mdb_value, op));
|
||||
}
|
||||
}
|
||||
return (status);
|
||||
}
|
||||
|
||||
@ -735,7 +769,6 @@ static int dict_lmdb_sequence(DICT *dict, int function,
|
||||
DICT_LMDB *dict_lmdb = (DICT_LMDB *) dict;
|
||||
MDB_val mdb_key;
|
||||
MDB_val mdb_value;
|
||||
MDB_txn *txn;
|
||||
MDB_cursor_op op;
|
||||
int status;
|
||||
|
||||
@ -774,14 +807,10 @@ static int dict_lmdb_sequence(DICT *dict, int function,
|
||||
break;
|
||||
|
||||
/*
|
||||
* Destroy cursor and read transaction.
|
||||
* End-of-database.
|
||||
*/
|
||||
case MDB_NOTFOUND:
|
||||
status = 1;
|
||||
txn = mdb_cursor_txn(dict_lmdb->cursor);
|
||||
mdb_cursor_close(dict_lmdb->cursor);
|
||||
mdb_txn_abort(txn);
|
||||
dict_lmdb->cursor = 0;
|
||||
break;
|
||||
|
||||
/*
|
||||
@ -845,8 +874,16 @@ DICT *dict_lmdb_open(const char *path, int open_flags, int dict_flags)
|
||||
if ((status = mdb_env_create(&env)))
|
||||
msg_fatal("env_create %s: %s", mdb_path, mdb_strerror(status));
|
||||
|
||||
if (stat(mdb_path, &st) == 0 && st.st_size > map_size)
|
||||
if (stat(mdb_path, &st) == 0 && st.st_size > map_size) {
|
||||
if (st.st_size / map_size < DICT_LMDB_SIZE_MAX / map_size) {
|
||||
map_size = (st.st_size / map_size + 1) * map_size;
|
||||
} else {
|
||||
map_size = st.st_size;
|
||||
}
|
||||
if (msg_verbose)
|
||||
msg_info("using %s:%s map size %lu",
|
||||
DICT_TYPE_LMDB, path, (unsigned long) map_size);
|
||||
}
|
||||
if ((status = mdb_env_set_mapsize(env, map_size)))
|
||||
msg_fatal("env_set_mapsize %s: %s", mdb_path, mdb_strerror(status));
|
||||
|
||||
@ -872,6 +909,9 @@ DICT *dict_lmdb_open(const char *path, int open_flags, int dict_flags)
|
||||
if ((status = mdb_open(txn, NULL, 0, &dbi)))
|
||||
msg_fatal("mdb_open %s: %s", mdb_path, mdb_strerror(status));
|
||||
|
||||
/*
|
||||
* Bundle up.
|
||||
*/
|
||||
dict_lmdb = (DICT_LMDB *) dict_alloc(DICT_TYPE_LMDB, path, sizeof(*dict_lmdb));
|
||||
dict_lmdb->dict.lookup = dict_lmdb_lookup;
|
||||
dict_lmdb->dict.update = dict_lmdb_update;
|
||||
@ -914,9 +954,10 @@ DICT *dict_lmdb_open(const char *path, int open_flags, int dict_flags)
|
||||
/* The following facilitate transparent error recovery. */
|
||||
dict_lmdb->dict_api_retries = 0;
|
||||
dict_lmdb->bulk_mode_retries = 0;
|
||||
dict_lmdb->open_flags = open_flags;
|
||||
dict_lmdb->env_flags = env_flags;
|
||||
dict_lmdb->dict_open_flags = open_flags;
|
||||
dict_lmdb->mdb_open_flags = env_flags;
|
||||
dict_lmdb->txn = txn;
|
||||
dict_lmdb->readers_full = 0;
|
||||
dict_lmdb_prepare(dict_lmdb);
|
||||
if (dict_flags & DICT_FLAG_BULK_UPDATE)
|
||||
dict_jmp_alloc(&dict_lmdb->dict); /* build into dict_alloc() */
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user