mir/postfix
2
0
Fork 0
mirror of https://github.com/vdukhovni/postfix synced 2025-09-03 15:45:24 +00:00
Code Issues Releases Wiki Activity

postfix-2.5-20071121

Browse Source
This commit is contained in:
Wietse Venema
2007-11-21 00:00:00 -05:00
committed by Viktor Dukhovni
parent 8130775303
commit 261e3f8965
15 changed files with 757 additions and 33 deletions
Download Patch File Download Diff File Expand all files Collapse all files

26
postfix/HISTORY
View File

@@ -13813,3 +13813,29 @@ Apologies for any names omitted.
Bugfix (introduced: 20071004) missing exception handling Bugfix (introduced: 20071004) missing exception handling
in smtp-sink per-command delay feature. Victor Duchovni. in smtp-sink per-command delay feature. Victor Duchovni.
File: smtpstone/smtp-sink.c. File: smtpstone/smtp-sink.c.
2007117-20
Revised queue manager with separate mechanisms for
per-destination concurrency control and dead destination
detection. The concurrency control supports non-integer
feedback for more gradual concurrency adjustments, and uses
hysteresis to avoid rapid oscillations. A destination is
declared "dead" after a configurable number of pseudo-cohorts
(number of deliveries equal to a destination's concurrency)
reports connection or handshake failure. This work began
with a discussion that Wietse started with Patrik Rak and
Victor Duchovni late January 2004, and that Victor revived
late October 2007. To establish a baseline for further
improvement, Wietse implemented a few simple mechanisms.
Configuration parameters: qmgr_concurrency_feedback_debug,
qmgr_negative_concurrency_feedback_hysteresis,
qmgr_negative_concurrency_feedback_style,
qmgr_positive_concurrency_feedback_hysteresis,
qmgr_positive_concurrency_feedback_style, qmgr_sacrifice_cohorts.
See postconf(5) for detailed information. Right now, the
defaults are compatible with older Postfix versions. After
further review the number of parameters will be consolidated
and the defaults will select the better algorithms. Files:
qmgr/qmgr_queue.c, qmgr/qmgr_deliver.c.

41
postfix/RELEASE_NOTES
View File

@@ -17,7 +17,46 @@ Incompatibility with Postfix 2.3 and earlier
If you upgrade from Postfix 2.3 or earlier, read RELEASE_NOTES-2.4 If you upgrade from Postfix 2.3 or earlier, read RELEASE_NOTES-2.4
before proceeding. before proceeding.
Major changes with Postfix snapshot 20071110 Major changes with Postfix snapshot 20071121
============================================
Revised queue manager with separate mechanisms for per-destination
concurrency control and for dead destination detection. The
concurrency control supports non-integer feedback to allow for more
gradual concurrency adjustments, and uses hysteresis to avoid rapid
oscillations. A destination is declared "dead" after a configurable
number of pseudo-cohorts(*) reports connection or handshake failure.
(*) A pseudo-cohort is a number of delivery requests equal to a
destination's delivery concurrency.
The drawbacks of the old +/-1 feedback scheduler are a) overshoot
due to exponential delivery concurrency growth with each pseudo-cohort(*)
(5-10-20...); b) throttling down to zero concurrency after a single
pseudo-cohort(*) failure. The second problem was especially an issue
with low-concurrency channels where a single failure could be
sufficient to mark a destination as "dead", and suspend further
deliveries.
The new code is a laboratory model with a multitude of configuration
parameters, so that developers can experiment with different feedback
functions and hysteresis values. This is a baseline against which
further improvements will be measured: a) is the additional improvement
worth the additional complexity; b) is the design sound, i.e. free
from arbitrary constants and other tweaks that optimize for a narrow
range of application.
New main.cf parameters: qmgr_concurrency_feedback_debug,
qmgr_negative_feedback_hysteresis, qmgr_negative_feedback_method,
qmgr_positive_feedback_hysteresis, qmgr_positive_feedback_method,
qmgr_sacrifice_cohorts. See postconf(5) for extensive descriptions.
The default parameter settings are backwards compatible with older
Postfix versions. However, after a testing period, the number of
parameters will be consolidated, and the default settings will be
changed to take advantage of the "better" algorithm.
Major changes with Postfix snapshot 20071111
============================================ ============================================
Header/body checks are now available in the SMTP client, after the Header/body checks are now available in the SMTP client, after the

142
postfix/html/postconf.5.html
View File

@@ -5915,6 +5915,18 @@ This feature is available in Postfix 2.0 and later.
</p> </p>
</DD>
<DT><b><a name="qmgr_concurrency_feedback_debug">qmgr_concurrency_feedback_debug</a>
(default: no)</b></DT><DD>
<p> Make the queue manager's feedback algorithm verbose for performance
analysis purposes. </p>
<p> This feature is temporarily available in Postfix 2.5; its final
form is likely to change. </p>
</DD> </DD>
<DT><b><a name="qmgr_fudge_factor">qmgr_fudge_factor</a> <DT><b><a name="qmgr_fudge_factor">qmgr_fudge_factor</a>
@@ -5966,6 +5978,136 @@ parameter is 1.
</p> </p>
</DD>
<DT><b><a name="qmgr_negative_concurrency_feedback_hysteresis">qmgr_negative_concurrency_feedback_hysteresis</a>
(default: 1)</b></DT><DD>
<p> The per-destination integer amount of negative concurrency
feedback that must accumulate between negative adjustments of a
destination's delivery concurrency. The concurrency adjustment is
equal in size to the negative hysteresis value, and is applied at
the <b>beginning</b> of a cycle of (hysteresis / feedback) steps.
At that same time, the destination's positive feedback hysteresis
cycle is reset to its beginning. </p>
<p> This feature is temporarily available in Postfix 2.5; its final
form is likely to change. The default setting is compatible with
earlier Postfix versions. </p>
</DD>
<DT><b><a name="qmgr_negative_concurrency_feedback_style">qmgr_negative_concurrency_feedback_style</a>
(default: fixed_1)</b></DT><DD>
<p> The per-destination amount of negative delivery concurrency
feedback, after a delivery completes with a connection or handshake
failure. </p>
<dl>
<dt> <b> inverse_concurrency </b> </dt> <dd> Variable feedback of
1 / (delivery concurrency). With this setting, and with
"<a href="postconf.5.html#qmgr_negative_concurrency_feedback_hysteresis">qmgr_negative_concurrency_feedback_hysteresis</a> = 1", the destination's
delivery concurrency is decremented by 1 after each failed
pseudo-cohort, and the destination is marked dead (further delivery
suspended) after the failed pseudo-cohort count reaches
$<a href="postconf.5.html#qmgr_sacrificial_cohorts">qmgr_sacrificial_cohorts</a>. </dd>
<dt> <b> inverse_sqrt_concurrency </b> </dt> <dd> Variable feedback
of 1 / (square root of delivery concurrency). This is an intermediate
form between the other two. It lacks sound justification, and is a
candidate for removal. </dd>
<dt> <b> fixed_1 </b> </dt> <dd> Constant feedback of 1. This setting
is compatible with Postfix versions before 2.5, where a destination's
delivery concurrency is throttled down to zero (and further delivery
suspended) after a single failed pseudo-cohort. </dd>
</dl>
<p> A pseudo-cohort is a number of deliveries equal to the destination's
delivery concurrency. </p>
<p> This feature is temporarily available in Postfix 2.5; its final
form is likely to change. The default setting is compatible with
earlier Postfix versions. </p>
</DD>
<DT><b><a name="qmgr_positive_concurrency_feedback_hysteresis">qmgr_positive_concurrency_feedback_hysteresis</a>
(default: 1)</b></DT><DD>
<p> The per-destination integer amount of positive concurrency
feedback that must accumulate before positive adjustments of a
destination's delivery concurrency. The concurrency adjustment is
equal in size to the positive hysteresis value, and is applied at
the <b>end</b> of a cycle of (hysteresis / feedback) steps. At that
same time, the destination's negative feedback hysteresis cycle is
reset to its beginning. </p>
<p> This feature is temporarily available in Postfix 2.5; its final
form is likely to change. The default setting is compatible with
earlier Postfix versions. </p>
</DD>
<DT><b><a name="qmgr_positive_concurrency_feedback_style">qmgr_positive_concurrency_feedback_style</a>
(default: fixed_1)</b></DT><DD>
<p> The per-destination amount of positive delivery concurrency
feedback, after a delivery completes without connection or handshake
failure. </p>
<dl>
<dt> <b> inverse_concurrency </b> </dt> <dd> Variable feedback of
1 / (delivery concurrency). With this setting, and with
"<a href="postconf.5.html#qmgr_positive_concurrency_feedback_hysteresis">qmgr_positive_concurrency_feedback_hysteresis</a> = 1", the destination's
delivery concurrency is incremented by 1 after each successful
pseudo-cohort, until it reaches the per-destination maximal concurrency
limit. </dd>
<dt> <b> inverse_sqrt_concurrency </b> </dt> <dd> Variable feedback
of 1 / (square root of delivery concurrency). This is an intermediate
form between the other two. It lacks sound justification, and is a
candidate for removal. </dd>
<dt> <b> fixed_1 </b> </dt> <dd> Constant feedback of 1. This setting
is compatible with Postfix versions before 2.5, where a destination's
delivery concurrency is doubled after each successful pseudo-cohort,
until it reaches the per-destination maximal concurrency limit.
</dd>
</dl>
<p> A pseudo-cohort is a number of deliveries equal to the destination's
delivery concurrency. </p>
<p> This feature is temporarily available in Postfix 2.5. The default
setting is compatible with earlier Postfix versions. </p>
</DD>
<DT><b><a name="qmgr_sacrificial_cohorts">qmgr_sacrificial_cohorts</a>
(default: 1)</b></DT><DD>
<p> How many pseudo-cohorts must suffer connection or handshake
failure before a specific destination is considered unavailable
(and further delivery is suspended). A pseudo-cohort is a number
of deliveries equal to a destination's concurrency. The pseudo-cohort
failure count is reset each time a delivery completes without
connection or handshake failure for that specific destination. </p>
<p> This feature is temporarily available in Postfix 2.5; its final
form is likely to change. The default setting is compatible with
earlier Postfix versions. </p>
</DD> </DD>
<DT><b><a name="qmqpd_authorized_clients">qmqpd_authorized_clients</a> <DT><b><a name="qmqpd_authorized_clients">qmqpd_authorized_clients</a>

48
postfix/html/qmgr.8.html
View File

@@ -262,6 +262,38 @@ QMGR(8) QMGR(8)
<b><a href="postconf.5.html#default_destination_concurrency_limit">tion_concurrency_limit</a>)</b> <b><a href="postconf.5.html#default_destination_concurrency_limit">tion_concurrency_limit</a>)</b>
Idem, for delivery via the named message <i>transport</i>. Idem, for delivery via the named message <i>transport</i>.
<b><a href="postconf.5.html#qmgr_concurrency_feedback_debug">qmgr_concurrency_feedback_debug</a> (no)</b>
Make the queue manager's feedback algorithm verbose
for performance analysis purposes.
<b><a href="postconf.5.html#qmgr_negative_concurrency_feedback_hysteresis">qmgr_negative_concurrency_feedback_hysteresis</a> (1)</b>
The per-destination integer amount of negative con-
currency feedback that must accumulate between neg-
ative adjustments of a destination's delivery con-
currency.
<b><a href="postconf.5.html#qmgr_negative_concurrency_feedback_style">qmgr_negative_concurrency_feedback_style</a> (fixed_1)</b>
The per-destination amount of negative delivery
concurrency feedback, after a delivery completes
with a connection or handshake failure.
<b><a href="postconf.5.html#qmgr_positive_concurrency_feedback_hysteresis">qmgr_positive_concurrency_feedback_hysteresis</a> (1)</b>
The per-destination integer amount of positive con-
currency feedback that must accumulate before posi-
tive adjustments of a destination's delivery con-
currency.
<b><a href="postconf.5.html#qmgr_positive_concurrency_feedback_style">qmgr_positive_concurrency_feedback_style</a> (fixed_1)</b>
The per-destination amount of positive delivery
concurrency feedback, after a delivery completes
without connection or handshake failure.
<b><a href="postconf.5.html#qmgr_sacrificial_cohorts">qmgr_sacrificial_cohorts</a> (1)</b>
How many pseudo-cohorts must suffer connection or
handshake failure before a specific destination is
considered unavailable (and further delivery is
suspended).
<b>RECIPIENT SCHEDULING CONTROLS</b> <b>RECIPIENT SCHEDULING CONTROLS</b>
<b><a href="postconf.5.html#default_destination_recipient_limit">default_destination_recipient_limit</a> (50)</b> <b><a href="postconf.5.html#default_destination_recipient_limit">default_destination_recipient_limit</a> (50)</b>
The default maximal number of recipients per mes- The default maximal number of recipients per mes-
@@ -305,21 +337,23 @@ QMGR(8) QMGR(8)
Idem, for delivery via the named message <i>transport</i>. Idem, for delivery via the named message <i>transport</i>.
<b>OTHER RESOURCE AND RATE CONTROLS</b> <b>OTHER RESOURCE AND RATE CONTROLS</b>
<b><a href="postconf.5.html#minimal_backoff_time">minimal_backoff_time</a> (version dependent)</b> <b><a href="postconf.5.html#minimal_backoff_time">minimal_backoff_time</a> (300s)</b>
The minimal time between attempts to deliver a The minimal time between attempts to deliver a
deferred message. deferred message; prior to Postfix 2.4 the default
value was 1000s.
<b><a href="postconf.5.html#maximal_backoff_time">maximal_backoff_time</a> (4000s)</b> <b><a href="postconf.5.html#maximal_backoff_time">maximal_backoff_time</a> (4000s)</b>
The maximal time between attempts to deliver a The maximal time between attempts to deliver a
deferred message. deferred message.
<b><a href="postconf.5.html#maximal_queue_lifetime">maximal_queue_lifetime</a> (5d)</b> <b><a href="postconf.5.html#maximal_queue_lifetime">maximal_queue_lifetime</a> (5d)</b>
The maximal time a message is queued before it is The maximal time a message is queued before it is
sent back as undeliverable. sent back as undeliverable.
<b><a href="postconf.5.html#queue_run_delay">queue_run_delay</a> (version dependent)</b> <b><a href="postconf.5.html#queue_run_delay">queue_run_delay</a> (300s)</b>
The time between <a href="QSHAPE_README.html#deferred_queue">deferred queue</a> scans by the queue The time between <a href="QSHAPE_README.html#deferred_queue">deferred queue</a> scans by the queue
manager. manager; prior to Postfix 2.4 the default value was
1000s.
<b><a href="postconf.5.html#transport_retry_time">transport_retry_time</a> (60s)</b> <b><a href="postconf.5.html#transport_retry_time">transport_retry_time</a> (60s)</b>
The time between attempts by the Postfix queue man- The time between attempts by the Postfix queue man-

97
postfix/man/man5/postconf.5
View File

@@ -3244,6 +3244,12 @@ clogging up the Postfix active queue. Specify 0 to disable.
This feature is enabled with the helpful_warnings parameter. This feature is enabled with the helpful_warnings parameter.
.PP .PP
This feature is available in Postfix 2.0 and later. This feature is available in Postfix 2.0 and later.
.SH qmgr_concurrency_feedback_debug (default: no)
Make the queue manager's feedback algorithm verbose for performance
analysis purposes.
.PP
This feature is temporarily available in Postfix 2.5; its final
form is likely to change.
.SH qmgr_fudge_factor (default: 100) .SH qmgr_fudge_factor (default: 100)
Obsolete feature: the percentage of delivery resources that a busy Obsolete feature: the percentage of delivery resources that a busy
mail system will use up for delivery of a large mailing list mail system will use up for delivery of a large mailing list
@@ -3263,6 +3269,97 @@ takes priority over any other in-memory recipient limits (i.e.,
the global qmgr_message_recipient_limit and the per transport the global qmgr_message_recipient_limit and the per transport
_recipient_limit) if necessary. The minimum value allowed for this _recipient_limit) if necessary. The minimum value allowed for this
parameter is 1. parameter is 1.
.SH qmgr_negative_concurrency_feedback_hysteresis (default: 1)
The per-destination integer amount of negative concurrency
feedback that must accumulate between negative adjustments of a
destination's delivery concurrency. The concurrency adjustment is
equal in size to the negative hysteresis value, and is applied at
the \fBbeginning\fR of a cycle of (hysteresis / feedback) steps.
At that same time, the destination's positive feedback hysteresis
cycle is reset to its beginning.
.PP
This feature is temporarily available in Postfix 2.5; its final
form is likely to change. The default setting is compatible with
earlier Postfix versions.
.SH qmgr_negative_concurrency_feedback_style (default: fixed_1)
The per-destination amount of negative delivery concurrency
feedback, after a delivery completes with a connection or handshake
failure.
.IP "\fB inverse_concurrency \fR"
Variable feedback of
1 / (delivery concurrency). With this setting, and with
"qmgr_negative_concurrency_feedback_hysteresis = 1", the destination's
delivery concurrency is decremented by 1 after each failed
pseudo-cohort, and the destination is marked dead (further delivery
suspended) after the failed pseudo-cohort count reaches
$qmgr_sacrificial_cohorts.
.IP "\fB inverse_sqrt_concurrency \fR"
Variable feedback
of 1 / (square root of delivery concurrency). This is an intermediate
form between the other two. It lacks sound justification, and is a
candidate for removal.
.IP "\fB fixed_1 \fR"
Constant feedback of 1. This setting
is compatible with Postfix versions before 2.5, where a destination's
delivery concurrency is throttled down to zero (and further delivery
suspended) after a single failed pseudo-cohort.
.PP
A pseudo-cohort is a number of deliveries equal to the destination's
delivery concurrency.
.PP
This feature is temporarily available in Postfix 2.5; its final
form is likely to change. The default setting is compatible with
earlier Postfix versions.
.SH qmgr_positive_concurrency_feedback_hysteresis (default: 1)
The per-destination integer amount of positive concurrency
feedback that must accumulate before positive adjustments of a
destination's delivery concurrency. The concurrency adjustment is
equal in size to the positive hysteresis value, and is applied at
the \fBend\fR of a cycle of (hysteresis / feedback) steps. At that
same time, the destination's negative feedback hysteresis cycle is
reset to its beginning.
.PP
This feature is temporarily available in Postfix 2.5; its final
form is likely to change. The default setting is compatible with
earlier Postfix versions.
.SH qmgr_positive_concurrency_feedback_style (default: fixed_1)
The per-destination amount of positive delivery concurrency
feedback, after a delivery completes without connection or handshake
failure.
.IP "\fB inverse_concurrency \fR"
Variable feedback of
1 / (delivery concurrency). With this setting, and with
"qmgr_positive_concurrency_feedback_hysteresis = 1", the destination's
delivery concurrency is incremented by 1 after each successful
pseudo-cohort, until it reaches the per-destination maximal concurrency
limit.
.IP "\fB inverse_sqrt_concurrency \fR"
Variable feedback
of 1 / (square root of delivery concurrency). This is an intermediate
form between the other two. It lacks sound justification, and is a
candidate for removal.
.IP "\fB fixed_1 \fR"
Constant feedback of 1. This setting
is compatible with Postfix versions before 2.5, where a destination's
delivery concurrency is doubled after each successful pseudo-cohort,
until it reaches the per-destination maximal concurrency limit.
.PP
A pseudo-cohort is a number of deliveries equal to the destination's
delivery concurrency.
.PP
This feature is temporarily available in Postfix 2.5. The default
setting is compatible with earlier Postfix versions.
.SH qmgr_sacrificial_cohorts (default: 1)
How many pseudo-cohorts must suffer connection or handshake
failure before a specific destination is considered unavailable
(and further delivery is suspended). A pseudo-cohort is a number
of deliveries equal to a destination's concurrency. The pseudo-cohort
failure count is reset each time a delivery completes without
connection or handshake failure for that specific destination.
.PP
This feature is temporarily available in Postfix 2.5; its final
form is likely to change. The default setting is compatible with
earlier Postfix versions.
.SH qmqpd_authorized_clients (default: empty) .SH qmqpd_authorized_clients (default: empty)
What clients are allowed to connect to the QMQP server port. What clients are allowed to connect to the QMQP server port.
.PP .PP

33
postfix/man/man8/qmgr.8
View File

@@ -235,6 +235,29 @@ The default maximal number of parallel deliveries to the same
destination. destination.
.IP "\fItransport\fB_destination_concurrency_limit ($default_destination_concurrency_limit)\fR" .IP "\fItransport\fB_destination_concurrency_limit ($default_destination_concurrency_limit)\fR"
Idem, for delivery via the named message \fItransport\fR. Idem, for delivery via the named message \fItransport\fR.
.IP "\fBqmgr_concurrency_feedback_debug (no)\fR"
Make the queue manager's feedback algorithm verbose for performance
analysis purposes.
.IP "\fBqmgr_negative_concurrency_feedback_hysteresis (1)\fR"
The per-destination integer amount of negative concurrency
feedback that must accumulate between negative adjustments of a
destination's delivery concurrency.
.IP "\fBqmgr_negative_concurrency_feedback_style (fixed_1)\fR"
The per-destination amount of negative delivery concurrency
feedback, after a delivery completes with a connection or handshake
failure.
.IP "\fBqmgr_positive_concurrency_feedback_hysteresis (1)\fR"
The per-destination integer amount of positive concurrency
feedback that must accumulate before positive adjustments of a
destination's delivery concurrency.
.IP "\fBqmgr_positive_concurrency_feedback_style (fixed_1)\fR"
The per-destination amount of positive delivery concurrency
feedback, after a delivery completes without connection or handshake
failure.
.IP "\fBqmgr_sacrificial_cohorts (1)\fR"
How many pseudo-cohorts must suffer connection or handshake
failure before a specific destination is considered unavailable
(and further delivery is suspended).
.SH "RECIPIENT SCHEDULING CONTROLS" .SH "RECIPIENT SCHEDULING CONTROLS"
.na .na
.nf .nf
@@ -274,15 +297,17 @@ Idem, for delivery via the named message \fItransport\fR.
.nf .nf
.ad .ad
.fi .fi
.IP "\fBminimal_backoff_time (version dependent)\fR" .IP "\fBminimal_backoff_time (300s)\fR"
The minimal time between attempts to deliver a deferred message. The minimal time between attempts to deliver a deferred message;
prior to Postfix 2.4 the default value was 1000s.
.IP "\fBmaximal_backoff_time (4000s)\fR" .IP "\fBmaximal_backoff_time (4000s)\fR"
The maximal time between attempts to deliver a deferred message. The maximal time between attempts to deliver a deferred message.
.IP "\fBmaximal_queue_lifetime (5d)\fR" .IP "\fBmaximal_queue_lifetime (5d)\fR"
The maximal time a message is queued before it is sent back as The maximal time a message is queued before it is sent back as
undeliverable. undeliverable.
.IP "\fBqueue_run_delay (version dependent)\fR" .IP "\fBqueue_run_delay (300s)\fR"
The time between deferred queue scans by the queue manager. The time between deferred queue scans by the queue manager;
prior to Postfix 2.4 the default value was 1000s.
.IP "\fBtransport_retry_time (60s)\fR" .IP "\fBtransport_retry_time (60s)\fR"
The time between attempts by the Postfix queue manager to contact The time between attempts by the Postfix queue manager to contact
a malfunctioning message delivery transport. a malfunctioning message delivery transport.

8
postfix/mantools/postlink
View File

@@ -334,6 +334,14 @@ while (<>) {
s;\bqmgr_message_recip[-</bB>]*\n* *[<bB>]*ient_limit\b;<a href="postconf.5.html#qmgr_message_recipient_limit">$&</a>;g; s;\bqmgr_message_recip[-</bB>]*\n* *[<bB>]*ient_limit\b;<a href="postconf.5.html#qmgr_message_recipient_limit">$&</a>;g;
s;\bqmgr_message_recip[-</bB>]*\n* *[<bB>]*ient_minimum\b;<a href="postconf.5.html#qmgr_message_recipient_minimum">$&</a>;g; s;\bqmgr_message_recip[-</bB>]*\n* *[<bB>]*ient_minimum\b;<a href="postconf.5.html#qmgr_message_recipient_minimum">$&</a>;g;
s;\bqmqpd_authorized_clients\b;<a href="postconf.5.html#qmqpd_authorized_clients">$&</a>;g; s;\bqmqpd_authorized_clients\b;<a href="postconf.5.html#qmqpd_authorized_clients">$&</a>;g;
s;\bqmgr_negative_concurrency_feedback_hysteresis\b;<a href="postconf.5.html#qmgr_negative_concurrency_feedback_hysteresis">$&</a>;g;
s;\bqmgr_negative_concurrency_feedback_style\b;<a href="postconf.5.html#qmgr_negative_concurrency_feedback_style">$&</a>;g;
s;\bqmgr_positive_concurrency_feedback_hysteresis\b;<a href="postconf.5.html#qmgr_positive_concurrency_feedback_hysteresis">$&</a>;g;
s;\bqmgr_positive_concurrency_feedback_style\b;<a href="postconf.5.html#qmgr_positive_concurrency_feedback_style">$&</a>;g;
s;\bqmgr_sacrificial_cohorts\b;<a href="postconf.5.html#qmgr_sacrificial_cohorts">$&</a>;g;
s;\bqmgr_concurrency_feedback_debug\b;<a href="postconf.5.html#qmgr_concurrency_feedback_debug">$&</a>;g;
s;\bqmqpd_error_delay\b;<a href="postconf.5.html#qmqpd_error_delay">$&</a>;g; s;\bqmqpd_error_delay\b;<a href="postconf.5.html#qmqpd_error_delay">$&</a>;g;
s;\bqmqpd_timeout\b;<a href="postconf.5.html#qmqpd_timeout">$&</a>;g; s;\bqmqpd_timeout\b;<a href="postconf.5.html#qmqpd_timeout">$&</a>;g;
s;\bqueue_directory\b;<a href="postconf.5.html#queue_directory">$&</a>;g; s;\bqueue_directory\b;<a href="postconf.5.html#queue_directory">$&</a>;g;

118
postfix/proto/postconf.proto
View File

@@ -10674,3 +10674,121 @@ that change the delivery time or destination are not available.
</p> </p>
<p> This feature is available in Postfix 2.5 and later. </p> <p> This feature is available in Postfix 2.5 and later. </p>
%PARAM qmgr_concurrency_feedback_debug no
<p> Make the queue manager's feedback algorithm verbose for performance
analysis purposes. </p>
<p> This feature is temporarily available in Postfix 2.5; its final
form is likely to change. </p>
%PARAM qmgr_sacrificial_cohorts 1
<p> How many pseudo-cohorts must suffer connection or handshake
failure before a specific destination is considered unavailable
(and further delivery is suspended). A pseudo-cohort is a number
of deliveries equal to a destination's concurrency. The pseudo-cohort
failure count is reset each time a delivery completes without
connection or handshake failure for that specific destination. </p>
<p> This feature is temporarily available in Postfix 2.5; its final
form is likely to change. The default setting is compatible with
earlier Postfix versions. </p>
%PARAM qmgr_negative_concurrency_feedback_hysteresis 1
<p> The per-destination integer amount of negative concurrency
feedback that must accumulate between negative adjustments of a
destination's delivery concurrency. The concurrency adjustment is
equal in size to the negative hysteresis value, and is applied at
the <b>beginning</b> of a cycle of (hysteresis / feedback) steps.
At that same time, the destination's positive feedback hysteresis
cycle is reset to its beginning. </p>
<p> This feature is temporarily available in Postfix 2.5; its final
form is likely to change. The default setting is compatible with
earlier Postfix versions. </p>
%PARAM qmgr_positive_concurrency_feedback_hysteresis 1
<p> The per-destination integer amount of positive concurrency
feedback that must accumulate before positive adjustments of a
destination's delivery concurrency. The concurrency adjustment is
equal in size to the positive hysteresis value, and is applied at
the <b>end</b> of a cycle of (hysteresis / feedback) steps. At that
same time, the destination's negative feedback hysteresis cycle is
reset to its beginning. </p>
<p> This feature is temporarily available in Postfix 2.5; its final
form is likely to change. The default setting is compatible with
earlier Postfix versions. </p>
%PARAM qmgr_negative_concurrency_feedback_style fixed_1
<p> The per-destination amount of negative delivery concurrency
feedback, after a delivery completes with a connection or handshake
failure. </p>
<dl>
<dt> <b> inverse_concurrency </b> </dt> <dd> Variable feedback of
1 / (delivery concurrency). With this setting, and with
"qmgr_negative_concurrency_feedback_hysteresis = 1", the destination's
delivery concurrency is decremented by 1 after each failed
pseudo-cohort, and the destination is marked dead (further delivery
suspended) after the failed pseudo-cohort count reaches
$qmgr_sacrificial_cohorts. </dd>
<dt> <b> inverse_sqrt_concurrency </b> </dt> <dd> Variable feedback
of 1 / (square root of delivery concurrency). This is an intermediate
form between the other two. It lacks sound justification, and is a
candidate for removal. </dd>
<dt> <b> fixed_1 </b> </dt> <dd> Constant feedback of 1. This setting
is compatible with Postfix versions before 2.5, where a destination's
delivery concurrency is throttled down to zero (and further delivery
suspended) after a single failed pseudo-cohort. </dd>
</dl>
<p> A pseudo-cohort is a number of deliveries equal to the destination's
delivery concurrency. </p>
<p> This feature is temporarily available in Postfix 2.5; its final
form is likely to change. The default setting is compatible with
earlier Postfix versions. </p>
%PARAM qmgr_positive_concurrency_feedback_style fixed_1
<p> The per-destination amount of positive delivery concurrency
feedback, after a delivery completes without connection or handshake
failure. </p>
<dl>
<dt> <b> inverse_concurrency </b> </dt> <dd> Variable feedback of
1 / (delivery concurrency). With this setting, and with
"qmgr_positive_concurrency_feedback_hysteresis = 1", the destination's
delivery concurrency is incremented by 1 after each successful
pseudo-cohort, until it reaches the per-destination maximal concurrency
limit. </dd>
<dt> <b> inverse_sqrt_concurrency </b> </dt> <dd> Variable feedback
of 1 / (square root of delivery concurrency). This is an intermediate
form between the other two. It lacks sound justification, and is a
candidate for removal. </dd>
<dt> <b> fixed_1 </b> </dt> <dd> Constant feedback of 1. This setting
is compatible with Postfix versions before 2.5, where a destination's
delivery concurrency is doubled after each successful pseudo-cohort,
until it reaches the per-destination maximal concurrency limit.
</dd>
</dl>
<p> A pseudo-cohort is a number of deliveries equal to the destination's
delivery concurrency. </p>
<p> This feature is temporarily available in Postfix 2.5. The default
setting is compatible with earlier Postfix versions. </p>

33
postfix/src/global/mail_params.h
View File

@@ -2830,6 +2830,39 @@ extern char *var_smtp_body_chks;
#define VAR_LMTP_BODY_CHKS "lmtp_body_checks" #define VAR_LMTP_BODY_CHKS "lmtp_body_checks"
#define DEF_LMTP_BODY_CHKS "" #define DEF_LMTP_BODY_CHKS ""
/*
* Scheduler concurrency feedback algorithms.
*/
#define VAR_QMGR_POS_FDBACK "qmgr_positive_concurrency_feedback_style"
#define DEF_QMGR_POS_FDBACK QMGR_FDBACK_NAME_FIXED_1
extern char *var_qmgr_pos_feedback;
#define VAR_QMGR_NEG_FDBACK "qmgr_negative_concurrency_feedback_style"
#define DEF_QMGR_NEG_FDBACK QMGR_FDBACK_NAME_FIXED_1
extern char *var_qmgr_neg_feedback;
#define QMGR_FDBACK_NAME_FIXED_1 "fixed_1"
#define QMGR_FDBACK_NAME_INVERSE_1 "inverse_1" /* deprecated */
#define QMGR_FDBACK_NAME_INVERSE_WIN "inverse_concurrency"
#define QMGR_FDBACK_NAME_INV_SQRT "inverse_sqrt" /* deprecated */
#define QMGR_FDBACK_NAME_INV_SQRT_WIN "inverse_sqrt_concurrency"
#define VAR_QMGR_POS_HYST "qmgr_positive_concurrency_feedback_hysteresis"
#define DEF_QMGR_POS_HYST 1
extern int var_qmgr_pos_hysteresis;
#define VAR_QMGR_NEG_HYST "qmgr_negative_concurrency_feedback_hysteresis"
#define DEF_QMGR_NEG_HYST 1
extern int var_qmgr_neg_hysteresis;
#define VAR_QMGR_SAC_COHORTS "qmgr_sacrificial_cohorts"
#define DEF_QMGR_SAC_COHORTS 1
extern int var_qmgr_sac_cohorts;
#define VAR_QMGR_FDBACK_DEBUG "qmgr_concurrency_feedback_debug"
#define DEF_QMGR_FDBACK_DEBUG 0
extern bool var_qmgr_feedback_debug;
/* LICENSE /* LICENSE
/* .ad /* .ad
/* .fi /* .fi

2
postfix/src/global/mail_version.h
View File

@@ -20,7 +20,7 @@
* Patches change both the patchlevel and the release date. Snapshots have no * Patches change both the patchlevel and the release date. Snapshots have no
* patchlevel; they change the release date only. * patchlevel; they change the release date only.
*/ */
#define MAIL_RELEASE_DATE "20071111" #define MAIL_RELEASE_DATE "20071121"
#define MAIL_VERSION_NUMBER "2.5" #define MAIL_VERSION_NUMBER "2.5"
#ifdef SNAPSHOT #ifdef SNAPSHOT

3
postfix/src/qmgr/Makefile.in
View File

@@ -14,7 +14,7 @@ CFLAGS = $(DEBUG) $(OPT) $(DEFS)
TESTPROG= TESTPROG=
PROG = qmgr PROG = qmgr
INC_DIR = ../../include INC_DIR = ../../include
LIBS = ../../lib/libmaster.a ../../lib/libglobal.a ../../lib/libutil.a LIBS = ../../lib/libmaster.a ../../lib/libglobal.a ../../lib/libutil.a -lm
.c.o:; $(CC) $(CFLAGS) -c $*.c .c.o:; $(CC) $(CFLAGS) -c $*.c
@@ -290,6 +290,7 @@ qmgr_queue.o: ../../include/htable.h
qmgr_queue.o: ../../include/mail_params.h qmgr_queue.o: ../../include/mail_params.h
qmgr_queue.o: ../../include/msg.h qmgr_queue.o: ../../include/msg.h
qmgr_queue.o: ../../include/mymalloc.h qmgr_queue.o: ../../include/mymalloc.h
qmgr_queue.o: ../../include/name_code.h
qmgr_queue.o: ../../include/recipient_list.h qmgr_queue.o: ../../include/recipient_list.h
qmgr_queue.o: ../../include/scan_dir.h qmgr_queue.o: ../../include/scan_dir.h
qmgr_queue.o: ../../include/sys_defs.h qmgr_queue.o: ../../include/sys_defs.h

50
postfix/src/qmgr/qmgr.c
View File

@@ -205,6 +205,29 @@
/* destination. /* destination.
/* .IP "\fItransport\fB_destination_concurrency_limit ($default_destination_concurrency_limit)\fR" /* .IP "\fItransport\fB_destination_concurrency_limit ($default_destination_concurrency_limit)\fR"
/* Idem, for delivery via the named message \fItransport\fR. /* Idem, for delivery via the named message \fItransport\fR.
/* .IP "\fBqmgr_concurrency_feedback_debug (no)\fR"
/* Make the queue manager's feedback algorithm verbose for performance
/* analysis purposes.
/* .IP "\fBqmgr_negative_concurrency_feedback_hysteresis (1)\fR"
/* The per-destination integer amount of negative concurrency
/* feedback that must accumulate between negative adjustments of a
/* destination's delivery concurrency.
/* .IP "\fBqmgr_negative_concurrency_feedback_style (fixed_1)\fR"
/* The per-destination amount of negative delivery concurrency
/* feedback, after a delivery completes with a connection or handshake
/* failure.
/* .IP "\fBqmgr_positive_concurrency_feedback_hysteresis (1)\fR"
/* The per-destination integer amount of positive concurrency
/* feedback that must accumulate before positive adjustments of a
/* destination's delivery concurrency.
/* .IP "\fBqmgr_positive_concurrency_feedback_style (fixed_1)\fR"
/* The per-destination amount of positive delivery concurrency
/* feedback, after a delivery completes without connection or handshake
/* failure.
/* .IP "\fBqmgr_sacrificial_cohorts (1)\fR"
/* How many pseudo-cohorts must suffer connection or handshake
/* failure before a specific destination is considered unavailable
/* (and further delivery is suspended).
/* RECIPIENT SCHEDULING CONTROLS /* RECIPIENT SCHEDULING CONTROLS
/* .ad /* .ad
/* .fi /* .fi
@@ -238,15 +261,17 @@
/* OTHER RESOURCE AND RATE CONTROLS /* OTHER RESOURCE AND RATE CONTROLS
/* .ad /* .ad
/* .fi /* .fi
/* .IP "\fBminimal_backoff_time (version dependent)\fR" /* .IP "\fBminimal_backoff_time (300s)\fR"
/* The minimal time between attempts to deliver a deferred message. /* The minimal time between attempts to deliver a deferred message;
/* prior to Postfix 2.4 the default value was 1000s.
/* .IP "\fBmaximal_backoff_time (4000s)\fR" /* .IP "\fBmaximal_backoff_time (4000s)\fR"
/* The maximal time between attempts to deliver a deferred message. /* The maximal time between attempts to deliver a deferred message.
/* .IP "\fBmaximal_queue_lifetime (5d)\fR" /* .IP "\fBmaximal_queue_lifetime (5d)\fR"
/* The maximal time a message is queued before it is sent back as /* The maximal time a message is queued before it is sent back as
/* undeliverable. /* undeliverable.
/* .IP "\fBqueue_run_delay (version dependent)\fR" /* .IP "\fBqueue_run_delay (300s)\fR"
/* The time between deferred queue scans by the queue manager. /* The time between deferred queue scans by the queue manager;
/* prior to Postfix 2.4 the default value was 1000s.
/* .IP "\fBtransport_retry_time (60s)\fR" /* .IP "\fBtransport_retry_time (60s)\fR"
/* The time between attempts by the Postfix queue manager to contact /* The time between attempts by the Postfix queue manager to contact
/* a malfunctioning message delivery transport. /* a malfunctioning message delivery transport.
@@ -390,6 +415,12 @@ int var_local_rcpt_lim;
int var_proc_limit; int var_proc_limit;
bool var_verp_bounce_off; bool var_verp_bounce_off;
int var_qmgr_clog_warn_time; int var_qmgr_clog_warn_time;
char *var_qmgr_pos_feedback;
char *var_qmgr_neg_feedback;
int var_qmgr_pos_hysteresis;
int var_qmgr_neg_hysteresis;
int var_qmgr_sac_cohorts;
int var_qmgr_feedback_debug;
static QMGR_SCAN *qmgr_scans[2]; static QMGR_SCAN *qmgr_scans[2];
@@ -614,6 +645,11 @@ static void qmgr_post_init(char *name, char **unused_argv)
qmgr_scans[QMGR_SCAN_IDX_DEFERRED] = qmgr_scan_create(MAIL_QUEUE_DEFERRED); qmgr_scans[QMGR_SCAN_IDX_DEFERRED] = qmgr_scan_create(MAIL_QUEUE_DEFERRED);
qmgr_scan_request(qmgr_scans[QMGR_SCAN_IDX_INCOMING], QMGR_SCAN_START); qmgr_scan_request(qmgr_scans[QMGR_SCAN_IDX_INCOMING], QMGR_SCAN_START);
qmgr_deferred_run_event(0, (char *) 0); qmgr_deferred_run_event(0, (char *) 0);
/*
* Scheduler initialization.
*/
qmgr_queue_feedback_init();
} }
MAIL_VERSION_STAMP_DECLARE; MAIL_VERSION_STAMP_DECLARE;
@@ -624,6 +660,8 @@ int main(int argc, char **argv)
{ {
static CONFIG_STR_TABLE str_table[] = { static CONFIG_STR_TABLE str_table[] = {
VAR_DEFER_XPORTS, DEF_DEFER_XPORTS, &var_defer_xports, 0, 0, VAR_DEFER_XPORTS, DEF_DEFER_XPORTS, &var_defer_xports, 0, 0,
VAR_QMGR_POS_FDBACK, DEF_QMGR_POS_FDBACK, &var_qmgr_pos_feedback, 1, 0,
VAR_QMGR_NEG_FDBACK, DEF_QMGR_NEG_FDBACK, &var_qmgr_neg_feedback, 1, 0,
0, 0,
}; };
static CONFIG_TIME_TABLE time_table[] = { static CONFIG_TIME_TABLE time_table[] = {
@@ -654,11 +692,15 @@ int main(int argc, char **argv)
VAR_LOCAL_RCPT_LIMIT, DEF_LOCAL_RCPT_LIMIT, &var_local_rcpt_lim, 0, 0, VAR_LOCAL_RCPT_LIMIT, DEF_LOCAL_RCPT_LIMIT, &var_local_rcpt_lim, 0, 0,
VAR_LOCAL_CON_LIMIT, DEF_LOCAL_CON_LIMIT, &var_local_con_lim, 0, 0, VAR_LOCAL_CON_LIMIT, DEF_LOCAL_CON_LIMIT, &var_local_con_lim, 0, 0,
VAR_PROC_LIMIT, DEF_PROC_LIMIT, &var_proc_limit, 1, 0, VAR_PROC_LIMIT, DEF_PROC_LIMIT, &var_proc_limit, 1, 0,
VAR_QMGR_POS_HYST, DEF_QMGR_POS_HYST, &var_qmgr_pos_hysteresis, 1, 0,
VAR_QMGR_NEG_HYST, DEF_QMGR_NEG_HYST, &var_qmgr_neg_hysteresis, 1, 0,
VAR_QMGR_SAC_COHORTS, DEF_QMGR_SAC_COHORTS, &var_qmgr_sac_cohorts, 1, 0,
0, 0,
}; };
static CONFIG_BOOL_TABLE bool_table[] = { static CONFIG_BOOL_TABLE bool_table[] = {
VAR_ALLOW_MIN_USER, DEF_ALLOW_MIN_USER, &var_allow_min_user, VAR_ALLOW_MIN_USER, DEF_ALLOW_MIN_USER, &var_allow_min_user,
VAR_VERP_BOUNCE_OFF, DEF_VERP_BOUNCE_OFF, &var_verp_bounce_off, VAR_VERP_BOUNCE_OFF, DEF_VERP_BOUNCE_OFF, &var_verp_bounce_off,
VAR_QMGR_FDBACK_DEBUG, DEF_QMGR_FDBACK_DEBUG, &var_qmgr_feedback_debug,
0, 0,
}; };

4
postfix/src/qmgr/qmgr.h
View File

@@ -198,6 +198,9 @@ struct QMGR_QUEUE {
int todo_refcount; /* queue entries (todo list) */ int todo_refcount; /* queue entries (todo list) */
int busy_refcount; /* queue entries (busy list) */ int busy_refcount; /* queue entries (busy list) */
int window; /* slow open algorithm */ int window; /* slow open algorithm */
double success; /* cumulative positive feedback */
double failure; /* cumulative negative feedback */
double fail_cohorts; /* pseudo-cohort failure count */
QMGR_TRANSPORT *transport; /* transport linkage */ QMGR_TRANSPORT *transport; /* transport linkage */
QMGR_ENTRY_LIST todo; /* todo queue entries */ QMGR_ENTRY_LIST todo; /* todo queue entries */
QMGR_ENTRY_LIST busy; /* messages on the wire */ QMGR_ENTRY_LIST busy; /* messages on the wire */
@@ -217,6 +220,7 @@ extern void qmgr_queue_done(QMGR_QUEUE *);
extern void qmgr_queue_throttle(QMGR_QUEUE *, DSN *); extern void qmgr_queue_throttle(QMGR_QUEUE *, DSN *);
extern void qmgr_queue_unthrottle(QMGR_QUEUE *); extern void qmgr_queue_unthrottle(QMGR_QUEUE *);
extern QMGR_QUEUE *qmgr_queue_find(QMGR_TRANSPORT *, const char *); extern QMGR_QUEUE *qmgr_queue_find(QMGR_TRANSPORT *, const char *);
extern void qmgr_queue_feedback_init(void);
#define QMGR_QUEUE_THROTTLED(q) ((q)->window <= 0) #define QMGR_QUEUE_THROTTLED(q) ((q)->window <= 0)

8
postfix/src/qmgr/qmgr_deliver.c
View File

@@ -317,9 +317,11 @@ static void qmgr_deliver_update(int unused_event, char *context)
if (VSTRING_LEN(dsb->reason) == 0) if (VSTRING_LEN(dsb->reason) == 0)
vstring_strcpy(dsb->reason, "unknown error"); vstring_strcpy(dsb->reason, "unknown error");
vstring_prepend(dsb->reason, SUSPENDED, sizeof(SUSPENDED) - 1); vstring_prepend(dsb->reason, SUSPENDED, sizeof(SUSPENDED) - 1);
qmgr_queue_throttle(queue, DSN_FROM_DSN_BUF(dsb)); if (queue->window > 0) {
if (queue->window == 0) qmgr_queue_throttle(queue, DSN_FROM_DSN_BUF(dsb));
qmgr_defer_todo(queue, &dsb->dsn); if (queue->window == 0)
qmgr_defer_todo(queue, &dsb->dsn);
}
} }
} }

177
postfix/src/qmgr/qmgr_queue.c
View File

@@ -50,8 +50,9 @@
/* transport. A null result means that the queue was not found. /* transport. A null result means that the queue was not found.
/* /*
/* qmgr_queue_throttle() handles a delivery error, and decrements the /* qmgr_queue_throttle() handles a delivery error, and decrements the
/* concurrency limit for the destination. When the concurrency limit /* concurrency limit for the destination, with a lower bound of 1.
/* for a destination becomes zero, qmgr_queue_throttle() starts a timer /* When the cohort failure bound is reached, qmgr_queue_throttle()
/* sets the concurrency limit to zero and starts a timer
/* to re-enable delivery to the destination after a configurable delay. /* to re-enable delivery to the destination after a configurable delay.
/* /*
/* qmgr_queue_unthrottle() undoes qmgr_queue_throttle()'s effects. /* qmgr_queue_unthrottle() undoes qmgr_queue_throttle()'s effects.
@@ -71,7 +72,7 @@
/* P.O. Box 704 /* P.O. Box 704
/* Yorktown Heights, NY 10598, USA /* Yorktown Heights, NY 10598, USA
/* /*
/* Scheduler enhancements: /* Pre-emptive scheduler enhancements:
/* Patrik Rak /* Patrik Rak
/* Modra 6 /* Modra 6
/* 155 00, Prague, Czech Republic /* 155 00, Prague, Czech Republic
@@ -81,6 +82,7 @@
#include <sys_defs.h> #include <sys_defs.h>
#include <time.h> #include <time.h>
#include <math.h>
/* Utility library. */ /* Utility library. */
@@ -88,11 +90,13 @@
#include <mymalloc.h> #include <mymalloc.h>
#include <events.h> #include <events.h>
#include <htable.h> #include <htable.h>
#include <name_code.h>
/* Global library. */ /* Global library. */
#include <mail_params.h> #include <mail_params.h>
#include <recipient_list.h> #include <recipient_list.h>
#include <mail_proto.h> /* QMGR_LOG_WINDOW */
/* Application-specific. */ /* Application-specific. */
@@ -100,6 +104,81 @@
int qmgr_queue_count; int qmgr_queue_count;
/*
* Lookup tables for main.cf feedback method names.
*/
#define QMGR_FDBACK_CODE_BAD 0
#define QMGR_FDBACK_CODE_FIXED_1 1
#define QMGR_FDBACK_CODE_INVERSE_WIN 2
#define QMGR_FDBACK_CODE_INVERSE_1 QMGR_FDBACK_CODE_INVERSE_WIN
#define QMGR_FDBACK_CODE_INV_SQRT_WIN 3
#define QMGR_FDBACK_CODE_INV_SQRT QMGR_FDBACK_CODE_INV_SQRT_WIN
NAME_CODE qmgr_feedback_map[] = {
QMGR_FDBACK_NAME_FIXED_1, QMGR_FDBACK_CODE_FIXED_1,
QMGR_FDBACK_NAME_INVERSE_WIN, QMGR_FDBACK_CODE_INVERSE_WIN,
QMGR_FDBACK_NAME_INVERSE_1, QMGR_FDBACK_CODE_INVERSE_1,
QMGR_FDBACK_NAME_INV_SQRT_WIN, QMGR_FDBACK_CODE_INV_SQRT_WIN,
QMGR_FDBACK_NAME_INV_SQRT, QMGR_FDBACK_CODE_INV_SQRT,
0, QMGR_FDBACK_CODE_BAD,
};
static int qmgr_pos_feedback_idx;
static int qmgr_neg_feedback_idx;
/*
* Choosing the right feedback method at run-time.
*/
#define QMGR_FEEDBACK_VAL(idx, window) ( \
(idx) == QMGR_FDBACK_CODE_INVERSE_1 ? (1.0 / (window)) : \
(idx) == QMGR_FDBACK_CODE_FIXED_1 ? (1.0) : \
(1.0 / sqrt(window)) \
)
#define QMGR_ERROR_OR_RETRY_QUEUE(queue) \
(strcmp(queue->transport->name, MAIL_SERVICE_RETRY) == 0 \
|| strcmp(queue->transport->name, MAIL_SERVICE_ERROR) == 0)
#define QMGR_LOG_FEEDBACK(feedback) \
if (var_qmgr_feedback_debug && !QMGR_ERROR_OR_RETRY_QUEUE(queue)) \
msg_info("%s: feedback %g", myname, feedback);
#define QMGR_LOG_WINDOW(queue) \
if (var_qmgr_feedback_debug && !QMGR_ERROR_OR_RETRY_QUEUE(queue)) \
msg_info("%s: queue %s: limit %d window %d success %g failure %g fail_cohorts %g", \
myname, queue->name, queue->transport->dest_concurrency_limit, \
queue->window, queue->success, queue->failure, queue->fail_cohorts);
/* qmgr_queue_feedback_init - initialize feedback selection */
void qmgr_queue_feedback_init(void)
{
/*
* Positive and negative feedback method indices.
*/
qmgr_pos_feedback_idx = name_code(qmgr_feedback_map, NAME_CODE_FLAG_NONE,
var_qmgr_pos_feedback);
if (qmgr_pos_feedback_idx == QMGR_FDBACK_CODE_BAD)
msg_fatal("%s: bad feedback method: %s",
VAR_QMGR_POS_FDBACK, var_qmgr_pos_feedback);
if (var_qmgr_feedback_debug)
msg_info("positive feedback method %d, value at %d: %g",
qmgr_pos_feedback_idx, var_init_dest_concurrency,
QMGR_FEEDBACK_VAL(qmgr_pos_feedback_idx,
var_init_dest_concurrency));
qmgr_neg_feedback_idx = name_code(qmgr_feedback_map, NAME_CODE_FLAG_NONE,
var_qmgr_neg_feedback);
if (qmgr_neg_feedback_idx == QMGR_FDBACK_CODE_BAD)
msg_fatal("%s: bad feedback method: %s",
VAR_QMGR_NEG_FDBACK, var_qmgr_neg_feedback);
if (var_qmgr_feedback_debug)
msg_info("negative feedback method %d, value at %d: %g",
qmgr_neg_feedback_idx, var_init_dest_concurrency,
QMGR_FEEDBACK_VAL(qmgr_neg_feedback_idx,
var_init_dest_concurrency));
}
/* qmgr_queue_unthrottle_wrapper - in case (char *) != (struct *) */ /* qmgr_queue_unthrottle_wrapper - in case (char *) != (struct *) */
static void qmgr_queue_unthrottle_wrapper(int unused_event, char *context) static void qmgr_queue_unthrottle_wrapper(int unused_event, char *context)
@@ -122,10 +201,21 @@ void qmgr_queue_unthrottle(QMGR_QUEUE *queue)
{ {
const char *myname = "qmgr_queue_unthrottle"; const char *myname = "qmgr_queue_unthrottle";
QMGR_TRANSPORT *transport = queue->transport; QMGR_TRANSPORT *transport = queue->transport;
double feedback;
double multiplier;
if (msg_verbose) if (msg_verbose)
msg_info("%s: queue %s", myname, queue->name); msg_info("%s: queue %s", myname, queue->name);
/*
* Don't restart the negative feedback hysteresis cycle with every
* positive feedback. Restart it only when we make a positive concurrency
* adjustment (i.e. at the end of a positive feedback hysteresis cycle).
* Otherwise negative feedback would be too aggressive: negative feedback
* takes effect immediately at the start of its hysteresis cycle.
*/
queue->fail_cohorts = 0;
/* /*
* Special case when this site was dead. * Special case when this site was dead.
*/ */
@@ -135,7 +225,13 @@ void qmgr_queue_unthrottle(QMGR_QUEUE *queue)
msg_panic("%s: queue %s: window 0 status 0", myname, queue->name); msg_panic("%s: queue %s: window 0 status 0", myname, queue->name);
dsn_free(queue->dsn); dsn_free(queue->dsn);
queue->dsn = 0; queue->dsn = 0;
queue->window = transport->init_dest_concurrency; /* Back from the almost grave, best concurrency is anyone's guess. */
if (queue->busy_refcount > 0)
queue->window = queue->busy_refcount;
else
queue->window = transport->init_dest_concurrency;
queue->success = queue->failure = 0;
QMGR_LOG_WINDOW(queue);
return; return;
} }
@@ -143,11 +239,35 @@ void qmgr_queue_unthrottle(QMGR_QUEUE *queue)
* Increase the destination's concurrency limit until we reach the * Increase the destination's concurrency limit until we reach the
* transport's concurrency limit. Allow for a margin the size of the * transport's concurrency limit. Allow for a margin the size of the
* initial destination concurrency, so that we're not too gentle. * initial destination concurrency, so that we're not too gentle.
*
* Why is the concurrency increment based on preferred concurrency and not
* on the number of outstanding delivery requests? The latter fluctuates
* wildly when deliveries complete in bursts (artificial benchmark
* measurements), and does not account for cached connections.
*
* Keep the window within reasonable distance from actual concurrency
* otherwise negative feedback will be ineffective. This expression
* assumes that busy_refcount changes gradually. This is invalid when
* deliveries complete in bursts (artificial benchmark measurements).
*/ */
if (transport->dest_concurrency_limit == 0 if (transport->dest_concurrency_limit == 0
|| transport->dest_concurrency_limit > queue->window) || transport->dest_concurrency_limit > queue->window)
if (queue->window < queue->busy_refcount + transport->init_dest_concurrency) if (queue->window < queue->busy_refcount + transport->init_dest_concurrency) {
queue->window++; feedback = QMGR_FEEDBACK_VAL(qmgr_pos_feedback_idx, queue->window);
QMGR_LOG_FEEDBACK(feedback);
queue->success += feedback;
/* Prepare for overshoot (feedback > hysteresis, rounding error). */
while (queue->success >= var_qmgr_pos_hysteresis) {
queue->window += var_qmgr_pos_hysteresis;
queue->success -= var_qmgr_pos_hysteresis;
queue->failure = 0;
}
/* Prepare for overshoot. */
if (transport->dest_concurrency_limit > 0
&& queue->window > transport->dest_concurrency_limit)
queue->window = transport->dest_concurrency_limit;
}
QMGR_LOG_WINDOW(queue);
} }
/* qmgr_queue_throttle - handle destination delivery failure */ /* qmgr_queue_throttle - handle destination delivery failure */
@@ -155,6 +275,7 @@ void qmgr_queue_unthrottle(QMGR_QUEUE *queue)
void qmgr_queue_throttle(QMGR_QUEUE *queue, DSN *dsn) void qmgr_queue_throttle(QMGR_QUEUE *queue, DSN *dsn)
{ {
const char *myname = "qmgr_queue_throttle"; const char *myname = "qmgr_queue_throttle";
double feedback;
/* /*
* Sanity checks. * Sanity checks.
@@ -167,13 +288,43 @@ void qmgr_queue_throttle(QMGR_QUEUE *queue, DSN *dsn)
myname, queue->name, dsn->status, dsn->reason); myname, queue->name, dsn->status, dsn->reason);
/* /*
* Decrease the destination's concurrency limit until we reach zero, at * Don't restart the positive feedback hysteresis cycle with every
* which point the destination is declared dead. Decrease the concurrency * negative feedback. Restart it only when we make a negative concurrency
* limit by one, instead of using actual concurrency - 1, to avoid * adjustment (i.e. at the start of a negative feedback hysteresis
* declaring a host dead after just one single delivery failure. * cycle). Otherwise positive feedback would be too weak (positive
* feedback does not take effect until the end of its hysteresis cycle).
*/ */
if (queue->window > 0)
queue->window--; /*
* This queue is declared dead after a configurable number of
* pseudo-cohort failures.
*/
if (queue->window > 0) {
queue->fail_cohorts += 1.0 / queue->window;
if (queue->fail_cohorts >= var_qmgr_sac_cohorts)
queue->window = 0;
}
/*
* Decrease the destination's concurrency limit until we reach 1. Base
* adjustments on the concurrency limit itself, instead of using the
* actual concurrency. The latter fluctuates wildly when deliveries
* complete in bursts (artificial benchmark measurements).
*/
if (queue->window > 1) {
feedback = QMGR_FEEDBACK_VAL(qmgr_neg_feedback_idx, queue->window);
QMGR_LOG_FEEDBACK(feedback);
queue->failure -= feedback;
/* Prepare for overshoot (feedback > hysteresis, rounding error). */
while (queue->failure < 0) {
queue->window -= var_qmgr_neg_hysteresis;
queue->success = 0;
queue->failure += var_qmgr_neg_hysteresis;
}
/* Prepare for overshoot. */
if (queue->window < 1)
queue->window = 1;
}
/* /*
* Special case for a site that just was declared dead. * Special case for a site that just was declared dead.
@@ -184,6 +335,7 @@ void qmgr_queue_throttle(QMGR_QUEUE *queue, DSN *dsn)
(char *) queue, var_min_backoff_time); (char *) queue, var_min_backoff_time);
queue->dflags = 0; queue->dflags = 0;
} }
QMGR_LOG_WINDOW(queue);
} }
/* qmgr_queue_done - delete in-core queue for site */ /* qmgr_queue_done - delete in-core queue for site */
@@ -241,6 +393,7 @@ QMGR_QUEUE *qmgr_queue_create(QMGR_TRANSPORT *transport, const char *name,
queue->busy_refcount = 0; queue->busy_refcount = 0;
queue->transport = transport; queue->transport = transport;
queue->window = transport->init_dest_concurrency; queue->window = transport->init_dest_concurrency;
queue->success = queue->failure = queue->fail_cohorts = 0;
QMGR_LIST_INIT(queue->todo); QMGR_LIST_INIT(queue->todo);
QMGR_LIST_INIT(queue->busy); QMGR_LIST_INIT(queue->busy);
queue->dsn = 0; queue->dsn = 0;