From 0efcf2c92e9830a84e1f6bb89feece1a8824630c Mon Sep 17 00:00:00 2001 From: Wietse Venema Date: Mon, 13 Sep 2010 00:00:00 -0500 Subject: [PATCH] postfix-2.8-20100913 --- postfix/HISTORY | 13 +- postfix/README_FILES/POSTSCREEN_README | 197 +++++++++++++---------- postfix/RELEASE_NOTES | 15 ++ postfix/html/POSTSCREEN_README.html | 210 +++++++++++++++---------- postfix/html/postscreen.8.html | 67 +++++--- postfix/man/man8/postscreen.8 | 68 +++++--- postfix/proto/POSTSCREEN_README.html | 210 +++++++++++++++---------- postfix/src/global/mail_version.h | 2 +- postfix/src/postscreen/postscreen.c | 66 +++++--- 9 files changed, 537 insertions(+), 311 deletions(-) diff --git a/postfix/HISTORY b/postfix/HISTORY index 0f7610f99..2e4cf235d 100644 --- a/postfix/HISTORY +++ b/postfix/HISTORY @@ -15968,4 +15968,15 @@ Apologies for any names omitted. command before you can use the file, and that it does not detect changes after the file is read. All information is read into memory. Files: util/dict_open.c, util/dict_thash.[hc], - proto/DATABSE_README.html, postconf/postconf.c + proto/DATABASE_README.html, postconf/postconf.c + +20100912 + + Feature: bare newline detection in postscreen. Real spambots + don't make this mistake but poorly-written software often does. + File: postscreen/smtpd.c. + + Documentation: POSTSCREEN_README including instructions for + turning postscreen(8) on without blocking mail, and more. + Trimmed the text in the postscreen(8) manpage. File: + proto/POSTSCREEN_README.html, postscreen/postscreen.c. diff --git a/postfix/README_FILES/POSTSCREEN_README b/postfix/README_FILES/POSTSCREEN_README index 3de9a9658..c5ee1dad0 100644 --- a/postfix/README_FILES/POSTSCREEN_README +++ b/postfix/README_FILES/POSTSCREEN_README @@ -14,16 +14,15 @@ wasting one SMTP server process per connection. A side benefit of postscreen (8)'s DNSBL lookups is that DNS records are already cached before the Postfix SMTP server looks them up later. -postscreen(8) maintains a temporary whitelist of positive decisions. Once an -SMTP client is whitelisted, it is immediately forwarded to a real Postfix SMTP -server process without further checking. +postscreen(8) maintains a temporary whitelist for clients that have passed a +number of tests. When an SMTP client IP address is whitelisted, postscreen(8) +hands off the connection immediately to a Postfix SMTP server process. This +minimizes the overhead for legitimate mail. -By default, the program logs only statistics, and it does not run any checks on -clients in mynetworks (primarily, to avoid problems with buggy SMTP -implementations in network appliances). - -Many of the ideas in postscreen(8) have been explored in earlier work by -Michael Tokarev, in OpenBSD spamd, and in MailChannels Traffic Control. +By default, postscreen(8) logs statistics and hands off every connection to a +Postfix SMTP server process, while excluding clients in mynetworks from all +tests (primarily, to avoid problems with non-standard SMTP implementations in +network appliances). This mode is useful for non-destructive testing. Topics in this document: @@ -36,17 +35,19 @@ Topics in this document: * Other errors * When all tests succeed * Configuring the postscreen(8) service + * Historical notes and credits TThhee bbaassiicc iiddeeaa bbeehhiinndd ppoossttssccrreeeenn((88)) Spambots have a limited amount of time to send out spam before they become blacklisted. For this reason, spambots make compromises in their SMTP protocol implementation to speed up spam deliveries. For example, they speak before -their turn. +their turn, or they ignore responses from SMTP servers. -Many spambots avoid spamming the same site repeatedly. Thus, postscreen(8) must -make a long-term decision after a single measurement. For example, allow a good -client to skip the DNSBL test for 24 hours. +Many spambots avoid spamming the same site repeatedly, in an attempt to fly +under the radar. Thus, postscreen(8) must make a long-term decision after a +single measurement. For example, allow a good client to skip the "pregreet" +test for 24 hours. To recognize spambots, postscreen(8) measures properties of the client IP address and of the client SMTP protocol implementation (the protocol @@ -67,16 +68,17 @@ passes all tests, its IP address is temporarily excluded from any tests, typically 24 hours for simple tests or 1 week for complex tests. This minimizes the impact of the tests on legitimate mail clients. -After logging the result of its tests, postscreen(8) by default forwards all -connections to a real SMTP server process. This mode is useful for non- -destructive testing. +After logging its findings, postscreen(8) by default hands off all connections +to a Postfix SMTP server process. This mode is useful for non-destructive +testing. In a typical production setting, postscreen(8) is configured to reject mail -from clients that fail one or more tests, after logging the sender and +from clients that fail one or more tests, after logging the helo, sender and recipient information. Note: postscreen(8) is not an SMTP proxy; this is intentional. The purpose is -to prioritize legitimate clients with as little overhead as possible. +to keep spambots away from Postfix, with minimal overhead for legitimate +clients. QQuuiicckk tteessttss bbeeffoorree eevveerryytthhiinngg eellssee @@ -95,8 +97,8 @@ matches the permanent whitelist, this is logged as: WWHHIITTEELLIISSTTEEDD address -The action is not configurable: immediately forward the connection to a real -SMTP server process. +The action is not configurable: immediately hand off the connection to a +Postfix SMTP server process. PPeerrmmaanneenntt bbllaacckklliisstt tteesstt @@ -123,9 +125,9 @@ logs this as: PPAASSSS OOLLDD address -The action is not configurable: immediately forward the connection to a real -SMTP server process. The client is excluded from further tests until its -temporary whitelist entry expires, as controlled with the postscreen_*_ttl +The action is not configurable: immediately hand off the connection to a +Postfix SMTP server process. The client is excluded from further tests until +its temporary whitelist entry expires, as controlled with the postscreen_*_ttl parameters. Expired entries are silently renewed if possible. TTeessttss bbeeffoorree tthhee 222200 SSMMTTPP sseerrvveerr ggrreeeettiinngg @@ -135,8 +137,8 @@ The postscreen_greet_wait parameter specifies a short time interval before the parallel. When a good client passes these tests, and no "deep protocol tests" are -configured, postscreen(8) adds the client to the temporary whitelist and passes -the "live" connection to a Postfix SMTP server process. The client can then +configured, postscreen(8) adds the client to the temporary whitelist and hands +off the "live" connection to a Postfix SMTP server process. The client can then continue as if postscreen(8) never even existed (except of course for the short postscreen_greet_wait delay). @@ -185,8 +187,8 @@ blocklist servers with optional filters and weight factors. These servers will be queried in parallel with the reverse client IP address. This test is disabled by default. - CAUTION: when postscreen rejects mail, it replies with the DNSBL domain - name. Use the postscreen_dnsbl_reply_map feature to hide "password" + CAUTION: when postscreen rejects mail, it's SMTP reply contains the DNSBL + domain name. Use the postscreen_dnsbl_reply_map feature to hide "password" information in DNSBL domain names. When the postscreen_greet_wait time has elapsed, and the combined DNSBL score @@ -222,28 +224,29 @@ ddrroopp TTeessttss aafftteerr tthhee 222200 SSMMTTPP sseerrvveerr ggrreeeettiinngg -The tests in this phase use an SMTP protocol engine that is built into the +In this phase of the protocol, postscreen(8) implements a number of "deep +protocol" tests. These tests use an SMTP protocol engine that is built into the postscreen(8) server. -Important notes: +Important note: deep protocol tests are disabled by default. They are more +intrusive than the pregreet and DNSBL tests, and they have limitations as +discussed next. - * These tests are disabled by default, because they are more intrusive than - the pregreet and DNSBL tests. + * When a good client passes the deep protocol tests, postscreen(8) adds the + client to the temporary whitelist but it cannot hand off the "live" + connection to a Postfix SMTP server process in the middle of the session. + Instead, postscreen(8) defers mail delivery attempts with a 4XX status, + logs the helo/sender/recipient information, and waits for the client to + disconnect. - When a good client passes the deep protocol tests, postscreen(8) adds the - client to the temporary whitelist but it cannot pass the "live" connection - to a Postfix SMTP server process in the middle of the session. Instead, - postscreen(8) defers mail delivery attempts with a 4XX status, logs the - helo/sender/recipient information, and waits for the client to disconnect. + The next time the client connects it will be allowed to talk to a Postfix + SMTP server process to deliver its mail. To minimize the impact of this + limitation, postscreen(8) gives deep protocol tests a relatively long + expiration time. - The next time the client connects it will be allowed to talk to a real SMTP - server process to deliver its mail. - - To minimize the impact of these tests, postscreen(8) gives them relatively - long expiration times. - - * postscreen(8) does not implement the AUTH, STARTTLS, XCLIENT, and XFORWARD - features. STARTTLS support may be added in a future version. + * postscreen(8)'s built-in SMTP engine does not implement the AUTH, STARTTLS, + XCLIENT, and XFORWARD features. STARTTLS and AUTH support may be added in a + future version. End-user clients should connect directly to the submission service. Other systems that require the above features should directly connect to a Postfix @@ -257,22 +260,21 @@ SMTP server, or they should be placed on the postscreen(8) whitelist. CCoommmmaanndd ppiippeelliinniinngg tteesstt By default, SMTP is a half-duplex protocol: the sender and receiver send one -command and one response at a time. Unlike the real Postfix SMTP server, -postscreen(8) does not announce support for ESMTP command pipelining. -Therefore, clients are not allowed to send multiple commands. This test is -disabled by default. +command and one response at a time. Unlike the Postfix SMTP server, postscreen +(8) does not announce support for ESMTP command pipelining. Therefore, clients +are not allowed to send multiple commands. postscreen(8)'s deep protocol test +for this is disabled by default. With "postscreen_pipelining_enable = yes", postscreen(8) detects spambots that send multiple commands, instead of sending one command and waiting for the server to reply. -This test is opportunistically enabled when enabled when postscreen(8) has to -use the built-in SMTP engine anyway, to make postscreen(8) logging more -informative. +This test is opportunistically enabled when postscreen(8) has to use the built- +in SMTP engine anyway. This is to make postscreen(8) logging more informative. When a client sends multiple commands, postscreen(8) logs this as: -CCOOMMMMAANNDD PPIIPPEELLIINNIINNGG aafftteerr time ffrroomm address + CCOOMMMMAANNDD PPIIPPEELLIINNIINNGG aafftteerr time ffrroomm address Translation: the SMTP client at address sent multiple SMTP commands, instead of sending one command and then waiting for the server to reply. This happened @@ -283,14 +285,20 @@ next. See "When tests fail after the 220 SMTP server greeting" below. NNoonn--SSMMTTPP ccoommmmaanndd tteesstt -With "postscreen_non_smtp_command_enable = yes", postscreen(8) detects spambots -that send non-SMTP commands, such as commands specified with the -postscreen_forbidden_commands parameter, and commands that have the syntax of a -message header label. +Some spambots send their mail through open proxies. A symptom of this is the +usage of commands such as CONNECT and other non-SMTP commands. Just like the +Postfix SMTP server's smtpd_forbidden_commands feature, postscreen(8) has an +equivalent postscreen_forbidden_commands feature to block these clients. +postscreen(8)'s deep protocol test for this is disabled by default. -This test is disabled by default. The test is opportunistically enabled when -postscreen(8) has to use the built-in SMTP engine anyway, to make postscreen(8) -logging more informative. +With "postscreen_non_smtp_command_enable = yes", postscreen(8) detects spambots +that send commands specified with the postscreen_forbidden_commands parameter. +This also detects commands with the syntax of a message header label. The +latter is a symptom that the client is sending message content after ignoring +all the responses from postscreen(8) that reject mail. + +This test is opportunistically enabled when postscreen(8) has to use the built- +in SMTP engine anyway. This is to make postscreen(8) logging more informative. When a client sends non-SMTP commands, postscreen(8) logs this as: @@ -306,15 +314,15 @@ taken next. See "When tests fail after the 220 SMTP server greeting" below. BBaarree nneewwlliinnee tteesstt SMTP is a line-oriented protocol: lines have a limited length, and are -terminated with . +terminated with . Lines ending in a "bare" , that is newline not +preceded by carriage return, are not allowed in SMTP. postscreen(8)'s deep +protocol test for this is disabled by default. -With "postscreen_bare_newline_enable = yes", postscreen(8) detects spambots -that send lines ending in bare newline characters, that is newline not preceded -by carriage return. +With "postscreen_bare_newline_enable = yes", postscreen(8) detects clients that +send lines ending in bare newline characters. -This test is disabled by default. The test is opportunistically enabled when -postscreen(8) has to use the built-in SMTP engine anyway, to make postscreen(8) -logging more informative. +This test is opportunistically enabled when postscreen(8) has to use the built- +in SMTP engine anyway. This is to make postscreen(8) logging more informative. When a client sends bare newline characters, postscreen(8) logs this as: @@ -391,17 +399,18 @@ whitelist entry that excludes the client IP address from further tests until the temporary whitelist entry expires, as controlled with the postscreen_*_ttl parameters. -When no "deep procol tests" are configured, postscreen(8) passes the "live" +When no "deep protocol tests" are configured, postscreen(8) passes the "live" connection to a Postfix SMTP server process. The client can then continue as if postscreen(8) never even existed (except for the short postscreen_greet_wait delay). -When any "deep procol tests" are configured, postscreen(8) cannot pass the -"live" connection to a Postfix SMTP server process. Instead, postscreen(8) -defers mail delivery attempts with a 4XX status, logs the helo/sender/recipient -information, and waits for the client to disconnect. The next time the client -connects it will be allowed to talk to a Postfix SMTP server process to deliver -its mail. +When any "deep protocol tests" are configured, postscreen(8) cannot hand off +the "live" connection to a Postfix SMTP server process in the middle of the +session. Instead, postscreen(8) defers mail delivery attempts with a 4XX +status, logs the helo/sender/recipient information, and waits for the client to +disconnect. The next time the client connects it will be allowed to talk to a +Postfix SMTP server process to deliver its mail. postscreen(8) mitigates the +impact of this limitation by giving deep protocol tests a long expiration time. CCoonnffiigguurriinngg tthhee ppoossttssccrreeeenn((88)) sseerrvviiccee @@ -454,6 +463,11 @@ mail: Notes: + * Some postscreen(8) configuration parameters implement stress-dependent + behavior. This is supported only when the default value is stress-dependent + (that is, it looks like ${stress?X}${stress:Y}). Other parameters always + evaluate as if the stress value is the empty string. + * See "Tests before the 220 SMTP server greeting" for details about the logging from these postscreen(8) tests. @@ -484,16 +498,21 @@ more of: than the pregreet or DNSBL tests. When a good client passes the "deep protocol tests", postscreen(8) adds the - client to the temporary whitelist but it cannot pass the "live" connection - to a Postfix SMTP server process in the middle of the session. Instead, - postscreen(8) defers mail delivery attempts with a 4XX status, logs the - helo/sender/recipient information, and waits for the client to disconnect. + client to the temporary whitelist but it cannot hand off the "live" + connection to a Postfix SMTP server process in the middle of the session. + Instead, postscreen(8) defers mail delivery attempts with a 4XX status, + logs the helo/sender/recipient information, and waits for the client to + disconnect. - When the client comes back in a later session, it is allowed to talk + When the good client comes back in a later session, it is allowed to talk directly to a Postfix SMTP server. See "after_220 Tests after the 220 SMTP server greeting above for limitations with STARTTLS, AUTH and other - features that clients may need. Wietse enables "deep protocol tests" on his - own internet-facing mail server. + features that clients may need. + + An unexpected benefit from "deep protocol tests" is that some "good" + clients don't return after the 4XX reply; these clients were not so good + after all. Wietse enables "deep protocol tests" on his own internet-facing + mail server. * There is also support for permanent blacklists and whitelists; see the description of the postscreen_whitelist_networks and @@ -517,3 +536,21 @@ processes: 5. Read the new configuration with "postfix reload". +HHiissttoorriiccaall nnootteess aanndd ccrreeddiittss + +Many ideas in postscreen(8) were explored in earlier work by Michael Tokarev, +in OpenBSD spamd, and in MailChannels Traffic Control. + +Wietse threw together a crude prototype with pregreet and dnsbl support in June +2009, because he needed something new for a Mailserver conference presentation +in July. Ralf Hildebrandt ran this code on several servers to collect real- +world evidence. This version used the dnsblog(8) ad-hoc DNS client program. + +Wietse needed new material for a LISA conference presentation in November 2010, +so he added support for DNSBL weights and filters in August, followed by a +major code rewrite, deep protocol tests, helo/sender/recipient logging, and +stress-adaptive behavior in September. Ralf Hildebrandt ran this code on +several servers to collect real-world evidence. This version still used the +same delay for pregreet and DNBL tests, as well as the embarrassing dnsblog(8) +ad-hoc DNS client. + diff --git a/postfix/RELEASE_NOTES b/postfix/RELEASE_NOTES index 36b2cec30..61acfaa9c 100644 --- a/postfix/RELEASE_NOTES +++ b/postfix/RELEASE_NOTES @@ -36,6 +36,21 @@ equal to the empty string. Incompatibility with snapshot 20100912 ====================================== +- If your DNSBL queries have a "secret" in the domain name, you + must now censor this information from the postscreen(8) SMTP + replies. For example: + + /etc/postfix/main.cf: + postscreen_dnsbl_reply_map = texthash:/etc/postfix/dnsbl_reply + + /etc/postfix/dnsbl_reply: + # Secret DNSBL name Name in postscreen(8) replies + secret.zen.spamhaus.org zen.spamhaus.org + + The texthash: format is similar to hash: except that there is no need to + run postmap(1) before the file can be used, and that it does not detect + changes after the file is read. It is new with Postfix version 2.8. + - The postscreen "continue" action is now called "ignore". The old name is still supported but no longer documented. diff --git a/postfix/html/POSTSCREEN_README.html b/postfix/html/POSTSCREEN_README.html index a7a40a94e..a22148f6a 100644 --- a/postfix/html/POSTSCREEN_README.html +++ b/postfix/html/POSTSCREEN_README.html @@ -28,18 +28,17 @@ benefit of postscreen(8)'s DNSBL lookups is that already cached before the Postfix SMTP server looks them up later.

-

postscreen(8) maintains a temporary whitelist of positive -decisions. Once an SMTP client is whitelisted, it is immediately -forwarded to a real Postfix SMTP server process without further -checking.

+

postscreen(8) maintains a temporary whitelist for clients that +have passed a number of tests. When an SMTP client IP address is +whitelisted, postscreen(8) hands off the connection immediately to +a Postfix SMTP server process. This minimizes the overhead for +legitimate mail.

-

By default, the program logs only statistics, and it does not -run any checks on clients in mynetworks (primarily, to avoid problems -with buggy SMTP implementations in network appliances).

- -

Many of the ideas in postscreen(8) have been explored in earlier -work by Michael Tokarev, in OpenBSD spamd, and in MailChannels -Traffic Control.

+

By default, postscreen(8) logs statistics and hands off every +connection to a Postfix SMTP server process, while excluding clients +in mynetworks from all tests (primarily, to avoid problems with +non-standard SMTP implementations in network appliances). This mode +is useful for non-destructive testing.

Topics in this document:

@@ -63,6 +62,8 @@ Traffic Control.

  • Configuring the postscreen(8) service +
  • Historical notes and credits +

    The basic idea behind postscreen(8)

    @@ -70,12 +71,14 @@ Traffic Control.

    Spambots have a limited amount of time to send out spam before they become blacklisted. For this reason, spambots make compromises in their SMTP protocol implementation to speed up spam deliveries. -For example, they speak before their turn.

    +For example, they speak before their turn, or they ignore responses +from SMTP servers.

    -

    Many spambots avoid spamming the same site repeatedly. Thus, -postscreen(8) must make a long-term decision after a single -measurement. For example, allow a good client to skip the DNSBL -test for 24 hours.

    +

    Many spambots avoid spamming the same site repeatedly, in an +attempt to fly under the radar. Thus, postscreen(8) must make a +long-term decision after a single measurement. For example, allow +a good client to skip the "pregreet" test +for 24 hours.

    To recognize spambots, postscreen(8) measures properties of the client IP address and of the client SMTP protocol implementation @@ -99,17 +102,17 @@ temporarily excluded from any tests, typically 24 hours for simple tests or 1 week for complex tests. This minimizes the impact of the tests on legitimate mail clients.

    -

    After logging the result of its tests, postscreen(8) by default -forwards all connections to a real SMTP server process. This mode -is useful for non-destructive testing.

    +

    After logging its findings, postscreen(8) by default hands off +all connections to a Postfix SMTP server process. This mode is +useful for non-destructive testing.

    In a typical production setting, postscreen(8) is configured to reject mail from clients that fail one or more tests, after -logging the sender and recipient information.

    +logging the helo, sender and recipient information.

    Note: postscreen(8) is not an SMTP proxy; this is intentional. -The purpose is to prioritize legitimate clients with as little -overhead as possible.

    +The purpose is to keep spambots away from Postfix, with minimal +overhead for legitimate clients.

    Quick tests before everything else

    @@ -138,8 +141,8 @@ logged as:

    WHITELISTED address -

    The action is not configurable: immediately forward the -connection to a real SMTP server process.

    +

    The action is not configurable: immediately hand off the +connection to a Postfix SMTP server process.

    Permanent blacklist test

    @@ -172,8 +175,8 @@ whitelist, postscreen(8) logs this as:

    PASS OLD address -

    The action is not configurable: immediately forward the -connection to a real SMTP server process. The client is +

    The action is not configurable: immediately hand off the +connection to a Postfix SMTP server process. The client is excluded from further tests until its temporary whitelist entry expires, as controlled with the postscreen_*_ttl parameters. Expired entries are silently renewed if possible.

    @@ -186,7 +189,7 @@ interval before the "220 text..." server greeting, where

    When a good client passes these tests, and no "deep protocol tests" are configured, postscreen(8) -adds the client to the temporary whitelist and passes the "live" +adds the client to the temporary whitelist and hands off the "live" connection to a Postfix SMTP server process. The client can then continue as if postscreen(8) never even existed (except of course for the short postscreen_greet_wait delay).

    @@ -253,9 +256,9 @@ client IP address. This test is disabled by default.

    -CAUTION: when postscreen rejects mail, it replies with the DNSBL -domain name. Use the postscreen_dnsbl_reply_map feature to hide -"password" information in DNSBL domain names. +CAUTION: when postscreen rejects mail, it's SMTP reply contains the +DNSBL domain name. Use the postscreen_dnsbl_reply_map feature to +hide "password" information in DNSBL domain names.

    @@ -304,33 +307,32 @@ this test the next time the client connects.

    Tests after the 220 SMTP server greeting

    -

    The tests in this phase use an SMTP protocol engine that is -built into the postscreen(8) server.

    +

    In this phase of the protocol, postscreen(8) implements a +number of "deep protocol" tests. These tests use an SMTP protocol +engine that is built into the postscreen(8) server.

    -

    Important notes:

    +

    Important note: deep protocol tests are disabled by default. +They are more intrusive than the pregreet and DNSBL tests, and they +have limitations as discussed next.

      -

    • These tests are disabled by default, because they -are more intrusive than the pregreet and DNSBL tests.

      - -

      When a good client passes the deep -protocol tests , postscreen(8) adds the client to the temporary -whitelist but it cannot pass the "live" connection to a Postfix +

    • When a good client passes the deep +protocol tests, postscreen(8) adds the client to the temporary +whitelist but it cannot hand off the "live" connection to a Postfix SMTP server process in the middle of the session. Instead, postscreen(8) defers mail delivery attempts with a 4XX status, logs the helo/sender/recipient information, and waits for the client to disconnect.

      The next time the client connects it will be allowed to talk -to a real SMTP server process to deliver its mail.

      +to a Postfix SMTP server process to deliver its mail. To minimize the +impact of this limitation, postscreen(8) gives deep protocol tests +a relatively long expiration time.

      -

      To minimize the impact of these tests, postscreen(8) gives them -relatively long expiration times.

      - -

    • postscreen(8) does not implement the AUTH, STARTTLS, -XCLIENT, and XFORWARD features. STARTTLS support may be added in -a future version.

      +

    • postscreen(8)'s built-in SMTP engine does not implement +the AUTH, STARTTLS, XCLIENT, and XFORWARD features. STARTTLS and +AUTH support may be added in a future version.

    @@ -355,22 +357,25 @@ should be placed on the postscreen(8) whitelist.

    By default, SMTP is a half-duplex protocol: the sender and receiver send one command and one response at a time. Unlike the -real Postfix SMTP server, postscreen(8) does not announce support +Postfix SMTP server, postscreen(8) does not announce support for ESMTP command pipelining. Therefore, clients are not allowed -to send multiple commands. This test is disabled by default.

    +to send multiple commands. postscreen(8)'s deep +protocol test for this is disabled by default.

    With "postscreen_pipelining_enable = yes", postscreen(8) detects spambots that send multiple commands, instead of sending one command -and waiting for the server to reply.

    +and waiting for the server to reply.

    -

    This test is opportunistically enabled when enabled when -postscreen(8) has to use the built-in SMTP engine anyway, to make -postscreen(8) logging more informative.

    +

    This test is opportunistically enabled when postscreen(8) has +to use the built-in SMTP engine anyway. This is to make postscreen(8) +logging more informative.

    When a client sends multiple commands, postscreen(8) logs this as:

    -COMMAND PIPELINING after time from address +
    +    COMMAND PIPELINING after time from address
+

    Translation: the SMTP client at address sent multiple SMTP commands, instead of sending one command and then waiting for @@ -383,14 +388,23 @@ after the 220 SMTP server greeting" below.

    Non-SMTP command test

    -

    With "postscreen_non_smtp_command_enable = yes", postscreen(8) -detects spambots that send non-SMTP commands, such as commands -specified with the postscreen_forbidden_commands parameter, and -commands that have the syntax of a message header label.

    +

    Some spambots send their mail through open proxies. A symptom +of this is the usage of commands such as CONNECT and other non-SMTP +commands. Just like the Postfix SMTP server's smtpd_forbidden_commands +feature, postscreen(8) has an equivalent postscreen_forbidden_commands +feature to block these clients. postscreen(8)'s deep +protocol test for this is disabled by default.

    -

    This test is disabled by default. The test is opportunistically -enabled when postscreen(8) has to use the built-in SMTP engine -anyway, to make postscreen(8) logging more informative.

    +

    With "postscreen_non_smtp_command_enable = yes", postscreen(8) +detects spambots that send commands specified with the +postscreen_forbidden_commands parameter. This also detects commands +with the syntax of a message header label. The latter is a symptom +that the client is sending message content after ignoring all the +responses from postscreen(8) that reject mail.

    + +

    This test is opportunistically enabled when postscreen(8) has +to use the built-in SMTP engine anyway. This is to make postscreen(8) +logging more informative.

    When a client sends non-SMTP commands, postscreen(8) logs this as:

    @@ -409,16 +423,19 @@ tests fail after the 220 SMTP server greeting" below.

    Bare newline test

    -

    SMTP is a line-oriented protocol: lines have a limited -length, and are terminated with <CR><LF>.

    +

    SMTP is a line-oriented protocol: lines have a limited length, +and are terminated with <CR><LF>. Lines ending in a +"bare" <LF>, that is newline not preceded by carriage return, +are not allowed in SMTP. postscreen(8)'s deep +protocol test for this is disabled by default.

    With "postscreen_bare_newline_enable = yes", postscreen(8) -detects spambots that send lines ending in bare newline -characters, that is newline not preceded by carriage return.

    +detects clients that send lines ending in bare newline characters. +

    -

    This test is disabled by default. The test is opportunistically -enabled when postscreen(8) has to use the built-in SMTP engine -anyway, to make postscreen(8) logging more informative.

    +

    This test is opportunistically enabled when postscreen(8) has +to use the built-in SMTP engine anyway. This is to make postscreen(8) +logging more informative.

    When a client sends bare newline characters, postscreen(8) logs this as: @@ -526,19 +543,22 @@ creates a temporary whitelist entry that excludes the client IP address from further tests until the temporary whitelist entry expires, as controlled with the postscreen_*_ttl parameters.

    -

    When no "deep procol tests" are +

    When no "deep protocol tests" are configured, postscreen(8) passes the "live" connection to a Postfix SMTP server process. The client can then continue as if postscreen(8) never even existed (except for the short postscreen_greet_wait delay).

    -

    When any "deep procol tests" are -configured, postscreen(8) cannot pass the "live" connection to a -Postfix SMTP server process. Instead, postscreen(8) defers mail -delivery attempts with a 4XX status, logs the helo/sender/recipient -information, and waits for the client to disconnect. The next time -the client connects it will be allowed to talk to a Postfix SMTP -server process to deliver its mail.

    +

    When any "deep protocol tests" are +configured, postscreen(8) cannot hand off the "live" connection to +a Postfix SMTP server process in the middle of the session. Instead, +postscreen(8) defers mail delivery attempts with a 4XX status, logs +the helo/sender/recipient information, and waits for the client to +disconnect. The next time the client connects it will be allowed +to talk to a Postfix SMTP server process to deliver its mail. +postscreen(8) mitigates the impact of this limitation by giving +deep protocol tests a long expiration +time.

    Configuring the postscreen(8) service

    @@ -618,6 +638,12 @@ Postfix version 2.8.

      +

    • Some postscreen(8) configuration parameters implement +stress-dependent behavior. This is supported only when the default +value is stress-dependent (that is, it looks like ${stress?X}${stress:Y}). +Other parameters always evaluate as if the stress value is the empty +string.

      +

    • See "Tests before the 220 SMTP server greeting" for details about the logging from these postscreen(8) tests.

      @@ -657,18 +683,23 @@ tests.

      When a good client passes the "deep protocol tests", postscreen(8) adds the client to the temporary -whitelist but it cannot pass the "live" connection to a Postfix +whitelist but it cannot hand off the "live" connection to a Postfix SMTP server process in the middle of the session. Instead, postscreen(8) defers mail delivery attempts with a 4XX status, logs the helo/sender/recipient information, and waits for the client to disconnect.

      -

      When the client comes back in a later session, it is allowed +

      When the good client comes back in a later session, it is allowed to talk directly to a Postfix SMTP server. See "after_220 Tests after the 220 SMTP server greeting above for limitations with STARTTLS, AUTH and other features that clients -may need. Wietse enables "deep protocol -tests" on his own internet-facing mail server.

      +may need.

      + +

      An unexpected benefit from "deep protocol +tests" is that some "good" clients don't return after the 4XX +reply; these clients were not so good after all. Wietse enables +"deep protocol tests" on his own internet-facing +mail server.

    • There is also support for permanent blacklists and whitelists; see the description of the postscreen_whitelist_networks and @@ -703,6 +734,27 @@ follow.

      +

      Historical notes and credits

      + +

      Many ideas in postscreen(8) were explored in earlier work by +Michael Tokarev, in OpenBSD spamd, and in MailChannels Traffic +Control.

      + +

      Wietse threw together a crude prototype with pregreet and dnsbl +support in June 2009, because he needed something new for a Mailserver +conference presentation in July. Ralf Hildebrandt ran this code on +several servers to collect real-world evidence. This version used +the dnsblog(8) ad-hoc DNS client program.

      + +

      Wietse needed new material for a LISA conference presentation +in November 2010, so he added support for DNSBL weights and filters +in August, followed by a major code rewrite, deep protocol tests, +helo/sender/recipient logging, and stress-adaptive behavior in +September. Ralf Hildebrandt ran this code on several servers to +collect real-world evidence. This version still used the same delay +for pregreet and DNBL tests, as well as the embarrassing dnsblog(8) +ad-hoc DNS client.

      + diff --git a/postfix/html/postscreen.8.html b/postfix/html/postscreen.8.html index fa87b1321..4463dfb53 100644 --- a/postfix/html/postscreen.8.html +++ b/postfix/html/postscreen.8.html @@ -19,31 +19,32 @@ POSTSCREEN(8) POSTSCREEN(8) processes, more Postfix SMTP server processes remain available for legitimate clients. - postscreen(8) maintains a temporary whitelist of positive - decisions. Once an SMTP client is whitelisted, it is imme- - diately forwarded to a real Postfix SMTP server process - without further checking. + postscreen(8) maintains a temporary whitelist for clients + that have passed a number of tests. When an SMTP client + IP address is whitelisted, postscreen(8) hands off the + connection immediately to a Postfix SMTP server process. + This minimizes the overhead for legitimate mail. - By default, the program logs only statistics, and it does - not run any tests against clients in mynetworks (primar- - ily, to avoid problems with non-standard SMTP implementa- - tions in network appliances and test programs). - postscreen(8) by default forwards all connections to a - real SMTP server process. This mode is useful for non- + By default, postscreen(8) logs statistics and hands off + every connection to a Postfix SMTP server process, while + excluding clients in mynetworks from all tests (primarily, + to avoid problems with non-standard SMTP implementations + in network appliances). This mode is useful for non- destructive testing. In a typical production setting, postscreen(8) is config- ured to reject mail from clients that fail one or more - tests, after logging the helo, sender and recipient infor- - mation. + tests. postscreen(8) logs rejected mail with the client + address, helo, sender and recipient information. postscreen(8) is not an SMTP proxy; this is intentional. The purpose is to keep spambots away from Postfix SMTP - server processes, not to control traffic flows. + server processes, while minimizing overhead for legitimate + traffic. SECURITY The postscreen(8) server is moderately security-sensitive. - It talks to untrusted clients on the network. The process + It talks to untrusted clients on the network. The process can be run chrooted at fixed low privilege. STANDARDS @@ -54,15 +55,26 @@ POSTSCREEN(8) POSTSCREEN(8) Problems and transactions are logged to syslogd(8). BUGS - When successful tests involve postscreen(8)'s built-in - SMTP protocol engine, postscreen(8) adds the client to the - temporary whitelist but it cannot not hand off the "live" - connection from a good SMTP client to a Postfix SMTP - server process. Instead, postscreen(8) defers attempts to + Some of the non-default protocol tests involve + postscreen(8)'s built-in SMTP protocol engine. When these + tests succeed, postscreen(8) adds the client to the tempo- + rary whitelist but it cannot not hand off the "live" con- + nection to a Postfix SMTP server process in the middle of + a session. Instead, postscreen(8) defers attempts to deliver mail with a 4XX status, and waits for the client to disconnect. The next time a good client connects, it will be allowed to talk to a Postfix SMTP server process - to deliver mail. + to deliver mail. postscreen(8) mitigates the impact of + this limitation by giving such tests a long expiration + time. + + The postscreen(8) built-in SMTP protocol engine does not + announce support for STARTTLS, AUTH, XCLIENT or XFORWARD + (support for STARTTLS and AUTH may be added in the + future). End-user clients should connect directly to the + submission service; other systems that require the above + features should directly connect to a Postfix SMTP server, + or they should be placed on the postscreen(8) whitelist. CONFIGURATION PARAMETERS Changes to main.cf are not picked up automatically, as @@ -72,11 +84,11 @@ POSTSCREEN(8) POSTSCREEN(8) The text below provides only a parameter summary. See postconf(5) for more details including examples. - NOTE: Some parameters implement stress-dependent behavior. - This is supported only when the default value is stress- - dependent (${stress?X}${stress:Y}). Other parameters - always evaluate as if the stress value is the empty - string. + NOTE: Some postscreen(8) parameters implement stress- + dependent behavior. This is supported only when the + default value is stress-dependent (that is, it looks like + ${stress?X}${stress:Y}). Other parameters always evaluate + as if the stress value is the empty string. TRIAGE PARAMETERS postscreen_bare_newline_action (ignore) @@ -295,6 +307,11 @@ POSTSCREEN(8) POSTSCREEN(8) The Secure Mailer license must be distributed with this software. +HISTORY + Many ideas in postscreen(8) were explored in earlier work + by Michael Tokarev, in OpenBSD spamd, and in MailChannels + Traffic Control. + AUTHOR(S) Wietse Venema IBM T.J. Watson Research diff --git a/postfix/man/man8/postscreen.8 b/postfix/man/man8/postscreen.8 index 7b09383dc..8c49e61ce 100644 --- a/postfix/man/man8/postscreen.8 +++ b/postfix/man/man8/postscreen.8 @@ -18,26 +18,28 @@ multiple inbound SMTP connections in parallel. While server processes, more Postfix SMTP server processes remain available for legitimate clients. -\fBpostscreen\fR(8) maintains a temporary whitelist of -positive decisions. Once an SMTP client is whitelisted, it -is immediately forwarded to a real Postfix SMTP server -process without further checking. +\fBpostscreen\fR(8) maintains a temporary whitelist for +clients that have passed a number of tests. When an SMTP +client IP address is whitelisted, \fBpostscreen\fR(8) hands +off the connection immediately to a Postfix SMTP server +process. This minimizes the overhead for legitimate mail. -By default, the program logs only statistics, and it does -not run any tests against clients in mynetworks (primarily, +By default, \fBpostscreen\fR(8) logs statistics and hands +off every connection to a Postfix SMTP server process, while +excluding clients in mynetworks from all tests (primarily, to avoid problems with non-standard SMTP implementations -in network appliances and test programs). \fBpostscreen\fR(8) -by default forwards all connections to a real SMTP server -process. This mode is useful for non-destructive testing. +in network appliances). This mode is useful for non-destructive +testing. In a typical production setting, \fBpostscreen\fR(8) is configured to reject mail from clients that fail one or -more tests, after logging the helo, sender and recipient -information. +more tests. \fBpostscreen\fR(8) logs rejected mail with the +client address, helo, sender and recipient information. \fBpostscreen\fR(8) is not an SMTP proxy; this is intentional. The purpose is to keep spambots away from Postfix SMTP -server processes, not to control traffic flows. +server processes, while minimizing overhead for legitimate +traffic. .SH "SECURITY" .na .nf @@ -58,15 +60,26 @@ Problems and transactions are logged to \fBsyslogd\fR(8). .SH BUGS .ad .fi -When successful tests involve \fBpostscreen\fR(8)'s built-in -SMTP protocol engine, \fBpostscreen\fR(8) adds the client +Some of the non-default protocol tests involve +\fBpostscreen\fR(8)'s built-in SMTP protocol engine. When +these tests succeed, \fBpostscreen\fR(8) adds the client to the temporary whitelist but it cannot not hand off the -"live" connection from a good SMTP client to a Postfix SMTP -server process. Instead, \fBpostscreen\fR(8) defers attempts -to deliver mail with a 4XX status, and waits for the client -to disconnect. The next time a good client connects, it -will be allowed to talk to a Postfix SMTP server process -to deliver mail. +"live" connection to a Postfix SMTP server process in the +middle of a session. Instead, \fBpostscreen\fR(8) defers +attempts to deliver mail with a 4XX status, and waits for +the client to disconnect. The next time a good client +connects, it will be allowed to talk to a Postfix SMTP +server process to deliver mail. \fBpostscreen\fR(8) mitigates +the impact of this limitation by giving such tests a long +expiration time. + +The \fBpostscreen\fR(8) built-in SMTP protocol engine does +not announce support for STARTTLS, AUTH, XCLIENT or XFORWARD +(support for STARTTLS and AUTH may be added in the future). +End-user clients should connect directly to the submission +service; other systems that require the above features +should directly connect to a Postfix SMTP server, or they +should be placed on the \fBpostscreen\fR(8) whitelist. .SH "CONFIGURATION PARAMETERS" .na .nf @@ -80,9 +93,10 @@ change. The text below provides only a parameter summary. See \fBpostconf\fR(5) for more details including examples. -NOTE: Some parameters implement stress-dependent behavior. -This is supported only when the default value is stress-dependent -(${stress?X}${stress:Y}). Other parameters always evaluate +NOTE: Some \fBpostscreen\fR(8) parameters implement +stress-dependent behavior. This is supported only when the +default value is stress-dependent (that is, it looks like +${stress?X}${stress:Y}). Other parameters always evaluate as if the stress value is the empty string. .SH "TRIAGE PARAMETERS" .na @@ -263,6 +277,14 @@ POSTSCREEN_README, Postfix Postscreen Howto .ad .fi The Secure Mailer license must be distributed with this software. +.SH "HISTORY" +.na +.nf +.ad +.fi +Many ideas in \fBpostscreen\fR(8) were explored in earlier +work by Michael Tokarev, in OpenBSD spamd, and in MailChannels +Traffic Control. .SH "AUTHOR(S)" .na .nf diff --git a/postfix/proto/POSTSCREEN_README.html b/postfix/proto/POSTSCREEN_README.html index 8d342384c..cbfaa53ed 100644 --- a/postfix/proto/POSTSCREEN_README.html +++ b/postfix/proto/POSTSCREEN_README.html @@ -28,18 +28,17 @@ benefit of postscreen(8)'s DNSBL lookups is that DNS records are already cached before the Postfix SMTP server looks them up later.

      -

      postscreen(8) maintains a temporary whitelist of positive -decisions. Once an SMTP client is whitelisted, it is immediately -forwarded to a real Postfix SMTP server process without further -checking.

      +

      postscreen(8) maintains a temporary whitelist for clients that +have passed a number of tests. When an SMTP client IP address is +whitelisted, postscreen(8) hands off the connection immediately to +a Postfix SMTP server process. This minimizes the overhead for +legitimate mail.

      -

      By default, the program logs only statistics, and it does not -run any checks on clients in mynetworks (primarily, to avoid problems -with buggy SMTP implementations in network appliances).

      - -

      Many of the ideas in postscreen(8) have been explored in earlier -work by Michael Tokarev, in OpenBSD spamd, and in MailChannels -Traffic Control.

      +

      By default, postscreen(8) logs statistics and hands off every +connection to a Postfix SMTP server process, while excluding clients +in mynetworks from all tests (primarily, to avoid problems with +non-standard SMTP implementations in network appliances). This mode +is useful for non-destructive testing.

      Topics in this document:

      @@ -63,6 +62,8 @@ Traffic Control.

    • Configuring the postscreen(8) service +
    • Historical notes and credits +

    The basic idea behind postscreen(8)

    @@ -70,12 +71,14 @@ Traffic Control.

    Spambots have a limited amount of time to send out spam before they become blacklisted. For this reason, spambots make compromises in their SMTP protocol implementation to speed up spam deliveries. -For example, they speak before their turn.

    +For example, they speak before their turn, or they ignore responses +from SMTP servers.

    -

    Many spambots avoid spamming the same site repeatedly. Thus, -postscreen(8) must make a long-term decision after a single -measurement. For example, allow a good client to skip the DNSBL -test for 24 hours.

    +

    Many spambots avoid spamming the same site repeatedly, in an +attempt to fly under the radar. Thus, postscreen(8) must make a +long-term decision after a single measurement. For example, allow +a good client to skip the "pregreet" test +for 24 hours.

    To recognize spambots, postscreen(8) measures properties of the client IP address and of the client SMTP protocol implementation @@ -99,17 +102,17 @@ temporarily excluded from any tests, typically 24 hours for simple tests or 1 week for complex tests. This minimizes the impact of the tests on legitimate mail clients.

    -

    After logging the result of its tests, postscreen(8) by default -forwards all connections to a real SMTP server process. This mode -is useful for non-destructive testing.

    +

    After logging its findings, postscreen(8) by default hands off +all connections to a Postfix SMTP server process. This mode is +useful for non-destructive testing.

    In a typical production setting, postscreen(8) is configured to reject mail from clients that fail one or more tests, after -logging the sender and recipient information.

    +logging the helo, sender and recipient information.

    Note: postscreen(8) is not an SMTP proxy; this is intentional. -The purpose is to prioritize legitimate clients with as little -overhead as possible.

    +The purpose is to keep spambots away from Postfix, with minimal +overhead for legitimate clients.

    Quick tests before everything else

    @@ -138,8 +141,8 @@ logged as:

    WHITELISTED address -

    The action is not configurable: immediately forward the -connection to a real SMTP server process.

    +

    The action is not configurable: immediately hand off the +connection to a Postfix SMTP server process.

    Permanent blacklist test

    @@ -172,8 +175,8 @@ whitelist, postscreen(8) logs this as:

    PASS OLD address -

    The action is not configurable: immediately forward the -connection to a real SMTP server process. The client is +

    The action is not configurable: immediately hand off the +connection to a Postfix SMTP server process. The client is excluded from further tests until its temporary whitelist entry expires, as controlled with the postscreen_*_ttl parameters. Expired entries are silently renewed if possible.

    @@ -186,7 +189,7 @@ postscreen(8) can run a number of tests in parallel.

    When a good client passes these tests, and no "deep protocol tests" are configured, postscreen(8) -adds the client to the temporary whitelist and passes the "live" +adds the client to the temporary whitelist and hands off the "live" connection to a Postfix SMTP server process. The client can then continue as if postscreen(8) never even existed (except of course for the short postscreen_greet_wait delay).

    @@ -253,9 +256,9 @@ client IP address. This test is disabled by default.

    -CAUTION: when postscreen rejects mail, it replies with the DNSBL -domain name. Use the postscreen_dnsbl_reply_map feature to hide -"password" information in DNSBL domain names. +CAUTION: when postscreen rejects mail, it's SMTP reply contains the +DNSBL domain name. Use the postscreen_dnsbl_reply_map feature to +hide "password" information in DNSBL domain names.

    @@ -304,33 +307,32 @@ this test the next time the client connects.

    Tests after the 220 SMTP server greeting

    -

    The tests in this phase use an SMTP protocol engine that is -built into the postscreen(8) server.

    +

    In this phase of the protocol, postscreen(8) implements a +number of "deep protocol" tests. These tests use an SMTP protocol +engine that is built into the postscreen(8) server.

    -

    Important notes:

    +

    Important note: deep protocol tests are disabled by default. +They are more intrusive than the pregreet and DNSBL tests, and they +have limitations as discussed next.

      -

    • These tests are disabled by default, because they -are more intrusive than the pregreet and DNSBL tests.

      - -

      When a good client passes the deep -protocol tests , postscreen(8) adds the client to the temporary -whitelist but it cannot pass the "live" connection to a Postfix +

    • When a good client passes the deep +protocol tests, postscreen(8) adds the client to the temporary +whitelist but it cannot hand off the "live" connection to a Postfix SMTP server process in the middle of the session. Instead, postscreen(8) defers mail delivery attempts with a 4XX status, logs the helo/sender/recipient information, and waits for the client to disconnect.

      The next time the client connects it will be allowed to talk -to a real SMTP server process to deliver its mail.

      +to a Postfix SMTP server process to deliver its mail. To minimize the +impact of this limitation, postscreen(8) gives deep protocol tests +a relatively long expiration time.

      -

      To minimize the impact of these tests, postscreen(8) gives them -relatively long expiration times.

      - -

    • postscreen(8) does not implement the AUTH, STARTTLS, -XCLIENT, and XFORWARD features. STARTTLS support may be added in -a future version.

      +

    • postscreen(8)'s built-in SMTP engine does not implement +the AUTH, STARTTLS, XCLIENT, and XFORWARD features. STARTTLS and +AUTH support may be added in a future version.

    @@ -355,22 +357,25 @@ should be placed on the postscreen(8) whitelist.

    By default, SMTP is a half-duplex protocol: the sender and receiver send one command and one response at a time. Unlike the -real Postfix SMTP server, postscreen(8) does not announce support +Postfix SMTP server, postscreen(8) does not announce support for ESMTP command pipelining. Therefore, clients are not allowed -to send multiple commands. This test is disabled by default.

    +to send multiple commands. postscreen(8)'s deep +protocol test for this is disabled by default.

    With "postscreen_pipelining_enable = yes", postscreen(8) detects spambots that send multiple commands, instead of sending one command -and waiting for the server to reply.

    +and waiting for the server to reply.

    -

    This test is opportunistically enabled when enabled when -postscreen(8) has to use the built-in SMTP engine anyway, to make -postscreen(8) logging more informative.

    +

    This test is opportunistically enabled when postscreen(8) has +to use the built-in SMTP engine anyway. This is to make postscreen(8) +logging more informative.

    When a client sends multiple commands, postscreen(8) logs this as:

    -COMMAND PIPELINING after time from address +
    +    COMMAND PIPELINING after time from address
+

    Translation: the SMTP client at address sent multiple SMTP commands, instead of sending one command and then waiting for @@ -383,14 +388,23 @@ after the 220 SMTP server greeting" below.

    Non-SMTP command test

    -

    With "postscreen_non_smtp_command_enable = yes", postscreen(8) -detects spambots that send non-SMTP commands, such as commands -specified with the postscreen_forbidden_commands parameter, and -commands that have the syntax of a message header label.

    +

    Some spambots send their mail through open proxies. A symptom +of this is the usage of commands such as CONNECT and other non-SMTP +commands. Just like the Postfix SMTP server's smtpd_forbidden_commands +feature, postscreen(8) has an equivalent postscreen_forbidden_commands +feature to block these clients. postscreen(8)'s deep +protocol test for this is disabled by default.

    -

    This test is disabled by default. The test is opportunistically -enabled when postscreen(8) has to use the built-in SMTP engine -anyway, to make postscreen(8) logging more informative.

    +

    With "postscreen_non_smtp_command_enable = yes", postscreen(8) +detects spambots that send commands specified with the +postscreen_forbidden_commands parameter. This also detects commands +with the syntax of a message header label. The latter is a symptom +that the client is sending message content after ignoring all the +responses from postscreen(8) that reject mail.

    + +

    This test is opportunistically enabled when postscreen(8) has +to use the built-in SMTP engine anyway. This is to make postscreen(8) +logging more informative.

    When a client sends non-SMTP commands, postscreen(8) logs this as:

    @@ -409,16 +423,19 @@ tests fail after the 220 SMTP server greeting" below.

    Bare newline test

    -

    SMTP is a line-oriented protocol: lines have a limited -length, and are terminated with <CR><LF>.

    +

    SMTP is a line-oriented protocol: lines have a limited length, +and are terminated with <CR><LF>. Lines ending in a +"bare" <LF>, that is newline not preceded by carriage return, +are not allowed in SMTP. postscreen(8)'s deep +protocol test for this is disabled by default.

    With "postscreen_bare_newline_enable = yes", postscreen(8) -detects spambots that send lines ending in bare newline -characters, that is newline not preceded by carriage return.

    +detects clients that send lines ending in bare newline characters. +

    -

    This test is disabled by default. The test is opportunistically -enabled when postscreen(8) has to use the built-in SMTP engine -anyway, to make postscreen(8) logging more informative.

    +

    This test is opportunistically enabled when postscreen(8) has +to use the built-in SMTP engine anyway. This is to make postscreen(8) +logging more informative.

    When a client sends bare newline characters, postscreen(8) logs this as: @@ -526,19 +543,22 @@ creates a temporary whitelist entry that excludes the client IP address from further tests until the temporary whitelist entry expires, as controlled with the postscreen_*_ttl parameters.

    -

    When no "deep procol tests" are +

    When no "deep protocol tests" are configured, postscreen(8) passes the "live" connection to a Postfix SMTP server process. The client can then continue as if postscreen(8) never even existed (except for the short postscreen_greet_wait delay).

    -

    When any "deep procol tests" are -configured, postscreen(8) cannot pass the "live" connection to a -Postfix SMTP server process. Instead, postscreen(8) defers mail -delivery attempts with a 4XX status, logs the helo/sender/recipient -information, and waits for the client to disconnect. The next time -the client connects it will be allowed to talk to a Postfix SMTP -server process to deliver its mail.

    +

    When any "deep protocol tests" are +configured, postscreen(8) cannot hand off the "live" connection to +a Postfix SMTP server process in the middle of the session. Instead, +postscreen(8) defers mail delivery attempts with a 4XX status, logs +the helo/sender/recipient information, and waits for the client to +disconnect. The next time the client connects it will be allowed +to talk to a Postfix SMTP server process to deliver its mail. +postscreen(8) mitigates the impact of this limitation by giving +deep protocol tests a long expiration +time.

    Configuring the postscreen(8) service

    @@ -618,6 +638,12 @@ Postfix version 2.8.

      +

    • Some postscreen(8) configuration parameters implement +stress-dependent behavior. This is supported only when the default +value is stress-dependent (that is, it looks like ${stress?X}${stress:Y}). +Other parameters always evaluate as if the stress value is the empty +string.

      +

    • See "Tests before the 220 SMTP server greeting" for details about the logging from these postscreen(8) tests.

      @@ -657,18 +683,23 @@ tests.

      When a good client passes the "deep protocol tests", postscreen(8) adds the client to the temporary -whitelist but it cannot pass the "live" connection to a Postfix +whitelist but it cannot hand off the "live" connection to a Postfix SMTP server process in the middle of the session. Instead, postscreen(8) defers mail delivery attempts with a 4XX status, logs the helo/sender/recipient information, and waits for the client to disconnect.

      -

      When the client comes back in a later session, it is allowed +

      When the good client comes back in a later session, it is allowed to talk directly to a Postfix SMTP server. See "after_220 Tests after the 220 SMTP server greeting above for limitations with STARTTLS, AUTH and other features that clients -may need. Wietse enables "deep protocol -tests" on his own internet-facing mail server.

      +may need.

      + +

      An unexpected benefit from "deep protocol +tests" is that some "good" clients don't return after the 4XX +reply; these clients were not so good after all. Wietse enables +"deep protocol tests" on his own internet-facing +mail server.

    • There is also support for permanent blacklists and whitelists; see the description of the postscreen_whitelist_networks and @@ -703,6 +734,27 @@ follow.

      +

      Historical notes and credits

      + +

      Many ideas in postscreen(8) were explored in earlier work by +Michael Tokarev, in OpenBSD spamd, and in MailChannels Traffic +Control.

      + +

      Wietse threw together a crude prototype with pregreet and dnsbl +support in June 2009, because he needed something new for a Mailserver +conference presentation in July. Ralf Hildebrandt ran this code on +several servers to collect real-world evidence. This version used +the dnsblog(8) ad-hoc DNS client program.

      + +

      Wietse needed new material for a LISA conference presentation +in November 2010, so he added support for DNSBL weights and filters +in August, followed by a major code rewrite, deep protocol tests, +helo/sender/recipient logging, and stress-adaptive behavior in +September. Ralf Hildebrandt ran this code on several servers to +collect real-world evidence. This version still used the same delay +for pregreet and DNBL tests, as well as the embarrassing dnsblog(8) +ad-hoc DNS client.

      + diff --git a/postfix/src/global/mail_version.h b/postfix/src/global/mail_version.h index 9a910abb1..33fe9315e 100644 --- a/postfix/src/global/mail_version.h +++ b/postfix/src/global/mail_version.h @@ -20,7 +20,7 @@ * Patches change both the patchlevel and the release date. Snapshots have no * patchlevel; they change the release date only. */ -#define MAIL_RELEASE_DATE "20100912" +#define MAIL_RELEASE_DATE "20100913" #define MAIL_VERSION_NUMBER "2.8" #ifdef SNAPSHOT diff --git a/postfix/src/postscreen/postscreen.c b/postfix/src/postscreen/postscreen.c index 7ea057e06..cca2ae066 100644 --- a/postfix/src/postscreen/postscreen.c +++ b/postfix/src/postscreen/postscreen.c @@ -12,26 +12,28 @@ /* server processes, more Postfix SMTP server processes remain /* available for legitimate clients. /* -/* \fBpostscreen\fR(8) maintains a temporary whitelist of -/* positive decisions. Once an SMTP client is whitelisted, it -/* is immediately forwarded to a real Postfix SMTP server -/* process without further checking. +/* \fBpostscreen\fR(8) maintains a temporary whitelist for +/* clients that have passed a number of tests. When an SMTP +/* client IP address is whitelisted, \fBpostscreen\fR(8) hands +/* off the connection immediately to a Postfix SMTP server +/* process. This minimizes the overhead for legitimate mail. /* -/* By default, the program logs only statistics, and it does -/* not run any tests against clients in mynetworks (primarily, +/* By default, \fBpostscreen\fR(8) logs statistics and hands +/* off every connection to a Postfix SMTP server process, while +/* excluding clients in mynetworks from all tests (primarily, /* to avoid problems with non-standard SMTP implementations -/* in network appliances and test programs). \fBpostscreen\fR(8) -/* by default forwards all connections to a real SMTP server -/* process. This mode is useful for non-destructive testing. +/* in network appliances). This mode is useful for non-destructive +/* testing. /* /* In a typical production setting, \fBpostscreen\fR(8) is /* configured to reject mail from clients that fail one or -/* more tests, after logging the helo, sender and recipient -/* information. +/* more tests. \fBpostscreen\fR(8) logs rejected mail with the +/* client address, helo, sender and recipient information. /* /* \fBpostscreen\fR(8) is not an SMTP proxy; this is intentional. /* The purpose is to keep spambots away from Postfix SMTP -/* server processes, not to control traffic flows. +/* server processes, while minimizing overhead for legitimate +/* traffic. /* SECURITY /* .ad /* .fi @@ -44,15 +46,26 @@ /* DIAGNOSTICS /* Problems and transactions are logged to \fBsyslogd\fR(8). /* BUGS -/* When successful tests involve \fBpostscreen\fR(8)'s built-in -/* SMTP protocol engine, \fBpostscreen\fR(8) adds the client +/* Some of the non-default protocol tests involve +/* \fBpostscreen\fR(8)'s built-in SMTP protocol engine. When +/* these tests succeed, \fBpostscreen\fR(8) adds the client /* to the temporary whitelist but it cannot not hand off the -/* "live" connection from a good SMTP client to a Postfix SMTP -/* server process. Instead, \fBpostscreen\fR(8) defers attempts -/* to deliver mail with a 4XX status, and waits for the client -/* to disconnect. The next time a good client connects, it -/* will be allowed to talk to a Postfix SMTP server process -/* to deliver mail. +/* "live" connection to a Postfix SMTP server process in the +/* middle of a session. Instead, \fBpostscreen\fR(8) defers +/* attempts to deliver mail with a 4XX status, and waits for +/* the client to disconnect. The next time a good client +/* connects, it will be allowed to talk to a Postfix SMTP +/* server process to deliver mail. \fBpostscreen\fR(8) mitigates +/* the impact of this limitation by giving such tests a long +/* expiration time. +/* +/* The \fBpostscreen\fR(8) built-in SMTP protocol engine does +/* not announce support for STARTTLS, AUTH, XCLIENT or XFORWARD +/* (support for STARTTLS and AUTH may be added in the future). +/* End-user clients should connect directly to the submission +/* service; other systems that require the above features +/* should directly connect to a Postfix SMTP server, or they +/* should be placed on the \fBpostscreen\fR(8) whitelist. /* CONFIGURATION PARAMETERS /* .ad /* .fi @@ -64,9 +77,10 @@ /* The text below provides only a parameter summary. See /* \fBpostconf\fR(5) for more details including examples. /* -/* NOTE: Some parameters implement stress-dependent behavior. -/* This is supported only when the default value is stress-dependent -/* (${stress?X}${stress:Y}). Other parameters always evaluate +/* NOTE: Some \fBpostscreen\fR(8) parameters implement +/* stress-dependent behavior. This is supported only when the +/* default value is stress-dependent (that is, it looks like +/* ${stress?X}${stress:Y}). Other parameters always evaluate /* as if the stress value is the empty string. /* TRIAGE PARAMETERS /* .ad @@ -233,6 +247,12 @@ /* .ad /* .fi /* The Secure Mailer license must be distributed with this software. +/* HISTORY +/* .ad +/* .fi +/* Many ideas in \fBpostscreen\fR(8) were explored in earlier +/* work by Michael Tokarev, in OpenBSD spamd, and in MailChannels +/* Traffic Control. /* AUTHOR(S) /* Wietse Venema /* IBM T.J. Watson Research