From 5d604c14b4d013693b4eb8bcff7226e5c583a509 Mon Sep 17 00:00:00 2001 From: Wietse Venema Date: Fri, 17 Dec 2010 00:00:00 -0500 Subject: [PATCH] postfix-2.8-20101217 --- postfix/HISTORY | 13 ++ postfix/README_FILES/POSTSCREEN_README | 30 +++- postfix/html/POSTSCREEN_README.html | 36 ++++- postfix/proto/POSTSCREEN_README.html | 36 ++++- postfix/src/global/mail_version.h | 2 +- postfix/src/postscreen/postscreen.c | 3 + postfix/src/postscreen/postscreen_smtpd.c | 6 +- postfix/src/tls/Makefile.in | 2 + postfix/src/tls/tls.h | 2 - postfix/src/tls/tls_bio_ops.c | 178 ++++------------------ postfix/src/tls/tls_client.c | 34 ++--- postfix/src/tls/tls_misc.c | 4 - postfix/src/tls/tls_server.c | 35 ++--- 13 files changed, 157 insertions(+), 224 deletions(-) diff --git a/postfix/HISTORY b/postfix/HISTORY index b7461ec9d..684305649 100644 --- a/postfix/HISTORY +++ b/postfix/HISTORY @@ -16250,3 +16250,16 @@ Apologies for any names omitted. handling of out-of-range numbers. Files: global/cfg_parser.c, global/conv_time.c, global/mail_conf_int.c, global/mail_conf_long.c, global/mail_conf_nint.c. + +20101217 + + Cleanup: eliminated the code that copied TLS protocol + messages between the OpenSSL TLS engine and the network. + This change hopefully simplifies the TLS library enough + that it can be used in an event-driven TLS proxy in front + of postscreen. Files: tls/tls_bio.c, tls/tls_server.c, + tls/tls_client.c. + + This change eliminates an obscure bug where the SMTP server + would wait for another $smtpd_timeout seconds after sending + the "421 Error: timeout exceeded" message to the client. diff --git a/postfix/README_FILES/POSTSCREEN_README b/postfix/README_FILES/POSTSCREEN_README index 5e3e02271..686d15255 100644 --- a/postfix/README_FILES/POSTSCREEN_README +++ b/postfix/README_FILES/POSTSCREEN_README @@ -9,10 +9,28 @@ connections in parallel. While a single postscreen(8) process keeps zombies away from Postfix SMTP server processes, more Postfix SMTP server processes remain available for legitimate clients. -By doing these checks in a single postscreen(8) process, Postfix can avoid -wasting one SMTP server process per zombie. A side benefit of postscreen(8)'s -DNSBL lookups is that DNS records will already be cached before the Postfix -SMTP server looks them up later. +postscreen(8) is the first layer in a multi-layer defense. + + * The postscreen(8) layer blocks connections from zombies and other spambots + that are responsible for about 90% of all spam. It is implemented as a + single process to make this defense as cheap as possible. + + * The second layer implements more complex SMTP-level access checks that are + available with Postfix SMTP servers, policy daemons, and Milter + applications. + + * The third layer performs light-weight content inspection with the Postfix + built-in header_checks and body_checks. This can block unacceptable + attachments such as executable programs, and worms or viruses with easy-to- + recognize signatures. + + * The fourth layer provides heavy-weight content inspection with external + content filters. Typical examples are Amavisd-new, SpamAssassin, and Milter + applications. + +Each layer reduces the spam volume. The general strategy is to eliminate spam +early with the less expensive defenses and to use the more expensive defenses +for the spam that remains. Topics in this document: @@ -39,9 +57,7 @@ The main challenge for postscreen(8) is to make an is-it-a-zombie decision based on a single measurement. This is necessary because many zombies avoid spamming the same site repeatedly, in an attempt to fly under the radar. Once postscreen(8) decides that a client is not-a-zombie, it whitelists the client -temporarily to avoid further delays for legitimate mail. Clients that pass -postscreen(8) are still subject to the checks that are built into Postfix smtpd -(8), Postfix built-in content filters, and external content filters. +temporarily to avoid further delays for legitimate mail. Zombies have challenges too: they have only a limited amount of time to deliver spam before their IP address becomes blacklisted. To speed up spam deliveries, diff --git a/postfix/html/POSTSCREEN_README.html b/postfix/html/POSTSCREEN_README.html index e5ef607a6..fc80d6db8 100644 --- a/postfix/html/POSTSCREEN_README.html +++ b/postfix/html/POSTSCREEN_README.html @@ -23,10 +23,33 @@ process keeps zombies away from Postfix SMTP server processes, more Postfix SMTP server processes remain available for legitimate clients.

-

By doing these checks in a single postscreen(8) process, Postfix -can avoid wasting one SMTP server process per zombie. A side benefit -of postscreen(8)'s DNSBL lookups is that DNS records will already be -cached before the Postfix SMTP server looks them up later.

+

postscreen(8) is the first layer in a multi-layer defense.

+ +

+ +

Each layer reduces the spam volume. The general strategy is to +eliminate spam early with the less expensive defenses and to use +the more expensive defenses for the spam that remains.

Topics in this document:

@@ -67,10 +90,7 @@ decision based on a single measurement. This is necessary because many zombies avoid spamming the same site repeatedly, in an attempt to fly under the radar. Once postscreen(8) decides that a client is not-a-zombie, it whitelists the client temporarily to avoid -further delays for legitimate mail. Clients that pass postscreen(8) -are still subject to the checks that are built into Postfix smtpd(8), -Postfix built-in content filters, and external content filters. -

+further delays for legitimate mail.

Zombies have challenges too: they have only a limited amount of time to deliver spam before their IP address becomes blacklisted. diff --git a/postfix/proto/POSTSCREEN_README.html b/postfix/proto/POSTSCREEN_README.html index f590a5fe9..97c62b11c 100644 --- a/postfix/proto/POSTSCREEN_README.html +++ b/postfix/proto/POSTSCREEN_README.html @@ -23,10 +23,33 @@ process keeps zombies away from Postfix SMTP server processes, more Postfix SMTP server processes remain available for legitimate clients.

-

By doing these checks in a single postscreen(8) process, Postfix -can avoid wasting one SMTP server process per zombie. A side benefit -of postscreen(8)'s DNSBL lookups is that DNS records will already be -cached before the Postfix SMTP server looks them up later.

+

postscreen(8) is the first layer in a multi-layer defense.

+ +

+ +

Each layer reduces the spam volume. The general strategy is to +eliminate spam early with the less expensive defenses and to use +the more expensive defenses for the spam that remains.

Topics in this document:

@@ -67,10 +90,7 @@ decision based on a single measurement. This is necessary because many zombies avoid spamming the same site repeatedly, in an attempt to fly under the radar. Once postscreen(8) decides that a client is not-a-zombie, it whitelists the client temporarily to avoid -further delays for legitimate mail. Clients that pass postscreen(8) -are still subject to the checks that are built into Postfix smtpd(8), -Postfix built-in content filters, and external content filters. -

+further delays for legitimate mail.

Zombies have challenges too: they have only a limited amount of time to deliver spam before their IP address becomes blacklisted. diff --git a/postfix/src/global/mail_version.h b/postfix/src/global/mail_version.h index 24aa548ef..0f4ffef58 100644 --- a/postfix/src/global/mail_version.h +++ b/postfix/src/global/mail_version.h @@ -20,7 +20,7 @@ * Patches change both the patchlevel and the release date. Snapshots have no * patchlevel; they change the release date only. */ -#define MAIL_RELEASE_DATE "20101210" +#define MAIL_RELEASE_DATE "20101217" #define MAIL_VERSION_NUMBER "2.8" #ifdef SNAPSHOT diff --git a/postfix/src/postscreen/postscreen.c b/postfix/src/postscreen/postscreen.c index 958e57f8f..fed34fb4f 100644 --- a/postfix/src/postscreen/postscreen.c +++ b/postfix/src/postscreen/postscreen.c @@ -464,6 +464,9 @@ static void ps_service(VSTREAM *smtp_client_stream, /* * This program handles all incoming connections, so it must not block. * We use event-driven code for all operations that introduce latency. + * + * Note: instead of using VSTREAM-level timeouts, we enforce limits on the + * total amount of time to receive a complete SMTP command line. */ non_blocking(vstream_fileno(smtp_client_stream), NON_BLOCKING); diff --git a/postfix/src/postscreen/postscreen_smtpd.c b/postfix/src/postscreen/postscreen_smtpd.c index c05d1fe5c..5e9e19e21 100644 --- a/postfix/src/postscreen/postscreen_smtpd.c +++ b/postfix/src/postscreen/postscreen_smtpd.c @@ -554,9 +554,9 @@ static void ps_smtpd_read_event(int event, char *context) /* * Try to match the current character desired by the state * machine. If that fails, try to restart the machine with a - * match for its first state. smtpd(8) incompatibility: we - * require that lines end in , while smtpd(8) allows - * lines ending in and bare . + * match for its first state. Like smtpd(8), we understand lines + * ending in and bare . Unlike smtpd(8), we may + * treat lines ending in bare as an offense. */ for (transp = cmd_trans; transp->state != state->read_state; transp++) if (transp->want == 0) diff --git a/postfix/src/tls/Makefile.in b/postfix/src/tls/Makefile.in index 918a9732f..74c04bcaa 100644 --- a/postfix/src/tls/Makefile.in +++ b/postfix/src/tls/Makefile.in @@ -112,6 +112,7 @@ tls_certkey.o: ../../include/vstring.h tls_certkey.o: tls.h tls_certkey.o: tls_certkey.c tls_client.o: ../../include/argv.h +tls_client.o: ../../include/iostuff.h tls_client.o: ../../include/mail_params.h tls_client.o: ../../include/msg.h tls_client.o: ../../include/mymalloc.h @@ -235,6 +236,7 @@ tls_seed.o: tls_seed.c tls_server.o: ../../include/argv.h tls_server.o: ../../include/dict.h tls_server.o: ../../include/hex_code.h +tls_server.o: ../../include/iostuff.h tls_server.o: ../../include/mail_params.h tls_server.o: ../../include/msg.h tls_server.o: ../../include/mymalloc.h diff --git a/postfix/src/tls/tls.h b/postfix/src/tls/tls.h index b66a09f86..b4be636e2 100644 --- a/postfix/src/tls/tls.h +++ b/postfix/src/tls/tls.h @@ -92,8 +92,6 @@ typedef struct { int cipher_algbits; /* Private. */ SSL *con; - BIO *internal_bio; /* postfix/TLS side of pair */ - BIO *network_bio; /* network side of pair */ char *cache_type; /* tlsmgr(8) cache type if enabled */ char *serverid; /* unique server identifier */ char *namaddr; /* nam[addr] for logging */ diff --git a/postfix/src/tls/tls_bio_ops.c b/postfix/src/tls/tls_bio_ops.c index 5699f8ced..72a08e289 100644 --- a/postfix/src/tls/tls_bio_ops.c +++ b/postfix/src/tls/tls_bio_ops.c @@ -2,7 +2,7 @@ /* NAME /* tls_bio_ops 3 /* SUMMARY -/* TLS network BIO management +/* TLS network basic I/O management /* SYNOPSIS /* #define TLS_INTERNAL /* #include @@ -36,76 +36,31 @@ /* int timeout; /* TLS_SESS_STATE *context; /* DESCRIPTION -/* This layer synchronizes the TLS network buffers with the network -/* while performing TLS handshake or input/output operations. +/* This module enforces timeouts on non-blocking I/O while +/* performing TLS handshake or input/output operations. /* -/* When the TLS layer is active, it converts plain-text -/* data from Postfix into encrypted network data and vice versa. -/* However, to handle network timeout conditions, Postfix -/* needs to maintain control over network input/output. This -/* rules out the usual approach of placing the TLS layer -/* between the application and the network socket. +/* The Postfix VSTREAM read/write routines invoke the +/* tls_bio_read/write routines to send and receive plain-text +/* data. In addition, this module provides tls_bio_connect/accept +/* routines that trigger the initial TLS handshake. The +/* tls_bio_xxx routines invoke the corresponding SSL routines +/* that translate the requests into TLS protocol messages. /* -/* As shown below, Postfix reads/writes plain-text data from/to -/* the TLS layer. The TLS layer informs Postfix when it needs -/* to read/write encrypted data from/to the network; Postfix -/* then reads/writes encrypted data from/to the TLS layer and -/* takes care of the network socket I/O. +/* Whenever an SSL operation indicates that network input (or +/* output) needs to happen, the tls_bio_xxx routines wait for +/* the network to become readable (or writable) within the +/* timeout limit, then retry the SSL operation. This works +/* because the network socket is in non-blocking mode. /* -/* The TLS layer to network interface is realized with a BIO pair: +/* tls_bio_connect() performs the SSL_connect() operation. /* -/* Postfix SMTP layer | TLS layer -/* | -/* smtp/smtpd | -/* /\ || | -/* || \/ | -/* vstream read/write <===> TLS read/write/etc -/* | /\ || -/* | || \/ -/* | BIO pair (internal_bio) -/* | BIO pair (network_bio) -/* Postfix socket layer | /\ || -/* | || \/ -/* socket read/write <===> BIO read/write -/* /\ || | -/* || \/ | -/* network | +/* tls_bio_accept() performs the SSL_accept() operation. /* -/* The Postfix VSTREAM read/write operations invoke the SSL -/* read/write operations to send and retrieve plain-text data. Inside -/* the TLS layer the data are converted to/from TLS protocol. +/* tls_bio_shutdown() performs the SSL_shutdown() operation. /* -/* Whenever an SSL operation reports success, or whenever it -/* indicates that network input/output needs to happen, Postfix -/* uses the BIO read/write routines to synchronize the -/* network_bio buffer with the network. Writing data to the -/* network has precedence over reading from the network. This -/* is necessary to avoid deadlock. +/* tls_bio_read() performs the SSL_read() operation. /* -/* The BIO pair buffer size is set to 8192 bytes. This is much -/* larger than the typical Path MTU, and avoids sending tiny TCP -/* segments. It is also larger than the default VSTREAM_BUFSIZE -/* (4096, see vstream.h), so that large write operations can -/* be handled within one request. The internal buffer in the -/* network/network_bio handling layer is set to the same -/* value, since this seems to be reasonable. The code is -/* however able to handle arbitrary values smaller or larger -/* than the buffer size in the BIO pair. -/* -/* tls_bio_connect() performs the SSL_connect() operation while -/* synchronizing the network_bio buffer with the network. -/* -/* tls_bio_accept() performs the SSL_accept() operation while -/* synchronizing the network_bio buffer with the network. -/* -/* tls_bio_shutdown() performs the SSL_shutdown() operation while -/* synchronizing the network_bio buffer with the network. -/* -/* tls_bio_read() performs the SSL_read() operation while -/* synchronizing the network_bio buffer with the network. -/* -/* tls_bio_write() performs the SSL_write() operation while -/* synchronizing the network_bio buffer with the network. +/* tls_bio_write() performs the SSL_write() operation. /* /* Arguments: /* .IP fd @@ -161,87 +116,6 @@ #define TLS_INTERNAL #include -/* Application-specific. */ - -#define NETLAYER_BUFFERSIZE 8192 - -/* network_biopair_interop - synchronize network with BIO pair */ - -static int network_biopair_interop(int fd, int timeout, BIO *network_bio) -{ - const char *myname = "network_biopair_interop"; - int want_write; - int num_write; - int write_pos; - int from_bio; - int want_read; - int num_read; - int to_bio; - char buffer[NETLAYER_BUFFERSIZE]; - - /* - * To avoid deadlock, write all pending data to the network before - * attempting to read from the network. - */ - while ((want_write = BIO_ctrl_pending(network_bio)) > 0) { - if (want_write > sizeof(buffer)) - want_write = sizeof(buffer); - from_bio = BIO_read(network_bio, buffer, want_write); - - /* - * Write the complete buffer contents to the network. - */ - for (write_pos = 0; write_pos < from_bio; /* see below */ ) { - if (timeout > 0 && write_wait(fd, timeout) < 0) - return (-1); - num_write = write(fd, buffer + write_pos, from_bio - write_pos); - if (num_write <= 0) { - if ((num_write < 0) && (timeout > 0) && (errno == EAGAIN)) { - msg_warn("write() returns EAGAIN on a writable file descriptor!"); - msg_warn("pausing to avoid going into a tight select/write loop!"); - sleep(1); - } else { - msg_warn("%s: error writing %d bytes to the network: %m", - myname, from_bio - write_pos); - return (-1); - } - } else { - write_pos += num_write; - } - } - } - - /* - * Read data from the network into the BIO pair. - */ - while ((want_read = BIO_ctrl_get_read_request(network_bio)) > 0) { - if (want_read > sizeof(buffer)) - want_read = sizeof(buffer); - if (timeout > 0 && read_wait(fd, timeout) < 0) - return (-1); - num_read = read(fd, buffer, want_read); - if (num_read == 0) - /* FIX 200412 Cannot return a zero read count. */ - return (-1); - if (num_read < 0) { - if ((num_read < 0) && (timeout > 0) && (errno == EAGAIN)) { - msg_warn("read() returns EAGAIN on a readable file descriptor!"); - msg_warn("pausing to avoid going into a tight select/write loop!"); - sleep(1); - } else { - msg_warn("%s: error reading %d bytes from the network: %m", - myname, want_read); - return (-1); - } - } else { - to_bio = BIO_write(network_bio, buffer, num_read); - if (to_bio != num_read) - msg_panic("%s: BIO_write error: to_bio != num_read", myname); - } - } - return (0); -} - /* tls_bio - perform SSL input/output operation with extreme prejudice */ int tls_bio(int fd, int timeout, TLS_SESS_STATE *TLScontext, @@ -254,7 +128,6 @@ int tls_bio(int fd, int timeout, TLS_SESS_STATE *TLScontext, int status; int err; int retval = 0; - int biop_retval; int done; /* @@ -319,13 +192,14 @@ int tls_bio(int fd, int timeout, TLS_SESS_STATE *TLScontext, case SSL_ERROR_NONE: /* success */ retval = status; done = 1; - /* FALLTHROUGH */ - case SSL_ERROR_WANT_WRITE: /* flush/update buffers */ + break; + case SSL_ERROR_WANT_WRITE: + if (write_wait(fd, timeout) < 0) + return (-1); /* timeout error */ + break; case SSL_ERROR_WANT_READ: - biop_retval = - network_biopair_interop(fd, timeout, TLScontext->network_bio); - if (biop_retval < 0) - return (-1); /* network read/write error */ + if (read_wait(fd, timeout) < 0) + return (-1); /* timeout error */ break; /* diff --git a/postfix/src/tls/tls_client.c b/postfix/src/tls/tls_client.c index 7fd32d478..b9b2788c3 100644 --- a/postfix/src/tls/tls_client.c +++ b/postfix/src/tls/tls_client.c @@ -139,6 +139,7 @@ #include #include #include +#include /* non-blocking */ /* Global library. */ @@ -817,21 +818,6 @@ TLS_SESS_STATE *tls_client_start(const TLS_CLIENT_START_PROPS *props) | ((protomask & TLS_PROTOCOL_SSLv3) ? SSL_OP_NO_SSLv3 : 0L) | ((protomask & TLS_PROTOCOL_SSLv2) ? SSL_OP_NO_SSLv2 : 0L)); - /* - * The TLS connection is realized by a BIO_pair, so obtain the pair. - * - * XXX There is no need to make internal_bio a member of the TLScontext - * structure. It will be attached to TLScontext->con, and destroyed along - * with it. The network_bio, however, needs to be freed explicitly. - */ - if (!BIO_new_bio_pair(&TLScontext->internal_bio, TLS_BIO_BUFSIZE, - &TLScontext->network_bio, TLS_BIO_BUFSIZE)) { - msg_warn("Could not obtain BIO_pair"); - tls_print_errors(); - tls_free_context(TLScontext); - return (0); - } - /* * XXX To avoid memory leaks we must always call SSL_SESSION_free() after * calling SSL_set_session(), regardless of whether or not the session @@ -876,11 +862,21 @@ TLS_SESS_STATE *tls_client_start(const TLS_CLIENT_START_PROPS *props) SSL_set_connect_state(TLScontext->con); /* - * Connect the SSL connection with the Postfix side of the BIO-pair for - * reading and writing. + * Connect the SSL connection with the network socket. */ - SSL_set_bio(TLScontext->con, TLScontext->internal_bio, - TLScontext->internal_bio); + if (SSL_set_fd(TLScontext->con, vstream_fileno(props->stream)) != 1) { + msg_info("SSL_set_fd error to %s: %d", props->namaddr, sts); + tls_print_errors(); + uncache_session(app_ctx->ssl_ctx, TLScontext); + tls_free_context(TLScontext); + return (0); + } + + /* + * Turn on non-blocking I/O so that we can enforce timeouts on network + * I/O. + */ + non_blocking(vstream_fileno(props->stream), NON_BLOCKING); /* * If the debug level selected is high enough, all of the data is dumped: diff --git a/postfix/src/tls/tls_misc.c b/postfix/src/tls/tls_misc.c index 1a324b939..4f592c32a 100644 --- a/postfix/src/tls/tls_misc.c +++ b/postfix/src/tls/tls_misc.c @@ -582,8 +582,6 @@ TLS_SESS_STATE *tls_alloc_sess_context(int log_level, const char *namaddr) TLScontext = (TLS_SESS_STATE *) mymalloc(sizeof(TLS_SESS_STATE)); memset((char *) TLScontext, 0, sizeof(*TLScontext)); TLScontext->con = 0; - TLScontext->internal_bio = 0; - TLScontext->network_bio = 0; TLScontext->cache_type = 0; TLScontext->serverid = 0; TLScontext->peer_CN = 0; @@ -609,8 +607,6 @@ void tls_free_context(TLS_SESS_STATE *TLScontext) */ if (TLScontext->con != 0) SSL_free(TLScontext->con); - if (TLScontext->network_bio) - BIO_free(TLScontext->network_bio); if (TLScontext->namaddr) myfree(TLScontext->namaddr); diff --git a/postfix/src/tls/tls_server.c b/postfix/src/tls/tls_server.c index 9ed6d20ed..4d78e1b24 100644 --- a/postfix/src/tls/tls_server.c +++ b/postfix/src/tls/tls_server.c @@ -110,6 +110,7 @@ #include #include #include +#include /* non-blocking */ /* Global library. */ @@ -598,22 +599,6 @@ TLS_SESS_STATE *tls_server_start(const TLS_SERVER_START_PROPS *props) return (0); } - /* - * The TLS connection is realized by a BIO_pair, so obtain the pair. - * - * XXX There is no need to store the internal_bio handle in the TLScontext - * structure. It will be attached to and destroyed with TLScontext->con. - * The network_bio, however, needs to be freed explicitly, so we need to - * store its handle in TLScontext. - */ - if (!BIO_new_bio_pair(&TLScontext->internal_bio, TLS_BIO_BUFSIZE, - &TLScontext->network_bio, TLS_BIO_BUFSIZE)) { - msg_warn("Could not obtain BIO_pair"); - tls_print_errors(); - tls_free_context(TLScontext); - return (0); - } - /* * Before really starting anything, try to seed the PRNG a little bit * more. @@ -629,11 +614,21 @@ TLS_SESS_STATE *tls_server_start(const TLS_SERVER_START_PROPS *props) SSL_set_accept_state(TLScontext->con); /* - * Connect the SSL connection with the Postfix side of the BIO-pair for - * reading and writing. + * Connect the SSL connection with the network socket. */ - SSL_set_bio(TLScontext->con, TLScontext->internal_bio, - TLScontext->internal_bio); + if (SSL_set_fd(TLScontext->con, vstream_fileno(props->stream)) != 1) { + msg_info("SSL_set_fd error to %s: %d", props->namaddr, sts); + tls_print_errors(); + uncache_session(app_ctx->ssl_ctx, TLScontext); + tls_free_context(TLScontext); + return (0); + } + + /* + * Turn on non-blocking I/O so that we can enforce timeouts on network + * I/O. + */ + non_blocking(vstream_fileno(props->stream), NON_BLOCKING); /* * If the debug level selected is high enough, all of the data is dumped: