From 19b6598b23b8c2558e7df1117d58c2329743bb4f Mon Sep 17 00:00:00 2001
From: Wietse Venema
Postfix assumes that a remote SMTP server will reject +unknown addresses in reply to the RCPT TO command. However, some +sites report this in reply to the DATA command. For such sites +you may configure a workaround with the smtp_address_verify_target +parameter (Postfix 2.12 and later).
+When verifying a remote address, Postfix probes the preferred MTAs for that address, without actually delivering mail. If a preferred MTA accepts the address, then Postfix assumes that the @@ -227,8 +233,8 @@ MTA for that address rejects mail from your machine for any reason. This is not a limitation, but it is mentioned here just in case people believe that it is a limitation.
-Unfortunately, some sites do not reject -unknown addresses in reply to the RCPT TO command, but report a +
Unfortunately, some sites do not reject unknown addresses +in reply to the RCPT TO or DATA command, but instead report a delivery failure in response to end of DATA after a message is transferred. Postfix address verification does not work with such sites.
diff --git a/postfix/html/lmtp.8.html b/postfix/html/lmtp.8.html index 37f1a1986..54d6807f1 100644 --- a/postfix/html/lmtp.8.html +++ b/postfix/html/lmtp.8.html @@ -833,6 +833,12 @@ SMTP(8) SMTP(8) Optional list of relay hosts for SMTP destinations that can't be found or that are unreachable. + Available with Postfix 2.12 and later: + + smtp_address_verify_target (rcpt) + In the context of email address verification, the SMTP protocol + stage that determines whether an email address is deliverable. + SEE ALSO generic(5), output address rewriting header_checks(5), message header content inspection diff --git a/postfix/html/postconf.5.html b/postfix/html/postconf.5.html index b3d7e8b5f..cace24f43 100644 --- a/postfix/html/postconf.5.html +++ b/postfix/html/postconf.5.html @@ -4021,6 +4021,17 @@ configuration parameter. See there for details.This feature is available in Postfix 2.8 and later.
+ + +The LMTP-specific version of the smtp_dns_support_level +configuration parameter. See there for details.
+ +This feature is available in Postfix 2.12 and later.
+ +This feature is available in Postfix 2.8 and later.
+ + +In the context of email address verification, the SMTP protocol +stage that determines whether an email address is deliverable. +Specify one of "rcpt" or "data". The latter is needed with remote +SMTP servers that reject recipients after the DATA command. Use +transport_maps to apply this feature selectively:
+ +++ ++/etc/postfix/main.cf: + transport_maps = hash:/etc/postfix/transport ++
++ ++/etc/postfix/transport: + smtp-domain_that_verifies_after_data smtp-data-target: + lmtp-domain_that_verifies_after_data lmtp-data-target: ++
++/etc/postfix/master.cf: + smtp-data-target unix - - n - - smtp + -o smtp_address_verify_target=data + lmtp-data-target unix - - n - - lmtp + -o lmtp_address_verify_target=data +++ +Unselective use of the "data" target does no harm, but will +result in unnecessary "lost connection after DATA" events at remote +SMTP/LMTP servers.
+ +This feature is available in Postfix 2.12 and later.
+ +
Postfix assumes that a remote SMTP server will reject +unknown addresses in reply to the RCPT TO command. However, some +sites report this in reply to the DATA command. For such sites +you may configure a workaround with the smtp_address_verify_target +parameter (Postfix 2.12 and later).
+When verifying a remote address, Postfix probes the preferred MTAs for that address, without actually delivering mail. If a preferred MTA accepts the address, then Postfix assumes that the @@ -227,8 +233,8 @@ MTA for that address rejects mail from your machine for any reason. This is not a limitation, but it is mentioned here just in case people believe that it is a limitation.
-Unfortunately, some sites do not reject -unknown addresses in reply to the RCPT TO command, but report a +
Unfortunately, some sites do not reject unknown addresses +in reply to the RCPT TO or DATA command, but instead report a delivery failure in response to end of DATA after a message is transferred. Postfix address verification does not work with such sites.
diff --git a/postfix/proto/postconf.proto b/postfix/proto/postconf.proto index eb6633cab..d3783b48d 100644 --- a/postfix/proto/postconf.proto +++ b/postfix/proto/postconf.proto @@ -15419,6 +15419,52 @@ units are: s (seconds), m (minutes), h (hours), d (days), w (weeks).This feature is available in Postfix 2.9 and later.
+%PARAM smtp_address_verify_target rcpt + +In the context of email address verification, the SMTP protocol +stage that determines whether an email address is deliverable. +Specify one of "rcpt" or "data". The latter is needed with remote +SMTP servers that reject recipients after the DATA command. Use +transport_maps to apply this feature selectively:
+ +++ ++/etc/postfix/main.cf: + transport_maps = hash:/etc/postfix/transport ++
++ ++/etc/postfix/transport: + smtp-domain_that_verifies_after_data smtp-data-target: + lmtp-domain_that_verifies_after_data lmtp-data-target: ++
++/etc/postfix/master.cf: + smtp-data-target unix - - n - - smtp + -o smtp_address_verify_target=data + lmtp-data-target unix - - n - - lmtp + -o lmtp_address_verify_target=data +++ +Unselective use of the "data" target does no harm, but will +result in unnecessary "lost connection after DATA" events at remote +SMTP/LMTP servers.
+ +This feature is available in Postfix 2.12 and later.
+ +%PARAM lmtp_address_verify_target rcpt + +The LMTP-specific version of the smtp_dns_support_level +configuration parameter. See there for details.
+ +This feature is available in Postfix 2.12 and later.
+ %PARAM daemon_table_open_error_is_fatal noHow a Postfix daemon process handles errors while opening lookup diff --git a/postfix/src/bounce/bounce_template.c b/postfix/src/bounce/bounce_template.c index d011f417b..ca8c8c86f 100644 --- a/postfix/src/bounce/bounce_template.c +++ b/postfix/src/bounce/bounce_template.c @@ -462,7 +462,7 @@ static const char *bounce_template_lookup(const char *key, int unused_mode, "non-ASCII input value: \"%s\"", tp->origin, key, asc_val); return (asc_val); - } else if ((utf8_val = midna_ascii_to_utf8(asc_val)) == 0) { + } else if ((utf8_val = midna_to_utf8(asc_val)) == 0) { msg_warn("%s: conversion \"%s\" failed: " "input value: \"%s\"", tp->origin, key, asc_val); diff --git a/postfix/src/global/mail_params.h b/postfix/src/global/mail_params.h index ffc0fbbfe..4ae37622f 100644 --- a/postfix/src/global/mail_params.h +++ b/postfix/src/global/mail_params.h @@ -2778,6 +2778,14 @@ extern char *var_vrfy_relay_maps; #define DEF_VRFY_XPORT_MAPS "$" VAR_TRANSPORT_MAPS extern char *var_vrfy_xport_maps; +#define SMTP_VRFY_TGT_RCPT "rcpt" +#define SMTP_VRFY_TGT_DATA "data" +#define VAR_LMTP_VRFY_TGT "lmtp_address_verify_target" +#define DEF_LMTP_VRFY_TGT SMTP_VRFY_TGT_RCPT +#define VAR_SMTP_VRFY_TGT "smtp_address_verify_target" +#define DEF_SMTP_VRFY_TGT SMTP_VRFY_TGT_RCPT +extern char *var_smtp_vrfy_tgt; + /* * Message delivery trace service. */ diff --git a/postfix/src/global/mail_parm_split.c b/postfix/src/global/mail_parm_split.c index 18aefd367..f0c67f5cf 100644 --- a/postfix/src/global/mail_parm_split.c +++ b/postfix/src/global/mail_parm_split.c @@ -17,7 +17,7 @@ /* /* Arguments: /* .IP name -/* Parameter name. This is used to privode context for +/* Parameter name. This is used to provide context for /* error messages. /* .IP value /* Parameter value. diff --git a/postfix/src/global/mail_version.h b/postfix/src/global/mail_version.h index 80e11f679..3fd05a693 100644 --- a/postfix/src/global/mail_version.h +++ b/postfix/src/global/mail_version.h @@ -20,7 +20,7 @@ * Patches change both the patchlevel and the release date. Snapshots have no * patchlevel; they change the release date only. */ -#define MAIL_RELEASE_DATE "20141226" +#define MAIL_RELEASE_DATE "20141228" #define MAIL_VERSION_NUMBER "2.12" #ifdef SNAPSHOT diff --git a/postfix/src/posttls-finger/posttls-finger.c b/postfix/src/posttls-finger/posttls-finger.c index 35036ddd7..ac4bc8770 100644 --- a/postfix/src/posttls-finger/posttls-finger.c +++ b/postfix/src/posttls-finger/posttls-finger.c @@ -1103,7 +1103,7 @@ static DNS_RR *domain_addr(STATE *state, char *domain) * IDNA support. */ #ifndef NO_EAI - if (!allascii(domain) && (aname = midna_utf8_to_ascii(domain)) != 0) { + if (!allascii(domain) && (aname = midna_to_ascii(domain)) != 0) { msg_info("%s asciified to %s", domain, aname); } else #endif @@ -1168,7 +1168,7 @@ static DNS_RR *host_addr(STATE *state, const char *host) * IDNA support. */ #ifndef NO_EAI - if (!allascii(host) && (ahost = midna_utf8_to_ascii(host)) != 0) { + if (!allascii(host) && (ahost = midna_to_ascii(host)) != 0) { msg_info("%s asciified to %s", host, ahost); } else #endif diff --git a/postfix/src/smtp/lmtp_params.c b/postfix/src/smtp/lmtp_params.c index 24274676f..9f405c4a4 100644 --- a/postfix/src/smtp/lmtp_params.c +++ b/postfix/src/smtp/lmtp_params.c @@ -33,6 +33,7 @@ VAR_LMTP_SASL_TYPE, DEF_LMTP_SASL_TYPE, &var_smtp_sasl_type, 1, 0, VAR_LMTP_BIND_ADDR, DEF_LMTP_BIND_ADDR, &var_smtp_bind_addr, 0, 0, VAR_LMTP_BIND_ADDR6, DEF_LMTP_BIND_ADDR6, &var_smtp_bind_addr6, 0, 0, + VAR_LMTP_VRFY_TGT, DEF_LMTP_VRFY_TGT, &var_smtp_vrfy_tgt, 1, 0, VAR_LMTP_HELO_NAME, DEF_LMTP_HELO_NAME, &var_smtp_helo_name, 1, 0, VAR_LMTP_HOST_LOOKUP, DEF_LMTP_HOST_LOOKUP, &var_smtp_host_lookup, 1, 0, VAR_LMTP_DNS_SUPPORT, DEF_LMTP_DNS_SUPPORT, &var_smtp_dns_support, 0, 0, diff --git a/postfix/src/smtp/smtp.c b/postfix/src/smtp/smtp.c index cd4e344b2..977f8eb95 100644 --- a/postfix/src/smtp/smtp.c +++ b/postfix/src/smtp/smtp.c @@ -685,6 +685,11 @@ /* .IP "\fBsmtp_fallback_relay ($fallback_relay)\fR" /* Optional list of relay hosts for SMTP destinations that can't be /* found or that are unreachable. +/* .PP +/* Available with Postfix 2.12 and later: +/* .IP "\fBsmtp_address_verify_target (rcpt)\fR" +/* In the context of email address verification, the SMTP protocol +/* stage that determines whether an email address is deliverable. /* SEE ALSO /* generic(5), output address rewriting /* header_checks(5), message header content inspection @@ -816,6 +821,7 @@ char *var_smtp_sasl_mechs; char *var_smtp_sasl_type; char *var_smtp_bind_addr; char *var_smtp_bind_addr6; +char *var_smtp_vrfy_tgt; bool var_smtp_rand_addr; int var_smtp_pix_thresh; int var_queue_run_delay; @@ -1077,6 +1083,11 @@ static void post_init(char *unused_name, char **unused_argv) */ smtp_dns_res_opt = name_mask(VAR_LMTP_SMTP(DNS_RES_OPT), dns_res_opt_masks, var_smtp_dns_res_opt); + + /* + * Address verification. + */ + smtp_vrfy_init(); } /* pre_init - pre-jail initialization */ diff --git a/postfix/src/smtp/smtp.h b/postfix/src/smtp/smtp.h index 8c8fa5700..f8c08ff9e 100644 --- a/postfix/src/smtp/smtp.h +++ b/postfix/src/smtp/smtp.h @@ -365,6 +365,7 @@ extern int smtp_connect(SMTP_STATE *); /* * smtp_proto.c */ +extern void smtp_vrfy_init(void); extern int smtp_helo(SMTP_STATE *); extern int smtp_xfer(SMTP_STATE *); extern int smtp_rset(SMTP_STATE *); @@ -515,6 +516,11 @@ extern void smtp_chat_notify(SMTP_SESSION *); #define DSN_BY_LOCAL_MTA ((char *) 0) /* DSN issued by local MTA */ +#define SMTP_RESP_SET_DSN(resp, _dsn) do { \ + vstring_strcpy((resp)->dsn_buf, (_dsn)); \ + (resp)->dsn = STR((resp)->dsn_buf); \ + } while (0) + /* * These operations implement a redundant mark-and-sweep algorithm that * explicitly accounts for the fate of every recipient. The interface is diff --git a/postfix/src/smtp/smtp_addr.c b/postfix/src/smtp/smtp_addr.c index fb1833326..7ac4607f4 100644 --- a/postfix/src/smtp/smtp_addr.c +++ b/postfix/src/smtp/smtp_addr.c @@ -378,7 +378,7 @@ DNS_RR *smtp_domain_addr(const char *name, DNS_RR **mxrr, int misc_flags, * IDNA support. */ #ifndef NO_EAI - if (!allascii(name) && (aname = midna_utf8_to_ascii(name)) != 0) { + if (!allascii(name) && (aname = midna_to_ascii(name)) != 0) { if (msg_verbose) msg_info("%s asciified to %s", name, aname); } else @@ -524,7 +524,7 @@ DNS_RR *smtp_host_addr(const char *host, int misc_flags, DSN_BUF *why) * IDNA support. */ #ifndef NO_EAI - if (!allascii(host) && (ahost = midna_utf8_to_ascii(host)) != 0) { + if (!allascii(host) && (ahost = midna_to_ascii(host)) != 0) { if (msg_verbose) msg_info("%s asciified to %s", host, ahost); } else diff --git a/postfix/src/smtp/smtp_params.c b/postfix/src/smtp/smtp_params.c index 2b2e860a4..cb66a6f27 100644 --- a/postfix/src/smtp/smtp_params.c +++ b/postfix/src/smtp/smtp_params.c @@ -34,6 +34,7 @@ VAR_SMTP_SASL_TYPE, DEF_SMTP_SASL_TYPE, &var_smtp_sasl_type, 1, 0, VAR_SMTP_BIND_ADDR, DEF_SMTP_BIND_ADDR, &var_smtp_bind_addr, 0, 0, VAR_SMTP_BIND_ADDR6, DEF_SMTP_BIND_ADDR6, &var_smtp_bind_addr6, 0, 0, + VAR_SMTP_VRFY_TGT, DEF_SMTP_VRFY_TGT, &var_smtp_vrfy_tgt, 1, 0, VAR_SMTP_HELO_NAME, DEF_SMTP_HELO_NAME, &var_smtp_helo_name, 1, 0, VAR_SMTP_HOST_LOOKUP, DEF_SMTP_HOST_LOOKUP, &var_smtp_host_lookup, 1, 0, VAR_SMTP_DNS_SUPPORT, DEF_SMTP_DNS_SUPPORT, &var_smtp_dns_support, 0, 0, diff --git a/postfix/src/smtp/smtp_proto.c b/postfix/src/smtp/smtp_proto.c index 4b17b5743..9bf9926ff 100644 --- a/postfix/src/smtp/smtp_proto.c +++ b/postfix/src/smtp/smtp_proto.c @@ -144,6 +144,8 @@ #include
#include #include +#include +#include #include #include #include @@ -261,6 +263,24 @@ HBC_CALL_BACKS smtp_hbc_callbacks[1] = { smtp_text_out, }; +static int smtp_vrfy_tgt; + +/* smtp_vrfy_init - initialize */ + +void smtp_vrfy_init(void) +{ + static const NAME_CODE vrfy_init_table[] = { + SMTP_VRFY_TGT_RCPT, SMTP_STATE_RCPT, + SMTP_VRFY_TGT_DATA, SMTP_STATE_DATA, + 0, + }; + + if ((smtp_vrfy_tgt = name_code(vrfy_init_table, NAME_CODE_FLAG_NONE, + var_smtp_vrfy_tgt)) == 0) + msg_fatal("bad protocol stage: \"%s = %s\"", + VAR_SMTP_VRFY_TGT, var_smtp_vrfy_tgt); +} + /* smtp_helo - perform initial handshake with SMTP server */ int smtp_helo(SMTP_STATE *state) @@ -1541,7 +1561,8 @@ static int smtp_loop(SMTP_STATE *state, NOCLOBBER int send_state, dsn_notify_str(rcpt->dsn_notify)); } if ((next_rcpt = send_rcpt + 1) == SMTP_RCPT_LEFT(state)) - next_state = DEL_REQ_TRACE_ONLY(request->flags) ? + next_state = (DEL_REQ_TRACE_ONLY(request->flags) + && smtp_vrfy_tgt == SMTP_STATE_RCPT) ? SMTP_STATE_ABORT : SMTP_STATE_DATA; break; @@ -1808,7 +1829,8 @@ static int smtp_loop(SMTP_STATE *state, NOCLOBBER int send_state, } ++nrcpt; /* If trace-only, mark the recipient done. */ - if (DEL_REQ_TRACE_ONLY(request->flags)) { + if (DEL_REQ_TRACE_ONLY(request->flags) + && smtp_vrfy_tgt == SMTP_STATE_RCPT) { translit(resp->str, "\n", " "); smtp_rcpt_done(state, resp, rcpt); } @@ -1822,7 +1844,8 @@ static int smtp_loop(SMTP_STATE *state, NOCLOBBER int send_state, } /* If trace-only, send RSET instead of DATA. */ if (++recv_rcpt == SMTP_RCPT_LEFT(state)) - recv_state = DEL_REQ_TRACE_ONLY(request->flags) ? + recv_state = (DEL_REQ_TRACE_ONLY(request->flags) + && smtp_vrfy_tgt == SMTP_STATE_RCPT) ? SMTP_STATE_ABORT : SMTP_STATE_DATA; /* XXX Also: record if non-delivering session. */ break; @@ -1833,6 +1856,7 @@ static int smtp_loop(SMTP_STATE *state, NOCLOBBER int send_state, * receiver can apply a course correction. */ case SMTP_STATE_DATA: + recv_state = SMTP_STATE_DOT; if (resp->code / 100 != 3) { if (nrcpt > 0) smtp_mesg_fail(state, STR(iter->host), resp, @@ -1842,7 +1866,30 @@ static int smtp_loop(SMTP_STATE *state, NOCLOBBER int send_state, xfer_request[SMTP_STATE_DATA]); nrcpt = -1; } - recv_state = SMTP_STATE_DOT; + + /* + * In the case of a successful address probe with target + * equal to DATA, the remote server is now in the DATA + * state, and therefore we must not make any further + * attempt to send or receive on this connection. This + * means that we cannot not reuse the general-purpose + * course-correction logic below which sends RSET (and + * perhaps QUIT). Instead we "jump" straight to the exit + * and force an unceremonious disconnect. + */ + else if (DEL_REQ_TRACE_ONLY(request->flags) + && smtp_vrfy_tgt == SMTP_STATE_DATA) { + for (nrcpt = 0; nrcpt < recv_rcpt; nrcpt++) { + rcpt = request->rcpt_list.info + nrcpt; + if (!SMTP_RCPT_ISMARKED(rcpt)) { + translit(resp->str, "\n", " "); + SMTP_RESP_SET_DSN(resp, "2.0.0"); + smtp_rcpt_done(state, resp, rcpt); + } + } + DONT_CACHE_THIS_SESSION; + send_state = recv_state = SMTP_STATE_LAST; + } break; /* diff --git a/postfix/src/smtpd/smtpd_check.c b/postfix/src/smtpd/smtpd_check.c index e7dd7d4e2..fd114c0d1 100644 --- a/postfix/src/smtpd/smtpd_check.c +++ b/postfix/src/smtpd/smtpd_check.c @@ -1417,7 +1417,7 @@ static int reject_unknown_mailhost(SMTPD_STATE *state, const char *name, * Fix 20140924: convert domain to ASCII. */ #ifndef NO_EAI - if (!allascii(name) && (aname = midna_utf8_to_ascii(name)) != 0) { + if (!allascii(name) && (aname = midna_to_ascii(name)) != 0) { if (msg_verbose) msg_info("%s asciified to %s", name, aname); name = aname; @@ -1916,7 +1916,7 @@ static int permit_mx_backup(SMTPD_STATE *state, const char *recipient, * Fix 20140924: convert domain to ASCII. */ #ifndef NO_EAI - if (!allascii(domain) && (adomain = midna_utf8_to_ascii(domain)) != 0) { + if (!allascii(domain) && (adomain = midna_to_ascii(domain)) != 0) { if (msg_verbose) msg_info("%s asciified to %s", domain, adomain); domain = adomain; @@ -2914,7 +2914,7 @@ static int check_server_access(SMTPD_STATE *state, const char *table, * Fix 20140924: convert domain to ASCII. */ #ifndef NO_EAI - if (!allascii(domain) && (adomain = midna_utf8_to_ascii(domain)) != 0) { + if (!allascii(domain) && (adomain = midna_to_ascii(domain)) != 0) { if (msg_verbose) msg_info("%s asciified to %s", domain, adomain); domain = adomain; @@ -3634,7 +3634,7 @@ static const SMTPD_RBL_STATE *find_dnsxl_domain(SMTPD_STATE *state, * Fix 20140706: convert domain to ASCII. */ #ifndef NO_EAI - if (!allascii(domain) && (adomain = midna_utf8_to_ascii(domain)) != 0) { + if (!allascii(domain) && (adomain = midna_to_ascii(domain)) != 0) { if (msg_verbose) msg_info("%s asciified to %s", domain, adomain); domain = adomain; diff --git a/postfix/src/tls/tls_client.c b/postfix/src/tls/tls_client.c index d0121bc62..f30f1e3ae 100644 --- a/postfix/src/tls/tls_client.c +++ b/postfix/src/tls/tls_client.c @@ -535,7 +535,7 @@ static int match_servername(const char *certid, */ if (!allascii(certid)) return (0); - if (!allascii(nexthop) && (aname = midna_utf8_to_ascii(nexthop)) != 0) { + if (!allascii(nexthop) && (aname = midna_to_ascii(nexthop)) != 0) { if (msg_verbose) msg_info("%s asciified to %s", nexthop, aname); nexthop = aname; @@ -586,7 +586,7 @@ static int match_servername(const char *certid, } } if (!allascii(domain) - && (aname = midna_utf8_to_ascii(domain)) != 0) { + && (aname = midna_to_ascii(domain)) != 0) { if (msg_verbose) msg_info("%s asciified to %s", domain, aname); domain = aname; diff --git a/postfix/src/util/Makefile.in b/postfix/src/util/Makefile.in index e8739d1b2..b634d30ad 100644 --- a/postfix/src/util/Makefile.in +++ b/postfix/src/util/Makefile.in @@ -515,7 +515,7 @@ tests: all valid_hostname_test mac_expand_test dict_test unescape_test \ dict_cidr_test attr_scan_plain_test htable_test hex_code_test \ myaddrinfo_test format_tv_test ip_match_test name_mask_tests \ base32_code_test dict_thash_test surrogate_test timecmp_test \ - dict_static_test dict_inline_test + dict_static_test dict_inline_test midna_test root_tests: @@ -750,6 +750,11 @@ dict_inline_test: dict_open dict_inline.ref diff dict_inline.ref dict_inline.tmp rm -f dict_inline.tmp +midna_test: midna midna_test.in midna_test.ref + $(SHLIB_ENV) ./midna midna_test.tmp 2>&1 + diff midna_test.ref midna_test.tmp + rm -f midna_test.tmp + depend: $(MAKES) (sed '1,/^# do not edit/!d' Makefile.in; \ set -e; for i in [a-z][a-z0-9]*.c; do \ diff --git a/postfix/src/util/midna.c b/postfix/src/util/midna.c index 70b393e32..0f4118ee1 100644 --- a/postfix/src/util/midna.c +++ b/postfix/src/util/midna.c @@ -8,21 +8,34 @@ /* /* int midna_cache_size; /* -/* const char *midna_utf8_to_ascii( +/* const char *midna_to_ascii( /* const char *name) /* -/* const char *midna_ascii_to_utf8( +/* const char *midna_to_utf8( +/* const char *name) +/* +/* const char *midna_suffix_to_ascii( +/* const char *name) +/* +/* const char *midna_suffix_to_utf8( /* const char *name) /* DESCRIPTION /* The functions in this module transform domain names from /* or to IDNA form. The result is cached to avoid repeated /* conversion of the same name. /* -/* midna_utf8_to_ascii() converts an UTF-8 domain name to -/* ASCII. The result is a null pointer in case of error. +/* midna_to_ascii() converts an UTF-8 or ASCII domain name to +/* ASCII. The result is a null pointer in case of error. This +/* function verifies that the result is a valid ASCII domainname. /* -/* midna_ascii_to_utf8() converts an ASCII domain name to -/* UTF-8. The result is a null pointer in case of error. +/* midna_to_utf8() converts an UTF-8 or ASCII domain name to +/* UTF-8. The result is a null pointer in case of error. This +/* function verifies that the result converts to a valid ASCII +/* domainname. +/* +/* midna_suffix_to_ascii() and midna_suffix_to_utf8() take a +/* name that starts with '.' and otherwise perform the same +/* operations as midna_to_ascii() and midna_to_utf8(). /* /* midna_cache_size specifies the size of the conversion result /* cache. This value is used only once, upon the first lookup @@ -31,7 +44,7 @@ /* msg(3) diagnostics interface /* DIAGNOSTICS /* Fatal errors: memory allocation problem. -/* Warnings: conversion error. +/* Warnings: conversion error or result validation error. /* LICENSE /* .ad /* .fi @@ -50,6 +63,7 @@ */ #include #include +#include #ifndef NO_EAI #include @@ -67,15 +81,18 @@ /* * Application-specific. */ -#define DEF_MIDNA_CACHE_SIZE 100 +#define DEF_MIDNA_CACHE_SIZE 256 int midna_cache_size = DEF_MIDNA_CACHE_SIZE; +static VSTRING *midna_buf; /* x.suffix */ -/* midna_utf8_to_ascii_create - convert UTF8 domain to ASCII */ +#define STR(x) vstring_str(x) -static void *midna_utf8_to_ascii_create(const char *name, void *unused_context) +/* midna_to_ascii_create - convert domain to ASCII */ + +static void *midna_to_ascii_create(const char *name, void *unused_context) { - static const char myname[] = "midna_utf8_to_ascii_create"; + static const char myname[] = "midna_to_ascii_create"; char buf[1024]; /* XXX */ UErrorCode error = U_ZERO_ERROR; UIDNAInfo info = UIDNA_INFO_INITIALIZER; @@ -85,11 +102,15 @@ static void *midna_utf8_to_ascii_create(const char *name, void *unused_context) /* * Paranoia: do not expose uidna_*() to unfiltered network data. */ - if (valid_utf8_string(name, strlen(name)) == 0) { - msg_warn("%s: Problem translating domain \"%s\" to IDNA form: %s", + if (allascii(name) == 0 && valid_utf8_string(name, strlen(name)) == 0) { + msg_warn("%s: Problem translating domain \"%s\" to ASCII form: %s", myname, name, "malformed UTF-8"); return (0); } + + /* + * Perform the requested conversion. + */ idna = uidna_openUTS46(UIDNA_DEFAULT, &error); anl = uidna_nameToASCII_UTF8(idna, name, strlen(name), @@ -97,20 +118,32 @@ static void *midna_utf8_to_ascii_create(const char *name, void *unused_context) &info, &error); uidna_close(idna); + + /* + * Paranoia: verify that the result is a valid ASCII domain name. A quick + * check shows that the UTS46 implementation will reject labels that + * start or end in '-' or that are over-long, but let's play safe here. + */ if (U_SUCCESS(error) && info.errors == 0 && anl > 0) { + buf[anl] = 0; /* XXX */ + if (!valid_hostname(buf, DONT_GRIPE)) { + msg_warn("%s: Problem translating domain \"%s\" to ASCII form: %s", + myname, name, "malformed ASCII label(s)"); + return (0); + } return (mystrndup(buf, anl)); } else { - msg_warn("%s: Problem translating domain \"%s\" to IDNA form: %s", + msg_warn("%s: Problem translating domain \"%s\" to ASCII form: %s", myname, name, u_errorName(error)); return (0); } } -/* midna_ascii_to_utf8_create - convert ASCII domain to UTF8 */ +/* midna_to_utf8_create - convert domain to UTF8 */ -static void *midna_ascii_to_utf8_create(const char *name, void *unused_context) +static void *midna_to_utf8_create(const char *name, void *unused_context) { - static const char myname[] = "midna_ascii_to_utf8_create"; + static const char myname[] = "midna_to_utf8_create"; char buf[1024]; /* XXX */ UErrorCode error = U_ZERO_ERROR; UIDNAInfo info = UIDNA_INFO_INITIALIZER; @@ -120,11 +153,15 @@ static void *midna_ascii_to_utf8_create(const char *name, void *unused_context) /* * Paranoia: do not expose uidna_*() to unfiltered network data. */ - if (valid_hostname(name, DONT_GRIPE) == 0) { - msg_warn("%s: Problem translating domain \"%s\" to UTF8 form: %s", - myname, name, "malformed ASCII"); + if (allascii(name) == 0 && valid_utf8_string(name, strlen(name)) == 0) { + msg_warn("%s: Problem translating domain \"%s\" to UTF-8 form: %s", + myname, name, "malformed UTF-8"); return (0); } + + /* + * Perform the requested conversion. + */ idna = uidna_openUTS46(UIDNA_DEFAULT, &error); anl = uidna_nameToUnicodeUTF8(idna, name, strlen(name), @@ -132,10 +169,18 @@ static void *midna_ascii_to_utf8_create(const char *name, void *unused_context) &info, &error); uidna_close(idna); + + /* + * Paranoia: UTS46 toUTF8 will accept and produce a name that does not + * convert to a valid ASCII domain name. So we enforce sanity here. + */ if (U_SUCCESS(error) && info.errors == 0 && anl > 0) { + buf[anl] = 0; /* XXX */ + if (midna_to_ascii(buf) == 0) + return (0); return (mystrndup(buf, anl)); } else { - msg_warn("%s: Problem translating domain \"%s\" to IDNA form: %s", + msg_warn("%s: Problem translating domain \"%s\" to UTF8 form: %s", myname, name, u_errorName(error)); return (0); } @@ -149,39 +194,76 @@ static void midna_cache_free(void *value, void *unused_context) myfree(value); } -/* midna_utf8_to_ascii - convert UTF8 hostname to ASCII */ +/* midna_to_ascii - convert name to ASCII */ -const char *midna_utf8_to_ascii(const char *name) +const char *midna_to_ascii(const char *name) { - static CTABLE *midna_utf8_to_ascii_cache = 0; + static CTABLE *midna_to_ascii_cache = 0; - if (midna_utf8_to_ascii_cache == 0) - midna_utf8_to_ascii_cache = ctable_create(midna_cache_size, - midna_utf8_to_ascii_create, - midna_cache_free, - (void *) 0); - return (ctable_locate(midna_utf8_to_ascii_cache, name)); + if (midna_to_ascii_cache == 0) + midna_to_ascii_cache = ctable_create(midna_cache_size, + midna_to_ascii_create, + midna_cache_free, + (void *) 0); + return (ctable_locate(midna_to_ascii_cache, name)); } -/* midna_ascii_to_utf8 - convert UTF8 hostname to ASCII */ +/* midna_to_utf8 - convert name to UTF8 */ -const char *midna_ascii_to_utf8(const char *name) +const char *midna_to_utf8(const char *name) { - static CTABLE *midna_ascii_to_utf8_cache = 0; + static CTABLE *midna_to_utf8_cache = 0; - if (midna_ascii_to_utf8_cache == 0) - midna_ascii_to_utf8_cache = ctable_create(midna_cache_size, - midna_ascii_to_utf8_create, - midna_cache_free, - (void *) 0); - return (ctable_locate(midna_ascii_to_utf8_cache, name)); + if (midna_to_utf8_cache == 0) + midna_to_utf8_cache = ctable_create(midna_cache_size, + midna_to_utf8_create, + midna_cache_free, + (void *) 0); + return (ctable_locate(midna_to_utf8_cache, name)); +} + +/* midna_suffix_to_ascii - convert .name to ASCII */ + +const char *midna_suffix_to_ascii(const char *suffix) +{ + const char *cache_res; + + /* + * If prepending x to .name causes the result to become too long, then + * the suffix is bad. + */ + if (midna_buf == 0) + midna_buf = vstring_alloc(100); + vstring_sprintf(midna_buf, "x%s", suffix); + if ((cache_res = midna_to_ascii(STR(midna_buf))) == 0) + return (0); + else + return (cache_res + 1); +} + +/* midna_suffix_to_utf8 - convert .name to UTF8 */ + +const char *midna_suffix_to_utf8(const char *name) +{ + const char *cache_res; + + /* + * If prepending x to .name causes the result to become too long, then + * the suffix is bad. + */ + if (midna_buf == 0) + midna_buf = vstring_alloc(100); + vstring_sprintf(midna_buf, "x%s", name); + if ((cache_res = midna_to_utf8(STR(midna_buf))) == 0) + return (0); + else + return (cache_res + 1); } #ifdef TEST /* - * Test program - reads hostnames from stdin, reports invalid hostnames to - * stderr. + * Test program - reads names from stdin, reports invalid names to stderr. */ #include #include @@ -207,26 +289,37 @@ int main(int argc, char **argv) temp_utf8_kludge = 1; while (vstring_fgets_nonl(buffer, VSTREAM_IN)) { - msg_info("testing: \"%s\"", bp = vstring_str(buffer)); + bp = STR(buffer); + msg_info("> %s", bp); + while (ISSPACE(*bp)) + bp++; + if (*bp == '#' || *bp == 0) + continue; if (!allascii(bp)) { - ascii = midna_utf8_to_ascii(bp); + utf8 = midna_to_utf8(bp); + if (utf8 != 0) + msg_info("\"%s\" ->utf8 \"%s\"", bp, utf8); + ascii = midna_to_ascii(bp); if (ascii != 0) { - msg_info("\"%s\" -> \"%s\"", bp, ascii); - utf8 = midna_ascii_to_utf8(ascii); + msg_info("\"%s\" ->ascii \"%s\"", bp, ascii); + utf8 = midna_to_utf8(ascii); if (utf8 != 0) { - msg_info("\"%s\" -> \"%s\" -> \"%s\"", + msg_info("\"%s\" ->ascii \"%s\" ->utf8 \"%s\"", bp, ascii, utf8); if (strcmp(utf8, bp) != 0) msg_warn("\"%s\" != \"%s\"", bp, utf8); } } } else { - utf8 = midna_ascii_to_utf8(bp); + ascii = midna_to_ascii(bp); + if (ascii != 0) + msg_info("\"%s\" ->ascii \"%s\"", bp, ascii); + utf8 = midna_to_utf8(bp); if (utf8 != 0) { - msg_info("\"%s\" -> \"%s\"", bp, utf8); - ascii = midna_utf8_to_ascii(utf8); + msg_info("\"%s\" ->utf8 \"%s\"", bp, utf8); + ascii = midna_to_ascii(utf8); if (ascii != 0) { - msg_info("\"%s\" -> \"%s\" -> \"%s\"", + msg_info("\"%s\" ->utf8 \"%s\" ->ascii \"%s\"", bp, utf8, ascii); if (strcmp(ascii, bp) != 0) msg_warn("\"%s\" != \"%s\"", bp, ascii); @@ -237,6 +330,6 @@ int main(int argc, char **argv) exit(0); } -#endif +#endif /* TEST */ -#endif +#endif /* NO_EAI */ diff --git a/postfix/src/util/midna.h b/postfix/src/util/midna.h index 1d2477773..599097832 100644 --- a/postfix/src/util/midna.h +++ b/postfix/src/util/midna.h @@ -14,8 +14,10 @@ /* * External interface. */ -extern const char *midna_utf8_to_ascii(const char *); -extern const char *midna_ascii_to_utf8(const char *); +extern const char *midna_to_ascii(const char *); +extern const char *midna_to_utf8(const char *); +extern const char *midna_suffix_to_ascii(const char *); +extern const char *midna_suffix_to_utf8(const char *); /* LICENSE /* .ad diff --git a/postfix/src/util/midna_test.in b/postfix/src/util/midna_test.in new file mode 100644 index 000000000..b309be965 --- /dev/null +++ b/postfix/src/util/midna_test.in @@ -0,0 +1,14 @@ +# Upper-case greek -> lower-case greek. +Δημοσθένους.example.com +# Upper-case ASCII -> lower-case ASCII. +Hello.example.com +# Invalid domain name ('-' at begin or end). +-bad-.example.com +# Invalid domain name (label > 62 bytes). +abcdef01234567890abcdef01234567890abcdef01234567890abcdef0123456789.example.com +# Valid domain name. +abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com +# Aliases for '.' -> '.'. +x。example.com +x.example.com +x。example.com diff --git a/postfix/src/util/midna_test.ref b/postfix/src/util/midna_test.ref new file mode 100644 index 000000000..1df1eb61e --- /dev/null +++ b/postfix/src/util/midna_test.ref @@ -0,0 +1,71 @@ +./midna: > # Upper-case greek -> lower-case greek. +./midna: > Δημοσθένους.example.com +./midna: ctable_locate: install entry key δημοσθένουσ.example.com +./midna: ctable_locate: install entry key Δημοσθένους.example.com +./midna: "Δημοσθένους.example.com" ->utf8 "δημοσθένουσ.example.com" +./midna: ctable_locate: install entry key Δημοσθένους.example.com +./midna: "Δημοσθένους.example.com" ->ascii "xn--ixanjetild6aev.example.com" +./midna: ctable_locate: move existing entry key δημοσθένουσ.example.com +./midna: ctable_locate: install entry key xn--ixanjetild6aev.example.com +./midna: "Δημοσθένους.example.com" ->ascii "xn--ixanjetild6aev.example.com" ->utf8 "δημοσθένουσ.example.com" +./midna: warning: "Δημοσθένους.example.com" != "δημοσθένουσ.example.com" +./midna: > # Upper-case ASCII -> lower-case ASCII. +./midna: > Hello.example.com +./midna: ctable_locate: install entry key Hello.example.com +./midna: "Hello.example.com" ->ascii "hello.example.com" +./midna: ctable_locate: install entry key hello.example.com +./midna: ctable_locate: install entry key Hello.example.com +./midna: "Hello.example.com" ->utf8 "hello.example.com" +./midna: ctable_locate: leave existing entry key hello.example.com +./midna: "Hello.example.com" ->utf8 "hello.example.com" ->ascii "hello.example.com" +./midna: warning: "Hello.example.com" != "hello.example.com" +./midna: > # Invalid domain name ('-' at begin or end). +./midna: > -bad-.example.com +./midna: warning: midna_to_ascii_create: Problem translating domain "-bad-.example.com" to ASCII form: U_ZERO_ERROR +./midna: ctable_locate: install entry key -bad-.example.com +./midna: warning: midna_to_utf8_create: Problem translating domain "-bad-.example.com" to UTF8 form: U_ZERO_ERROR +./midna: ctable_locate: install entry key -bad-.example.com +./midna: > # Invalid domain name (label > 62 bytes). +./midna: > abcdef01234567890abcdef01234567890abcdef01234567890abcdef0123456789.example.com +./midna: warning: midna_to_ascii_create: Problem translating domain "abcdef01234567890abcdef01234567890abcdef01234567890abcdef0123456789.example.com" to ASCII form: U_ZERO_ERROR +./midna: ctable_locate: install entry key abcdef01234567890abcdef01234567890abcdef01234567890abcdef0123456789.example.com +./midna: ctable_locate: leave existing entry key abcdef01234567890abcdef01234567890abcdef01234567890abcdef0123456789.example.com +./midna: ctable_locate: install entry key abcdef01234567890abcdef01234567890abcdef01234567890abcdef0123456789.example.com +./midna: > # Valid domain name. +./midna: > abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com +./midna: ctable_locate: install entry key abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com +./midna: "abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com" ->ascii "abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com" +./midna: ctable_locate: leave existing entry key abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com +./midna: ctable_locate: install entry key abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com +./midna: "abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com" ->utf8 "abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com" +./midna: ctable_locate: leave existing entry key abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com +./midna: "abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com" ->utf8 "abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com" ->ascii "abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com" +./midna: > # Aliases for '.' -> '.'. +./midna: > x。example.com +./midna: ctable_locate: install entry key x.example.com +./midna: ctable_locate: install entry key x。example.com +./midna: "x。example.com" ->utf8 "x.example.com" +./midna: ctable_locate: install entry key x。example.com +./midna: "x。example.com" ->ascii "x.example.com" +./midna: ctable_locate: move existing entry key x.example.com +./midna: ctable_locate: install entry key x.example.com +./midna: "x。example.com" ->ascii "x.example.com" ->utf8 "x.example.com" +./midna: warning: "x。example.com" != "x.example.com" +./midna: > x.example.com +./midna: ctable_locate: leave existing entry key x.example.com +./midna: ctable_locate: install entry key x.example.com +./midna: "x.example.com" ->utf8 "x.example.com" +./midna: ctable_locate: install entry key x.example.com +./midna: "x.example.com" ->ascii "x.example.com" +./midna: ctable_locate: move existing entry key x.example.com +./midna: "x.example.com" ->ascii "x.example.com" ->utf8 "x.example.com" +./midna: warning: "x.example.com" != "x.example.com" +./midna: > x。example.com +./midna: ctable_locate: move existing entry key x.example.com +./midna: ctable_locate: install entry key x。example.com +./midna: "x。example.com" ->utf8 "x.example.com" +./midna: ctable_locate: install entry key x。example.com +./midna: "x。example.com" ->ascii "x.example.com" +./midna: ctable_locate: move existing entry key x.example.com +./midna: "x。example.com" ->ascii "x.example.com" ->utf8 "x.example.com" +./midna: warning: "x。example.com" != "x.example.com" diff --git a/postfix/src/util/valid_utf8_hostname.c b/postfix/src/util/valid_utf8_hostname.c index f1fac55e5..f42e21f53 100644 --- a/postfix/src/util/valid_utf8_hostname.c +++ b/postfix/src/util/valid_utf8_hostname.c @@ -68,7 +68,7 @@ int valid_utf8_hostname(int enable_utf8, const char *name, int gripe) */ #ifndef NO_EAI if (enable_utf8 && !allascii(name)) { - if ((aname = midna_utf8_to_ascii(name)) == 0) { + if ((aname = midna_to_ascii(name)) == 0) { if (gripe) msg_warn("%s: malformed UTF-8 domain name", myname); return (0);