Courier: courier/doc/draft-varshavchik-verp-smtpext.txt

"Fossies" - the Fresh Open Source Software Archive

Member "courier-1.2.2/courier/doc/draft-varshavchik-verp-smtpext.txt" (25 Aug 2013, 22760 Bytes) of package /linux/misc/courier-1.2.2.tar.bz2:

As a special service "Fossies" has tried to format the requested text file into HTML format (style: standard) with prefixed line numbers. Alternatively you can here view or download the uninterpreted source code file.

1 2 3 4 5 6 7 1. Abstract 8 9 This document describes an extension to the SMTP service [1], called 10 Variable Envelope Return Path (VERP). The VERP extension implements 11 a way of automatically identifying undeliverable mail recipients, 12 even when non-delivery reports originate from mail systems that do 13 not implement delivery status notifications, as specified in [2] and 14 [3]. 15 16 2. Introduction 17 18 All E-mail software can expect to deal with undeliverable mail. [2] 19 and [3] specify a machine-readable format for delivery status 20 notifications (DSNs, or non-delivery reports). DSNs allow 21 undeliverable mail to be handled in a totally automatic fashion, 22 without requiring manual intervention. For example, mailing list 23 managers can automatically identify addresses that are no longer 24 deliverable, and remove them from the mailing list. 25 26 Although [2] and [3] are now widely implemented, there are still many 27 systems that do not use them. This makes it impractical to 28 completely rely on DSNs for automatic mailing list management. 29 Undeliverable addresses accumulate quickly even from a very small 30 percentage of non-DSN systems. This results in a non-trivial amount 31 of manual work to identify undeliverable addresses and purge them 32 from the mailing list. 33 34 Mailing list software began to use VERPs (the acronym stands for 35 Variable Envelope Return Path) after DSNs were found to be 36 impractical for totally automatic mailing list management. VERPs are 37 an alternative way to handle non-delivery notices. The advantage of 38 VERPs is that they can be made to work automatically, even when non- 39 delivery notices are not in the format specified by [2]. 40 41 Unfortunately, VERPs require much more bandwidth and network 42 resources than DSNs because VERPs cannot be used to send one copy of 43 a mailing list message addressed to all the recipients in the same E- 44 mail domain. 45 46 This SMTP service extension allows E-mail software to send a single 47 VERP message to all addresses in the same mail domain, for as long as 48 mail servers, which relay the message, support the VERP SMTP 49 extension. 50 51 The VERP message may be eventually relayed to a mail server that does 52 not support this extension. Separate messages - with variable 53 envelope return paths - will be sent when this happens. 54 55 56 57 58 S. Varshavchik Expires XXX XX, XXXX [Page 1] 59 60 VERP SMTP Extension S. Varshavchik XXX XX, XXXX 61 62 63 So the worst case scenario results in the same situation where 64 traditional VERPs are used right from the start. The best case 65 scenario results in significant savings of network resources and 66 bandwidth, from eliminating hundreds (or more) copies of the same 67 message. 68 69 Essentially, the VERP extension postpones the generation of multiple 70 messages with different return paths as much as possible, until it is 71 absolutely required. 72 73 2.1 VERP overview 74 75 The traditional VERP message encodes the recipient address as a 76 portion of the return address. When undeliverable mail comes back, 77 the mail software decodes the return address (now the recipient 78 address) and obtains the address responsible for the non-delivery 79 notice. 80 81 For example: mail sent by a mailing list manager to the address 82 <john@example.org> carries a return address of <mlist-return- 83 john=example.org@domain.com>. The mailing list software at 84 domain.com handles all mail with the local portion of the address 85 starting with "mlist-return-". If a non-delivery notice is generated 86 because the address is not deliverable, the mailing list software 87 takes the address where the non-delivery report was sent, retrieves 88 the remaining portion of the local address, "john=example.org", and 89 determines that the undeliverable address was <john@example.org>. 90 91 This does not rely on RFC 1894, and will work for all non-delivery 92 notices. 93 94 3. Framework for the VERP SMTP transport extension 95 96 This SMTP transport extension [1] is laid out as follows. 97 98 (1) The name of the SMTP transport extension defined here is 99 Variable Envelope Return Path. 100 101 (2) The EHLO keyword associated with this extension is VERP. 102 103 (3) The VERP EHLO keyword takes no parameters. 104 105 (4) One optional ESMTP keyword VERP is associated with the MAIL 106 FROM command. This parameter takes no values. 107 108 (5) No additional ESMTP verbs are defined by this extension. 109 110 (6) The next section specifies how support for this extension 111 112 113 114 S. Varshavchik Expires XXX XX, XXXX [Page 2] 115 116 VERP SMTP Extension S. Varshavchik XXX XX, XXXX 117 118 119 affects the behavior of a server and client SMTP. 120 121 4. The VERP SMTP extension 122 123 When a VERP keyword is present in the MAIL FROM command, [4], some 124 additional restrictions are imposed on the RFC 822 address [5], 125 specified by that MAIL FROM command, and on all RFC 822 addresses in 126 the subsequent RCPT TO commands that refer to the same message (that 127 is, until the next DATA, RSET, or QUIT command). The term "VERP 128 message" refers to any E-mail message whose MAIL FROM command 129 includes the VERP keyword. The term "VERP-compliant server" refers 130 to any E-mail server that supports the Variable Envelope Return Path 131 SMTP extension. When a VERP keyword is present in the MAIL FROM 132 command: 133 134 (1) The address specified by the MAIL FROM verb MUST contain at 135 least one @ character. 136 137 (2) The address in every RCPT TO verb referring to the same 138 message MUST contain at least one @ character. 139 140 (3) The domain portion of the address in the MAIL FROM and RCPT 141 TO verbs MUST be compliant with the definition of <domain> in 142 [6]. That is, it MUST contain only letters, digits, hyphens, 143 and periods. The domain portion of the address is the one 144 that follows the last @ character. 145 146 4.1 Delivery failures 147 148 When a VERP-compliant server is unable to deliver a VERP message to 149 one or more recipients, the VERP server MUST do one of the following: 150 151 1) Return an RFC 1891 delivery status notification to the return 152 address, or: 153 154 2) Transmit a separate non-delivery notice for each failed 155 recipient. The return address for each non-delivery notice 156 MUST be the address that’s formed by applying the procedure 157 described in section 7 of this document to the return address 158 of the message and the failed recipient’s address. If more 159 than one recipient was undeliverable a separate notice MUST 160 be sent for each undeliverable address. 161 162 5. Final delivery 163 164 Section 4.3.1 of [5] specifies that the mail server performing final 165 delivery of a message will generate a Return-Path: header containing 166 the return address of the message. 167 168 169 170 S. Varshavchik Expires XXX XX, XXXX [Page 3] 171 172 VERP SMTP Extension S. Varshavchik XXX XX, XXXX 173 174 175 This return address MUST be formed by applying the procedure 176 described in section 7 of this document to the return address and the 177 recipient’s address. 178 179 This also applies if the mail server invokes some other external 180 process to handle final delivery, instead of placing the message into 181 the recipient’s mailbox. In all cases, the return address specified 182 by the mail server to any external environment or process MUST be 183 derived by applying the procedure in section 7 to the return address 184 and the recipient’s address. 185 186 6. Relaying 187 188 When a VERP-compliant server determines that a recipient of a VERP 189 message is not a local mailbox, and the message must be relayed to 190 another server, the VERP-compliant server MUST: 191 192 (1) If the VERP-compliant server’s local policies require the 193 return and/or recipient addresses are to be rewritten, the 194 VERP-compliant server MUST make sure that delivery notices 195 MUST NOT be sent to a rewritten return address, or reflect a 196 rewritten recipient address, unless separate arrangements are 197 made with the sender of the VERP message. This is because 198 the sender expects to be able to resolve non-delivery notices 199 to some recipient address it knows about (irrespective 200 whether the addresses are resolved from a VERP, or from a 201 delivery status notification in the format specified by [2]). 202 When local policies of the relaying server require the 203 recipient or the sender address to be rewritten, the sender 204 will obviously be not aware of this rewriting or expansion. 205 If a subsequent delivery failure results in a non-delivery 206 notice being sent to a VERP containing the rewritten address, 207 the sender will not be able to associate the address encoded 208 in the VERP with any recipient address it has on file. 209 210 A suggested way to handle this situation is to set specify a 211 delivery status notification of "NEVER" - as specified by [3] 212 - for any recipient address generated by local aliasing or 213 rewriting, and treat the original address as being either 214 delivered or relayed, resulting in a "delivered" or "relayed" 215 delivery status notification if the original recipient 216 address specified a "SUCCESS" notification. 217 218 Note that this clause does not prohibit the relaying server 219 to have an established agreement with the sender to act as a 220 mailing list exploder, as long as this agreement is handled 221 in a transparent way. For example, the relaying server can 222 set its own return address on all VERP messages it exploded, 223 224 225 226 S. Varshavchik Expires XXX XX, XXXX [Page 4] 227 228 VERP SMTP Extension S. Varshavchik XXX XX, XXXX 229 230 231 and handle non-delivery notices all by itself. 232 233 (2) If the VERP-compliant server determines that the remote 234 server is also a VERP compliant server, the VERP keyword MUST 235 be included in the MAIL FROM command used to relay the VERP 236 message to the remote server. 237 238 (3) If the remote server is not a VERP compliant server, The VERP 239 compliant server SHOULD send a separate copy of the message 240 for every recipient. The return address of each copy of the 241 message MUST be formed by applying the procedure described in 242 section 7 of this document to the original return address, 243 and the address of each individual recipient. Although the 244 message SHOULD NOT be returned as undeliverable, if it is 245 then the rules defined in section 4.1 MUST be applied. 246 247 These rules also apply if the SMTP-compliant server 248 determines that the VERP message must be forwarded via some 249 other protocol to a non-SMTP gateway, unless the non-SMTP 250 protocol has equivalent features that are completely 251 identical in function to Variable Envelope Return Path SMTP 252 service extension (including any translations of E-mail 253 addresses to and from the non-RFC822 format). 254 255 7. Variable envelope return path encoding 256 257 This encoding method starts with a return address and one recipient 258 address. As mentioned previously, both addresses MUST be valid 259 RFC822 addresses, [5], and MUST contain at least one @ character. 260 The portion of each address following the last @ character MUST be 261 compliant with [6], or must be an explict network address literal, 262 for example: john43@[192.68.0.4]. 263 264 Let "sdomain" represent the portion of the return address that 265 follows the last @ character. 266 267 Let "slocal" represent the portion of the return address that 268 precedes the last @ character. 269 270 Let "rdomain" represent the portion of the recipient address that 271 follows the last @ character. 272 273 Let "rlocal" represent the portion of the recipient address that 274 precedes the last @ character. 275 276 To encode the recipient address within the envelope sender address, 277 create an address of the following form: 278 279 280 281 282 S. Varshavchik Expires XXX XX, XXXX [Page 5] 283 284 VERP SMTP Extension S. Varshavchik XXX XX, XXXX 285 286 287 slocal-encodedrlocal=encodedrdomain@sdomain 288 289 Where "encodedrlocal" and "encodedrdomain" are formed by taking 290 rlocal and rdomain, respectively, and encoding each one as follows: 291 292 1) Each @, :, %, !, -, [, ], and + character in rlocal is 293 replaced by a single ’+’ character followed by two uppercase 294 hexadecimal characters whose value is the ASCII code of the 295 replaced character. 296 297 Taking into account poorly-written mail relays that ignore 298 case-sensitivity, both uppercase and lowercase hexadecimal 299 characters SHOULD be recognized when decoding envelope return 300 paths. 301 302 2) All other characters are unchanged. Other characters MAY, 303 but SHOULD NOT be also encoded in the same fashion. 304 305 This can be represented using BNF as follows: 306 307 encodedverp: slocal "-" encodedrlocal "=" encodedrdomain "@" sdomain 308 309 encodedrlocal: * (char-literal / char-encoded ) 310 311 encodedrdomain: * (char-literal / char-encoded ) 312 313 char-literal: any character valid in an RFC821 address [4], 314 except @, :, %, !, [, ], -, and + 315 316 char-encoded: "+" hexdigit hexdigit 317 318 hexdigit: ("0" / "1" / "2" / "3" / "4" / "5" / "6" / "7" / "8" / 319 "9" / "A" / "B" / "C" / "D" / "E" / "F" / 320 "a" / "b" / "c" / "d" / "e" / "f") 321 322 8. Variable envelope return path decoding 323 324 Non-delivery notices for VERP messages will be sent to either the 325 original address, <slocal@sdomain>, or to the VERP-encoded address, 326 <slocal-encodedrlocal=encodedrdomain@sdomain>. 327 328 Messages sent to <slocal@sdomain> will be RFC 1891-compliant delivery 329 status notifications. These messages will be machine-readable, and 330 the mail software will be able to identify failed addresses from the 331 RFC 1891 delivery report. 332 333 Non-delivery notices will also be sent to the VERP-encoded address, 334 and the mail software will be able to reconstruct the failed address 335 336 337 338 S. Varshavchik Expires XXX XX, XXXX [Page 6] 339 340 VERP SMTP Extension S. Varshavchik XXX XX, XXXX 341 342 343 from the VERP-encoded address by simply reversing the steps used in 344 encoding: 345 346 1) Extract encodedrlocal and rdomain from the recipient address. 347 There will be at least one = character in the encoded portion 348 of the return address. encodedrlocal is everything up to the 349 last = character. Everything following the last = character 350 is encodedrdomain. 351 352 2) Replacing all occurrences of "+" followed by two hexadecimal 353 digits in encodedrlocal and encodedrdomain with the 354 equivalent ASCII character. 355 356 3) Using the decoded rlocal, @, then rdomain. 357 358 9. Examples 359 360 Suppose that a VERP-compliant server named "example.com" receives a 361 message via the following SMTP conversation (for brevity, non- 362 relevant headers have been omitted): 363 364 220 example.com ESMTP 365 EHLO domain.com 366 250-example.com ESMTP 367 250-SIZE 368 250-DSN 369 250-VERP 370 250 HELP 371 MAIL FROM:<itny-out@domain.com> VERP SIZE=100 372 250 Ok 373 RCPT TO:<alex@example.com> 374 250 Ok 375 RCPT TO:<node42!ann@old.example.com> 376 250 Ok 377 RCPT TO:<tom@old.example.com> 378 250 Ok 379 RCPT TO:<lisa@new.example.com> 380 250 Ok 381 RCPT TO:<dave+priority@new.example.com> 382 250 Ok 383 DATA 384 354 Ok 385 From: "John" <john@domain.com> 386 Date: Thu, 16 Jan 1997 14:49:31 -0500 (EST) 387 Subject: Meeting canceled. 388 389 Today’s 2pm meeting has been rescheduled for tomorrow, 9am, due 390 to a scheduling conflict. 391 392 393 394 S. Varshavchik Expires XXX XX, XXXX [Page 7] 395 396 VERP SMTP Extension S. Varshavchik XXX XX, XXXX 397 398 399 . 400 401 The message is delivered to the local mailbox for <alex@example.com>. 402 The message looks like this: 403 404 Return-Path: <itny-out-alex=example.com@domain.com> 405 From: "John" <john@domain.com> 406 Date: Thu, 16 Jan 1997 14:49:31 -0500 (EST) 407 Subject: Meeting canceled. 408 409 Today’s 2pm meeting has been rescheduled for tomorrow, 9am, due 410 to a scheduling conflict. 411 412 The VERP-compliant server at example.com connects to the mail server 413 for old.example.com. old.example.com does not support the Variable 414 Envelope Return Path extension. Therefore, old.example.com receives 415 two messages. The SMTP conversation for the first message is as 416 follows: 417 418 220 old.example.com ESMTP 419 EHLO example.com 420 250-old.example.com ESMTP 421 250-SIZE 422 250-DSN 423 250 HELP 424 MAIL FROM:<itny-out-node42+21ann=old.example.com@domain.com> 425 250 Ok 426 RCPT TO:<node42!ann@old.example.com> 427 250 Ok 428 DATA 429 354 Ok 430 From: "John" <john@domain.com> 431 Date: Thu, 16 Jan 1997 14:49:31 -0500 (EST) 432 Subject: Meeting canceled. 433 434 Today’s 2pm meeting has been rescheduled for tomorrow, 9am, due 435 to a scheduling conflict. 436 . 437 438 The SMTP conversation for the second message is as follows: 439 440 MAIL FROM:<itny-out-tom=old.example.com@domain.com> 441 250 Ok 442 RCPT TO:<tom@old.example.com> 443 250 Ok 444 DATA 445 354 Ok 446 From: "John" <john@domain.com> 447 448 449 450 S. Varshavchik Expires XXX XX, XXXX [Page 8] 451 452 VERP SMTP Extension S. Varshavchik XXX XX, XXXX 453 454 455 Date: Thu, 16 Jan 1997 14:49:31 -0500 (EST) 456 Subject: Meeting canceled. 457 458 Today’s 2pm meeting has been rescheduled for tomorrow, 9am, due 459 to a scheduling conflict. 460 . 461 462 example.com connects to new.example.com and determines that 463 new.example.com runs a modern ESMTP server that supports the VERP 464 keyword. The SMTP conversation then goes like this: 465 466 220 new.example.com ESMTP 467 EHLO example.com 468 250-new.example.com ESMTP 469 250-SIZE 470 250-DSN 471 250-VERP 472 250 HELP 473 MAIL FROM:<itny-out@domain.com> VERP SIZE=100 474 250 Ok 475 RCPT TO:<lisa@new.example.com> 476 250 Ok 477 RCPT TO:<dave+priority@new.example.com> 478 250 Ok 479 DATA 480 354 Ok 481 From: "John" <john@domain.com> 482 Date: Thu, 16 Jan 1997 14:49:31 -0500 (EST) 483 Subject: Meeting canceled. 484 485 Today’s 2pm meeting has been rescheduled for tomorrow, 9am, due 486 to a scheduling conflict. 487 . 488 489 10. Security concerns 490 491 All the usual security considerations applicable to SMTP are also 492 applicable to this extension. Relay of VERP messages to non-VERP 493 servers requires a single message with many recipients to be exploded 494 into many messages with one recipient. In all cases, however, there 495 will never be any additional overhead beyond the resources that are 496 required when VERPs are manually implemented by the mail sender, 497 instead of the VERP SMTP extension. 498 499 Mail systems which support the VERP extension SHOULD have adequate 500 security measures, including blocks against unauthorized access and 501 relaying. 502 503 504 505 506 S. Varshavchik Expires XXX XX, XXXX [Page 9] 507 508 VERP SMTP Extension S. Varshavchik XXX XX, XXXX 509 510 511 10.1 Vacation programs, and other autoresponders 512 513 "Vacation" type autoresponders are often used in practice. A 514 vacation autoresponder is a program that automatically replies to 515 every message, informing the sender that the recipient is on 516 vacation, or is generally unavailable at this time. 517 518 Vacation autoresponders MUST NOT generate autoresponses to mailing 519 list messages, but people often forget to do set them up to do so. 520 Because autoresponses are sent to the same address that’s used to 521 receive non-delivery reports, malfunctioning autoresponders result in 522 the recipient being removed from mailing lists. 523 524 Advanced autoresponders send automatic replies in the format 525 specified by [2], as a "delayed" notification. DSN-aware software 526 will not remove addresses from mailing lists due to delayed 527 notifications. 528 529 Section 5 of this document specifies that the mail server MUST 530 replace the original return address with a VERP-modified address when 531 delivering the message to a mailbox or an external process. 532 533 Therefore it is possible that RFC 1891 reports may also be sent to a 534 VERP-encoded address, as specified by sections 5 and 7 of this 535 document. Mail software SHOULD ignore any RFC 1891 "delayed" or 536 "success" reports that sent to a VERP-encoded address. If it is a 537 "failed" report, note that the VERP address will be more reliable 538 than the address specified in the report itself. 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 S. Varshavchik Expires XXX XX, XXXX [Page 10] 563 564 VERP SMTP Extension S. Varshavchik XXX XX, XXXX 565 566 567 11. References 568 569 [1] Klensin, J., Freed, N., Rose, M., Stefferud, E., Crocker, D. 570 "SMTP Service Extensions", RFC 1425, United Nations 571 University, Innosoft International, Inc., Dover Beach 572 Consulting, Inc., Network Management Associates, Inc., The 573 Branch Office, February 1993 574 575 [2] Moore, K., and G. Vaudreuil, "An Extensible Message Format 576 for Delivery Status Notifications", RFC 1894, University of 577 Tennessee, Octel Network Services, January 1996. 578 579 [3] Moore, K. "SMTP Service Extension for Delivery Status 580 Notifications", RFC 1891, University of Tennessee, January 581 1996. 582 583 [4] Postel, J., "Simple Mail Transfer Protocol", STD 10, RFC 821, 584 USC/Information Sciences Institute, August 1982. 585 586 [5] Crocker, D., "Standard for the Format of ARPA Internet Text 587 Messages", STD 11, RFC 822, UDEL, August 1982. 588 589 [6] Mockapetris, P., "Domain Names - Implementation and 590 Specification", RFC 1035, ISI, November 1987 591 592 12. Author’s address 593 594 Sam Varshavchik 595 Double Precision, Inc. 596 402 Main Street Suite 100-241 597 Metuchen, NJ 08840 598 <mrsam@courier-mta.com> 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 S. Varshavchik Expires XXX XX, XXXX [Page 11]