Mailfromd NEWS -- history of user-visible changes. 2008-02-10 Copyright (C) 2005, 2006, 2007, 2008 Sergey Poznyakoff See the end of file for copying conditions. Please send mailfromd bug reports to Version 4.3, 2008-02-10 * write built-in The `write' built-in function takes an optional third argument, specifying the number of bytes to write. This form is particualry useful in `body' handler for writing $1, because it is not null- terminated, e.g.: prog body do write(fd, $1, $2) ... done * sieve New built-in function `sieve' provides an interface to Sieve interpreter. * Restore previous meaning of --enable-syslog-async. Unless this option is given to ./configure, asynchronous syslog implementation will not be compiled. * Bugfixes: ** Fix compilation on Sun. ** Fix header deletion (delete action). ** Variable shadowing was broken if a rehash happened between vardcl and forget_autos. Version 4.2, 2007-10-23 * Licensed under the GPLv3 * New command line options --syslog-async and --no-syslog-async These options allow to select the implementation of syslog to use at run time. * RFC 2821 compatibility The sender verification engine used to send whitespace character between `MAIL FROM:' and `RCPT TO:' commands and their argument. This violated RFC 2821, which requires the argument to follow the command without any intermediate whitespace. It is fixed in this release. * Sample threshold for `rate' function. The `rate' function takes an optional third argument allowing to specify the minimum number of received mails needed to obtain the sending rate value. The default is 2, which is probably too conservative. The following example raises it to 10: if rate($f "-" ${client_addr}, interval("1 hour"), 10) > %maxrate ... fi * mtasim is improved In particular, it accepts MAIL FROM: and RCPT TO: without extra space after the colon, as required by RFC. The milter interface is also improved. * The `resolve' function ignores TXT records. In other words, resolve and primitive_resolve are guaranteed to return either an A or a PTR record. Version 4.1, 2007-06-11 * National Language Support. The program includes National Language Support. Polish and Ukrainian translations are available. * NLS Functions NLS functions allow to localize your filter scripts for a particular language. The following functions are implemented: bindtextdomain, dgettext, dngettext, textdomain, gettext, ngettext. In addition, macros _() and N_() are also provided. * GNU Emacs MFL Mode This release comes with the file `mfl-mode.el', providing MFL mode for GNU Emacs. This mode facilitates editing MFL source files. By default, the new mode is installed whenever configure determines the presense of GNU Emacs on your machine. See the documentation, node `Using MFL Mode' for the detailed discussion of this mode including customization information. * Input files are preprocessed before compilation. The default preprocessor is M4, but this can be changed (or disabled) at configuration time (see `DEFAULT_PREPROCESSOR' variable and `--with-preprocessor' command line option). * New atom $# Returns the number of the arguments passed to the function. * New atom @parm Returns the position of parameter `parm' in the function argument list. It can be used, for example, to check whether an optional argument value is passed to the function, e.g.: func foo(string x; number n) do if $# > @n /* `n' is passed */ ... The default preprocessor setup script provides a macro `define' designed to be used for this purpose: func foo(string x; number n) do if defined(n) /* `n' is passed */ ... * sprintf The built-in function `sprintf' is available with the same semantics as its C counterpart. * Discontinued support for deprecated features: ** `&code' form to specify an exception code is discontinued. ** pragma options retry, io-retry, and connect-retry * Bugfixes: ** Built-in listen ignored optional second argument. ** Debug specification incorrectly gave preference to the global level over the source level. This is fixed, so that `--debug=40,dns=10' means level 10 for calls from `dns.c', and level 40 for all the rest. Version 4.0, 2007-05-12 Note for users of 3.1.x: see also the notes for previous alpha (3.1.9x) versions. * SIGHUP handling SIGHUP instructs `mailfromd' to restart itself. * rc.mailfromd reload The `reload' option given to `rc.mailfromd' instructs it to send SIGHUP to the running instance of the program. * mtasim The `mtasim' utility is an MTA simulator for testing and debugging mailfromd filter scripts. It supports stdio (-bs) and daemon (-bd) modes, has GNU readline support and `expect' facility, which makes it useful in automated test cases. See the documentation, chapter `mtasim'. * `begin'/`end' handlers The `begin' and `end' special handlers may be used to supply startup and cleanup code for the filter program. The `begin' special handler is executed once for each SMTP session, after the connection has been established but before the first milter handler has been called. Similarly, an `end' handler is executed exactly once, after the connection has been closed. Neither of handlers takes any arguments. See the documentation, section `begin/end'. * Cache control Use function `db_set_active' to enable or disable given cache database. E.g. # Disable DNS cache: db_set_active("dns", 0) # Enable it back again: db_set_active("dns", 1) Similarly, the function `db_get_active' returns a number indicating whether the given cache database is used or not. ** Bugfixes * Fix a long-standing bug in parsing the --predict option argument. Is there anybody using this option at all? Version 3.1.91, 2007-04-23 * Non-blocking syslog This version is shipped with non-blocking syslog implementation by Simon Kelley. You may wish to enable it if you noticed that the number of mailfromd processes grows uncontrollably and the processes are hung for prolonged amounts of time. Usually this indicates that the daemon blocks in syslog() calls. Read the description of `--enable-syslog-async' option in chapter `Building' for the detailed discussion of this (try `info -f doc/mailfromd.info --index-search syslog-async'). * SPF support The function check_host() tests the SPF record for the given identity/host name. The syntax is: number check_host(string ip, string domain, string sender, string helo) See the documentation, node `SPF functions' for the detailed description of this function and the related functions and variables. * next and pass Use `pass' instead of `next'. The `next' keyword has changed its semantics: it is now used to resume the next iteration of the enclosing loop statement (see below). For compatibility with the previous versions, its use outside of a loop statement is still allowed, but a warning is issued. You are encouraged to replace all occurrences of `next' in your configuration scripts with `pass'. * Loop The loop statement is implemented. Its syntax is: loop [name] [for ,] [while ,] [] do ... done [while ] See the documentation, section `Loop Statements'. * break and next The `break' statement exits from the enclosing loop. The `next' statement resumes the next iteration of the enclosing loop statement. Both statements take an optional argument specifying the identifier (name) of the loop to break from (or continue), this allows to build complex iterations consisting of nested loops. For example, in this code (line numbers added for clarity): 1 loop outer for set i 1, while %i < %N 2 do 3 ... 4 loop for set j 1, while %j < %i 5 do 6 if foo(%j) 7 break outer 8 fi 9 done 10 done 11 accept if the call to `foo' in line 6 returns true, the control is immediately passed to `accept' in line 11. * Resizable stack The runtime stack of the MFL grows automatically as the need arises. Thus, `#pragma stacksize' sets the initial size of the stack, and the `Out of stack space' error, which was common in the previous versions, now can occur only if there is no more virtual memory left. Whenever the stack gets expanded, mailfromd issues a warning message to the logs, notifying of the new stack size, e.g.: warning: stack segment expanded, new size=8192 You can use these messages to adjust your stack size configuration settings. * Runtime stack traces New command line option --stack-trace enables dumping stack traces on runtime errors. This might help localize the source of the error. The trace looks like: mailfromd: RUNTIME ERROR near ../mflib/match_cidr.mf:30: invalid CIDR (boo%) mailfromd: Stack trace: mailfromd: 0077: test.mf:7: match_cidr mailfromd: 0096: test.mf:13: bar mailfromd: 0110: test.mf:18: foo mailfromd: Stack trace finishes mailfromd: Execution of the configuration program was not finished Each trace line describes one stack frame, the lines appear in the order of most recently called to least recently called. Each frame consists of: 1. Value of program counter at the time of its execution 2. Source code location, if available 3. Name of the function called The same output can be obtained by calling function stack_trace() in your filter program. See the documentation, section `Runtime Errors', for the detailed description and examples. * connect handler Connect handler is implemented. * envfrom and envrcpt Both handlers take an additional second argument, containing the rest of the SMTP command line. * New functions - string db_name(string fmt) Return full file name of the database file corresponding to format `fmt' - number db_expire_interval(string fmt) Return the expiration period for db format `fmt' - string getmx(string domain [, number resolve]) Returns a whitespace-separated list of MX names (if `resolve' is not given or is 0) or MX IP addresses (if `resolve'==1) for `domain'. If `domain' has no MX records, empty string is returned. If the DNS query fails, `getmx' raises an appropriate exception. This interface differs from that of version 3.1.4 in that the calls to getmx(domain) and getmx(domain,1) can return different number of entries (see the docs for an example). * #pragma regex stack The `#pragma regex' statement can keep a stack of regex flags. The stack is maintained using `push' and `pop' commands. The statement #pragma regex push [options] saves current regex flags on stack and then optionally modifies them as requested by options. The statement #pragma regex pop [options] does the opposite: restores the current regex flags from the top of stack and applies [options] to it. This statement is useful in include files to avoid disturbing user regex settings. E.g.: #pragma regex push +extended +icase . . . #pragma regex pop * Optional arguments in user-defined functions User-defined functions can take optional arguments. In a declaration, optional arguments are separated from the mandatory ones by a semicolon. For example: func foo(number a, number b; string c) This function is declared with two mandatory arguments (a and b), and an optional one (c). Subsequently it can be invoked either as foo(x, y, z) or foo(x, y) When invoking such functions, any missing arguments are replaced with default values: - 0 for numeric arguments - "" for string arguments Thus, continuing our previous example, the invocation `foo(x, y)' is equivalent to `foo(x, y, "")'. * New statement #include_once This statement works exactly like `#include' except that it keeps track of the included files. If the requested file has already been included, `#include_once' returns silently, while `#include' issues an error message. * New statement #require Requires use of the named module, e.g.: #require dns See the documentation, section `Modules', for the description of MFL module system. * Internet address manipulation functions - number ntohl (number N) - number htonl (number N) - number ntohs (number N) - number htons (number N) - number inet_aton (string S) - string inet_ntoa (number N) - number len_to_netmask (number N) - number netmask_to_len (number N) * DNS functions DNS functions are reimplemented in two layers: 1. Primitive calls: - string primitive_hostname (string IP) - string primitive_resolve (string S [, string DOMAIN]) - number primitive_hasmx (string DOMAIN) - number primitive_ismx (string DOMAIN, string IP) These functions throw an exception if the requested lookup operation fails. 2. Traditional calls: - string hostname (string IP) - string resolve (string S [, string DOMAIN]) - number hasmx (string DOMAIN) - number ismx (string DOMAIN, string IP) These are implemented in MFL and work exactly as their predecessors in 3.1.x branch. To use the traditional calls, add the following statement at the beginning of your script file: #require dns (see the documentatio, section `Modules' for the description of #require statement) * Function `match_cidr' This function has been reimplemented in MFL. To use it, add `#require match_cidr' at the top of your script source (see the documentatio, section `Modules' for the description of #require statement) * Catch arguments Catch takes two positional arguments: $1 gives the exception code, $2 is the diagnostic string, explaining what happened in detail. * New statement `throw' The `throw' statement throws the given exception. For example: throw invcidr "invalid CIDR (%cidr)" * New command line option -v (--variable) allows to alter initial value of a global variable, e.g: mailfromd -v ehlo_domain=mydomain.org The old way of assigning values to the globals from the command line (by prefixing an assignment with a percent sign) is discontinued. * Pragmatic options `mailfrom' and `ehlo' Both options create ambiguities in the language and are therefore deprecated. They are still understood but a deprecation warning is issued when the parser sees them. To update your scripts: Change #pragma option mailfrom @var{value} to set mailfrom_address @var{value} Change #pragma option ehlo @var{value} to set ehlo_domain @var{value} * Code generation redone to decrease memory requirements and make compiled code self-sufficient. * New statement `const' defines a named constant, e.g.: const n 10 const s "String" if $f != s ... fi In program text, constants are referred to by their name. In strings, they are referred to using variable syntax (e.g. "%s"). * It is allowed to initialize variables in declarations. For example, instead of number x set x 1 you can write number x 1 * Built-in macro __statedir__ New built-in macro `__statedir__' expands to the name of the default program state directory. * Changing state directory at run-time It is possible using `--state-directory' command line option or `#pragma option state-directory' statement. * Bugfixes ** Fix incorrect packet length calculation in connect Milter handler. The bug manifested itself with the following log messages: - In mailfromd log: MailfromFilter: shan_connect: wrong length - In Sendmail log: milter_read(mailfrom): cmd read returned 0, expecting 5 Milter(mailfrom): to error state ** Fix coredumps on printing void returns with --dump-tree ** Fix coredumps on optimizing conditionals like if 0 do_something fi ** Fix function context checks. Previous versions bailed out on using context-limited built-ins (like `sa' and `clamav') in functions. It is fixed. The context limit of the built-in propagates to the function it is used in, that is defining func sa_wrapper(string url) do if sa(%url, 3) discard fi done makes function `sa_wrapper' limited for use in `prog eom' only. ** Fix passing of string handler arguments between handlers, as in prog helo do set x $1 done prog envfrom do if %x = "dom.ain" ... (from 3.1.3) ** If, during sender verification, the remote server replies with 4xx to MAIL FROM command, do not try next sender address, but tempfail immediately. Version 3.1.90, 2006-12-13 * `==' can be used as well as `=' to test for equality This is a bit of syntactic sugar for seasoned C programmers, who seem to type == instinctively (like the author does). * resolve takes an optional second argument The argument specifies the domain. For example, the following calls are equivalent: resolve("puszcza", "gnu.org.ua") = resolve("puszcza.gnu.org.ua") resolve("22.0.120.213", "in-addr.arpa") = hostname("213.130.0.22") * listens takes an optional second argument The second argument specifies the port to use instead of the default 25. * New functions - message_header_decode(string TEXT, [string CS]) The TEXT must be encoded as per RFC 2047. The function decodes it and returns the resulting string. Optional argument CS specifies the character set for the output string. Default is UTF-8. - message_header_encode(string TEXT, [string ENC, string CS]) Encodes TEXT according to RFC 2047. Optional arguments: ARG Meaning Default value ------+------------------+------------------- ENC Encoding quoted-printable CS Character set UTF-8 Valid values for ENC are quoted-printable, Q, base64, B - unfold (string TEXT) Unfold TEXT as defined in RFC 2822, section 2.2.3. Return unfolded string. * Default logging facility is LOG_MAIL * While checking sender validity, issue RSET if the previous MAIL FROM returned 4xx. Bugfixes: * `Make install' creates state directory if it does not exist * `clamav' function could cause coredumps when using socket ports. * `resolve' function altered its argument if it was a CNAME. * A typo in gram.y prevented some correct `matches' conditions from being compiled. In particular, strip_domain_part.mf triggered this error. Version 3.1, 2006-12-07 * Incompatible changes. For detailed instructions on how to upgrade from version 3.0, please see http://mailfromd.software.gnu.org.ua/upgrade.html 1. The package refuses to compile without DBM 2. The command line option --config-file (-c) is no longer supported. To use an alternative filter script, give its name as an argument in the command line, e.g. mailfromd my-script.rc For backward compatibility, the invocation `mailfromd --config-file my-script.rc' still works but produces a warning message. The semantics of `-c' will change in the next release. 3. The function `dbmap' takes an optional third argument. If it is 1, then the length of the lookup key will include the terminating null character. In previous versions dbmap always counted the terminating null character in the key length. So, you should add the non-zero third argument to the calls to dbmap to preserve their functionality. 4. Variables, implicitly declared within a function, are automatic. Previous versions created them as global. * Language changes ** Hex and octal constants Usual C notation (0xNNN for hex and 0NNN for octal) is accepted. ** Bitwise operators: &, |, ^ (logical and, or, xor) and ~ (twos-complement) ** Search path for include files The `#include' statement handles its argument the same way C preprocessor does: the argument is searched in the include file path, if enclosed in angle brackets (<>), and in the current working directory first and then in the include file path, if it is enclosed in double quotes. The default include file path is /usr/share/mailfromd/include:/usr/local/share/mailfromd/include plus $prefix/share/mailfromd/include if $prefix is not `/usr' or `/usr/local'. The command line option -I (--include) adds the named directory in front of the default include path. ** Code optimization Parse tree is optimized before code generation. This can be controlled using -Olevel option, where `level' is the optimization level. Currently implemented levels are 0 (no optimization) and 1 (full optimization), which is the default. ** All variables are now strongly typed. The declaration of the variable has the form: `TYPE NAME', where TYPE is one of `string' or `number', and NAME is the variable name. For compatibility with the previous versions, the declaration is optional. If it is absent, the first assignment to the variable defines its type. Subsequent assignments will implicitly cast the value being assigned to the type of the variable. ** New style of function declarations. Named parameters. Functions should be defined as: func NAME (PARAM-LIST) returns TYPE where TYPE is as described in the previous paragraph and PARAM-LIST is a comma-separated list of parameter declarations in the form TYPE NAME. Consequently, instead of the positional notation, parameters can be referenced by their names: func sum(number a, number b) returns number do return %a + %b done Within the function body, the named parameters can be handled the same way as other variables, in particular they can be assigned new values using `set' instruction. For compatibility with the previous version, old type of function declarations is supported as well. ** Automatic variables Automatic variables are defined within a function or handler. Their scope of visibility ends with the terminating `done' statement. Automatic variables are declared and referenced the same way as global ones. To declare an automatic variable, use `TYPE NAME' notation. Variable declarations can be intermixed with executable statements. The following example defines two automatic variables for the function `foo': func foo() do number a string s ... If a variable is declared implicitly within a function or handler, it is declared automatic. See the documentation for the detailed description and examples. ** New functions: *** I/O functions: open, close, write, getline See http://mailfromd.software.gnu.org.ua/manual/bi/io.html *** Time functions: time, strftime See http://mailfromd.software.gnu.org.ua/manual/bi/system.html *** System functions: system See http://mailfromd.software.gnu.org.ua/manual/bi/system.html *** DBM functions: dbput, dbdel See http://mailfromd.software.gnu.org.ua/manual/bi/db.html *** String functions: substr, index, rindex See http://mailfromd.software.gnu.org.ua/manual/bi/string.html *** Debugging functions: debug, cancel_debug, program_trace, cancel_program_trace See http://mailfromd.software.gnu.org.ua/manual/bi/debug.html *** Mail sending functions: send_mail, send_text, send_dsn See http://mailfromd.software.gnu.org.ua/manual/bi/mail.html ** The legacy function numrcpt() has been withdrawn Use %rcpt_count instead. ** Built-in macros Built-in macros have names beginning and ending with double underscore. As their name implies, the macros are expanded to constant values. The following built-in macros are defined: 1. __file__ expands to the name of the current source file 2. __line__ expands to the number of line in the current source file 3. __function__ expands to the name of the current lexical context, i.e. the function or handler name. 4. __package__ expands to the string containing package name ("mailfromd") 5. __version__ expands to the textual representation of the program version (e.g. "3.0.90") 6. __major__ expands to the major version number 7. __minor__ expands to the minor version number 8. __patch__ expands to the version patch level, or 0 if it is not defined. Built-in macros can be used in variable context. For example, to use them within a string or here-document, prepend them with % as if they were regular variables, e.g.: echo "%__file__:%__line__: Checkpoint" * The envfrom and envrcpt handlers print entire argument array in the debugging output. * New DNS caching scheme. All DNS lookups are cached on global basis, as opposed to the per-session basis in previous versions. The cache is stored in the DBM database `dns'. It can be listed and otherwise operated upon using usual mailfromd commands. If a lookup gives a positive result, the TTL from the DNS record is used as the record expiration interval. For negative lookups, the default interval of 3600 seconds is used. It can be altered by the following pragmatic comment: #pragma database dns negative-expire-interval N * New command line option --xref Produces a cross-reference listing of global variables. * Fuller SMTP timeout control In order to more fully control SMTP transactions, new timeout value is introduced: initial-response-timeout. This is the maximum time to wait for the remote to issue the initial SMTP response. This value is especially useful for dealing with the servers that impose a delay before the initial reply (most notably "CommuniGate Pro" ones"). The default value is 30 seconds which should be enough for most normal servers. See the documentation, node "SMTP Timeouts" for the detailed discussion. * No more `retry' options. The `retry' options and pragmas have been removed. The new timeout control scheme warrants that the polling will take at most the given interval of time. In particular, that affects: ** Command line option `--retry' ** Pragma options io-retry and connect-retry * Bugfixes ** Switch statements without the default branch produced incorrect code (the very first branch was used as the default one). This is fixed. ** Fix handling of escape sequences at the beginning of a string and before the beginning of an interpreted sequence within the string. ** Fix the declarations of the built-in functions `toupper' and `tolower'. ** Fix storing the macro values obtained from Sendmail ** Collect zombie subprocesses as soon as possible ** Fix arithmetical expression syntax in rc.mailfromd ** Fix multiple from address handling ** Fix race condition when using GDBM Version 3.0, 2006-11-05 * The mailfromd binary is now installed in ${prefix}/sbin. Please, update your scripts. You are encouraged to update the startup script (run `cp etc/rc.mailfromd /wherever-your-startup-lives'), since the new version contains lots of enhancements (see below). * The package no longer uses libmilter. * Several `from' addresses can be specified both with polling functions and in `#pragma option mailfrom' statement. In this case the probing will try each address until either the remote party accepts it or the list of addresses is exhausted, whichever happens first. This can help if a remote host is picky about sender addresses. * After discussions with Jan, the final part of the standard poll method has been redone. Now the last-resort poll (i.e. querying the domain part of the sender email, treated as an MX) is done only if the domain has no MX records. * New option --dump-macros shows all Sendmail macros used in the configuration file, by milter states. It is useful to create Sendmail `Milter.macros.*' (confMILTER_MACROS_*) statements. * rc.mailfromd stript two new options: - rc.mailfromd configtest [FILE] Checks configuration file syntax. If FILE is not given, the default one is assumed. - rc.mailfromd macros [-c] [FILE] Generate milter export statements for Sendmail configuration files. Optional FILE specifies alternative mailfromd configuration file. By default, `.mc' statements are generated. Specifying `-c' option instructs the script to create `.cf' statements instead. * New pragmatic options `connect-retry' and `connect-timeout' set retry count and timeout values for initial connections. The corresponding values for I/O operations are set using `io-retry' and `io-timeout' options. The pragmatic options `retry' and `timeout' are retained for backward compatibility. They are synonymous to their `io-' counterparts. The default values are: #pragma option connect-retry 1 #pragma option connect-timeout 30 #pragma option io-retry 3 #pragma option io-timeout 3 * New function `sa' checks the message for spam via SpamAssasin spamd interface. It supplies additional data via the global variables sa_score, sa_threshold, and sa_keywords. * New function `clamav' checks the message for viruses via ClamAV daemon interface. Additional data (the virus name, if found) is stored in the global variable clamav_virus_name. * New function `ismx' returns true if the IP address or hostname given by its second argument is one of the MX records of the domain name given by the first argument. * New variables `last_poll_host', `last_poll_send', `last_poll_recv' contain the host name of the lastly polled host, the command sent and the first line of the reply received. The variables are set by polling functions. * New variable `cache_used' is set to true if cache data were used instead of polling, and to false otherwise. The variable is set by stdpoll and strictpoll built-in functions (and by `on poll' statement, accordingly). * New string functions: `length' and `substring' * Here document syntax expanded. ** Removing leading whitespace Inserting a single space between the dash and the terminator word in the beginning of the here-document construct, as in: <<- WORD ... WORD instructs parser to remove leading white space characters from each line of the document body, including terminating WORD. ** Expansion of macros and variables Variables and sendmail macros are expanded when used within a double-quoted string or a here-document body. The variable expansion and backslash interpretation is suppressed by quoting the WORD, e.g.: <<\WORD ... WORD or <<'WORD' ... WORD ** Numeric escape sequences Two new kinds of escape sequences are supported: \0ooo, where o is any octal digit \xhh, where h is any hex digit ** Back-references. References to the parenthesized subexpressions of the previous regular expression are expanded both in the code and in double-quoted string literals. For example: if domainpart $f matches '\(.*\).com' set d \1 fi * Bugfixes ** Fix berkeley 4.x support ** Fix expiration of the greylist and rate databases. ** Fix returning multiline replies. The last line of the reply was not taken into account unless it ended with a newline. ** Fix type casting of arguments to user-defined functions ** Fix argument passing in function calls generated for `on poll' statements. Version 2.0 * Program requires Mailutils version 1.0 or newer * Support for old DBM and NDBM has been withdrawn. * Added support for Berkeley DB versions 3.x and 4.x * INCOMPATIBLE CHANGES ** To use version 1.x configuration files, the following changes should be applied to them: 1. The entire code section should be enclosed in the following statement: prog envfrom do ... done See the section `Handler declarations' below for the detailed description. 2. Convert any `rate' statements to function calls, e.g.: if rate $f 180 / minute should be rewritten as if rate($f, interval(minute)) > 180 See the section `rate(key, interval)' below for the detailed description. See also section "Special test functions" in the mailfromd documentation. ** Format of cache and rates database has changed: the key field now includes the trailing nul character, which is also reflected in its length. This allows for empty (zero-length) keys. To convert existing databases to the new format, run mailfromd --compact --all after compiling the package. ** The database management options (--list,--delete,--expire) do not take any argument. To specify that the option refers to the rate database, use --format=rate option. * Language features ** Compiled code Configuration file handling completely rewritten. The file is parsed into a pseudo-code program, which is executed considerably faster than the parse tree in 1.x branch. ** Sendmail macros Multiletter Sendmail macros can be used with and without surrounding curly braces, i.e. ${rcpt_count} and $rcpt_count are both valid. ** File inclusion Configuration file can include other files. The syntax is: #include "FILENAME" Inclusion depth is not limited. ** Adjacent expressions concatenate: $f => "gray@gnu.org.ua" ${client_addr} => "127.0.0.1" $f "-" $client_addr => "gray@gnu.org.ua-127.0.0.1" ** Arithmetical expressions The four usual arithmetical expressions are now supported. ** Comparisons In addition to equal (=) and not equal (!=) the following comparison operators are supported: <, <=, >, >=. Their precedence and associativity is the same as in C. ** Type casting The rules for implicit type casts are: 1. Both arguments to an arithmetical operation are cast to numeric types. 2. All arguments of the concatenation operation are cast to string type. 3. Both arguments to `match' or `fnmatch' function are cast to string type. 4. The argument of the unary negation (arithmetical or boolean) is cast to numeric type. 5. Otherwise, if the arguments to a binary operation have different types, the right-hand side argument is cast to the type of the left-hand side argument. There is no special syntactic sugar for explicit type casting. To cast an expression to string, concatenate an empty string to it: %var "" To cast an expression to numeric, add it to zero: %var + 0 ** Single-quoted strings In addition to double-quoted strings, single-quoted ones are also supported. The single-quoted strings are not subject to backslash expansion, thus they are particularly useful for writing regular expressions: if $f matches '.*\.com' ** Splitting strings between several lines. Double-quoted strings can be split over several lines, by placing a backslash immediately before the newline. For example: "A very\ long string" produces "A very long string". ** Here document syntax and multiline sendmail replies Mailfromd supports "here document" syntax: <<[-]WORD ... WORD Optional '-' instructs parser to remove leading tabulation characters from each line of the document body, including terminating WORD. This allows for here documents to follow normal program indentation. The backslash expansion is performed on the contents of the here document, unless WORD is quoted (i.e. either 'WORD' or \WORD). This is particularly useful in providing multiline Sendmail replies: reject 550 5.0.0 <<-EOT This service is not available now. Please, refer to http://some.site for more information on the subject. EOT ** Handler declarations The program consists of a set of handler declarations. Each handler is defined as prog STATE do ... done where `...' represents the actual code, and STATE is the milter state this handler is defined for. Recognized milter states are: helo, envfrom, envrcpt, header, eoh, body, eom. ** Positional arguments ($N notation) Handlers can take several positional arguments, which can be dereferenced in the program using $N notation, where N is a decimal number. The semantics of the positional arguments depends on the state for which the handler is designed. For example: prog helo do /* For helo, $1 means helo domain */ if $1 matches "gnu.org.ua" ... fi done ** Internal variables (% notation) Mailfromd variables can be defined using `set' statement: set VAR expr where VAR is the variable name. The variables are dereferenced using %VAR notation. Their values are retained between handlers, for example: prog helo do set helo_domain $1 done prog envfrom do if $f = "gray" and %helo_domain matches "gnu.org.ua" ... fi done The `set' statement can be used both inside and outside of program handlers or functions. When used outside them, it declares a global variable and assigns it an initial value. The variable will be initialized to that value at the beginning of each message. ** Built-in function syntax Built-in functions of more than 1 argument are invoked using the usual C-like syntax, e.g.: name(arg1, arg2, arg3) Built-in functions of one argument can be invoked with or without parentheses: name arg name(arg) ** User defined functions User defined functions are declared using the following syntax: func NAME ( TYPELIST ) returns TYPE do ... done where NAME is the name of the function, TYPELIST is a comma-separated list of parameter types, and TYPE is the return type of the function. Types are designated by a single letter: 'n' denotes the numeric type, 's' denotes the string type. Within the function body, the arguments are denoted using positional argument notation (see `Positional arguments' above). The `return' statement is used to return a value from the function. For example: func sum(n,n) returns n do return $1 + $2; done ** Switch statement The mailfromd language now has switch statement. Its syntax is similar to that of C: switch EXPR do case VAL [or VAL...] : STMT . . . default: STMT done where EXPR is any valid mailfromd expression, VAL is a literal value (numeric or string), STMT is any mailfromd statement or a list of statements. For example: switch %x do case 1 or 2: echo "Accepting mail" accept default: reject done ** Catch statement The new `catch' statement can be used to handle exceptional conditions occurring within a function. An exceptional condition is signalled when the filter script program encounters a condition it is not able to handle. See the documentation, section "Exceptions" for the details. The syntax of the `catch' statement is: catch EXCEPTION-LIST do HANDLER-BODY done where EXCEPTION-LIST is the list of exception types, separated by the word `or'. Special form `catch *' catches all exceptions. The HANDLER-BODY is the list of statements comprising the handler body. ** On statement The support for `on' statement has been entirely rewritten. The statement is implemented as a wrapper around `catch'. Instead of `poll' you can specify any function call as its selector value. For example, this statement: prog envfrom do on poll $f do when success: accept when not_found or failure: reject 550 5.1.0 "Sender validity not confirmed" when temp_failure: tempfail 450 4.1.0 "Try again later" done done is actually a shortcut for: function poll_wrapper(s) returns n do catch &success or ¬_found or &failure or &temp_failure do return $1 done return stdpoll($1, %ehlo_domain, %mailfrom_address) done prog envfrom do switch poll_wrapper($f) do case &success: accept case ¬_found or &failure: reject 550 5.1.0 "Sender validity not confirmed" case &temp_failure: tempfail 450 4.1.0 "Try again later" done done (See also the description of %ehlo_domain and %mailfrom_address variables) ** MX matching The `matches' and `fnmatches' operations has been extended to allow operations on MX lists. The test if ${client_addr} mx matches '.*\.gnu\.org' yields true if the list of MXs for the IP address ${client_addr} contains a host name that matches the regular expression '.*\.gnu\.org'. The left side argument can be either a hostname or IP address or an RFC 2822 email address. In the latter case, its domain part is used to query for the MX records. The similar 'mx fnmatches' construction is also available. Both `mx' tests can throw one of the following exceptions: not_found, failure, temp_failure. ** #pragma database New pragma `database' controls various aspects of databases used by the program. The pragma has four forms: 1. Store the database DBNAME in the given FILENAME #pragma database DBNAME file FILENAME 2. Set the expiration interval for DBNAME: #pragma database DBNAME expire-interval INTERVAL INTERVAL can be any valid interval specification (see the next section). If DBNAME is "cache", two additional forms are understood: 3. Set the expiration period for positive cache records: #pragma database cache positive-expire-interval INTERVAL 4. Set the expiration period for negative cache records: #pragma database cache negative-expire-interval INTERVAL The forms 3. and 4. are equivalent to #pragma option positive-expire-interval INTERVAL and #pragma option negative-expire-interval INTERVAL correspondingly. ** Time interval specification Time intervals can be specified as an English text, e.g. "1 hour 35 minutes". The following pragmas take a time interval specification as their argument: #pragma option timeout #pragma option milter-timeout #pragma option expire-interval #pragma option negative-expire-interval #pragma option positive-expire-interval #pragma option rates-expire-interval #pragma option lock-retry-timeout * New built-in functions: ** dbmap(dbname, key) Returns true if `key' is found in the database file `dbname', E.g.: dbmap("/etc/mail/aliases.db", $f) ** domainpart(str) Returns the domain part of `str' if it is a valid email address, otherwise returns `str' itself. ** greylist(key, interval) Returns true if the `key' is found in the greylist database (controlled by `#pragma database greylist' pragma). The argument `interval' gives the greylisting interval in seconds. The function sets internal variable %greylist_seconds_left to the number of seconds left to the end of greylisting period. Sample usage: set gltime 500 if greylist(${client_addr} "-" $f "-" ${rcpt_addr}, %gltime) if %greylist_seconds_left = %gltime tempfail 470 "You are greylisted for " %gltime " seconds" else tempfail 470 "Still greylisted for " %greylist_seconds_left "seconds" fi fi ** hasmx(host) Returns true if `host' has any MX records. The function can throw one of the following exceptions: not_found, failure, temp_failure. ** interval(string) Converts its argument, which should be a valid time interval specification, to seconds ** localpart(str) Returns the local part of `str' if it is a valid email address, otherwise returns unchanged `str'. ** match_cidr(ip, cidr) Returns true if the IP address `ip' matches the network block `cidr'. For example: match_cidr(${client_addr}, "213.130.0.0/19") Possible exceptions: invip, if the first parameter is not a valid IP, and invcidr if the second parameter is not a valid CIDR. ** stdpoll(email, domain, mailfrom) Performs standard poll for `email', using `domain' as EHLO domain and `mailfrom' as MAIL FROM: address. Returns 0 or 1 depending on the result of the test. Can raise one of the following exceptions: failure, temp_failure. In `on' statement context, it is synonymous to `poll' without explicit `host'. ** strictpoll(email, domain, mailfrom, host) Performs strict poll for `email' on host `host'. See the description of stdpoll for the detailed information. In `on' statement context, it is synonymous to `poll host'. ** _pollhost(ip, email, domain, mailfrom) Poll SMTP host `ip' for email address `email', using `domain' as EHLO domain and `mailfrom' as MAIL FROM: address. Returns 0 or 1 depending on the result of the test. In contrast to strictpoll function, this function does not use cache database and does not fall back to MX poll if the poll tempfails. The function can throw one of the following exceptions: failure, temp_failure. ** _pollmx(domain, email, domain, mailfrom) Poll MX-s of the `domain' for email address `email', using `domain' as EHLO domain and `mailfrom' as MAIL FROM: address. Returns 0 or 1 depending on the result of the test. In contrast to stdpoll function, _pollmx function does not use cache database and does not fall back to host poll if the poll fails. The function can throw one of the following exceptions: failure, temp_failure. ** tolower(string) Returns a copy of the string str, with all the upper-case characters in string translated to their corresponding lower-case counterparts. Non-alphabetic characters are left unchanged. ** toupper(string) Returns a copy of the string str, with all the lower-case characters in string translated to their corresponding upper-case counterparts. Non-alphabetic characters are left unchanged. ** rate(key, interval) Returns the mail sending rate for `key' per `interval'. This function replaces `rate' statement from 1.x branch. To convert old rate statements use the following algorithm: Old statement: if rate KEY LIMIT '/' EXPR New statement: if rate(KEY, interval("EXPR")) > LIMIT For example: Old statement: rate $f 180 / 1 hour 25 minutes New statement: if rate($f, interval("1 hour 25 minutes")) > 180 ** resolve(host) Returns the IP address corresponding to `host' or "0" if it cannot be resolved. ** validuser(user) Returns true if `user' is a valid local account. It uses mailutils authentication mechanisms. * Global variables ** %ehlo_domain Name of the domain used by polling functions in EHLO or HELO command. It is set by `#pragma option ehlo' directive, or via --ehlo command line option. ** %mailfrom_address Email address used by polling functions in 'MAIL FROM' command. Set by `#pragma option mailfrom' directive or via --mailfrom command line option. ** %rcpt_count The variable %rcpt_count keeps the number of recipients given so far. The variable is defined in the envrcpt state. * Database expiration The operation of `mailfromd --expire' has been completely redesigned to avoid skipping some keys when using GDBM. * Database compaction New option --compact starts "database compaction" process, which removes all expired entries and empty blocks from the database. It also converts any obsolete (not nul-terminated) keys to nul-terminated ones. During this process the original database file is locked for writing, so the running mailfromd instance is able to read entries from it, but cannot write or update it. The existed database will be replaced with the compacted version only if there were no errors during the process. If you wish to ignore any failed reads (keys that were not retrieved), use the --ignore-failed-reads option. * Database locking Before accessing, any database file is locked using kernel locking. By default, if the first attempt to lock the file fails, two more attempts are undertaken in 1 second intervals. If the lock cannot be acquired after the last attempt, the database file is opened in read- only mode. The number of locking attempt and the timeout value are controlled by command line options --lock-retry-count and --lock-retry-timeout, or the corresponding pragmas: #pragma option lock-retry-count #pragma option lock-retry-timeout * Selecting the database format and file ** New option -H (--format) specifies which format is the database being operated upon by any of the database management options. Recognized formats are: cache Poll cache database rate Sending rate database greylist Greylisting database (see below) ** New option --all can be used with one of the options --expire or --compact to apply the operation to all configured databases. This is useful to invoke mailfromd as a crontab job. ** New option --file allows to explicitely specify the database file name. Notice, that in contrast to the previous version, the name should include the suffix. * Debugging ** New option --lint (-l, --syntax-check): check the configuration file syntax. ** The argument to --debug option should be the numeric debug level. The use of characters 'c', 'd', 'l', 'y' is discouraged (although still supported for a while). See below for the alternatives. ** New option --dump-code dumps the listing of the assembled code on screen (similar to the earlier --debug=c) ** New option --dump-tree dumps the parse tree in human-readable form (earlier --debug=d option) ** New option --dump-grammar-trace prints grammar parser traces while parsing the configuration file (earlier --debug=g option). ** New option --dump-lex-trace dumps lexical analizer traces (earlier --debug=l option). ** New option -L (--log-tag) sets the identifier used in syslog messages. ** New option --source-info includes source line information into the debugging messages. Previously this information was included by default. * New option --group (-g) allows to retain the given supplementary group when switching to user privileges. By default mailfromd does not retain any supplementary groups. The use of this option may be necessary if your mailfromd script needs to access some databases that have restrictive access privileges. For example, if mailfromd runs with the privileges of user 'mail' (the default) and needs to access /etc/mail/aliases.db, which is usually owned by root.smmsp and has access rights 0640, you should run mailfromd --group smmsp * New option --source (-S) sets the source address mailfromd will use for any TCP connections. The configuration file equivalent is #pragma option source * To set up the local account validation (see the description of `validuser' function above) mailfromd uses authentication options from mailutils. See the mailutils documentation, chapter `Authorization and Authentication Principles' for the detailed description of these. * Code generation optimized to avoid unnecessary instructions and to reduce code size. * MX lookups no longer recurse to parent domains. Previously, if the domain "some.domain.com" had no MX records, mailfromd would lookup for MXs of "domain.com" and use these instead. This is no longer the case. * Added testsuite Version 1.4 * Configuration Added possibility to link against the forked version of libmilter (--with-forks). The patch for sendmail-8.13.1 is included (etc/sendmail-8.13.1.diff). * Configuration file ** New unary expression `listens' checks if the host listens on port 25. ** Several `#pragma option relay' statements accumulate * Bugfixes ** Fixed coredump on incorrect libmilter socket specification. ** Fixed `poll for EMAIL as EMAIL'. Version 1.3 * Rewritten DNS resolver functions in order to take into account CNAMEs. * Updated Makefiles to allow for compilation with the CVS Mailutils * Improved documentation. Version 1.2 * Implemented sending rate control. This feature allows to impose a limit on the number of messages a user can send within a given interval. If this number is exceeded, the connection is refused until enough time passes to keep the rate within the given limit. Version 1.1 Mostly bugfixes. Version 1.0 Lots of major improvements. Implemented two methods of sender address verification, controlled by a sophisticated configuration file. Sender domains and emails can also be distinguished basing on POSIX regex or shell-style globbing patterns. Version 0.2 First release. ========================================================================= Copyright information: Copyright (C) 2005, 2006, 2007 Sergey Poznyakoff Permission is granted to anyone to make or distribute verbatim copies of this document as received, in any medium, provided that the copyright notice and this permission notice are preserved, thus giving the recipient permission to redistribute in turn. Permission is granted to distribute modified versions of this document, or of portions of it, under the above conditions, provided also that they carry prominent notices stating who last changed them. Local variables: mode: outline paragraph-separate: "[ ]*$" eval: (add-hook 'write-file-hooks 'time-stamp) time-stamp-start: "changes. " time-stamp-format: "%:y-%02m-%02d" time-stamp-end: "\n" end: