@c Copyright (C) 2005-2006, 2009-2011 Sergey Poznyakoff @c Permission is granted to copy, distribute and/or modify this document @c under the terms of the GNU Free Documentation License, Version 1.3 or @c any later version published by the Free Software Foundation; with no @c Invariant Sections, with the Front-Cover texts being ``Mailfromd Manual'', @c and with the Back-Cover Texts at your option. The following sections describe procedures for upgrading between the consecutive Mailfromd releases. @menu * 710-700:: Upgrading from 7.0 to 7.1 * 600-700:: Upgrading from 6.0 to 7.0 * 5x0-600:: Upgrading from 5.x to 6.0 * 500-510:: Upgrading from 5.0 to 5.1 * 440-500:: Upgrading from 4.4 to 5.0 * 43x-440:: Upgrading from 4.3.x to 4.4 * 420-43x:: Upgrading from 4.2 to 4.3.x * 410-420:: Upgrading from 4.1 to 4.2 * 400-410:: Upgrading from 4.0 to 4.1 * 31x-400:: Upgrading from 3.1.x to 4.0 * 30x-31x:: Upgrading from 3.0.x to 3.1 * 2x-30x:: Upgrading from 2.x to 3.0.x * 1x-2x:: Upgrading from 1.x to 2.x @end menu @node 710-700 @appendixsec Upgrading from 7.0 to 7.1 The 7.1 release fixes the bugs found in 7.0. No special actions are needed when upgrading. @node 600-700 @appendixsec Upgrading from 6.0 to 7.0 @cindex Upgrading from 6.0 to 7.0 The release 7.0 removes the features which were declared as obsolete in 6.0 and introduces important new features, both syntactical, at the @acronym{MFL} level, and operational. Unless your filter used any deprecated features, it should work correctly after upgrade to this version. It will, however, issue warning messages regarding the deprecated features (e.g. the use of @samp{%} in front of identifiers, as described below). To fix these, follow the upgrade procedure described in @ref{upgrade procedure}. The removed features are: @itemize @bullet @item Old-style functional notation @item The use of functional operators @item Implicit concatenations @item #pragma option @item #pragma database @end itemize The @acronym{MFL} syntax has changed: it is no longer necessary to use @samp{%} in front of a variable to get its value. To reference a variable, simply use its name, e.g.: @smallexample set x var + z @end smallexample The old syntax is still supported, so the following statement will also work: @smallexample set x %var + %z @end smallexample It will, however, generate a warning message. Of course, the use of @samp{%} to reference variables within a string literal remains mandatory. Another important changes to @acronym{MFL} are user-defined exceptions (@pxref{User-defined Exceptions}) and the @dfn{try--catch} construct (@pxref{Catch and Throw,try--catch}). Several existing @acronym{MFL} functions have been improved. In particular, it is worth noticing that the @code{open} function, when opening a pipe to or from a command, provides a way to control where the command's standard error would go (@pxref{open, standard error}). The @code{accept} function (or action) issues a warning if its use would cancel any modifications to the message applied by, e.g., @code{header_add} and similar functions. @xref{Message modification queue}, for a detailed discussion of this feature. The most important change in @command{mailfromd} operation is that the version 7.0 is able to run several servers (@pxref{conf-server}). Dedicated @dfn{callout servers} make it possible to run sender verifications in background, using a set of long timeouts, as prescribed by RFC 2822 (@pxref{SMTP Timeouts}). This diminishes the number of false positives, allows for coping with servers showing large delays and also reduces the number of callouts performed for such servers. This release no longer includes the @command{smap} utility. It was moved into a self-standing project, which by now provides much more functionality and is way more flexible than this utility was. If you are interested in using @command{smap}, visit @uref{http://www.gnu.org.ua/software/smap}, for a detailed information, including pointers to file downloads. @node 5x0-600 @appendixsec Upgrading from 5.x to 6.0 @cindex Upgrading from 5.x to 6.0 The 6.0 release is aimed to fix several logical inconsistencies that affected the previous versions. The most important one is that until version 5.2, the filter script file contained both the actual filter script, and the run-time configuration for @command{mailfromd} (in form of @samp{#pragma option} and @samp{#pragma database} statements). The new version separates run-time configuration from the filter script by introducing a special configuration file @file{mailfromd.conf} (@pxref{Mailfromd Configuration}). Consequently, the @samp{#pragma option} and @samp{#pragma database} statements become deprecated. Furthermore, the following deprecated pragmas are removed: @samp{#pragma option ehlo}, @samp{#pragma option mailfrom}. These pragmas became deprecated in version 4.0 (@pxref{31x-400}). The second problem was that the default filter script file had @samp{.rc} suffix, which usually marks a configuration file, not the source. In version 6.0 the script file is renamed to @file{mailfromd.mf}. In the absence of this file, the legacy file @file{mailfromd.rc} is recognized and parsed. This ensures backward compatibility. This release also fixes various inconsistencies and dubious features in the @acronym{MFL} language. The support for unquoted literals is discontinued. This feature was marked as deprecated in version 3.0. @anchor{deprecated-6.0} The following features are deprecated: @samp{#pragma option} (@histref{6,pragma_002doption,pragma-option} and @samp{#pragma database} (@histref{6,pragma_002ddatabase,pragma-database}) directives, the legacy style of function declarations (@histref{6,old_002dstyle-function-declarations, old-style function declarations}), calls to functions of one argument without parentheses (@histref{6,implicit-concatenation, operational notation}), the @samp{#require} statement (@xref{import}, for the new syntax) and implicit concatenation (@histref{6,implicit-concatenation, implicit concatenation}). See @histref{6,Deprecated-Features,Deprecated Features}, for more information about these. This release also introduces important new features, which are summarized in the table below: @multitable @columnfractions .50 .50 @headitem Feature @tab Reference @item Configuration @tab @xref{Mailfromd Configuration}. @item Module system @tab @xref{Modules}. @item Explicit type casts @tab @xref{explicit type casts}. @item Concatenation operator @tab @xref{Concatenation}. @item Scope of visibility @tab @xref{scope of visibility}. @item Precious variables @tab @xref{rset}. @end multitable @command{Mailfromd} version @samp{6.0} will work with unchanged scripts from @samp{5.x}. When started, it will verbosely warn you about any deprecated constructs that are used in your filter sources and will create a script for upgrading them. @anchor{upgrade procedure} To upgrade your filter scripts, follow the steps below: @enumerate 1 @item Run @samp{mailfromd --lint}. You will see a list of warnings similar to this: @smallexample mailfromd: Warning: using legacy script file /usr/local/etc/mailfromd.rc mailfromd: Warning: rename it to /usr/local/etc/mailfromd.mf or use script-file statement in /usr/local/etc/mailfromd.conf to disable this warning mailfromd: /usr/local/etc/mailfromd.rc:19: warning: this pragma is deprecated: use relayed-domain-file configuration statement instead mailfromd: /usr/local/etc/mailfromd.rc:23: warning: this pragma is deprecated: use io-timeout configuration statement instead mailfromd: Info: run script /tmp/mailfromd-newconf.sh to fix the above warnings @dots{} @end smallexample @item At the end of the run @command{mailfromd} will create a shell script @file{/tmp/mailfromd-newconf.sh} for fixing these warnings. Run it: @smallexample $ sh /tmp/mailfromd-newconf.sh @end smallexample @item When the script finishes, run @command{mailfromd --lint} again. If it shows no more deprecation warnings, the conversion went correctly. Now you can remove the upgrade script: @smallexample $ rm /tmp/mailfromd-newconf.sh @end smallexample @end enumerate Notice, that the conversion script attempts to fix only deprecation warnings. It will not try to correct any other type of warnings or errors. For example, you may get warning messages similar to: @smallexample mailfromd: /etc/mailfromd.mf:7: warning: including a module file is unreliable and may cause subtle errors mailfromd: /etc/mailfromd.mf:7: warning: use `require dns' instead @end smallexample This means that you use @samp{#include} where you should have used @samp{require}. You will have to fix such warnings manually, as suggested in the warning message. If, for some reason, you cannot upgrade your scripts right now, you may suppress deprecation warnings by setting the environment variable @env{MAILFROMD_DEPRECATION} to @samp{no} before starting @command{mailfromd}. Nonetheless, I recommend to upgrade as soon as possible, because the deprecated features will be removed in version @samp{6.1}. @node 500-510 @appendixsec Upgrading from 5.0 to 5.1 @cindex Upgrading from 5.0 to 5.1 Upgrading from 5.0 to 5.1 does not require any changes in your filter scripts. Notice, however, the following important points: @itemize @bullet @item Starting from this release @command{mailfromd} supports Milter protocol version 6, which is compatible with Sendmail 8.14.0 and newer. While being backward compatible with earlier Sendmail releases, it allows you to use the new @samp{prog data} handler (@pxref{Handlers, data}). It also supports macro negotiation, a feature that enables Mailfromd to ask @acronym{MTA} to export the macros it needs for each particular handler. This means that if you are using Sendmail 8.14.0 or higher (or Postfix 2.5 or higher), you no longer need to worry about exporting macro names in @file{sendmail.cf} file. The same feature is also implemented on the server side, in @code{mtasim} and @code{pmult}. Consequently, using @code{define-macros} in @code{pmult} configuration file is not strictly necessary. However, keep in mind that due to the specifics of @acronym{MeTA1}, the number of symbols that may be exported for each stage is limited (@pxref{pmult-macros}). @item The semantics of @code{__preproc__} and @code{__statedir__} built-in constant is slightly different from what it used to be in 5.0. These constants now refer to the @emph{current} values of the preprocessor command line and program state directory, correspondingly. This should not affect your script, unless you redefine the default values on run time. If your script needs to access default values, use @code{__defpreproc__} and @code{__defstatedir__}, correspondingly (@pxref{Built-in constants}). The following example explains the difference between these: @smallexample $ cat pval.mf prog envfrom do echo "Default value: " __defstatedir__ echo "Current value: " __statedir__ done $ mailfromd --state-directory=/var/mfd --test pval.mf Default value: /usr/local/var/mailfromd Current value: /var/mfd @end smallexample @item If your filter uses the @code{rate} function, you might consider using the new function @code{rateok} or @code{tbf_rate} instead. For a detailed discussion of these functions, see @ref{Sending Rate}. @item If your script extensively uses database access functions, you might be interested in the new @code{#pragma dbprop} (@pxref{dbprop}). @end itemize @node 440-500 @appendixsec Upgrading from 4.4 to 5.0 @cindex Upgrading from 4.4 to 5.0 This version of Mailfromd requires @uref{http://www.gnu.org/software/mailutils, GNU mailutils} version 2.0 or later. Upgrading from version 4.4 to 5.0 requires no additional changes. The major differences between these two versions are summarized below: @enumerate 1 @item Support for @samp{MeTA1}. @item New @samp{Mailutils} configuration file. @item New @acronym{MFL} functions. @enumerate a @item Message functions. @xref{Message functions}. @item Mailbox functions. @xref{Mailbox functions}. @item Mail body functions. @xref{Mail body functions}. @item Header modification functions. @xref{Header modification functions}. @item Envelope modification functions. @xref{Envelope modification functions}. @item Quarantine functions. @xref{Quarantine functions}. @item @code{getopt} and @code{varptr}. @xref{getopt}. @item Macro access functions. @xref{Macro access}. @item Character type functions. @xref{Character Type}. @item New string functions (@pxref{String manipulation}): @code{verp_extract_user}, @code{sa_format_report_header}, @code{sa_format_score}. @item Sequential access to DBM files. @xref{dbm-seq}. @end enumerate @item Changes in @acronym{MFL} @enumerate 1 @item @xref{variadic functions}. @item @xref{function alias}. @end enumerate @item New operation mode: @xref{Run Mode}. @item Improved stack growth technique. The stack can be grown either by fixed size blocks, or exponentially. Upper limit can be specified. @xref{stacksize}. @item Milter ports can be specified using @acronym{URL} notation. @item Removed deprecated features. Support for some deprecated features has been withdrawn. These are: @enumerate a @item Command line options @option{--ehlo}, @option{--postmaster-email}, and @option{--mailfrom}. These became deprecated in version 4.0. @xref{31x-400}. @end enumerate @end enumerate @node 43x-440 @appendixsec Upgrading from 4.3.x to 4.4 @cindex Upgrading from 4.3.x to 4.4 The deprecated @option{--domain} command line option has been withdrawn. The short option @option{-D} now defines a preprocessor symbol (@pxref{Preprocessor Options}). This version correctly handles name clashes between constants and variables, which remained unnoticed in previous releases. @xref{variable--constant shadowing}, for a detailed description of it. To minimize chances of name clashes, all symbolic exception codes has been renamed by prefixing them with the @samp{e_}, thus, e.g. @code{divzero} became @code{e_divzero}, etc. The @code{ioerr} exception code is renamed to @code{e_io}. @xref{status.mf}, for a full list of the new exception codes. @cindex @code{OLD_EXCEPTION_CODES}, preprocessor symbol For consistency, the following most often used codes are available without the @samp{e_} prefix: success, not_found, failure, temp_failure. This makes most existing user scripts suitable for use with version 4.4 without any modification. If your script refers to any exception codes other than these four, you can still use it by defining a preprocessor symbol @code{OLD_EXCEPTION_CODES}, for example: @smallexample $ mailfromd -DOLD_EXCEPTION_CODES @end smallexample @node 420-43x @appendixsec Upgrading from 4.2 to 4.3.x @cindex Upgrading from 4.2 to 4.3.x @cindex @code{DEFAULT_SYSLOG_ASYNC}, @command{configure} variable Upgrading from 4.2 to 4.3 or 4.3.1 does not require any changes to your configuration and scripts. The only notable change in these versions is the following: The asynchronous syslog implementation was reported to malfunction on some systems (notably on Solaris), so this release does not enable it by default. The previous meaning of the @option{--enable-syslog-async} configuration option is also restored. Use this option in order to enable asynchronous syslog feature. To set default syslog implementation, use @code{DEFAULT_SYSLOG_ASYNC} configuration variable (@pxref{syslog-async}). The following deprecated features are removed: @enumerate @item @code{#pragma option ehlo} statement. It became deprecated in version 4.0. @xref{pragma-option-ehlo}. @item @code{#pragma option mailfrom} statement. It became deprecated in version 4.0. @xref{pragma-option-ehlo}. @item The @option{--config-file} command line option. It became deprecated in version 3.1. @xref{30x-31x}. @item Built-in exception codes in catch statements. They are deprecated since version 4.0. @xref{31x-400}. @end enumerate @node 410-420 @appendixsec Upgrading from 4.1 to 4.2 @cindex Upgrading from 4.1 to 4.2 Upgrading to this version does not require any special efforts. You can use your configuration files and filter scripts without any changes. The only difference worth noticing is that starting from this version @command{mailfromd} is always compiled with asynchronous syslog implementation. The @option{--enable-syslog-async} configuration file option is still available, but its meaning has changed: it sets the @emph{default} syslog implementation to use (@pxref{syslog-async}). Thus, it can be used the same way it was in previous versions. You can also select the syslog implementation at run time, see @ref{Logging and Debugging, --syslog-async option}, for more detailed information. @node 400-410 @appendixsec Upgrading from 4.0 to 4.1 @cindex Upgrading from 4.0 to 4.1 Upgrading to this version does not require any special efforts. You can use your configuration files and filter scripts without any changes. Notice only the following major differences between 4.1 and 4.0: @itemize @bullet @item Input files are preprocessed before compilation. @xref{Preprocessor}, for more information. @item There is a way to discern between a not-supplied optional parameter, and a supplied one, having null value (@pxref{defined}). @item The version 4.1 implements @code{sprintf} function (@pxref{String formatting}) and @code{printf} macro (@pxref{Preprocessor, printf}). @item Support for some obsolete features is withdrawn. These include: @enumerate 1 @item Using @samp{&@var{code}} to specify exception codes @item Pragma options: @code{retry}, @code{io-retry}, and @code{connect-retry}. @end enumerate @end itemize @node 31x-400 @appendixsec Upgrading from 3.1.x to 4.0 @cindex upgrading from 3.1.x to 4.0 Before building this version, please re-read the chapter @xref{Building}, especially the section @ref{syslog-async, Using non-blocking syslog}. Starting from the version 4.0, @acronym{MFL} no longer uses the predefined symbolic names for exception codes (previous versions used the @samp{&} prefix to dereference them). Instead, it relies on constants defined in the include file @file{status.mfh} (@pxref{status.mf}). However, the script files from 3.1 series will still work, but the following warning messages will be displayed: @smallexample Warning: obsolete constant form used: &failure Warning: remove leading '&' and include Warning: Using built-in exception codes is deprecated Warning: Please include @end smallexample @anchor{pragma-option-ehlo} Another important difference is that pragmatic options @samp{ehlo} and @samp{mailfromd} are now deprecated, as well as their command line equivalents @option{--ehlo} and @option{--domain}. These options became superfluous after the introduction of @code{mailfrom_address} and @code{ehlo_domain} built-in variables. For compatibility with the previous versions, they are still supported by @command{mailfromd} 4.0, but a warning message is issued if they are used: @smallexample @group warning: `#pragma option ehlo' is deprecated, consider using `set ehlo_domain "domain.name"' instead @end group @end smallexample To update your startup scripts for the new version follow these steps: @enumerate 1 @item Change @code{#pragma option mailfrom @var{value}} to @code{set mailfrom_address @var{value}}. Refer to @ref{mailfrom_address}, for a detailed discussion of this variable. @item Change @code{#pragma option ehlo @var{value}} to @code{set ehlo_domain @var{value}}. Refer to @ref{ehlo_domain}, for a detailed discussion of this variable. @item Include @file{status.mfh}. Add the following line to the top of your startup file: @smallexample #include_once @end smallexample @item Remove all instances of @samp{&} in front of the constants. You can use the following @command{sed} expression: @samp{s/&\([a-z]\)/\1/g}. @item If your code uses any of the following functions: @code{hostname}, @code{resolve}, @code{hasmx} or @code{ismx}, add the following line to the top of your script: @smallexample #require dns @end smallexample @xref{Modules}, for a detailed description of the module system. @item Replace all occurrences of @code{next} with @code{pass}. @item If your code uses function @code{match_cidr}, add the following line to the top of your script: @smallexample #require match_cidr @end smallexample @xref{Modules}, for a description of @acronym{MFL} module system. @end enumerate @node 30x-31x @appendixsec Upgrading from 3.0.x to 3.1 @cindex upgrading from 3.0.x to 3.1 @enumerate 1 @item The @command{mailfromd} binary no longer supports @option{--config-file} (@option{-c}) option. To use an alternative script file, give it as an argument, i.e. instead of: @smallexample $ @kbd{mailfromd --config-file @var{file.rc}} @end smallexample @noindent write: @smallexample $ @kbd{mailfromd @var{file.rc}} @end smallexample For backward compatibility, the old style invocation still works but produces a warning message. However, if @command{mailfromd} encounters the @option{-c} option it will print a diagnostic message and exit immediately. This is because the semantics of this option will change in the future releases. @item If a variable is declared implicitly within a function, it is created as automatic. This differs from the previous versions, where all variables were global. It is a common practice to use global variables to pass additional information between handlers (@xref{HELO Domain}, for an example of this approach). If your filter uses it, make sure the variable is declared as global. For example, this code: @float Figure, old-code @caption{Implicit declaration, old style} @smallexample prog helo do # @r{Save the host name for further use} set helohost $s done @end smallexample @end float @noindent has to be rewritten as follows: @float Figure, new-code @caption{Implicit declaration, new style} @smallexample set helohost "" prog helo do # @r{Save the host name for further use} set helohost $s done @end smallexample @end float @item Starting from version 3.1 the function @code{dbmap} takes an optional third argument indicating whether or not to count the terminating null character in key (@pxref{dbmap}). If your startup script contained any calls to @code{dbmap}, change them as follows: @multitable @columnfractions 0.5 0.5 @headitem in 3.0.x @tab in 3.1 @item dbmap(@var{db}, @var{key}) @tab dbmap(@var{db}, @var{key}, 1) @end multitable @end enumerate @node 2x-30x @appendixsec Upgrading from 2.x to 3.0.x @cindex upgrading from 2.x to 3.0.x Update your startup scripts and/or crontab entries. The @command{mailfromd} binary is now installed in @file{$@{prefix@}/sbin}. We also encourage you to update the startup script (run @kbd{cp etc/rc.mailfromd /wherever-your-startup-lives}), since the new version contains lots of enhancements. @node 1x-2x @appendixsec Upgrading from 1.x to 2.x @cindex upgrading from 1.x to 2.x If you are upgrading from version 1.x to 2.0, you will have to do the following: @enumerate 1 @item Edit your script file and enclose the entire code section into: @smallexample @group prog envfrom do @dots{} done @end group @end smallexample @noindent @xref{Handlers}, for more information about the @code{prog} statement. @item If your code contained any @code{rate} statements, convert them to function calls (@pxref{Rate limiting functions, rate}), using the following scheme: @smallexample @group Old statement: if rate @var{key} @var{limit} / @var{expr} New statement: if rate(@var{key}, interval("@var{expr}")) > @var{limit} @end group @end smallexample For example, @smallexample rate $f 180 / 1 hour 25 minutes @end smallexample @noindent should become @smallexample rate($f, interval("1 hour 25 minutes")) > 180 @end smallexample @item Rebuild your databases using the following command: @smallexample mailfromd --compact --all @end smallexample @noindent This is necessary since the format of @command{mailfromd} databases has changed in version 2.0: the key field now includes the trailing @samp{NUL} character, which is also reflected in its length. This allows for empty (zero-length) keys. @xref{Database Maintenance}, for more information about the database compaction. @end enumerate