path: root/doc/upgrade.texi

@c Copyright (C) 2005--2024 Sergey Poznyakoff
@c Permission is granted to copy, distribute and/or modify this document
@c under the terms of the GNU Free Documentation License, Version 1.3 or
@c any later version published by the Free Software Foundation; with no
@c Invariant Sections, with the Front and Back-Cover texts at your option. 


The following sections describe procedures for upgrading
between the consecutive Mailfromd releases. The absence of a section for
a pair of versions @var{x}-@var{y} numbers means that no specific
actions are required for upgrading from @var{x} to @var{y}.

@menu
* 8170-9000:: Upgrading from 8.17 to 9.00
* 8160-8170:: Upgrading from 8.16 to 8.17
* 8140-8150:: Upgrading from 8.14 to 8.15
* 8130-8140:: Upgrading from 8.13 to 8.14
* 870-880::  Upgrading from 8.7 to 8.8
* 850-860::  Upgrading from 8.5 to 8.6
* 820-830::  Upgrading from 8.2 to 8.3 (or 8.4)
* 700-800::  Upgrading from 7.0 to 8.0
* 600-700::  Upgrading from 6.0 to 7.0
* 5x0-600::  Upgrading from 5.x to 6.0
* 500-510::  Upgrading from 5.0 to 5.1
* 440-500::  Upgrading from 4.4 to 5.0
* 43x-440::  Upgrading from 4.3.x to 4.4
* 420-43x::  Upgrading from 4.2 to 4.3.x
* 410-420::  Upgrading from 4.1 to 4.2
* 400-410::  Upgrading from 4.0 to 4.1
* 31x-400::  Upgrading from 3.1.x to 4.0
* 30x-31x::  Upgrading from 3.0.x to 3.1
* 2x-30x::   Upgrading from 2.x to 3.0.x
* 1x-2x::    Upgrading from 1.x to 2.x
@end menu

@node 8170-9000
@appendixsec Upgrading from 8.17 to 9.00
@cindex Upgrading from 8.17 to 9.00

@subsubheading Old @samp{.mf} suffix gone

The @samp{.mf} suffix, that has long been used for MFL files and
has been deprecated since version 8.15 (@pxref{8140-8150}) is no
longer supported.  MFL files have suffix @samp{.mfl}.

@subsubheading Begin and end handlers must be defined using @code{prog}

The old syntax was:

@example
@group
begin
do
  ...
end
@end group
@end example

It is no longer supported.  The correct way to declare @code{begin}
and @code{end} handlers is

@example
@group
prog begin
do
  ...
end
@end group
@end example

@subsubheading Module and include search paths

Since version 8.15 if a module was not found in module search path,
the search was retried using include search path.  This is no longer
the case: the two paths serve different purposes and don't interact
in any way.  MFL modules are searched in module search path only.

@subsubheading IPv6

Version 9.0 features full IPv6 support.

The following modules used in previous releases are now deprecated.
They are retained for backward compatibility.  Any existing code that
uses any of these will compile and work as in previous releases, except
that a warning will be printed to draw your attention to the fact.
You are advised to remove uses of these modules, as they will be
removed in future versions:

@table @code
@item match_cidr
@flindex match_cidr.mfl
The function @code{match_cidr} is reimplemented as a built-in
function.  Remove any instances of @code{require 'match_cidr'} or
any equivalent @code{import} statements from your code.

@xref{match_cidr}.

@item is_ip
@flindex is_ip.mfl
This module defines the @code{is_ip} function:

@deftypefn {Library Function} boolean is_ip (string @var{str})
Returns 1 if @var{str} is a string representation of an IPv4 address
and 0 otherwise.
@end deftypefn

This function has been obsoleted by @code{is_ip4str} function.
You are advised to use it, or a more general @code{is_ipstr},
function instead.  @xref{Internet address manipulation
functions}, for a detailed discussion of these functions,

@anchor{revip}
@item revip
@flindex revip.mfl
This module defines the following function:

@deftypefn {Library Function} string revip (string @var{ip})
Reverses octets in @var{ip}, which must be a valid string
representation of an @acronym{IP}v4 address.
@end deftypefn

This function has been obsoleted by @code{reverse_ipstr}, discussed
in @ref{Internet address manipulation functions}.
@end table

@node 8160-8170
@appendixsec Upgrading from 8.16 to 8.17

No special actions are required after upgrading.  However, one point
is worth noticing.  Until now, it was an error to specify two
@code{prog} definitions with the same handler name.  This rule
however, applied only to generic milter handlers, special handlers
(@code{begin}, @code{end}, @code{startup}, @code{shutdown}) being exempt
from it.  Starting with version 8.17, multiple @code{prog}
declarations with the same handler name are allowed for all handlers.
@xref{Multiple handler definitions}, for details.

@node 8140-8150
@appendixsec Upgrading from 8.14 to 8.15
@cindex Upgrading from 8.14 to 8.15

This release introduces certain changes that require the installer's
attention.

@subsubheading MFL file suffix changed

MFL source files now have the suffix @samp{.mfl} instead of
@samp{.mf} which was used in previous versions.  This is to avoid
confusion with Metafont files, which also have suffix @samp{.mf}.

All installed MFL module files have been renamed.

As of version 8.15, this change is of advisory nature, and the
legacy @samp{.mf} suffix is still supported.  Thus, for example
if module @samp{X} was required and the file @samp{X.mfl} was not
found, @command{mailfromd} will look for file @samp{X.mf}.
Nevertheless, module authors are urged to rename their module
files accordingly.

In particular, the default script file is now named
@file{mailfromd.mfl}.  If it does not exist in the system
configuration directory, then the file @file{mailfromd.mf} is
looked up.  Installers are advised to rename @file{mailfromd.mf}
to @file{mailfromd.mfl}.

@subsubheading MFL module search path

Until now, @command{mailfromd} searched for module files in
@dfn{include search path}, which was mostly due to historical
reasons.  Version 8.15 introduces a separate @dfn{module search
path} for this purpose.  @xref{module search path}, for details.

All modules shipped with @command{mailfromd} are installed to
the new location: @file{@var{prefix}/share/mailfromd/8.15}.

For backward compatibility, if unable to find module file in
the module search path, @command{mailfromd} retries the search
using include search path, so that existing installation will still
work.  A warning message is issued for each module file found in
include search path, to help you locate files that need moving
to the new place.

This additional search phase will remain active during a transitional
period of couple of new releases to help users accommodate to
the change.

@subsubheading Special handlers

Special handlers (@pxref{begin/end}) are defined using the
same syntax as standard milter stage handlers, i.e.:

@example
@group
prog begin
do
  @dots{}
done
@end group
@end example

Old syntax (without the @code{prog} keyword) is still supported, but
causes a deprecation warning.  Module writers are advised to update
their sources accordingly.

Two new special handlers are provided.  @xref{startup/shutdown}, for
details.

@subsubheading Deprecated features removed

Deprecated configuration statements @code{lock-retry-count} and
@code{lock-retry-timeout} were removed in this version.  Please
use the @code{locking} statement instead.  @xref{locking statement,
Mailutils Configuration File,,mailutils, GNU Mailutils Manual}, for
a detailed decription.  To make the long story short:

@multitable @columnfractions .50 .50
@headitem Deprecated keyword @tab Use this instead
@item @code{lock-retry-count} @tab @code{retry-count}
@item @code{lock-retry-timeout} @tab @code{retry-sleep}
@end multitable

@noindent
So that, instead of

@example
@group
lock-retry-count 10;
lock-retry-timeout 1;
@end group
@end example

@noindent
you would write:

@example
@group
locking @{
  retry-count 10;
  retry-sleep 1;
@}
@end group
@end example

@node 8130-8140
@anchor{813-814} @c to not break the existing links
@appendixsec Upgrading from 8.13 to 8.14
@cindex Upgrading from 8.13 to 8.14

No special actions are required.  There are several important changes
in version 8.14, most notable ones being improved @code{dkim_sign}
function, DNS resolver tolerance for CNAME chains, and buffered I/O.

@subheading dkim_sign

This change is important for those who use @code{dkim_sign} with
Sendmail.  @xref{dkim_sign and sendmail}, for a detailed discussion.

@subheading CNAME chains

Mailfromd dropped support for CNAME chains in version 8.3
(@pxref{820-830}).  This change was forced by the behavior of
the @code{adns} resolver library, which had apparently been grounded
on somewhat rigorous reading of @acronym{RFC}s
1034@footnote{@uref{https://www.rfc-editor.org/rfc/rfc1034#section-3.6.2}}
and
2181@footnote{@uref{https://www.rfc-editor.org/rfc/rfc2181#section-10.1}}.
Reportedly, many existing DNS servers still employ CNAME chains leading to
TXT records (in particular, to DKIM TXT records), so this change made certain
DKIM signatures unverifiable for @command{mailfromd}.

To fix this, starting from version 8.14 @command{mailfromd} implements
a work-around that allows it to follow CNAME chains of limited
length.  The default length is 2 (which means CNAME pointing to a
CNAME, pointing to a valid RR).  It can be changed using the
@code{dns.max-cname-chain} configuration statement
(@pxref{conf-resolver}).

@subheading I/O buffering

Buffered I/O can tremendously improve performance of @code{getdelim}
and @code{getline}.

The global variables @code{io_buffering} and @code{io_buffer_size}
(@pxref{io_buffering}) define buffering mode and associated buffer
size for file descriptors returned by the subsequent calls to
@code{open} or @code{spawn}.  Buffering mode of an already open file
descriptor can be changed using the @code{setbuf} function.

The @code{io_buffering} variable defines the buffering mode.  By
default it is 0 (@code{BUFFER_NONE}), which disables buffering for
backward compatibility with the previous versions.  Another
possible values are: 1 (@code{BUFFER_FULL}) and 2 (@code{BUFFER_LINE}).

When set to @code{BUFFER_FULL}, all I/O operations become fully buffered.
The buffer size is defined by the @code{io_buffer_size} global variable.

@code{BUFFER_LINE} is similar to @code{BUFFER_FILE} when used for
input.  When used for the output, the data are accumulated in buffer
and actually sent to the underlying transport stream when the newline
character is seen.  The @code{io_buffer_size} global variable sets the
initial value for the buffer size in this mode.  The actual size can
grow as needed during the I/O.

The default value for @code{io_buffer_size} is the size of the system page.

The symbolic constants @code{BUFFER_NONE}, @code{BUFFER_FULL} and
@code{BUFFER_LINE} are defined in the @file{status.mf} module.  E.g.:

@example
@group
  require status

  begin
  do
    io_buffering = BUFFER_FULL
  done
@end group
@end example

will set up all the subsequent buffer-aware @ref{I/O functions}
for buffered reads and writes using the maximum buffer capacity.

The @code{setbuf} function (@pxref{setbuf}) changes the buffering
mode and/or buffer size for an already opened stream, e.g.:

@example
  setbuf(fd, BUFFER_FULL, 4096)
@end example

  For detailed discussion of various buffering modes and their effect
on the I/O, see @ref{I/O functions}.

@node 870-880
@appendixsec Upgrading from 8.7 to 8.8
@cindex Upgrading from 8.7 to 8.8

  DKIM support (@pxref{DKIM}) introduced in this version requires the
Nettle cryptographic
library@footnote{@uref{http://www.gnu.org/software/nettle}}.  It you
need DKIM, make sure Nettle is installed prior to compiling
@command{mailfromd}.  Otherwise, no special actions are required.

@node 850-860
@appendixsec Upgrading from 8.5 to 8.6
@cindex Upgrading from 8.5 to 8.6

  New configure option @option{--with-dbm} allows you to select any
DBM flavor supported by GNU mailutils as the default DBM implementation
for @command{mailfromd}.

@node 820-830
@appendixsec Upgrading from 8.2 to 8.3 (or 8.4)
@cindex Upgrading from 8.2 to 8.3
@cindex Upgrading from 8.2 to 8.4

  Versions 8.3 and 8.4 differ only in required minimal version of
mailutils (3.3 and 3.4, correspondingly).  Apart from that, the
following instructions apply to both versions.

  In version 8.3 I abandoned the legacy DNS resolver and switched to 
@dfn{GNU adns}.  GNU ands is a standalone resolver library, which provides
a number of advanced features.  It is included in most distributions.  The 
source code of the recent release is available from
@uref{http://www.chiark.greenend.org.uk/~ian/adns/adns.tar.gz}.

  This change brought a number of user-visible changes.  In particular,
arbitrary limits on the sizes of the RRs are removed.  Consequently,
the following configurations statements are withdrawn:

@table @code
@item runtime.max-dns-reply-a

@item runtime.max-dns-reply-ptr

@item runtime.max-dns-reply-mx

@item max-match-mx

@item max-callout-mx
@end table

  Secondly, the new resolver is less tolerant to deviations from the
standard.  This means that remote DNS misconfigurations that would have
slipped unnoticed in previous versions, will be noticed by mailfromd
8.3.  For example, a CNAME record pointing to another CNAME is treated
as an error.

  A new command line option was added to @command{mailfromd} and
@command{calloutd}: @option{--resolv-conf-file}.  This option instructs
the programs to read resolver settings from the supplied file, instead
of the default @file{/etc/resolv.conf}.

  Another user-visible change is in handling of SPF checks.  Previously,
results if SPF checks were cached in a database.  This proved to cause
more problems than solutions and was removed in this version.  As a
result, the following MFL global variables have been withdrawn:

@table @code
@item spf_ttl

@item spf_cached

@item spf_database

@item spf_negative_ttl
@end table

@node 700-800
@appendixsec Upgrading from 7.0 to 8.0
@cindex Upgrading from 7.0 to 8.0

  Version 8.0 is a major rewrite, that introduces a lot of new
concepts and features.  Nevertheless, it is still able to run
the MFL scripts from version 7.0 without modifications.

Note the following important points:

@itemize @bullet
@item The @code{listen} configuration statement withdrawn

Use the @code{server milter} statement instead.  @xref{conf-server}.

@item The @option{--remove} option withdrawn

This option was a noop since version 7.0.91.

@item The use of @samp{%} before variable names is no longer supported

The @samp{%} characters is used as modulo operator.  @xref{Arithmetic
operations}.

@item The @code{debug_spec} built-in function changed signature.

@xref{debug_spec}.

@item @code{listens} and @code{portprobe}

The @code{listens} function was moved to the @file{portprobe} module.  It is
actually an alias to the @code{portprobe} function.  If your filter uses
@code{listens}, make sure to @code{require} the @file{portprobe}
module.

@xref{Special test functions, portprobe}.

@item @code{_pollhost}, @code{_pollmx}, @code{stdpoll}, @code{strictpoll}

These functions have been moved to the @file{poll} module, which must be
required prior to using any of them.

@item The @code{message_header_count} function.

This function takes an optional string argument, supplying the header
name.  @xref{message_header_count}.
@end itemize

@node 600-700
@appendixsec Upgrading from 6.0 to 7.0
@cindex Upgrading from 6.0 to 7.0

  The release 7.0 removes the features which were declared as obsolete
in 6.0 and introduces important new features, both syntactical, at the
@acronym{MFL} level, and operational.

  Unless your filter used any deprecated features, it should work
correctly after upgrade to this version.  It will, however, issue
warning messages regarding the deprecated features (e.g. the use
of @samp{%} in front of identifiers, as described below).  To
fix these, follow the upgrade procedure described in @ref{upgrade
procedure}.

  The removed features are:

@itemize @bullet
@item Old-style functional notation
@item The use of functional operators
@item Implicit concatenations
@item #pragma option
@item #pragma database
@end itemize

  The @acronym{MFL} syntax has changed: it is no longer necessary to
use @samp{%} in front of a variable to get its value.  To reference a
variable, simply use its name, e.g.: 

@example
set x var + z
@end example

  The old syntax is still supported, so the following statement
will also work:

@example
set x %var + %z
@end example

  It will, however, generate a warning message.   

  Of course, the use of @samp{%} to reference variables within a string
literal remains mandatory.

  Another important changes to @acronym{MFL} are user-defined
exceptions (@pxref{User-defined Exceptions}) and the @dfn{try--catch}
construct (@pxref{Catch and Throw,try--catch}).

  Several existing @acronym{MFL} functions have been improved.  In
particular, it is worth noticing that the @code{open} function, when
opening a pipe to or from a command, provides a way to control where
the command's standard error would go (@pxref{open, standard error}).

  The @code{accept} function (or action) issues a warning if its use would
cancel any modifications to the message applied by, e.g.,
@code{header_add} and similar functions.  @xref{Message modification
queue}, for a detailed discussion of this feature.

  The most important change in @command{mailfromd} operation is that
the version 7.0 is able to run several servers (@pxref{conf-server}).
Dedicated @dfn{callout servers} make it possible to run sender
verifications in background, using a set of long timeouts, as
prescribed by RFC 2822 (@pxref{SMTP Timeouts}).  This diminishes
the number of false positives, allows for coping with servers showing
large delays and also reduces the number of callouts performed for
such servers.

  This release no longer includes the @command{smap} utility.  It was
moved into a self-standing project, which by now provides much more
functionality and is way more flexible than this utility was.  If you
are interested in using @command{smap}, visit
@uref{http://www.gnu.org.ua/software/smap}, for a detailed
information, including pointers to file downloads.

@node 5x0-600
@appendixsec Upgrading from 5.x to 6.0
@cindex Upgrading from 5.x to 6.0

  The 6.0 release is aimed to fix several logical inconsistencies that
affected the previous versions.  The most important one is that until
version 5.2, the filter script file contained both the actual filter
script, and the run-time configuration for @command{mailfromd} (in
form of @samp{#pragma option} and @samp{#pragma database} statements).
The new version separates run-time configuration from the
filter script by introducing a special configuration file
@file{mailfromd.conf} (@pxref{Mailfromd Configuration}).

  Consequently, the @samp{#pragma option} and @samp{#pragma database}
statements become deprecated.  Furthermore, the following deprecated
pragmas are removed: @samp{#pragma option ehlo}, @samp{#pragma option mailfrom}.
These pragmas became deprecated in version 4.0 (@pxref{31x-400}).

  The second problem was that the default filter script file had
@samp{.rc} suffix, which usually marks a configuration file, not
the source.  In version 6.0 the script file is renamed to
@file{mailfromd.mf}.  In the absence of this file, the legacy file
@file{mailfromd.rc} is recognized and parsed.  This ensures backward
compatibility.

  This release also fixes various inconsistencies and dubious features
in the @acronym{MFL} language.    
 
  The support for unquoted literals is discontinued.  This feature
was marked as deprecated in version 3.0.

@anchor{deprecated-6.0}
  The following features are deprecated: @samp{#pragma option}
(@histref{6,pragma_002doption,pragma-option} and @samp{#pragma database}
(@histref{6,pragma_002ddatabase,pragma-database}) directives, the
legacy style of function declarations
(@histref{6,old_002dstyle-function-declarations, 
old-style function declarations}), calls to
functions of one argument without parentheses
(@histref{6,implicit-concatenation, operational
notation}), the @samp{#require} statement (@xref{import}, for the
new syntax) and implicit concatenation
(@histref{6,implicit-concatenation, implicit concatenation}).  See @histref{6,Deprecated-Features,Deprecated Features}, for more information
about these.

  This release also introduces important new features, which are
summarized in the table below:

@multitable @columnfractions .50 .50
@headitem Feature            @tab Reference
@item Configuration          @tab @xref{Mailfromd Configuration}.
@item Module system          @tab @xref{Modules}.
@item Explicit type casts    @tab @xref{explicit type casts}.
@item Concatenation operator @tab @xref{Concatenation}.
@item Scope of visibility    @tab @xref{scope of visibility}.
@item Precious variables     @tab @xref{rset}.
@end multitable

  @command{Mailfromd} version @samp{6.0} will work with unchanged
scripts from @samp{5.x}.  When started, it will verbosely warn you
about any deprecated constructs that are used in your filter sources
and will create a script for upgrading them.

@anchor{upgrade procedure}
  To upgrade your filter scripts, follow the steps below:

@enumerate 1
@item
Run @samp{mailfromd --lint}.  You will see a list of warnings similar
to this:

@example
mailfromd: Warning: using legacy script file
 /usr/local/etc/mailfromd.rc
mailfromd: Warning: rename it to /usr/local/etc/mailfromd.mf
 or use script-file statement in /usr/local/etc/mailfromd.conf
 to disable this warning
mailfromd: /usr/local/etc/mailfromd.rc:19: warning: this pragma is
 deprecated: use relayed-domain-file configuration statement instead
mailfromd: /usr/local/etc/mailfromd.rc:23: warning: this pragma is
 deprecated: use io-timeout configuration statement instead
mailfromd: Info: run script /tmp/mailfromd-newconf.sh
 to fix the above warnings
@dots{} 
@end example

@item
  At the end of the run @command{mailfromd} will create a shell script
@file{/tmp/mailfromd-newconf.sh} for fixing these warnings.  Run it:

@example
$ sh /tmp/mailfromd-newconf.sh
@end example


@item
  When the script finishes, run @command{mailfromd --lint} again.  If
it shows no more deprecation warnings, the conversion went correctly.
Now you can remove the upgrade script:

@example
$ rm /tmp/mailfromd-newconf.sh
@end example
@end enumerate

  Notice, that the conversion script attempts to fix only deprecation
warnings.  It will not try to correct any other type of warnings or
errors.  For example, you may get warning messages similar to:

@example
mailfromd: /etc/mailfromd.mf:7: warning: including a module file is unreliable and may cause subtle errors
mailfromd: /etc/mailfromd.mf:7: warning: use `require dns' instead
@end example

This means that you use @samp{#include} where you should have used
@samp{require}.  You will have to fix such warnings manually, as
suggested in the warning message.

  If, for some reason, you cannot upgrade your scripts right now, you
may suppress deprecation warnings by setting the environment variable
@env{MAILFROMD_DEPRECATION} to @samp{no} before starting
@command{mailfromd}.  Nonetheless, I recommend to upgrade as soon as
possible, because the deprecated features will be removed in version
@samp{6.1}.

@node 500-510
@appendixsec Upgrading from 5.0 to 5.1
@cindex Upgrading from 5.0 to 5.1

  Upgrading from 5.0 to 5.1 does not