Register the "i" macro for use in the lowest SMTP state declared in the program.

* mfd/engine.c (first_used_state): New variable.
(mlfi_negotiate): If first_used_state is set, register "i"
macro for it.
(milter_enable_state): Set first_used_state.
* mfd/gram.y (register_macro): Remove static qualifier.

* mfd/builtin/macro.bi (miltermacros): New pragma.

* mfd/lex.l (paren_follows): Remove tie-in variable.
* mfd/mailfromd.h (register_macro): New proto.
(paren_follows): Remove extern.

* NEWS: Update.
* doc/mailfromd.texi: Document #pragma miltermacros.
* doc/functions.texi: Likewise.

4 files changed, 148 insertions, 77 deletions

diff --git a/NEWS b/NEWS
index 28963c0d..b2d364b4 100644
--- a/NEWS
+++ b/NEWS
@@ -1,4 +1,4 @@
-Mailfromd NEWS -- history of user-visible changes. 2010-07-22
+Mailfromd NEWS -- history of user-visible changes. 2010-08-02
 Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010 Sergey Poznyakoff
 See the end of file for copying conditions.
 
@@ -136,6 +136,17 @@ value of I/O timeout (see the `io-timeout' configuration
 statement and the `--timeout' command line option), which
 defaults to 3s.
 
+* Automatic export of the "i" macro.
+
+The "i" Sendmail macro is automatically requested for the lowest
+used stage handler.  This ensures the log messages are marked with
+the corresponding queue ID.
+
+Notice, however, that the MTA is free to honor or ignore the request.
+Generally speaking, this works well with Sendmail starting from 8.14.0
+and MeTA1 starting from 1.0.PreAlpha29.0.  The "i" macro is alost useless
+with Postfix 2.6 or higher, because it defines it only at the EOM stage.
+
 * Language changes
 
 ** Use of % in front of identifiers
@@ -247,7 +258,17 @@ If you see this diagnostics in your log, do the following:
  - if it was not, read the manual, subsection `4.12.1.8 Message
    Modification Queue', for information on how to handle it.
    FIXME: section number is subject to change.
- 
+
+** #pragma miltermacros
+
+The new `#pragma miltermacros' declares macros that are referenced at a
+given stage.  It is helpful when automatic macro negotiation 
+is in use and mailfromd is unable to trace all macros referenced from
+each handler, i.e. when at least one of the handlers:
+
+- calls functions that reference MTA macros;
+- refers to macros via macro_defined() and getmacro() functions. 
+   
 ** MFL functions
 
 *** progress
diff --git a/doc/functions.texi b/doc/functions.texi
index 4d754b72..17804684 100644
--- a/doc/functions.texi
+++ b/doc/functions.texi
@@ -67,6 +67,37 @@ macro names programmatically, e.g.:
 Return true if Sendmail macro @var{name} is defined.
 @end deftypefn
 
+  Notice, that if your @acronym{MTA} supports macro name
+negotiation@footnote{That is, if it supports Milter protocol 6 and
+upper.  Sendmail 8.14.0 and Postfix 2.6 and newer do.  MeTA1 (via
+@command{pmult}) does as well.  @xref{MTA Configuration}, for more
+details.}, you will have to export macro names used by these two
+functions using @samp{#pragma miltermacros} construct.  Consider this
+example:
+
+@smallexample
+func authcheck(string name)
+do
+  string macname "auth_%name"
+  if macro_defined(macname)
+    if getmacro(macname)
+      @dots{}
+    fi
+  fi
+done
+
+#pragma miltermacros envfrom auth_authen
+
+prog envfrom
+do
+  authcheck("authen")
+done
+@end smallexample
+
+In this case, the parser cannot deduce that the @code{envfrom} handler
+will attempt to reference the @samp{auth_authen} macro, therefore the
+@samp{#pragma miltermacros} is used to help it.
+
 @node String manipulation
 @section String Manipulation Functions
 @deftypefn {Built-in Function} string domainpart (string @var{str})
diff --git a/doc/mailfromd.texi b/doc/mailfromd.texi
index 353744c3..9384edf0 100644
--- a/doc/mailfromd.texi
+++ b/doc/mailfromd.texi
@@ -3790,7 +3790,8 @@ tabulations or newlines).
   The following sections describe @acronym{MFL} syntax in detail.
 
 @menu
-* Comments::                    Usual and pragmatic comments.
+* Comments::                    Comments.
+* Pragmas::                     Pragmatic comments.
 * Data Types::                  
 * Numbers::                     
 * Literals::                    
@@ -3926,7 +3927,8 @@ diagnostics, that the line number of the next source line is given by
 @var{number} and the current input file is named by @var{identifier}.
 If the identifier is absent, the remembered file name does not change.
 
-@anchor{Pragmatic comments}
+@node Pragmas
+@section Pragmatic comments
 @cindex pragmatic comments
 @cindex #pragma statement
 @kwindex #pragma
@@ -3938,48 +3940,47 @@ configuration setting.
   The available pragma types are described in the following subsections.
 
 @menu
-* stacksize::   Pragma stacksize.
-* regex::       Pragma regex.
-* dbprop::      Pragma dbprop.
-* greylist::    Pragma greylist.
+* stacksize::    Pragma stacksize.
+* regex::        Pragma regex.
+* dbprop::       Pragma dbprop.
+* greylist::     Pragma greylist.
+* miltermacros:: Pragma miltermacros.
 @end menu
 
 @node stacksize
 @subsection Pragma stacksize
-@cindex stacksize, pragma
-@cindex #pragma stacksize
-
-@subheading Syntax
-@smallexample
-#pragma stacksize @var{size}
-#pragma stacksize @var{size} twice|@var{incr}
-#pragma stacksize @var{size} twice|@var{incr} @var{max}
-@end smallexample
 
-@subheading Description
   The @code{stacksize} pragma sets the initial size of the run-time
 stack and may also define the policy of its growing, in case it
-becomes full.  It takes one mandatory and two optional arguments.  The
-first argument gives the desired initial stack size, in words.  It
-defaults to @value{STACK-SIZE}.  You may wish to increase this number
-if your configuration program uses recursive functions or does an
-excessive amount of string manipulations.  For example:
-
+becomes full.  The default stack size is @value{STACK-SIZE}.  You may
+need to increase this number if your configuration program uses
+recursive functions or does an excessive amount of string manipulations.
+
+@deffn {pragma} stacksize size [incr [max]]
+  Sets stack size to @var{size} units.  Optional @var{incr} and
+@var{max} define stack growth policy (see below).  The default
+@dfn{units} are words.  The following example sets the stack 
+size to 7168 words:
+  
 @smallexample
 #pragma stacksize 7168
 @end smallexample
 
-The size may end with a usual size suffix, i.e. one of:
+The @var{size} may end with a @dfn{unit size} suffix:
 
+@float Table, unit-size
+@caption{Unit Size Suffix}
 @multitable @columnfractions 0.3 0.6
-@headitem Suffix @tab Meaning
+@headitem Suffix @tab Meaning 
 @item k @tab Kiloword, i.e. 1024 words
 @item m @tab Megawords, i.e. 1048576 words
 @item g @tab Gigawords, 
 @item t @tab Terawords (ouch!)
 @end multitable
+@end float
 
-File suffixes are case-insensitive:
+File suffixes are case-insensitive, so the following two pragmas are
+equivalent and set the stack size to @code{7*1048576 = 7340032} words:
 
 @smallexample
 @group
@@ -3996,22 +3997,27 @@ execution of the filter stops.
 
 @cindex stack growth policy
 @cindex growth policy, stack
-Optional second argument to @code{#pragma stacksize} defines growth
-policy for the stack.  A positive number in its place is taken for the
-@dfn{expansion chunk} size.  In this case, the stack will be grown in
-blocks that contain integer number of expansion chunks.  The following
-example sets initial stack size to 10240, and expansion chunk size to
-2048 words: 
+The optional @var{incr} argument to @code{#pragma stacksize} defines growth
+policy for the stack.  Two growth policies are implemented:
+@dfn{fixed increment policy}, which expands stack in a fixed
+number of @dfn{expansion chunks}, and @dfn{exponential growth policy}, which
+duplicates the stack size until it is able to accomodate the needed
+number of words.  The fixed increment policy is the default.  The default
+chunk size is @value{STACK-INCR}.
+
+  If @var{incr} is the word @samp{twice}, the duplicate policy is
+selected.  Otherwise @var{incr} must be a positive number optionally
+suffixed with a size suffix (see above).  This indicates the expansion
+chunk size for the fixed increment policy.
+
+  The following example sets initial stack size to 10240, and
+expansion chunk size to 2048 words: 
 
 @smallexample
 #pragma stacksize 10M 2K
 @end smallexample
 
-This @dfn{incremental growth policy} is the default.  The default
-chunk size is @value{STACK-INCR}.
-
-The word @samp{twice} in the second argument enables @dfn{exponential
-stack growth} policy:
+  The pragma below enables exponential stack growth policy:
 
 @smallexample
 #pragma stacksize 10240 twice
@@ -4020,13 +4026,14 @@ stack growth} policy:
 In this case, when the run-time evaluator hits the stack size limit,
 it expands the stack to twice the size it had before.  So, in the
 example above, the stack will be sequentially expanded to the
-following sizes: 20480, 40960, 81920, 163840.
+following sizes: 20480, 40960, 81920, 163840, etc.
 
-Third argument to @code{#pragma stacksize} defines maximum size of
+  The optional @var{max} argument defines the maximum size of
 the stack.  If stack grows beyond this limit, the execution of the
 script will be aborted.
+@end deffn
 
-If you are concerned about the execution time of your script, you
+  If you are concerned about the execution time of your script, you
 may wish to avoid stack reallocations.  To help you find out the
 optimal stack size, each time the stack is expanded,
 @command{mailfromd} issues a warning in its log file, which looks like
@@ -4041,28 +4048,21 @@ settings.
 
 @node regex
 @subsection Pragma regex
-@cindex regex, pragma
-@cindex #pragma regex
 @anchor{pragma regex}
-  Another pragmatic comment, @samp{#pragma regex}, controls
+  The @samp{#pragma regex}, controls
 compilation of expressions.  You can use any number
 of such pragma directives in your @file{mailfromd.mf}.  The scope of
 @samp{#pragma regex} extends to the next occurrence of this directive
 or to the end of the script file, whichever occurs first.
 
-  The syntax of this pragma is:
-
-@smallexample
-#pragma regex [push|pop] [@var{list-of-flags}]
-@end smallexample
-
-@noindent
-where @code{push} and @code{pop} are optional sub-commands discussed in
-detail below, and @var{list-of-flags} is a whitespace-separated list
-of @dfn{regex flags}.  Each regex-flag is a word specifying some regex
-feature.  It can be preceded by @samp{+} to enable this feature (this is the
-default), by @samp{-} to disable it or by @samp{=} to reset regex
-flags to its value.  Valid regex-flags are:
+@deffn {pragma} regex [push|pop] flags
+  The optional @var{push|pop} parameter is one of the words @samp{push} or
+@samp{pop} and is discussed in detail below.  The @var{flags}
+parameter is a whitespace-separated list of @dfn{regex flags}.  Each
+regex-flag is a word specifying some regex feature.  It can be
+preceded by @samp{+} to enable this feature (this is the default), by
+@samp{-} to disable it or by @samp{=} to reset regex flags to its
+value.  Valid regex-flags are:
 
 @table @samp
 @item extended
@@ -4094,8 +4094,9 @@ case insensitive matching (a good thing to start your
 @smallexample
 #pragma regex +extended +icase
 @end smallexample
+@end deffn
 
-Optional modifiers @code{push} and @code{pop} can be used to maintain
+Optional modifiers @samp{push} and @samp{pop} can be used to maintain
 a stack of regex flags.  The statement
 
 @smallexample
@@ -4131,26 +4132,17 @@ regex settings.  E.g.:
 
 @node dbprop
 @subsection Pragma dbprop
-@cindex dbprop, pragma
-@cindex #pragma dbprop
-
-@smallexample
-#pragma dbprop @var{pattern} @var{prop} [@var{prop}]
-@end smallexample
 
-This pragma configures properties for a @acronym{DBM} database.
+@deffn {pragma} dbprop pattern prop @dots{}
+  This pragma configures properties for a @acronym{DBM} database.
 @xref{Database functions}, for its detailed description.
+@end deffn
 
 @node greylist
 @subsection Pragma greylist
-@cindex greylist, pragma
-@cindex #pragma greylist
-
-@smallexample
-#pragma greylist @var{type}
-@end smallexample
 
-  This pragma selects the greylisting implementation to use.  Allowed
+@deffn {pragma} greylist type
+  Selects the greylisting implementation to use.  Allowed
 values for @var{type} are:
 
 @table @asis
@@ -4163,13 +4155,40 @@ Use the traditional greylisting implementation.  This is the default.
 Use Con Tassios greylisting implementation.
 @end table
 
-@xref{greylisting types}, for a detailed description of available
+@xref{greylisting types}, for a detailed description of these
 greylisting implementations.
+@end deffn
 
-Notice, that this pragma can be used only once.  Second use of this
-pragma constitutes an error, because you cannot use both greylisting
+Notice, that this pragma can be used only once.  A second use of this
+pragma would constitute an error, because you cannot use both greylisting
 implementations in the same program.
 
+@node miltermacros
+@subsection Pragma miltermacros
+@deffn {pragma} miltermacros handler macro @dots{}
+Declare that the Milter stage @var{handler} uses @acronym{MTA} macro
+listed as the rest of arguments.  The @var{handler} must be a valid
+handler name (@pxref{Handlers}).
+@end deffn
+
+  The @command{mailfromd} parser collects the names of the macros
+referred to by a @samp{$@var{name}} construct within a handler
+(@pxref{Sendmail Macros}) and declares them automatically for
+corresponding handlers.  It is, however, unable to track macros
+used in functions called from handler as well as those referred to
+via @code{getmacro} and @code{macro_defined} functions.  Such
+macros should be declared using @samp{#pragma miltermacros}.
+
+  During initial negotiation with the @acronym{MTA},
+@command{mailfromd} will ask it to export the macro names declared
+automatically or by using the @samp{#pragma miltermacros}.  The
+@acronym{MTA} is free to honor or to ignore this request.  In
+particular, Sendmail versions prior to 8.14.0 and Postfix versions
+prior to 2.5 do not support this feature.  If you use one of these,
+you will need to export the needed macros explicitly in the
+@acronym{MTA} configuration.  For more details, refer to the section
+in @ref{MTA Configuration} corresponding to your @acronym{MTA} type.
+
 @node Data Types
 @section Data Types
 
diff --git a/mfd/builtin/macro.bi b/mfd/builtin/macro.bi
index 2300005f..cd63306e 100644
--- a/mfd/builtin/macro.bi
+++ b/mfd/builtin/macro.bi
@@ -14,8 +14,8 @@
    You should have received a copy of the GNU General Public License
    along with this program.  If not, see <http://www.gnu.org/licenses/>. */
 
-/* #pragma mta_state_macros <handler> <name> [name...] */
-MF_PRAGMA(mta_state_macros, 4, 0)
+/* #pragma miltermacros <handler> <name> [name...] */
+MF_PRAGMA(miltermacros, 4, 0)
 {
 	enum smtp_state state;
 	int i = 1;