manpagesection(6. SPECIFICATION FILE(S))

    Flc() expects an input file containing directives and the regular
expressions that should be recognized by objects of the scanner class
generated by flc(). In this man page the elements and organization of flc()'s
input file is described. 

    Flc()'s input file consists of two sections, separated from each other by
a line merely containing two consecutive percent characters:
    verb(
%%
    )
    The section before this separator contains directives; the section
following this separator contains regular expressions and possibly actions to
perform when these regular expressions are matched by the object of the
scanner class generated by flc(). 

    White space is usually ignored, as is comment, which may be of the
traditional bf(C) form (i.e., tt(/*), followed by (possibly multi-line)
comment text, followed by tt(*/), and it may be bf(C++) end-of-line comment:
two consecutive slashes (tt(//)) start the comment, which continues up to
the next newline character.

manpagesection(6.1. FILE SWITCHING)

Flc()'s input file may be split into multiple files. This allows for the
definition of logically separate elements of the specifications in different
files. Include directives must be specified on a line of their own. To
switch to another specification file the following stanza is used:
        verb(
//include file-location
        )
    The tt(//include) directive starts in the line's first column. File
locations can be absolute or relative to the location of the file containing
the tt(//include) directive. White space characters following tt(//include)
and before the end of the line are ignored. The file specification may be
surrounded by double quotes, but these double quotes are not required and are
ignored (removed) if present. All remaining characters are expected to define
the name of the file where flc()'s rules specifications continue. Once end of
file of a sub-file has been reached, processing continues at the line beyond
the tt(//include) directive of the previously scanned file. The end-of-file of
the file that was initially specified when flc() was called indicates the end
of flc()'s rules specification.

manpagesection(6.2. DIRECTIVES)

    The first section of flc()'s input file consists of directives. In
addition it may associate regular expressions with symbolic names, allowing
you to use these identifiers in the rules section. Each directive is defined
on a line of its own. When available, directives are overridden by flc()
command line options.

    includefile(../manual/lexer/directives.yo)

manpagesection(6.3. MINI SCANNERS)

    Mini scanners come in two flavors: inclusive mini scanners and exclusive
mini scanners. The rules that apply to an inclusive mini scanner are the mini
scanner's own rules as well as the rules which apply to no mini scanners in
particular (i.e., the rules that apply to the default (or tt(INITIAL)) mini
scanner). Exclusive mini scanners only use the rules that were defined for
them. 

    To define an inclusive mini scanner use tt(%s), followed by one
or more identifiers specifying the name(s) of the mini-scanner(s). To define
an exclusive mini scanner use tt(%x), followed by or more identifiers
specifying the name(s) of the mini-scanner(s). The following example defines
the names of two mini scanners: tt(string) and tt(comment): 
        verb(
    %x string comment 
        )
    Following this, rules defined in the context of the tt(string) mini
scanner (see below) will only be used when that mini scanner is active.

    A flc() input file may contain multiple tt(%s) and tt(%x)
specifications.

manpagesection(6.4. DEFINITIONS)

    Definitions are of the form
        verb(
identifier  regular-expression
        )
    Each definition must be entered on a line of its own. Definitions
associate identifiers with regular expressions, allowing the use of
tt(${identifier}) as synonym for its regular expression in the rules section
of the flc() input file. One defined, the identifiers representing regular
expressions can also be used in subsequent definitions.

Example:
        verb(
FIRST                   [A-Za-z_]
NAME                    {FIRST}[-A-Za-z0-9_]*
        )

manpagesection(6.5. %% SEPARATOR)

Following directives and definitions a line merely containing two consecutive
tt(%) characters is expected. Following this line the rules are defined. Rules
consist of regular expressions which should be recognized, possibly followed
by actions to be executed once a rule's regular expression has been matched.

manpagesection(6.6. REGULAR EXPRESSIONS)

The regular expressions defined in flc()'s rules files are matched against 
the information passed to the scanner's tt(lex) function. 

Regular expressions begin as the first non-blank character on a line. Comment
is interpreted as comment as long as it isn't part of the regular
expresssion. To define a  regular expression starting with two slashes (at
least) the first slash can be escaped or double quoted. (E.g., tt("//".*)
defines bf(C++) comment to end-of-line).

Regular expressions end at the first blank character (to add a blank character,
e.g., a space character, to a regular expression, prefix it by a backslash or
put it in a double-quoted string).

Actions may be associated with regular expressions.  At a match the action
that is associated with the regular expression is executed, after which
scanning continues when the lexical scanning function (e.g., tt(lex)) is
called again. Actions are not required, and regular expressions can be defined
without any actions at all. If such action-less regular expressions are
matched then the match is performed silently, after which processing
continues.

Flc() tries to match as many characters of the input file as possible (i.e.,
it uses `greedy matching'). Non-greedy matching is accomplished by a
combination of a scanner and parser and/or by using the `lookahead' operator
(tt(/)).

The following regular expression `building blocks' are available. More complex
regular expressions are created by combining them:

description(
dit(tt(x))
    the character `x'
dit(tt(.))
    any character (byte) except newline
dit(tt([xyz]))
   a character class; in this case, the pattern matches either an `x',
    a `y', or a `z'
dit(tt([abj-oZ]))
   a character class containing a range; matches an `a', a `b', any
    letter from `j' through `o', or a `Z'
dit(tt([^A-Z])) a negated character class, i.e., any character except
    for those in the class.  In this example, any non-capital character.
dit(tt("[xyz]\"foo"))
    text between double quotes matches the literal string: tt([xyz]"foo).
dit(tt(\X))
   if X is `a', `b', `f', `n', `r', `t', or `v', then the ANSI-C
    interpretation of `\x' is matched. Otherwise, a literal `X' is matched
    (this is used to escape operators such as `*').
dit(tt(\0))
    a NUL character (ASCII code 0).
dit(tt(\123))
    the character with octal value 123.
dit(tt(\x2a))
    the character with hexadecimal value 2a.
dit(tt((r)))
   the regular expression `r'; parentheses are used to override
    precedence (see below)
dit(tt({name}))
   the expansion of the `name' definition.
dit(tt(r*)) 
   zero or more regular expressions `r'. This also matches the empty
    string.
dit(tt(r+))
   one or more regular expressions `r'.
dit(tt(r?))
   zero or one regular expression `r'.  This also matches the empty
    string.
dit(tt(rs))
   the regular expression `r' followed by the regular expression `s';
    called concatenation
dit(tt(r{m, n})) 
   regular expression `r' at least m, but at most n times (tt(1 <= m
    <= n)).
dit(tt(r{m,}))
   regular expression `r' m or more times (tt(1 <= m)). 
dit(tt(r{m}))
   regular expression `r' exactly m times (tt(1 <= m)).
dit(tt(r|s))
   either regular expression `r' or regular expression `s'
dit(tt(r/s))
   regular expression `r' if it is followed by regular expression
    `s'. The text matched by `s' is included when determining whether this
    rule results in the longest match, but `s' is then returned to the input
    before the rule's action (if defined) is executed.
dit(tt(^r))
   a regular expression `r' at the beginning of a line or file.
dit(tt(r$))
   a regular expression `r', occurring  at the end of a line. This
    pattern is identical to `r/\n'.
dit(tt(<s>r))
   a regular exprression `r' in start condition `s'
dit(tt(<s1,s2,s3>r))
   a regular exprression `r' in start conditions s1, s2, or s3.
dit(tt(<*>r))
   a regular exprression `r' in all start conditions.
dit(tt(<)tt(<EOF>)tt(>))
   an end-of-file.
dit(tt(<s1,s2><)tt(<EOF>)tt(>))
   an end-of-file when in start conditions s1 or s2 
)

Inside a character class all regular expression operators lose their special
meanings, except for the escape character (tt(\)) and the character class
operators tt(-), tt(]]), and, at the beginning of the class, tt(^). To add a
closing bracket to a character class use tt([]). To add a closing bracket to a
negated character class use tt([^]). Once a character class has started, all
subsequent character (ranges) are added to the set, until the final closing
bracket (tt(])) has been reached.

The regular expressions listed above are grouped according to precedence, from
highest precedence at the top to lowest at the bottom. From lowest to highest
precedence, the operators are:
        itemization(
        itt(|): the or-operator at the end of a line (instead of an action)
indicates that this expression's action is identical to the action of the next
rule. 
        itt(/): the look-ahead operator;
        itt(|): the or-operator withn a regular expression;
        itt(CHAR): individual elements of the regular expression: characters,
strings, quoted characters, escaped characters, character sets etc. are all
considered tt(CHAR) elements. Multiple tt(CHAR) elements can be combined by
enclosing them in parentheses (e.g., tt((abc)+) indicates sequences of tt(abc)
characters, like tt(abcabcabc));
        itt(*, ?, +, {): multipliers:nl()
            tt(?): zero or one occurrence  of the previous element;nl()
            tt(+): one or more repetitions of the previous element;nl()
            tt(*): zero or more repetitions of the previous element;nl()
            tt({...}): interval specification: a specified number of
                repetitions of the previous element (see above for specific
                forms of the interval specification)
        itt({+}, {-}): set operators (tt({+}) computing the union of two sets,
                tt({-}) computing the difference of the left-hand side set
                minus the elements in the right-hand side set);
        )

The lex standard defines concatenation as having a higher precedence than the
interval expression. This is different from many other regular expression
engines, and flc() follows these latter engines, giving all `multiplication
operators' equal priority.

Name expansion has the same precedence as grouping (using parentheses to
influence the precedence of the other operators in the regular expression).
Since the name expansion is treated as a group in flc(), it is not allowed to
use the lookahead operator in a name definition (a named pattern, defined in
the definition section).

Character classes can also contain character class expressions. These are
expressions enclosed inside tt([:) and tt(:]) delimiters (which themselves
must appear between the tt([) and tt(]) of the character class. Other elements
may occur inside the character class as well). The character class expressions
are:
        verb(     
     [:alnum:] [:alpha:] [:blank:]
     [:cntrl:] [:digit:] [:graph:]
     [:lower:] [:print:] [:punct:]
     [:space:] [:upper:] [:xdigit:]
        )

    Character class expressions designate a set of characters equivalent to
the corresponding standard bf(C) isXXX function. For example, tt([:alnum:])
designates those characters for which tt(isalnum) returns true - i.e., any
alphabetic or numeric character.  For example, the following character classes
are all equivalent:
        verb( 
    [[:alnum:]]
    [[:alpha:][:digit:]]
    [[:alpha:][0-9]]
    [a-zA-Z0-9]
        )

A negated character class such as the example tt([^A-Z]) above will match a
newline unless tt(\n) (or an equivalent escape sequence) is one of the
characters explicitly present in the negated character class (e.g.,
tt([^A-Z\n])). This differs from the way many other regular expression tools
treat negated character classes, but unfortunately the inconsistency is
historically entrenched. Matching newlines means that a pattern like tt([^"]*)
can match the entire input unless there's another quote in the input.

Flc() allows negation of character class expressions by prepending tt(^) to
the POSIX character class name.
        verb(                
    [:^alnum:] [:^alpha:] [:^blank:]
    [:^cntrl:] [:^digit:] [:^graph:]
    [:^lower:] [:^print:] [:^punct:]
    [:^space:] [:^upper:] [:^xdigit:]
        )      

The tt({-}) operator computes the difference of two character classes. For
example, tt([a-c]{-}[b-z]) represents all the characters in the class
tt([a-c]) that are not in the class tt([b-z]) (which in this case, is just the
single character tt(a)). The tt({-}) operator is left associative, so
tt([abc]{-}[b]{-}[c]) is the same as tt([a]).

The tt({+}) operator computes the union of two character classes. For example,
tt([a-z]{+}[0-9]) is the same as tt([a-z0-9]). This operator is useful when
preceded by the result of a difference operation, as in,
tt([[:alpha:]]{-}[[:lower:]]{+}[q]), which is equivalent to tt([A-Zq]) in the
bf(C) locale.

A rule can have at most one instance of trailing context (the tt(/) operator
or the tt($) operator). The start condition, tt(^), and tt(<<EOF>>) patterns
can only occur at the beginning of a pattern, and cannot be surrounded by
parentheses. The characters tt(^) and tt($) only have their special properties
at, respectively, the beginning and end of regular expressions. In all other
cases they are treated as a normal characters.

manpagesection(6.7. SPECIFICATION EXAMPLE)

    verb(
%option debug

%x comment

NAME    [[:alpha:]][_[:alnum:]]*

%%

"//".*          // ignore

"/*"            begin(comment);

<comment>.|\n   // ignore
<comment>"*/"   begin(INITIAL);

^a              return 1;
a               return 2;
a$              return 3;
{NAME}          return 4;

.|\n            // ignore
        )

    )
