Copyright © 1999 - 2020  Free Software Foundation, Inc.

     Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, with the Front-Cover texts being ``A GNU Manual'', and with the Back-Cover Texts as in (a) below.  A copy of the license is included in the section entitled ``GNU Free Documentation License''.

     (a) The FSF's Back-Cover Text is: ``You have the freedom to copy and modify this GNU manual.  Buying copies from the FSF supports it in developing GNU and promoting software freedom.''

Top
***

Wisi Version 3.1.2

Overview
********

``wisi'' used to be an acronym, but now it's just a name.

The wisi package provides an elisp interface to an external parser. It assumes the parser generator package WisiToken (http://stephe-leake.org/ada/wisitoken.html, implemented in Ada), but can use any parser that meets the same API. wisi provides several grammar actions, to implement indentation, navigating, and syntax highlighting (fontification).

wisi also provides an extension to Emacs project.el, providing operations useful for compilation and cross-reference.

Grammar Actions
***************

Grammar actions are specified in the grammar file, in a nonterminal declaration. We assume the user is familiar with parser grammars and grammar actions. For example, a simple ``if'' statement can be declared as:

     if_statement
       : IF expression THEN statements elsif_list ELSE statements END IF SEMICOLON
         %((wisi-statement-action [1 statement-start 3 motion 6 motion 10 statement-end])
           (wisi-motion-action [1 3 5 6 10])
           (wisi-indent-action [nil
                                [(wisi-hanging% ada-indent-broken (* 2 ada-indent-broken))
                                 ada-indent-broken]
                                nil
                                [ada-indent ada-indent] nil nil
                                [ada-indent ada-indent] nil nil nil]))%

The item before : is the ``left hand side'', or ``nonterminal''. The list of tokens after : is the ``right hand side''; in general there can be more than one right hand side for each nonterminal (separated by |).

The items enclosed in ``%()%'' are the grammar actions. They are specified as list of elisp forms; an earlier version of the wisi package generated an elisp parser. We keep the elisp form because it is compact, and easier to read and write than the equivalent Ada code. The wisi-bnf-generate tool converts the elisp into the required Ada statements.

There are two classes of actions; in-parse and post-parse. WisiToken calls these ``semantic checks'' and ``user actions''. The in-parse actions are done as parsing procedes; they provide extra checks that can cause the parse to fail. Currently the only one provided is match-names; it is used to check that the declaration and end names in named Ada blocks are the same (which can aid significantly in error correction). In the grammar file, in-parse actions are specified in a second %()% block, which can be omitted if empty. In this document, the term ``action'' means ``post-parse action'', we use ``in-parse action'' unless the meaning is clear from context.

Executing the wisi grammar actions creates text properties in the source file; those text properties are then used by elisp code for various purposes.

The text properties applied are:

wisi-cache
     This should be named wisi-navigate, but isn't for historical reasons (there used to be only one kind of text property).

     The property contains a wisi-cache object, containing:

     nonterm
          The nonterminal in the grammar production that specified the action that produced this text property.

     token
          A token identifier naming a token in the production right hand side containing the text this text property is applied to.

     last
          The position of the last character in the token, relative to the first character (0 indexed). The text property is only applied to the first character in the token (mostly for historical reasons).

     class
          A token class; see the list of possible values in wisi-statement-action below.

     containing
          A marker pointing to the start of the containing token for this token; only nil for the outermost containing token in a file.

     prev
          A marker pointing to the previous ``motion token'' in the statement or declaration. These are normally language keywords, but can be other things.

     next
          A marker pointing to the next ``motion token'' in the statement or declaration.

     end
          A marker pointing to the end of the statement or declaration.

     wisi provides motion commands for going to the various markers.

wisi-name
     Contains no data, applied to a ``name'' of some sort. wisi provides commands for finding the next/previous name, and returning the text. Useful for the names of subprograms, which can then be used to build a completion table; see wisi-xref-identifier-completion-table.

font-lock-face
     The standard font-lock property, specifying the face for the text.

     Some major modes do not use this for simple keywords; they use font-lock regular expressions instead. One reason for this is so keywords are still highlighted when the parser fails, which can happen if there are severe syntax errors.

     Other items, like function and package names, are typically marked with font-lock-face by the parser.

fontified
     Another standard font-lock text property; applied whenever font-lock-face is.

wisi-indent
     Contains the indent (in characters) for the next line; applied to the newline character on the preceding line. The first line in a buffer is assumed to have indent 0.

Each action is classified as one of navigate, face, indent, in-parse; when actions are executed, only one of the first three classes is executed (in-parse is always executed). This reflects the reasons the parser is run; to figure out how to go somehere (end of current statement, start of current procedure, etc), to apply faces for syntax highlighting, or to indent the code.

Navigate actions
================

wisi-statement-action [TOKEN CLASS ...]
     The argument is a vector; alternating items are a token index (an integer or label indicating a token in the right hand side) and a ``token class''; one of:

     motion
          Create a wisi-cache text property on the token, for use in a subsequent wisi-motion-action.

     statement-end
          Create a wisi-cache text property on the token, enter a pointer to it in the other wisi-cache objects in the statement or declaration.

     statement-start
          Create a wisi-cache text property on the token, enter a pointer to it in the other wisi-cache objects (in the containing slot) in the statement or declaration.

     statement-override
          Same as statement-start; marks the token to be used as the statement start if the first token is optional.

     misc
          Create a wisi-cache text property on the token, to be used for some other purpose. It is good style to indicate the purpose in a comment.

          For example, ada-mode uses a 'misc' property on left parentheses that start a subprogram parameter list; this distinquishes them from other left parentheses, and makes it possible to automatically call ada-format-paramlist to format the parameter list, instead of using the standard Emacs align.

wisi-motion-action [TOKEN ...]
     The argument is a vector, where each element is either a token index or a vector [INDEX ID].

     Each terminal token must already have a wisi-cache created by a wisi-statement-action (this is checked at action execution, not during grammar generation). This action sets the prev, next slots for the chain of tokens, creating a chain of motion tokens.

     If TOKEN is a nonterminal without an ID specified, the wisi-cache must be on the first token in the nonterminal, and it is assumed to have a valid pointer in the next slot, indicating a chain of motion tokens. That chain is linked into the chain for the current right hand side.

     If TOKEN is a nonterminal with an ID, the region contained by the nonterminal is searched for all wisi-cache with that token ID, and for each one where prev/next is not already set, it is linked into the motion chain.

     Note that the ``search'' described here is done in the parser process, on a tree data structure containing the data that will eventually be stored in Emacs text properties.

wisi-name-action TOKEN
     TOKEN is a token index. Create a wisi-name text property on the token.

Face actions
============

wisi-face-mark-action [INDEX CLASS ...]
     The argument is a vector; alternating elements form pairs of INDEX CLASS, where class is one of prefix, suffix.

     Mark the tokens as part of a compound name, for use by later face actions.

wisi-face-apply-action [TOKEN PREFIX-FACE SUFFIX-FACE ...]
     The argument is a vector; triples of items specify TOKEN, PREFIX-FACE, SUFFIX-FACE. The faces are the elisp names of face objects (which must declared by an %elisp_face declaration).

     If the token is a nonterminal, and it has been marked by a previous wisi-face-mark-action, the specified faces are applied to the prefix and suffix in the token as font-lock-face text properties.

     If the token is a terminal, or a non-terminal with no face mark, the suffix face is applied to the entire text contained by the token.

wisi-face-apply-list-action [TOKEN PREFIX-FACE SUFFIX-FACE ...]
     Similar to ’wisi-face-apply-action’, but applies faces to all tokens marked by wisi-face-mark-action in each indicated production token, and does not apply a face if there are no such marks.

Indent actions
==============

Indents are computed for each line in a cumulative way as the grammar actions are executed. Initially, each indent is set to nil, which means ``not computed''; this is not the same as the value 0. The grammar actions are executed in a bottom-up fashion; low level productions are executed before higher level ones. In general, the indent action for a production specifies a ``delta indent''; the indent is incremented by that amount. When all productions have been processed, the indent has been computed for all lines.

Indents are often given as a function call; the arguments to the function can be other function calls, or integer expressions. wisitoken-bnf-generate supports only simple integer expressions; those using integers, integer-valued variables, parenthesis, + (plus), - (minus), and * (multiply).

wisi-indent-action [INDENT ...]
     The argument is a vector, giving an indent for each token in the production right-hand side.

     For terminals, the indents only have meaning, and are only computed, if the token is the first on a line. For nonterminals where the indent is not a variant of wisi-hanging, the indent is only computed if the first terminal token in the nonterminal is the first on a line. See wisi-hanging in Indent functions for the remaining case.

     An indent can have several forms.  In the descriptions below, the ``current token'' is given by the position of the indent expression in the wisi-indent-action argument list.

     An integer
          This gives a delta indent; it is added to the total indent for the line.

     A variable name
          The name is translated to an Ada identifier by replacing ``-'' with ``_'', and applying Camel_Case. The translated name must identify a directly visible run-time Ada integer variable; this is checked at Ada compile time. It provides an integer delta indent.

          For example, in Ada two indent variable names are ada-indent and ada-indent-broken, giving the basic ident, and the continuation line indent. They are runtime variables so different projects can specify them as part of a coding standard.

     A function call
          A function that computes a delta indent. See Indent functions.

     [CODE-INDENT , COMMENT-INDENT]
          A vector giving separate indents for code and comments.

          Normally, the indent for trailing comments (on lines with no code, after all code in the token) is given by the indent of the following token in the production. When the current token is the last, or the following tokens may be empty, or the indent of the following token would be wrong for some reason (for example, it is a block end), the comment indent may be specified separately. If it is not specified, and the indent from the next token is not available, the indent for the current token is used for code and comments.

          Comment lines that are not trailing are indented by CODE-INDENT.

     (label . INDENT)
          If token labels are used in a right hand side, they must be given explicitly in the indent arguments, using he lisp ``cons'' syntax. Labels are normally only used with EBNF grammars, which expand into multiple right hand sides, with optional tokens simply left out. Explicit labels on the indent arguments allow them to be left out as well.

Indent functions
----------------

wisi-anchored TOKEN OFFSET
     Sets the indent for the current token to be OFFSET (an integer expression) from the start of TOKEN (a token index); the current token is ``anchored to'' TOKEN.

wisi-anchored* TOKEN OFFSET
     Sets the indent for the current token to be OFFSET from the start of TOKEN, but only if TOKEN is the first token on a line; otherwise no indent

wisi-anchored*- TOKEN OFFSET
     Sets the indent for the current token to be OFFSET from the start of TOKEN, but only if TOKEN is the first token on a line and the indent for the current token accumulated so far is nil.

wisi-anchored% TOKEN OFFSET
     If there is an opening parenthesis containing TOKEN in the line containing TOKEN, set the current indent to OFFSET from that parenthesis. Otherwise, OFFSET gives an indent delta.

wisi-anchored%- TOKEN OFFSET
     Same as wisi-anchored%, but only if the current token accumulated indent is nil.

wisi-hanging DELTA-1 DELTA-2
     The current token is assumed to be a nonterminal. If the text it contains spans multiple lines, use DELTA-1 for the first line, DELTA-2 for the rest. If the current token is only on one line, use DELTA-1.

     DELTA-1 and DELTA-2 can be any IDENT expression, except a variant of wisi-hanging.

wisi-hanging% DELTA-1 DELTA-2
     Similar to wisi-hanging; if the first terminal token in the current nonterminal is the first token on the first line, use DELTA-1 for the first line and DELTA-2 for the rest. Otherwise, use DELTA-1 for all lines.

wisi-hanging%- DELTA-1 DELTA-2
     Same as wisi-hanging%, except applied only if the current token accumulated indent is nil.

Language-specific function
     Language-specific indent functions are specified by an %elisp_indent declaration in the grammar file. Each function specifies how many arguments it accepts; this is checked at action runtime, not during grammar generation. Each argument is an INDENT as described above, or a token ID prefixed by ' (to allow distinguishing token IDs from variable names).

Indent example
--------------

The example if_statement grammar nonterminal is:

     if_statement
       : IF expression THEN statements elsif_list ELSE statements END IF SEMICOLON
         %((wisi-indent-action [nil
                                [(wisi-hanging% ada-indent-broken (* 2 ada-indent-broken))
                                 ada-indent-broken]
                                nil
                                [ada-indent ada-indent] nil nil
                                [ada-indent ada-indent] nil nil nil]))%

We trace how the indent is computed for this sample Ada code:

      1: if A < B and
      2:   C < D
      3:   --  comment on expression
      4: then
      5:    if E then
      6:       Do_E;
      7:       -- comment on statement
      8:    elsif F then
      9:       G := A + Compute_Something
     10:         (arg_1, arg_2);
     11:    end if;
     12: end if;

First, the indent for the lower-level nonterminals (expression, statements, elsif_list) are computed. Assume they set the indent for line 10 to 2 (for the hanging expression) and leave the rest at nil.

Next, the action for the inner if_statement is executed. Most of the tokens specify an indent of nil, which means the current accumulated indent is not changed. For the others, the action is as follows:

expression:
     The expression E is contained on one line, and it is not the first token on that line, so the indent for line 5 is not changed.

statements: [ada-indent ada-indent]
     This specifies separate indents for code and trailing comments, because otherwise the trailing comments would be indented with the following THEN; instead they are indented with the expression code; see the comment on line 7.

     Here ada-indent is 3, so the indent for lines 6 and 7 (for the first occurence of statments) is incremented from nil to 3.

     For the second occurence of statements, line 9 is incremented from nil to 3, and line 10 from 2 to 5.

At this point, the accumulated indents are (the indent is given after the line number):

      1: nil : if A < B and
      2: nil :   C < D
      3: nil :   --  comment on expression
      4: nil : then
      5: nil :    if E then
      6:   3 :       Do_E;
      7:   3 :       -- comment on statement
      8: nil :    elsif F then
      9:   3 :       G := A + Compute_Something
     10:   5 :         (arg_1, arg_2);
     11: nil :    end if;
     12: nil : end if;

Then the action is executed for the outer if_statement:

expression: [(wisi-hanging% ada-indent-broken (* 2 ada-indent-broken)) ada-indent-broken]
     This specifies separate indents for code and trailing comments, because otherwise the trailing comments would be indented with the following THEN; instead they are indented with the expression code; see the comment on line 3.

     In this case, wisi-hanging% returns DELTA-1, which is ada-indent-broken, which is 2. So the indent for line 2 is incremented from nil to 2.

     The indent for line 3 is also incremented from nil to 2.

statements: [ada-indent ada-indent]
     Here there is only one statement; the nested if_statement. The indent for lines 5 .. 11 are each incremented by 3.

The final result is:

      1: nil : if A < B and
      2:   2 :   C < D
      3:   2 :   --  comment on expression
      4: nil : then
      5:   3 :    if E then
      6:   6 :       Do_E;
      7:   6 :       -- comment on statement
      8:   3 :    elsif F then
      9:   6 :       G := A + Compute_Something
     10:   8 :         (arg_1, arg_2);
     11:   6 :    end if;
     12: nil : end if;

In a full grammar, the top production should specify an indent of 0, not nil, for tokens that are not indented; then every line will have a non-nil indent. However, in normal operation a nil indent is treated as 0; the wisi-indent text property is not set for lines that have nil indent, and wisi-indent-region detects that and uses 0 for the indent. You can set the variable wisi-debug to a value > 0 to signal an error for nil indents; this is useful to catch indent errors during grammar development.

In-parse actions
================

wisi-propagate-name TOKEN
     The argument is a token index. Set the name component of the left-hand-side parse-time token object to the name component of the identified token, if it is not empty. Otherwise use the byte_region component.

wisi-merge-name FIRST-TOKEN, LAST-TOKEN
     The arguments are token indices, giving a range of tokens. LAST-TOKEN may be omitted if it is the same as FIRST-TOKEN.

     Set the name component of the left-hand-side to the merger of the name or byte-region components of the identified tokens.

wisi-match-name START-TOKEN END-TOKEN
     The arguments are token indices. Compare the text contained by the name (or byte_region if name is empty) token components for START-TOKEN and END-TOKEN; signal a parse error if they are different.

     The behavior when a name is missing is determined by the runtime language variable given in the %end_names_optional_option declaration; if True, a missing name that is supposed to match a present name is an error. Both names missing is not an error (assuming that is allowed by the grammar).

Project extension
*****************

wisi defines the cl-defstuct wisi-prj, with operations suitable for compilation and cross-reference.

In order to use wisi projects, the user must write project files and customize project-find-functions and xref-backend-functions.

Project files
=============

Project file names must have an extension given by wisi-prj-file-extensions (default .adp, .prj).

Project files have a simple syntax; they may be edited directly. Each line specifies a project variable name and its value, separated by ``='':

     src_dir=/Projects/my_project/src_1
     src_dir=/Projects/my_project/src_2

There must be no space between the variable name and ``='', and no trailing spaces after the value.

Any line that does not have an ``='' is a comment.

Some variables (like src_dir) are lists; each line in the project file specifies one element of the list. The value on the last line is the last element in the list.

A variable name that starts with $ is set as a process environment variable, for processes launched from Emacs for the project.

In values, process environment variables can be referenced using the normal $var syntax.

In values, relative file names are expanded relative to the directory containing the project file.

Here is the list of project variables defined by wisi; major modes may add more.

casing         [slot: case-exception-files]
     List of files containing casing exceptions. See Casing exception files.

src_dir        [slot: source-path]
     A list of directories to search for source files.

Selecting projects
==================

The current project can either be indicated by a global variable (called a ``selected project''), or depend on the current buffer.

In addition, the project file can be parsed each time it is needed, or the result cached to improve response time,

One reason to use a selected project is to handle a hierarchy of projects; if projects B and C both depend on library project A, then when in a file of project A, there is no way to determine which of the three projects to return. So the user must indicate which is active, by using one of wisi-prj-select-file or wisi-prj-select-cache.

In addition, if changing from one project to another requires setting global resources that must also be unset (such as a syntax propertize hook or compilation filter hook), then the project should define wisi-prj-deselect in addition to wisi-prj-select. Such projects require having a selected current project, so it can be deselected before a new one is selected. One example of such projects is ada-mode.

One way to declare each project is to add a Local Variables section in the main Makefile for the project; when the Makefile is first visited, the project is declared. In the examples here, we assume that approach is used; each gives an :eval line.

Note that wisi-prj-current-parse and wisi-prj-current-cached always succeed after some project is selected; no functions after them on project-find-functions will be called. That's why the depth is 90 for those in the examples.

No caching, current project depends on current buffer

          (add-hook 'project-find-functions #'wisi-prj-find-dominating-parse 0)

          :eval (wisi-prj-set-dominating "foo.prj" (foo-prj-default "prj-name"))

     wisi-prj-set-dominating declares the name of a project file with a default project object, and ensures that the current buffer file name is in wisi-prj--dominating.

     wisi-prj-find-dominating-parse looks for the filenames in wisi-prj--dominiating in the parent directories of the current buffer. When one is found, the associated project file is parsed, using the default project object to dispatch to the appropriate parsers. Then the final project object is returned.

Caching, current project depends on current buffer

          (add-hook 'project-find-functions #'wisi-prj-find-dominating-cached 0)

          :eval (wisi-prj-cache-dominating "foo.prj" (foo-prj-default "prj-name"))

     wisi-prj-cache-dominating declares the project file, parses it, and saves the project object in a cache indexed by the absolute project file name.

     wisi-prj-find-dominating-cached finds the dominating project file, and retrieves the object from the cache.

No caching, last selected project is current

          (add-hook 'project-find-functions #'wisi-prj-current-parse 90)

          :eval: (wisi-prj-select-file <prj-file> (foo-prj-default "prj-name"))

     wisi-prj-select-file sets the project file as the current project, and saves the default project object.

     wisi-prj-current-parse parses the current project file, using the saved default project object, and returns the project object.

Caching, last selected project is current

          (add-hook 'project-find-functions #'wisi-prj-current-cached 90)

          :eval: (wisi-prj-select-cache <prj-file> (foo-prj-default "prj-name"))

     wisi-prj-select-cache parses the project file, caches the project object.

     wisi-prj-current-cached returns the cached current project object.

In addition, the user should set xref-backend-functions. Currently, there is only one choice for wisi projects:

     (add-to-list 'xref-backend-functions #'wisi-prj-xref-backend 90)

wisi-prj-xref-backend returns the current wisi project object.

Casing exception files
======================

Each line in a case exception file specifies the casing of one word or word fragment. If an exception is defined in multiple files, the first occurrence is used.

If the word starts with an asterisk (*), it defines the casing of a word fragment (or ``substring''); part of a word between two underscores or word boundary.

For example:

     DOD
     *IO
     GNAT

The word fragment *IO applies to any word containing ``_io''; Text_IO, Hardware_IO, etc.

Other project functions
=======================

wisi-refresh-prj-cache (not-full)
     Refreshes all cached data in the project, and re-selects the project. If NOT-FULL is non-nil, slow refresh operations are skipped.

     This reparses the project file, and any cross reference information.

wisi-prj-select-dominating (dominating-file)
     Find a wisi-prj matching DOMINATING-FILE (defaults to the current buffer file). If the associated project is current, do nothing. If it is not current, select it.

     This is useful before running `compilation-start', to ensure the correct project is current.

GNU Free Documentation License
******************************

     [file doclicense.texi not included]