Emacs Auto-Overlays Manual
**************************

This manual describes the Emacs Auto-Overlays package, version 0.10.9

Copyright © 2007-2015 Free Software Foundation, Inc

     Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled "GNU Free Documentation License".

An Emacs overlay demarcates a region of text in a buffer, often giving it a different face or changing other properties for that region. There are many circumstance in which it might be useful to create, update, and delete overlays automatically when text matches some criterion, specified for example by regular expressions. This is what the auto-overlays package addresses. It is intended as an Elisp library, providing functions to be used by other Elisp packages, so does not itself define any new interactive commands or minor modes.

Overview
********

The auto-overlays package automatically creates, updates and destroys overlays based on regular expression matches in the buffer text. The overlay is created when text is typed that matches an auto-overlay regexp, and is destroyed if and when the matching text is changed so that it no longer matches.

The regexps are grouped into sets, and any number of different sets of regexps can be active in the same buffer simultaneously. Regexps in different sets are completely independent, and each set can be activated and deactivated independently (see Defining Regexps). This allows different Emacs modes to simultaneously make use of auto-overlays in the same buffer.

There are different ``classes'' of auto-overlay, used to define different kinds of overlay behaviour. Some classes only require a single regexp, others require separate regexps to define the start and end of the overlay (see Defining Regexps). Any additional regexps, beyond the minimum requirements, act as alternatives; if more than one of the regexps matches overlapping regions of text, the one that appears earlier in the list will take precedence. The predefined regexp classes are: word, line, self, nested and flat, but the auto-overlays package can easily be extended with new classes.

word
     These are used to define overlays that cover the text matched by the regexp itself, so require a single regexp. An example use would be to create overlays covering single words.

line
     These are used to define overlays that stretch from the text matching the regexp to the end of the line, and require a single regexp to define the start of the overlay. An example use would be to create overlays covering single-line comments in programming languages such as c.

self
     These are used to define overlays that stretch from one regexp match to the next match for the same regexp, so naturally require a single regexp. An example use would be to create overlays covering strings delimited by "".

     Note that for efficiency reasons, self overlays are not fully updated when a new match is found. Instead, when a modification is subsequently made at any position in the buffer after the new match, the overlays are updated up to that position. The update occurs just before the modification is made. Therefore, the overlays at a given buffer position will not necessarily be correct until a modification is made at or after that position (see To-Do).

nested
     These are used to define overlays that start and end at different regexp matches, and that can be nested one inside another. This class requires separate start and end regexps. An example use would be to create overlays between matching braces {}.

flat
     These are used to define overlays that start and end at different regexp matches, but that can not be nested. Extra start matches within one of these overlays are ignored. This class requires separate start and end regexps. An example use would be to create overlays covering multi-line comments in code, e.g. c++ block comments delimited by /* and */.

By default, the entire text matching a regexp acts as the ``delimeter''. For example, a word overlay will cover all the text matching its regexp, and a nested overlay will start at the end of the text matching its start regexp. Sometimes it is useful to be able to have only part of the regexp match act as the delimeter. This can be done by grouping that part of the regexp (see Defining Regexps). Overlays will then start and end at the text matching the group, instead of the text matching the entire regexp.

Of course, automatically creating overlays isn't much use without some way of setting their properties too. Overlay properties can be defined along with the regexp, and are applied to any overlays created by a match to that regexp. Certain properties have implications for auto-overlay behaviour:

priority
     This is a standard Emacs overlay property (see Overlay Properties (GNU Emacs Lisp Reference Manual)), but it is also used to determine which regexp takes precedence when two or more regexps in the same auto-overlay definition match overlapping regions of text. It is also used to determine which regexp's properties take precedence for overlays that are defined by separate start and end matches.

exclusive
     Normally, different auto-overlay regexps coexist, and act completely independently of one-another. However, if an auto-overlay has non-nil exclusive and priority properties, regexp matches within the overlay are ignored if they have lower priority. An example use is ignoring other regexp matches within comments in code.

Auto-Overlay Functions
**********************

To use auto-overlays in an Elisp package, you must load the package by including a line of the form

     (require 'auto-overlays)

near the beginning of your package.

This section describes the functions that are needed in order to make use of auto-overlays in an Elisp package. Functions related to extending the auto-overlays package are explained in a later section. See Extending the Auto-Overlays Package.

Defining Regexps
================

An auto-overlay definition is a list of the form:

     (CLASS &optional :id ENTRY-ID REGEXP1 REGEXP2 ...)

CLASS is one of the regexp classes described in the previous section (see Overview).

The optional :id property should be a symbol that can be used to uniquely identify the auto-overlay definition.

Each REGEXP defines one of the regexps that make up the auto-overlay definition. It should be a list of the form

     (RGXP &optional :id SUBENTRY-ID :edge EDGE :exclusitve EXCLUSIVE @rest PROPERTY1 PROPERTY2 ...)

The optional :id property should be a symbol that can be used to uniquely identify the regexp.

The :edge property should be one of the symbols 'start or 'end, and determines which edge of the auto-overlay this regexp corresponds to. If :edge is not specified, it is assumed to be 'start. Auto-overlay classes that do not require separate start and end regexps ignore this property.

The optional exclusive property specifies that this regexp matches exclusively; no lower-priority regexp is allowed to match the same text. (Contrast with the exclusive overlay property, which prevents any auto-overlay matches within the auto-overlay itself. see Overview)

Any further elements in the list are cons cells of the form (property . value), where PROPERTY is an overlay property name (a symbol) and VALUE its value.

In its simplest form, RGXP is a single regular expression. If only part of the regexp should act as the delimeter (see Overview), RGXP should instead be a cons cell:

     (RX . GROUP)

where RX is a regexp that contains at least one group (see Regular Expressions (GNU Emacs Lisp Reference Manual)), and GROUP is an integer identifying which group should act as the delimeter.

If the overlay class requires additional groups to be specified, RGXP should instead be a list:

     (RX GROUP0 GROUP1 ...)

where RX is a regexp. The first GROUP0 still specifies the part that acts as the delimeter, as before. If the entire regexp should act as the delimeter, GROUP0 must still be supplied but should be set to 0 (meaning the entire regexp). None of the standard classes make use of any additional groups, but extensions to the auto-overlays package that define new classes may. See Extending the Auto-Overlays Package.

The following functions are used to load and unload regexp definitions:

(auto-overlay-load-set DEFINITIONS)
     Load a new set of auto-overlay DEFINITIONS into the set identified by the symbol SET-ID. DEFINITIONS should be a list of definitions, each element being a definition of the form described above.

(auto-overlay-load-definition SET-ID DEFINITION &optional POS)
     Load a new auto-overlay DEFINITION, which should be a list of the form described above, into the set identified by the symbol SET-ID. The optional parameter POS determines where in the set's regexp list the new regexp is inserted. If it is nil, the regexp is added at the end. If it is t, the regexp is added at the beginning. If it is an integer, the regexp is added at that position in the list. Whilst the position in the list has no effect on overlay behaviour, it does determine the order in which regexps are checked, so can affect efficiency.

(auto-overlay-load-regexp SET-ID ENTRY-ID REGEXP &optional POS)
     Load a new REGEXP, which should be a list of the form described above, into the auto-overlay definition identified by the symbol ENTRY-ID, in the set identified by the symbol SET-ID. REGEXP should be a list of the form described above. The optional POS determines the position of the regexp in the list of regexps defining the auto-overlay, which can be significant for overlay behaviour since it determines which regexp takes precedence when two match the same text.

(auto-overlay-unload-set SET-ID)
     Unload the entire regexp set identified by the symbol SET-ID.

(auto-overlay-unload-definition SET-ID ENTRY-ID)
     Unload the auto-overlay definition identified by the symbol ENTRY-ID from the set identified by the symbol SET-ID.

(auto-overlay-unload-regexp SET-ID ENTRY-ID SUBENTRY-ID)
     Unload the auto-overlay regexp identified by the symbol SUBENTRY-ID from the auto-overlay definition identified by the symbol ENTRY-ID in the set identified by the symbol SET-ID.

(auto-overlay-share-regexp-set SET-ID FROM-BUFFER @optional TO-BUFFER)
     Share the set of regexp definitions identified by the symbol SET-ID in buffer from-buffer with the buffer TO-BUFFER, or the current buffer if TO-BUFFER is null. The regexp set becomes common to both buffers, and any changes made to it in one buffer, such as loading and unloading regexp definitions, are also reflected in the other buffer. However, the regexp set can still be enabled and disabled independently in both buffers. The same regexp set can be shared between any number of buffers. To remove a shared regexp set from one of the buffers, simply unload the entire set from that buffer using auto-overlay-unload-regexp. The regexp set will remain defined in all the other buffers it was shared with.

Starting and Stopping Auto-Overlays
===================================

A set of regexps is not active until it has been ``started'', and can be deactivated by ``stopping'' it. When a regexp set is activated, the entire buffer is scanned for regexp matches, and the corresponding overlays created. Similarly, when a set is deactivated, all the overlays are deleted. Note that regexp definitions can be loaded and unloaded whether the regexp set is active or inactive, and that deactivating a regexp set does not delete its regexp definitions.

Since scanning the whole buffer for regexp matches can take some time, especially for large buffers, auto-overlay data can be saved to an auxiliary file so that the overlays can be restored more quickly if the same regexp set is subsequently re-activated. Of course, if either the text in the buffer or the overlay definitions are modified whilst the regexp set is disabled, then the saved data will be out of date. Auto-overlays automatically checks whether the text or overlay definitions have been modified since the data was saved. If so, it ignores the saved data and re-scans the buffer.

The usual time to save and restore overlay data is when a regexp set is deactivated or activated. The auxilliary file name is then constructed automatically from the buffer name and the set-id. However, auto-overlays can also be saved and restored manually.

(auto-overlay-start SET-ID @optional BUFFER SAVE-FILE NO-REGEXP-CHECK)
     Activate the auto-overlay regexp set identified by the symbol SET-ID in BUFFER, or the current buffer if the latter is nil. If there is a file called auto-overlay-BUFFER-NAME-SET-ID containing up-to-date overlay data, it will be used to restore the auto-overlays (BUFFER-NAME is the name of the file visited by the buffer, or the buffer name itself if there is none). Otherwise, the entire buffer will be scanned for regexp matches.

     The string SAVE-FILE specifies the where to look for the file of saved overlay data. If it is nil, it defaults to the current directory. If it is a string specifying a relative path, then it is relative to the current directory, whereas an absolute path specifies exactly where to look. If it is a string specifying a file name (with or without a full path, relative or absolute), then it overrides the default file name and/or location. Any other value of SAVE-FILE will cause the file of overlay data to be ignored, even if it exists.

     If the overlays are being loaded from a file, but optional argument no-regexp-check is non-nil, the file of saved overlays will be used, but no check will be made to ensure regexp refinitions are the same as when the overlays were saved.

(auto-overlay-stop SET-ID @optional BUFFER SAVE-FILE LEAVE-OVERLAYS)
     Deactivate the auto-overlay regexp set identified by the symbol SET-ID in BUFFER, or the current buffer if the latter is nil. All corresponding overlays will be deleted (unless the LEAVE-OVERLAYS option is non-nil, which should only be used if the buffer is about to be killed), but the regexp definitions are preserved and can be reactivated later.

     If SAVE-FILE is non-nil, overlay data will be saved in an auxilliary file called auto-overlay-BUFFER-NAME-SET-ID in the current directory, to speed up subsequent reactivation of the regexp set in the same buffer (BUFFER-NAME is the name of the file visited by the buffer, or the buffer name itself if there is none). If SAVE-FILE is a string, it overrides the default save location, overriding either the directory if it only specifies a path (relative paths are relative to the current directory), or the file name if it only specifies a filename, or both if it specifies a full path.

(auto-overlay-save-overlays SET-ID @optional BUFFER FILE)
     Save auto-overlay data for the regexp set identified by the symbol SET-ID in BUFFER, or the current buffer if nil, to an auxilliary file called FILE. If FILE is nil, the overlay data are saved to a file called auto-overlay-BUFFER-NAME-SET-ID in the current directory (BUFFER-NAME is the name of the file visited by the buffer, or the buffer name itself if it's not visiting a file). If file is a directory name (either an absolute path or relative to the current directory), the overlay data are saved to the default file name under that directory.

(auto-overlay-load-overlays SET-ID @optional BUFFER FILE NO-REGEXP-CHECK)
     Load auto-overlay data for the regexp set identified by the symbol SET-ID into BUFFER, or the current buffer if nil, from an auxilliary file called FILE. If FILE is nil, it attempts to load the overlay data from a file called auto-overlay-BUFFER-NAME-SET-ID in the current directory (BUFFER-NAME is the name of the file visited by the buffer, or the buffer name itself if it's not visiting a file). If file is a directory name (either an absolute path or relative to the current directory), it attempts to load the overlay data from the default file name under that directory. If NO-REGEXP-CHECK is no-nil, the saved overlays will be loaded even if different regexp definitions were active when the overlays were saved. Returns t if the overlays were successfully loaded, nil otherwise.

Searching for Overlays
======================

Auto-overlays are just normal Emacs overlays, so any of the standard Emacs functions can be used to search for overlays and retrieve overlay properties. The auto-overlays package provides some additional functions.

(auto-overlays-at-point @optional POINT @key INACTIVE ALL-OVERLAYS @rest PROP-TESTS)
     Return a list of auto-overlays overlapping POINT, or the point if POINT is null. The list can be filtered to only return overlays with properties matching criteria specified by PROP-TESTS. This should be a list defining a property test, with one of the following forms (or a list of such lists, if more than one property test is required):

          (FUNCTION PROPERTY)
          (FUNCTION PROPERTY VALUE)
          (FUNCTION (PROPERTY1 PROPERTY2 ...) (VALUE1 VALUE2 ...))

     where FUNCTION is a function, PROPERTY is an overlay property name (a symbol), and VALUE can be any value or lisp expression. For each overlay, first the values corresponding to the PROPERTY names are retrieved from the overlay and any VALUEs that are lisp expressions are evaluated. Then FUNCTION is called with the property values followed by the other values as its arguments. The test is satisfied if the result is non-nil, otherwise it fails. Tests are evaluated in order, but only up to the first failure. Only overlays that satisfy all property tests are returned.

     For efficiency reasons, the auto-overlays package sometimes leaves overlays hanging around in the buffer even when they should have been deleted. These are marked with a non-nil inactive property. By default, auto-overlays-at-point ignores these. A non-nil INACTIVE keyword argument will override this, causing inactive overlays to be included in the returned list (assuming they pass all property tests).

     By default, auto-overlays-at-point only returns overlays created by the auto-overlays package. If you want to include all overlays, pass a non-nil ALL-OVERLAYS keyword argument.

(auto-overlays-in START END @key WITHIN INACTIVE ALL-OVERLAYS @rest PROP-TESTS)
     Return a list of overlays overlapping the region between START and END. A non-nil WITHIN keyword argument, restricts the results to overlays that are entirely within the region from START to END, not overlays that extend outside that region. The remaining arguments are as for auto-overlays-at-point, above.

(auto-overlay-highest-priority-at-point @optional POINT @key INACTIVE ALL-OVERLAYS @rest PROP-TESTS)
     Return the highest priority overlay at POINT (or the point, if POINT is null). An overlay's priority is determined by the value of its priority property (see Overlay Properties (GNU Emacs Lisp Reference Manual)). If two overlays have the same priority, the innermost one takes precedence (i.e. the one that begins later in the buffer, or if they begin at the same point the one that ends earlier; if two overlays have the same priority and extend over the same region, there is no way to predict which will be returned). The remaining arguments are as for auto-overlays-at-point.

(auto-overlay-local-binding SYMBOL @optional POINT)
     Return the ``overlay-local'' binding of SYMBOL at POINT (or the point if POINT is null), or the current local binding if there is no overlay binding. An ``overlay-local'' binding for SYMBOL is the value of the overlay property called SYMBOL. If more than one overlay at POINT has a non-nil SYMBOL property, the value from the highest priority overlay is returned (see auto-overlay-highest-priority-at-point, above, for an explanation of ``highest priority'').

Worked Example
**************

The interaction of all the different regexp definitions, overlay properties and auto-overlay classes provided by the auto-overlays package can be a little daunting. This section will go through an example of how the auto-overlay regexps could be defined to create overlays for a subset of LaTeX, which is complex enough to demonstrate most of the features.

LaTeX is a markup language, so a LaTeX document combines markup commands with normal text. Commands start with \, and end at the first non-word-constituent character. We want to highlight all LaTeX commands in blue. Two commands that will particularly interest us are \begin and \end, which begin and end a LaTeX environment. The environment name is enclosed in braces: \begin{ENVIRONMENT-NAME}, and we want it to be highlighted in pink. LaTeX provides many environments, used to create lists, tables, titles, etc. We will take the example of an equation environment, used to typeset mathematical equations. Thus equations are enclosed by \begin{equation} and \end{equation}, and we would like to highlight these equations in yellow. Another example we will use is the $ delimiter. Pairs of $s delimit mathematical expressions that appear in the middle of a paragraph of normal text (whereas equation environments appear on their own, slightly separated from surrounding text). Again, we want to highlight these mathematical expressions, this time in green. The final piece of LaTeX markup we will need to consider is the % character, which creates a comment that lasts till the end of the line (i.e. text after the % is ignored by the LaTeX processor up to the end of the line).

LaTeX commands are a good example of when to use word regular expressions (see Overview). The appropriate regexp definition is loaded by

     (auto-overlay-load-definition
      'latex
      '(word ("\\\\[[:alpha:]]*?\\([^[:alpha:]]\\|$\\)"
              (face . (background-color . "blue")))))

We have called the regexp set latex. The face property is a standard Emacs overlay property that sets font properties within the overlay. See Overlay Properties (GNU Emacs Lisp Reference Manual). "\\\\" is the string defining the regexp that matches a single \. (Note that the \ character has a special meaning in regular expressions, so to include a literal one it must be escaped: \\. However, \ also has a special meaning in lisp strings, so both \ characters must be escaped there too, giving \\\\.) [[:alpha:]]*? matches a sequence of zero or more letter characters. The ? ensures that it matches the shortest sequence of letters consistent with matching the regexp, since we want the region to end at the first non-letter character, matched by [^[:alpha:]]. The \| defines an alternative, to allow the LaTeX command to be terminated either by a non-letter character or by the end of the line ($). See Regular Expressions (GNU Emacs Lisp Reference Manual), for more details on Emacs regular expressions.

However, there's a small problem. We only want the blue background to cover the characters making up a LaTeX command. But as we've defined things so far, it will cover all the text matched by the regexp, which includes the leading \ and the trailing non-letter character. To rectify this, we need to group the part of the regexp that matches the command (i.e. by surround it with \( and \)), and put the regexp inside a cons cell containing the regexp in its car and a number indicating which subgroup to use in its cdr:

     (auto-overlay-load-definition
      'latex
      '(word (("\\\\[[:alpha:]]*?\\([^[:alpha:]]\\|$\\)" . 1)
              (face . (background-color . "blue")))))

The $ delimiter is an obvious example of when to use a self regexp (see Overview). We can update our example to include this (note that $ also has a special meaning in regular expressions, so it must be escaped with \ which itself must be escaped in lisp strings):

     (auto-overlay-load-definition
      'latex
      '(word (("\\\\[[:alpha:]]*?\\([^[:alpha:]]\\|$\\)" . 1)
              (face . (background-color . "blue")))))

     (auto-overlay-load-definition
      'latex
      '(self ("\\$" (face . (background-color . "green")))))

This won't quite work though. LaTeX maths commands also start with a \ character, which will match the word regexp. For the sake of example we want the entire equation highlighted in green, without highlighting any LaTeX maths commands it contains in blue. Since the word overlay will be within the self overlay, the blue highlighting will take precedence. We can change this by giving the self overlay a higher priority (any priority is higher than a non-existent one; we use 3 here for later convenience). For efficiency reasons, it's a good idea to put higher priority regexp definitions before lower priority ones, so we get:

     (auto-overlay-load-definition
      'latex
      '(self ("\\$" (priority . 3) (face . (background-color . "green")))))

     (auto-overlay-load-definition
      'latex
      '(word (("\\\\[[:alpha:]]*?\\([^[:alpha:]]\\|$\\)" . 1)
              (face . (background-color . "blue")))))

The \begin{equation} and \end{equation} commands also enclose maths regions, which we would like to highlight in yellow. Since the opening and closing delimiters are different in this case, we must use nested overlays (see Overview). Our example now looks like:

     (auto-overlay-load-definition
      'latex
      '(self ("\\$" (priority . 3) (face . (background-color . "green")))))

     (auto-overlay-load-definition
      'latex
      '(nested
        ("\\begin{equation}"
         :edge start
         (priority . 1)
         (face . (background-color . "yellow")))
        ("\\end{equation}"
         :edge end
         (priority . 1)
         (face . (background-color . "yellow")))))

     (auto-overlay-load-definition
      'latex
      '(word (("\\\\[[:alpha:]]*?\\([^[:alpha:]]\\|$\\)" . 1)
              (face . (background-color . "blue")))))

Notice how we've used separate start and end regexps to define the auto-overlay. Once again, we have had to escape the \ characters, and increase the priority of the new regexp definition to avoid any LaTeX commands within the maths region being highlighted in blue.

LaTeX comments start with % and last till the end of the line: a perfect demonstration of a line regexp. Here's a first attempt:

     (auto-overlay-load-definition
      'latex
      '(self ("\\$" (priority . 3) (face . (background-color . "green")))))

     (auto-overlay-load-definition
      'latex
      '(nested
        ("\\begin{equation}"
         :edge start
         (priority . 1)
         (face . (background-color . "yellow")))
        ("\\end{equation}"
         :edge end
         (priority . 1)
         (face . (background-color . "yellow")))))

     (auto-overlay-load-definition
      'latex
      '(word (("\\\\[[:alpha:]]*?\\([^[:alpha:]]\\|$\\)" . 1)
              (face . (background-color . "blue")))))

     (auto-overlay-load-definition
      'latex
      `(line ("%" (face . (background-color
                           . ,(face-attribute 'default :background))))))

We use the standard Emacs face-attribute function to retrieve the default background colour, which is evaluated before the regexp definition is loaded. (This will of course go wrong if the default background colour is subsequently changed, but it's sufficient for this example). Let's think about this a bit. We probably don't want anything within a comment to be highlighted at all, even if it matches one of the other regexps. In fact, creating overlays for \begin and \end commands which are within a comment could cause havoc! If they don't occur in pairs within the commented region, they will erroneously pair up with ones outside the comment. We need comments to take precedence over everything else, and we need them to block other regexp matches, so we boost the overlay's priority and set the exclusive property:

     (auto-overlay-load-definition
      'latex
      `(line ("%" (priority . 4) (exclusive . t)
                  (face . (background-color
                           . ,(face-attribute 'default :background))))))

     (auto-overlay-load-definition
      'latex
      '(self ("\\$" (priority . 3) (face . (background-color . "green")))))

     (auto-overlay-load-definition
      'latex
      '(nested
        ("\\begin{equation}"
         :edge start
         (priority . 1)
         (face . (background-color . "yellow")))
        ("\\end{equation}"
         :edge end
         (priority . 1)
         (face . (background-color . "yellow")))))

     (auto-overlay-load-definition
      'latex
      '(word (("\\\\[[:alpha:]]*?\\([^[:alpha:]]\\|$\\)" . 1)
              (face . (background-color . "blue")))))

We're well on our way to creating a useful setup, at least for the LaTeX commands we're considering in this example. There is one last type of overlay to create, but it is the most complicated. We want environment names to be highlighted in pink, i.e. the region between \begin{ and }. A first attempt at this might result in:

     (auto-overlay-load-definition
      'latex
      `(line ("%" (priority . 4) (exclusive . t)
                  (face . (background-color
                           . ,(face-attribute 'default :background))))))

     (auto-overlay-load-definition
      'latex
      '(self ("\\$" (priority . 3) (face . (background-color . "green")))))

     (auto-overlay-load-definition
      'latex
      '(nested
        ("\\begin{"
         :edge start
         (priority . 2)
         (face . (background-color . "pink")))
        ("}"
         :edge end
         (priority . 2)
         (face . (background-color . "pink")))))

     (auto-overlay-load-definition
      'latex
      '(nested
        ("\\begin{equation}"
         :edge start
         (priority . 1)
         (face . (background-color . "yellow")))
        ("\\end{equation}"
         :edge end
         (priority . 1)
         (face . (background-color . "yellow")))))

     (auto-overlay-load-definition
      'latex
      '(word (("\\\\[[:alpha:]]*?\\([^[:alpha:]]\\|$\\)" . 1)
              (face . (background-color . "blue")))))

However, we'll hit a problem with this. The } character also closes the \end{ command. Since we haven't told auto-overlays about \end{, every } that should close an \end{ command will instead be interpreted as the end of a \start{ command, probably resulting in lots of unmatched } characters, creating pink splodges everywhere! Clearly, since we also want environment names between \end{ and } to be pink, we need something more along the lines of:

     (auto-overlay-load-definition
      'latex
      `(line ("%" (priority . 4) (exclusive . t)
                  (face . (background-color
                           . ,(face-attribute 'default :background))))))

     (auto-overlay-load-definition
      'latex
      '(self ("\\$" (priority . 3) (face . (background-color . "green")))))

     (auto-overlay-load-definition
      'latex
      '(nested
        ("\\begin{"
         :edge start
         (priority . 2)
         (face . (background-color . "pink")))
        ("\\end{"
         :edge start
         (priority . 2)
         (face . (background-color . "pink")))
        ("}"
         :edge end
         (priority . 2)
         (face . (background-color . "pink")))))

     (auto-overlay-load-definition
      'latex
      '(nested
        ("\\begin{equation}"
         :edge start
         (priority . 1)
         (face . (background-color . "yellow")))
        ("\\end{equation}"
         :edge end
         (priority . 1)
         (face . (background-color . "yellow")))))

     (auto-overlay-load-definition
      'latex
      '(word (("\\\\[[:alpha:]]*?\\([^[:alpha:]]\\|$\\)" . 1)
              (face . (background-color . "blue")))))

We still haven't solved the problem though. The } character doesn't only close \begin{ and \end{ commands in LaTeX. All arguments to LaTeX commands are surrounded by { and }. We could add all the commands that take arguments, but we don't really want to bother about any other commands (at least in this example). All we want to do is prevent predictive mode incorrectly pairing the } characters used for other commands. Instead, we can just add { to the list:

     (auto-overlay-load-definition
      'latex
      `(line ("%" (priority . 4) (exclusive . t)
                  (face . (background-color
                           . ,(face-attribute 'default :background))))))

     (auto-overlay-load-definition
      'latex
      '(self ("\\$" (priority . 3) (face . (background-color . "green")))))

     (auto-overlay-load-definition
      'latex
      '(nested
        ("{"
         :edge start
         (priority . 2))
        ("\\begin{"
         :edge start
         (priority . 2)
         (face . (background-color . "pink")))
        ("\\end{"
         :edge start
         (priority . 2)
         (face . (background-color . "pink")))
        ("}"
         :edge end
         (priority . 2))))

     (auto-overlay-load-definition
      'latex
      '(nested
        ("\\begin{equation}"
         :edge start
         (priority . 1)
         (face . (background-color . "yellow")))
        ("\\end{equation}"
         :edge end
         (priority . 1)
         (face . (background-color . "yellow")))))

     (auto-overlay-load-definition
      'latex
      '(word (("\\\\[[:alpha:]]*?\\([^[:alpha:]]\\|$\\)" . 1)
              (face . (background-color . "blue")))))

Notice how the { and } regexps do not define a background colour (or indeed any other properties), so that any overlays they create will have no effect other than making sure all { and } characters are correctly paired.

We've made one mistake though: by putting the { regexp at the beginning of the list, it will take priority over any other regexp in the list that could match the same text. And since { will match whenever \begin{ or \end{ matches, environments will never be highlighted! The { regexp must come after the \begin{ and \end{ regexps, to ensure it is only used if neither of them match (it doesn't matter whether it appears before or after the { regexp, since the latter will never match the same text):

     (auto-overlay-load-definition
      'latex
      `(line ("%" (priority . 4) (exclusive . t)
                  (face . (background-color
                           . ,(face-attribute 'default :background))))))

     (auto-overlay-load-definition
      'latex
      '(self ("\\$" (priority . 3) (face . (background-color . "green")))))

     (auto-overlay-load-definition
      'latex
      '(nested
        ("\\begin{"
         :edge start
         (priority . 2)
         (face . (background-color . "pink")))
        ("\\end{"
         :edge start
         (priority . 2)
         (face . (background-color . "pink")))
        ("{"
         :edge start
         (priority . 2))
        ("}"
         :edge end
         (priority . 2))))

     (auto-overlay-load-definition
      'latex
      '(nested
        ("\\begin{equation}"
         :edge start
         (priority . 1)
         (face . (background-color . "yellow")))
        ("\\end{equation}"
         :edge end
         (priority . 1)
         (face . (background-color . "yellow")))))

     (auto-overlay-load-definition
      'latex
      '(word (("\\\\[[:alpha:]]*?\\([^[:alpha:]]\\|$\\)" . 1)
              (face . (background-color . "blue")))))

There is one last issue. A literal { or } character can be included in a LaTeX document by escaping it with \: \{ and \}. In this situation, the characters do not match anything and should not be treated as delimiters. We can modify the { and } regexps to exclude these cases:

     (auto-overlay-load-definition
      'latex
      `(line ("%" (priority . 4) (exclusive . t)
                  (face . (background-color
                           . ,(face-attribute 'default :background))))))

     (auto-overlay-load-definition
      'latex
      '(self ("\\$" (priority . 3) (face . (background-color . "green")))))

     (auto-overlay-load-definition
      'latex
      '(nested
        ("\\begin{"
         :edge start
         (priority . 2)
         (face . (background-color . "pink")))
        ("\\end{"
         :edge start
         (priority . 2)
         (face . (background-color . "pink")))
        ("\\([^\\]\\|^\\){"
         :edge start
         (priority . 2))
        ("\\([^\\]\\|^\\)}"
         :edge end
         (priority . 2))))

     (auto-overlay-load-definition
      'latex
      '(nested
        ("\\begin{equation}"
         :edge start
         (priority . 1)
         (face . (background-color . "yellow")))
        ("\\end{equation}"
         :edge end
         (priority . 1)
         (face . (background-color . "yellow")))))

     (auto-overlay-load-definition
      'latex
      '(word (("\\\\[[:alpha:]]*?\\([^[:alpha:]]\\|$\\)" . 1)
              (face . (background-color . "blue")))))

The new, complicated-looking regexps will only match { and } characters if they are not preceded by a \ character (see Regular Expressions (GNU Emacs Lisp Reference Manual)). Note that the character alternative [^\]\|^ can match any character that isn't a \ or the start of a line. This is required because matches to auto-overlay regexps are not allowed to span more than one line. If { or } appear at the beginning of a line, there will be no character in front (the newline character doesn't count, since it isn't on the same line), so the [^\] will not match.

However, when it does match, the } regexp will now match an additional character before the }, causing the overlay to end one character early. (The { regexp will also match one additional character before the {, but since the beginning of the overlay starts from the end of the start delimiter, this poses less of a problem.) We need to group the part of the regexp that should define the delimiter, i.e. the }, by surrounding it with \( and \), and put the regexp in the car of a cons cell whose cdr specifies the new subgroup (i.e. the 2nd subgroup, since the regexp already included a group for other reasons; we could alternatively replace the original group by a shy-group, since we don't actually need to capture match data for that group). Our final version looks like this:

     (auto-overlay-load-definition
      'latex
      `(line ("%" (priority . 4) (exclusive . t)
                  (face . (background-color
                           . ,(face-attribute 'default :background))))))

     (auto-overlay-load-definition
      'latex
      '(self ("\\$" (priority . 3) (face . (background-color . "green")))))

     (auto-overlay-load-definition
      'latex
      '(nested
        ("\\begin{"
         :edge start
         (priority . 2)
         (face . (background-color . "pink")))
        ("\\end{"
         :edge start
         (priority . 2)
         (face . (background-color . "pink")))
        ("\\([^\\]\\|^\\){"
         :edge start
         (priority . 2))
        (("\\([^\\]\\|^\\)\\(}\\)" . 2)
         :edge end
         (priority . 2))))

     (auto-overlay-load-definition
      'latex
      '(nested
        ("\\begin{equation}"
         :edge start
         (priority . 1)
         (face . (background-color . "yellow")))
        ("\\end{equation}"
         :edge end
         (priority . 1)
         (face . (background-color . "yellow")))))

     (auto-overlay-load-definition
      'latex
      '(word (("\\\\[[:alpha:]]*?\\([^[:alpha:]]\\|$\\)" . 1)
              (face . (background-color . "blue")))))

With these regexp definitions, LaTeX commands will automatically be highlighted in blue, equation environments in yellow, inline maths commands in green, and environment names in pink. LaTeX markup within comments will be ignored. And { and } characters from other commands will be correctly taken into account. All this is done in ``real-time''; it doesn't wait until Emacs is idle to update the overlays. Not bad for a bundle of regexps!

Of course, this could all be done more easily using Emacs' built-in syntax highlighting features, but the highlighting was only an example to show the location of the overlays. The main point is that the overlays are automatically created and kept up to date, and can be given any properties you like and used for whatever purpose is required by your Elisp package.

Extending the Auto-Overlays Package
***********************************

The auto-overlays package can easily be extended by adding new overlay classes (Or rather, it is easy to integrate new overlay classes into the package. Whether writing a new overlay class is easy or not depends on what you're trying to do, and how good your coding skills are ;-)). The next sections document the functions and interfaces provided by the auto-overlays package for this purpose.

Often, a new class is a minor modification of one of the standard classes. For example, it may work exactly like one of the standard classes, but in addition call some function whenever an overlay is created or destroyed. In this case, it is far better to build the new class on top of the existing class, using functions from the class-specific Elisp files, rather than starting from scratch. See Standard Parse and Suicide Functions.

Auto-Overlays in Depth
======================

In order to write new classes, a deeper understanding is required of how the auto-overlays package works. In fact, two kinds of overlays are automatically created, updated and destroyed when auto-overlays are active: the auto-overlays themselves, and ``match'' overlays, used to mark text that matches an auto-overlay regexp.

For overlay classes that only require one regexp to fully define an overlay (the word and line classes are the only standard classes like this (Although the self class only requires one regexp definition, the auto-overlays themselves require two matches to that same regexp to set the start and end of the overlay.)), the auto-overlays are always matched with the corresponding match overlay. For classes that require two regexp matches to define the start and end of an overlay (all other standard classes), each edge of an auto-overlay can be matched with a match overlay. The match overlays define where the edge of the auto-overlay is located. There will always be at least one matched edge, since an auto-overlay is only created when a regexp match is found, but it is possible for the second edge to not yet be matched (for many classes, the unmatched edge will be located at the beginning or end of the buffer).

If a match overlay delimits the start of an auto-overlay, the match overlay is stored in the auto-overlay's start property. The match overlay is also stored in the start property for auto-overlays that only require a single match. If a match overlay delimits the end of an auto-overlay, the match overlay is stored in the auto-overlay's end property. Conversely, a ``link'' to the auto-overlay is always stored in the match overlay's parent property (The ``parent'' terminology is admittedly very poor, and is a relic of a previous incarnation of the auto-overlays package, when it made more sense.).

Whenever a buffer is modified, the lines containing the modifications are scanned for new regexp matches. If one is found, a new match overlay is created covering the matching text, and then passed as an argument to the appropriate ``parse'' function (More bad terminology.) for its class. This deals with creating or updating the auto-overlays, as appropriate. If the text within a match overlay is modified, the match overlay checks whether the text it covers still matches the regexp. If it no longer matches, the match overlay is passed as an argument to the appropriate ``suicide'' function for its class, which deals with updating or deleting its parent auto-overlay (and possibly more besides).

To summarise, the core of the auto-overlays package deals with searching for regexp matches, and creating or deleting the corresponding match overlays. It then hands over the task of creating, updating or deleting the auto-overlays themselves to class-specific functions, which implement the correct behaviour for that class.

Integrating New Overlay Classes
===============================

To add a new overlay class, all that is required is to write new ``parse'' and ``suicide'' functions, and inform the auto-overlays package of their existence. A ``match'' function can also optionally be defined. It is called whenever a match overlay in the class becomes matched with the edge of an auto-overlay (see Functions for Modifying Overlays). The parse, suicide and match functions are conventionally called auto-o-parse-CLASS-match, auto-o-CLASS-suicide and auto-o-match-CLASS, where CLASS is the name of the class, though the convention is not enforced in any way.

parse function
     A parse function is passed a single argument containing a match overlay. It should return a list containing any new auto-overlays it creates, or nil if none were created.

          O-LIST = (auto-o-parse-CLASS-match O-MATCH)

     Note that the parse function itself is responsible for calling the auto-o-update-exclusive function if a new exclusive overlay is created. See Functions for Modifying Overlays.

suicide function
     A suicide function is passed a single argument containing a match overlay. Its return value is ignored.

          (auto-o-CLASS-suicide O-MATCH)

     The text covered by the match overlay should be considered to no longer match its regexp, although in certain cases matches are ignored for other reasons and this may not really be the case (for example if a new, higher-priority, exclusive overlay overlaps the match, see Overview).

match function
     A match function is passed a single argument containing a match overlay that has just been matched with an edge of an auto-overlay (see Functions for Modifying Overlays). Its return value is ignored.

          (auto-o-match-CLASS O-MATCH)

     The auto-overlay it is matched with is stored in the match overlay's parent property.

To integrate the new class into the auto-overlays package, the parse and suicide functions must be added to the property list of the symbol used to refer to the new class, denoted here by CLASS:

     (put 'CLASS 'auto-overlay-parse-function
          'auto-o-parse-CLASS-match)
     (put 'CLASS 'auto-overlay-suicide-function
          'auto-o-CLASS-suicide)

If the optional match function is defined, it should similarly be added to the symbol's property list:

     (put 'CLASS 'auto-overlay-match-function
          'auto-o-match-CLASS)

Functions for Writing New Overlay Classes
=========================================

Some functions are provided by the auto-overlays package for use in new parse and suicide functions. The functions that modify overlays carry out tasks that require interaction with the core of the auto-overlays package, and provide the only reliable way of carrying out those tasks. The other functions are used to query various things about auto-overlays and match overlays. Again, they are the only reliable interface for this, since the internal implementation may change between releases of the auto-overlays package.

Standard Parse and Suicide Functions
------------------------------------

All the standard overlay classes define their own parse and suicide functions (none of them require a match function), which can be used to create new ``derived'' classes based on the standard ones. This is the easiest and most common way to create a new class. For example, the new class may behave exactly like one of the standard classes, but perform some additional processing whenever an overlay is created, destroyed, or matched. The parse and suicide functions for the new class should perform whatever additional processing is required, and call the standard class functions to deal with creating and destroying the overlay.

All the standard parse and suicide functions follow the same naming convention (see Integrating New Overlay Classes), where CLASS is the name of the overlay class (one of word, line, self, nested or flat, see Overview):

(auto-o-parse-CLASS-match O-MATCH)
     Parse a new match overlay O-MATCH whose class is CLASS. This will create or update auto-overlays, as appropriate for the class.

(auto-o-CLASS-suicide O-MATCH)
     Delete or update auto-overlays as appropriate for overlay class CLASS, due to the match overlay O-MATCH no longer matching.

Functions for Modifying Overlays
--------------------------------

These functions modify auto-overlays and match overlays as necessary to perform a particular update. They should always be used to carry out their corresponding tasks, rather than doing it separately, since these tasks require interaction with the core of the auto-overlays package.

(auto-o-update-exclusive SET-ID BEG END OLD-PRIORITY NEW-PRIORITY)
     Update the region between BEG and END in the current buffer as necessary due to the priority of an exclusive overlay overlapping the region changing from OLD-PRIORITY to NEW-PRIORITY. If the exclusive overlay did not previously overlap the region, OLD-PRIORITY should be null. If it no longer overlaps the region, NEW-PRIORITY should be null. (If both are null, nothing will happen!) The return value is meaningless.

(auto-o-match-overlay OVERLAY START @optional END NO-PROPS NO-PARSE PROTECT-MATCH)
     Match or unmatch the start and end of the auto-overlay OVERLAY, update all appropriate properties (such as parent, start and end properties, and any properties specified in regexp definitions), and update other auto-overlays in the region covered by OVERLAY as necessary (usually because the exclusive or priority properties of OVERLAY have changed).

     If START or END are match overlays, match the corresponding edge of OVERLAY. The edge is moved to the location defined by the match overlay, and the parent property of the match overlay and the start and end properties of OVERLAY are updated accordingly. The START argument should be a match overlay corresponding either to the unique regexp if only one is needed for that overlay class, or to a start regexp if the overlay class uses separate start and end regexps. The END argument should then be a match overlay corresponding to an end regexp in the same class (see Overview). You're responsible for enforcing this; no check is made.

     If START or END are numbers or markers, move the corresponding edge of OVERLAY to that location and set it as unmatched. The start or end property of OVERLAY and the parent property of any corresponding match overlay are set to nil. If START or END are non-nil but neither of the above, leave the corresponding edge of OVERLAY where it is, but set it unmatched (as described above). If START or END are null, don't change the corresponding edge. However, for convenience, if END is null but START is a match overlay corresponding to a match for an end-regexp, match the end of OVERLAY rather than the start.

     The remaining arguments disable some of the tasks normally carried out by auto-o-match-overlay. If NO-PROPS is non-nil, overlay properties specified in regexp definitions are ignored and not updated. If NO-PARSE is non-nil, auto-overlays in the region covered by OVERLAY are not updated, even if the exclusive or priority properties of OVERLAY have changed. If PROTECT-MATCH is non-nil, the parent properties of the START and END match overlays are left alone.

(auto-o-delete-overlay OVERLAY @optional NO-PARSE PROTECT-MATCH)
     Delete auto-overlay OVERLAY from the buffer, and update overlays and overlay properties as necessary. The optional arguments disable parts of the updating process, as for auto-o-match-overlay, above.

Functions for Querying Overlays
-------------------------------

These functions query certain things about auto-overlays or match overlays, or retrieve certain values associated with them. A few are merely convenience functions, but most depend on the internal implementation details of the auto-overlays package, and provide the only reliable interface for whatever they return.

(auto-o-class O-MATCH)
     Return the class of match overlay O-MATCH.

(auto-o-regexp O-MATCH)
     Return the regular expression matched by the text covered by match overlay O-MATCH.

(auto-o-regexp-group O-MATCH)
     Return the regexp group defined in the regexp definition corresponding to match overlay O-MATCH (see Defining Regexps).

(auto-o-props O-MATCH)
     Return the list of overlay properties defined in the regexp definition corresponding to match overlay O-MATCH (see Defining Regexps).

(auto-o-edge O-MATCH)
     Return edge (the symbol start or end) of match overlay O-MATCH.

(auto-o-parse-function O-MATCH)
     Return appropriate parse function for match overlay O-MATCH.

(auto-o-suicide-function O-MATCH)
     Return appropriate suicide function for match overlay O-MATCH.

(auto-o-match-function O-MATCH)
     Return match function for match overlay O-MATCH, if any.

(auto-o-edge-matched-p OVERLAY EDGE)
     Return non-nil if EDGE (the symbol start or end) of auto-overlay overlay is matched.

(auto-o-start-matched-p OVERLAY)
     Return non-nil if auto-overlay OVERLAY is start-matched.

(auto-o-end-matched-p OVERLAY)
     Return non-nil if auto-overlay OVERLAY is end-matched.

Auto-Overlay Hooks
==================

The auto-overlays package defines two hooks, that are called when auto-overlays are enabled and disabled in a buffer. These are intended to be used by overlay classes to set up any extra buffer-local variables and settings they require, and clean them up afterwards. (There is no point leaving auto-overlay variables and settings hanging around in a buffer when auto-overlays are not in use.)

auto-overlay-load-hook
     This hook is run when the first auto-overlay regexp set in a buffer is started, using the auto-overlay-start function. See Starting and Stopping Auto-Overlays.

auto-overlay-unload-hook
     This hook is run when the last auto-overlay regexp set in a buffer is stopped, using the auto-overlay-stop function. See Starting and Stopping Auto-Overlays.

Auto-Overlay Modification Pseudo-Hooks
======================================

The auto-overlays package adds functions to buffer and overlay modification hooks in order to update the overlays as the buffer text is modified (see Modification Hooks (GNU Emacs Lisp Reference Manual)). The order in which all these modification hooks are called is undefined in Emacs (Or at least undocumented, and therefore unreliable.). Therefore, the auto-overlays package provides a mechanism to schedule functions to run at particular points during the overlay update process.

There are two stages to the overlay update process: first, any match overlay suicide functions are called, then modified buffer lines are scanned for new regexp matches. Three pseudo-hooks are defined that are called before, after and in between these stages. Their values are lists containing elements of the form:

     (FUNCTION ARG1 ARG2 ...)

where FUNCTION is the function to be called by the hook, and the ARG's are the arguments to be passed to that function. The list elements are evaluated in order. The pseudo-hooks are cleared each time after they have been called.

auto-o-pending-pre-suicide
     Pseudo-hook called before any suicide functions.

auto-o-pending-post-suicide
     Pseudo-hook called after any suicide functions but before scanning for regexp matches.

auto-o-pending-post-update
     Pseudo-hook called after scanning for regexp matches.

These pseudo-hooks can be used to ensure that a function that would normally be added to a modification hook will be called at a particular point in the auto-overlay update process. To achieve this, a helper function must be added to the modification hook instead. The helper function should add the function itself to the appropriate pseudo-hook by adding a list element with the form described above. The push and add-to-list Elisp functions are the most useful ways to add elements to the list.

To-Do
*****

Things that still need to be implemented (in no particular order):

 1. There needs to be an eager-self overlay class, similar to the existing self class but updated immediately, rather than waiting for buffer modifications. This will be significantly less efficient, but is necessary for applications that require overlays to be up to date all the time, not just when the buffer is being modified.

 2. Currently, it's difficult to deal with nested class regexps for which the end regexps match some start regexps of interest but also others that are irrelevant. E.g. { and } in LaTeX when you're only interested in \somecommand{ start regexps. Or matching parens in LISP, when you're only interested in function bodies, say. The only solution is to include all start regexps, but not set any of their properties. This can end up creating a lot of overlays! A variant of the nested class that avoids this problem is needed.

Copying this Manual
*******************

     [file fdl.texi not included]