Nim/doc/manual/templates.txt

Templates
=========

A template is a simple form of a macro: It is a simple substitution
mechanism that operates on Nim's abstract syntax trees. It is processed in
the semantic pass of the compiler.

The syntax to *invoke* a template is the same as calling a procedure.

Example:

.. code-block:: nim
  template `!=` (a, b: untyped): untyped =
    # this definition exists in the System module
    not (a == b)

  assert(5 != 6) # the compiler rewrites that to: assert(not (5 == 6))

The ``!=``, ``>``, ``>=``, ``in``, ``notin``, ``isnot`` operators are in fact
templates:

| ``a > b`` is transformed into ``b < a``.
| ``a in b`` is transformed into ``contains(b, a)``.
| ``notin`` and ``isnot`` have the obvious meanings.

The "types" of templates can be the symbols ``untyped``,
``typed`` or ``typedesc`` (stands for *type
description*). These are "meta types", they can only be used in certain
contexts. Real types can be used too; this implies that ``typed`` expressions
are expected.


Typed vs untyped parameters
---------------------------

An ``untyped`` parameter means that symbol lookups and type resolution is not
performed before the expression is passed to the template. This means that for
example *undeclared* identifiers can be passed to the template:

.. code-block:: nim

  template declareInt(x: untyped) =
    var x: int

  declareInt(x) # valid
  x = 3


.. code-block:: nim

  template declareInt(x: typed) =
    var x: int

  declareInt(x) # invalid, because x has not been declared and so has no type

A template where every parameter is ``untyped`` is called an `immediate`:idx:
template. For historical reasons templates can be explicitly annotated with
an ``immediate`` pragma and then these templates do not take part in
overloading resolution and the parameters' types are *ignored* by the
compiler. Explicit immediate templates are now deprecated.

**Note**: For historical reasons ``stmt`` is an alias for ``typed`` and
``expr`` an alias for ``untyped``, but new code should use the newer,
clearer names.


Passing a code block to a template
----------------------------------

You can pass a block of statements as a last parameter to a template via a
special ``:`` syntax:

.. code-block:: nim
  template withFile(f, fn, mode, actions: untyped): untyped =
    var f: File
    if open(f, fn, mode):
      try:
        actions
      finally:
        close(f)
    else:
      quit("cannot open: " & fn)

  withFile(txt, "ttempl3.txt", fmWrite):
    txt.writeLine("line 1")
    txt.writeLine("line 2")

In the example the two ``writeLine`` statements are bound to the ``actions``
parameter.


Usually to pass a block of code to a template the parameter that accepts
the block needs to be of type ``untyped``. Because symbol lookups are then
delayed until template instantiation time:

.. code-block:: nim
  template t(body: typed) =
    block:
      body

  t:
    var i = 1
    echo i

  t:
    var i = 2  # fails with 'attempt to redeclare i'
    echo i

The above code fails with the mysterious error message that ``i`` has already
been declared. The reason for this is that the ``var i = ...`` bodies need to
be type-checked before they are passed to the ``body`` parameter and type
checking in Nim implies symbol lookups. For the symbol lookups to succeed
``i`` needs to be added to the current (i.e. outer) scope. After type checking
these additions to the symbol table are not rolled back (for better or worse).
The same code works with ``untyped`` as the passed body is not required to be
type-checked:

.. code-block:: nim
  template t(body: untyped) =
    block:
      body

  t:
    var i = 1
    echo i

  t:
    var i = 2  # compiles
    echo i


Varargs of untyped
------------------

In addition to the ``untyped`` meta-type that prevents type checking there is
also ``varargs[untyped]`` so that not even the number of parameters is fixed:

.. code-block:: nim
  template hideIdentifiers(x: varargs[untyped]) = discard

  hideIdentifiers(undeclared1, undeclared2)

However, since a template cannot iterate over varargs, this feature is
generally much more useful for macros.

**Note**: For historical reasons ``varargs[expr]`` is not equivalent
to ``varargs[untyped]``.


Symbol binding in templates
---------------------------

A template is a `hygienic`:idx: macro and so opens a new scope. Most symbols are
bound from the definition scope of the template:

.. code-block:: nim
  # Module A
  var
    lastId = 0

  template genId*: untyped =
    inc(lastId)
    lastId

.. code-block:: nim
  # Module B
  import A

  echo genId() # Works as 'lastId' has been bound in 'genId's defining scope

As in generics symbol binding can be influenced via ``mixin`` or ``bind``
statements.



Identifier construction
-----------------------

In templates identifiers can be constructed with the backticks notation:

.. code-block:: nim

  template typedef(name: untyped, typ: typedesc) =
    type
      `T name`* {.inject.} = typ
      `P name`* {.inject.} = ref `T name`

  typedef(myint, int)
  var x: PMyInt

In the example ``name`` is instantiated with ``myint``, so \`T name\` becomes
``Tmyint``.


Lookup rules for template parameters
------------------------------------

A parameter ``p`` in a template is even substituted in the expression ``x.p``.
Thus template arguments can be used as field names and a global symbol can be
shadowed by the same argument name even when fully qualified:

.. code-block:: nim
  # module 'm'

  type
    Lev = enum
      levA, levB

  var abclev = levB

  template tstLev(abclev: Lev) =
    echo abclev, " ", m.abclev

  tstLev(levA)
  # produces: 'levA levA'

But the global symbol can properly be captured by a ``bind`` statement:

.. code-block:: nim
  # module 'm'

  type
    Lev = enum
      levA, levB

  var abclev = levB

  template tstLev(abclev: Lev) =
    bind m.abclev
    echo abclev, " ", m.abclev

  tstLev(levA)
  # produces: 'levA levB'


Hygiene in templates
--------------------

Per default templates are `hygienic`:idx:\: Local identifiers declared in a
template cannot be accessed in the instantiation context:

.. code-block:: nim

  template newException*(exceptn: typedesc, message: string): untyped =
    var
      e: ref exceptn  # e is implicitly gensym'ed here
    new(e)
    e.msg = message
    e

  # so this works:
  let e = "message"
  raise newException(EIO, e)


Whether a symbol that is declared in a template is exposed to the instantiation
scope is controlled by the `inject`:idx: and `gensym`:idx: pragmas: gensym'ed
symbols are not exposed but inject'ed are.

The default for symbols of entity ``type``, ``var``, ``let`` and ``const``
is ``gensym`` and for ``proc``, ``iterator``, ``converter``, ``template``,
``macro`` is ``inject``. However, if the name of the entity is passed as a
template parameter, it is an inject'ed symbol:

.. code-block:: nim
  template withFile(f, fn, mode: untyped, actions: untyped): untyped =
    block:
      var f: File  # since 'f' is a template param, it's injected implicitly
      ...

  withFile(txt, "ttempl3.txt", fmWrite):
    txt.writeLine("line 1")
    txt.writeLine("line 2")


The ``inject`` and ``gensym`` pragmas are second class annotations; they have
no semantics outside of a template definition and cannot be abstracted over:

.. code-block:: nim
  {.pragma myInject: inject.}

  template t() =
    var x {.myInject.}: int # does NOT work


To get rid of hygiene in templates, one can use the `dirty`:idx: pragma for
a template. ``inject`` and ``gensym`` have no effect in ``dirty`` templates.



Limitations of the method call syntax
-------------------------------------

The expression ``x`` in ``x.f`` needs to be semantically checked (that means
symbol lookup and type checking) before it can be decided that it needs to be
rewritten to ``f(x)``. Therefore the dot syntax has some limiations when it
is used to invoke templates/macros:

.. code-block:: nim
  template declareVar(name: untyped) =
    const name {.inject.} = 45

  # Doesn't compile:
  unknownIdentifier.declareVar


Another common example is this:

.. code-block:: nim
  from sequtils import toSeq

  iterator something: string =
    yield "Hello"
    yield "World"

  var info = toSeq(something())

The problem here is that the compiler already decided that ``something()`` as
an iterator is not callable in this context before ``toSeq`` gets its
chance to convert it into a sequence.


Macros
======

A macro is a special kind of low level template. Macros can be used
to implement `domain specific languages`:idx:.

While macros enable advanced compile-time code transformations, they
cannot change Nim's syntax. However, this is no real restriction because
Nim's syntax is flexible enough anyway.

To write macros, one needs to know how the Nim concrete syntax is converted
to an abstract syntax tree.

There are two ways to invoke a macro:
(1) invoking a macro like a procedure call (`expression macros`)
(2) invoking a macro with the special ``macrostmt`` syntax (`statement macros`)


Expression Macros
-----------------

The following example implements a powerful ``debug`` command that accepts a
variable number of arguments:

.. code-block:: nim
  # to work with Nim syntax trees, we need an API that is defined in the
  # ``macros`` module:
  import macros

  macro debug(n: varargs[untyped]): untyped =
    # `n` is a Nim AST that contains the whole macro invocation
    # this macro returns a list of statements:
    result = newNimNode(nnkStmtList, n)
    # iterate over any argument that is passed to this macro:
    for i in 0..n.len-1:
      # add a call to the statement list that writes the expression;
      # `toStrLit` converts an AST to its string representation:
      add(result, newCall("write", newIdentNode("stdout"), toStrLit(n[i])))
      # add a call to the statement list that writes ": "
      add(result, newCall("write", newIdentNode("stdout"), newStrLitNode(": ")))
      # add a call to the statement list that writes the expressions value:
      add(result, newCall("writeLine", newIdentNode("stdout"), n[i]))

  var
    a: array [0..10, int]
    x = "some string"
  a[0] = 42
  a[1] = 45

  debug(a[0], a[1], x)

The macro call expands to:

.. code-block:: nim
  write(stdout, "a[0]")
  write(stdout, ": ")
  writeLine(stdout, a[0])

  write(stdout, "a[1]")
  write(stdout, ": ")
  writeLine(stdout, a[1])

  write(stdout, "x")
  write(stdout, ": ")
  writeLine(stdout, x)


Arguments that are passed to a ``varargs`` parameter are wrapped in an array
constructor expression. This is why ``debug`` iterates over all of ``n``'s
children.


BindSym
-------

The above ``debug`` macro relies on the fact that ``write``, ``writeLine`` and
``stdout`` are declared in the system module and thus visible in the
instantiating context. There is a way to use bound identifiers
(aka `symbols`:idx:) instead of using unbound identifiers. The ``bindSym``
builtin can be used for that:

.. code-block:: nim
  import macros

  macro debug(n: varargs[typed]): untyped =
    result = newNimNode(nnkStmtList, n)
    for x in n:
      # we can bind symbols in scope via 'bindSym':
      add(result, newCall(bindSym"write", bindSym"stdout", toStrLit(x)))
      add(result, newCall(bindSym"write", bindSym"stdout", newStrLitNode(": ")))
      add(result, newCall(bindSym"writeLine", bindSym"stdout", x))

  var
    a: array [0..10, int]
    x = "some string"
  a[0] = 42
  a[1] = 45

  debug(a[0], a[1], x)

The macro call expands to:

.. code-block:: nim
  write(stdout, "a[0]")
  write(stdout, ": ")
  writeLine(stdout, a[0])

  write(stdout, "a[1]")
  write(stdout, ": ")
  writeLine(stdout, a[1])

  write(stdout, "x")
  write(stdout, ": ")
  writeLine(stdout, x)

However, the symbols ``write``, ``writeLine`` and ``stdout`` are already bound
and are not looked up again. As the example shows, ``bindSym`` does work with
overloaded symbols implicitly.


Statement Macros
----------------

Statement macros are defined just as expression macros. However, they are
invoked by an expression following a colon.

The following example outlines a macro that generates a lexical analyzer from
regular expressions:

.. code-block:: nim
  import macros

  macro case_token(n: untyped): untyped =
    # creates a lexical analyzer from regular expressions
    # ... (implementation is an exercise for the reader :-)
    discard

  case_token: # this colon tells the parser it is a macro statement
  of r"[A-Za-z_]+[A-Za-z_0-9]*":
    return tkIdentifier
  of r"0-9+":
    return tkInteger
  of r"[\+\-\*\?]+":
    return tkOperator
  else:
    return tkUnknown


**Style note**: For code readability, it is the best idea to use the least
powerful programming construct that still suffices. So the "check list" is:

(1) Use an ordinary proc/iterator, if possible.
(2) Else: Use a generic proc/iterator, if possible.
(3) Else: Use a template, if possible.
(4) Else: Use a macro.


Macros as pragmas
-----------------

Whole routines (procs, iterators etc.) can also be passed to a template or
a macro via the pragma notation:

.. code-block:: nim
  template m(s: untyped) = discard

  proc p() {.m.} = discard

This is a simple syntactic transformation into:

.. code-block:: nim
  template m(s: untyped) = discard

  m:
    proc p() = discard