Nim/doc/tut3.rst

=======================
Nim Tutorial (Part III)
=======================

:Author: Arne Döring
:Version: |nimversion|

.. contents::


Introduction
============

  "With Great Power Comes Great Responsibility." -- Spider Man's Uncle

This document is a tutorial about Nim's macro system.
A macro is a function that is executed at compile time and transforms
a Nim syntax tree into a different tree.

Examples of things that can be implemented in macros:

* An assert macro that prints both sides of a comparison operator, if
  the assertion fails. ``myAssert(a == b)`` is converted to
  ``if a != b: quit($a " != " $b)``

* A debug macro that prints the value and the name of the symbol.
  ``myDebugEcho(a)`` is converted to ``echo "a: ", a``

* Symbolic differentiation of an expression.
  ``diff(a*pow(x,3) + b*pow(x,2) + c*x + d, x)`` is converted to
  ``3*a*pow(x,2) + 2*b*x + c``


Macro Arguments
---------------

The types of macro arguments have two faces. One face is used for
the overload resolution, and the other face is used within the macro
body. For example, if ``macro foo(arg: int)`` is called in an
expression ``foo(x)``, ``x`` has to be of a type compatible to int, but
*within* the macro's body ``arg`` has the type ``NimNode``, not ``int``!
Why it is done this way will become obvious later, when we have seen
concrete examples.

There are two ways to pass arguments to a macro, an argument can be
either ``typed`` or ``untyped``.


Untyped Arguments
-----------------

Untyped macro arguments are passed to the macro before they are
semantically checked. This means the syntax tree that is passed down
to the macro does not need to make sense for Nim yet, the only
limitation is that it needs to be parsable. Usually the macro does
not check the argument either but uses it in the transformation's
result somehow. The result of a macro expansion is always checked
by the compiler, so apart from weird error messages nothing bad
can happen.

The downside for an ``untyped`` argument is that these do not play
well with Nim's overloading resolution.

The upside for untyped arguments is that the syntax tree is
quite predictable and less complex compared to its ``typed``
counterpart.


Typed Arguments
---------------

For typed arguments, the semantic checker runs on the argument and
does transformations on it, before it is passed to the macro. Here
identifier nodes are resolved as symbols, implicit type
conversions are visible in the tree as calls, templates are
expanded and probably most importantly, nodes have type information.
Typed arguments can have the type ``typed`` in the arguments list.
But all other types, such as ``int``, ``float`` or ``MyObjectType``
are typed arguments as well, and they are passed to the macro as a
syntax tree.


Static Arguments
----------------

Static arguments are a way to pass values as values and not as syntax
tree nodes to a macro. For example for ``macro foo(arg: static[int])``
in the expression ``foo(x)``, ``x`` needs to be an integer constant,
but in the macro body ``arg`` is just like a normal parameter of type
``int``.

.. code-block:: nim

  import macros

  macro myMacro(arg: static[int]): untyped =
    echo arg # just an int (7), not ``NimNode``

  myMacro(1 + 2 * 3)


Code Blocks as Arguments
------------------------

It is possible to pass the last argument of a call expression in a
separate code block with indentation. For example the following code
example is a valid (but not a recommended) way to call ``echo``:

.. code-block:: nim

  echo "Hello ":
    let a = "Wor"
    let b = "ld!"
    a & b

For macros this way of calling is very useful; syntax trees of arbitrary
complexity can be passed to macros with this notation.


The Syntax Tree
---------------

In order to build a Nim syntax tree one needs to know how Nim source
code is represented as a syntax tree, and how such a tree needs to
look like so that the Nim compiler will understand it. The nodes of the
Nim syntax tree are documented in the `macros <macros.html>`_ module.
But a more interactive way to explore the Nim
syntax tree is with ``macros.treeRepr``, it converts a syntax tree
into a multi line string for printing on the console. It can be used
to explore how the argument expressions are represented in tree form
and for debug printing of generated syntax tree. ``dumpTree`` is a
predefined macro that just prints its argument in tree representation,
but does nothing else. Here is an example of such a tree representation:

.. code-block:: nim

  dumpTree:
    var mt: MyType = MyType(a:123.456, b:"abcdef")

  # output:
  #   StmtList
  #     VarSection
  #       IdentDefs
  #         Ident "mt"
  #         Ident "MyType"
  #         ObjConstr
  #           Ident "MyType"
  #           ExprColonExpr
  #             Ident "a"
  #             FloatLit 123.456
  #           ExprColonExpr
  #             Ident "b"
  #             StrLit "abcdef"


Custom Semantic Checking
-----------------------

The first thing that a macro should do with its arguments is to check
if the argument is in the correct form. Not every type of wrong input
needs to be caught here, but anything that could cause a crash during
macro evaluation should be caught and create a nice error message.
``macros.expectKind`` and ``macros.expectLen`` are a good start. If
the checks need to be more complex, arbitrary error messages can
be created with the ``macros.error`` proc.

.. code-block:: nim

  macro myAssert(arg: untyped): untyped =
    arg.expectKind nnkInfix


Generating Code
---------------

There are two ways to generate the code. Either by creating the syntax
tree with expressions that contain a lot of calls to ``newTree`` and
``newLit``, or with ``quote do:`` expressions. The first option offers
the best low level control for the syntax tree generation, but the
second option is much less verbose. If you choose to create the syntax
tree with calls to ``newTree`` and ``newLit`` the macro
``macros.dumpAstGen`` can help you with the verbosity. ``quote do:``
allows you to write the code that you want to generate literally,
backticks are used to insert code from ``NimNode`` symbols into the
generated expression. This means that you can't use backticks within
``quote do:`` for anything else than injecting symbols.  Make sure to
inject only symbols of type ``NimNode`` into the generated syntax
tree. You can use ``newLit`` to convert arbitrary values into
expressions trees of type ``NimNode`` so that it is safe to inject
them into the tree.


.. code-block:: nim
    :test: "nim c $1"

  import macros

  type
    MyType = object
      a: float
      b: string

  macro myMacro(arg: untyped): untyped =
    var mt: MyType = MyType(a:123.456, b:"abcdef")

    # ...

    let mtLit = newLit(mt)

    result = quote do:
      echo `arg`
      echo `mtLit`

  myMacro("Hallo")

The call to ``myMacro`` will generate the following code:

.. code-block:: nim
  echo "Hallo"
  echo MyType(a: 123.456'f64, b: "abcdef")


Building Your First Macro
-------------------------

To give a starting point to writing macros we will show now how to
implement the ``myDebug`` macro mentioned earlier. The first thing to
do is to build a simple example of the macro usage, and then just
print the argument. This way it is possible to get an idea of a
correct argument should be look like.

.. code-block:: nim
    :test: "nim c $1"

  import macros

  macro myAssert(arg: untyped): untyped =
    echo arg.treeRepr

  let a = 1
  let b = 2

  myAssert(a != b)

.. code-block::

  Infix
    Ident "!="
    Ident "a"
    Ident "b"


From the output it is possible to see that the argument is an infix
operator (node kind is "Infix"), as well as that the two operands are
at index 1 and 2. With this information the actual macro can be
written.

.. code-block:: nim
    :test: "nim c $1"

  import macros

  macro myAssert(arg: untyped): untyped =
    # all node kind identifiers are prefixed with "nnk"
    arg.expectKind nnkInfix
    arg.expectLen 3
    # operator as string literal
    let op  = newLit(" " & arg[0].repr & " ")
    let lhs = arg[1]
    let rhs = arg[2]

    result = quote do:
      if not `arg`:
        raise newException(AssertionDefect,$`lhs` & `op` & $`rhs`)

  let a = 1
  let b = 2

  myAssert(a != b)
  myAssert(a == b)


This is the code that will be generated. To debug what the macro
actually generated, the statement ``echo result.repr`` can be used, in
the last line of the macro. It is also the statement that has been
used to get this output.

.. code-block:: nim
  if not (a != b):
    raise newException(AssertionDefect, $a & " != " & $b)

With Power Comes Responsibility
-------------------------------

Macros are very powerful. A good advice is to use them as little as
possible, but as much as necessary. Macros can change the semantics of
expressions, making the code incomprehensible for anybody who does not
know exactly what the macro does with it. So whenever a macro is not
necessary and the same logic can be implemented using templates or
generics, it is probably better not to use a macro. And when a macro
is used for something, the macro should better have a well written
documentation. For all the people who claim to write only perfectly
self-explanatory code: when it comes to macros, the implementation is
not enough for documentation.

Limitations
-----------

Since macros are evaluated in the compiler in the NimVM, macros share
all the limitations of the NimVM. They have to be implemented in pure Nim
code. Macros can start external processes on the shell, but they
cannot call C functions except from those that are built in the
compiler.


More Examples
=============

This tutorial can only cover the basics of the macro system. There are
macros out there that could be an inspiration for you of what is
possible with it.


Strformat
---------

In the Nim standard library, the ``strformat`` library provides a
macro that parses a string literal at compile time. Parsing a string
in a macro like here is generally not recommended. The parsed AST
cannot have type information, and parsing implemented on the VM is
generally not very fast. Working on AST nodes is almost always the
recommended way. But still ``strformat`` is a good example for a
practical use case for a macro that is slightly more complex than the
``assert`` macro.

`Strformat <https://github.com/nim-lang/Nim/blob/5845716df8c96157a047c2bd6bcdd795a7a2b9b1/lib/pure/strformat.nim#L280>`_

Ast Pattern Matching
--------------------

Ast Pattern Matching is a macro library to aid in writing complex
macros. This can be seen as a good example of how to repurpose the
Nim syntax tree with new semantics.

`Ast Pattern Matching <https://github.com/krux02/ast-pattern-matching>`_

OpenGL Sandbox
--------------

This project has a working Nim to GLSL compiler written entirely in
macros. It scans recursively though all used function symbols to
compile them so that cross library functions can be executed on the GPU.

`OpenGL Sandbox <https://github.com/krux02/opengl-sandbox>`_