Building A Macro-Expansion Helper for IEx

Macros in Elixir can seem a bit murky if you haven't used Lisp-style macros extensively before. Let's open a window on what they're doing, by iteratively building a "macro-expansion" a helper for our Interactive Elixir (IEx) shell.

First, let's state our goal: we'd like to see what Elixir code various macros are generating. For convenience, we'd like to be able to do so in our interactive Elixir shell. If something goes wrong, and a macro can't fully finish expanding, we'd like to have an idea of why.

That's a feasible goal! Macro.expand and Macro.to_string will let us expand quoted code on demand, and turn it back into a Elixir code that a human can read. Let's see how far that gets us, and then we can ask what's missing to have an interactive development experience closer to what Lisp hackers have.

A little helper for .iex.exs

We'll make a helper named mex, for Macro EXpansion.

Open up (or create) your .iex.exs file, either in a Mix project, or the global one in ~/.iex.exs.

Then paste this in, and reload your IEx.

defmodule Mex do

 defmacro mex( do: block ) do
    block
    |> Macro.expand(__CALLER__)
    |> Macro.to_string         
    |> IO.puts                 
    quote do: :ok
  end

end  

Straightforward! We use it from IEx as follows:

iex(1)> import Mex  
nil  
iex(2)> mex do some_expression end  
expansion_of_some_expression  
:ok

We have a first draft. Let's try it out on some built-in Elixir macros.

Watching Elixir's Own Macros

As in Common Lisp, with so much of the language implemented as macros, we can learn a lot by expanding things built into the language.

Integer.is_even is a simple macro:

ex(2)> require Integer  
nil  
iex(3)> mex do: Integer.is_even 3  
(3 &&& 1) == 0
:ok

Neat! What would be a function call in other expressive languages expands to a 'bitwise and' operation in Elixir. Also, notice that require Integer didn't have to be typed inside of the "mex do .. end" block; Macro.expand(__CALLER__) expanded in the correct context.

Go ahead — use it to peek inside some other macros! Some ideas: use, if, and in.

Something's Not Right Here!

I said mex was a first draft, and that we'd have to improve it. Why is that?

Well, look at the following example. Can you see what hasn't expanded?

iex(2)> mex do  
...(2)>   if 1 == 2 do
...(2)>     "I don't believe you. Liar!"
...(2)>   else
...(2)>     "I guess."
...(2)>   end
...(2)> end
case(1 == 2) do  
  x when x in [false, nil] ->
    "I guess."
  _ ->
    "I don't believe you. Liar!"
end  
:ok

The expanded output contains x in [false, nil] -- but 'in' is a macro call! I thought these were getting expanded. So where did we go wrong?

(re)curse macro-expansion!

Here's the thing. The documentation for Macro.expand says, ahem,

"Macro.expand(tree, env) — Receives an AST node and expands it until it can no longer be expanded."

Macro.expand isn't recursive. It will turn an 'unless' into an 'if', and then into a 'case,' but any macros further down the tree, in the conditional or the do/else blocks will not be touched.

However, not to fear. We can make a recursive version of our helper, by using one more facility that the Macro module provides: Macro.prewalk.

Macro.prewalk is a great function, and occasionally convenient outside of macros!1 It's a kind of function sometimes called a 'mapping walk' or 'replacing walk', and it works the same way that 'map' works, but on potentially nested trees composed of tuples, lists and other basic Elixir data types. Like 'map,' you supply a function to be called for every element in the tree, and it will return a transformed version.

We will use it to recursively expand macros.

defmodule Mex do

  def expand_all(n, env) do
    Macro.prewalk n, &Macro.expand( &1, env )
  end

  defmacro mex( do: block ) do
    block
    |> expand_all(__CALLER__)   # <-- use our recursive walker
    |> Macro.to_string
    |> IO.puts
  end  

end  

How does this handle, for instance, nested if statements? Let's look:

iex(4)> mex do  
...(4)>   unless 2==1 do
...(4)>     if confusing_statement? do
...(4)>       "success!"
...(4)>     end
...(4)>   end
...(4)> end
case(2 == 1) do  
  x when Enum.member?([false, nil], x) ->
    case(confusing_statement?) do
      x when Enum.member?([false, nil], x) ->
        nil
      _ ->
        "success!"
    end
  _ ->
    nil
end  
:ok

Nice! Everything expanded.

Technical Limitations

It seems pretty simple. Is it always that easy to see what macros are doing? Not all of them. Macros that are tightly coupled to module definition are more opaque.

Elixir is all about modules, and our code is peppered with module-specific concepts like 'def', '@attributes', 'defstruct', and 'use'. The first macros people write are often attempts to do code reuse with __using__2. But look at what a module with a one-line function in it expands to:

iex(24)> quote do  
...(24)>  defmodule M do
...(24)>   def m, do: 2
...(24)>  end
...(24)> end |> Macro.expand(__ENV__) |> Macro.to_string |> IO.puts
(
  alias(M, as: nil, warn: false)
  :elixir_module.compile(M, 
    {:def, [context: Elixir, import: Kernel], 
       [{:m, [context: Elixir], Elixir}, 
        [do: 2]]
    }, [], __ENV__
  )
)
:ok

It expands to a block of code that throws a raw, un-expanded AST over the wall to :elixir_module.compile3. So modules are a bit more opaque. Code that interacted deeply enough with the language internals could maybe give us back the quoted form of an expanded 'def' in the context of a module, but we'd rather depend on language-level APIs like the Macro and Code modules.

However, if we break our macros down sufficiently, and avoid monolithic __using__ macros, we should also be able to inspect them in isolation.

Wrapping It Up

We can make this a more useful tool in a few mechanical ways, though unrelated to macroexpansion:

  • Installation. We could have a Hex package, so people don't have to rely on copy-pasted snippets in their .iex.exs files.
  • Multi-column display — we've seen how to display 3 levels of macroexpansion: expanding once, expanding a node completely, and recursively expanding its child nodes. As an aid to understanding, it'd be nice to see them side-by-side in the shell.
  • Error handling. If a macro can expand, but can't finish expanding its children, that is helpful information for the macro author. This means our helper would need to handle errors during expansion.

The result of making those changes is up on github, and also installable as a dev dependency via Hex as the package mex. Let me know if you find it useful!


  1. For instance, stringifying leaf nodes in a tree could be just "Macro.prewalk list, &((is_list(&1) && &1) || to_string(&1))". Node-walking is less preferable for most tasks in Elixir/Erlang, as pattern-matching on function heads makes it feasible to quickly write a performant walker for your data structure -- (2018 update: there is also the amazing

    undefined protocol in Elixir, which lets you do almost everything you'd want out of a replacing walk in a much more performant way). But sometimes the extreme waste could be worth the convenience of having a general way to do transforms on arbitrarily nested data.

  2. Many attempts at meta-programming with 'use' are clearly inspired by Ruby "macro methods" — though macros are not always necessary for nice DSLs in Elixir, and the value prop of pulling in a bunch of implicit behavior sounds hasn't aged well.

  3. If you're in the mood for a good whodunnit, trace what happens to a 'def' during module compilation, or to a macro call by the time it winds its way to ':elixir_dispatch.expand_macro_fun.' It's a rollicking read full of surprises, regardless of whether you've mucked about with language internals before.