By: Mosè Giordano
Re-posted from: http://giordano.github.io/blog/2022-06-18-first-macro/
This post isn’t about the first macro I wrote in the Julia programming
language, but it can be about your first macro.
Frequently Asked Questions
Question: What are macros in Julia?
Answer: Macros are sort of functions which take as input unevaluated expressions
(Expr) and return as output
another expression, whose code is then regularly evaluated at runtime. This post isn’t a
substitute for reading the section about macros in the Julia
documentation, it’s
more complementary to it, a light tutorial for newcomers, but I warmly suggest reading the
manual to learn more about them.
Calls to macros are different from calls to regular functions because you need to prepend
the name of the macro with the at-sign @ sign: @view A[4, 1:5] is a call to the
@view macro, view(A, 4, 1:5)
is a call to the view function.
There are a couple of important reasons why macro calls are visually distinct from function
calls (which sometimes upsets Lisp purists):
- the arguments of function calls are always evaluated before entering the body of the
function: inf(g(2, 5.6)), we know that the functiong(2, 5.6)will always be
evaluated before callingfon its result. Instead, the expression which is given as
input to a macro can in principle be discarded and never taken into account and the macro
can be something completely different: in@f g(2, 5.6), the expressiong(2, 5.6)is
taken unevaluated and rewritten into something else. The functiongmay actually never
be called, or it may be called with different arguments, or whatever the macro@fhas
been designed to rewrite the given expression; - nested macro calls are evaluated left-to-right, while nested function calls are evaluated
right-to-left: in the expressionf(g()), the functiong()is first called and then fed
intof, instead in@f @g xthe macro@fwill first rewrite the unevaluated
expression@g x, and then, if it is still there after the expression-rewrite operated by
@f(remember the previous point?), the macro@gis expanded.
Writing a macro can also be an unpleasant experience the first times you try it, because a
macro operates on a new level (expressions, which you want to turn into other expressions)
and you need to get familiar with new concepts, like
hygiene (more on this below), which can be
tricky to get initially right. However it may help remembering that the Expr macros
operate on are regular Julia objects, which you can access and modify like any other Julia
structure. So you can think of a macro as a regular function which operates on Expr
objects, to return a new Expr. This isn’t a simplification: many macros are actually
defined to only call a regular function which does all the expression rewriting business.
Q: Should I write a macro?
A: If you’re asking yourself this question, the answer is likely “no”. Steven G. Johnson
gave an interesting keynote speech at JuliaCon
2019 about metaprogramming (not just in
Julia), explaining when to use it, and more importantly when not to use it.
Also, macros don’t compose very well: remember that any macro can rewrite an expression in a
completely arbitrary way, so nesting macros can sometimes have unexpected results, if the
outermost macro doesn’t anticipate the possibility the expressions it operates on may
contain another macro which expects a specific expression. In practice, this is less of a
problem than it may sound, but it can definitely happens if you overuse many complicated
macros. This is one more reason why you should not write a macro in your code unless it’s
really necessary to substantially simplify the code.
Q: So, what’s the deal with macros?
A: Macros are useful to programmatically generate code which would be tedious, or very
complicated, to type manually. Keep in mind that the goal of a macro is to eventually get a
new expression, which is run when the macro is called, so the generated code will be
executed regularly, no shortcut about that, which is sometimes a misconception. There is
very little that plain (i.e. not using macros) Julia code cannot do that macros can.
Q: Why I should keep reading this post?
A: Writing a simple macro can still be a useful exercise to learn how it works, and
understand when they can be useful.
Our first macro: @stable
Globals in Julia are usually a major performance
hit, when their type is not
constant, because the compiler have no idea what’s their actual type. When you don’t need
to reassign a global variable, you can mark it as constant with the
const keyword, which greatly improves
performance of accessing a global variable, because the compiler will know its type and can
reason about it.
Julia v1.8 introduces a new way to have non-horribly-slow global variables: you can annotate
the type of a global variable, to say that its type won’t change:
julia> x::Float64 = 3.14
3.14
julia> x = -4.2
-4.2
julia> x = 10
10
julia> x
10.0
julia> x = nothing
ERROR: MethodError: Cannot `convert` an object of type Nothing to an object of type Float64
Closest candidates are:
convert(::Type{T}, ::T) where T<:Number at number.jl:6
convert(::Type{T}, ::Number) where T<:Number at number.jl:7
convert(::Type{T}, ::Base.TwicePrecision) where T<:Number at twiceprecision.jl:273
...
This is different from a const variable, because you can reassign a type-annotated global
variable to another value with the same type (or
convertible to the annotated
type), while reassigning a const variable isn’t possible (nor recommended). But the type
would still be constant, which helps the compiler optimising code which accesses this global
variable. Note that x = 10 returns 10 (an Int) but the actual value of x is 10.0
(a Float64) because assignment returns the right-hand side but the value 10 is converted to Float64 before
assigning it to x.
Wait, there is no macro so far! Right, we’ll get there soon. The problem with
type-annotated globals is that in the expression x::Float64 = 3.14 it’s easy to predict
the type we want to attach to x, but if you want to make x = f() a type-annotated global
variable and the type of f() is much more involved than Float64, perhaps a type with
few parameters, then doing
the type annotation can be annoying. Mind, not impossible, just tedious. So, that’s where
a macro could come in handy!
The idea is to have a macro, which we’ll call @stable, which operates like this:
@stable x = 2
will automatically run something like
x::typeof(2) = 2
so that we can automatically infer the type of x from the expression on the right-hand
side, without having to type it ourselves. A useful tool when dealing with macros is
Meta.@dump (another
macro, yay!).
julia> Meta.@dump x = 2
Expr
head: Symbol =
args: Array{Any}((2,))
1: Symbol x
2: Int64 2
This tells us how the expression x = 2 is parsed into an Expr, when it’s fed into a
macro. So, this means that our @stable x = 2 macro will see an Expr whose ex.args[1]
field is the name of the variable we want to create and ex.args[2] is the value we want to
assign to it, which means the expression we want to generate will be something like
ex.args[1]::typeof(ex.args[2]) = ex.args[2], but remember that you need to
interpolate
variables inside a quoted
expression:
julia> macro stable(ex::Expr)
return :( $(ex.args[1])::typeof($(ex.args[2])) = $(ex.args[2]) )
end
@stable (macro with 1 method)
julia> @stable x = 2
2
Well, that was easy, it worked at the first try! Now we can use our brand-new type-stable
x! Let’s do it!
julia> x
ERROR: UndefVarError: x not defined
Waaat! What happened to our x, we just defined it above! Didn’t we? Well, let’s use
yet another macro,
@macroexpand, to see
what’s going on:
julia> @macroexpand @stable x = 2
:(var"#2#x"::Main.typeof(2) = 2)
Uhm, that looks weird, we were expecting the expression to be x::typeof(2), what’s that
var"#2#x"? Let’s see:
julia> var"#2#x"
ERROR: UndefVarError: #2#x not defined
Another undefined variable, I’m more and more confused. What if that 2 in there is a
global counter? Maybe we need to try with 1:
julia> var"#1#x"
2
julia> var"#1#x" = 5
5
julia> var"#1#x"
5
julia> var"#1#x" = nothing
ERROR: MethodError: Cannot `convert` an object of type Nothing to an object of type Int64
Closest candidates are:
convert(::Type{T}, ::T) where T<:Number at number.jl:6
convert(::Type{T}, ::Number) where T<:Number at number.jl:7
convert(::Type{T}, ::Base.TwicePrecision) where T<:Number at twiceprecision.jl:273
...
Hey, here is our variable, and it’s working as we expected! But this isn’t as convenient
as calling the variable x as we wanted. I don’t like that. what’s happening? Alright,
we’re now running into hygiene, which we mentioned above: this isn’t about washing your
hands, but about the fact macros need to make sure the variables in the returned expression
don’t accidentally clash with variables in the scope they expand to. This is achieved by
using the gensym function to
automatically generate unique identifiers (in the current module) to avoid clashes with
local variables.
What happened above is that our macro generated a variable with a gensym-ed name, instead
of the name we used in the expression, because macros in Julia are hygienic by default. To
opt out of this mechanism, we can use the
esc function. A rule of thumb is
that you should apply esc on input arguments if they contain variables or identifiers from
the scope of the calling site that you need use as they are, but for more details do read
the section about hygiene in the Julia
manual. Note also that
the pattern var"#N#x", with increasing N at every macro call, in the gensym-ed
variable name is an implementation detail which may change in future versions of Julia,
don’t rely on it.
Now we should know how to fix the @stable macro:
julia> macro stable(ex::Expr)
return :( $(esc(ex.args[1]))::typeof($(esc(ex.args[2]))) = $(esc(ex.args[2])) )
end
@stable (macro with 1 method)
julia> @stable x = 2
2
julia> x
2
julia> x = 4.0
4.0
julia> x
4
julia> x = "hello world"
ERROR: MethodError: Cannot `convert` an object of type String to an object of type Int64
Closest candidates are:
convert(::Type{T}, ::T) where T<:Number at number.jl:6
convert(::Type{T}, ::Number) where T<:Number at number.jl:7
convert(::Type{T}, ::Base.TwicePrecision) where T<:Number at twiceprecision.jl:273
...
julia> @macroexpand @stable x = 2
:(x::Main.typeof(2) = 2)
Cool, this is all working as expected! Are we done now? Yes, we’re heading into the
right direction, but no, we aren’t quite done yet. Let’s consider a more sophisticated
example, where the right-hand side is a function call and not a simple literal number, which
is why we started all of this. For example, let’s define a new type-stable variable with a
rand() value, and let’s print
it with one more macro, @show,
just to be sure:
julia> @stable y = @show(rand())
rand() = 0.19171602949009747
rand() = 0.5007039099074341
0.5007039099074341
julia> y
0.5007039099074341
Ugh, that doesn’t look good. We’re calling rand() twice and getting two different
values? Let’s ask again our friend @macroexpand what’s going on (no need to use @show
this time):
julia> @macroexpand @stable y = rand()
:(y::Main.typeof(rand()) = rand())
Oh, I think I see it now: the way we defined the macro, the same expression, rand(), is
used twice: once inside typeof, and then on the right-hand side of the assignment, but
this means we’re actually calling that function twice, even though the expression is the
same. Correct! And this isn’t good for at least two reasons:
- the expression on the right-hand side of the assignment can be expensive to run, and
calling it twice wouldn’t be a good outcome: we wanted to create a macro to simply things,
not to spend twice as much time; - the expression on the right-hand side of the assignment can have side effects, which is
precisely the case of therand()function: every time you callrand()you’re advancing
the mutable state of the random number generator, but if you call it twice instead of
once, you’re doing something unexpected. By simply looking at the code@stable y =, someone would expect that
rand()rand()is called exactly once, it’d be bad if users
of your macro would experience undesired side effects, which can make for hard-to-debug
issues.
In order to avoid double evaluation of the expression, we can assign it to another temporary
variable, and then use its value in the assignment expression:
julia> macro stable(ex::Expr)
quote
tmp = $(esc(ex.args[2]))
$(esc(ex.args[1]))::typeof(tmp) = tmp
end
end
@stable (macro with 1 method)
julia> @stable y = @show(rand())
rand() = 0.5954734423582769
0.5954734423582769
This time rand() was called only once! That’s good, isn’t it? It is indeed, but I think
we can still improve the macro a little bit. For example, let’s look at the list of all
names defined in the current module with
names. Can you spot anything
strange?
julia> names(@__MODULE__; all=true)
13-element Vector{Symbol}:
Symbol("##meta#58")
Symbol("#1#2")
Symbol("#1#x")
Symbol("#5#tmp")
Symbol("#@stable")
Symbol("@stable")
:Base
:Core
:InteractiveUtils
:Main
:ans
:x
:y
I have a baad feeling about that Symbol("#5#tmp"). Are we leaking the temporary variable
in the returned expression? Correct! Admittedly, this isn’t a too big of a deal, the
variable is gensym-ed and so it won’t clash with any other local variables thanks to
hygiene, many people would just ignore this minor issue, but I believe it’d still be nice to
avoid leaking it in the first place, if possible. We can do that by sticking the
local keyword in front of the
temporary variable:
julia> macro stable(ex::Expr)
quote
local tmp = $(esc(ex.args[2]))
$(esc(ex.args[1]))::typeof(tmp) = tmp
end
end
@stable (macro with 1 method)
julia> @stable y = rand()
0.7029553059625194
julia> @stable y = rand()
0.04552255224129409
julia> names(@__MODULE__; all=true)
13-element Vector{Symbol}:
Symbol("##meta#58")
Symbol("#1#2")
Symbol("#1#x")
Symbol("#5#tmp")
Symbol("#@stable")
Symbol("@stable")
:Base
:Core
:InteractiveUtils
:Main
:ans
:x
:y
Yay, no other leaked temporary variables! Are we done now? Not yet, we can still make it
more robust. At the moment we’re assuming that the expression fed into our @stable macro
is an assignment, but what if it isn’t the case?
julia> @stable x * 12
4
julia> x
4
Uhm, it doesn’t look like anything happened, x is still 4, maybe we can ignore also
this case and move on. Not so fast:
julia> 1 * 2
ERROR: MethodError: objects of type Int64 are not callable
Maybe you forgot to use an operator such as *, ^, %, / etc. ?
Aaargh! What does that even mean?!? Let’s ask our dear friends Meta.@dump and
@macroexpand:
julia> Meta.@dump x * 2
Expr
head: Symbol call
args: Array{Any}((3,))
1: Symbol *
2: Symbol x
3: Int64 2
julia> @macroexpand @stable x * 12
quote
var"#5#tmp" = x
(*)::Main.typeof(var"#5#tmp") = var"#5#tmp"
end
julia> *
4
Let me see if I follow: with @stable x * 12 we’re assigning x (which is now
ex.args[2]) to the temporary variable, and then the assignment is basically * = 4,
because ex.args[1] is now *. Ooops. Brilliant! In particular we’re shadowing * in
the current scope (for example the Main module in the REPL, if you’re following along in
the REPL) with the number 4, the expression 1 * 2 is actually equivalent to *(1, 2),
and since * is 4
julia> 4(1, 2)
ERROR: MethodError: objects of type Int64 are not callable
Maybe you forgot to use an operator such as *, ^, %, / etc. ?
Gotcha! So we should validate the input? Indeed, we should make sure the expression
passed to the macro is what we expect, that is an assignment. We’ve already seen before
that this means ex.head should be the symbol =. We should also make sure the left-hand
side is only a variable name, we don’t want to mess up with indexing expressions like A[1]:
= 2
julia> Meta.@dump A[1] = 2
Expr
head: Symbol =
args: Array{Any}((2,))
1: Expr
head: Symbol ref
args: Array{Any}((2,))
1: Symbol A
2: Int64 1
2: Int64 2
Right, so ex.head should be only = and ex.args[1] should only be another symbol. In
the other cases we should throw a useful error message. You’re getting the hang of it!
julia> macro stable(ex::Expr)
(ex.head === :(=) && ex.args[1] isa Symbol) || throw(ArgumentError("@stable: `$(ex)` is not an assigment expression."))
quote
tmp = $(esc(ex.args[2]))
$(esc(ex.args[1]))::typeof(tmp) = tmp
end
end
@stable (macro with 1 method)
julia> @stable x * 12
ERROR: LoadError: ArgumentError: @stable: `x * 12` is not an assigment expression.
julia> @stable A[1] = 2
ERROR: LoadError: ArgumentError: @stable: `A[1] = 2` is not an assigment expression.
Awesome, I think I’m now happy with my first macro! Love it! Yes, now it works pretty
well and it has also good handling of errors! Nice job!
Conclusions
I hope this post was instructive to learn how to write a very basic macro in Julia. In the
end, the macro we wrote is quite short and not very complicated, but we ran into many
pitfalls along the way: hygiene, thinking about corner cases of expressions, avoiding
repeated undesired evaluations and introducing extra variables in the scope of the macro’s
call-site. This also shows the purpose of macros: rewriting expressions into other ones, to
simplify writing more complicated expressions or programmatically write more code.
This post is inspired by a macro which I wrote some months ago for the Seven Lines of
Julia
thread on JuliaLang Discourse.
This was cool! Where can I learn more about macros? Good to hear! I hope now you aren’t
going to abuse macros though! But if you do want to learn something more about macros, in
addition to the official documentation, some useful resources are:
- A Brief Introduction to Julia Metaprogramming, Macros, and Generated
Functions - Introduction to macros | Week 3 | 18.S191 MIT Fall
2020 - Julia Macro hygiene made easy! – Tom Kwong
- Julia macros for
beginners