What is metaprogramming?

Question

Sometimes we hear about "Metaprogramming".

• What is?
• What good is?
• How and when to use?

Answer 1

To reply from Guilherme Oderdenge demonstrates well the concept of metaprogramming: "programs that create programs". In my reply, I will give a more practical approach (rarely does anyone write a program whose output is other, except for fun or for being writing a compiler...).

Many programming tasks are repetitive. Although the purpose of the computer is to help us with repetition, in practice this is not always possible. Due to limitations in programming languages (among other computational tools), we often find ourselves writing the same things over and over and over again. If you cannot directly write a program that avoids these repetitions, what remains is to write a different program and then change it through an automated routine (a "meta-program").

In general, metaprogramming is all programming that acts on another program, either in source, binary, or in an abstract representation in memory.

Transforming the source code

The best known example is #define of C. It allows replacing a complex expression with a simpler one, and the replacement is done by the compiler itself:

#define CURSOR(top, bottom) (((top) << 8) | (bottom))

Another example is the eval, supported by several languages. It allows a chunk of code to be formed using not only static text but also dynamic components:

eval("function(x) { alert(x + " + y + "); }"); // Ex.: function(x) { alert(x + 10); }

Transforming the binary code

The Aspectj is a tool that brings the Programming Oriented by Aspects to the Java language. One of its modes of use is to insert new code to existing classes directly in binary code (bytecode-Weaving). One or more well-defined points are chosen in the execution of a program, and code is written to be executed at those points:

pointcut set() : execution(* set*(..) ) && this(Point);

after () : set() {
    Display.update();
}

The above code executes Display.update(); after the execution of every class method Point whose name begins with set.

Transforming an abstract representation

Every language that has first-class functions opens space for them to be programmatically modified. The simplest way is through decorators:

function checarNulos(f) {
    return function() {
        for ( var i = 0 ; i < arguments.length ; i++ )
            if ( arguments[i] == null )
                throw 'Parâmetro nulo';
        return f.apply(this, arguments);
    }
}

var func = checarNulos(function(a, b) { return a + b; });

In this case it is turning a simple function - without any error check - into another more robust (fail-fast). This is done in a generic way and can be applied to different types of function, with different quantity and format of parameters.

Another possibility, if the language has "first-class classes", is to apply transformations in a class as a whole (e.g..: metaclasses).

In addition to these cases, which I particularly call "black box" (because only the "boundary" of the code is changed, without knowledge of its content), languages homoiconic (as Lisp, Prolog or XSLT) also allow the transformation of the code as a whole - including specific instructions, expressions, reaching up to the terminal symbols. These transformations are generically called "macros" (although they are much more flexible than the "macros" of the C family).

term_expansion((Cabeca :- Cauda), (Cabeca :- NovaCauda)) :-
    transforma(Cauda, NovaCauda).

transforma((A, B), (NA, NB)) :-
    transforma(A, NA),
    transforma(B, NB).

transforma(A is B + C, A is C + B).
transforma(X, X).

The above code takes every "add two variables" statement and reverses the order of the operands. If there are two or more instructions separated by And, It transforms both the first and the second part. Although this code performs "at compile time" (i.e. before the function definition is incorporated into the program), it already acts on an abstract representation of the source code, not on the text itself (it would be like the AST, but in Prolog there is no difference between "code" and "data", so the program’s AST is represented by the language’s own data structures).

Finally, even languages without "first-class X" can still support some form of metaprogramming, in the form of reflection (Reflection). It consists of inspecting the structure of your data and acting on it in a generic way:

void funcaoDemorada(int argumento, float argumento2, Object callback, String nomeMetodo) {
    ...
    Method m = callback.getClass().getMethod(nomeMetodo);
    m.invoke(callback);
}

This example defines a function that can call any other function [public, zero parameters] on an object, simply knowing its name. Without this, it would be necessary that the object callback implement a specific interface, and define a single method in accordance with that interface. In this way, objects of totally different classes and with different methods can be used equally.

Note: several of the above examples cannot be considered good practice, and there are probably better ways to achieve the same goal. They are there only to exemplify the techniques mentioned.

In conclusion, metaprogramming is much broader than "a program generating another program as output", being several cases in which a program (or excerpt from a program) is used to manipulate another program - turn it into something other than what it would be if it were interpreted literally, as the programmer wrote it. It is likely that someone disagrees with the limit of what is "normal programming" (i.e. the simple use of the language’s own functionalities) and what is "meta-programming", but I hope I have conveyed the general idea.

Answer 2

Imagine a man who builds cars.

Sometimes he realizes he’s always doing the same thing. So he builds factories to build his cars, which is much more productive. He is now programming!

Time goes by and yet, once again, he realizes that he is always doing the same thing again: building factories to build his cars. Now, then, he decides to build factories that manufacture factories to build cars. That is meta programming.

Metaprogramming is immensely useful and powerful, but a problem in its architecture can make all the advantages turn into a monstrous headache. So when you’ve perfected yourself, feel free to use... or stay away!

_{Response adapted freely from SOEN.}

An analogy of a few words

Metaprogramming is creating programs that create programs.

And technically speaking, how do we look?

Metaprogramming consists of a variety of ways about how a program can manipulate itself: we can use macros, Reflection, templates, etc and before you ask yourself which is the best of them I tell you to forget. It’s a bit of a cliché, but it will depend not only on your problem, but on your language, knowledge and dominance - they’re a set of factors and measuring which is definitely the best, at the chance.

How and when to use?

I’m gonna kill two birds with one stone free translation (source):

Erika is an intelligent first-year computer science student. She already knows several programming languages, including C and Ruby. During her introduction to programming class, Professor Gomez, the course’s instructor, caught her chatting via chat on her laptop. As a form of punishment, he ordered Erika to write a program in C that displayed 1,000 lines of the following text:

1) I won’t talk in class.

2) I won’t talk in class.

[...]

999) I won’t talk in class.

1000) I won’t talk in class.

An additional constraint was that the program could not have any kind of loop or "goto" instructions. It should contain only a large function with 1,000 printfs. Something like that:

#include <stdio.h>
int main(void) {
    printf("1) Eu não vou conversar em aula.\n");
    printf("2) Eu não vou conversar em aula.\n");

   /* 996 printfs omitidos. */

    printf("999) Eu não vou conversar em aula.\n");
    printf("1000) Eu não vou conversar em aula.\n");
    return 0;
}

Only Professor Gomez was naive, so he basically waited for Erika to write the instruction printf once and copy to the Clipboard to then paste 999 times to finally change the numbers. He believed that this tiresome and evocative work was enough to teach her a lesson. But Erika immediately found a solution: metaprogramming. Instead of writing this program in hand, why not write another program that writes this program automatically for her? So she wrote the following ruby script:

File.open('punishment.c', 'w') do |output|
    output.puts '#include <stdio.h>'
      output.puts 'int main(void) {'
      1.upto(1000) do |i|
          output.puts "    printf(\"#{i}. " +
          "Eu não vou conversar em aula.\\n\");"
      end

    output.puts '    return 0;'
    output.puts '}'
end

The above code created a file called Punishment. c providing for 1000+ lines of code C.

Do you understand the usefulness of metaprogramming? As even the article mentions, this is a very figurative example. In real life, I will use the example that the same [article] gave, only that I will translate with my words:

You need to include a PNG image in a C program. Unfortunately, for some reason, the delivery engine of this program accepts only one file: the executable. So, somehow, we need to include the PNG image information inside that file. To solve this, we can also do in the previous example: read the PNG image with a Ruby script and through it inject into the C code.

Code example:

INPUT_FILE_NAME = 'ljlogo.png'
OUTPUT_FILE_NAME = 'ljlogo.h'
DATA_VARIABLE_NAME = 'ljlogo'


File.open(INPUT_FILE_NAME, 'r') do |input|
  File.open(OUTPUT_FILE_NAME, 'w') do |output|
    output.print "unsigned char #{DATA_VARIABLE_NAME}[] = {"
    data = input.read.unpack('C*')
    data.length.times do |i|
      if i % 8 == 0
        output.print "\n    "
      end
      output.print '0x%02X' % data[i]
      output.print ', ' if i < data.length - 1
    end
    output.puts "\n};"
  end
end

Basically, this is it.