← Contents

Chapter 1

What Raku++ Is

Raku++ is a Raku implementation written from scratch in C++17. It has no third-party dependencies: no parser generator, no regex library, no bignum library, no ICU, no libffi at link time, no tracing garbage collector. The binary it produces is one file, and the same source builds it on macOS, Linux, the BSDs, Windows, and — compiled to WebAssembly — inside a browser tab.

That constraint is not asceticism for its own sake. It has a specific consequence which shapes almost every chapter of this book: when something is slow or wrong, the code responsible is in the tree, and it can be changed. A good deal of what follows is the story of doing exactly that.

The pipeline

                            ┌─ interpret ─► tree-walk (Interpreter)
                            │
 source ─► Lexer ─► Parser ─┼─ --aot ─► AstEmit ─► cc ─► binary
                  (─► AST)  │              (rebuilds the tree, walks it)
                            │
                            └─ --exe ─► Codegen ─► cc ─► binary
                                           (native; no interpreter inside)

 source ─────────── --bundle ─► embed the bytes ─► cc ─► binary
                                   (parses and walks at run time)

A single front end feeds five back ends, and since v5.6.0 a sixth, which writes an Arduino sketch; after v5.8.0 a seventh, the same emitter with a second board, writes C++ for a BBC micro:bit. The front end is a hand-written character-level lexer and a recursive-descent parser with a precedence-climbing core for expressions; it produces a Program, which is a vector of statement nodes. Everything downstream consumes that one artifact.

Everything a running program needs is compiled into a static library, librakupp_rt.a — "the runtime" — and four small archives beside it that a compiled binary can leave out (Chapter 29). The rakupp executable links against them, and so does every binary the compiling modes produce. What stays out of the library is what only the command line can reach: the driver, the REPL, and the C++, JavaScript and board code generators. That is why a feature implemented once behaves identically whether a program is interpreted or natively compiled: there is one implementation of Value semantics, one set of built-ins, one method dispatcher, one regex engine, one Unicode subsystem.

The four modes, in one paragraph each

Interpret (the default) lexes, parses, and tree-walks. say "Hello, World!" takes 4.4 ms, process startup included (v5.5.0, on an Apple M4); throughput pays the per-node dispatch cost. This is the mode the language is developed against, because it is the only one that can run every construct.

--bundle embeds the source bytes in a generated C++ stub and links it against the runtime. The result is standalone, but at run time it still lexes, parses and tree-walks — it is the interpreter in a box. The win is distribution, not speed.

--aot parses at build time and emits C++ that rebuilds the identical AST at startup, then interprets it. Parse errors surface at build time rather than run time; execution speed is the same as bundling, because it is the same tree walk. The tree travels as bytes: the serialiser of Chapter 30 writes it, the generated program carries the blob as an array, and the same serialiser reads it back. It used to emit one builder function per node, listing each node type's fields by hand, and a field nobody listed did not survive.

--exe parses at build time and transpiles the tree into C++ that implements the program directly: native control flow, native calls, with the runtime called only for value semantics. With -O it additionally runs its own optimizer before the C++ compiler sees anything. Anything it cannot transpile throws a CodegenError, which the driver catches and answers by bundling the whole program instead — so --exe never refuses a program.

Chapter 25 works one small program through all four.

Three further targets produce something other than a program for this machine. --target=js transpiles to JavaScript over a runtime written in JavaScript (Chapter 31). --target=arduino, new in v5.6.0, writes a subset of Raku — native scalars, shaped native arrays, subs with native signatures — as one Arduino sketch, and --target=microbit writes the same subset as C++ for a BBC micro:bit's runtime, CODAL. A board has no Raku runtime to call, so there every construct either becomes plain C++ or is refused, with its line, before anything is built.

What the design is optimised for

Three decisions dominate everything else, and each gets a full chapter later.

One value type. Every Raku value at runtime is the same C++ struct, Value: a one-byte tag, a word for a number, a string, and one slot for whichever larger payload the tag calls for. Not a std::variant, not a class hierarchy — a fat struct, 32 bytes on 64-bit macOS and Linux. Chapter 8 argues the case, which is stronger than it first looks, and prices the memory it costs.

The C++ stack is the Raku stack. There is no bytecode and no separate operand stack. eval(Expr*) returns a Value; exec(Stmt*) runs a statement. This makes the implementation small and makes native recursion depth the Raku recursion budget, which is why the entry point runs the program on a thread with a very large stack.

Compile time runs almost nothing. BEGIN becomes a tagged block that the interpreter schedules after the parse; constant is not folded; use loads no ordinary module during parsing. There are two parse-time side effects. The first is registering a user-declared operator in the parser's own tables, and Chapter 6 is about why that one is enough to support custom operators with real precedence in a single forward pass. The second is use of a slang: that module alone is run during the parse, so that the tokens it registers can be spliced into the lexer for the rest of the unit. Macros need a third thing neither provides — a user-built AST substituted for the call site — and are not supported.

A compiled binary is a separate matter, and since v5.6.0 it does run code while it is built. --exe, --aot and --bundle run the program's compile-time part in an interpreter at build time — BEGIN, CHECK, use, the declarations, the constants, and no mainline — and write each plain-data value into the binary in place of the code that computed it, as a precompiled Rakudo module keeps its constants. That happens after the parse and does not feed it: the parser still learns nothing from a BEGIN block.

What it deliberately is not

It is not the reference implementation. Rakudo is, and Raku++ is measured against Rakudo and against the Roast suite continuously rather than aspirationally. The current position is every test Roast expects an implementation to pass: 219,395 of 219,395 at v5.6.0, in all 1,425 files of its spectest.data.

It has no tracing garbage collector, and no JIT unless one is asked for: the two tier-up back ends, --jit and --cnp, are off by default and are Chapter 44. Lifetime is reference counting, which frees a value the moment its last reference goes away. Beside it runs a backup cycle collector that looks only for groups of objects that refer to each other, which counting alone would leak; Chapter 12 describes it.

It implements compile-time metaprogramming only in part. macro and quasi are absent, for the structural reason given above. Swapping the grammar for a lexical scope — a slang — works at a fixed set of seams: the module is run and its tokens are called where the lexer would have started the productions it overrode, and a slang that reaches past them is refused by name. RakuAST turned out not to belong in that group: the tree a program asks for is a view built over the parse that has already happened, so nothing has to run mid-parse and nothing has to rewrite a grammar. It is implemented — the classes, .AST, .DEPARSE, .EVAL, visit-children and .rakudoc. What is supported is everything that can be done by adding an entry to a table during a single forward pass: all six custom-operator categories with precedence and associativity traits, plus the whole runtime meta-object surface (augment, .^add_method, does/but mixins, &routine.wrap).

The ecosystem around the compiler

Several things in this book are not in the rakupp binary but are load-bearing for it, and appear where they are relevant:

ThingWhat it is
Raku.jsthe same runtime compiled to WebAssembly — the interpreter in a browser
the Roast harnesstools/run-roast.raku, written in Raku, run by rakupp against the spec suite
perf-guardtools/perf-guard.raku, the release gate that fails a regression instead of eyeballing one
the showcase interpretersJavaScript, Perl 5, Python 3, Lisp and Forth interpreters written in Raku, used as beyond-Roast tests
the ::Native modulesextension modules in C — CSV::Native, TOML::Native, YAML::Native and others; the first, JSON::Native 0.0.x, was the proof the extension ABI works, and has shipped no C since 0.1.0

The showcase programs deserve a note here because they recur. An interpreter for another language, written in Raku and run by Raku++, exercises grammars, recursion, string handling, closures and dispatch simultaneously and at a scale no unit test reaches. Each of them, when first written, produced a list of genuine bugs in Raku++; several fixes in later chapters were found that way.

A map of this book

PartCovers
Ithis chapter, the layout of the source, and where the design sits in the compiler taxonomy
IIlexer, parser, user-defined operators, the AST
IIIValue, strings, interning, numbers, containers
IVthe tree walk, calls, control flow, dispatch, objects, laziness
Vthe regex engine, the grammar engine, longest-token matching
VIUnicode: graphemes, normalization, collation
VIIthe five run modes, the two code generators, -O, what a binary keeps, serialization, the browser
VIIImodules, use nqp, NativeCall, the extension ABI, concurrency
IXtooling built on the AST, and how any of this is proved
Xthe speed campaign: constant factors, other engines, and compiling while it runs