← Contents

Preface

This is a book about the inside of a compiler.

Raku++ is a hand-written implementation of the Raku programming language, written in C++17, with no third-party dependencies. It lexes, parses, interprets, and — in two of its five run modes — transpiles Raku to C++ and to JavaScript, and compiles it to a native binary. Two more targets write a subset of the language for a microcontroller board: as an Arduino sketch, or as C++ for a BBC micro:bit. It carries its own regular-expression and grammar engine, its own Unicode subsystem built from the pinned UCD tables, its own arbitrary-precision arithmetic, a foreign-function interface that loads libffi at run time rather than linking it, a C ABI for native extension modules, and a concurrency runtime that runs interpreter compute on all cores by default, with a global interpreter lock kept as the opt-in escape hatch.

None of that is unusual for a language implementation. What is unusual is that all of it fits in about a quarter of a million hand-written lines that one person can read, and that almost every non-obvious decision in it was made against a measurement. This book is an attempt to write down both: the mechanisms, and the reasons.

Who this is for

You should be comfortable reading C++ and comfortable reading Raku, but you do not need to be an expert in either, and you certainly do not need to have implemented a language before. Where a piece of compiler folklore is load-bearing — precedence climbing, Thompson construction, packrat memoisation, copy-on-write — it is explained where it is used rather than assumed, and Appendix D collects the whole vocabulary, general and local, in one alphabetical list.

Three kinds of reader were in mind:

What this book is not

It is not a manual for the language, and not a manual for the rakupp command. Those exist: the guide in docs/guide/ covers using Raku++, and docs/guide/REFERENCE.md is the exhaustive per-routine lookup sheet. Nor is it a specification of Raku; the reference implementation, Rakudo, and the Roast test suite hold that role, and this book is careful to say where Raku++ diverges from them rather than pretending it does not.

It is also not a tour of every function. The source is the normative reference, and it moves. What is written down here is the shape — the data structures that everything else is built on, the invariants they rely on, and the reasons a particular design was chosen over the obvious alternative.

How to read it

The ten parts are ordered the way a program flows through the system: source text, then the tree, then the values, then execution, then the specialised engines, then the back ends, then the boundaries with the outside world. Read straight through and it is a narrative. Read a single part and it should still stand on its own; cross-references say where the rest of a thread lives.

If you are here for one thing in particular:

You wantStart at
the overall mapChapter 1, then Chapter 25
what kind of compiler this isChapter 3
how source becomes a treePart II
what a runtime value isChapter 8
how a Raku call actually happensChapters 14 to 16
regexes and grammarsPart V
the native compilerPart VII
Raku in a browserChapter 32
installing modules, and the store zef sharesChapter 34
calling C, or being called from itChapters 36 and 37
a loop that compiles itself while it runsChapter 44
a term you have not met beforeAppendix D, the glossary

Conventions

Source excerpts are tagged with the file they come from:

// src/Value.h
static Value integer(long long x) { Value v; v.t = VT::Int; v.i = x; return v; }

They are lightly trimmed for the page — an ellipsis, a dropped comment, an elided error string — but are otherwise verbatim. File names, not line numbers, are the anchors. The code moves; grep for the function or for the quoted line.

Raku code is shown as Raku:

my @primes = (2..*).grep(*.is-prime);
say @primes.head(5);          # (2 3 5 7 11)

Measured numbers carry their conditions. Unless a chapter says otherwise, they were taken on arm64 macOS with Apple clang, using the project's benchmark policy: interleaved A/B runs, a discarded warm-up, medians or minima of several repetitions, and a control kernel that the change under test cannot affect. A number without a control is an anecdote, and this book tries not to print anecdotes.

Sections headed What went wrong are not decoration. Several of the designs here are the second or third attempt, and the discarded attempts are usually more instructive than the surviving one.

A word about honesty

Raku is a very large language, and Raku++ does not implement all of it. Every test the Roast suite expects an implementation to pass does pass — 219,395 of 219,395 at v5.6.0, skip and todo left out — and that is a statement about the suite, not about the language. Method dispatch walks the parents depth-first rather than in C3 order, so a diamond hierarchy can pick a different method than Rakudo does. Macros are not implemented at all, and slangs only where the lexer has a seam to hang a token on. Two entries that stood in this list when the book was first written have gone: multiple dispatch now raises X::Multi::Ambiguous on a tie, and a module's importer sees its exports and not the rest of its environment.

Every one of those is stated in the chapter where it belongs, under a heading that says so. A book about compiler internals that only described the parts that work would be a brochure.