Raku++ Internals

How a hand-written C++ implementation of Raku works — from source text to native code.

Download the PDF 320 pages · 1.1 MB

  1. Prefacewho this is for, and the conventions it uses

Ground

  1. Chapter 1What Raku++ Isthe goals, the four run modes, the shape of the system
  2. Chapter 2The Shape of the Sourcethe file map, and how to navigate it
  3. Chapter 3Where It Sits in the Taxonomywhy neither LL nor LR describes this compiler

The Front End

  1. Chapter 4The Lexertokens, significant whitespace, quotes, heredocs
  2. Chapter 5The Parserrecursive descent, a Pratt core, and no compile-time execution
  3. Chapter 6User-Defined Operators in a Single Passlive tables, precedence traits, and lexical rollback
  4. Chapter 7The ASTthe node hierarchy, and fields that answer questions about syntax

The Value Model

  1. Chapter 8Value: the Fat Tagged Structone fat tagged struct for every runtime value
  2. Chapter 9Strings: CowStrthe copy-on-write string, and the quadratic it removed
  3. Chapter 10Interning and Comparison: IStr and MNameinterned tags and packed-prefix name comparison
  4. Chapter 11Numbersnative ints, bignums, rationals, and the spill between them
  5. Chapter 12Containers, Binding, and Copy Semanticsscopes, binding, and where sharing is broken on purpose

The Interpreter

  1. Chapter 13The Tree Walkeval, exec, and the per-thread execution registers
  2. Chapter 14Calls and Parameter Bindingbuilding arguments, activating a routine, binding parameters
  3. Chapter 15Cooperative Control Flowreturn and friends without a C++ throw
  4. Chapter 16Dispatchsubs, multis, the method ladder, and re-dispatch
  5. Chapter 17The Object Systemclasses, roles, mixins, and the meta-object surface
  6. Chapter 18Laziness, Junctions, and the Wilder Valuesinfinite sequences, gather without coroutines, junctions
  7. Chapter 19Node Specializationone optimisation, measured end to end

The Regex and Grammar Engine

  1. Chapter 20Compiling a Regexa second recursive-descent compiler inside the runtime
  2. Chapter 21The Backtracking Matchercontinuation passing, backtracking, and a step budget
  3. Chapter 22Grammarsrules, packrat memoisation, actions, protos
  4. Chapter 23Longest-Token Matchingranking alternations by declarative prefix

Unicode

  1. Chapter 24Graphemes, Normalization, Collationgraphemes, normalization, collation — from pinned tables

The Back Ends

  1. Chapter 25Four Ways to Run a Programone program through interpret, bundle, AOT and native
  2. Chapter 26The Code GeneratorAST to C++, calling the same runtime the interpreter calls
  3. Chapter 27The -O Optimizerthree passes, and three defaults that are not passes
  4. Chapter 28Dispatch Cost in Compiled Codewhat each call shape costs, and an analysis that was wrong
  5. Chapter 29What a Compiled Binary Keepshow a binary drops what the program cannot reach
  6. Chapter 30AST Serialization and the Precompiled Parsethe binary AST format, and what invalidates a cache entry
  7. Chapter 31Raku in the Browserthe same runtime in a browser tab, Live mode, and embedded editors

Boundaries

  1. Chapter 32Moduleswhat use Foo actually does, and where it diverges
  2. Chapter 33The Installer and the Store
  3. Chapter 34use nqp: a Compatibility Subseta compatibility subset that costs nothing when unused
  4. Chapter 35NativeCall and the libffi Backendlibffi loaded at run time, and its ABI probed not tabulated
  5. Chapter 36The Extension ABIthe C ABI an extension compiles against, and why it never sees Value
  6. Chapter 37Concurrencythe GIL, per-thread registers, and switching the lock off

Around the Compiler

  1. Chapter 38Tooling Built on the ASTlint, highlight, profile, the REPL, and tools written in Raku
  2. Chapter 39Measuring, and Provingthe benchmark policy, the gates, and what did not pay

The Speed Campaign

  1. Chapter 40Constant Factors
  2. Chapter 41Reading Other Engines

Appendices

  1. Appendix AThe Tag Setsthe VT, NK and regex-node tag sets
  2. Appendix BFlags and Environment Variablesevery flag and environment variable
  3. Appendix CSource Map and Glossarya source map, the rules that are easy to break, a glossary