← Contents

Chapter 35

Measuring, and Proving

Almost every decision in this book was made against a number. This closing chapter is about how those numbers are produced, why several of them were wrong the first time, and what a release has to pass.

The benchmark policy

Stated once, applied everywhere:

• interleave the variants. Run A, B, A, B — not all of A then all of B — so machine drift and thermal state hit both equally.
• discard a warm-up, then report the median or the minimum of several repetitions. Which one depends on the noise shape; the important thing is to say which.
• keep a control kernel that the change under test cannot possibly affect.
• state the conditions: machine, architecture, compiler, date.

The control is the part people skip and the part that carries the argument. Chapter 18's table has a row for a pure method-dispatch loop, containing nothing node specialisation can touch:

Without that last row the others are only evidence that the machine was quieter the second time.

The performance gate

build/rakupp tools/perf-guard.raku --check

perf-guard compares the current binary against a recorded baseline and fails on a regression. The release checklist gates on it.

That it exists at all is a lesson learned twice: a release has shipped with a performance regression because someone eyeballed the numbers and thought they looked fine. Eyeballing does not work — the noise band is a few per cent, the regressions that matter are often a few per cent, and human judgement about which is which is unreliable at exactly that scale.

The rule that follows: a performance change is an interleaved A/B against perf-guard, not an opinion.

The correctness gates

Compiler agreement is the one that catches the most. Three execution paths must produce byte-identical output for the same program; a divergence is a bug in one of them, and it is found automatically rather than by someone noticing.

Roast is reported two ways, and the difference matters. Files fully passing is an all-or-nothing bar; assertions passing gives partial credit. Roughly ninety per cent of declared assertions pass. Both numbers are published, because either alone is misleading — a file can fail on one obscure assertion out of two hundred, and a file can "pass" by skipping everything.

Oracles

A test needs something to be right against. Four are used, in decreasing order of authority.

Roast. The specification. If Roast asserts it, it is the answer.

Rakudo. For anything Roast does not cover, the reference implementation is run side by side. The documentation harness does this in bulk: every documented example, on both engines, three-way classified.

The previous rakupp binary. For a change that is supposed to be semantically invisible — an optimisation — the baseline binary is the oracle, not another implementation. That habit came directly from a bug: the first evalIndex fast path wrapped negative subscripts from the end, and my $i = -1; @a[$i] must throw. Rakudo could not have caught it, because Rakudo rejects that spelling at compile time. Diffing against the previous binary did.

The language being implemented. The showcase interpreters compare their output against node, perl, python3 and the real Lisp — which is an oracle for thousands of lines of Raku that nobody wrote assertions for.

Error messages are not behaviour

A rule with a sharp edge: do not copy Rakudo's message prose unless Roast or the documentation asserts it.

Behaviour must match. Message wording need not, and chasing it produces churn that no test protects. Where a diagnostic is asserted, it is asserted by exception class and attributes rather than by text — which is why typed exceptions are built with attributes rather than formatted strings (Chapter 14).

What has and has not paid

The pattern across every optimisation in this book is consistent enough to state as a finding.

What paid — all of it removing work or allocation:

What did not pay:

• constant folding — measured first, with a corpus of 51,353 nodes; 0.07% of nodes, none of it in a loop. Not built.
• link-time optimisation and -mcpu=native — measured, no effect.
• -Os — smaller by nothing that matters, slower by 20 to 50% of the lane speed-up.
• a dispatch table for the method ladder — would chase 8.5% of a profile, and is not a drop-in because the ladder is guarded and order-sensitive.

The generalisation: on this codebase, removing an allocation has always paid and removing a branch almost never has. That is a property of a tree-walker over a fat value type, and it is worth re-deriving before assuming it holds somewhere else.

Three ways the measurement itself was wrong

Worth more than the successes.

Benchmarking a rename instead of the change it enables. An early analysis concluded that turning a builtin into a named C++ function was worth about 1 nanosecond per call. That was true for an out-of-line function still taking a ValueList — and completely missed what a real named function unlocks: direct Value arguments, no allocation, and an inlinable body. The corrected measurement was 5.6 times on an abs loop (Chapter 27).

Restructuring the shared path while adding a fast one. The first node specialisation made the control 5.7% slower, because binding an operand through an optional cost the general path its copy elision. The rule extracted: add an early exit, never restructure the code underneath it.

Assuming where the cost was. "I/O dominates, so call plumbing does not matter" was false: a buffered one-argument say costs about 125 nanoseconds, of which the plumbing was 45%. And the method dispatch ladder was exonerated twice by a correct observation — that a method 177 comparisons in cost the same as one 812 in — before a profiler showed the cost was strlen on a literal, not the ladder's length.

Where the remaining time is

For an interpreted method-heavy loop, after everything above:

Forty-two per cent in allocation and value churn. The shape of the fix is known — pass the invocant and argument list by reference rather than by value, and shrink Value — and both are being approached carefully rather than quickly, because the first trades away an accidental safety property and the second is a representation change that the extension ABI was specifically designed to survive (Chapter 32).

Honesty as a practice

Every chapter in this book has a "honest limitations" section, and that is a deliberate convention rather than a stylistic tic.

The list for the project as a whole: about ninety per cent of Roast; depth-first method resolution rather than C3; no ambiguity error in multiple dispatch; role composition that is last-writer-wins; modules that publish their whole environment; a flat class registry; no macros, no RakuAST, no slangs; laziness capped at a million elements; a gather block that can run more than once; a reference cycle that leaks.

Every one of those is a real difference from the reference implementation, and every one is written down where someone hitting it would look. That is the difference between a document and a brochure, and it is the reason the divergence lists in docs/dev/findings/ are maintained as running logs rather than occasionally cleaned up.

A last observation, which is the closest thing this book has to a thesis. The mechanisms in it are mostly ordinary — a Pratt parser, a tagged struct, a backtracking matcher, a transpiler. What made them work was not cleverness. It was measuring before building, keeping a control, writing down the reason next to the code, and being specific in public about what does not work yet.