Chapter 6 · July – September

Other people's code

Roast tests the language in the small: it isolates each feature. Real programs combine a dozen modules, a database driver and a lot of string handling, and they phrase things the way people actually write them. From mid-July the project ran other people's code as well, and diffed its output against Rakudo.

148 → 14real divergences across 3,068 course snippets

2,663 → 4,056Weekly Challenge solutions byte-identical to Rakudo

596 → 1,006documentation examples identical on both engines

624 → 1,019ecosystem distributions passing their own test suites

The turnOn 31 July (v1.5.2) the bar for a module changed. Loading it and calling one function no longer counted. A module counts as working only when its own test suite passes, the files zef runs at install time. Nearly every fix that followed was a general engine fix, found by a module and not written for it.

Two real programs

covid.observer, a sizeable Raku site generator, came first. To compile it, the engine needed heredocs, quote-aware regex lexing, literal multi parameters, and the ability to tell a hash from a block. To run it against a live MySQL database, it needed real module loading, feed operators, hyper method calls, and enough of the object model to hold a dozen modules at once. It now runs end to end and writes its HTML.

The generator behind the Raku course went further. It reads its table of contents through YAMLish, an indentation-sensitive YAML grammar that uses nearly every advanced regex feature at once. The regex chapter covers what that took. The generator also highlights code, and rakupp --highlight replaced the Python highlighter it used. It emits the same CSS classes, knows a method named role from the keyword because it parses rather than lexes, and starts in about 13 ms against about 110.

Two corpora

13 JulThe course, as a test suite. Every fenced code block in the course, plus its exercises, was run under both engines with stdin closed and a timeout: 3,068 comparisons. After discarding blocks that do not run under Rakudo either, 148 real divergences were left. Two rounds of fixes (containers and binding, list typing, gists, numeric coercion, quoting adverbs, regex corners) brought them to 14.
14 JulThe Weekly Challenge. 10,428 solutions, by many authors in many styles. About 6,800 are comparable: the rest need arguments, modules or input that Rakudo itself cannot run without. The first pass found 2,663 byte-identical to Rakudo. Fifteen fix batches later there were 4,056, 60% of the comparable set. The last finding was about leverage: six prolific authors account for about half the remaining mismatches, because each reuses one template.

The documentation, example by example

Every runnable example in the official documentation is run on both engines, and each one gets a verdict. Either all three agree (the documentation, Rakudo and Raku++), or one of them is the odd one out. The count where all three agree started at 596 on 25 July and reached 835 the next day, after sixteen gated batches. It is 1,006 now.

Documentation examples. Blue: identical on both engines. Orange: examples where Raku++ is the one that differs.

A matrix of 833 expressions over 121 operators is compared on both engines too. Divergences went from 72 to 30 in v1.5.0, then 24, 21, and 14 in v5.0.0.

Modules, by their own test suites

A battery of 59 distributions went from 11 passing their own suites (v1.5.2) to 18 (v1.7.0), 32 (v1.8.0) and 50 (v2.0.0). One of them, URI 0.3.8, went from 88 of 222 tests to 222 of 222 in a day, through twenty general interpreter fixes and not one line of URI. The battery has held at 48–50 of 59 since. The distributions it still misses need something the machine does not have, or a behaviour the engine does not yet share.

The whole ecosystem

From late August the whole ecosystem was swept: every distribution in the index, installed and tested under Raku++. The count of those passing their own suites rose from 624 to 1,019.

Ecosystem sweep. Distributions passing their own test suites, out of about 2,530 in the index. The last point is a fresh sweep of 2,547.

A pass rate against the whole catalogue would be misleading, so every distribution Raku++ did not pass was re-run under Rakudo on the same machine, through the same harness. Rakudo passes 794 of them, which puts the ceiling here at 1,791 of 2,529. 738 cannot pass under either engine on this machine: they need libgsl, fontconfig, /sbin/ldconfig or a network. That ceiling is the figure the sweep should be read against.

What the fronts taught

Each source has a blind spot the others cover. Roast misses what is not in the passing set. A real program misses whatever it happens not to use. The corpora find the idioms nobody isolated. The documentation finds behaviour that is plausible but not what the language defines. Every batch from these fronts still had to pass the Roast gate before it counted.