← Contents

Chapter 35

use nqp: a Compatibility Subset

Some ecosystem modules — most importantly JSON::Fast, which sits under about 170 other distributions — are written in ordinary Raku but reach for NQP ops in their hot paths: nqp::iseq_i, nqp::substr, nqp::push_s.

NQP is Rakudo's bootstrap language, and its nqp:: operations are the low-level primitives the Raku runtime is built on. Rakudo documents them as internal and unspecified; every implementation provides whatever subset its ecosystem happens to need.

Raku++ provides that subset. This chapter is about how, and — more interestingly — about why it costs nothing for the programs that do not use it.

There is no NQP compiler here

The single most important thing to understand: nqp::foo(...) is already valid Raku syntax. It is a call to a package-qualified routine named nqp::foo. The normal parser reads it with no special help.

There is no NQP grammar, no NQP compiler, and no QAST anywhere in Raku++. What is added is a semantic layer: when a program opts in with use nqp, the parser recognises those already-parsed calls and gives them meaning, compiling them to dedicated AST nodes instead of leaving them as calls to undefined routines.

use nqp;                       # (or `use MONKEY-GUTS`)
my int $x = nqp::add_i(2, 3);  # 5

Without use nqp, nqp::add_i(...) is exactly what it looks like — a call to a routine that does not exist — and running it gives the usual undefined-routine error.

Zero cost when unused

This is a hard design constraint, not an aspiration, and it holds by construction rather than by optimisation.

A regression test asserts the invariant directly: that nqp::add_i still reports "undefined routine" without use nqp.

How the ops compile

Inside a use nqp unit, a recognised nqp::op(...) becomes one of three things at parse time.

Constants fold to literals. nqp::const::CCLASS_NUMERIC is replaced by the integer 8 in the parser; it never exists as a call. All 56 constants are handled this way: the character-class flags, the normalization modes, the BINARY_ size and endian flags, the STAT_ field selectors and the CONTROL_ exception categories.

Control forms become native or lazy nodes, because their arguments must not all evaluate eagerly. nqp::if and nqp::unless compile to Raku++'s own ternary node, so the untaken branch never runs. nqp::while, nqp::until, nqp::repeat_while, nqp::repeat_until, nqp::stmts, nqp::ifnull and nqp::handle become NqpOp nodes whose evaluator drives its own argument evaluation. The two repeat_ forms differ from their plain counterparts in one way that matters: the body runs once before the test is ever taken.

Leaf ops become eager NqpOp nodes — integer maths, string and list primitives — which evaluate their arguments normally and then do the low-level operation over Raku++'s own Values. An op that takes no arguments may be written without parentheses, nqp::null or nqp::list_i, and is recognised the same way.

An nqp:: op outside the implemented subset is left as an ordinary call and fails loudly at run time rather than silently misbehaving. The subset grows by measured demand, never by guessing.

In code

The parser branch, condensed:

// src/Parser.cpp — the term is already lexed as the qualified name `nqp::add_i`
if (useNqp_ && name.compare(0, 5, "nqp::") == 0) {
    if (name.compare(0, 12, "nqp::const::") == 0) {
        long long cv;
        if (nqpConstValue(name.substr(12), cv))
            return std::make_unique<IntLit>(cv);      // folded here
    }
    if (ExprPtr n = makeNqpOp(name.substr(5), callArgs))
        return n;                                     // an NqpOp, or a Ternary
}

The node is deliberately tiny:

// src/Ast.h
enum class NqpOpc : uint16_t {
    Stmts, While, Until, IseqI, AddI, Substr, /* … */ };
struct NqpOp : Expr {
    NqpOpc op;
    std::vector<ExprPtr> args;
};

and the evaluator handles the lazy forms before touching their arguments:

// src/BuiltinsNqp.cpp — condensed
Value Interpreter::evalNqpOp(NqpOp* n) {
    auto& a = n->args;
    switch (n->op) {                        // lazy forms drive their own args
        case NqpOpc::While:
            while (boolify(eval(a[0].get())))
                for (size_t i = 1; i < a.size(); i++) eval(a[i].get());
            return Value::nil();
        default: break;
    }
    ValueList v;                            // everything else: eval once
    for (auto& e : a) v.push_back(eval(e.get()));
    auto I = [&](size_t i){ return v[i].toInt(); };
    switch (n->op) {
        case NqpOpc::AddI:  return Value::integer(I(0) + I(1));   // int64
        case NqpOpc::IseqI: return Value::integer(I(0) == I(1));
        // … about 165 more leaf ops …
    }
}

The argument buffers

An nqp-heavy program is nqp-heavy in the extreme: a tokenizer written in Raku runs about 1.5 million operations on a 278 KB document. Building a fresh ValueList per operation was a malloc and a free per op, for a list of one to four values.

(Those are now exactly the sizes ValueList keeps on a free list — see Chapter 12 — so the same op costs a pop and a push instead. The buffers below still pay, because they also save the clear and refill, and because the re-entrancy argument that picks the container has nothing to do with allocation. But this section was written when the allocator was the whole cost, and it no longer is.)

// src/Interpreter.h — ExecContext
std::deque<ValueList> nqpArgs;   // one reusable buffer per nesting depth
size_t nqpDepth = 0;

The buffers are kept, so their capacity is kept; the contents are still cleared on the way out, so argument lifetimes are exactly what they were.

It is a deque, not a vector, and the comment says why: an argument's own evaluation can re-enter evalNqpOp, and growing a vector would reallocate under the outer frame's live reference. A deque never moves the elements it already holds.

That is a small illustration of a recurring theme — the reusable-buffer optimisation is easy, and the container choice that makes it correct under re-entrancy is the part that takes thought.

Native compilation

The subset is a first-class part of the transpiler, not just the interpreter. A use nqp program native-compiles like any other; it does not fall back to bundling.

$ rakupp --cpp -e 'use nqp; say nqp::add_i(1, 2)'
…
rtSinkStmt(RT, rtCallB(RT, __bfp0, "say", ValueList{
    ([&]()->Value{ ValueList __na = ValueList{Value::integer(1LL),
                                              Value::integer(2LL)};
                   return rtNqpOp(NqpOpc(10), __na); }())}));

Two emission strategies, mirroring how the ops evaluate:

The two repeat_ forms are the exception, and it is a gap: the generator has no case for them, emits them as leaf ops, and rtNqpOp has no such op. A natively compiled nqp::repeat_while dies with "nqp op not implemented in this build" where the interpreter runs the loop.

The zero-cost guarantee extends here too: a program without use nqp emits zero references to rtNqpOp or any nqp machinery.

What is covered

176 opcodes, reached under 196 spellings in the parser's table — unbox_s, unbox_i, decont_s and hllize are all the one Decont — plus nqp::if and nqp::unless, which have no opcode. They were chosen from the actual inventory of the modules in the ecosystem compatibility battery. The table below is the shape of the coverage rather than the whole list — read NqpOpc in src/Ast.h for that:

GroupOps
Control (lazy)if unless while until repeat_while repeat_until stmts ifnull handle
Integer (int64, no bignum promotion)iseq_i isne_i islt_i isle_i isge_i isgt_i cmp_i add_i sub_i mul_i div_i mod_i not_i bitand_i bitor_i bitxor_i bitshiftl_i bitshiftr_i
Bignum (the _I family)add_I sub_I mul_I div_I mod_I pow_I gcd_I lcm_I neg_I abs_I cmp_I iseq_I isbig_I tostr_I fromstr_I and the bit ops
String (codepoint-indexed)ordat eqat substr chars concat join index rindex indexic flip split x chr iseq_s cmp_s strfromcodes strtocodes findcclass findnotcclass iscclass
Listlist list_i list_s elems atpos atpos_i bindpos push push_i push_s pop pop_s shift shift_i splice
Hashhash bindkey atkey existskey deletekey
Object / attrcreate clone istype can what where eqaddr isconcrete getattr bindattr p6bindattrinvres null isnull decont box_i isnanorinf
Buffersreaduint readint readnum writeuint writeint writenum slice decode setelems
Filesopen readfh closefh stat lstat stat_time opendir nextfiledir closedir filereadable fileislink readlink
Unicode propertiesunipropcode unipvalcode getuniprop_str getuniprop_bool getuniprop_int matchuniprop hasuniprop
Exceptionsnewexception setmessage setextype setpayload getmessage getextype getpayload throw
Othergetcomp sha1 time print say lock unlock hllbool
Constantsconst::CCLASS_* NORMALIZE_* BINARY_* STAT_* CONTROL_*

The _i integer ops use plain int64 semantics, matching NQP's native integers — no overflow to bignum — and string ops index by codepoint rather than by grapheme. Both of those are deliberate: an NQP op is supposed to be the low-level primitive, and a module reaching for one is reaching past Raku's semantics on purpose. The _I family is the other half of the same design: it is the bignum-safe spelling of arithmetic Raku++ already does, so those ops delegate to the ordinary arithmetic rather than re-deriving it.

MoarVM is not self-consistent in this corner, and the ops follow what it does rather than what would be tidy: div_i and div_I floor (-7 div 2 is -4), mod_i truncates (-7 mod 2 is -1), and mod_I floors. The values were read off Rakudo, and t/regression/lizmat-nqp-ops.raku pins them.

Scope and non-goals

Why this shape is the right one

The temptation with a compatibility layer is to implement the source language. That would have meant an NQP grammar, an NQP compiler, and a second execution model to keep in step with the first — for a feature whose entire purpose is to let a handful of modules load.

Recognising that nqp::foo(…) is already valid syntax in the host language turned a compiler project into about two thousand lines of parser branch and evaluator arm. The general principle generalises: before implementing a foreign language, check whether its surface syntax is already legal in yours.