← Contents

Chapter 12

Containers, Binding, and Copy Semantics

A Value copy shares its payload. Raku sometimes wants that and sometimes wants the opposite, and the rules for which are the subtlest part of the value model. This chapter is about where the sharing is broken on purpose, and how the things Raku calls containers are modelled without a container object.

Scopes are hash maps with a parent pointer

// src/Interpreter.h
struct Env {
    std::unordered_map<std::string, Value> vars;   // "$x", "@a", "%h", "&sub"
    std::shared_ptr<Env> parent;
    bool routineFrame = false;   // $/ scopes here
    bool loopFrame = false;      // a loop's `state` frame
    std::unique_ptr<EnvExtras> ex;

    Value* find(const std::string& name) {
        auto it = vars.find(name);
        if (it != vars.end()) return &it->second;
        return parent ? parent->find(name) : nullptr;
    }
    Value& define(const std::string& name, Value v) {
        return vars[name] = std::move(v);
    }
};

The full sigilled name is the key. Subs live in the same map under a &-prefixed key, so &foo and $foo occupy different namespaces without a second table. Lexical scoping is the parent walk in find; there is no separate symbol table anywhere in the interpreter.

Reading a variable returns a copy of the slot. Writing goes through lvalue(Expr*), which hands back a Value* pointing straight into the owning Env's map:

// read  $x
if (Value* p = tctx_.cur->find(ve->name)) return *p;      // a COPY

// my $x — make the slot here, hand back its address
tctx_.cur->define(ve->name, std::move(init));
return &tctx_.cur->vars[ve->name];

// $x = 5
*lv = rhs;

The three declarators differ only in which Env holds the slot:

DeclaratorStorage
mythe current lexical Env
ourthe package env, ultimately the global one; also republished under a qualified name when the package block closes
statea per-Callable stateEnv, created exactly once and shared across calls

state is the only one that needs care under concurrency, and it gets a std::once_flag on the Callable so the persistent environment is created exactly once even if two threads call the routine simultaneously.

EnvExtras: the eight rarely-used maps

An Env is constructed for every routine call and every block. It also needs somewhere to keep is rw write-through links, temp/let restorations, is default values, and the set of names declared is dynamic — eight containers which are empty in the overwhelming majority of scopes.

Constructing and destroying eight empty maps per block is pure overhead for the ordinary case, so they live behind one lazily-allocated pointer:

// src/Interpreter.h
EnvExtras& x() { if (!ex) ex = std::make_unique<EnvExtras>(); return *ex; }
const EnvExtras& xr() const {
    static const EnvExtras kEmpty;
    return ex ? *ex : kEmpty;
}

Writers call x() and materialise it; readers call xr() and get a shared empty instance, so a lookup never allocates. This is a small pattern but a recurring one: make the common case cost nothing, and pay only where the feature is actually used.

Where the sharing is broken

Copying a Value bumps refcounts on the shared_ptr members, so a raw copy of an Array shares its backing ValueList. Raku wants:

my @a = 1, 2, 3;
my @b = @a;      # a COPY
@b[0] = 99;
say @a;          # (1 2 3) — unchanged

So the interpreter deliberately breaks the sharing at = assignment, and only for the @ and % sigils:

// src/Interpreter.cpp — coerceArray
if (v.t == VT::Array) {
    if (v.itemized) { … }             // an itemized array is ONE element
    if (v.ext) return v;              // a lazy seq stays lazy
    Value r = Value::array(*v.arr);   // *v.arr copies the element list
    r.isList = false; return r;
}

*v.arr dereferences the pointer and copies the underlying ValueList, so @b gets its own storage. That copy is the most visible O(n) event in the language, and it is the reason ValueList is a container this project owns rather than a std::vector — see The list container below. Hashes copy the same way. Scalar assignment is a plain struct overwrite, so my $x = @a stores an Array value that does still share @a's buffer — because an item container is a reference to one thing.

Two consequences to keep straight:

The compiled backend has its own mirror of this rule, rtArrayVal, with the same fresh-buffer semantics, so interpreted and compiled code agree.

Binding: := and the Proxy

= copies a value into an existing container. := rebinds the container itself: after $y := $x, the two names are the same container and a write to either is seen by both.

But Env stores Values by value in a map, so two map slots cannot literally be the same storage. The alias is faked with a Proxy:

// src/Interpreter.cpp — $y := $x, scalar case
Value proxy = Value::makeHash(); proxy.hashKind = "Proxy";
(*proxy.hash)["FETCH"] = fetch;    // a builtin Code closing over the owning Env
(*proxy.hash)["STORE"] = store;
*lvalue(target) = proxy;

$y's slot holds a Hash tagged Proxy whose two closures read and write $x's slot in the environment that owns it. Reading a variable notices the tag and calls FETCH instead of returning the hash:

// src/Interpreter.cpp — reading a variable
Value* p = tctx_.cur->find(ve->name);
if (p) {
    if (p->t == VT::Hash && p->hashKind == "Proxy" && p->hash) {
        auto it = p->hash->find("FETCH");
        if (it != p->hash->end())
            return callCallable(it->second, { *p });
    }
    return *p;
}

Binding chains dereference one extra level so $z := $y := $x works.

This costs a call on every read of a bound variable, which is why the check is placed inside the slow half of the variable-read path and why the fast paths in Chapter 19 decline any value with a non-empty hashKind.

Array binding is much cheaper, because sharing a buffer is exactly what a shared_ptr copy already does:

// src/Interpreter.cpp — @a := @b
if (a->op == ":=" && rhs.t == VT::Array) {
    Value b = rhs; b.itemized = false; *lv = b;
}

The cleared flag is itemized, not isList, and the difference is two bugs rather than a detail. Clearing isList would make a bound List become an Array, which is wrong — my @a := @b where @b is a List must stay a List, and URI::Path is where that surfaced. What has to be cleared is the itemization: $(1,2,3) is one scalar item that does not flatten, and binding it to an @ variable that then flattens it is how DBDish::Pg overflowed its stack.

Value b = rhs copies the struct and shares rhs.arr. This is the deliberate opposite of @a = @b above. constant reuses the binding path — a constant is := in disguise.

The user-visible Proxy type — Proxy.new(:FETCH{…}, :STORE{…}) — is the same mechanism exposed, so a program can build one explicitly.

The scalar-container metaphor

Raku's $ variable is really a Scalar container holding a value, with introspectable machinery: .VAR, is default, type constraints. Raku++ models that without a separate container object, using flags on the Value plus per-Env side tables.

.VAR builds a Hash tagged "Scalar" reporting the variable's name, value and default.

is default(v) and typed defaults live in EnvExtras::varDefault, a per-scope map. my Int $x stores (Int) as both the initial value and the reset default. Assigning Nil walks the chain to find it:

// src/Interpreter.cpp — $x = Nil restores the container's default
for (Env* en = tctx_.cur.get(); en; en = en->parent.get()) {
    auto di = en->varDefault.find(nm);
    if (di != en->varDefault.end()) { dv = di->second; break; }
    if (en->vars.count(nm)) break;   // owner scope, no declared default
}
*lv = dv;                             // else Any

Type constraints on a scalar are enforced at assignment for the core nominal types, throwing X::TypeCheck::Assignment on a mismatch.

readonly is a flag on the Value itself, set when binding a plain $ parameter. Mutating operations such as s/// check it and die.

temp and let push restoration closures onto the scope's extras. tempRestores run whenever the scope leaves; letRestores run only on an unsuccessful exit, which is the distinction the language draws.

Sigils are mostly a parse-time fact

The runtime dispatches on the VT tag, not the sigil. The sigil — the first character of the name — is consulted exactly twice: to pick the empty container shape at declaration (@ gives an empty Array, % an empty Hash, $ an Any), and to choose the assignment coercion (coerceArray, coerceHash, or a scalar overwrite).

After that, behaviour follows the tag and two flags:

That pair carries a surprising amount of Raku's list semantics, and most of the one-argument-rule subtleties in the built-ins reduce to testing them.

The list container

ValueList is the type behind every Array — Value::arr is a shared_ptr<ValueList> — and it is also the argument list of every call. Until 2026-09 it was a typedef for std::vector<Value>. It is now RVec<Value>, a container this project owns (src/ValueVec.h), implementing the std::vector subset the tree uses with std::vector's semantics: raw-pointer iterators, growth invalidating everything, push_back(v[0]) still legal.

It differs in two places, both of them measured.

It grows in one pass. std::vector reallocating means filling a second buffer element by element and then walking the old buffer again to destroy each source. RVec constructs the new element and destroys the old one in a single walk. Over a hundred megabytes of Value, one pass instead of two is the larger half of this change: arrayops retires 30% fewer instructions, listbuild 26%, sortnums 19%.

Where the standard library permits it, that pass is a memcpy. A Value is trivially relocatable — moving its bytes to a new address and not destroying the source is equivalent to move-constructing and then destroying, which is exactly what a reallocation does. Nothing in it points at itself: i/n are scalars, IStr is an interned pointer, the payload and cold-block slots are shared_ptr pairs.

Nothing except one member. CowStr's inline std::string is relocatable on libc++ and MSVC, which compute a short string's data address from this, and is not on libstdc++, which stores _M_p aimed at its own inline buffer — a bitwise copy there reads the original object's storage and dangles the moment it is reused. bitwiseRelocOk() asks the library at run time rather than trusting a macro, relocating a short string and checking where the copy's characters come from, and a library that fails the probe keeps correct behaviour and loses only the speedup.

That probe is worth a warning to anyone editing Value. Its first version asked whether a short string's data() lay inside the string object — which is true everywhere, because that is what the small-buffer optimisation is — so it answered "not relocatable" on every platform, the memcpy path never ran for a whole day of measurement, and nothing failed, because the fallback is correct. tools/reloc-probe.cpp prints which path is live, and exists precisely because a correctness-preserving fallback can hide a dead fast path indefinitely.

Small blocks come off a free list

A ValueList is not only a list. It is the argument list of every interpreted call, and for a one- or two-element list the heap block is the cost: 32 ns against a call that takes about 265. RVec keeps a thread-local free list of blocks for capacities 1 through 4, so allocation is a pop and release is a push. The one-argument shape drops to 9.5 ns; fib retires 8.8% fewer instructions, a two-argument call loop 13%.

The lists are kept per exact capacity, not per rounded-up size class, and that is the whole design decision. One four-element block for every small request is simpler and marginally faster on call-shaped work — but a ValueList is also every Array's payload, and a program holding a million one-element arrays then holds a million four-element blocks: 560–663 MB against 292–296, paying 22% of its cycles in the extra memory traffic. The two users of the type want opposite things from a block-sizing policy, and per-capacity lists are what give both of them what they want.

This is Perl 5's arena-and-free-list discipline (sv.c) applied to the one allocation two separate investigations had already named as the thing to remove — see Chapter 41.

Shaped arrays and typed containers

// src/Value.h
std::shared_ptr<std::vector<long long>> shape;  // my @a[2;3]
std::string ofType;                              // Array[Int], my Int @a

A shaped array is a fixed row-major structure with its dimensions recorded; an unshaped one leaves the pointer null. makeShapedContainer builds one, optionally filling it from a flat list.

ofType carries the element type for a typed container, comma-joined when there is more than one parameter (Hash[Int,Str] becomes "Int,Str"). It also drives native-width masking for my uint8 @a, through the same natWidthOfType table as scalars.

is rw, and writing back to the caller

Because arguments are passed as Value copies, a mutated is rw parameter has to be written back into the caller's variable. The call site passes the argument expressions alongside the values, and the binder records the link:

// src/Interpreter.h — EnvExtras
std::map<std::string, std::pair<Expr*, std::shared_ptr<Env>>> rwLinks;
std::map<std::string, Value> rwSynced;
std::map<std::string, Value*> rwDirect;
std::set<std::string> rwDead;

There are two mechanisms because there are two situations. rwLinks holds the caller's argument expression and environment, so an assignment to the parameter can re-resolve the lvalue and push the value through immediately — the caller sees the change mid-call, which is what Raku specifies. rwSynced records what was last pushed, so the copy-out backstop at return can skip parameters that are already up to date; without it a late copy-out would re-apply stale values over the callee's own later edits.

rwDirect is the simpler case: a hyper-operator element call already has the caller's slot as a pointer, with no expression to re-evaluate. rwDead marks a raw or rw parameter bound to a literal, so assigning it dies with X::Assignment::RO rather than silently writing into a temporary.

This works for direct calls, where the caller supplied the argument expressions. Multi-dispatched and indirect calls can lose that fidelity — a known limitation, documented rather than hidden.

Honest limitations