GPU Acceleration (--gpu)

--gpu routes linalg's matrix product, its element-wise transcendentals and its broadcast / axis-fold / axes-transpose shapes to a GPU: an NVIDIA device driven straight through the CUDA driver, or a Mac's own through Metal. It is one of three orthogonal acceleration flags: --simd lowers the vectorizable vec: and linalg: kernels to CPU vector instructions, --blas replaces the matrix product with a tuned library call, and --gpu moves the work off the CPU entirely. Any combination of the three, or none -- How the three flags compose covers what happens when more than one is on.

--blas puts the matrix product on the fastest thing the CPU has. --gpu puts it on a different machine altogether.

rontolisp prog.lisp --gpu                 # interpreter
rontolisp prog.lisp -o Prog.class --gpu   # JVM class output
rontolisp prog.lisp --simd --blas --gpu   # all three, chained; the device is asked first

A GPU is recommended, never required, exactly as a tuned BLAS is. Nothing is bundled and nothing is downloaded, and there is no CUDA toolkit to install: libcuda.so.1, which ships with the NVIDIA driver, is the entire runtime requirement, and the kernels travel inside rontolisp as a text that the driver compiles for whatever card it finds. On a Mac there is nothing to install at all -- the frameworks and the shader compiler are part of macOS. A machine with no device, no driver, or a card older than Turing (compute capability 7.5) runs the same programs to the same output, only slower, and the interpreter says so on standard error rather than failing.

Everything in the next section is written for an NVIDIA card; On Apple Silicon is where the same flag differs, and the differences are large enough to plan around.

What is accelerated, and what declines

The matrix product, in both of its shapes. linalg:dot over two rank-2 arrays -- and therefore linalg:matmul at rank 2 and linalg:solve, which are written over it. And the stacked product behind linalg:matmul at rank 3 or more, which is torch.bmm: every attention layer, and every torch:linear over a (B T C) activation. A stack costs one round trip and one launch however many matrices are in it, because the device carries the batch on an axis of its own; an operand that broadcasts over the batch -- the rank-2 weight matrix under a rank-3 activation -- is copied to the device once rather than once per matrix.

And the twelve element-wise transcendentals: exp, log, tanh, sin, cos, tan, asin, acos, atan, sinh, cosh and erf -- so torch:gelu, torch:softmax and torch:log-softmax, which are written over them, reach the device too. These are the members with the highest ratio in the whole flag, not the matrix product: linalg:erf over 1.5 M double-floats is 103 ms on a SIMD CPU and 0.9 ms on the device.

And ten more members, each at ONE call shape. add, sub, mul, div, maximum and minimum when their two operands have DIFFERENT shapes and numpy broadcasts them -- (4 256 256) against (4 256 1), an array against its own per-row reduction, which is what torch:softmax and torch:layer-norm are built from; sum, amax and amin in their :axis form; and transpose with an axes list. mean, var, std, linalg:softmax and linalg:log-softmax reach the device through those, exactly as they reach the lane kernels on the CPU. What these shapes have in common is not their arithmetic: it is that the CPU walks them one element at a time with an index odometer rather than in vector lanes, so the CPU cost they have to beat is five to eight times the cost of the same operation on two equally shaped arrays. Measured on the JVM class output at a transformer's own shapes, single-float: a broadcast sub over 393216 elements is 660 us on the CPU and 118 on the device, sum :axis is 297 against 70, an axes transpose 335 against 75, and a whole linalg:softmax -- five of these members chained, five round trips -- 1915 against 402.

And the seeded generator's fill. linalg:rand, linalg:randn and linalg:uniform -- so every weight initialization and every dropout mask -- fill their array on the device from 8192 elements up, and this is the one member whose device result is byte-for-byte the CPU's: each thread jumps to the generator state its element starts from by the closed form a^k s mod m (exact integer arithmetic) and then draws exactly as the sequential walk does, so linalg:seed keeps its promise across the flag. Nothing is copied up -- only the draws come back -- which is why the threshold is the lowest of the set. A dropout mask over 393216 single-floats is about 4 ms on the CPU and about 50 us on the device, and a standard-normal fill (twelve draws per element) over a million elements is 95 ms against 2.1.

And one member outside linalg: vec:matvec, the matrix-by-vector product -- the GEMV a decode loop is made of (examples/llama2 runs thirteen of them per layer per token and little else). A GEMV is memory-bound: its whole cost is one pass over the matrix, so a trip that has to carry the matrix to the device loses to the CPU until about half a million elements. The member is therefore accepted only over a matrix that is already on the device. The first time a matrix is offered the call declines and remembers it; the second time -- the matrix not having been written in between -- it is uploaded and computed; every later call finds it there and pays only for the vector up, the launch and the result down, about 10 microseconds. A matrix the program writes between calls is never uploaded and never pays for a trip it would lose. Above 131072 matrix elements (rows * cols) a resident 384x384 single-float GEMV is 10.7 microseconds against 23 on the CPU; llama2's 288x288 projections stay on the CPU (12.5 against about 10, a tie) while its 768x288 feed-forward matrices and its 32000x288 classifier head move (30 against 12, and 1467 against 169). The kernel accumulates in double at both widths, as the portable vec.lisp definition does and the --simd lane kernel does not, so at single float it lands on the portable definition's own bits in practice (Reach and precision). Only vec:matvec itself: vec:matvec-into and the rest of vec: stay where they were.

The same names at an EQUAL shape are refused as a round trip, and refused by measurement -- and since 2026-08-23 they are members over an operand that is ALREADY on the device. sqrt, abs, negative and sign, add, sub, mul and div with two equal-shaped operands or with a number on either side, the comparison masks greater, greater-equal, less, less-equal and equal, where (and through it torch:masked-fill), the fused Adam update behind torch:adam / torch:adamw, and the copies -- reshape (so expand-dims, squeeze, flatten), the plain matrix transpose, slice, concatenate (torch:cat) and gradient clipping's in-place scale -- decline whenever their operands have to be carried to the device: there the CPU runs a vector lane loop or a memcpy, so its cost is already just the cost of walking the array, and a device has to walk it twice, over a link slower than memory, before it can start. Measured over 1.5 M elements, linalg:sqrt is 700 us on the CPU against 502 on the device (and 500 against 245 at single width), while linalg:add is 900 us against 780 -- and at single width the CPU wins, 350 us against 382. But the result of one device member is not carried anywhere: since 2026-08-23 it stays on the device until something on the host reads it, and a member whose operand is already there is a launch with no copy at all -- so over such an operand every one of those names runs on the device, at any size, and its result stays too. That is what lets softmax, layer-norm, attention and the optimizer step run as a chain that moves nothing over the link. Every one of these computes in double and narrows on the store, the CPU kernel's own rule, so all of them are bit-identical to what the CPU would have computed (Reach and precision).

And, over a resident operand, three index-driven copies and gradient clipping's sum. linalg:take-rows (the embedding lookup behind torch:index-select), linalg:gather (the per-row pick a class-index torch:cross-entropy-loss finds its target logit with) and take-rows' in-place adjoint linalg::%la-scatter-rows are copies chosen by an index vector rather than by a stride, so, exactly like the copies above, they cannot pay for a round trip and, over an operand that is already on the device, they cost one launch. All three are bit-identical to the CPU kernels. Two are pure gathers with nothing to reorder; the scatter-add is not, because a repeated index -- which is what a token embedding's gradient is made of -- makes the order it accumulates in part of its value, and it keeps that order by giving each DESTINATION cell one thread and grouping the indices by destination beforehand rather than by reaching for atomics. Alongside them, torch:clip-grad-norm's sum of squares over every gradient of the model, which was the largest thing a training step still read back on the host. That one is not bit-identical, and it is the only member of this flag that is not -- see Reach and precision.

Everything else declines and runs exactly what it ran before -- the tuned library when --blas is on too, the lane kernel when --simd is, the portable linalg.lisp definition otherwise. That includes the two linalg matrix-by-vector shapes --blas does take (they are memory-bound, so a trip that carries the matrix cannot pay for itself -- which is why vec:matvec above is taken only over a resident one), a rank-1 operand on either side of a stacked product, a batch shape whose slabs no single stride can reach (a broadcast axis sitting under a non-broadcast one), general boxed arrays, mixed widths, a scalar operand, and a shape mismatch, which signals the same error as ever.

It also includes everything small, and there are two thresholds because there are two kinds of work. A round trip to a device costs about 15 microseconds however little data rides on it, so a product below roughly 51x51x51 (n * m * p under 131072) declines and stays on the CPU; for a stack the same threshold applies to the TOTAL work, batch * n * m * p, because the round trip is paid once for the whole stack rather than once per matrix. An element-wise call is measured in elements instead -- one library call each -- and declines below 16384 of them; a broadcast or an axes transpose declines below 32768 result elements, an axis fold below 131072 input elements or 256 output slices -- 32 of them when the operand is already on the device, where the alternative is not a free CPU walk but a copy home (a fold with one output slice is a single-threaded loop on a device, and loses to any CPU), a generator fill below 8192 elements, and a vec:matvec below 131072 matrix elements -- or on the first sight of any matrix. Every threshold is one more decline rather than a mechanism of its own, which is why every example in this repository, all of which run shapes far below them, prints byte-identical output with the flag and without it.

On Apple Silicon

The flag is the same flag and the programs are the same programs. What differs is which calls the device accepts.

It is single-float only. Metal's shading language has no double at all, so a double-float array -- which is what linalg builds by default -- always stays on the CPU. torch: builds single-float tensors by default and needs nothing; a linalg program has to ask, with :element-type 'single-float or the #f reader syntax. Without that the flag is inert on a Mac, and it says nothing about it, because a decline is an ordinary outcome rather than an error.

Everything small stays on the CPU for longer. A round trip costs about 77 microseconds here against about 16 on an NVIDIA card, because on Metal that cost is paid once per command buffer rather than per launch. The thresholds move up with it: a product is offered from about 166x166x166 (n * m * p at 4194304) rather than 51 cubed, an element-wise call from 131072 elements rather than 16384, and a broadcast or an axes transpose from 262144 result elements rather than 32768.

The axis folds are not round-trip members here. sum, amax and amin in their :axis form stay on the CPU at every size unless their operand is already on the device, for two independent reasons: the portable definition accumulates them in double, which no single-float kernel can reproduce bit for bit as a plain float sum, and the half that would not have needed to -- amax and amin, which only compare -- measures a tie against the CPU rather than a win. mean, var, std, linalg:softmax and linalg:log-softmax still reach the device through their other links.

Everything else is the same member set: both product shapes, the twelve transcendentals, and the broadcast binary ops and the axes transpose. A rank-2 product above about 512x512x512 is handed to MetalPerformanceShaders, which is in the OS and is one and a half to four times faster than a hand-written kernel at those sizes; below that, and for every stacked product, the kernel rontolisp carries runs it. The two agree bit for bit, so which one ran is not observable in the results.

vec:matvec is a member here too, from a higher threshold, and it lands on the same bits without a double. The rule is the one above -- a matrix is taken only once it has been offered twice without being written, because carrying it to the device is a copy of the very bytes the CPU would have streamed -- and that matrix is the only array this backend keeps on the device: every other operand is copied in per call, which on unified memory is a plain memcpy and measured cheaper than keeping it. But the floor is per command buffer, so a resident single-float GEMV costs about 80 microseconds however small it is, and the member is offered from 2097152 matrix elements (rows * cols): a resident 1536x1536 GEMV is 94 microseconds against 267 on the CPU, 2048x2048 is 105 against 500, and llama2's 32000x288 classifier head is 185 against 800 -- while 1024x1024 is a tie (90 against 100) and stays on the CPU. One caveat a decode loop meets: this GPU lowers its clocks once it has been idle for more than about a millisecond, and the first command buffer after such a gap costs roughly half a millisecond more, so a GEMV called once every few milliseconds with CPU work in between wins far less than those figures say -- examples/llama2 decodes stories15M at the same speed with the flag and without it on an M4 Max (about 370 tokens per second either way, the story unchanged), because its one matrix above the threshold, the classifier head, is called once per 2.7-millisecond token. Metal's shading language has no double to accumulate in; the kernel keeps its running sum as a compensated pair of single floats instead, which carries about 48 bits and lands on the portable definition's bits on every one of 1024 measured rows of 768, exactly as the double accumulator does on an NVIDIA card (Reach and precision). Single float only, like everything else here.

Results come home after every call here, and that is a measurement too. The rule two sections up -- a device result stays on the device until something on the host reads it, and the members whose operands are already there run as launches with no copy -- is built for Apple Silicon as well (single-float only, and bit-identical to the CPU even where the CPU computes in double, because the shader carries that arithmetic in software), but the interpreter and the compiled class output do not switch it on here: measured on train-gpt-soseki on an M4 Max it is a tie at the notebook's shapes (0.102 against 0.104 seconds a step) and a loss at the book's (10 to 19 seconds a step against a steady 8.9), because every Metal call waits for its command buffer where a CUDA launch overlaps the host, because on unified memory the copy it saves is a memcpy, and because a result kept on the device is a second copy of an array that already lives in the Java heap. So on a Mac each device result is in its array when the call returns, as before.

What it is worth, single-float, at a transformer's own shapes on an Apple M4 Max -- microseconds per call on the JVM class output, best of three timed rounds:

The last two rows are declines: an axis fold is never offered here, and that transpose is 196608 elements, just under the threshold. A declined call costs a little more than it does without the flag -- a few microseconds for the size check, and for a shape between the two thresholds the work of describing it as well, which is the 40 microseconds on the transpose row.

And one n x n linalg:matmul on the interpreter, single-float, microseconds per call, warm:

n=128 is below the threshold and declines. From n=192 the device is four times the CPU, and by n=512 it is forty-five. Your machine, and the shapes your program actually runs, will differ; measure.

A runnable example

examples/ml/gpu-matmul.lisp is one linalg:matmul over a 256x256 matrix and a timing loop, and nothing else -- nine lines, at single-float width, the width a Mac's GPU can take. Another size is the program's own argument -- rontolisp examples/ml/gpu-matmul.lisp --gpu --simd -- 2048, where -- is the point the compiler's options end and the program's begin. Run it three ways:

rontolisp examples/ml/gpu-matmul.lisp               # the portable definition
rontolisp examples/ml/gpu-matmul.lisp --simd        # CPU vector lanes
rontolisp examples/ml/gpu-matmul.lisp --gpu --simd  # the device, lanes below it

Per 256x256 product on an Apple M4 Max: the native interpreter goes 14846 ms -> 2.54 ms with --simd -> 0.24 ms with --gpu --simd, and the JVM class output 1.60 ms -> 0.18 ms. Compiled to wasm-GC, where there is no foreign function API and so no --gpu, it goes 474 ms -> 6.02 ms with --simd alone. The program times itself -- it repeats the product for half a second and divides -- so only the flagless interpreter run is slow to finish: one product, about 15 seconds.

How the three flags compose

Each flag adds one attempt in front of the others, and every attempt that declines hands the same arguments to the next:

--gpu --blas --simd   ->   device -> library gemm -> lane kernel -> portable definition
--gpu --simd          ->   device ->                 lane kernel -> portable definition
--gpu                 ->   device ->                                portable definition

With --parallel the lane-kernel rung is the row-parallel one for the matrix products, and nothing else moves: the device is still asked first, on the calling thread, and only what it declines is split across threads. --blas takes only the rank-2 product, so a stacked one -- or an element-wise call -- has no library rung at all: --gpu --blas --simd chains those device -> lane kernel -> portable definition. The device is asked first because its size threshold is three orders of magnitude above the tuned library's: it turns down everything small before touching the driver at all, and on an NVIDIA card it is ahead of a threaded CPU BLAS at both widths from about n=256 up. So what the device declines lands on the fastest CPU path the invocation asked for, never back on the portable definition. The exception is a narrow band just above the threshold -- roughly n=64 to n=96 -- where --gpu --blas together accept a product the library alone would have finished sooner. Both sides are far under a millisecond there, and asking the library first instead would give away the several-fold win at the sizes the flag exists for.

On Apple Silicon that ordering does not hold below about n=1500. Accelerate's sgemm does not run in vector lanes at all: it reaches 2.1 TFLOP/s on an M4 Max, more than sixteen cores at the --simd column's per-core rate could give, so it is neither lanes nor threads but the CPU cluster's matrix coprocessor -- and it is a plain call on the same memory, with none of the ~80 microsecond command-buffer floor a Metal launch pays. The library therefore wins everything small, and the device pulls ahead only once the n^3 work has outgrown the fixed cost of the trip. One n x n single-float product on the interpreter, ms per call:

So between the Metal threshold (about n=166) and about n=1500, --gpu --blas hands the product to the slower of the two: on a Mac whose matrix products all live in that band, --blas alone is the faster invocation.

Reach and precision

--gpu reaches the interpreter (including the native binary) and the JVM class output. The device is called through the foreign function API, which WASM does not have, so --gpu with a .wasm output is an error rather than a silent no-op; a WASM program has --simd.

A class compiled with --gpu is still standalone -- both bindings travel inside it, whichever kind of machine compiled it, so there is nothing to put on the classpath and java Prog is the whole command on either kind of machine. It does call a restricted method, so run it as java --enable-native-access=ALL-UNNAMED Prog to keep the JVM's warning off standard error. On an NVIDIA card each device call in the native binary currently costs 20 to 50 times more than on the JVM (one n=512 double-float product measured 17.4 ms against 0.74), enough that on that build --gpu --blas is slower than --blas alone at every size measured (the Metal binding does not pay it -- the Apple Silicon figures above are a native binary); --gpu still beats --simd there by more than 2x, and the portable definition by four orders of magnitude. Compiling the program to a class is the way around that cost -- the class the native binary emits is the one java -jar emits, and it runs at the speeds in the second table below.

--gpu is the first flag whose results you should not expect to match the other backends digit for digit. Two separate reasons, and the second is the new one:

• An accelerated product is close to the portable definition rather than equal to it. The device kernel folds each output cell in the portable definition's own order, but it fuses every multiply and add into a single instruction, so each term is rounded once where the portable definition rounds twice. Over inputs that are exact at the operand width (integers, powers of two) that cannot show and the results match exactly; over inexact ones they differ -- measured on an NVIDIA GB10 over operands of magnitude 1, by up to 5e-15 at #d and 3e-6 at #f.

• An accelerated transcendental has no such exempt class of inputs, because the device carries its own implementation of exp, erf and the rest. Two correct libraries disagree in their last digits and neither is wrong. At #f there is a second cause on top: the device evaluates at the operand width, where every CPU kernel here evaluates in double and narrows only on the store. Measured across each member's own range on the same machine, the worst relative difference from the portable definition is 2e-16 to 1e-15 at #d (one to five units in the last place) and 1.1e-7 to 1.7e-7 at #f (one to two). erf is the largest at #d, and that is on rontolisp's side rather than the device's: the portable definition is a series expansion, not a correctly rounded erf. One difference is visible rather than microscopic: an accelerated erf of a negative zero prints -0.0 where the portable definition prints 0.0.

• The matrix-by-vector product vec:matvec sits between the two. Its kernel accumulates in double at both widths and narrows only on the store -- the portable definition's own rule -- so at #f every product of two elements is exact in the accumulator and only the ORDER of a double sum differs from the portable definition, which moves the narrowed result only when that sum lies within about 1e-16 of a rounding boundary: measured, on none of 1024 rows of 768 -- where the --simd lane kernel, which sums in single precision, differs from the portable definition on most rows. So an accepted GEMV under --gpu --simd is CLOSER to the portable definition than the lane kernel it replaces, not further; at #d it is the product's few-ulp story. Neither is promised as byte-identity. On Apple Silicon the kernel has no double; its accumulator is a compensated pair of single floats carrying about 48 bits, and it lands on the portable definition's bits on the same measured rows (1024 of 1024) -- the same contract, not a weaker one.

• torch:clip-grad-norm's norm is the one member of this flag whose result is not the portable definition's arithmetic in the portable definition's ORDER. The portable definition folds the squares one element at a time from left to right, and a whole-array sum has a single output cell, so there is nothing for a device to divide without changing the association. The kernel folds in blocks instead -- each block a strided slice summed into a double and added up in a tree, the block partials then added in block order -- which rounds every term exactly where the portable definition rounds it and only groups them differently. It is the closer of the two to the exact sum, and it is not equal to it: expect a few units in the last place. The number of blocks is a function of the array's length alone, so the answer is the same on every run. The norm is used as the scale max-norm / (norm + 1e-6), so this moves a training run the way an accelerated exp already does.

• The broadcast, axis fold and axes transpose members -- and every member taken over a resident operand since 2026-08-23: the equal-shape and scalar forms of the binary ops and the masks, sqrt / abs / negative / sign, where, the Adam step, the copies and the three index-driven ones -- are the exception: they stay byte-for-byte identical to the portable definition at both widths. Their kernels read every element widened to double, compute in double and narrow only on the store, which is the portable definition's own rule, and there is no library function anywhere in them for two implementations to disagree about. A program whose accelerated calls are only those prints exactly what it prints without the flag.

So a program that sums a million accelerated erf values will print a slightly different number with the flag on -- and a training run will diverge from the CPU one after enough steps, exactly as it would between two GPUs. The portable definition remains the cross-backend oracle, and --gpu is deliberately absent from the cross-backend test suite. If you need identity, do not pass the flag; if you want to check that a program is unchanged in every other respect, run it with CUDA_VISIBLE_DEVICES= set, which makes every device call decline and the output byte-identical. On a Mac, running without the flag is the same check, since there is no environment variable that hides the GPU.

What it is worth

One n x n linalg:matmul on the interpreter, microseconds per call, warm. The machine is an NVIDIA GB10 (Grace Blackwell, 20 CPU cores), and the --blas column is the best this machine has: OpenBLAS across all twenty of them. Your device, driver and library will all differ, so measure.

Read it in two directions. Against the lane kernel the device is 7x at n=128 and 28x at n=512, and 53x at n=512 in single float -- a different order of magnitude, which is the point of the flag. Against a tuned BLAS on twenty cores it is a wash until about n=256 and then 1.6x to 2.3x at double width, 2.5x to 10x at single: double-float is the width this class of device is worst at, so --gpu pays most for single-float data, which is what torch: builds by default. The single-float rows from n=512 up were re-measured on 2026-08-22, when the f32 product gained two register-tiled kernels (a 64x64 and a 128x128 block tile, picked per shape by the device's SM count) that fold each cell in exactly the 16x16 kernel's order and so land on exactly its bits -- 2.7x at n=1024 and 3.5x at n=2048 on the kernel alone; the double-float product is unchanged, because on this device the scarce double units make every tile the same speed.

The same products compiled to a .class and run on the JVM, best of three timed rounds after 400 warm-up calls:

It is the same table, which is the point: once the product is one device call, the backend around it no longer matters. Warm carefully before you compare anything near the threshold -- at n=64 and n=128 the device drops back to its idle clock between calls, and a single cold round there can measure several times these figures.

And the stacked product, which is the shape a transformer is made of: one linalg:matmul of batch n x n slabs, microseconds per call, interpreter, same machine and same warm-up. --blas has no column here because it does not take this member.

The batch is what the device is for: the CPU pays for every matrix in the stack while the round trip is paid once, so the ratio grows with the batch as much as with the matrix -- 1.25x at the threshold, 26x at 12 x 256 double-float and 55x single (that last cell is the register-tiled kernel's, re-measured 2026-08-22; a third of it is the 9 MB of copies around a 51-microsecond kernel).

The element-wise members, on the JVM class output: one linalg: call over 1.5 M elements -- the feed-forward activation of the transformer below -- microseconds per call, best of five timed rounds after 30 warm-up calls.

Nine to twelve times at double width, twenty-three to twenty-nine at single, and 115x for erf -- the member the CPU is slowest at, and the one the exact torch:gelu is written over. The device column is flat because at this size it is the copy and not the arithmetic: every member costs what 12 MB up and 12 MB back costs, which is also why single float is worth twice double float here rather than the fraction the arithmetic would suggest. The refused members are not in the table because the flag does not change what runs for them; the numbers that refused them are in the section above. Measure each width in a process of its own if you repeat this -- on the CPU the second width measured through the same call site is 1.5x to 2x slower than the first, which is a JIT artifact and not a property of the width.

And the members whose CPU twin is an index odometer rather than a lane loop, on the JVM class output, at a transformer's own shapes: microseconds per call, best of three timed rounds after 50 warm-up calls.

Three to six times. The last row is the same operation at an equal shape: the flag refuses it, so both columns are the CPU running the same lane loop -- offering it to the device instead measures 112 us, which is why it is refused. That contrast is the whole selection rule for this group.

End to end, examples/llm-from-scratch/chapter03/train-gpt-soseki.lisp at the notebook's own shapes (*n-embd* 384, *block-size* 256, which the file says is a one-line change) runs a training step about sixteen times faster on the JVM class output with the flag on -- 0.79 s against 0.050 s, from a 5-step and a 40-step run so that setup and sampling fall out of the slope, medians of three interleaved rounds (it was 0.89 against 0.21 when the flag first landed, 0.80 against 0.13 just before device residency and 0.11 the day before this figure; the difference since is that the AdamW update, the generator, and the selects and copies behind torch:masked-fill, torch:index-select and gradient clipping moved onto the acceleration seams, the generator onto the device, since 2026-08-22 the arrays themselves stay on the device between calls, and since 2026-08-23 a RESULT stays there until the host reads it, with the members whose operands are there running as launches with no copy, and since that evening the launch pipeline runs ahead of the device -- the post-launch waits and the strided layout copies that drained the queue are gone). Against --simd --parallel on twenty cores (0.37 s) it is four times; --blas changes nothing on that program, whose every product is the stacked one the library does not take. Quote the ratio rather than the digits: the same program varies by about 15% run to run on this machine. Past the first hundred steps the step settles near 0.016 s -- 50 times --simd's 0.85 s there -- and what is left there is the device kernels themselves, not the copies, the launches or the host: the 200-step run moves 2.3 GB down in 6737 copies against the 44 GB and 37534 it moved the day before, when every result came home after every call and the register-tiled product kernel of 2026-08-22 -- 2-4x on the kernel at this program's feed-forward shapes -- had moved the step by nothing measurable. The example's README has the per-flag table.

The member outside linalg, on the program it was measured for. examples/llama2 decoding stories15M -- 60 MB of single-float weights, 79 GEMVs per token -- on the JVM class output, 256 greedy tokens, three runs each on the same machine: --simd 220 to 226 tokens per second, --gpu --simd 282 to 292, about 1.3x, and the story byte-identical. Less than the weights' bandwidth would suggest, and the reason is what the device does NOT take: the four 288x288 projections per layer are a tie at about 12 microseconds and stay on the CPU, and the attention, RoPE and sampling around the GEMVs are not GEMVs at all. The three feed-forward matrices per layer and the classifier head -- two thirds of the multiply-adds -- are what moved, and the classifier head alone went from 1.5 milliseconds to 0.17.

What the copies cost, and what is left. Since 2026-08-22 the flag keeps a copy of every operand and result on the device between calls -- keyed by the array's identity and held weakly, so an array the program has dropped takes its copy with it -- and an operand a recent call uploaded or produced is not uploaded again. Since 2026-08-23 the host array is no longer the source of truth while a chain runs: a result stays on the device and comes home on the first host read of it -- an aref, a print, a write-sequence, a kernel that runs on the host -- and writing into a packed array in place ((setf (aref ...)), fill, the -into forms, a bulk read-sequence) first brings such a result home and then tells the device to forget its copy; nothing you write changes, and a program that never reads an intermediate result never pays for it. The resident copies are bounded by a quarter of free device memory (an evicted result is downloaded, never dropped), and downloads are staged through a pinned buffer, because on the machine this was measured on a device copy into a freshly allocated array costs a hundred times a warm one. A result that stays on the device also has no host array behind it until something reads it: what the program holds is the array's header, and the elements are allocated on the first host read -- so a training step whose activations are never read on the host costs the heap nothing for them, and at the book's shapes that took the step from 6.3 s to 0.65 (the collector had been most of it). With all of that in, a training step is DEVICE-bound: the profile's first lines are the kernels themselves, and the launches are overlapped -- nothing after a launch waits, and the layout of a strided kernel rides in its parameter block instead of a synchronous copy that would drain the queue (.kb/gpu.md). What can still leave the device idle is an operand the host has to SEND: an upload waits for the queued kernels before it copies, so a program that stages tens of megabytes a step keeps the device busy about three quarters of a step rather than nearly all of it -- the per-head Transformer of the second example is one, and its README says where those uploads come from. On the interpreter the same program still shows no change at all -- 26.1 s per step against 25.5 -- and the reason is not the device: an interpreted step is 32 times a compiled one at the same shapes, so what dominates it is the tree walk around the kernels rather than the kernels. Compile the program before you measure a flag.

On an NVIDIA card, on a long run the heap's PAGES matter, and one JVM flag can be worth 15%. A result that stays on the device holds device memory until the collector notices the program has dropped it, so the flag asks for a collection when its budget fills -- about 50 milliseconds, some 3% of a training run, whichever collector answers, and refusing the request outright (-XX:+DisableExplicitGC) costs several times the run. What is not free is what the answer does to the heap's pages: a compacting collection moves every surviving array to a new address and gives the regrown heap fresh pages, and a device copy to or from a page the GPU has never touched costs about a hundred times a warm one. So for a long run either tell the default collector not to compact on request -- -XX:+ExplicitGCInvokesConcurrent, worth 15% at a 13 M-parameter model's shapes and nothing at all at shapes small enough that the budget is never reached -- or give it a young generation the program FILLS and therefore recycles (-XX:+UseParallelGC -Xmn8g where a step allocates gigabytes, the fastest measured there). What not to do is hand-size a young generation the program does NOT fill: under the parallel collector -Xmn4g at a shape that allocates megabytes a step is 4 GB of pages the device has never touched, and it makes the same program's run half again as long. On Apple Silicon none of this applies, and following it costs. Results do not stay on the device there, so the library never asks for a collection at all -- System.gc() appears zero times in a training run under every configuration measured -- and there is no such thing as a page the device has not touched, because the buffers it reads are the same unified memory the heap is written into. Every flag combination above lands within one and a half per cent of leaving the collector alone at a shape that allocates megabytes a step, and at the shapes that allocate gigabytes -XX:+UseParallelGC -Xmn8g is 13% slower than leaving it alone, most of the difference being its own pause time. So on a Mac: set a heap big enough for the program and nothing else.