Neural Networks (torch)

The torch package is a PyTorch-style layer over linalg: a tensor that remembers how it was computed, and a torch:backward that walks that history in reverse to fill in gradients. Everything a hand-written backpropagation pass used to do -- tracking which arrays fed which, deriving each operation's adjoint, summing gradients over broadcast axes -- happens automatically, one operation at a time.

The package is implemented once in Lisp source and behaves identically on every backend. Every operation computes through the linalg kernels, so a torch program is accelerated under --simd for free, and the numerical results are the linalg results. How much the flag buys depends on which kernels a model leans on: batched (rank 3 or more) matmul is not in the accelerated set yet, so a transformer -- whose attention layers are almost entirely that call -- gains far less from it than a plain multilayer perceptron does. See Accelerating linalg.

Tensors

torch:tensor builds a leaf tensor from a number, a list, an array or a linalg array. :requires-grad t marks a parameter -- a tensor whose gradient the backward pass should fill in. A tensor prints as #<TENSOR data> (:REQUIRES-GRAD T appended for a parameter) -- the same text on every backend, since the printer shows only the data. Read values back with torch:data (the array), torch:item (the number inside a one-element tensor) and torch:shape:

Operations accept tensors, numbers, raw arrays and lists interchangeably; non-tensors become constants that no gradient flows to:

Recording and the backward pass

An operation whose operand participates in autograd records the operation on a tape. torch:backward on a scalar (one-element) tensor seeds its gradient with 1.0, visits the recorded operations in reverse topological order, and accumulates each input's gradient -- so a tensor used twice (a residual connection, a reused embedding row) collects the sum of both paths. Read the result with torch:grad:

Gradients accumulate across backward calls (+=), which is what a mini-batch loop wants; torch:zero-grad clears a tensor's slot between steps:

Broadcasting and gradients

Elementwise operations broadcast like numpy, and the backward pass reduces each gradient back to its operand's shape by summing over the broadcast axes. A (d) bias added to an (n d) activation therefore gets a (d) gradient -- the sum over the batch axis:

Staying off the tape

torch:no-grad runs its body with recording disabled -- the values are computed, nothing is remembered. This is how a training loop's parameter update (and inference in general) stays off the tape. torch:detach is the per-tensor spelling: a leaf sharing the same data, cut off from its history:

A training loop: fit y = 2x

Gradient descent needs nothing beyond what is above: a forward pass building the loss, torch:backward, and an update inside torch:no-grad. Fitting y = 2x by minimizing the mean squared error (the values are chosen so every quantity is an exact dyadic rational -- the printed result is identical on every backend):

Modules

A module owns parameters, composes, and has a forward pass. torch:module builds one from a kind keyword, a plist of fields and a forward function; torch:forward runs it. The fields plist is the parameter registration -- torch:parameters walks it -- so a layer's forward reads its parameters back with torch:field rather than from a closed-over variable, and a parameter that exists cannot be missing from the walk:

The walk descends into submodules and into lists of them, and deduplicates by identity -- a weight shared by two layers is one parameter, and a list of N blocks needs no ModuleList type. A tensor field without requires-grad is a buffer and is skipped:

torch:train and torch:eval switch the training flag through the same walk, and torch:zero-grad accepts a module -- it clears every parameter's gradient.

The built-in layers

torch:linear, torch:embedding, torch:layer-norm, torch:dropout and torch:sequential are ordinary callers of torch:module. Their parameters are initialized exactly as PyTorch's are, from the seeded linalg:seed generator, so a seeded run reproduces on every backend. Replacing a parameter with torch:set-field pins a layer to given weights:

torch:sequential threads its argument through each element, and an element may be a module or a plain function -- which is why there is no activation-module type, and why a reshape can sit in a chain:

The activations are torch:relu, torch:tanh and torch:gelu, each an ordinary function of a tensor. torch:gelu is the exact x * (1 + erf(x / sqrt(2))) / 2 by default (nn.GELU's own default), over the differentiable torch:erf; :approximate :tanh selects the GPT/BERT formulation instead.

torch:fields answers a module's whole fields plist, which is what makes the tree WALKABLE from outside: nn.Module.apply and nn.Module.named_parameters have no counterpart here because a walk is written over that plist plus torch:module-kind -- what a layer IS, rather than a substring of a dotted parameter name.

Losses

torch:mse-loss and torch:cross-entropy-loss are plain functions returning a scalar tensor. Cross entropy takes raw logits (never softmax outputs -- it is computed from -log-softmax, the numerically stable form) and flattens all but the last axis, so (batch seq vocab) works directly. Its target is either integer class indices, with :ignore-index dropping padding positions from both the sum and the mean's denominator, or a full probability distribution of the logits' own shape -- PyTorch's soft-label form, -sum(target * log-softmax(logits)):

A LIST target is always class indices, so the probability spelling needs a tensor or an array; a one-hot distribution and the matching index give the same loss.

Optimizers

An optimizer owns the update rule and its state. torch:sgd, torch:adam and torch:adamw take a model (or a plain list of parameters), keep their hyper-parameters and buffers in a fields plist exactly as a module does, and apply the rule to every parameter when torch:step runs:

The update writes each parameter's data in place and uses no torch operation, so it records nothing on the tape and needs no torch:no-grad around it -- unlike a hand-written update built from torch:set-data. The state (a momentum buffer, Adam's two moments, the step count its bias correction divides by) lives in the optimizer and never on the parameter, so two optimizers over the same weights keep separate state.

Hyper-parameters are ordinary fields, which is all a learning-rate schedule needs, and torch:zero-grad accepts an optimizer as well as a model:

torch:optimizer is the constructor all three are built on -- a kind keyword, the parameters, a fields plist and a step function -- so a rule this package does not ship is a plain defun over the same record.

torch:adam and torch:adamw are the SAME rule with the decay in a different place: Adam's :weight-decay adds wd * param to the gradient, AdamW's shrinks the parameter directly so the adaptive denominator never rescales it. There is no parameter-GROUP object; two optimizers over disjoint parameter lists are what a group is here, which is how a transformer decays its weight matrices and leaves its biases, LayerNorm gains and embedding tables alone.

torch:clip-grad-norm goes between torch:backward and torch:step: it returns the total L2 norm of every gradient -- as measured, so the loop can log it -- and scales them in place when that exceeds the bound.

Training a network

Everything above composes into the loop PyTorch writes: forward, loss, torch:zero-grad, torch:backward, torch:step.

Writing the update by hand instead needs torch:no-grad around it, because torch:sub on a parameter would record on the tape; torch:set-data then writes the new value into the very tensor the layer's fields point at, so the model keeps using it:

Batching, padding and masks

There is no Dataset/DataLoader hierarchy: a batch is an ordinary list. torch:shuffled-batches cuts a list of examples -- or an integer n, standing for the index list 0..n-1, which is how several parallel arrays get batched at once -- into mini-batches ordered by the seeded generator, so an epoch reproduces on every backend:

torch:pad-sequence turns a batch of variable-length index sequences into one padded rank-2 tensor, batch first, and the two mask constructors build the constants an attention layer fills with -infinity:

Both masks are raw linalg arrays -- a mask carries no gradient -- shaped to broadcast over a (batch query-length key-length) score: (batch 1 length) for the padding mask, (1 n n) for the causal one. They combine with linalg:add, since torch:masked-fill treats every non-zero as masked. The padding value chosen here is also the :ignore-index to pass to torch:cross-entropy-loss, so the padded positions leave the loss alone.

Masked attention scores

torch:masked-fill writes a constant where a mask is non-zero; filling with -infinity before torch:softmax is the masked-attention idiom, and the masked weight comes out as exactly 0.0 -- including through the backward pass:

A worked example

examples/llm-from-scratch/ is chapter 2 of 『作ってわかる大規模言語モデルの仕組み』 rewritten on this package: scaled dot-product and multi-head attention, sinusoidal positional encoding, the encoder/decoder Transformer with its padding and causal masks, and a Japanese-English training loop with greedy decoding. Its README carries the PyTorch-to-torch mapping table.

Packages

torch does not use cl, so programs stay in cl-user and call the qualified names; #'torch:name works (every function is a plain defun). The differentiable operations mirror their linalg counterparts -- the full list is in the function reference, and torch:no-grad on the macros page.