torch:adamw

(torch:adamw params &key lr betas eps weight-decay)

Returns an AdamW optimizer (PyTorch's torch.optim.AdamW) over params, a module or a list of parameter tensors. torch:adam's rule with DECOUPLED weight decay: the parameter shrinks on its own before the Adam step, instead of the decay entering the gradient and being rescaled by the adaptive denominator. Per element, torch:step computes

param <- param - lr * weight-decay * param
m <- beta1 * m + (1 - beta1) * grad
v <- beta2 * v + (1 - beta2) * grad^2
param <- param - lr * (m / (1 - beta1^t)) / (sqrt(v / (1 - beta2^t)) + eps)

:lr defaults to 0.001, :betas to (0.9 0.999), :eps to 1.0e-8 and :weight-decay to 0.01 -- PyTorch's default, against torch:adam's 0.

This is the rule a transformer is trained with, and a parameter that must NOT decay (a bias, a LayerNorm gain, an embedding table) belongs in a SECOND optimizer built with :weight-decay 0.0: two optimizers over disjoint parameter lists are what torch.optim's parameter GROUPS express here.