torch:padding-mask

(torch:padding-mask tokens &key pad-id)

Returns the padding mask of a (batch length) token matrix: 1.0 at every position holding pad-id (0 by default) and 0.0 elsewhere, with a query axis inserted -- (batch 1 length) -- so it broadcasts over an attention score's (batch query-length key-length).

The result is a raw linalg array, not a tensor: a mask is a constant, and torch:masked-fill takes it as one. Combine it with torch:subsequent-mask using linalg:add or linalg:maximum -- every non-zero counts as masked.