tokenizer:token-id

(tokenizer:token-id tk token)

The id of the token string token, or nil when the vocabulary has no such token. The argument is the byte-level spelling, the same one tokenizer:token-string hands back; a vocabulary that repeats a string keeps the lowest id, as the reference implementations do.