(rontolisp) docs
← tokenizer Package Functions

tokenizer:decode-bytes

(tokenizer:decode-bytes tk ids)

The raw bytes the token ids stand for, as a fill-pointer byte vector. This is the streaming half of tokenizer:decode: a generation loop accumulates the bytes of the tokens it has produced and decodes what is complete, which is the only way a character split across two tokens ever prints correctly. Nothing is stripped or rewritten.