tokenizer:eos-id
(tokenizer:eos-id tk)
The tokenizer's end-of-sequence id, or nil when it has none. It is what (tokenizer:encode tk text :eos t) appends, and the id a generation loop stops on.
(tokenizer:eos-id tk)
The tokenizer's end-of-sequence id, or nil when it has none. It is what (tokenizer:encode tk text :eos t) appends, and the id a generation loop stops on.