tokenizer:bos-id
(tokenizer:bos-id tk)
The tokenizer's beginning-of-sequence id, or nil when it has none. It is what (tokenizer:encode tk text :bos t) puts in front of the ids; a base model that wants no BOS simply leaves the keyword off.
(tokenizer:bos-id tk)
The tokenizer's beginning-of-sequence id, or nil when it has none. It is what (tokenizer:encode tk text :bos t) puts in front of the ids; a base model that wants no BOS simply leaves the keyword off.