Model Family:ByteLevel BPE
Vocabulary Size: 6,400 tokens
Special Tokens:
<|im_start|> (BOS),
<|im_end|> (EOS),
<|endoftext|> (Pad/Unknown)
Context Length: 32,768 tokens
ByteLevel Encoding: This tokenizer uses byte-level Byte-Pair Encoding (BPE).
It first converts text to bytes, then applies learned merge rules.
Spaces appear as Ġ and newlines as Ċ in raw token output.
💡 Try long texts, code snippets, or non-English languages to see how tokenization varies!