I'm one of the authors of this blog post. Figuring out how to integrate the LLM, the inference and serving engine, and the constrained decoding implementation such that you get the intended output and overall quality definitely requires careful engineering and testing. Please feel free to AMA. :)
To the LLM, the <|...|> tags are single, special tokens. They are rendered as <|...|> tags for readability, but to the model the tags are single tokens.
I do not know why these tokens are visualized as <|...|>, but I speculate that this way they are easily recognizable as special-purpose tokens, and difficult to confuse with other types of existing markup.
I do not know why these tokens are visualized as <|...|>, but I speculate that this way they are easily recognizable as special-purpose tokens, and difficult to confuse with other types of existing markup.
https://github.com/huggingface/transformers.js/blob/main/pac...