About Anaconda Help Download Anaconda

r / packages / r-tokenizers.bpe

Unsupervised text tokenizer focused on computational efficiency. Wraps the 'YouTokenToMe' library <https://github.com/VKCOM/YouTokenToMe> which is an implementation of fast Byte Pair Encoding (BPE) <https://aclanthology.org/P16-1162/>.


© 2025 Anaconda, Inc. All Rights Reserved. (v4.0.7) Legal | Privacy Policy