Files | Anaconda.org

r / packages / r-tokenizers.bpe

0

Unsupervised text tokenizer focused on computational efficiency. Wraps the 'YouTokenToMe' library <https://github.com/VKCOM/YouTokenToMe> which is an implementation of fast Byte Pair Encoding (BPE) <https://aclanthology.org/P16-1162/>.

Filters

Type	Size	Name	Uploaded	Downloads	Labels
conda	1.0 MB	\| linux-64/r-tokenizers.bpe-0.1.3-r43h884c59f_0.tar.bz2	1 year and 29 days ago	20	main
conda	1.0 MB	\| linux-64/r-tokenizers.bpe-0.1.0-r42h884c59f_0.tar.bz2	2 years and 7 months ago	52	main
conda	1.0 MB	\| win-64/r-tokenizers.bpe-0.1.0-r36h796a38f_0.tar.bz2	4 years and 11 months ago	94	main
conda	1.0 MB	\| osx-64/r-tokenizers.bpe-0.1.0-r36h466af19_0.tar.bz2	4 years and 11 months ago	18	main
conda	1.0 MB	\| linux-64/r-tokenizers.bpe-0.1.0-r36h29659fb_0.tar.bz2	4 years and 11 months ago	64	main