Manage | Anaconda.org

r / packages / r-tokenizers.bpe

0

Unsupervised text tokenizer focused on computational efficiency. Wraps the 'YouTokenToMe' library <https://github.com/VKCOM/YouTokenToMe> which is an implementation of fast Byte Pair Encoding (BPE) <https://aclanthology.org/P16-1162/>.

Filters

Type	Size	Name	Uploaded	Downloads	Labels
conda	1.0 MB	\| linux-64/r-tokenizers.bpe-0.1.3-r43h884c59f_0.tar.bz2	2 years and 2 months ago	42	main
conda	1.0 MB	\| linux-64/r-tokenizers.bpe-0.1.0-r42h884c59f_0.tar.bz2	3 years and 9 months ago	72	main
conda	1.0 MB	\| win-64/r-tokenizers.bpe-0.1.0-r36h796a38f_0.tar.bz2	6 years and 23 days ago	109	main
conda	1.0 MB	\| osx-64/r-tokenizers.bpe-0.1.0-r36h466af19_0.tar.bz2	6 years and 23 days ago	33	main
conda	1.0 MB	\| linux-64/r-tokenizers.bpe-0.1.0-r36h29659fb_0.tar.bz2	6 years and 23 days ago	85	main