About Anaconda Help Download Anaconda
Sign In
Anaconda.cloud

conda-forge / packages / r-tokenizers

Convert natural language text into tokens. Includes tokenizers for shingled n-grams, skip n-grams, words, word stems, sentences, paragraphs, characters, shingled characters, lines, tweets, Penn Treebank, regular expressions, as well as functions for counting characters, words, and sentences, and a function for splitting longer texts into separate documents, each with the same number of words. The tokenizers have a consistent interface, and the package is built on the 'stringi' and 'Rcpp' packages for fast yet correct tokenization in 'UTF-8'.

copied from cf-staging / r-tokenizers

Filters
Type Size Name Uploaded Downloads Labels
conda 621.7 kB | win-64/r-tokenizers-0.3.0-r43h8ae3a7c_2.conda  8 months and 27 days ago 307 main
conda 620.4 kB | win-64/r-tokenizers-0.3.0-r44h8ae3a7c_2.conda  8 months and 27 days ago 323 main
conda 615.9 kB | osx-64/r-tokenizers-0.3.0-r44h25d921d_2.conda  8 months and 27 days ago 262 main
conda 614.8 kB | osx-arm64/r-tokenizers-0.3.0-r44hd76f289_2.conda  8 months and 27 days ago 213 main
conda 615.8 kB | osx-arm64/r-tokenizers-0.3.0-r43hd76f289_2.conda  8 months and 27 days ago 226 main
conda 624.9 kB | linux-aarch64/r-tokenizers-0.3.0-r43h6170684_2.conda  8 months and 27 days ago 147 main
conda 626.3 kB | linux-ppc64le/r-tokenizers-0.3.0-r43h0cea4bf_2.conda  8 months and 27 days ago 132 main
conda 623.9 kB | linux-aarch64/r-tokenizers-0.3.0-r44h6170684_2.conda  8 months and 27 days ago 115 main
conda 616.9 kB | osx-64/r-tokenizers-0.3.0-r43h25d921d_2.conda  8 months and 27 days ago 279 main
conda 625.7 kB | linux-ppc64le/r-tokenizers-0.3.0-r44h0cea4bf_2.conda  8 months and 27 days ago 130 main
conda 624.8 kB | linux-64/r-tokenizers-0.3.0-r44h0d4f4ea_2.conda  8 months and 27 days ago 1335 main
conda 625.3 kB | linux-64/r-tokenizers-0.3.0-r43h0d4f4ea_2.conda  8 months and 27 days ago 1662 main
conda 616.1 kB | osx-arm64/r-tokenizers-0.3.0-r42h65f505e_1.conda  1 year and 2 months ago 208 main
conda 615.5 kB | osx-arm64/r-tokenizers-0.3.0-r43h65f505e_1.conda  1 year and 2 months ago 216 main
conda 616.5 kB | osx-64/r-tokenizers-0.3.0-r43h29979af_1.conda  1 year and 2 months ago 256 main
conda 616.8 kB | osx-64/r-tokenizers-0.3.0-r42h29979af_1.conda  1 year and 2 months ago 277 main
conda 620.6 kB | win-64/r-tokenizers-0.3.0-r41ha856d6a_1.conda  1 year and 9 months ago 557 main
conda 617.1 kB | osx-64/r-tokenizers-0.3.0-r42hac7d2d5_1.conda  1 year and 9 months ago 275 main
conda 617.0 kB | osx-64/r-tokenizers-0.3.0-r43hac7d2d5_1.conda  1 year and 9 months ago 294 main
conda 626.0 kB | linux-ppc64le/r-tokenizers-0.3.0-r42hd301276_1.conda  1 year and 9 months ago 142 main
conda 625.8 kB | linux-ppc64le/r-tokenizers-0.3.0-r43hd301276_1.conda  1 year and 9 months ago 153 main
conda 624.3 kB | linux-aarch64/r-tokenizers-0.3.0-r43ha6e910b_1.conda  1 year and 9 months ago 156 main
conda 625.5 kB | linux-64/r-tokenizers-0.3.0-r43ha503ecb_1.conda  1 year and 9 months ago 2632 main
conda 625.0 kB | linux-64/r-tokenizers-0.3.0-r42ha503ecb_1.conda  1 year and 9 months ago 2433 main
conda 625.1 kB | linux-aarch64/r-tokenizers-0.3.0-r42ha6e910b_1.conda  1 year and 9 months ago 183 main
conda 627.4 kB | linux-ppc64le/r-tokenizers-0.3.0-r41he86bb61_0.conda  2 years and 3 months ago 151 main
conda 627.2 kB | linux-ppc64le/r-tokenizers-0.3.0-r42he86bb61_0.conda  2 years and 3 months ago 141 main
conda 624.6 kB | win-64/r-tokenizers-0.3.0-r41ha856d6a_0.conda  2 years and 3 months ago 709 main
conda 615.9 kB | osx-64/r-tokenizers-0.3.0-r42h49197e3_0.conda  2 years and 3 months ago 479 main
conda 616.1 kB | osx-64/r-tokenizers-0.3.0-r41h49197e3_0.conda  2 years and 3 months ago 481 main
conda 624.9 kB | linux-64/r-tokenizers-0.3.0-r42h38f115c_0.conda  2 years and 3 months ago 2269 main
conda 624.6 kB | linux-aarch64/r-tokenizers-0.3.0-r41hc3d7334_0.conda  2 years and 3 months ago 162 main
conda 624.6 kB | linux-aarch64/r-tokenizers-0.3.0-r42hc3d7334_0.conda  2 years and 3 months ago 145 main
conda 624.4 kB | linux-64/r-tokenizers-0.3.0-r41h38f115c_0.conda  2 years and 3 months ago 2100 main
conda 631.0 kB | linux-ppc64le/r-tokenizers-0.2.3-r42ha35a809_1.conda  2 years and 3 months ago 135 main
conda 620.5 kB | osx-64/r-tokenizers-0.2.3-r41h49197e3_1.conda  2 years and 3 months ago 458 main
conda 658.7 kB | win-64/r-tokenizers-0.2.3-r41ha856d6a_1.tar.bz2  2 years and 6 months ago 565 main
conda 657.6 kB | linux-ppc64le/r-tokenizers-0.2.3-r41ha35a809_1.tar.bz2  2 years and 6 months ago 32 main
conda 647.5 kB | osx-64/r-tokenizers-0.2.3-r42h49197e3_1.tar.bz2  2 years and 6 months ago 71 main
conda 657.0 kB | linux-64/r-tokenizers-0.2.3-r41h7525677_1.tar.bz2  2 years and 6 months ago 2125 main
conda 656.9 kB | linux-64/r-tokenizers-0.2.3-r42h7525677_1.tar.bz2  2 years and 6 months ago 2420 main
conda 657.2 kB | linux-aarch64/r-tokenizers-0.2.3-r42hf13958e_1.tar.bz2  2 years and 6 months ago 59 main
conda 657.5 kB | linux-aarch64/r-tokenizers-0.2.3-r41hf13958e_1.tar.bz2  2 years and 6 months ago 59 main
conda 658.5 kB | win-64/r-tokenizers-0.2.3-r41ha856d6a_0.tar.bz2  2 years and 6 months ago 557 main
conda 663.7 kB | win-64/r-tokenizers-0.2.3-r40ha856d6a_0.tar.bz2  2 years and 6 months ago 554 main
conda 657.0 kB | linux-ppc64le/r-tokenizers-0.2.3-r40ha35a809_0.tar.bz2  2 years and 6 months ago 32 main
conda 657.8 kB | linux-ppc64le/r-tokenizers-0.2.3-r41ha35a809_0.tar.bz2  2 years and 6 months ago 33 main
conda 647.8 kB | osx-64/r-tokenizers-0.2.3-r41h49197e3_0.tar.bz2  2 years and 6 months ago 77 main
conda 647.5 kB | osx-64/r-tokenizers-0.2.3-r40h49197e3_0.tar.bz2  2 years and 6 months ago 70 main
conda 657.5 kB | linux-aarch64/r-tokenizers-0.2.3-r41hf13958e_0.tar.bz2  2 years and 6 months ago 60 main
© 2025 Anaconda, Inc. All Rights Reserved. (v4.0.9) Legal | Privacy Policy