About Anaconda Help Download Anaconda
Sign In
Anaconda.cloud

conda-forge / packages / r-tokenizers

Convert natural language text into tokens. Includes tokenizers for shingled n-grams, skip n-grams, words, word stems, sentences, paragraphs, characters, shingled characters, lines, tweets, Penn Treebank, regular expressions, as well as functions for counting characters, words, and sentences, and a function for splitting longer texts into separate documents, each with the same number of words. The tokenizers have a consistent interface, and the package is built on the 'stringi' and 'Rcpp' packages for fast yet correct tokenization in 'UTF-8'.

copied from cf-staging / r-tokenizers

Filters
Type Size Name Uploaded Downloads Labels
conda 621.7 kB | win-64/r-tokenizers-0.3.0-r43h8ae3a7c_2.conda  7 months and 15 days ago 296 main
conda 620.4 kB | win-64/r-tokenizers-0.3.0-r44h8ae3a7c_2.conda  7 months and 15 days ago 313 main
conda 615.9 kB | osx-64/r-tokenizers-0.3.0-r44h25d921d_2.conda  7 months and 15 days ago 255 main
conda 614.8 kB | osx-arm64/r-tokenizers-0.3.0-r44hd76f289_2.conda  7 months and 15 days ago 203 main
conda 615.8 kB | osx-arm64/r-tokenizers-0.3.0-r43hd76f289_2.conda  7 months and 15 days ago 217 main
conda 624.9 kB | linux-aarch64/r-tokenizers-0.3.0-r43h6170684_2.conda  7 months and 15 days ago 144 main
conda 626.3 kB | linux-ppc64le/r-tokenizers-0.3.0-r43h0cea4bf_2.conda  7 months and 15 days ago 131 main
conda 623.9 kB | linux-aarch64/r-tokenizers-0.3.0-r44h6170684_2.conda  7 months and 15 days ago 111 main
conda 616.9 kB | osx-64/r-tokenizers-0.3.0-r43h25d921d_2.conda  7 months and 15 days ago 271 main
conda 625.7 kB | linux-ppc64le/r-tokenizers-0.3.0-r44h0cea4bf_2.conda  7 months and 15 days ago 129 main
conda 624.8 kB | linux-64/r-tokenizers-0.3.0-r44h0d4f4ea_2.conda  7 months and 15 days ago 1152 main
conda 625.3 kB | linux-64/r-tokenizers-0.3.0-r43h0d4f4ea_2.conda  7 months and 15 days ago 1490 main
conda 616.1 kB | osx-arm64/r-tokenizers-0.3.0-r42h65f505e_1.conda  1 year and 1 month ago 205 main
conda 615.5 kB | osx-arm64/r-tokenizers-0.3.0-r43h65f505e_1.conda  1 year and 1 month ago 213 main
conda 616.5 kB | osx-64/r-tokenizers-0.3.0-r43h29979af_1.conda  1 year and 1 month ago 254 main
conda 616.8 kB | osx-64/r-tokenizers-0.3.0-r42h29979af_1.conda  1 year and 1 month ago 276 main
conda 620.6 kB | win-64/r-tokenizers-0.3.0-r41ha856d6a_1.conda  1 year and 8 months ago 554 main
conda 617.1 kB | osx-64/r-tokenizers-0.3.0-r42hac7d2d5_1.conda  1 year and 8 months ago 274 main
conda 617.0 kB | osx-64/r-tokenizers-0.3.0-r43hac7d2d5_1.conda  1 year and 8 months ago 292 main
conda 626.0 kB | linux-ppc64le/r-tokenizers-0.3.0-r42hd301276_1.conda  1 year and 8 months ago 141 main
conda 625.8 kB | linux-ppc64le/r-tokenizers-0.3.0-r43hd301276_1.conda  1 year and 8 months ago 152 main
conda 624.3 kB | linux-aarch64/r-tokenizers-0.3.0-r43ha6e910b_1.conda  1 year and 8 months ago 154 main
conda 625.5 kB | linux-64/r-tokenizers-0.3.0-r43ha503ecb_1.conda  1 year and 8 months ago 2540 main
conda 625.0 kB | linux-64/r-tokenizers-0.3.0-r42ha503ecb_1.conda  1 year and 8 months ago 2329 main
conda 625.1 kB | linux-aarch64/r-tokenizers-0.3.0-r42ha6e910b_1.conda  1 year and 8 months ago 180 main
conda 627.4 kB | linux-ppc64le/r-tokenizers-0.3.0-r41he86bb61_0.conda  2 years and 2 months ago 149 main
conda 627.2 kB | linux-ppc64le/r-tokenizers-0.3.0-r42he86bb61_0.conda  2 years and 2 months ago 140 main
conda 624.6 kB | win-64/r-tokenizers-0.3.0-r41ha856d6a_0.conda  2 years and 2 months ago 707 main
conda 615.9 kB | osx-64/r-tokenizers-0.3.0-r42h49197e3_0.conda  2 years and 2 months ago 478 main
conda 616.1 kB | osx-64/r-tokenizers-0.3.0-r41h49197e3_0.conda  2 years and 2 months ago 480 main
conda 624.9 kB | linux-64/r-tokenizers-0.3.0-r42h38f115c_0.conda  2 years and 2 months ago 2175 main
conda 624.6 kB | linux-aarch64/r-tokenizers-0.3.0-r41hc3d7334_0.conda  2 years and 2 months ago 160 main
conda 624.6 kB | linux-aarch64/r-tokenizers-0.3.0-r42hc3d7334_0.conda  2 years and 2 months ago 143 main
conda 624.4 kB | linux-64/r-tokenizers-0.3.0-r41h38f115c_0.conda  2 years and 2 months ago 2007 main
conda 631.0 kB | linux-ppc64le/r-tokenizers-0.2.3-r42ha35a809_1.conda  2 years and 2 months ago 133 main
conda 620.5 kB | osx-64/r-tokenizers-0.2.3-r41h49197e3_1.conda  2 years and 2 months ago 457 main
conda 658.7 kB | win-64/r-tokenizers-0.2.3-r41ha856d6a_1.tar.bz2  2 years and 4 months ago 564 main
conda 657.6 kB | linux-ppc64le/r-tokenizers-0.2.3-r41ha35a809_1.tar.bz2  2 years and 4 months ago 32 main
conda 647.5 kB | osx-64/r-tokenizers-0.2.3-r42h49197e3_1.tar.bz2  2 years and 4 months ago 71 main
conda 657.0 kB | linux-64/r-tokenizers-0.2.3-r41h7525677_1.tar.bz2  2 years and 4 months ago 2035 main
conda 656.9 kB | linux-64/r-tokenizers-0.2.3-r42h7525677_1.tar.bz2  2 years and 4 months ago 2316 main
conda 657.2 kB | linux-aarch64/r-tokenizers-0.2.3-r42hf13958e_1.tar.bz2  2 years and 4 months ago 57 main
conda 657.5 kB | linux-aarch64/r-tokenizers-0.2.3-r41hf13958e_1.tar.bz2  2 years and 4 months ago 57 main
conda 658.5 kB | win-64/r-tokenizers-0.2.3-r41ha856d6a_0.tar.bz2  2 years and 5 months ago 556 main
conda 663.7 kB | win-64/r-tokenizers-0.2.3-r40ha856d6a_0.tar.bz2  2 years and 5 months ago 553 main
conda 657.0 kB | linux-ppc64le/r-tokenizers-0.2.3-r40ha35a809_0.tar.bz2  2 years and 5 months ago 31 main
conda 657.8 kB | linux-ppc64le/r-tokenizers-0.2.3-r41ha35a809_0.tar.bz2  2 years and 5 months ago 33 main
conda 647.8 kB | osx-64/r-tokenizers-0.2.3-r41h49197e3_0.tar.bz2  2 years and 5 months ago 77 main
conda 647.5 kB | osx-64/r-tokenizers-0.2.3-r40h49197e3_0.tar.bz2  2 years and 5 months ago 70 main
conda 657.5 kB | linux-aarch64/r-tokenizers-0.2.3-r41hf13958e_0.tar.bz2  2 years and 5 months ago 58 main
© 2025 Anaconda, Inc. All Rights Reserved. (v4.0.7) Legal | Privacy Policy