About Anaconda Help Download Anaconda
Sign In
Anaconda.cloud

conda-forge / packages / r-tokenizers

Convert natural language text into tokens. Includes tokenizers for shingled n-grams, skip n-grams, words, word stems, sentences, paragraphs, characters, shingled characters, lines, tweets, Penn Treebank, regular expressions, as well as functions for counting characters, words, and sentences, and a function for splitting longer texts into separate documents, each with the same number of words. The tokenizers have a consistent interface, and the package is built on the 'stringi' and 'Rcpp' packages for fast yet correct tokenization in 'UTF-8'.

copied from cf-staging / r-tokenizers

Filters
Type Size Name Uploaded Downloads Labels
conda 656.2 kB | linux-aarch64/r-tokenizers-0.2.3-r40hf13958e_0.tar.bz2  2 years and 5 months ago 58 main
conda 657.0 kB | linux-64/r-tokenizers-0.2.3-r41h7525677_0.tar.bz2  2 years and 5 months ago 1855 main
conda 656.2 kB | linux-64/r-tokenizers-0.2.3-r40h7525677_0.tar.bz2  2 years and 5 months ago 2342 main
conda 655.1 kB | win-64/r-tokenizers-0.2.1-r40ha856d6a_1002.tar.bz2  3 years and 9 months ago 791 main
conda 649.9 kB | win-64/r-tokenizers-0.2.1-r41ha856d6a_1002.tar.bz2  3 years and 9 months ago 812 main
conda 643.2 kB | osx-64/r-tokenizers-0.2.1-r40h9951f98_1002.tar.bz2  3 years and 9 months ago 140 main
conda 643.2 kB | osx-64/r-tokenizers-0.2.1-r41h9951f98_1002.tar.bz2  3 years and 9 months ago 176 main
conda 651.3 kB | linux-ppc64le/r-tokenizers-0.2.1-r40h6203a36_1002.tar.bz2  3 years and 9 months ago 42 main
conda 651.5 kB | linux-ppc64le/r-tokenizers-0.2.1-r41h6203a36_1002.tar.bz2  3 years and 9 months ago 42 main
conda 648.9 kB | linux-aarch64/r-tokenizers-0.2.1-r41hecdc70b_1002.tar.bz2  3 years and 9 months ago 71 main
conda 648.8 kB | linux-aarch64/r-tokenizers-0.2.1-r40hecdc70b_1002.tar.bz2  3 years and 9 months ago 67 main
conda 646.2 kB | linux-64/r-tokenizers-0.2.1-r40h03ef668_1002.tar.bz2  3 years and 9 months ago 3384 main
conda 646.5 kB | linux-64/r-tokenizers-0.2.1-r41h03ef668_1002.tar.bz2  3 years and 9 months ago 3354 main
conda 671.0 kB | linux-ppc64le/r-tokenizers-0.2.1-r40he54295a_1002.tar.bz2  4 years and 7 months ago 53 main
conda 666.2 kB | linux-aarch64/r-tokenizers-0.2.1-r40h0357c0b_1002.tar.bz2  4 years and 7 months ago 77 main
conda 666.4 kB | linux-aarch64/r-tokenizers-0.2.1-r36h0357c0b_1002.tar.bz2  4 years and 7 months ago 75 main
conda 671.3 kB | linux-ppc64le/r-tokenizers-0.2.1-r36he54295a_1002.tar.bz2  4 years and 7 months ago 51 main
conda 656.5 kB | win-64/r-tokenizers-0.2.1-r36h796a38f_1002.tar.bz2  4 years and 9 months ago 1101 main
conda 656.5 kB | win-64/r-tokenizers-0.2.1-r40h796a38f_1002.tar.bz2  4 years and 9 months ago 1032 main
conda 643.6 kB | osx-64/r-tokenizers-0.2.1-r36hc5da6b9_1002.tar.bz2  4 years and 9 months ago 368 main
conda 643.7 kB | osx-64/r-tokenizers-0.2.1-r40hc5da6b9_1002.tar.bz2  4 years and 9 months ago 331 main
conda 652.0 kB | linux-64/r-tokenizers-0.2.1-r36h0357c0b_1002.tar.bz2  4 years and 9 months ago 6032 main
conda 651.6 kB | linux-64/r-tokenizers-0.2.1-r40h0357c0b_1002.tar.bz2  4 years and 9 months ago 3635 main
conda 655.0 kB | win-64/r-tokenizers-0.2.1-r36h796a38f_1001.tar.bz2  5 years and 7 months ago 1402 main cf202003
conda 653.7 kB | win-64/r-tokenizers-0.2.1-r35h796a38f_1001.tar.bz2  5 years and 7 months ago 1370 main cf202003
conda 640.4 kB | osx-64/r-tokenizers-0.2.1-r36hf99fc2c_1001.tar.bz2  5 years and 7 months ago 382 main cf202003
conda 692.8 kB | osx-64/r-tokenizers-0.2.1-r35hf99fc2c_1001.tar.bz2  5 years and 7 months ago 354 main cf202003
conda 646.6 kB | linux-64/r-tokenizers-0.2.1-r35h0357c0b_1001.tar.bz2  5 years and 7 months ago 4896 main cf202003
conda 647.4 kB | linux-64/r-tokenizers-0.2.1-r36h0357c0b_1001.tar.bz2  5 years and 7 months ago 4124 main cf202003
conda 645.9 kB | linux-64/r-tokenizers-0.2.1-r351h29659fb_1000.tar.bz2  6 years and 3 months ago 4654 main gcc7 cf202003
conda 652.7 kB | win-64/r-tokenizers-0.2.1-r351h6115d3f_1000.tar.bz2  6 years and 3 months ago 1629 main cf202003 cf201901
conda 637.5 kB | osx-64/r-tokenizers-0.2.1-r351h466af19_1000.tar.bz2  6 years and 3 months ago 393 main gcc7 cf202003
conda 651.7 kB | win-64/r-tokenizers-0.2.1-r351h6115d3f_0.tar.bz2  6 years and 5 months ago 1719 main cf202003 cf201901
conda 634.3 kB | win-64/r-tokenizers-0.2.1-r341h6115d3f_0.tar.bz2  6 years and 5 months ago 1736 main cf202003 cf201901
conda 674.6 kB | osx-64/r-tokenizers-0.2.1-r351h9d2a408_0.tar.bz2  6 years and 5 months ago 360 main cf202003 cf201901
conda 657.4 kB | osx-64/r-tokenizers-0.2.1-r341h9d2a408_0.tar.bz2  6 years and 5 months ago 360 main cf202003 cf201901
conda 656.0 kB | linux-64/r-tokenizers-0.2.1-r341h9d2a408_0.tar.bz2  6 years and 5 months ago 4676 main cf202003 cf201901
conda 673.4 kB | linux-64/r-tokenizers-0.2.1-r351h9d2a408_0.tar.bz2  6 years and 5 months ago 4689 main cf202003 cf201901
conda 94.0 kB | win-64/r-tokenizers-0.1.4-r341hca4a3dc_1.tar.bz2  6 years and 7 months ago 1731 main cf202003 cf201901
conda 110.6 kB | osx-64/r-tokenizers-0.1.4-r341hfc679d8_1.tar.bz2  6 years and 7 months ago 343 main cf202003 cf201901
conda 121.9 kB | linux-64/r-tokenizers-0.1.4-r341hfc679d8_1.tar.bz2  6 years and 7 months ago 4817 main cf202003 cf201901
conda 110.7 kB | osx-64/r-tokenizers-0.1.4-r3.4.1_0.tar.bz2  7 years and 17 days ago 349 main cf202003 cf201901
conda 94.3 kB | win-64/r-tokenizers-0.1.4-r3.4.1_0.tar.bz2  7 years and 6 months ago 1852 main cf202003 cf201901
conda 116.9 kB | linux-64/r-tokenizers-0.1.4-r3.4.1_0.tar.bz2  7 years and 6 months ago 5023 main cf202003 cf201901
conda 81.2 kB | win-64/r-tokenizers-0.1.4-r3.3.2_0.tar.bz2  7 years and 10 months ago 1876 main cf202003 cf201901
conda 93.3 kB | osx-64/r-tokenizers-0.1.4-r3.3.2_0.tar.bz2  7 years and 10 months ago 354 main cf202003 cf201901
conda 107.0 kB | linux-64/r-tokenizers-0.1.4-r3.3.2_0.tar.bz2  7 years and 10 months ago 5145 main cf202003 cf201901
© 2025 Anaconda, Inc. All Rights Reserved. (v4.0.7) Legal | Privacy Policy