About Anaconda Help Download Anaconda
Sign In
Anaconda.cloud

r / packages / r-tokenizers

Convert natural language text into tokens. Includes tokenizers for shingled n-grams, skip n-grams, words, word stems, sentences, paragraphs, characters, shingled characters, lines, Penn Treebank, regular expressions, as well as functions for counting characters, words, and sentences, and a function for splitting longer texts into separate documents, each with the same number of words. The tokenizers have a consistent interface, and the package is built on the 'stringi' and 'Rcpp' packages for fast yet correct tokenization in 'UTF-8'.

Filters
Type Size Name Uploaded Downloads Labels
conda 648.9 kB | linux-64/r-tokenizers-0.3.0-r43h884c59f_0.tar.bz2  1 year and 29 days ago 47 main
conda 653.7 kB | linux-64/r-tokenizers-0.2.1-r42h884c59f_0.tar.bz2  2 years and 7 months ago 87 main
conda 652.7 kB | win-64/r-tokenizers-0.2.1-r351h6f4ce42_0.tar.bz2  4 years and 11 months ago 105 main
conda 655.6 kB | win-64/r-tokenizers-0.2.1-r36h6115d3f_0.tar.bz2  4 years and 11 months ago 205 main
conda 1.1 MB | win-64/r-tokenizers-0.2.1-mro351_0.tar.bz2  4 years and 11 months ago 106 main
conda 94.6 kB | win-64/r-tokenizers-0.1.4-r343h889e2dd_0.tar.bz2  4 years and 11 months ago 100 main
conda 94.9 kB | win-64/r-tokenizers-0.1.4-r342h1e42628_0.tar.bz2  4 years and 11 months ago 94 main
conda 95.1 kB | win-64/r-tokenizers-0.1.4-r3.4.1_0.tar.bz2  4 years and 11 months ago 94 main
conda 81.2 kB | win-64/r-tokenizers-0.1.4-r3.3.2_0.tar.bz2  4 years and 11 months ago 93 main
conda 605.9 kB | win-64/r-tokenizers-0.1.4-mro343h889e2dd_0.tar.bz2  4 years and 11 months ago 105 main
conda 659.9 kB | win-32/r-tokenizers-0.2.1-r36h6115d3f_0.tar.bz2  4 years and 11 months ago 15 main
conda 656.8 kB | win-32/r-tokenizers-0.2.1-r351h6f4ce42_0.tar.bz2  4 years and 11 months ago 15 main
conda 95.9 kB | win-32/r-tokenizers-0.1.4-r343h889e2dd_0.tar.bz2  4 years and 11 months ago 15 main
conda 96.2 kB | win-32/r-tokenizers-0.1.4-r342h8c7424a_0.tar.bz2  4 years and 11 months ago 16 main
conda 82.2 kB | win-32/r-tokenizers-0.1.4-r3.3.2_0.tar.bz2  4 years and 11 months ago 16 main
conda 96.0 kB | win-32/r-tokenizers-0.1.4-r3.4.1_0.tar.bz2  4 years and 11 months ago 16 main
conda 640.4 kB | osx-64/r-tokenizers-0.2.1-r36h466af19_0.tar.bz2  4 years and 11 months ago 62 main
conda 637.0 kB | osx-64/r-tokenizers-0.2.1-r351h32998d9_0.tar.bz2  4 years and 11 months ago 25 main
conda 639.1 kB | osx-64/r-tokenizers-0.2.1-r350h1104f25_0.tar.bz2  4 years and 11 months ago 26 main
conda 85.0 kB | osx-64/r-tokenizers-0.1.4-r343he124285_0.tar.bz2  4 years and 11 months ago 26 main
conda 84.9 kB | osx-64/r-tokenizers-0.1.4-r342h2f5c633_0.tar.bz2  4 years and 11 months ago 25 main
conda 107.8 kB | osx-64/r-tokenizers-0.1.4-r3.4.1_0.tar.bz2  4 years and 11 months ago 25 main
conda 105.4 kB | osx-64/r-tokenizers-0.1.4-r3.3.2_0.tar.bz2  4 years and 11 months ago 23 main
conda 647.6 kB | linux-64/r-tokenizers-0.2.1-r36h29659fb_0.tar.bz2  4 years and 11 months ago 218 main
conda 645.0 kB | linux-64/r-tokenizers-0.2.1-r351h29659fb_0.tar.bz2  4 years and 11 months ago 74 main
conda 644.3 kB | linux-64/r-tokenizers-0.2.1-r350hebe7666_0.tar.bz2  4 years and 11 months ago 65 main
conda 645.4 kB | linux-64/r-tokenizers-0.2.1-mro351hebc1506_0.tar.bz2  4 years and 11 months ago 71 main
conda 645.1 kB | linux-64/r-tokenizers-0.2.1-mro350hc05d2f9_0.tar.bz2  4 years and 11 months ago 68 main
conda 91.6 kB | linux-64/r-tokenizers-0.1.4-r343h599a50d_0.tar.bz2  4 years and 11 months ago 68 main
conda 91.3 kB | linux-64/r-tokenizers-0.1.4-r342h0fb3b52_0.tar.bz2  4 years and 11 months ago 66 main
conda 117.9 kB | linux-64/r-tokenizers-0.1.4-r3.4.1_0.tar.bz2  4 years and 11 months ago 75 main
conda 107.5 kB | linux-64/r-tokenizers-0.1.4-r3.3.2_0.tar.bz2  4 years and 11 months ago 80 main
conda 91.3 kB | linux-64/r-tokenizers-0.1.4-mro343h599a50d_0.tar.bz2  4 years and 11 months ago 64 main
conda 643.5 kB | linux-32/r-tokenizers-0.2.1-r350hebe7666_0.tar.bz2  4 years and 11 months ago 22 main
conda 88.2 kB | linux-32/r-tokenizers-0.1.4-r343h599a50d_0.tar.bz2  4 years and 11 months ago 21 main
conda 88.0 kB | linux-32/r-tokenizers-0.1.4-r342h5c34818_0.tar.bz2  4 years and 11 months ago 20 main
conda 119.7 kB | linux-32/r-tokenizers-0.1.4-r3.4.1_0.tar.bz2  4 years and 11 months ago 21 main
conda 108.8 kB | linux-32/r-tokenizers-0.1.4-r3.3.2_0.tar.bz2  4 years and 11 months ago 19 main
© 2025 Anaconda, Inc. All Rights Reserved. (v4.0.7) Legal | Privacy Policy