Convert natural language text into tokens. Includes tokenizers for shingled n-grams, skip n-grams, words, word stems, sentences, paragraphs, characters, shingled characters, lines, tweets, Penn Treebank, regular expressions, as well as functions for counting characters, words, and sentences, and a function for splitting longer texts into separate documents, each with the same number of words. The tokenizers have a consistent interface, and the package is built on the 'stringi' and 'Rcpp' packages for fast yet correct tokenization in 'UTF-8'.
copied from cf-staging / r-tokenizers| Label | Latest Version |
|---|---|
| main | 0.3.0 |
| cf201901 | 0.2.1 |
| cf202003 | 0.2.1 |
| gcc7 | 0.2.1 |
