Convert natural language text into tokens. The tokenizers have a consistent interface and are compatible with Unicode, thanks to being built on the 'stringi' package. Includes tokenizers for shingled n-grams, skip n-grams, words, word stems, sentences, paragraphs, characters, lines, and regular expressions.
https://anaconda.org/kite-eating-tree/r-tokenizers/badges/version.svg
https://anaconda.org/kite-eating-tree/r-tokenizers/badges/latest_release_date.svg
https://anaconda.org/kite-eating-tree/r-tokenizers/badges/latest_release_relative_date.svg
https://anaconda.org/kite-eating-tree/r-tokenizers/badges/platforms.svg
https://anaconda.org/kite-eating-tree/r-tokenizers/badges/license.svg
https://anaconda.org/kite-eating-tree/r-tokenizers/badges/downloads.svg