Tokenizers | Anaconda.org

main / packages / tokenizers 0.20.1

0

Fast State-of-the-Art Tokenizers optimized for Research and Production

License: Apache-2.0
Home: https://github.com/huggingface/tokenizers
Development: https://github.com/huggingface/tokenizers
Documentation: https://huggingface.co/docs/tokenizers/index
6599 total downloads
Last upload: 1 month and 6 days ago

Installers

linux-64 v0.20.1
linux-aarch64 v0.20.1
linux-s390x v0.20.1
osx-64 v0.20.1
osx-arm64 v0.20.1
win-64 v0.20.1
linux-ppc64le v0.13.3
win-32 v0.10.3

conda install

To install this package run one of the following:
conda install main::tokenizers

Description

Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.

© 2024 Anaconda, Inc. All Rights Reserved. (v4.0.6) Legal | Privacy Policy