Manage | Anaconda.org

If you were automatically logged out you may need to refresh the page. You're trying to access a page that requires authentication. ×

anaconda / packages / tokenizers 0.23.1

0

Fast State-of-the-Art Tokenizers optimized for Research and Production

License: Apache-2.0
Home: https://github.com/huggingface/tokenizers
Development: https://github.com/huggingface/tokenizers
Documentation: https://huggingface.co/docs/tokenizers/index
45272 total downloads
Last upload: 19 days and 13 hours ago

Installers

linux-aarch64 v0.23.1
linux-s390x v0.21.0
osx-64 v0.21.0
osx-arm64 v0.23.1
win-64 v0.23.1
linux-64 v0.23.1
linux-ppc64le v0.13.3
win-32 v0.10.3

conda install

To install this package run one of the following:
conda install anaconda::tokenizers

Description

Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.

© 2026 Anaconda, Inc. All Rights Reserved. (v4.2.18) Legal | Privacy Policy