Tokenizers | Anaconda.org

Anaconda-python-gte-3.10 / packages / tokenizers 0.21.0

0

Fast State-of-the-Art Tokenizers optimized for Research and Production

License: Apache-2.0
Home: https://github.com/huggingface/tokenizers
Development: https://github.com/huggingface/tokenizers
Documentation: https://huggingface.co/docs/tokenizers/index
0 total downloads
Last upload: 5 months and 25 days ago

Installers

linux-64 v0.21.0

conda install

To install this package run one of the following:
conda install anaconda-python-gte-3.10::tokenizers

Description

Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.

© 2025 Anaconda, Inc. All Rights Reserved. (v4.2.2) Legal | Privacy Policy