CMD + K
rocketce/tokenizers

tokenizers

Community

Fast State-of-the-Art Tokenizers optimized for Research and Production

Installation

To install this package, run one of the following:

Conda
$conda install rocketce::tokenizers

Usage Tracking

0.15.2
0.13.3
2 / 8 versions selected
Total downloads: 0

About

Summary

Fast State-of-the-Art Tokenizers optimized for Research and Production

Information Last Updated

Aug 20, 2025 at 15:51

License

Apache-2.0

Total Downloads

1.5K

Platforms

Linux ppc64le Version: 0.15.2

Home

https://pypi.org/project/tokenizers/

GitHub Repository

https://github.com/huggingface/tokenizers

Documentation

https://huggingface.co/tokenizers/