tokenizers
Fast State-of-the-Art Tokenizers optimized for Research and Production
- Overview
- Files 8
- Labels 1
- Badges
Installation
To install this package, run one of the following:
Usage Tracking
0.7.0
0.6.0
2 / 8 versions selected
Total downloads: 0
About
Summary
Fast State-of-the-Art Tokenizers optimized for Research and Production
Information Last Updated
Mar 25, 2025 at 16:23
License
Apache-2.0
Total Downloads
2.2K
Platforms
Linux ppc64le Version: 0.7.0
Linux 64 Version: 0.7.0
Documentation
https://huggingface.co/tokenizers/