CMD + K

tokenizers

Community

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Installation

To install this package, run one of the following:

Conda
$conda install huggingface::tokenizers

Usage Tracking

0.13.2
0.13.1
0.13.0
0.13.0.dev0
0.12.1
5 / 8 versions selected
Downloads (Last 6 months): 0

About

Summary

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Last Updated

Nov 7, 2022 at 07:59

License

Apache License 2.0

Total Downloads

428.5K

Supported Platforms

linux-64
macOS-64
win-64