tokenizers
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
Installation
To install this package, run one of the following:
Usage Tracking
0.13.2
0.13.1
0.13.0
0.13.0.dev0
0.12.1
5 / 8 versions selected
Downloads (Last 6 months): 0
About
Summary
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
Last Updated
Nov 7, 2022 at 07:59
License
Apache License 2.0
Total Downloads
428.5K
Supported Platforms
linux-64
macOS-64
win-64