Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.
| Uploaded | Tue Apr 1 00:31:55 2025 |
| md5 checksum | 0e1feb495d364155e760eee03fc43597 |
| arch | x86_64 |
| build | py311h5e3497e_0 |
| depends | huggingface_hub >=0.16.4,<1.0, libgcc-ng >=11.2.0, libstdcxx-ng >=11.2.0, openssl >=3.0.16,<4.0a0, python >=3.11,<3.12.0a0 |
| license | Apache-2.0 |
| license_family | Apache |
| md5 | 0e1feb495d364155e760eee03fc43597 |
| name | tokenizers |
| platform | linux |
| sha1 | a9ac31b3a09c00705855410add76f183b85e476a |
| sha256 | 419b558d5bb875ca1d1780a4d979af2c8c61af7858939851c0df0b3e6995bcd9 |
| size | 2432057 |
| subdir | linux-64 |
| timestamp | 1741370830694 |
| version | 0.21.0 |