Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.
| Uploaded | Tue Apr 1 00:32:03 2025 |
| md5 checksum | ee8b8886b8d1a51c0f86960017f21f3d |
| arch | x86_64 |
| build | py311h22610ee_0 |
| depends | huggingface_hub >=0.16.4,<1.0, libgcc-ng >=11.2.0, libstdcxx-ng >=11.2.0, openssl >=3.0.13,<4.0a0, python >=3.11,<3.12.0a0 |
| license | Apache-2.0 |
| license_family | Apache |
| md5 | ee8b8886b8d1a51c0f86960017f21f3d |
| name | tokenizers |
| platform | linux |
| sha1 | 906ea974ecc6bf7300884104d7e84efa94cf7b10 |
| sha256 | aad7de517e648c66a6c2928718b7730492ccedf41c09bdc519ef8bc4a4ee3b27 |
| size | 3156041 |
| subdir | linux-64 |
| timestamp | 1708633996789 |
| version | 0.15.1 |