Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.
| Uploaded | Tue Apr 1 00:32:00 2025 |
| md5 checksum | 991741666e419f65240174749ad4481b |
| arch | x86_64 |
| build | py310hff361bb_0 |
| depends | huggingface_hub >=0.16.4,<1.0, libgcc-ng >=11.2.0, libstdcxx-ng >=11.2.0, openssl >=3.0.14,<4.0a0, python >=3.10,<3.11.0a0 |
| license | Apache-2.0 |
| license_family | Apache |
| md5 | 991741666e419f65240174749ad4481b |
| name | tokenizers |
| platform | linux |
| sha1 | 0ed565b8a07c84430586b0f30f55a62635271238 |
| sha256 | 052b8628da11750e9d44f68e89d1764dce7d8d84ecb8d04e1efd32d9f0ea672d |
| size | 2961331 |
| subdir | linux-64 |
| timestamp | 1721139673677 |
| version | 0.19.1 |