Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.
| Uploaded | Tue Apr 1 00:32:01 2025 |
| md5 checksum | ac3e68cd09ad42a9720c2366736dc9d4 |
| arch | x86_64 |
| build | py312ha11519a_0 |
| depends | huggingface_hub >=0.16.4,<1.0, libgcc-ng >=11.2.0, libstdcxx-ng >=11.2.0, openssl >=3.0.14,<4.0a0, python >=3.12,<3.13.0a0 |
| license | Apache-2.0 |
| license_family | Apache |
| md5 | ac3e68cd09ad42a9720c2366736dc9d4 |
| name | tokenizers |
| platform | linux |
| sha1 | e2fde9463080d7eaf089cee3408fc42d1c40723c |
| sha256 | 6368020f07a1ab17b6daa9c53a560682777f9e2db0d64136ffdb796d30ba31e3 |
| size | 2968422 |
| subdir | linux-64 |
| timestamp | 1721140189116 |
| version | 0.19.1 |