Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.
| Uploaded | Tue Apr 1 00:31:59 2025 |
| md5 checksum | ab2c27849af87e8e172c31a9e85b46fa |
| arch | x86_64 |
| build | py312h28f53f8_0 |
| depends | huggingface_hub >=0.16.4,<1.0, libgcc-ng >=11.2.0, libstdcxx-ng >=11.2.0, openssl >=3.0.15,<4.0a0, python >=3.12,<3.13.0a0 |
| license | Apache-2.0 |
| license_family | Apache |
| md5 | ab2c27849af87e8e172c31a9e85b46fa |
| name | tokenizers |
| platform | linux |
| sha1 | 1df2fdca6c16a95968a994aa41eba175bb524b1f |
| sha256 | 9b30b8256f653281e797ee2712929c8fc189718fb7b700f08460e428faf72586 |
| size | 3095986 |
| subdir | linux-64 |
| timestamp | 1729696804826 |
| version | 0.20.1 |