Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.
| Uploaded | Tue Apr 1 00:31:59 2025 |
| md5 checksum | 352190769b4b3b301731c17b2e970d17 |
| arch | x86_64 |
| build | py312h526025d_1 |
| build_number | 1 |
| depends | huggingface_hub >=0.16.4,<1.0, libgcc-ng >=11.2.0, libstdcxx-ng >=11.2.0, openssl >=3.0.15,<4.0a0, python >=3.12,<3.13.0a0 |
| license | Apache-2.0 |
| license_family | Apache |
| md5 | 352190769b4b3b301731c17b2e970d17 |
| name | tokenizers |
| platform | linux |
| sha1 | 95073504ac0a840d65916a6b61bbb480318c40cf |
| sha256 | 71f7d1ef3e32986e20f51b9a46cae69077136541f2ccefcb0ff6f3c0dcc13747 |
| size | 2426795 |
| subdir | linux-64 |
| timestamp | 1732129910978 |
| version | 0.20.1 |