Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.
| Uploaded | Tue Apr 1 00:31:57 2025 |
| md5 checksum | 0f4337ece7f94e2c064f19fe8ea7af52 |
| arch | x86_64 |
| build | py310h7d74088_0 |
| depends | huggingface_hub >=0.16.4,<1.0, libgcc-ng >=11.2.0, libstdcxx-ng >=11.2.0, openssl >=3.0.15,<4.0a0, python >=3.10,<3.11.0a0 |
| license | Apache-2.0 |
| license_family | Apache |
| md5 | 0f4337ece7f94e2c064f19fe8ea7af52 |
| name | tokenizers |
| platform | linux |
| sha1 | d690373e1e3cb59ca0b19d11949dbd4ab29ec375 |
| sha256 | 5ec0bfec7979f8e4f4b8fddc40bd511dd76e70610777e0b0df623b88c7faf673 |
| size | 3086344 |
| subdir | linux-64 |
| timestamp | 1729697350475 |
| version | 0.20.1 |