Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.
| Uploaded | Tue Apr 1 00:32:00 2025 |
| md5 checksum | c7871e334c3f30994bb6da50954c3ed8 |
| arch | x86_64 |
| build | py311h585d796_0 |
| depends | huggingface_hub >=0.16.4,<1.0, libgcc-ng >=11.2.0, libstdcxx-ng >=11.2.0, openssl >=3.0.14,<4.0a0, python >=3.11,<3.12.0a0 |
| license | Apache-2.0 |
| license_family | Apache |
| md5 | c7871e334c3f30994bb6da50954c3ed8 |
| name | tokenizers |
| platform | linux |
| sha1 | fe4995ce2825320cacb0f71f60ff16e07c32e82c |
| sha256 | 2e9f00a8337018448853cf2eeb3aa3549a99cc683ac58c99eb1be0880a6edfb0 |
| size | 2968940 |
| subdir | linux-64 |
| timestamp | 1721139935157 |
| version | 0.19.1 |