Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.
| Uploaded | Tue Apr 1 00:32:08 2025 |
| md5 checksum | 184da442ef1f746b1e2ff56f1938d2b7 |
| arch | x86_64 |
| build | py311h22610ee_1 |
| build_number | 1 |
| depends | libgcc-ng >=11.2.0, libstdcxx-ng >=11.2.0, openssl >=3.0.8,<4.0a0, python >=3.11,<3.12.0a0 |
| license | Apache-2.0 |
| license_family | Apache |
| md5 | 184da442ef1f746b1e2ff56f1938d2b7 |
| name | tokenizers |
| platform | linux |
| sha1 | fe276350cdb9521824c8ac13561c6dc682d992e8 |
| sha256 | 431345783c5abb8c15ecc675cfe1d77d94de9355d2062e2fec3fad31b7d68ec6 |
| size | 4352671 |
| subdir | linux-64 |
| timestamp | 1687193772816 |
| version | 0.13.2 |