Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.
| Uploaded | Tue Apr 1 00:32:08 2025 |
| md5 checksum | a0c3c16ced4f126379cf164f0cc513df |
| arch | x86_64 |
| build | py311he7d60b5_0 |
| depends | libgcc-ng >=11.2.0, libstdcxx-ng >=11.2.0, openssl >=1.1.1t,<1.1.2a, python >=3.11,<3.12.0a0 |
| license | Apache-2.0 |
| license_family | Apache |
| md5 | a0c3c16ced4f126379cf164f0cc513df |
| name | tokenizers |
| platform | linux |
| sha1 | c684e7e4d4e7ce91fd9fc45108505e4deb5b9160 |
| sha256 | 92cdc6c13490d3362f7641073fb807ecd8955a61bd340a256ed57510d0bdf3f7 |
| size | 3776054 |
| subdir | linux-64 |
| timestamp | 1676863258866 |
| version | 0.13.2 |