Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.
| Uploaded | Tue Apr 1 00:32:05 2025 |
| md5 checksum | dba1bbe593e8fb3249a59b102c5e11bb |
| arch | x86_64 |
| build | py312h22610ee_0 |
| depends | libgcc-ng >=11.2.0, libstdcxx-ng >=11.2.0, openssl >=3.0.11,<4.0a0, python >=3.12,<3.13.0a0 |
| license | Apache-2.0 |
| license_family | Apache |
| md5 | dba1bbe593e8fb3249a59b102c5e11bb |
| name | tokenizers |
| platform | linux |
| sha1 | 4f95f31922f68fb6470c42221861d464b86c437c |
| sha256 | 7de03f96c51455b5918f61be231cb4a22b0dc3dccce914b57d196f777eba1f37 |
| size | 4152039 |
| subdir | linux-64 |
| timestamp | 1698903169251 |
| version | 0.13.3 |