Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.
| Uploaded | Tue Apr 1 00:31:58 2025 |
| md5 checksum | d66fff14725e717cb923a4e630f0f86b |
| arch | x86_64 |
| build | py311h766900b_0 |
| depends | huggingface_hub >=0.16.4,<1.0, libgcc-ng >=11.2.0, libstdcxx-ng >=11.2.0, openssl >=3.0.15,<4.0a0, python >=3.11,<3.12.0a0 |
| license | Apache-2.0 |
| license_family | Apache |
| md5 | d66fff14725e717cb923a4e630f0f86b |
| name | tokenizers |
| platform | linux |
| sha1 | f173e8331d00159b89db82b20223cd34bdbaa969 |
| sha256 | 63dae445a1eba48368967f0a19008adb04feb034ac340da49ae9139c33e7ed6a |
| size | 3097174 |
| subdir | linux-64 |
| timestamp | 1729697083831 |
| version | 0.20.1 |