Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.
| Uploaded | Tue Apr 1 00:32:02 2025 |
| md5 checksum | 32951725bd0d5a49051555f0e642137b |
| arch | x86_64 |
| build | py310h22610ee_0 |
| depends | huggingface_hub >=0.16.4,<1.0, libgcc-ng >=11.2.0, libstdcxx-ng >=11.2.0, openssl >=3.0.13,<4.0a0, python >=3.10,<3.11.0a0 |
| license | Apache-2.0 |
| license_family | Apache |
| md5 | 32951725bd0d5a49051555f0e642137b |
| name | tokenizers |
| platform | linux |
| sha1 | 4c0091c5550888eea3fb07cc64ed8dbfe4f231dd |
| sha256 | 1c32d8c7b1692a584c0ea2609a6afcd1979e25e908c6e3b5344d6e1f2ab8b4f5 |
| size | 3145492 |
| subdir | linux-64 |
| timestamp | 1708634325373 |
| version | 0.15.1 |