Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.
| Uploaded | Tue Apr 1 00:32:04 2025 |
| md5 checksum | 175dd62d7990253ef45b1576fc5ce973 |
| arch | x86_64 |
| build | py310h22610ee_0 |
| depends | libgcc-ng >=11.2.0, libstdcxx-ng >=11.2.0, openssl >=3.0.11,<4.0a0, python >=3.10,<3.11.0a0 |
| license | Apache-2.0 |
| license_family | Apache |
| md5 | 175dd62d7990253ef45b1576fc5ce973 |
| name | tokenizers |
| platform | linux |
| sha1 | 46dbcda3db12dda2d72d301e39ac8090814f2b27 |
| sha256 | 658a8cf3fe502fd209e44d8e14a6229809d18451c81bea2e1826c3e4f8cdfd71 |
| size | 4637344 |
| subdir | linux-64 |
| timestamp | 1696540504144 |
| version | 0.13.3 |