Fast State-of-the-Art Tokenizers optimized for Research and Production
conda install nogil::tokenizers
Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.
