tiktoken
tiktoken is a fast BPE tokeniser for use with OpenAI's models
Installation
To install this package, run one of the following:
Usage Tracking
0.6.0
1 / 8 versions selected
Downloads (Last 6 months): 0
About
Summary
tiktoken is a fast BPE tokeniser for use with OpenAI's models
Last Updated
Jul 12, 2024 at 11:24
License
MIT
Total Downloads
57
Supported Platforms
linux-ppc64le