tiktoken
tiktoken is a fast BPE tokeniser for use with OpenAI's models
Installation
To install this package, run one of the following:
Usage Tracking
0.9.0
0.5.1
2 / 8 versions selected
Downloads (Last 6 months): 0
Description
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
About
Summary
tiktoken is a fast BPE tokeniser for use with OpenAI's models
Last Updated
Mar 13, 2025 at 09:10
License
MIT
Total Downloads
26.6K
Supported Platforms
linux-aarch64
linux-s390x
macOS-64
macOS-arm64
linux-64
win-64
Unsupported Platforms
linux-ppc64le Last supported version: 0.5.1
GitHub Repository
https://github.com/openai/tiktokenDocumentation
https://github.com/openai/tiktoken/blob/main/README.md