tiktoken is a fast BPE tokeniser for use with OpenAI's models
conda install anaconda::tiktoken
tiktoken is a fast BPE tokeniser for use with OpenAI's models.