SentencePiece is an unsupervised text tokenizer and detokenizer mainly for Neural Network-based text generation systems where the vocabulary size is predetermined prior to the neural model training.
Label | Latest Version |
---|---|
main | 0.1.91 |
prerelease | 0.2.0 |