CMD + K

sentencepiece-python

Anaconda Verified

Unsupervised text tokenizer for Neural Network-based text generation.

Installation

To install this package, run one of the following:

Conda
$conda install anaconda::sentencepiece-python

Usage Tracking

0.2.1
0.2.0
2 / 8 versions selected
Downloads (Last 6 months): 0

Description

SentencePiece is an unsupervised text tokenizer and detokenizer mainly for Neural Network-based text generation systems where the vocabulary size is predetermined prior to the neural model training. SentencePiece implements subword units (e.g., byte-pair-encoding (BPE) [Sennrich et al.]) and unigram language model [Kudo]) with the extension of direct training from raw sentences. SentencePiece allows us to make a purely end-to-end system that does not depend on language-specific pre/postprocessing.

About

Summary

Unsupervised text tokenizer for Neural Network-based text generation.

Last Updated

Dec 11, 2025 at 14:13

License

Apache-2.0

Total Downloads

18.6K

Supported Platforms

linux-64
linux-aarch64
macOS-arm64
win-64

Unsupported Platforms

linux-s390x Last supported version: 0.2.0
macOS-64 Last supported version: 0.2.0