WordPiece: A Subword Segmentation Algorithm

WordPiece is a subword tokenization algorithm that breaks down words into smaller units called “wordpieces.”

These wordpieces can be common prefixes, suffixes, or other sub-units that appear frequently in the training data. By using subword units, WordPiece enables models to handle out-of-vocabulary words, capture semantic relationships between words, and reduce the size of the vocabulary.

It was initially introduced in 2016 to enhance the performance of models dealing with complex linguistic structures by breaking down words into smaller, manageable units known as subword tokens.

The Need for Subword Segmentation

Traditional NLP models typically rely on words as the basic units of language. However, this approach presents several challenges:

• Rare words: Many words, especially in languages with rich morphology like English, are infrequent. This leads to sparse data and poor model performance.
• Out-of-vocabulary (OOV) words: Models trained on a finite vocabulary may encounter words they haven’t seen during training, leading to errors.
• Morphological variations: Many words have different morphological forms (e.g., singular/plural, verb conjugations). Representing these variations accurately can be difficult.

Subword segmentation addresses these issues by breaking down words into smaller units, such as:

• Characters: The most basic units, but may not capture meaningful linguistic units.
• Syllables: Can be language-specific and difficult to define consistently.
• Subwords: Meaningful units smaller than words, such as prefixes, suffixes, and common roots.

Mechanism of WordPiece Tokenization

WordPiece is a data-driven approach to subword segmentation. It aims to find a vocabulary of subwords that maximizes the likelihood of the training data.

Here’s a simplified overview of the algorithm:

1. Initialization: The vocabulary starts with individual characters from the training corpus.
2. Language Model Training: A language model is trained on the training data using the initial vocabulary.
3. Iterative Refinement: The algorithm iteratively merges two existing wordpieces in the vocabulary to create a new wordpiece.
   • For each pair of consecutive subwords in the training data, consider merging them into a single subword.
   • Calculate the increase in the likelihood of the training data if this merge were to occur.
   • Select the merge that results in the greatest increase in likelihood.
   • Add the new merged subword to the vocabulary.
4. Repeat: Continue iteratively merging subwords until a desired vocabulary size is reached.

WordPiece vs. Byte-Pair Encoding (BPE)

WordPiece is closely related to another popular subword tokenization algorithm called Byte-Pair Encoding (BPE). The key difference lies in how they select which subword units to merge.

• BPE merges the most frequent pair of symbols (bytes or characters). It is a greedy, data-driven approach that focuses on frequency.
• WordPiece chooses the pair of symbols that maximizes the likelihood of the training data. It evaluates all possible subword combinations and selects the one that results in the largest increase in likelihood upon merging. This makes WordPiece more computationally intensive but potentially more effective in capturing semantic relationships.

In essence, BPE is a simpler, frequency-based approach, while WordPiece is a more complex, likelihood-based approach.

Advantages of WordPiece

• Out-of-Vocabulary Handling: By decomposing words into subword tokens, WordPiece effectively addresses the challenges posed by rare or unseen words. This capability minimizes OOV instances, which are common hurdles in NLP tasks.
• Efficient Text Encoding: Compared to word-level tokenization, WordPiece reduces the vocabulary size significantly, leading to more efficient data representation and processing. This efficiency is crucial for computational resources and model complexity.
• Language Agnostic: The algorithm’s design makes it applicable across multiple languages, enhancing its versatility for multilingual applications.
• Reduces Vocabulary Size: Compared to word-level tokenization, WordPiece can significantly reduce the vocabulary size, which leads to more efficient models.

References

1. “Google’s neural machine translation system: Bridging the gap between human and machine translation.” arXiv preprint (2016).
2. wordpiece wiki
3. Hugging face course