FastText Word Embedding Technique 

FastText is an extension of Word2Vec that represents words as bags of character n-grams. It is particularly effective for handling morphologically rich languages and dealing with out-of-vocabulary words. Let's go through a simplified manual example to understand the basic concept of FastText.

Example Sentence:

"The quick brown fox jumped over the lazy dog."

Step 1: Tokenization: Tokenize the sentence into individual words.

["The", "quick", "brown", "fox", "jumped", "over", "the", "lazy", "dog", "."]

Step 2: Create Character N-grams: Generate character n-grams for each word. Let's choose an example with n=3 (trigrams).

["<Th", "The", "he>", "<qu", "qui", "uic", "ick", "ck>", ...]

Each word is represented as a bag of character trigrams, including boundary symbols < and > to mark the beginning and end of words.

Step 3: Initialize Word Vectors: Initialize word vectors for each word and character n-gram in the vocabulary. For simplicity, let's assume each word vector has a length of 3, and each character n-gram vector has a length of 2.

Example:

• "quick" Word Vector: [0.1, 0.2, -0.3]
• "<qu" Character N-gram Vector: [0.4, -0.5]
• ... and so on for other words and n-grams.

Step 4: Train the FastText Model: Train the FastText model to predict the target word based on its character n-grams. The model is trained using a shallow neural network with an embedding layer for word vectors and character n-gram vectors.

Step 5: Updated Word Vectors: After training, the word vectors are updated to better capture the context relationships based on character n-grams.

Step 6: Obtain Word Vectors:

The learned word vectors represent words in a continuous vector space. These vectors are composed not only of the traditional word embeddings but also incorporate information from character n-grams.

FastText can handle out-of-vocabulary words because it can generate embeddings for subwords. For example, if a new word appears with a character n-gram that was present during training, FastText can still provide a meaningful vector representation.

This is a simplified illustration of how FastText works. In practice, FastText considers subword information during training, leading to embeddings that capture morphological nuances and improve performance in various natural language processing tasks