Word2Vec (Word to Vector) Model
Let's go through a simplified manual example to illustrate the basic concept of Word2Vec, specifically focusing on the Continuous Bag of Words (CBOW) approach. In CBOW, the model is trained to predict a target word based on its context (surrounding words).
Example Sentence:
"The quick brown fox jumped over the lazy dog."
Step 2: Create Context-Target Pairs: Define context-target pairs by selecting a target word and its surrounding context words. Let's choose a window size of 2 (i.e., considering two words on each side).
For the target word "fox," create pairs:
- Context: ["quick", "brown", "jumped", "over"]
- Target: "fox"
Similarly, create pairs for other words in the sentence.
Step 3: One-Hot Encoding: Encode words using one-hot encoding, where each word is represented by a binary vector with a 1 at its index in the vocabulary.
Example:
- "quick" might be represented as [0, 1, 0, 0, 0, 0, 0, 0, 0, 0].
Step 4: Initialize Word Vectors: Initialize word vectors for each word in the vocabulary. For simplicity, let's assume each word vector has a length of 3.
Example:
- "quick": [0.2, 0.5, -0.1]
Step 5: Train the CBOW Model: Train the CBOW model to predict the target word "fox" based on its context words. Adjust the word vectors during training to minimize the prediction error.
Step 6: Updated Word Vectors: After training, the word vectors are updated to better capture the context relationships. For example, the vector for "quick" might shift to [0.3, 0.6, -0.2].
Step 7: Word Similarities: The trained word vectors can be used to calculate similarities between words. Similar words will have vectors that are closer in distance.
This is a simplified illustration of Word2Vec's CBOW approach. The actual model involves a neural network with an embedding layer, and the training process uses backpropagation to adjust word vectors based on the prediction error. The result is word vectors that capture semantic relationships, allowing similar words to have similar vector representations.
