GloVe (Global Vectors for Word Representation)

let's go through a manual example to understand the basic concept of GloVe (Global Vectors for Word Representation). GloVe is an unsupervised learning algorithm for obtaining vector representations for words by considering global word co-occurrence statistics. It constructs a word-word co-occurrence matrix and factorizes it to obtain word vectors.

Example Sentence:

"The quick brown fox jumped over the lazy dog."

Step 1: Tokenization: Tokenize the sentence into individual words.

["The", "quick", "brown", "fox", "jumped", "over", "the", "lazy", "dog", "."]

Step 2: Create Word-Context Pairs: Define word-context pairs by selecting a word and its surrounding context words. Let's choose a context window size of 2 (i.e., considering two words on each side).

For the word "fox," create pairs:

• Context: ["quick", "brown", "jumped", "over"]
• Word: "fox"

Similarly, create pairs for other words in the sentence.

Step 3: Create Co-occurrence Matrix: Create a word co-occurrence matrix based on the word-context pairs. The matrix represents how often words appear together in the given context window.

Example Co-occurrence Matrix:

| The | quick | brown | fox | jumped | over | the | lazy | dog | . -------- | --- | ----- | ----- | --- | ------ | ---- | --- | ---- | --- | - The | 0 | 1 | 0 | 1 | 0 | 0 | 1 | 0 | 0 | 0 quick | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 0 | 0 brown | 0 | 1 | 0 | 1 | 0 | 0 | 1 | 0 | 0 | 0 ... (similar entries for other words)

Step 4: Compute Word Probabilities: Calculate the probabilities of word co-occurrence using the formula: $�(�\mathrm{\mid }�)=\frac{\text{Co-occurrence count of words i and j}}{\text{Total co-occurrence count of word j}}$

Step 5: Define Objective Function: Define the objective function for GloVe that the model aims to minimize. The objective is to learn word vectors such that the dot product of the vectors approximates the logarithm of the word co-occurrence probabilities. $�(�)={\sum }_{�,�}�({�}_{��})({\mathbf{�}}_{�}^{�}{\mathbf{�}}_{�}+{�}_{�}+{�}_{�}-\log ({�}_{��}){)}^2$

Here, ${�}_{��}$ is the probability of co-occurrence between words i and j, ${\mathbf{�}}_{�}$ and ${\mathbf{�}}_{�}$ are the word vectors, ${�}_{�}$ and ${�}_{�}$ are bias terms, and $�$ is a weighting function.

Step 6: Training: Use optimization algorithms like stochastic gradient descent (SGD) to minimize the objective function. Update word vectors and biases iteratively during training.

Step 7: Obtain Word Vectors: After training, the learned word vectors represent words in a continuous vector space.

This is a simplified explanation of how GloVe works. The actual GloVe algorithm involves more sophisticated optimization techniques and hyperparameter tuning. GloVe vectors are known for capturing global semantic relationships and are often used in natural language processing tasks.