Machine Learning

Search This Blog

Self- Attention Mechanism - Part 2

on September 03, 2025

Get link
Facebook
X
Pinterest
Email
Other Apps

How the self-attention mechanism works

Summary of Self attention steps

Transformers

Get link
Facebook
X
Pinterest
Email
Other Apps

Comments

NLP - Tokenization

What is tokenization? Tokenization is splitting raw text into smaller units called tokens—these might be words, subwords, or individual characters. Why bother? Imagine you’re handed a dense, unformatted sentence without spaces or punctuation: "GenerativeAIisfascinatingandisthefuture" . Without tokenization, deciphering meaningful segments becomes nearly impossible. Humans can intuitively parse this into generative AI is fascinating and is the future , but machines require explicit instructions to recognize word boundaries. Tokenization bridges this gap, enabling machines to identify and separate individual words or meaningful subunits within the text. In reality, tokenization results may vary depending on the tokenizer used. The above image is for demonstration purposes only. Moreover, languages vary widely in their structure and word boundaries. For instance: English: Spaces separate words, making basic tokenization relatively straightforward. Chin...

NLP - Text Preprocessing

Text Preprocessing Essentials Learn how preprocessing transforms raw, messy text into clean, normalized data that fuels advanced NLP models. Imagine you’re building an email spam filter, and a user sends a message with ALL CAPS, random punctuation, and maybe even an emoji or two. You can quickly spot the yelling as a human and decide whether to shrug it off or block it. But to a computer, that message is just a blob of characters—completely chaotic without further processing. It struggles to understand the true intention unless you clean things up first. That’s exactly what text preprocessing does: it tames the messy realm of human language, giving your models the clarity they need to do their job effectively. We’ll explore three essential techniques that transform raw, chaotic text into digestible input for NLP: tokenization, stemming, and lemmatization. We’ll also see how these steps emerged from real-world needs—like improving search engines—and evolved into indispensable tool...

Self attention Mechanism - Step 1

Step 1: Calculating the similarity score The first step in the self-attention mechanism is to compute the dot product between the query matrix, Q, and the key matrix K^ T. The following shows the result of the dot product between the query matrix Q and K^T. The similarity score for ' I' Let's look into the first row of the Q.K^T matrix as shown in the following figure. We can observe that we are computing the dot product between the query vector q1(I) and all the key vectors— k1(I), k2(am), k3(good). C omputing the dot product between two vectors tells us how similar they are. Therefore, computing the dot product between the query vector (q1) and key vectors (k1, k2, k3) tells us how similar the query vector q1 is to key vectors k1(I), k2(am)and k3(good). By looking at the first row of the Q.K^T matrix, we can understand that the word ' I' is more related to itself than the words ' am' and ' good' since the dot produ...

Powered by Blogger

Self- Attention Mechanism - Part 2

Comments

Post a Comment