Self-Attention

term_id: self_attention

Category: basic_concepts

Definition

Self-attention enables models to capture dependencies between all positions in a sequence simultaneously, regardless of distance. By computing attention scores between every pair of tokens, it allows the network to dynamically focus on relevant context, forming the foundational layer of Transformer architectures used in modern natural language processing and computer vision tasks.

Summary

A mechanism allowing a neural network to weigh the importance of different parts of the input sequence relative to each other.

Key Concepts

  • Query-Key-Value
  • Attention Scores
  • Contextual Weighting
  • Parallel Processing

Use Cases

  • Machine translation
  • Text summarization
  • Image classification via Vision Transformers

Code Example

1
2

import torch.nn as nn
attn = nn.MultiheadAttention(embed_dim=512, num_heads=8)

Related