Transformers & Attention
You'll be able to
- Understand the attention mechanism at a high level
- Explain why transformers process sequences in parallel
- Connect transformers to modern language models
The transformer, introduced in 2017, uses an attention mechanism that lets every token in a sequence look at every other token and weigh their relevance. Instead of reading a sentence strictly left-to-right, it builds a rich, context-aware representation all at once.
This parallel, context-aware design scales remarkably well with data and compute — it is the architecture behind virtually every modern large language model.
Challenge
Why context matters
Write one sentence about how a model knowing whether 'bank' means a riverbank or a financial institution depends on attention to surrounding words.
Knowledge Check
Transformers
What does the attention mechanism let each token do?
Transformers are the foundation of most modern large language models.
Answer all questions to submit.