Facts about Transformers (in AI)
- 07
In 2023, researchers at Google introduced FlashAttention, reducing transformer attention computation from O(n²) to O(n log n) complexity by computing attention in GPU memory blocks rather than full matrices.
- 06
Multi-head attention in transformers splits the embedding dimension into 8 to 12 parallel attention heads, allowing the model to simultaneously attend to information from different representation subspaces.
- 05
Positional encoding in transformers uses sinusoidal functions with wavelengths ranging from 2π to 10000·2π to allow models to extrapolate to sequence lengths longer than those seen during training.
- 04
Transformer models require O(n²) memory during training due to computing full attention matrices, making it prohibitively expensive to train on sequences longer than 2048 tokens without techniques like gradient checkpointing or sparse attention patterns.
- 03
During inference, transformer models use key-value caching to reduce computational complexity from O(n²) to O(n) per token, enabling practical deployment of billion-parameter models in production systems.
- 02
GPT-3, released by OpenAI in 2020, contains 175 billion parameters and demonstrates that transformer models can perform novel tasks without task-specific fine-tuning through in-context learning.
- 01
The 2017 Attention Is All You Need paper introduced transformers with a 65,000-parameter base model that outperformed recurrent neural networks on machine translation tasks.