AI peer review tools for research papers.

Transformers are slow and memory-hungry on long sequences, since the time and memory complexity of self-attention are quadratic in sequence length. We propose FlashAttention, an IO-aware exact attention algorithm that uses tiling to reduce the number of memory reads/writes between GPU high bandwidth memory (HBM) and on-chip SRAM. FlashAttention is 3x faster than the existing PyTorch implementation on GPT-2 and 15% faster than baseline implementations on long-range arena. We extend FlashAttention to block-sparse attention, yielding an approximate attention algorithm that is faster than any existing approximate attention method.