event

GPT-2 Performance Analysis

4 items · started 2026-10-01 · closed 2026-10-04

  1. Why didn't we get GPT-2 in 2005? The ancient Romans were never great at building ships and never tried to explore the Atlantic.

  2. Should low-precision transformer inference use stochastic rounding (SR) or round-to-nearest (RN)? The answer depends on where in the network you look.

  3. Attention-head ablation, zeroing a head and measuring the resulting change in task performance, is a common method for inferring which components of a language model are causally responsible for a behavior. We show using GPT-2 small that t…

  4. Why do OpenAI's GPT-2 weights beat mine? Part five: data quality When I finished learning how to build an LLM from scratch, I was left with a mystery: my own models were not as good as OpenAI's original GPT-2 models, despite being based on…

← all threads