Why didn't we get GPT-2 in 2005? (dynomight.net via hn) 6w
event
GPT-2 Performance Analysis
-
Why didn't we get GPT-2 in 2005? The ancient Romans were never great at building ships and never tried to explore the Atlantic.
-
Should low-precision transformer inference use stochastic rounding (SR) or round-to-nearest (RN)? The answer depends on where in the network you look.
-
Attention-head ablation, zeroing a head and measuring the resulting change in task performance, is a common method for inferring which components of a language model are causally responsible for a behavior. We show using GPT-2 small that t…
-
Why do OpenAI's GPT-2 weights beat mine? Part five: data quality (www.gilesthomas.com via hn) openai 6w
Why do OpenAI's GPT-2 weights beat mine? Part five: data quality When I finished learning how to build an LLM from scratch, I was left with a mystery: my own models were not as good as OpenAI's original GPT-2 models, despite being based on…