Facts about GPT-2
- 06
In 2019, GPT-2 demonstrated that a language model could learn to perform basic arithmetic operations on numbers within its training data despite having no explicit mathematical training objective.
- 05
Reproducing GPT-2's 1.5 billion parameter model required approximately 256 V100 GPUs and cost OpenAI roughly $50,000 in compute resources for a single training run.
- 04
GPT-2's training on Common Crawl data enabled it to generate coherent multi-paragraph text samples conditioning on just a single sentence prompt, a capability that impressed researchers in 2019.
- 03
The 117-million-parameter version of GPT-2 achieved 88.5% accuracy on the SuperGLUE benchmark without task-specific fine-tuning, demonstrating strong zero-shot performance.
- 02
When trained on 40GB of internet text, GPT-2 demonstrated emergent abilities like summarization and translation without explicit task-specific training data.
- 01
OpenAI released GPT-2 in February 2019 with 1.5 billion parameters, initially withholding the full model due to safety concerns.