factjuice
Technology  /  gpt-2

Facts about GPT-2

6 facts squeezed so far
  1. 06

    In 2019, GPT-2 demonstrated that a language model could learn to perform basic arithmetic operations on numbers within its training data despite having no explicit mathematical training objective.

    GPT-2May 14capabilityemergencelanguagemodels
  2. 05

    Reproducing GPT-2's 1.5 billion parameter model required approximately 256 V100 GPUs and cost OpenAI roughly $50,000 in compute resources for a single training run.

    GPT-2May 14trainingcomputecost
  3. 04

    GPT-2's training on Common Crawl data enabled it to generate coherent multi-paragraph text samples conditioning on just a single sentence prompt, a capability that impressed researchers in 2019.

    GPT-2May 14languageaitraining
  4. 03

    The 117-million-parameter version of GPT-2 achieved 88.5% accuracy on the SuperGLUE benchmark without task-specific fine-tuning, demonstrating strong zero-shot performance.

    GPT-2May 14benchmarkmeasurementperformance
  5. 02

    When trained on 40GB of internet text, GPT-2 demonstrated emergent abilities like summarization and translation without explicit task-specific training data.

    GPT-2May 14machinelearningtrainingcapabilities
  6. 01

    OpenAI released GPT-2 in February 2019 with 1.5 billion parameters, initially withholding the full model due to safety concerns.

    GPT-2May 14technologyai2019