Leo Gao

Open-source work

Gao led The Pile, an English text corpus assembled from 22 diverse subsets for training large language models and published with an EleutherAI affiliation at the end of 2020. Gao is also the first-listed author in the citation entry for EleutherAI's LM Evaluation Harness, a framework for few-shot evaluation of language models.¹²³

Gao co-authored the 2022 GPT-NeoX-20B paper describing EleutherAI's open-source autoregressive language model; its contributions appendix lists Gao, alongside other authors, under evaluations, tokenizer experiments, and alignment implications.⁴

Research

Gao is first author of "Scaling Laws for Reward Model Overoptimization" (2022), written with John Schulman and Jacob Hilton under an OpenAI affiliation, which studies how optimizing against a learned reward model in reinforcement learning from human feedback diverges from the true objective.⁵

Gao's later interpretability papers, also listed with OpenAI, include "Scaling and evaluating sparse autoencoders" (2024), on which Gao was the primary contributor, and "Weight-sparse transformers have interpretable circuits" (2025), which trains transformers with sparse weights so that the circuits behind their behavior are easier to study.⁶⁷

Work and contributions

• The Pile Published research

  First author of The Pile paper (2020), which describes an 825 GiB English text corpus built from 22 diverse subsets for training large-scale language models.¹²

• LM Evaluation Harness Open-source code

  First-listed author in the project's citation entry for the evaluation framework.³

• GPT-NeoX-20B Published research

  Co-author of the GPT-NeoX-20B paper; its contributions appendix credits Gao with evaluations, tokenizer experiments, and the alignment-implications section.⁴

Timeline

1. Dec 31, 2020Released The Pile, an 800GB dataset of diverse text for language modeling, as first author.¹
2. Apr 14, 2022Co-authored the GPT-NeoX-20B paper on EleutherAI's open-source language model.⁴
3. Oct 19, 2022Released "Scaling Laws for Reward Model Overoptimization" as first author with an OpenAI affiliation.⁵
4. Jun 6, 2024Released "Scaling and evaluating sparse autoencoders" as first author and primary contributor.⁶
5. Nov 17, 2025Released "Weight-sparse transformers have interpretable circuits" as first author.⁷

Documented roles

The sources do not confirm a current role; these are the roles they document.

• Lead author of The Pile (2020), published with an EleutherAI affiliation, EleutherAI²

Companies in their history

• EleutherAI 5 people in the wiki

Related people

• Stella BidermanConnected to EleutherAI
• Connor LeahyConnected to EleutherAI
• Sid BlackConnected to EleutherAI
• Katherine CrowsonConnected to EleutherAI
• Adina WilliamsEvaluation and benchmarks · Language models
• Rohan TaoriLanguage models · Evaluation and benchmarks
• Yacine JerniteDatasets and data · Language models
• Jesse DodgeEvaluation and benchmarks · Datasets and data
• Eugene YanLanguage models · Evaluation and benchmarks
• Ruchir PuriLanguage models · Datasets and data