Jacob Steinhardt

Research

Jacob Steinhardt's research aims to keep machine learning systems understood by and aligned with the people who use them. Steinhardt's work examines how models exploit errors in reward functions, including a 2022 study finding that more capable reinforcement learning agents often exploit misspecified rewards, and earlier work on certified defenses against data poisoning attacks.¹⁵⁶

Other lines of Steinhardt's research study models' latent activations to understand and adjust how outputs are produced, and build systems that summarize large datasets in natural language. Steinhardt co-authored the MMLU benchmark for measuring multitask language understanding and the study that traced an indirect object identification circuit in GPT-2 small.¹³⁴

Teaching and Transluce

As a graduate student, Steinhardt was advised by Percy Liang, and during a postdoc year Steinhardt worked at OpenAI and Open Philanthropy. At Berkeley Steinhardt has taught courses on robust statistics, forecasting, and data, inference and decisions.¹⁷

Steinhardt co-founded Transluce and is its CEO; Transluce is a nonprofit lab building public technology for scalable oversight of AI. Its tools include Docent, which turns AI agent transcripts into traceable measurements for safety evaluations, reinforcement learning monitoring and agent development.²⁸⁹

Work and contributions

• Measuring Massive Multitask Language Understanding (MMLU) Published research

  Last-listed author of the 2020 paper proposing a test of a text model's multitask accuracy across 57 tasks, including elementary mathematics and law.³

• Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small Published research

  Last-listed author of the 2022 paper explaining how GPT-2 small performs indirect object identification in terms of groups of attention heads.⁴

• Transluce

  Founder and CEO of Transluce, which describes itself as a non-profit research lab building the public tech stack for scalable oversight of AI.¹²

Timeline

1. Jun 2017Posted "Certified Defenses for Data Poisoning Attacks" as first author.⁶
2. 2019Taught STAT260, Robust Statistics, at UC Berkeley in fall 2019.⁷
3. Sep 2020Co-authored the MMLU paper on measuring massive multitask language understanding.³
4. Jan 2022Co-authored a study of reward misspecification and reward hacking in reinforcement learning.⁵
5. Nov 2022Co-authored the indirect object identification circuit study of GPT-2 small.⁴

Other documented roles

Roles the sources document that are not confirmed as current.

• Worked at OpenAI during a postdoc year, OpenAI¹

Where they work

• UC Berkeley 27 people in the wiki
• Transluce 1 person in the wiki

Related people

• Owain EvansAlso at UC Berkeley
• Collin BurnsAlso at UC Berkeley
• David PattersonAlso at UC Berkeley
• Andy KonwinskiAlso at UC Berkeley
• Ion StoicaAlso at UC Berkeley
• Kush R. VarshneySafety and alignment · Interpretability
• Sameer SinghInterpretability · Evaluation and benchmarks
• Nicholas CarliniSafety and alignment · Evaluation and benchmarks
• Eric Michael SmithSafety and alignment · Evaluation and benchmarks
• James GlasbrennerEvaluation and benchmarks · Safety and alignment