Stephen Roller

Career

Stephen Roller's homepage divides Roller's publications into undergraduate work, a PhD period from 2010 to 2017, and a period at Meta's Fundamental AI Research team from 2017 to 2023. The PhD in Computer Science was completed at the University of Texas at Austin under the supervision of Katrin Erk, with a 2017 dissertation on identifying lexical relationships and entailments with distributional semantics.¹

After FAIR, Roller led pretraining and training infrastructure work at Character.AI and then worked as a Senior Staff Research Scientist at Google DeepMind. The homepage now lists a role as a Member of the Technical Staff at Thinky, linking to Thinking Machines Lab, and notes that Roller's pretraining work since leaving FAIR has not been published.¹

Research

Roller's doctoral research, much of it with Katrin Erk, studied distributional semantics, including hypernymy detection, lexical substitution, and lexical entailment. At FAIR, Roller continued this line with the 2018 ACL paper on Hearst patterns for hypernym detection before moving into open-domain dialogue.¹

Roller was first author of Recipes for Building an Open-Domain Chatbot, submitted in April 2020, which describes building chatbots at several model sizes and releases the models and code. Roller later co-authored the BlenderBot 3 paper in August 2022 and the Science paper on playing the game Diplomacy by combining language models with strategic reasoning.³⁴¹

On large language models, Roller was a joint first author of OPT: Open Pre-trained Transformer Language Models, submitted in May 2022, whose code is published in the MIT-licensed metaseq repository. Roller was also first author of Hash Layers for Large Sparse Models, a 2021 NeurIPS paper that replaces learned mixture-of-experts routing with token hashing.²⁵⁶¹

Work and contributions

• OPT and metaseq Open-source code

  One of three equal-contribution first authors of OPT: Open Pre-trained Transformer Language Models (2022), with Susan Zhang and Naman Goyal; the paper releases code for experimenting with the models, and the metaseq repository is published under the MIT License.²⁵¹

• Open-domain chatbot research (Recipes and BlenderBot 3) Published research

  First author of Recipes for Building an Open-Domain Chatbot (2020), which builds 90M, 2.7B, and 9.4B parameter models and makes the models and code publicly available, and co-author of BlenderBot 3 (2022), a 175B parameter dialogue model whose paper says the weights and code were released.³⁴¹

• Hash Layers for Large Sparse Models Published research

  First author of a 2021 NeurIPS paper that routes each token to a set of feedforward weights by hashing, as an alternative to learned mixture-of-experts routing.⁶¹

Timeline

1. 2010Began doctoral studies in computer science, according to Roller's publication list.¹
2. 2017Completed a PhD dissertation on lexical relationships and entailments with distributional semantics.¹
3. 2017Joined Meta's Fundamental AI Research team, where Roller worked until 2023.¹
4. Apr 28, 2020Submitted Recipes for Building an Open-Domain Chatbot as first author.³
5. Jun 8, 2021Submitted Hash Layers for Large Sparse Models, later presented at NeurIPS 2021.⁶¹
6. May 2, 2022Submitted OPT, Open Pre-trained Transformer Language Models, as a joint first author.²
7. Aug 5, 2022Co-authored the BlenderBot 3 paper on a deployed conversational agent.⁴

Other documented roles

Roles the sources document that are not confirmed as current.

• Senior Staff Research Scientist, Google DeepMind¹
• Member of the Technical Staff (pretraining and training infrastructure), Character.AI¹
• Senior Staff Research Engineer (2017–2023), Meta (Fundamental AI Research, FAIR)¹²

Honors and recognition

• Best paper award, NLP for Conversational AI Workshop (Human Evaluation of Conversations is an Open Problem) · NLP for Conversational AI Workshop¹

Where they work

• Thinking Machines Lab 13 people in the wiki

Related people

• Liliang RenAlso at Thinking Machines Lab
• Danqi ChenAlso at Thinking Machines Lab
• Myle OttAlso at Thinking Machines Lab
• Kenneth LiAlso at Thinking Machines Lab
• John SchulmanAlso at Thinking Machines Lab
• Noam ShazeerLanguage models · Training systems
• Jeffrey QuesnelleLanguage models · Training systems
• Luke ZettlemoyerLanguage models · Training systems
• Christopher RéTraining systems · Language models
• Yonghui WuLanguage models · Training systems