Horace He

PyTorch and compilers

Horace He works at the meeting point of machine learning and compilers. A resume on Horace He's site records a 2019 internship on the PyTorch TorchScript/JIT team at Facebook, and Horace He later worked on PyTorch and torch.compile at Meta, including the GPT-Fast example of PyTorch-native transformer text generation.²⁵⁶

Horace He co-authored the ASPLOS 2024 PyTorch 2 paper on TorchDynamo and TorchInductor and the 2024 FlexAttention paper on generating attention kernels. PyTorch's governance page lists Horace He as a core maintainer and as an emeritus maintainer of TorchInductor and the PT2 dispatcher.³⁷¹

Research and writing

Horace He's earlier research includes "Combining Label Propagation and Simple Models Out-performs Graph Neural Networks" (ICLR 2021) and "Better Set Representations for Relational Reasoning" (NeurIPS 2020), both as co-first author, and a reinforcement learning approach to optimizing deep learning kernels presented at the NeurIPS Workshop on ML for Systems.⁸⁹

Horace He's essays include "The State of Machine Learning Frameworks in 2019" and "Making Deep Learning Go Brrrr From First Principles" (2022). In September 2025 Thinking Machines Lab published "Defeating Nondeterminism in LLM Inference" by Horace He in collaboration with others at the lab, explaining batch-invariant kernels for reproducible inference.¹⁰⁴

Work and contributions

• PyTorch (torch.compile) Open-source code

  Listed as a PyTorch core maintainer and an emeritus maintainer of TorchInductor and the PT2 dispatcher; co-author of the PyTorch 2 paper introducing TorchDynamo and TorchInductor, which implement torch.compile. The repository is under a BSD-style license.¹³¹¹

• Defeating Nondeterminism in LLM Inference Published research

  Credited author, with Thinking Machines Lab, of this Connectionism post, which examines why LLM inference results are not reproducible in practice and describes batch-invariant kernels for RMSNorm, matrix multiplication, and attention.⁴

• GPT-Fast Open-source code

  Horace He's homepage cites GPT-Fast as an example of Horace He's work on PyTorch and torch.compile; the BSD-3-Clause repository provides simple, PyTorch-native transformer text generation.²⁶

• FlexAttention Published research

  Co-author of "Flex Attention: A Programming Model for Generating Optimized Attention Kernels" (2024), which describes generating fused attention kernels for many attention variants from short PyTorch functions.⁷

Timeline

1. Mar 2017Became a core maintainer and developer of the VSCodeVim extension.⁵
2. Jun 2019Began an internship on the PyTorch TorchScript/JIT team at Facebook.⁵
3. Oct 27, 2020Released "Combining Label Propagation and Simple Models Out-performs Graph Neural Networks" (ICLR 2021).⁹
4. Mar 15, 2022Published the essay "Making Deep Learning Go Brrrr From First Principles".¹⁰
5. Apr 2024Co-authored the PyTorch 2 paper on TorchDynamo and TorchInductor at ASPLOS 2024.³
6. Dec 7, 2024Co-authored the FlexAttention paper on generating optimized attention kernels.⁷
7. Sep 10, 2025Thinking Machines Lab published "Defeating Nondeterminism in LLM Inference" by Horace He.⁴

Other documented roles

Roles the sources document that are not confirmed as current.

• Credited "in collaboration with others at Thinking Machines" on a September 2025 research post, Thinking Machines Lab⁴
• Worked on PyTorch and torch.compile, Meta²³

Where they work

• PyTorch 11 people in the wiki

Related people

• Joe SpisakAlso at PyTorch
• Greg ChananAlso at PyTorch
• Jane XuAlso at PyTorch
• Mikayla GawareckiAlso at PyTorch
• Piotr BialeckiAlso at PyTorch
• Lin QiaoInference and serving · ML frameworks and libraries
• Ian BuckML frameworks and libraries · Inference and serving
• Mark SaroufimML frameworks and libraries · Inference and serving
• Chris LattnerML frameworks and libraries · Inference and serving
• Dmytro DzhulgakovInference and serving · ML frameworks and libraries