Aran Komatsuzaki

Career

Komatsuzaki's own page describes work on generative AI since 2017, with a focus on Transformer decoders, and a PhD in machine learning from Georgia Tech completed in 2023. In 2022 Komatsuzaki was an intern at Google Research, where the sparse upcycling work was carried out.¹³

Open models and datasets

GPT-J-6B, a six-billion-parameter JAX-based Transformer language model, was released in June 2021 with Ben Wang. The release post credits Komatsuzaki with proposing the project, designing its high-level plan and configurations, writing the article, and advising Wang, who wrote the code and ran the experiments; the model was built with assistance from EleutherAI.⁴²

Komatsuzaki describes co-leading the development of the LAION image-text datasets and is a co-author of the LAION-400M paper (2021), which released an open dataset of CLIP-filtered image-text pairs.¹⁵

Research

In "One Epoch Is All You Need" (2019), Komatsuzaki proposed training language models on a larger dataset for only one or a few epochs instead of many passes over a smaller one, adjusting regularization and model size to the compute budget. The 2022 sparse upcycling paper, with Google Research co-authors, initializes a sparsely activated mixture-of-experts model from a dense pretrained checkpoint to add capacity with modest extra training.⁶³¹

Work and contributions

• GPT-J-6B Open weights

  Named with Ben Wang as an author in the repository's citation entry for the GPT-J-6B weights, which the README says are licensed under Apache 2.0, and thanked there for advice on experiment design and for writing the blog posts.²¹

• Sparse Upcycling Published research

  First author of "Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints" (2022), which reuses a dense checkpoint to initialize a sparsely activated mixture-of-experts model.³¹

Timeline

1. Jun 16, 2019Released "One Epoch Is All You Need", on training language models for one or a few epochs on more data.⁶
2. Jun 4, 2021Released GPT-J-6B, a 6B-parameter JAX-based Transformer language model, with Ben Wang.⁴²
3. Nov 3, 2021Co-authored the LAION-400M paper on an open dataset of CLIP-filtered image-text pairs.⁵
4. 2022Interned at Google Research, where Komatsuzaki co-proposed sparse upcycling.¹
5. Dec 9, 2022Released the Sparse Upcycling paper as first author.³
6. 2023Completed a PhD in machine learning at Georgia Tech.¹

Documented roles

The sources do not confirm a current role; these are the roles they document.

• Intern (2022), Google Research¹

Companies in their history

• Google 121 people in the wiki

Related people

• John X. MorrisConnected to Google
• George HalalConnected to Google
• Sridhar RamaswamyConnected to Google
• Ashish VaswaniConnected to Google
• Sara HookerConnected to Google
• Christopher RéLanguage models · Datasets and data
• Rangan MajumderLanguage models · Datasets and data
• Loubna Ben AllalLanguage models · Datasets and data
• Sriram RaghavanLanguage models · Datasets and data
• Zach NussbaumLanguage models · Datasets and data