Jeremy Howard

Career

Howard's site lists early work as an analytical specialist at McKinsey & Co and in a machine learning and data analytics practice at A.T. Kearney, followed by the startups FastMail and Optimal Decisions Group, where Howard developed profit-optimized pricing for insurance. Howard was later the founding President and Chief Scientist of Kaggle, the data science competition community, and founded Enlitic, a company applying deep learning to medicine.¹

Howard is a founding researcher at fast.ai and an honorary professor at the University of Queensland. In December 2023 Howard, as founding CEO, and Eric Ries, as founding director, launched Answer.AI, an R&D lab that builds practical end-user products from foundational research.¹⁴²

Research

With Sebastian Ruder, Howard wrote "Universal Language Model Fine-tuning for Text Classification" (2018), which introduced ULMFiT, a method for pretraining a language model and then fine-tuning it for text classification tasks. The fastai library's design is described in "fastai: A Layered API for Deep Learning" (2020), written with Sylvain Gugger.⁵⁶

At Answer.AI, Howard took part in the FSDP/QLoRA project, released in March 2024, which combines fully sharded data parallel training with quantized low-rank adaptation so that a 70-billion-parameter model can be fine-tuned on two 24 GB consumer GPUs.³⁷⁸

Teaching and writing

Howard created course.fast.ai, the free Practical Deep Learning for Coders course for people with some coding experience, whose second part covers topics from matrix multiplication and backpropagation to Stable Diffusion and latent diffusion. The book "Deep Learning for Coders with Fastai and PyTorch: AI Applications Without a PhD", listed on Howard's site among Howard's works, accompanies the course and is written entirely as Jupyter notebooks that can also be read free online.⁹¹⁰¹

Howard has also created open-source tools beyond fastai, including nbdev, which the fastai library itself uses for its tests and development workflow, and fastpages, a blogging platform.¹¹¹

Work and contributions

• FSDP/QLoRA

  Answer.AI's announcement credits Howard with discussing the idea of combining FSDP and QLoRA in late 2023 and with exploring and documenting the issues that occurred when the two were combined. The Apache-2.0 repository documents fine-tuning Llama 2 70B on two 24 GB GPUs, with 4-bit quantization through bitsandbytes or HQQ.³⁷⁸

• Hugging Face Transformers in FSDP/QLoRA

  According to the announcement, Howard studied how Meta's Llama-Recipes worked with bitsandbytes and Hugging Face's PEFT, Transformers, and Accelerate libraries, then built a minimal standalone script covering every step needed to fine-tune a model. The project's installation steps use the Transformers library.³⁷

Timeline

1. Jan 18, 2018Released the ULMFiT paper with Sebastian Ruder on fine-tuning language models for text classification.⁵
2. Feb 11, 2020Released "fastai - A Layered API for Deep Learning" with Sylvain Gugger.⁶
3. Dec 12, 2023Launched Answer.AI as its founding CEO, with Eric Ries as founding director.⁴
4. Mar 6, 2024Answer.AI released FSDP/QLoRA for fine-tuning 70B-parameter models on two consumer GPUs.³

Where they work

• Answer.AI 1 person in the wiki
• fast.ai 2 people in the wiki

Related people

• Andrew NgTeaching and writing · Training systems
• Mu LiTeaching and writing · Training systems
• Sylvain GuggerTraining systems · Teaching and writing
• Emily WebberTraining systems · Teaching and writing
• Christopher RéTraining systems
• Furu WeiTraining systems
• Duen Horng "Polo" ChauTeaching and writing
• Tomasz TunguzTeaching and writing
• Ce ZhangTraining systems
• Jeff DelaneyTeaching and writing