David Harwath

Career

Harwath spent nearly a decade at MIT's Computer Science and Artificial Intelligence Laboratory, first as a graduate research assistant from 2011 and then as a research scientist from March 2018 to August 2020. Along the way, Harwath interned with the Human Language Technologies Group at MIT Lincoln Laboratory and with the speech team at Google.⁴

Harwath completed a PhD in computer science at MIT in 2018 in the Spoken Language Systems Group, advised by James Glass. Harwath joined the computer science faculty at The University of Texas at Austin as an assistant professor in August 2020 and leads the Speech, Audio, and Language Technologies (SALT) Lab there.¹⁴

Research

Harwath's research centers on automatic speech recognition, spoken language understanding, and multimodal machine learning across speech, environmental sound, and vision. The goal, as the UT Austin faculty page puts it, is speech and language models that are robust, flexible, and able to learn from several input modalities at once.²

Much of Harwath's MIT work studied visually grounded speech, in which models learn words and sub-word units by pairing spoken captions with images, as in papers at NeurIPS 2016, ACL 2017 and ECCV 2018. At UT Austin, Harwath's group has extended this line toward speech language models, including VoiceCraft, a neural codec language model for speech editing and zero-shot text-to-speech presented at ACL 2024.¹⁴⁵

Teaching

At UT Austin, Harwath has taught the graduate course Spoken Language Technologies (CS395T/EE381V) and an undergraduate Introduction to Speech and Audio Processing. At MIT, Harwath was a teaching assistant for courses on automatic speech recognition and introductory machine learning.¹⁴

Work and contributions

• Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech Published research

  First author, with Wei-Ning Hsu and James Glass, of this ICLR 2020 paper, which UT Austin's faculty page lists among Harwath's selected publications.³²

• Spoken caption corpora Open data

  Harwath's homepage links to a site for downloading all of the spoken caption corpora Harwath collected, along with modeling code.¹

• VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild Published research

  Senior author, with first author Puyuan Peng and colleagues, of this ACL 2024 paper introducing a token-infilling neural codec language model; the authors released data, code and model weights.⁵

Timeline

1. Aug 2011Began graduate research at MIT CSAIL in the Spoken Language Systems Group.⁴
2. 2016Presented Unsupervised Learning of Spoken Language with Visual Context at NeurIPS 2016, with Antonio Torralba and James Glass.¹
3. 2018Completed a PhD in computer science at MIT, advised by James Glass.⁴
4. 2018Presented Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input at ECCV 2018.⁴
5. Mar 2018Became a research scientist at MIT CSAIL.⁴
6. Nov 21, 2019Posted Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech, later published at ICLR 2020.³
7. Aug 2020Joined The University of Texas at Austin as an assistant professor of computer science.⁴
8. May 2022Began a visiting faculty researcher appointment at Google.⁴
9. Mar 25, 2024Co-authored VoiceCraft, a neural codec language model for speech editing and text-to-speech, presented at ACL 2024.⁵

Other documented roles

Roles the sources document that are not confirmed as current.

• Research scientist (2018–2020), MIT CSAIL¹

Honors and recognition

• NSF CAREER Award · National Science Foundation, 2023⁴
• George M. Sprowls Award for doctoral thesis in computer science · Massachusetts Institute of Technology, 2018²⁴

Where they work

• The University of Texas at Austin 2 people in the wiki

Related people

• James E. DavisAlso at The University of Texas at Austin
• Alec RadfordMultimodal models · Speech and audio
• Xu TanSpeech and audio · Multimodal models
• Alexandre DéfossezSpeech and audio · Multimodal models
• Wei-Ning HsuSpeech and audio · Multimodal models
• Alexei BaevskiSpeech and audio · Multimodal models
• Jong Wook KimSpeech and audio · Multimodal models
• Edouard GraveSpeech and audio
• Michael AuliSpeech and audio
• Saining XieMultimodal models