Alex Graves

Research on sequence learning

Graves's publication list centres on recurrent neural networks, and in particular LSTM, applied to labelling sequences such as speech and handwriting. With Jürgen Schmidhuber and colleagues, Graves published work on bidirectional LSTM for phoneme classification in 2005 and introduced Connectionist Temporal Classification at ICML 2006, a way to train recurrent networks on sequence data that has not been segmented in advance. The list also includes multidimensional recurrent networks for offline handwriting recognition at NIPS 2008 and a 2009 IEEE TPAMI article on unconstrained handwriting recognition.¹

Graves's 2008 dissertation at Technische Universität München, "Supervised Sequence Labelling with Recurrent Neural Networks", was later extended into a 2012 Springer textbook of the same name. In 2013 Graves, Abdel-rahman Mohamed and Geoffrey Hinton reported deep LSTM networks trained end to end for speech recognition, evaluated on the TIMIT phoneme recognition benchmark.¹⁴

Generative models and adaptive computation

In "Generating Sequences With Recurrent Neural Networks" (2013), Graves showed LSTM networks generating text and online handwriting by predicting one data point at a time, and extended the approach to synthesizing cursive handwriting from a given text. In 2016 Graves introduced Adaptive Computation Time, an algorithm that lets a recurrent network learn how many computation steps to take between receiving an input and producing an output.⁵⁶

In 2023 Graves, Rupesh Kumar Srivastava, Timothy Atkinson and Faustino Gomez, writing with NNAISENSE affiliations, introduced Bayesian Flow Networks. The paper derives loss functions for continuous, discretised and discrete data and reports experiments on image modelling and character-level language modelling.⁷

Neural networks with external memory

With Greg Wayne and Ivo Danihelka, Graves wrote "Neural Turing Machines" (2014), which couples a neural network to an external memory through attention so that the whole system can be trained by gradient descent and can learn simple algorithms such as copying and sorting.²

The follow-up, "Hybrid computing using a neural network with dynamic external memory", was published in Nature on 12 October 2016 by Graves, Wayne and colleagues at Google DeepMind. It describes the differentiable neural computer, a network that reads from and writes to an external memory matrix. The paper's contributions statement records that Graves and Wayne conceived the project and that the two contributed equally.³

Work and contributions

• Differentiable neural computer Published research

  First author of "Hybrid computing using a neural network with dynamic external memory" (Nature, October 2016), whose authors are listed with Google DeepMind affiliations.³

• Neural Turing Machines Published research

  First author, with Greg Wayne and Ivo Danihelka, of the 2014 paper that couples neural networks to external memory resources through attention, giving a system that is differentiable end to end and can infer simple algorithms such as copying and sorting.²

• Connectionist Temporal Classification Published research

  First author, with Santiago Fernández, Faustino Gomez and Jürgen Schmidhuber, of the ICML 2006 paper "Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks", listed on Graves's homepage.¹

• Generating Sequences With Recurrent Neural Networks Published research

  Sole author of the 2013 paper showing how LSTM networks can generate text and online handwriting one data point at a time, extended to handwriting synthesis conditioned on a text sequence.⁵

• Bayesian Flow Networks Published research

  First author of the 2023 paper introducing Bayesian Flow Networks, a class of generative model that updates the parameters of simple distributions with Bayesian inference and passes them to a neural network, without the forward process used in diffusion models.⁷

Timeline

1. 2005Published work with Jürgen Schmidhuber on framewise phoneme classification with bidirectional LSTM networks.¹
2. 2006Presented Connectionist Temporal Classification at ICML 2006 as first author.¹
3. Jul 2008Completed the dissertation "Supervised Sequence Labelling with Recurrent Neural Networks" at Technische Universität München.¹
4. May 2009Published "A novel connectionist system for unconstrained handwriting recognition" in IEEE TPAMI as first author.¹
5. 2012Published the Springer textbook "Supervised Sequence Labelling with Recurrent Neural Networks".¹
6. Mar 22, 2013Posted "Speech Recognition with Deep Recurrent Neural Networks" with Abdel-rahman Mohamed and Geoffrey Hinton.⁴
7. Aug 4, 2013Posted "Generating Sequences With Recurrent Neural Networks" on arXiv as sole author.⁵
8. Oct 20, 2014Posted "Neural Turing Machines" with Greg Wayne and Ivo Danihelka.²
9. Mar 29, 2016Posted "Adaptive Computation Time for Recurrent Neural Networks" on arXiv as sole author.⁶
10. Oct 12, 2016Published the differentiable neural computer paper in Nature as first author with Google DeepMind colleagues.³
11. Aug 14, 2023Posted "Bayesian Flow Networks" with NNAISENSE co-authors.⁷

Documented roles

The sources do not confirm a current role; these are the roles they document.

• Author affiliated with Google DeepMind on the 2016 Nature paper on the differentiable neural computer, Google DeepMind³
• CIFAR Junior Fellow supervised by Geoffrey Hinton, University of Toronto (Department of Computer Science)¹
• Author affiliated with NNAISENSE on the 2023 paper "Bayesian Flow Networks", NNAISENSE⁷

Companies in their history

• Google DeepMind 64 people in the wiki

Related people

• Mathieu BlondelConnected to Google DeepMind
• Shane LeggConnected to Google DeepMind
• Jascha Sohl-DicksteinConnected to Google DeepMind
• Heiga ZenConnected to Google DeepMind
• Jeff CluneConnected to Google DeepMind
• Anima AnandkumarMachine-learning theory
• Sanyuan ChenSpeech and audio
• Michael I. JordanMachine-learning theory
• Edouard GraveSpeech and audio
• Ju-Chieh ChouSpeech and audio