Razvan Pascanu

Career

Pascanu earned an MSc at Jacobs University in 2009, supervised by Herbert Jaeger, and a PhD at the University of Montreal in 2014, supervised by Yoshua Bengio. Pascanu was also involved in developing the Theano library, helped write its deep learning tutorials and co-authored a 2012 paper on its new features. Pascanu is now a Research Scientist at DeepMind and an Affiliate Member of Mila, and has co-supervised students jointly with researchers at Johannes Kepler University, Imperial College London and University College London.¹²

Research

Pascanu's early work dealt with optimization and recurrent networks. Besides the 2012 analysis of exploding and vanishing gradients in RNNs, Pascanu wrote "Revisiting Natural Gradient for Deep Networks" (2013) with Yoshua Bengio and co-authored "On the Number of Linear Regions of Deep Neural Networks" (2014), on how depth affects the functions a network can represent.³⁴⁵¹

Later research has addressed learning across many tasks, including Progressive Neural Networks (2016) and "Overcoming catastrophic forgetting in neural networks" (2016), as well as graph neural networks through the 2018 paper "Relational inductive biases, deep learning, and graph networks". Returning to recurrent models, Pascanu co-authored "Resurrecting Recurrent Neural Networks for Long Sequences" (2023).¹⁶⁷⁸⁹

Teaching and community

Pascanu is one of the organizers of EEML, the Eastern European Machine Learning Summer School, and of the AIRomania community, through which Pascanu has organized RomanianAIDays since 2020 and helped build an AI course for high school students. Pascanu has also helped organize CoLLAs, a conference on lifelong learning, and the Learning on Graphs (LoG) conference.^1101112

Work and contributions

• On the difficulty of training Recurrent Neural Networks Published research

  First author, with Tomas Mikolov and Yoshua Bengio, of the 2012 paper proposing norm-based gradient clipping for exploding gradients and a soft constraint for vanishing gradients.³

• Theano Open-source code

  Involved in developing Theano and in writing some of its deep learning tutorials; co-author of "Theano: new features and speed improvements" (2012).¹²¹³

Timeline

1. 2009Completed an MSc at Jacobs University under Herbert Jaeger.¹
2. Nov 2012Posted "On the difficulty of training Recurrent Neural Networks" as first author with Tomas Mikolov and Yoshua Bengio.³
3. Nov 2012Co-authored "Theano - new features and speed improvements".²
4. Jan 2013Posted "Revisiting Natural Gradient for Deep Networks" with Yoshua Bengio.⁴
5. 2014Completed a PhD at the University of Montreal under Yoshua Bengio.¹
6. Jun 2016Co-authored Progressive Neural Networks, on transferring knowledge across a sequence of tasks.⁶
7. Dec 2016Co-authored "Overcoming catastrophic forgetting in neural networks".⁷
8. Jun 2018Co-authored "Relational inductive biases, deep learning, and graph networks".⁸
9. 2020Began organizing RomanianAIDays as part of the AIRomania community.¹
10. Mar 2023Co-authored "Resurrecting Recurrent Neural Networks for Long Sequences".⁹

Where they work

• Google DeepMind 64 people in the wiki
• Mila 2 people in the wiki

Related people

• Mathieu BlondelAlso at Google DeepMind
• Roy FrostigAlso at Google DeepMind
• Laurent SartranAlso at Google DeepMind
• Jascha Sohl-DicksteinAlso at Google DeepMind
• Alex GravesAlso at Google DeepMind
• Alex SmolaMachine-learning theory · ML frameworks and libraries
• Martín AbadiML frameworks and libraries · Machine-learning theory
• Fabian PedregosaML frameworks and libraries · Machine-learning theory
• Lorenzo StellaML frameworks and libraries · Machine-learning theory
• Joshua LochnerML frameworks and libraries