Tuomas Haarnoja

Research

Haarnoja's doctoral work at Berkeley centred on maximum entropy reinforcement learning for robots. A 2017 paper with Haoran Tang, Pieter Abbeel and Sergey Levine introduced soft Q-learning, which learns energy-based policies for continuous states and actions. The 2018 Soft Actor-Critic paper, presented at ICML 2018, followed with an off-policy actor-critic algorithm built on the same framework.¹⁴²

A follow-up report, "Soft Actor-Critic Algorithms and Applications" (December 2018), described refinements to the algorithm and its use on real robots. A companion paper, "Learning to Walk via Deep Reinforcement Learning", presented at RSS 2019, applied the method to learning robot locomotion with little manual tuning.⁵⁶

Robot soccer at Google DeepMind

At Google DeepMind, Haarnoja was first author of a 2023 study that used deep reinforcement learning to train a miniature humanoid robot with 20 actuated joints to play one-versus-one soccer, with skills such as fall recovery, turning and kicking. Haarnoja is also a co-author of the 2024 follow-up, "Learning Robot Soccer from Egocentric Vision with Deep Reinforcement Learning", which trained soccer policies using onboard cameras and computation.³⁷

Open-source work

The Softlearning repository from Berkeley's RAIL lab is a reinforcement learning framework for training maximum entropy policies and includes the official implementation of Soft Actor-Critic. Its README lists Haarnoja as an author of the soft Q-learning, Soft Actor-Critic and related papers the code accompanies.⁸

Work and contributions

• Soft Actor-Critic (SAC) Published research

  First author of "Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor" (2018).²

• Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning Published research

  First author of the 2023 paper, with Google DeepMind as the listed affiliation, using deep reinforcement learning to train a low-cost miniature humanoid robot to play a simplified one-versus-one soccer game.³

Timeline

1. Feb 2017Posted "Reinforcement Learning with Deep Energy-Based Policies", introducing soft Q-learning.⁴
2. Jan 2018Posted the Soft Actor-Critic paper as first author, later presented at ICML 2018.²
3. Dec 2018Completed a Berkeley PhD thesis on maximum entropy deep reinforcement learning for robot skills.¹
4. Dec 2018Posted "Soft Actor-Critic Algorithms and Applications" and "Learning to Walk via Deep Reinforcement Learning".⁵⁶
5. Apr 2023Posted "Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning" with Google DeepMind colleagues.³
6. May 2024Co-authored "Learning Robot Soccer from Egocentric Vision with Deep Reinforcement Learning".⁷

Documented roles

The sources do not confirm a current role; these are the roles they document.

• Research Scientist, DeepMind (Google DeepMind)¹³

Companies in their history

• Google DeepMind 64 people in the wiki

Related people

• Justin FuConnected to Google DeepMind
• Misha DenilConnected to Google DeepMind
• Adam GleaveConnected to Google DeepMind
• Volodymyr MnihConnected to Google DeepMind
• Hado van HasseltConnected to Google DeepMind
• Sergey LevineRobotics · Reinforcement learning
• Bob McGrewRobotics · Reinforcement learning
• Vikash KumarRobotics · Reinforcement learning
• Pieter AbbeelRobotics · Reinforcement learning
• Deepak PathakRobotics · Reinforcement learning