Angelos Katharopoulos

Research

Angelos Katharopoulos's research interest, as stated on the homepage, is training deep neural networks with less compute, less memory and fewer samples. Katharopoulos completed a PhD at the Idiap Research Institute and EPFL under the supervision of François Fleuret, and earlier worked with Christos Diou and Anastasios Delopoulos while studying electrical engineering.¹

With Fleuret, Katharopoulos published ICML papers on importance sampling for deep learning (2018) and on attention sampling for processing megapixel images (2019). The 2020 ICML paper "Transformers are RNNs" introduced linear attention, showing that autoregressive transformers can run as recurrent networks, and a NeurIPS 2020 paper with Apoorv Vyas and Fleuret proposed clustered attention. Later co-authored work includes Neural Parts (CVPR 2021) on 3D shape abstractions and a CVPR 2023 study of self supervision in large-scale image-text training.¹⁶

Open-source work

The MLX README credits Katharopoulos, with Awni Hannun, Jagrit Digani and Ronan Collobert, with the initial development of the MLX software suite, and its 2023 citation entry lists the four as authors. Katharopoulos's commits to MLX in 2026 include work on JACCL, the distributed backend that the MLX documentation describes for low-latency communication using RDMA over Thunderbolt.³⁴⁵

Katharopoulos's homepage lists further open-source projects that Katharopoulos started or contributes to: Fast Transformers for PyTorch, the simple-3dviz visualization library, attention-sampling and Keras importance-sampling libraries, the LDA++ topic-modeling library, and NlpTools, a PHP natural language processing library.¹

Work and contributions

• MLX

  Credited in the README, together with Awni Hannun, Jagrit Digani, and Ronan Collobert, with the initial development of the MLX software suite, and listed among the authors in its citation entry. Katharopoulos continues to author commits to the repository, including to the JACCL distributed backend.³⁴⁵

• Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention Published research

  First author of the ICML 2020 paper that expresses self-attention as a linear dot product of kernel feature maps, reducing its cost from quadratic to linear in sequence length.⁶¹

• Fast Transformers Open-source code

  Listed on Katharopoulos's homepage as a PyTorch library of efficient transformer implementations focused on efficient self-attention.¹

Timeline

1. Oct 2016Presented fsLDA, a fast supervised LDA method for video datasets, at ACM Multimedia 2016.¹
2. 2018Published "Not All Samples Are Created Equal" on importance sampling for deep learning at ICML 2018.¹
3. 2019Published the attention-sampling paper on processing megapixel images at ICML 2019.¹
4. 2020Co-authored "Fast Transformers with Clustered Attention" at NeurIPS 2020.¹
5. Jun 29, 2020Posted "Transformers are RNNs," the linear attention paper later presented at ICML 2020.⁶
6. 2021Co-authored Neural Parts, on 3D shape abstractions with invertible neural networks, at CVPR 2021.¹
7. 2023MLX, initially developed by Katharopoulos with Awni Hannun, Jagrit Digani and Ronan Collobert, carries a 2023 citation entry.³

Where they work

• Apple 27 people in the wiki

Related people

• Samy BengioAlso at Apple
• John GiannandreaAlso at Apple
• Chris LattnerAlso at Apple
• Carlos GuestrinAlso at Apple
• Jonathan CohenAlso at Apple
• Tong HeML frameworks and libraries
• Jane XuML frameworks and libraries
• Sayak PaulML frameworks and libraries
• Lewis TunstallML frameworks and libraries
• Pieter NoordhuisML frameworks and libraries