Matei Zaharia

Open-source work

Matei Zaharia started the Apache Spark project during PhD work and co-started other datacenter software, including Apache Mesos and Spark Streaming. At Databricks, which Zaharia co-founded and serves as CTO, Zaharia helped develop MLflow and Delta Lake; Zaharia's homepage notes that most of Zaharia's research work is released as open source.¹

Software from Zaharia's Berkeley group listed on the homepage includes MegaBlocks, ColBERT, DSPy, LOTUS, LEANN, Continual Learning Bench and GEPA. DSPy, described in a 2023 paper with Omar Khattab and colleagues, is maintained in the MIT-licensed stanfordnlp/dspy repository.¹²³

Research

Zaharia's research covers computer systems for large-scale workloads such as AI, data analytics and cloud computing, along with security and privacy for communication and Internet queries. Representative papers include ColBERT, a passage search method based on late interaction over BERT (SIGIR 2020), the Lakehouse platform paper (CIDR 2021), and GEPA, a reflective prompt evolution method presented at ICLR 2026.¹

Teaching

At UC Berkeley, Zaharia works in the Sky Lab and has taught CS 294-162 on AI systems in Fall 2023 and Fall 2024, as well as a Fall 2023 course on responsible generative AI and decentralized intelligence. Zaharia advises PhD students and postdocs, several of them jointly with other faculty members.¹

Work and contributions

• MLflow Open-source code

  Zaharia's homepage says Zaharia helped develop MLflow at Databricks. The mlflow/mlflow repository is published under the Apache-2.0 license.¹⁴

• DSPy Open-source code

  Co-author of "DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines" (2023); Zaharia's homepage lists DSPy among Zaharia's recent projects. The DSPy repository is MIT-licensed.²¹³

• Apache Spark Open-source code

  Started the Apache Spark project during PhD work, according to Zaharia's homepage.¹

Timeline

1. 2014Received the ACM Doctoral Dissertation Award for PhD work.¹
2. Dec 2018Co-authored "Accelerating the Machine Learning Lifecycle with MLflow" in the IEEE Data Engineering Bulletin.¹
3. 2020Published ColBERT, a passage search method using contextualized late interaction, with Omar Khattab at SIGIR.¹
4. 2021Co-authored the Lakehouse paper on open platforms unifying data warehousing and advanced analytics (CIDR).¹
5. Oct 5, 2023Co-authored the DSPy paper on compiling declarative language model calls into self-improving pipelines.²
6. 2025Received the ACM Prize in Computing.¹
7. 2026Co-authored GEPA, a reflective prompt evolution method, presented as an oral at ICLR 2026.¹

Honors and recognition

• ACM Prize in Computing · Association for Computing Machinery, 2025¹
• Mark Weiser Award · ACM SIGOPS, 2023¹
• SIGMOD Systems Award (for Apache Spark) · ACM SIGMOD, 2022¹
• Sloan Research Fellowship · Alfred P. Sloan Foundation, 2022¹
• NSDI Test of Time Paper Award (for Mesos) · USENIX Symposium on Networked Systems Design and Implementation, 2021¹
• EuroSys Test of Time Paper Award (for Delay Scheduling) · EuroSys Conference, 2020¹
• Presidential Early Career Award for Scientists and Engineers (PECASE) · United States Government, 2019¹
• NSF CAREER Award · National Science Foundation, 2017¹
• ACM Doctoral Dissertation Award · Association for Computing Machinery, 2014¹

Where they work

• UC Berkeley 27 people in the wiki
• Databricks 11 people in the wiki

Related people

• Yangqing JiaAlso at UC Berkeley
• Shreya ShankarAlso at UC Berkeley
• Ali GhodsiAlso at UC Berkeley
• Hao ZhangAlso at UC Berkeley
• David PattersonAlso at UC Berkeley
• Lukas BiewaldDeveloper tools · Training systems
• Sara HookerDatasets and data · Training systems
• Steven HuelsDeveloper tools · Datasets and data
• Werner VogelsDeveloper tools · Datasets and data
• Reynold XinDatasets and data · Developer tools