Ali Ghodsi

Career

Ghodsi is a co-founder of Databricks. Before becoming chief executive in January 2016, Ghodsi served as the company's VP of Engineering and Product Management. Databricks' speaker biography also lists Ghodsi as an adjunct professor at UC Berkeley and a board member of the university's RiseLab, and notes a PhD in distributed computing from KTH Royal Institute of Technology in 2006.¹

In July 2023 Databricks announced it had completed its acquisition of MosaicML; Ghodsi, quoted as co-founder and CEO, described it as a way for customers to train and deploy their own language models on their data. Ghodsi is among the bylined authors of the March 2024 post introducing DBRX, an open mixture-of-experts language model, and of the June 2024 post announcing the open-sourcing of Unity Catalog.⁵³⁴⁶

Research

Databricks' biography notes that ideas from Ghodsi's academic research on resource management, scheduling and data caching have been applied in Apache Mesos and Apache Hadoop. At NSDI 2011 Ghodsi co-authored the Mesos paper on fine-grained resource sharing and was first author of "Dominant Resource Fairness", which addresses the fair allocation of multiple resource types in a shared cluster.¹²⁷

Ghodsi is described as one of the creators of the open-source Apache Spark project, which is developed under the Apache-2.0 license. With Michael Armbrust, Reynold Xin and Matei Zaharia, Ghodsi co-wrote the CIDR 2021 paper "Lakehouse", which argues for data platforms built on open direct-access file formats with built-in support for machine learning and data science.¹⁸⁹

Work and contributions

• Apache Spark Open-source code

  Databricks' speaker biography names Ghodsi as one of the creators of the open-source Apache Spark project. The apache/spark repository is published under the Apache-2.0 license.¹⁸

• Unity Catalog Open-source code

  Co-author, with Matei Zaharia, Reynold Xin, Arsalan Tavakoli-Shiraji and Patrick Wendell, of the June 2024 Databricks post announcing that Unity Catalog was being open-sourced under the Apache 2.0 license and hosted at LF AI & Data. The unitycatalog/unitycatalog repository is Apache-2.0.⁴⁶

• Mesos Published research

  Co-author of "Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center" (NSDI 2011), whose eight authors are all listed at the University of California, Berkeley.²

• DBRX

  One of six bylined authors of Databricks' March 2024 post announcing DBRX, a mixture-of-experts language model built on the MegaBlocks research and open-source project.³

Timeline

1. 2011Co-authored the Mesos paper on fine-grained resource sharing in the data center at NSDI 2011.²
2. 2011Presented "Dominant Resource Fairness" as first author at NSDI 2011.⁷
3. Jan 2016Became chief executive of Databricks after serving as VP of Engineering and Product Management.¹
4. 2021Co-authored the CIDR 2021 paper describing the Lakehouse architecture for open data platforms.⁹
5. Jul 19, 2023Databricks announced the completion of its acquisition of MosaicML, with Ghodsi quoted as co-founder and CEO.⁵
6. Mar 27, 2024Co-authored the Databricks post announcing the open DBRX language model.³
7. Jun 13, 2024Co-authored the Databricks post announcing the open-sourcing of Unity Catalog.⁴

Where they work

• Databricks 11 people in the wiki
• UC Berkeley 27 people in the wiki

Related people

• Jonathan FrankleAlso at Databricks
• Andy KonwinskiAlso at Databricks
• Nikhil ThoratAlso at Databricks
• Arsalan Tavakoli-ShirajiAlso at Databricks
• Patrick WendellAlso at Databricks
• Pradeep DasigiLanguage models · Datasets and data
• Kezhi KongLanguage models · Datasets and data
• Rangan MajumderLanguage models · Datasets and data
• Vaishaal ShankarDatasets and data · Language models
• Guilherme PenedoDatasets and data · Language models