Bio

I am currently a postdoctoral researcher at RISE Lab, UC Berkeley, working with Prof. Ion Stoica.

I am recently working on building end-to-end composable and automated systems for large-scale distributed DL. The most recent projects is Alpa that automates model-parallel training on large-scale distributed GPU/TPU clusters.

I study large-scale distributed ML in the joint context of ML and systems, concerning both performance and usability. My work spans across parallel ML programmability, representations of parallelisms, performance optimizations, system architectures, auto-parallelization techniques, and AutoML, with applications in computer vision, natural language processing, and healthcare.

I completed my Ph.D. at the School of Computer Science, Carnegie Mellon University. My advisor was Prof. Eric Xing. My Ph.D. thesis is Machine Learning Parallelism Could Be Adaptive, Composable and Automated. Several of works including Poseidon, Cavs, GeePS, and Alpa are parts of the CASL project and the Ray project, and now being commercialized at multiple start-ups including Petuum and AnyScale.

News:

• Alpa paper accepted to OSDI'22.
• Alpa code is open sourced. Check the project out and watch my introductory talk for Alpa.
• I am on the academic job market for 2021 - 2022! Here is my application package.
• I organized the RISE Camp 2021. Check the talks and tutorials.
• I gave a half-day tutorial at KDD 2021: Simple and Automated Distributed Machine Learning on Ray.
• I gave a half-day tutorial at AAAI 2021: Simplifying and Automating Parallel Machine Learning via a Programmable and Composable Parallel ML System.

Publications

[ Selected / ML systems / Scalable ML models&algorithms / AutoML / Large ML applications / All by date ]

Software

• Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning
• AutoDist: A Composable and Automated Synchronization System for Distributed Deep Learning
• AdaptDL: Resource-adaptive Cluster Scheduler for Deep Learning Training
• Cavs: An Efficient Programming Interface for Dynamic Neural Networks
• Poseidon v2.0: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusters
• DyNet: The Dynamic Neural Network Toolkit
• GeePS: a GPU-specialized Parameter Server
• Poseidon v1.0: Distributed Deep Learning on CPU/GPU Clusters

Teaching

• TA of 10-708 Probabilistic Graphical Models, Spring 2019, Carnegie Mellon University
• TA of 16-791 Applied Data Science, Spring 2019, Carnegie Mellon University
• TA of 10-701 Introduction to Machine Learning, Fall 2015, Carnegie Mellon University.
• TA of Artificial Intelligence, Spring 2012, Shanghai Jiao Tong University.