Bio

I am currently a postdoctoral researcher at RISE Lab, UC Berkeley, working with Prof. Ion Stoica.

I am recently working on the Alpa project that automates model-parallel training and serving of large models like GPT-3.

I study large-scale distributed ML in the joint context of ML and systems, concerning both performance and usability. My work spans across parallel ML programmability, representations of parallelisms, performance optimizations, system architectures, auto-parallelization techniques, and AutoML, with applications in computer vision, natural language processing, and healthcare.

I completed my Ph.D. at the School of Computer Science, Carnegie Mellon University. My advisor was Prof. Eric Xing. My Ph.D. thesis is Machine Learning Parallelism Could Be Adaptive, Composable and Automated. Several of works including Poseidon, Cavs, GeePS, and Alpa are parts of the CASL project and the Ray project, and now being commercialized at multiple start-ups including Petuum and AnyScale.

News:

• 🌟 We recently launched an open sourced version of GPT-3 text prompting service. Try it and share your results!
• 🌟 I gave a tutorial at ICML 2022. Big Model Tutorial: Techniques and Systems to Train and Serve Bigger Models.
• Alpa paper accepted to OSDI'22.
• Alpa code is open sourced. Check the project out and watch my introductory talk for Alpa.

Publications

[ Selected / ML systems / Scalable ML models&algorithms / AutoML / Large ML applications / All by date ]

Software

• Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning
• AutoDist: A Composable and Automated Synchronization System for Distributed Deep Learning
• AdaptDL: Resource-adaptive Cluster Scheduler for Deep Learning Training
• Cavs: An Efficient Programming Interface for Dynamic Neural Networks
• Poseidon v2.0: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusters
• DyNet: The Dynamic Neural Network Toolkit
• GeePS: a GPU-specialized Parameter Server
• Poseidon v1.0: Distributed Deep Learning on CPU/GPU Clusters

Teaching

• TA of 10-708 Probabilistic Graphical Models, Spring 2019, Carnegie Mellon University
• TA of 16-791 Applied Data Science, Spring 2019, Carnegie Mellon University
• TA of 10-701 Introduction to Machine Learning, Fall 2015, Carnegie Mellon University.
• TA of Artificial Intelligence, Spring 2012, Shanghai Jiao Tong University.