Optimization for Modern Data Analysis

Benjamin Recht and Stephen J. Wright

This book explores theory and algorithms for nonlinear optimization with a particular focus on problems that arise in machine learning and data analysis. The text balances worst-case analysis against implementation concerns and aims to highlight the core theoretical tools that provide reasonable guidelines to optimization practice.

This book covers material appropriate for a quarter-length course on optimization aimed at graduate students in computer science, industrial engineering, electrical engineering, and related fields.

This online textbook is an incomplete work in progress. We welcome your feedback to improve the existing material and shape the chapters to come.


  1. Introduction

  2. Foundations

  3. Elementary Descent Methods

  4. Gradient Methods Using Momentum

  5. Stochastic Gradient Methods

  6. Coordinate Descent Methods

  7. First-Order Methods for Constrained Optimization

  8. Nonsmooth Functions and Subgradients

  9. Nonsmooth Optimization Methods

Contact us

We welcome your feedback. For all questions and comments, please send an email to opt4mlbook@gmail.com.


To cite this book, please use this bibtex entry:

  title = {Optimization for Modern Data Analysis},
  author = {Benjamin Recht and Stephen J.~Wright},
  note = {Preprint available at \url{http://eecs.berkeley.edu/~brecht/opt4mlbook}},
  year = {2019}