I'm a Senior Research Scientist at nuTonomy, where I work on the computer vision system for their fleet of autonomous vehicles. Before joining nuTonomy I was a postdoctoral scholar in Trevor Darrell's lab at Berkeley AI Research. There, I worked on automated quantification of scientific image-data using deep learning.
Previously, I was lead developer at Hövding where I created the algorithmic framework and hardware design for their invisible bicycle helmet. I have also worked on automated dietary logging systems for consumer applications and focusing algorithms for image-based cell analysis.
I also manage and develop CoralNet, deploying deep convolutional neural networks to help coral reef ecologists mine image data.
Some of my favorite previous and current research projects.
UCSD, San Diego, California
With the world's coral reef in unprecedented decline, much effort is focused on research, restoration, and conservation. Towards these ends, ecological composition across large spatial and temporal scales needs to be measured. Fortunately, progress in digital imagery and underwater robotics allow for rapid collection of large image corpora. Unfortunately, however, subsequent manual image analysis is prohibitively time consuming creating a "manual annotation bottleneck" between collected images and tabulated data.
Addressing this bottleneck was the over-arching goal of my PhD, and culminated with the creation of CoralNet. CoralNet implements the computer vision methods that I developed, and allows researchers, agencies, and private partners to processed their survey images in several automated and semi-automated annotation modes. The site also serves as a repository and collaboration platform for the scientific community. Current users include the NOAA Coral Reef Ecosystem Program, the Catlin Seaview Survey, the the Australian Institute of Marine Science, Washington State University, University of Washington, University of North Carolina, Scripps Institution of Oceanography, Colby College, and Stanford@Sea. Photo credit: Catlin Seaview Survey
UC Berkeley, California
Quantifying the content of a single image, or a set of images is important in many practical situation. Visual sampling surveys, for example, which is used throughout the natural sciences, require just this. With this in mind, there is surprisingly little work on this topic in computer vision, as it falls outside the standard tasks of classification, segmentation or detection.
I have looked at a few aspects of this problem. For example: how do you best integrate an automated annotation system when designing a random sampling survey, or what is the optimal way to quantify the class-distribution under domain-shift given a finite set of images?
UCSD, San Diego, California
Supervised machine learning algorithms are typically studied for cost-balanced binary classification, where the goal is to separate two equally important classes. However, in many practical situations, there are multiple classes, and in addition, misclassification costs may vary. This leads to the more challenging problem of cost-sensitive multiclass learning.
I worked with Steve Branson on a structured Support Vector Machine solver for cost-sensitive multiclass learning that converges orders of magnitudes faster than previous solvers. In another project I examined loss functions for cost-sensitive multiclass learning together with Mohammad Saberian, and identified a property, guess-averseness, which has strong empirical importance.
Microsoft Research, Seattle, Washington, USA
The World Health Organization (WHO) predicts that overweight and obesity may soon replace more traditional public health concerns such as under-nutrition and infectious diseases as the most significant cause of poor health. Logging food and calorie intake has been shown to facilitate weight management, but current food logging methods are time-consuming and cumbersome, which limits their effectiveness.
During an internship at Microsoft Research, I worked with Neel Joshi and Dan Morris, to develop a practical method for food-logging from images. The method utilize a data-base of menu items to estimate the nutritional content of the query image, and we demonstrate robust performance on a dataset of realistic food images.
Hövding AB, Malmö Sweden
From Hövding's website: "Hövding is a bicycle helmet unlike any other currently on the market. It's ergonomic, it's practical, it complies with all the safety requirements, and it's also subtle and blends in with what else you are wearing. Hövding is a collar for bicyclists, worn around the neck. The collar contains a folded up airbag that you'll only see if you happen to have an accident. The airbag is shaped like a hood, surrounding and protecting the bicyclist's head. The trigger mechanism is controlled by sensors which pick up the abnormal movements of a bicyclist in an accident. Hövding is a practical accessory that's easy to carry around, it's got a great-looking yet subtle design, and it will save your life."
I was fortunate to be the first employee at Hövding where my task was to develop the accident detection system from scratch. This involved everything from selecting hardware sensors, collecting train data, setting up a computational infrastructure, defining target performances, and developing the actual algorithm. I worked with a small team of engineers on this task.
Hövding has been covered in thousands of video and news-releases, including stories from NBC, WIRED, The Washington Post, and der Spiegel. Filmmaker Fredrik Gerttens short-film about Hövding have been viewed over 20 million times. Some of my other personal favorites are this story by a Swiss independent reporter, and this one by a German talk show host. Please see Hövdings press page for a complete list.
Cellavision AB, Lund Sweden
From Cellavisions website: "CellaVision develops and markets products for the health care sector, enabling fast and firm blood cell analysis and quality assurance of morphology diagnosis. The company has cutting-edge expertise within sophisticated digital image analysis, artificial intelligence and automated microscopy. For laboratories, this means increased efficiency, a simplification of the procedures and confirmed proficiency. The product line includes systems for automatic blood cell differentials and software for differential proficiency testing and education. The products are sold to hospitals and laboratories in Europe, North America and Asia."
I did my masters thesis at Cellavision, where I worked on a single-image focus level assessment method for blood cell images. This is in contrast with standard methods for image focusing that use multiple images together to determine focus level. The method I developed has an international patent and is currently deployed in Cellavision products.
Full list of publications is available at Google Scholar profile
Below are links to some data-sets that I have packaged or published
This data-set is an aggregate of two marine ecology data-sets packaged for quantification and domain adaptation experiments. One data-set is a Caribbean coral reef survey by the Catlin Seaview Survey, and one a plankton time-series from Martha's Vineyard Coastal Observatory. The two data-sets, while superficially different, pose the same question: how can we quantify the label-distribution across multiple repeated data-set shifts at the lowest possible manual annotation effort?
Pacific Labeled Corals (PLC) is an aggregate dataset containing 5090 coral reef survey images from four Pacific monitoring projects in Moorea (French Polynesia), the northern Line Islands, Nanwan Bay (Taiwan) and Heron Reef (Australia). PLC contains a total of 251,988 expert annotations across 4 pacific reef locations, and can be used as a benchmark dataset for evaluating object recognition methods and texture descriptors as well as for domain transfer learning research.