CS 289A: Machine Learning
(Spring 2019)
Project
20% of final grade.
The project should be done in teams of 2–3 students.
Please find a partner.
Teaching Assistants Faraz Tavakoli
ft@berkeley.edu,
Panna Felsen panna@eecs.berkeley.edu, and
Carlos Florensa florensacc@berkeley.edu
are in charge of project supervision.
Please discuss your ideas with at least one of them
before submitting your initial proposal.
Deliverables
- Initial proposal, due Monday, April 8
- Project video (maximum 3 minutes), due Friday, May 10,
50% of score
- Final report (maximum 8 pages), due Friday, May 10,
50% of score
Overview
The project theme may be anything related to machine learning techniques
discussed in class, including
- critically revisiting a published paper (including reproducing
the experiments, generating new graphs and visual representations, and
discussing the results);
- writing a literature review in a specific domain
(e.g., adversarial examples, transfer learning, active learning,
meta-learning) and making a critical comparison
(ideally on a standard benchmark dataset);
- conducting original theoretical research (e.g., attack one of the COLT
open problems:
2014,
2015);
- conducting original practical research by applying
machine learning methods to a public or private dataset
(see project ideas below).
You are welcome and encouraged to design a project that
is related to your research outside this course.
However, please be honorable and don't suggest a project that
you've already implemented as part of your research.
Initial
proposal
The initial proposal is primarily a proposal, and need not be long.
Write a few paragraphs describing what you have decided to do.
You may have any number of figures and references.
- Include background information: What is the application domain or
field of research? Why is the problem important?
What specific questions will you try to answer?
- Talk about what data sources you plan to use, including
the number(s) of samples and number(s) of features.
If it's visual data, include an illustration if possible.
- Explain what methods you are planning to use and why.
- If you have done preliminary work, explain what you have already done
(e.g., downloaded and played with data, tried k-nearest neighbors,
did a mock-up of the user interface, etc.).
- What will be the core of the work for your project?
(That is, how do you expect to spend most of your time?)
Do you expect to be judged primarily on your writing, your programming,
your exhaustive exploration of methods and data, something else, or
some combination thereof?
(You are welcome to use any and all libraries and codes written by
others, but you should be clear on what your substantial new
contribution will be.)
- The initial proposal is not graded. Its purpose is to make sure
you start early and we give you feedback on your idea.
- Please submit the initial proposal (like the final) through Gradescope.
Video
- The video should be clear and understandable, describing everything
you think is important about your project
(motivation, description, techniques, results, etc.).
- The video needs to be self-contained: any CS 289A student should
be able to understand what you did (at least at a high level)
without consulting any other materials.
- You can make the video as simple as slides with a voice overlay, or
as fancy as you want.
- As long as it is clear and understandable, you will not be graded on
the fanciness of the video. Content is what will matter.
(Fanciness might be fun, though.)
- You must upload the video on YouTube and provide us with the link.
You may choose to keep the video private (i.e., only those with
the link can view it), in which case only the instructors will view it.
You can make the video public if you want to.
- Important:
The video can be at most 3 minutes long.
This is a very strict requirement;
a video of length 3 minutes and 1 second does not count.
The length is counted as whatever YouTube says it is.
In case you are worried about how you will fit an entire class project
into three minutes, take a look at
these
videos, which fit an entire Ph.D. thesis into three minutes.
Final
report
- We encourage you to use a template from
your favorite machine learning conference (e.g.,
NIPS
or ICML).
- There is no minimum length requirement. The maximum length is 8 pages.
- The submission must be made through Gradescope.
Any one person from the group can submit the proposal.
Please include the full names, student IDs, and email addresses for
all the members of the group.
- Also, for inspiration, here are some of
the final projects from
a Neural Networks class at Stanford.
It is a truth universally acknowledged that
your work will be better than Stanford's.
Grading
criteria
The video and the final report will be graded with 5 criteria.
- Relevance: should be related to machine learning techniques.
- Usefulness: should answer good questions or
solve problems worth solving.
(The questions should be clearly stated in the proposal.)
- Soundness: choose data sets with enough examples to get
statistically significant results;
conduct sound numerical experiments
(split the data into training/validation/test sets);
make comparative result tables using validation or cross-validation;
use the test set only for final assessment;
include error bars if appropriate;
add graphs and other good means of visualization
(e.g., projections onto principal components);
provide sound proofs;
if you choose a literature review, mention the most important papers in
the area and give proper credit.
- Clarity/presentation: good paper organization, good bibliography,
enough graphs and visual support, length should not exceed 8 pages.
- Novelty/originality: we do not require novelty/originality, but
it could add a few points.
Project
ideas
The ideas in this list fall mainly under the fourth category,
practical research.
If you prefer to revisit an important paper, simply pick a paper.
If you prefer to conduct a literature review,
simply pick a machine learning topic that interests you.
If you prefer to conduct theoretical research, you'd better already know
what you're doing.
- Research and implement an attack method to generate adversarial examples for neural networks used for classification. Can you develop a novel attack that notably reduces the accuracy of neural networks trained to be robust? If you're interested in this project, contact Marc Khoury, khoury@berkeley.edu, for starter code and advice.
- Applications of machine learning to 3D computer vision and computer graphics have been abundant in the last few years. One interesting problem is point set classification. That is, given as a training set a set of points in 3D representing some scene, and a label for each point, which labels the object the point is a part of, can we train a classifier to classify the objects in a new point set on a per-point basis? See this paper. This project will investigate if we can achieve better results using a much simpler approach, specifically the approach described in this paper. If you're interested in this project, contact Marc Khoury khoury@berkeley.edu for starter code and advice.
- Training neural networks to be robust to adversarial examples is a difficult problem with few empirically successful approaches. This project will attempt to combine neural network and nearest neighbor classifiers. We will use neural networks to learn an embedding into a metric space in which nearest neighbor classifiers achieve high classification accuracy. Since nearest neighbors is not differentiable, it cannot be backpropagated through, and so traditional gradient attacks cannot be used to attack the model. We will investigate the robustness of the resulting model by applying decision based attacks, as well as developing new attacks to target nearest neighbor classifiers. If you're interested in this project, contact Marc Khoury khoury@berkeley.edu for starter code and advice.
- The method of integrated gradients is an attribution technique that attempts to attribute some fraction of the decision made by a neural network to individual features of the input. Implement integrated gradients and conduct an evaluation of adversarially perturbed images from MNIST on both naturally trained and robustly trained models. If you're interested in this project, contact Marc Khoury khoury@berkeley.edu for starter code and advice.
- Businesses love to promote themselves on Yelp, leaving false positive reviews. With this dataset, predict which reviews are likely to have been written by the business itself.
- Browsers typically download files to the Downloads folder (or another fixed, set folder). Develop a method for automatically placing files in the appropriate folder. You can constrain yourself to text documents, or to images.
- Image captioning using RNN/LSTM is also an important topic. Try to generate meaningful text from images/videos. For example, take a look at the COCO Captioning Challenge.
- Can you train a fine-grained image classifier? There are many specialized datasets, like this one for dog breeds. This can be applied with the project above or in isolation. Architectures? See this paper.
- Reinforcement Learning tackles sequential decision-making problems where the only information about the system is received through interaction. A great resource to understand RL algorithms is Spinning Up. Pick your favorite algorithm from there and apply it to more complex simulated environment (e.g., MuJoCo is free for students).
- If you want harder problems, involving various aspects of ML, you can also check this: OpenAI Requests for Research
- Fake news 1. Can you train a system to decide if two articles are related and agree? The Fake News Challenge gives access to a dataset for this. You can propose your own solution and see if you get close to the winners!
- Fake news 2. Can you classify articles by bias and factuality? There is a dataset (and SVM algorithm) built with this purpose. Their code is also available, therefore a substantial innovation should be attempted.
- Can you fool a classifier with a real object? There are works that make traffic signs classification systems (trained on the LISA dataset) predict that a stop sign is a 45mph speed limit sign. Or that a 3D printed turtle is a rifle.
- Can you visualize the features learned by a Deep Neural Network? When training Deep Neural Networks, the hidden state at each layer can be understood as features or representations useful to perform the desired task. One such tool is guided backpropagation, and more recently other dataset-wide visualizations have been proposed. See this blog post.
- Dependently typed programming languages, such as Idris, provide safety guarantees that are not available in the more mainstream languages. Can any of these features be used to improve the data analytic stack, or developer ergonomics?
Other useful data sources: