I build tools to help people understand, debug, and interpret ML models by combining traditional software engineering and explainable AI techniques. My work employs design and systems-building methodologies at the intersection of HCI and ML research. My previous projects have fused sensing, ML, and novel AR interfaces to guide users through physical tasks.
Umlaut supports users in debugging neural networks by running heuristic checks on the ML program's structure and model behavior during runtime. Umlaut's checking infrastructure can be added to a Keras runtime with minimal changes to the source program (2 lines of code). Raised errors are presented in an interface which tightly couples tutorial-based explanations, visualizations of model output, context from program runtime, and links back to source code.
IMACS is a method that combines gradient-based model attributions with aggregation and visualization techniques to summarize differences in attributions between two image classification models. IMACS extracts influential image regions from an evaluation dataset, clusters them based on similarity, then visualizes differences in model attributions for clusters of visually similar regions. This approach can be used to discover the behavioral differences between models as well as uncover the influential features and concepts for a given class in a single model.
HindSight increases the environmental awareness of cyclists by warning them of vehicles approaching from outside their visual field. A panoramic camera mounted on a bicycle helmet streams real-time, 360-degree video to a laptop running YOLOv2, a neural object detector designed for real-time use. Detected vehicles are passed through a filter bank to find the most relevant. Resulting vehicles are sonified using bone conduction headphones, giving cyclists added margin to react.
The goal of Drill Sergeant is to reduce the barrier of entry for using power tools to create physical artifacts. The project consisted of an ecosystem of augmented power tools (a smart drill, miter saw, CNC router, and measuring tape) which use sensors and embedded displays provide real-time, corrective feedback and coach users through completing assemblies. A server keeps state and coordinates instructions across tools, recalculating the assembly from real-world measurements.
Eldon Schoop, Xin Zhou, Gang Li, Zhourong Chen, Björn Hartmann, Yang Li. Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysis. To appear in CHI 2022. ArXiv preprint.
Eldon Schoop, Ben Wedin, Andrei Kapishnikov, Tolga Bolukbasi, Michael Terry. IMACS: Image Model Attribution Comparison Summaries. ArXiv preprint.
Forrest Huang, Eldon Schoop, David Ha, Jeffrey Nichols, John Canny. Sketch-Based Creativity Support Tools Using Deep Learning. Book chapter in Artificial Intelligence for Human Computer Interaction: A Modern Approach (2021).
Eldon Schoop, Forrest Huang, Björn Hartmann. UMLAUT: Debugging Deep Learning Programs using Program Structure and Model Behavior CHI 2021.
Eldon Schoop, Forrest Huang, Björn Hartmann. SCRAM: Simple Checks for Realtime Analysis of Model Training for Non-Expert ML Programmers. ICML 2020 Workshop on Human in the Loop Learning.
Eldon Schoop, Forrest Huang, Björn Hartmann. SCRAM: Simple Checks for Realtime Analysis of Model Training for Non-Expert ML Programmers. CHI 2020 Extended Abstracts.
Forrest Huang, Eldon Schoop, David Ha, and John Canny. Scones: Towards Conversational Authoring of Sketches. IUI 2020.
Eldon Schoop, Forrest Huang, Nathan Khuu, Björn Hartmann. MakerLens: What Sign-In, Reservation and Training Data Can (and Cannot) Tell You About Your Makerspace ISAM 2018.
Eldon Schoop, James Smith, Björn Hartmann. HindSight: Enhancing Spatial Awareness by Sonifying Detected Objects in Real-Time 360-Degree Video CHI 2018.
Eldon Schoop, James Smith, Björn Hartmann. HindSight: Enhancing Spatial Awareness by Sonifying Detected Objects in Real-Time 360-Degree Video 2018 Berkeley EECS Tech. Report. Masters Thesis.
Eldon Schoop, Chris Myers, Björn Hartmann. MakerPass: A Multiplatform Access Control System for Academic Makerspaces. ISAM 2017 Posters.
Michelle Nguyen and Eldon Schoop, Mitchell Karchemsky, Valkyrie Savage, Björn Hartmann, Sean Follmer. Drill Sergeant: Supporting Physical Construction Projects through an Ecosystem of Augmented Tools. 2016 Berkeley EECS Tech. Report.
Eldon Schoop and Michelle Nguyen, Daniel Lim, Valkyrie Savage, Sean Follmer, Björn Hartmann. Drill Sergeant: Supporting Physical Construction Projects through an Ecosystem of Augmented Tools. CHI 2016 Extended Abstracts. Best Paper, late-breaking work.
Thinking Through Art and Design@Berkeley: Responsible Design from Bits to Buildings focuses on design and its connection to the arts across a range of disciplinary contexts. This course explores the role of design in imagining the built environment, in framing our experience of new media and technology, as well as design’s intersections with allied creative fields. Throughout, the class considers how design offers a toolkit for framing and approaching complex problems and how responsible design emerges through new forms of collaboration with a broad field of experts—including users, artists, engineers, scientists, and others.
Global Product Development is a project-based course that teaches students how to translate a functional prototype into a commercial, consumer-ready product. This course provides an overview of product commercialization, design for manufacturing, supply chain and logistics, intellectual property, regulatory certification, and more. The class features several guest lecturers with real-world experience, in-class critique, lectures on product development, and a one-week field trip to Hong Kong to explore the hardware startup ecosystem.
Interactive Device Design teaches concepts and skills required to design, prototype, and fabricate interactive devices—physical objects that intelligently respond to user input and enable new types of interactions. The first half of the semester is dedicated to a survey of relevant techniques in 3D modeling and fabrication; electronics and circuit board design; sensing and actuation for interaction; embedded software development, wired and wireless communication with mobile devices, computers, and networks; and user interface programming. In the second half of the semester, students propose and carry out a significant design project. Images above showcase selected student work.
CS 160 is an introduction to Human-Computer Interaction (HCI). In this class, students learn to prototype, evaluate, and design user interfaces. In contrast to most classes at Berkeley, CS 160 does not (only) focus on particular algorithmic techniques or technologies. Students develop a broad set of skills needed for user-centered design. These skills include ideation, needs assessment, communication, rapid prototyping, implementation and evaluation.
The MakerPass system originally started as my undergraduate design project to create an access control system for the shops at Jacobs Hall. Today, MakerPass is responsible for managing user access for many student shops across campus. The system consists of card readers which light up and emit audio when scanned, desktop software that restricts login, and an admin portal for managing users and their permissions.