Project Goals
Our goal is to build next-generation intelligent tutoring systems utilizing deep learning approaches.
Our project focuses on explaining problems to students via a dialogue setting in which interactions are grounded in an image. Ultimately, the domains our project will apply to include foreign language learning, math, and science questions which have an accompanying image or diagram. As a starting point, we began with the foreign language domain.
Current Work
Colored Shapes
We first collect a dataset of Amazon Mechanical Turk Crowdworkers role-playing the tutor and student learning the name of a colored shape. We have Turkers reply with the next utterance in a conversation as well as a classification of the action taken at that utterance. For each conversation, we collect three tutoring responses.
An example of this domain can be seen below:
Once data is collected, we experiment with language generation models to play the role of the tutor in a dialogue.
Prepositional Phrases
We have also extended our project the language learning domain of prepositional phrases involving colored objects. This domain includes more complex vocabulary and grammar rules. We collect a dataset utilizing Amazon Mechanical Turk of Turkers role-playing students and tutors.
Utilizing our resulting dataset, we are examining many deep learning approaches to generating the next tutoring utterance.
Publications and Presentations
A preliminary version of this work was presented at the AWS re:Invent conference in October 2018. Slides for this presentation can be viewed here.
A publication for this project is under review.
People
Kimberly Kao
Kevin Lu
Kiran Girish
Avni Prasad
Project Support
This research is supported in part by an AWS / Amazon ML grant, an Allen Institute for Artificial Intelligence grant, and a UC Berkeley Chancellor's fellowship.