CS 294 Spring 2003 Project Topic Suggestions
============================================

I can supply references and/or code for all these topics; also, more
references can be found easily on the web.


Applications
------------

Disk spin-down (when to turn off the disk on a laptop) and related
problems in power management for mobile systems.

Solve a more realistic version of Randlov and Alstrom's bicycle problem.

Make a (simulated) noisy three-link arm throw a dart accurately.

Make a (simulated) human figure walk stably

Solve larger and more realistic versions of the vacuum world.

Simulated car racing - RARS (rars.sourceforge.net)

RoboCup Simulated soccer (sserver.sourceforge.net)

Discrete games: backgammon, othello, chess, go, blackjack, poker

Continuous games: golf, croquet, tennis

Elevator scheduling (possibly in Soda Hall ...)

Solve Minesweeper for as large a board as you can

Various applications of RL to operations research problems, e.g.,
inventory control, equipment maintenance, equipment resale


Theoretical topics, survey projects
-----------------------------------

Continuous-state and continuous-time problems:
  what problems are exactly solvable in closed form?
  what problems are exactly solvable by iteration?
  what problems are approximately solvable by iteration+projection?

Exploration:
  what exploration problems are exactly solvable, under what assumptions?

Reinforcement learning with instance-based function approximators

Applications of RL to model biological learning

Inverse reinforcement learning - find  R given \pi

Can anything be said about RL algorithms when the environment is not
Markov in s?

What would be reasonable optimality criteria for nonstationary MDPs?

Convergence proof for RL methods when R is partially observable (as in
the elevator scheduling problem)