Video-based Motion Retargeting
A.A. Efros, A. Berg, G. Mori and J. Malik
This paper presents a simple technique for retargeting motion performed by a ``driver'' actor in a video sequence onto a ``target'' actor. The core requirement is a sufficiently rich set of video clips of the target actor performing a variety of actions. The algorithm is based on a three step procedure. First the motion of the driver and target actors are extracted from their backgrounds as two stabilized video layers or ``sprites''. The next step is the computation of motion similarity between frames from the driver and target sprites. This measure needs to be robust under variations in appearance (e.g. clothing, lighting, etc.) between the target and the driver. A key component of our work is such a measure which is based on robust comparison of spatio-temporal patterns of optical flow. Synthesis of the target sequence is formulated as a dynamic programming problem using a cost function that enforces motion similarity between corresponding frames of driver and target as well as temporal coherence of the synthesized sequence. We show results with two kinds of motion retargeting: using motion of another actor (``Do as I Do'' synthesis) or motion labels (``Do as I Say'' synthesis). Our technique can use video footage from either a stationary or moving camera.
The paper is in review. Please do not circulate.