System and Method for Performing Motion Capture and Image Reconstruction

1. A computer-implemented system for performing motion capture of a subject comprising: a plurality of cameras for capturing a sequence of image frames of the subject over a period of time, each frame having a plurality of vertices defining a captured surface of the subject; a computing system for processing the sequence of image frames, the computing system having a memory for storing program code and a processor for processing the program code to perform the operations of: establishing a reference frame having one or more of the plurality of vertices and specifying a location for each of the vertices; performing frame-to-frame tracking to identify locations of vertices within an N′th frame based on locations of vertices within an (N−1)′th frame or an earlier frame; performing reference-to-frame tracking to identify locations of vertices within the N′th frame based on the locations of vertices in the reference frame to counter potential drift between the frames; storing the locations of vertices for use in subsequent reconstruction of the motion of the subject; and performing the frame-to-frame and reference-to-frame tracking again using a different set of parameters, the parameters defining a search area for the vertices of each frame wherein multiple correlation passes are performed with the different sets of parameters; and wherein for passes after the first, the search area is shrunk by using an estimate of the position of a vertex based on the position of nearby vertices that were successfully tracked in the previous passes.

2. The system as in claim 1 wherein a camera closest to a normal of the surface on which each vertex is located is selected to perform the frame-to-frame and reference-to-frame tracking.

3. The system as in claim 1 wherein the frame-to-frame tracking is performed using a relatively larger window for matching vertices and the reference-to-frame tracking is performed using a relatively smaller window for matching vertices.

4. The system as in claim 1 further comprising: estimating the location of vertices not found in each frame N based on known locations of neighboring vertices.

5. The system as in claim 1 wherein the computing system includes additional program code executed by the processor to perform the additional operations of: correlating lines or random patterns captured by one of the plurality of cameras with images from other of the plurality of cameras.

6. The system as in claim 5 further comprising: generating range information for each vertex based on the correlation.

7. The system as in claim 6 wherein range information from multiple cameras is combined by performing the operations of: treating a 3-dimensional (3D) capture volume of the subject as a scalar field; using a marching cubes or marching tetrahedrons process to locate an isosurface of the scalar field and create a polygon mesh representing the surface of the subject; and removing false surfaces.

8. The system as in claim 7 wherein a scalar value of each point in the 3D capture volume is computed based on a weighted sum of scalar values from each of the plurality of cameras.

9. The system as in claim 8 wherein a first weighting component is associated with each camera based on an angle at which the camera is pointed relative to the isosurface normal.

10. The system as in claim 9 wherein a second weighting component is a function of distance of the vertex from the isosurface.

11. The system as in claim 1 wherein the subject is a performer and wherein a random pattern of material is applied to regions of the performer's face to create the vertices to be tracked.

12. The system as in claim 11 wherein the material is phosphorescent paint.

13. The system as in claim 11 further comprising: a light source to be strobed on and off in response to control signals from the computer system, the light source charging the random pattern when on; and wherein shutters of the plurality of cameras are strobed synchronously with the strobing of the light source to capture sequences of images of the random pattern (“glow frames”) as the performer moves or changes facial expressions during a performance, wherein the shutters of the plurality of cameras are open when the light source is off and the shutters are closed when the light source is on.

14. The system as in claim 13 further comprising: a second plurality of cameras having shutters strobed synchronously with the strobing of the light source to capture images of the performer (“lit frames”), wherein the shutters of the second plurality of cameras are open when the light source is on and the shutters of the second plurality of cameras are closed when the light source is off.

15. The system as in claim 14 wherein the first plurality of cameras are grayscale cameras and the second plurality of cameras are color cameras.

16. The system as in claim 13 wherein the light source comprises a light emitting diode (LED) array.

17. The system as in claim 13 wherein strobing the shutters comprises opening the shutters for a first period of time and closing the shutters for a second period of time, the second period of time being of a different duration than the first period of time.

18. The system as in claim 17 wherein the first period of time is longer than the second period of time.

19. The system as in claim 14 wherein the lit frames and glow frames are separated to generate two separate sets of image data.

20. The system as in claim 14 wherein cameras capturing the lit frames have a sensitivity which is different from cameras capturing the glow frames.

21. The system as in claim 13 wherein the shutters are opened for a first period of time when the light source is on and for a second period of time when the light source is off, wherein the first and second periods of time are unequal.

22. The system as in claim 11 wherein applying the random pattern comprises: applying phosphorescent material to a sponge; and applying the sponge upon the performer's face.

23. The system as in claim 11 wherein applying the random pattern comprises: spraying the random pattern on the performer's face with an airbrush.

24. The system as in claim 11 wherein applying the random pattern comprises: applying paint to the performer's face through a stencil.

25. The system as in claim 11 wherein the material is phosphorescent paint and wherein applying the random pattern comprises flicking a wire brush containing the phosphorescent paint such that droplets of phosphorescent paint are splattered onto the performer's face.

26. The system as in claim 1 wherein the estimate comprises a least squares estimate.