Systems and Methods for Object Tracking Based on User Refinement Input

1. A method implemented in a media editing device for tracking an object in a plurality of frames, comprising: obtaining a contour of an object in a frame; generating a local region list for storing one or more of: local regions added to the object contour and local regions removed from the object contour; for each of the remaining frames of the plurality of frames, performing the steps of: updating the local region list; based on the content of a current frame, the content and the obtained contour of a prior frame, and the local regions in the local region list, generating, by a probability map generator, a plurality of probability maps containing probability values for pixels in the current frame, wherein a probability value of each pixel in a first probability map corresponds to a likelihood of the pixel being located within the object, and wherein a probability value of each pixel in a second probability map corresponds to a likelihood of the pixel being located at a boundary of the object; estimating, for the current frame, a contour of the object based on the plurality of probability maps; determining whether user input for refining the estimated contour is received; in response to receiving user input, identifying, based on the user input, one of: at least one local region added to the estimated contour, at least one local region removed from the estimated contour, or any combination thereof; for the current frame, recording the identified local regions in the local region list; and setting the obtained contour of the current frame to one of: the user refined contour or the estimated contour.

2. The method of claim 1 , wherein the contour is obtained from a user via an object selection component in a user interface displayed to the user.

3. The method of claim 1 , wherein the first probability map comprises a color model map, and wherein the second probability map comprises an edge map.

4. The method of claim 1 , wherein generating, by the probability map generator, the plurality of probability maps containing probability values for pixels in the current frame comprises: constructing a foreground color model according to a region inside the contour in a prior frame; and constructing a background color model according to a region outside the contour in the same frame or a different prior frame.

5. The method of claim 4 , further comprising estimating the probability values in the first probability map according to similarities of pixel values in a current frame with respect to the foreground and background models.

6. The method of claim 4 , further comprising estimating the probability values in the second probability map according to differences between pixel values and neighboring pixels, wherein larger probability values are assigned to pixels with relatively large difference values.

7. The method of claim 1 , wherein estimating, for the current frame, the contour of the object based on the plurality of probability maps comprises: selecting a contour as the estimated contour based on a highest contour score, wherein each contour score is calculated according to at least one of the following: a total length of a contour boundary, wherein a higher contour score is assigned in response to a total length of the contour boundary being shorter; the probability values of the pixels in the frame within the contour representing a likelihood of being located within the object, wherein a higher contour score is assigned in response to the probability values being higher; and the probability values of the pixels in the frame on the contour representing a likelihood of being located at the boundary of the object, wherein a higher contour score is assigned in response to the probability values being higher.

8. The method of claim 1 , wherein obtaining user input refining the estimated contour comprises receiving a modification of the estimated contour from a user via an object selection component on a user interface.

9. The method of claim 1 , wherein identifying one of: at least one local region added to the estimated contour, at least one local region removed from the estimated contour, or any combination thereof comprises: determining at least one difference between the estimated contour and the refined estimated contour, wherein the at least one local region comprises the at least one difference.

10. The method of claim 1 , wherein generating, by the probability map generator, the plurality of probability maps containing probability values for pixels in the current frame comprises: configuring the probability map generator to generate higher probability values in the first probability map corresponding to the pixels within the added at least one local region; and configuring the probability map generator to generate lower probability values in the first probability map corresponding to the pixels within the removed at least one local region.

11. The method of claim 1 , wherein generating, by the probability map generator, the plurality of probability maps containing probability values for pixels in the current frame comprises: configuring the probability map generator to generate higher probability values in the second probability map corresponding to the pixels on a boundary of the added at least one local region or on a boundary of the removed at least one local region.

12. The method of claim 1 , wherein recording the identified local regions in the local region list comprises: for each identified local region, recording a contour and content of the local region in the local region list.

13. The method of claim 12 , wherein updating the local region list based on content of the current frame comprises: applying motion estimation to the current frame and a previous frame; for each local region in the local region list, performing the steps of: adjusting a contour of the local region in the current frame based on the applied motion estimation; comparing the content of the local region recorded in the local region list with a content of the local region in the current frame; and removing the recorded local region from the local region list based on the comparison.

14. The method of claim 13 , wherein the content of the recorded local region comprise pixels within the contour of the recorded local region, and wherein comparing the content of the recorded local region in the local region list with a content of the local region in the current frame is performed based on a sum of absolute difference between the pixels in the recorded local region and the pixels within the current contour of the local region in the current frame.

15. The method of claim 14 , wherein the local region is removed if the computed sum of absolute difference is greater than a threshold.

16. The method of claim 1 , further comprising: selecting a test frame from the plurality of frames; selecting one of the frames with the obtained contour as a base frame; generating a reference contour according to the test frame, the base frame, and the obtained contour of the base frame; estimating, for the test frame, a contour of the object; based on the reference contour and the estimated contour, identifying one of: at least one local region included in the reference contour and not included in the estimated contour, at least one local region included in the estimated contour and not included in the reference contour, or any combination thereof; and based on the identification of the at least one local region included in the reference contour and not included in the estimated contour, the at least one local region included in the estimated contour and not included in the reference contour, or any combination thereof, recording the local regions in the local region list, wherein a local region included in the reference contour is recorded as an added local region, and a local region not included in the reference contour is recorded as a removed local region.

17. The method of claim 16 , wherein generating the reference contour according to the test frame, the base frame, and the obtained contour of the base frame comprises: applying motion estimation to the test frame and the base frame; generating a reference contour by modifying the obtained contour based on the motion estimation.

18. The method of claim 16 , wherein a same frame is selected as both the base frame and the test frame, and wherein the reference contour is identical to the obtained contour of the frame.

19. A system for tracking an object in a plurality of frames, comprising: a computing device including a processing device; an application executable in the computing device for processing the plurality of frames, the application comprising: an object selector for obtaining a contour of an object in a frame; a local region analyzer for generating a local region list for storing one or more of: local regions added to the object contour and local regions removed from the object contour, wherein the local region analyzer is further configured to update the local region list for each of the remaining frames of the plurality of frames; a probability map generator for generating, for each of the remaining frames of the plurality of frames, a plurality of probability maps containing probability values for pixels in the current frame based on the content of a current frame, the content and the obtained contour of a prior frame, and the local regions in the local region list, wherein a probability value of each pixel in a first probability map corresponds to a likelihood of the pixel being located within the object, and wherein a probability value of each pixel in a second probability map corresponds to a likelihood of the pixel being located at a boundary of the object; a contour estimator for estimating, for the current frame, a contour of the object based on the plurality of probability maps; a refinement module for determining whether user input for refining the estimated contour is received; wherein the local region analyzer is further configured to identify, in response to receiving user input, one of: at least one local region added to the estimated contour, at least one local region removed from the estimated contour, or any combination thereof based on the user input; wherein the local region analyzer is further configured to record, for the current frame, the identified local regions in the local region list; and wherein the contour estimator is further configured to set the obtained contour of the current frame to one of: the user refined contour or the estimated contour.

20. The system of claim 19 , wherein the probability map generator generates the plurality of probability maps containing probability values for pixels in the current frame by constructing a foreground color model according to a region inside the contour in a prior frame; and constructing a background color model according to a region outside the contour in the same frame or a different prior frame.

21. The system of claim 20 , wherein the probability map generator estimates the probability values in the first probability map according to similarities of pixel values in a current frame with respect to the foreground and background models.

22. The system of claim 20 , wherein the probability map generator estimates the probability values in the second probability map according to differences between pixel values and neighboring pixels, wherein larger probability values are assigned to pixels with relatively large difference values.

23. The system of claim 19 , wherein the contour estimator estimates, for the current frame, the contour of the object based on the plurality of probability maps by selecting a contour as the estimated contour based on a highest contour score, wherein each contour score is calculated according to at least one of the following: a total length of a contour boundary, wherein a higher contour score is assigned in response to a total length of the contour boundary being shorter; the probability values of the pixels in the frame within the contour representing a likelihood of being located within the object, wherein a higher contour score is assigned in response to the probability values being higher; and the probability values of the pixels in the frame on the contour representing a likelihood of being located at the boundary of the object, wherein a higher contour score is assigned in response to the probability values being higher.

24. The system of claim 19 , wherein the local region analyzer identifies one of: at least one local region added to the estimated contour, at least one local region removed from the estimated contour, or any combination thereof by determining at least one difference between the estimated contour and the refined estimated contour, wherein the at least one local region comprises the at least one difference.

25. A non-transitory computer-readable medium embodying a program executable in a computing device for tracking an object in a plurality of frames, comprising: code that obtains a contour of an object in a frame; code that generates a local region list for storing one or more of: local regions added to the object contour and local regions removed from the object contour; code that updates the local region list for each of the remaining frames of the plurality of frames; code that generates, based on the content of a current frame, the content and the obtained contour of a prior frame, and the local regions in the local region list, a plurality of probability maps containing probability values for pixels in the current frame for each of the remaining frames of the plurality of frames, wherein a probability value of each pixel in a first probability map corresponds to a likelihood of the pixel being located within the object, and wherein a probability value of each pixel in a second probability map corresponds to a likelihood of the pixel being located at a boundary of the object; code that estimates, for the current frame, a contour of the object based on the plurality of probability maps for each of the remaining frames of the plurality of frames; code that determines whether user input for refining the estimated contour is received for each of the remaining frames of the plurality of frames; code that identifies, in response to receiving user input and based on the user input, one of: at least one local region added to the estimated contour, at least one local region removed from the estimated contour, or any combination thereof for each of the remaining frames of the plurality of frames; code that records, for the current frame, the identified local regions in the local region list for each of the remaining frames of the plurality of frames; and code that sets the obtained contour of the current frame to one of: the user refined contour or the estimated contour for each of the remaining frames of the plurality of frames.

26. The non-transitory computer-readable medium of claim 25 , wherein the code that identifies one of: at least one local region added to the estimated contour, at least one local region removed from the estimated contour, or any combination thereof by determining at least one difference between the estimated contour and the refined estimated contour, wherein the at least one local region comprises the at least one difference.