Dynamic Hand Gesture Recognition Using Depth Data

1. In a computing environment, a method performed at least in part on at least one processor, the method comprising: sensing depth data for a plurality of frames that include hand movement; for the plurality of frames, processing the depth data, wherein processing the depth data comprises: segmenting the depth data to isolate a hand represented in the plurality of frames; determining that a principal direction of the hand is not pointing in a predefined direction in an image plane; based on the determining, rotating the hand such that the palm of the hand is substantially parallel to the image plane and such that the principal direction of the hand is pointing toward the predefined direction in the image plane; performing normalization on the rotated hand to provide normalized hand data to compensate for scale and a relative hand size of the hand; and extracting feature values corresponding to the normalized hand data; and recognizing the hand movement as a hand gesture based upon the feature values provided to a classifier.

2. The method of claim 1 further comprising detecting the hand by: segmenting the depth data and the classified human body into a human shape; and detecting the hand based upon depth data of the hand relative to depth data of the human shape.

3. The method of claim 2 wherein detecting the hand comprises refining an object that includes an arm portion and a hand portion to remove the arm portion.

4. The method of claim 3 , wherein refining an object that includes an arm portion and a hand portion to remove the arm portion comprises: identifying a hand region; determining that the identified hand region includes a portion of an arm and a portion of the hand; locating a thinnest portion of the hand region; classifying the thinnest portion of the hand region as a wrist; and removing points beyond the wrist such that the arm portion is removed.

5. The method of claim 1 wherein extracting the feature values corresponding to the hand comprises extracting feature values based upon hand velocity data, one or more hand rotation parameters, or at least one shape descriptor, or any combination of hand velocity data, one or more hand rotation parameters, or at least one shape descriptor.

6. The method of claim 1 wherein extracting the feature values corresponding to the hand comprises extracting shape descriptor feature values based upon one or more occupancy features.

7. The method of claim 1 wherein extracting the feature values corresponding to the hand comprises extracting shape descriptor feature values based upon one or more silhouette features.

8. The method of claim 1 , wherein processing the depth data further comprises: dividing an original depth map for a frame into a plurality of blobs by connecting adjacent pixels if a difference between depth values of the pixels is less than a pre-defined threshold; determining a largest blob of the plurality of blobs; identifying blobs within a predefined distance of the largest blob; and classifying the largest blob and the blobs within the predefined distance of the largest blob as a human body.

9. One or more computer-readable storage devices having computer-executable instructions, which when executed perform operations comprising: processing sensed depth data for a plurality of frames that include hand movement, wherein processing the sensed depth data comprises: segmenting the depth data to isolate a hand represented in the frames of depth data; determining that a principal direction of the hand is not pointing in a predefined direction in an image plane; based on the determining, rotating the hand such that palm of the hand is substantially parallel to an image plane and such that the principal direction of the hand is pointing toward the predefined direction in the image plane; performing normalization on the rotated hand to provide normalized hand data to compensate for scale and a relative hand size of the hand; and extracting feature values corresponding to the normalized hand data; and recognizing the hand movement as a hand gesture based upon the feature values provided to a classifier.

10. The one or more computer-readable storage devices of claim 9 wherein extracting the feature values corresponding to the hand comprises extracting a hand velocity feature value set, a hand rotation feature value set, or a hand shape descriptor feature set.

11. The one or more computer-readable storage devices of claim 9 having further computer-executable instructions, which when executed perform operations comprising training the classifier with feature values extracted from frames of depth data that are associated with intended hand gestures, and wherein recognizing the hand movement as a hand gesture based upon the feature values comprises inputting a feature vector representative of the feature values to the classifier.

12. A system comprising: a memory; a computing device; and a processor programmed to: sense depth data for a plurality of frames that include hand movement; for the plurality of frames, process the depth data, wherein processing the depth data comprises: segmenting the depth data to isolate a hand represented in the plurality of frames; determine that a principal direction of the hand is not pointing in a predefined direction in an image plane; based on the determining, rotate the hand such that the palm of the hand is substantially parallel to the image plane and such that the principal direction of the hand is pointing toward the predefined direction in the image plane; perform normalization on the rotated hand to provide normalized hand data to compensate for scale and relative hand size of the hand; and extract feature values corresponding to the normalized hand data; and recognize the hand movement as a hand gesture based upon the feature values provided to a classifier.

13. The system of claim 12 , wherein processing the depth data further comprises: dividing an original depth map for a frame into a plurality of blobs by connecting adjacent pixels if a difference between depth values of the pixels is less than a pre-defined threshold; determining a largest blob of the plurality of blobs; identifying blobs within a predefined distance of the largest blob; and classifying the largest blob and the blobs within the predefined distance of the largest blob as a human body.

14. The system of claim 13 , wherein processing the depth data further comprises detecting the hand by: segmenting the depth data and the classified human body into a human shape; and detecting the hand based upon depth data of the hand relative to depth data of the human shape.

15. The system of claim 14 , wherein detecting the hand comprises refining an object that includes an arm portion and a hand portion to remove the arm portion, and wherein refining an object that includes an arm portion and a hand portion to remove the arm portion comprises: identifying a hand region; determining that the identified hand region includes a portion of an arm and a portion of the hand; locating a thinnest portion of the hand region; classifying the thinnest portion of the hand region as a wrist; and removing points beyond the wrist such that the arm portion is removed.

16. The system of claim 12 , wherein processing the depth data further comprises detecting the hand by: determining a plurality of hypothesized hand regions; and determining a hand region from among the hypothesized hand regions based upon processing one or more previous frames of depth data.

17. The system of claim 12 , wherein extracting the feature values corresponding to the hand comprises extracting feature values based upon hand velocity data, one or more hand rotation parameters, or at least one shape descriptor, or any combination of hand velocity data, one or more hand rotation parameters, or at least one shape descriptor.

18. The system of claim 12 , wherein extracting the feature values corresponding to the hand comprises extracting shape descriptor feature values based upon one or more occupancy features.

19. The system of claim 12 , wherein extracting the feature values corresponding to the hand comprises extracting shape descriptor feature values based upon one or more silhouette features.

20. The system of claim 12 , wherein processing the depth data further comprises detecting the hand by: identifying a hand region; determining that the identified hand region includes a portion of an arm and a portion of the hand; locating a thinnest portion of the hand region; classifying the thinnest portion of the hand region as a wrist; and removing points beyond the wrist such that the arm portion is removed.