System and Method for Audio-Visual Speech Recognition

1. A method of performing speech recognition at a distance comprising: obtaining audio information from a plurality of microphones positioned at a distance from a speaker; obtaining visual information from an image capture device; pre-processing the visual information, wherein pre-processing comprises: using a recurrent deep neural network model to identify relevant image features by classifying a pixel in a first frame of the visual information using features from a pixel in the same location from a prior frame of the visual information; identifying a region of interest in the visual information; and using the region of interest, aligning the visual information with the acoustic information and classifying individual frames in the visual information into context-dependent phonetic states; combining the audio information and visual information within a single deep neural network classifier; and performing a speech recognition process on the combined audio information and visual information, wherein the speech recognition process comprises: generating observation probabilities for context-dependent phonetic states using a joint audio-visual observation model, and conducting a search in a standard speech recognition engine using the observation probabilities.

2. The method of claim 1 , wherein pre-processing further comprises: determining whether a speaker is present in the visual information.

3. The method of claim 1 , wherein the standard speech recognition engine is a WFST-based speech recognition engine.

4. The method of claim 1 , wherein pre-processing further comprises: defining the region of interest by generating a probability distribution of classes for each pixel in the visual information.

5. The method of claim 1 , further comprising: identifying a location of the speaker relative to the image capture device.

6. The method of claim 1 , wherein pre-processing further comprises: scaling the region of interest.

7. The method of claim 1 , wherein combining the audio information and visual information comprises: combining the audio information and the visual information in an output layer of the single deep neural network classifier.

8. The method of claim 1 , wherein combining the audio information and visual information comprises: combining the audio information and the visual information in a second hidden layer of the single deep neural network classifier.