A speaker estimation method that estimate the speaker from audio and image includes: inputting audio; extracting a feature quantity representing a voice characteristic from the input audio; inputting an image; detecting person regions of respective persons from the input image; estimating feature quantities representing voice characteristics from the respective detected person regions; Performing a change such that an image taken from another position and with another angle is input when any person is not detected; calculating a similarity between the feature quantity representing the voice characteristic extracted from the audio and the feature quantity representing the voice characteristic estimated from the person region in the image; and estimating a speaker from the calculated similarity.
Legal claims defining the scope of protection, as filed with the USPTO.
1. A speaker estimation method that estimates a speaker from audio and image, comprising: inputting audio; extracting a feature quantity comprising a first vector representing a voice characteristic from the input audio; inputting an image of respective persons, the image not taken during speaking by the respective persons; detecting person regions of the respective persons from the input image: estimating a feature quantity comprising a second vector representing a voice characteristic from the respective detected person regions; performing a change such that an image taken from another position and with another angle is input when any person is not detected; calculating a similarity between the feature quantity representing the voice characteristic extracted from the input audio and the feature quantity representing the voice characteristic estimated from the person region in the image by comparing at least an L2 norm between the first vector and the second vector; and estimating a speaker from the calculated similarity.
2. The speaker estimation method according to claim 1 , comprising changing a position and an angle for inputting an image when the calculated similarity has a maximum value smaller than a threshold value.
3. The speaker estimation method according to claim 1 , comprising outputting audio and a behavior for identifying the speaker when the calculated similarity has a maximum value smaller than a threshold value.
4. The speaker estimation method according to claim 3 , comprising outputting the audio and the behavior for identifying the speaker when the calculated similarity has two or more values equal to or more than the threshold value.
5. The speaker estimation method according to claim 1 , comprising: estimating a sound-source direction from the audio; calculating a sound-source score for each person from the estimated sound-source direction and a person detection direction in the image, the sound-source score indicating a probability of the speaker; and estimating the speaker from the calculated similarity and the sound-source score.
6. A speaker estimation device that estimates a speaker from audio and image, comprising: a processor including a memory and a control component; a microphone that receives audio; and a camera that receives an image of respective persons, the image not taken during speaking by the respective persons, the processor configured to perform: an image input controller function to control a position and an angle for inputting the image of the camera; and a control function to control the microphone, the camera, and the image input controller function, wherein the processor performs the control function to: extract a feature quantity comprising a first vector representing a voice characteristic from the audio input from the microphone; detect person regions of the respective persons from the image input from the camera; estimate a feature quantity comprising a second vector representing a voice characteristic from the respective detected person regions; control the image input controller function to perform a change such that an image taken from another position and with another angle is input when any person is not detected; calculate a similarity between the feature quantity representing the voice characteristic extracted from the input audio and the feature quantity representing the voice characteristic estimated from the person region in the image by comparing at least an L2 norm between the first vector and the second vector; and estimate a speaker from the calculated similarity.
7. The speaker estimation device according to claim 6 , wherein the processor performs the control function to control the image input controller to change a position and an angle for inputting an image when the calculated similarity has a maximum value smaller than a threshold value.
8. The speaker estimation device according to claim 6 , comprising an output device that outputs audio and a behavior, wherein the processor is further configured to perform the control function to control the output device to output the audio and the behavior for identifying the speaker when the calculated similarity has a maximum value smaller than a threshold value.
9. The speaker estimation device according to claim 8 , wherein the processor is further configured to perform the control function to control the output device to output the audio and the behavior for identifying the speaker when the calculated similarity has two or more values equal to or more than the threshold value.
10. The speaker estimation device according claim 6 , wherein the microphone includes a plurality of audio input units, the processor further configured to perform the control function to: estimate a sound-source direction from a plurality of pieces of audio input from the microphone; calculate a sound-source score for each person from the estimated sound-source direction and a person detection direction in the image, the sound-source score indicating a probability of the speaker; and estimate the speaker from the calculated similarity and the sound-source score.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
February 26, 2019
August 31, 2021
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.