Deep Neural Network Based Audio Processing Method, Device and Storage Medium

1. A deep neural network (DNN) based audio processing method, comprising: obtaining a DNN-based speech extraction model, wherein the speech extraction model is created through the following steps: obtaining a mixed audio training dataset having multiple mixed audio data frames each containing mixed speech data and non-speech data, the speech data and the non-speech data both being represented in time domain data format; acquiring at least one audiogram and at least one set of predetermined gain compensation coefficients associated with the at least one audiogram, wherein each audiogram corresponds to a set of predetermined gain compensation coefficients, and each set of predetermined gain compensation coefficients include multiple predetermined gain compensation coefficients corresponding to respective audio signal frequencies; performing, for each of the mixed audio data frames, gain compensation on the speech data included therein with the at least one set of predetermined gain compensation coefficients to generate compensated speech data; and training the DNN-based speech extraction model with the mixed audio training dataset and the compensated speech data corresponding to each of the mixed audio data frames of the mixed audio training dataset to obtain a trained speech extraction model; receiving an audio input object having a speech portion and a non-speech portion, wherein the audio input object includes one or more audio data frames each having a set of audio data samples sampled at a predetermined sampling interval and represented in time domain data format; obtaining a user audiogram and a set of user gain compensation coefficients associated with the user audiogram; and inputting the audio input object and the set of user gain compensation coefficients into the trained speech extraction model to obtain an audio output result represented in time domain data format outputted by the trained speech extraction model, wherein the non-speech portion of the audio input object is at least partially attenuated in or removed from the audio output result.

2. The audio processing method of claim 1 , wherein the speech data and the non-speech data included in each of the mixed audio data frames are mixed at a predetermined gain ratio.

3. The audio processing method of claim 1 , wherein the step of performing, for each of the mixed audio data frames, gain compensation on the speech data included therein with the at least one set of predetermined gain compensation coefficients to generate compensated speech data comprises: performing Fourier transform on the speech data included in each of the mixed audio data frames to obtain corresponding speech data represented in frequency domain data format; performing, for each of the mixed audio data frames, gain compensation on the speech data represented in frequency domain data format with the at least one set of predetermined gain compensation coefficients to generate compensated speech data represented in frequency domain data format; and performing, for each of the mixed audio data frames, inverse Fourier transform on the compensated speech data represented in frequency domain data format to generate the compensated speech data represented in time domain data format.

4. The audio processing method of claim 1 , wherein the step of training the DNN-based speech extraction model with the mixed audio training dataset and the compensated speech data corresponding to each of the mixed audio data frames of the mixed audio training dataset to obtain a trained speech extraction model comprises: training the speech extraction model by using the mixed audio training dataset and the at least one set of predetermined gain compensation coefficients associated with the at least one audiogram as inputs to an input layer of the speech extraction model and using the compensated speech data corresponding to each of the mixed audio data frames of the mixed audio training dataset as outputs of an output layer of the speech extraction model.

5. The audio processing method of claim 1 , wherein the speech extraction model is trained with an Error Back Propagation algorithm.

6. The audio processing method of claim 1 , wherein the trained speech extraction model has a weighting coefficient set and an offset coefficient set, and the trained speech extraction model comprises multiple processing sublayers each weighting the audio data frames with at least one set of weighting coefficients included in the weighting coefficient set.

7. The audio processing method of claim 6 , wherein the DNN is a recurrent neural network.

8. The audio processing method of claim 7 , wherein the multiple processing sublayers include at least one Gated Recurrent Unit processing sublayer or a Long Short Time Memory network processing sublayer.

9. The audio processing method of claim 1 , wherein an input layer of the speech extraction model comprises a first plurality of neurons for receiving the audio input object, and an output layer of the speech extraction model includes a second plurality of neurons for outputting the audio output result, and wherein a number of the first plurality of neurons is equal to a number of the second plurality of neurons.

10. The audio processing method of claim 9 , wherein the input layer of the speech extraction model further comprises a third plurality of neurons for receiving the set of user gain compensation coefficients.

11. The audio processing method of claim 1 , wherein the step of acquiring at least one audiogram and at least one set of predetermined gain compensation coefficients associated with the at least one audiogram comprises: acquiring at least one audiogram; and for each audiogram, selecting multiple different audio signal frequencies within a frequency range of the audiogram and determining multiple predetermined gain compensation coefficients respectively corresponding to the multiple different audio signal frequencies with a Wide Dynamic Range Compression algorithm.

12. The audio processing method of claim 11 , wherein the at least one audiogram is generated randomly.

13. The audio processing method of claim 11 , wherein the at least one audiogram includes the user audiogram.

14. The audio processing method of claim 1 , wherein each audio data frame of the audio input object has a frame length of 1 to 50 milliseconds and a sampling frequency not less than 10 kHz.

15. The audio processing method of claim 1 , wherein for each audio signal frequency, each set of predetermined gain compensation coefficients include one or more predetermined gain compensation coefficients corresponding to different loudness respectively.

16. A deep neural network (DNN) based audio processing device, wherein the audio processing device comprises a non-transitory computer storage medium for storing one or more executable instructions that, when executed by a processor, causes the processor to perform: obtaining a DNN-based speech extraction model, wherein the speech extraction model is created through the following steps: obtaining a mixed audio training dataset having multiple mixed audio data frames each containing mixed speech data and non-speech data, the speech data and the non-speech data both being represented in time domain data format; acquiring at least one audiogram and at least one set of predetermined gain compensation coefficients associated with the at least one audiogram, wherein each audiogram corresponds to a set of predetermined gain compensation coefficients, and each set of predetermined gain compensation coefficients include multiple predetermined gain compensation coefficients corresponding to respective audio signal frequencies; performing, for each of the mixed audio data frames, gain compensation on the speech data included therein with the at least one set of predetermined gain compensation coefficients to generate compensated speech data; and training the DNN-based speech extraction model with the mixed audio training dataset and the compensated speech data corresponding to each of the mixed audio data frames of the mixed audio training dataset to obtain a trained speech extraction model; receiving an audio input object having a speech portion and a non-speech portion, wherein the audio input object includes one or more audio data frames each having a set of audio data samples sampled at a predetermined sampling interval and represented in time domain data format; obtaining a user audiogram and a set of user gain compensation coefficients associated with the user audiogram; and inputting the audio input object and the set of user gain compensation coefficients into the trained speech extraction model to obtain an audio output result represented in time domain data format outputted by the trained speech extraction model, wherein the non-speech portion of the audio input object is at least partially attenuated in or removed from the audio output result.

17. The audio processing device of claim 16 , wherein the audio processing device is a hearing assistance device.

18. A non-transitory computer storage medium having stored therein one or more executable instructions that, when executed by a processor, causes the processor to perform: obtaining a DNN-based speech extraction model, wherein the speech extraction model is created through the following steps: obtaining a mixed audio training dataset having multiple mixed audio data frames each containing mixed speech data and non-speech data, the speech data and the non-speech data both being represented in time domain data format; acquiring at least one audiogram and at least one set of predetermined gain compensation coefficients associated with the at least one audiogram, wherein each audiogram corresponds to a set of predetermined gain compensation coefficients, and each set of predetermined gain compensation coefficients include multiple predetermined gain compensation coefficients corresponding to respective audio signal frequencies; performing, for each of the mixed audio data frames, gain compensation on the speech data included therein with the at least one set of predetermined gain compensation coefficients to generate compensated speech data; and training the DNN-based speech extraction model with the mixed audio training dataset and the compensated speech data corresponding to each of the mixed audio data frames of the mixed audio training dataset to obtain a trained speech extraction model; receiving an audio input object having a speech portion and a non-speech portion, wherein the audio input object includes one or more audio data frames each having a set of audio data samples sampled at a predetermined sampling interval and represented in time domain data format; obtaining a user audiogram and a set of user gain compensation coefficients associated with the user audiogram; and inputting the audio input object and the set of user gain compensation coefficients into the trained speech extraction model to obtain an audio output result represented in time domain data format outputted by the trained speech extraction model, wherein the non-speech portion of the audio input object is at least partially attenuated in or removed from the audio output result.