Operating a hearing device to assist the user in engaging in a healthy living style

The present application claims priority to EP Patent Application No. 22212819.1, filed Dec. 12, 2022, the contents of which are hereby incorporated by reference in their entirety.

Hearing devices may be used to improve the hearing capability or communication capability of a user, for instance by compensating a hearing loss of a hearing-impaired user, in which case the hearing device is commonly referred to as a hearing instrument such as a hearing aid, or hearing prosthesis. A hearing device may also be used to output sound based on an audio signal which may be communicated by a wire or wirelessly to the hearing device. A hearing device may also be used to reproduce a sound in a user's ear canal detected by an input transducer such as a microphone or a microphone array. The reproduced sound may be amplified to account for a hearing loss, such as in a hearing instrument, or may be output without accounting for a hearing loss, for instance to provide for a faithful reproduction of detected ambient sound and/or to add audio features of an augmented reality in the reproduced ambient sound, such as in a hearable. A hearing device may also provide for a situational enhancement of an acoustic scene, e.g. beamforming and/or active noise cancelling (ANC), with or without amplification of the reproduced sound. A hearing device may also be implemented as a hearing protection device, such as an earplug, configured to protect the user's hearing. Different types of hearing devices configured to be be worn at an ear include earbuds, earphones, hearables, and hearing instruments such as receiver-in-the-canal (RIC) hearing aids, behind-the-ear (BTE) hearing aids, in-the-ear (ITE) hearing aids, invisible-in-the-canal (IIC) hearing aids, completely-in-the-canal (CIC) hearing aids, cochlear implant systems configured to provide electrical stimulation representative of audio content to a user, a bimodal hearing system configured to provide both amplification and electrical stimulation representative of audio content to a user, or any other suitable hearing prostheses. A hearing system comprising two hearing devices configured to be worn at different ears of the user is sometimes also referred to as a binaural hearing device. A hearing system may also comprise a hearing device, e.g., a single monaural hearing device or a binaural hearing device, and a user device, e.g., a smartphone and/or a smartwatch, communicatively coupled to the hearing device.

Hearing devices are often employed in conjunction with communication devices, such as smartphones or tablets, for instance when listening to sound data processed by the communication device and/or during a phone conversation operated by the communication device. More recently, communication devices have been integrated with hearing devices such that the hearing devices at least partially comprise the functionality of those communication devices. A hearing system may comprise, for instance, a hearing device and a communication device.

In recent times, some hearing devices are also increasingly equipped with different sensor types. Traditionally, those sensors often include a sound sensor to detect a sound and to output an amplified and/or signal processed version of the sound to the user. In an effort to provide the user with even more information about himself and/or the ambient environment, various other sensor types are progressively implemented, in particular sensors which are not directly related to the sound reproduction and/or amplification function of the hearing device. Those sensors include inertial sensors, such as accelerometers, allowing to monitor the user's movements. Physiological sensors, such as optical sensors and bioelectric sensors, are mostly employed for monitoring the user's health.

Hearing impairment is often paired with other health risks. Those hearing loss comorbidities can affect the physical health, e.g., an increased risk of cardiovascular disease, dehydration, diabetes, stumbling or falling during walking, visual impairment, sleep deprivation, and insufficient bodily hygiene. Further, the mental health of a hearing device user can be affected, e.g., social isolation, unhappiness, depressive symptoms, and cognitive decline. Those health risks could be reduced or mitigated by the user, at least to a certain extent, by engaging in a healthier living style. However, often the user is not aware or tends to ignore daily opportunities to improve his health. Especially older users tend to be stuck in their daily routine and find it hard to adopt a new behavior benefiting their health situation.

For example, a user suffering from dehydration could decrease the risk by increasing his fluid intake but may instead stick to his habit of not drinking enough. A user feeling unhappy or depressed may improve his mood by directing his awareness to a subject in his environment suitable to lighten him up but may instead be distracted coping with inner negative emotions. A user living in social isolation could improve his situation by taking opportunities to engage in conversations with other people but may instead choose to ignore those possibilities to meet people or to strengthen relationships with acquaintances. A user neglecting his body hygiene such as his oral care could improve on that by brushing his teeth more carefully and/or more regularly but may not feel inclined to do so out of habit. A user at risk of a cardiovascular disease could improve his fitness by regularly exercising but may postpone or forget about it in his daily routine. A user suffering from sleep deprivation could take precautions making it easier to him falling asleep, e.g., by tuning down a disturbing sound before going to bed, but may not take those measures by force of habit or for convenience.

The disclosure relates to a method of operating a hearing device configured to be worn at an ear of a user to assist the user in engaging in a healthy living style. The disclosure also relates to a hearing device for performing the method and a computer-readable medium storing instructions for performing the method.

It is a feature of the present disclosure to avoid at least one of the above mentioned disadvantages and to propose a method of operating a hearing device which assists the user in engaging in a healthier living style, in particular relative to a subject in the user's environment which is suitable to impact the physical and/or mental health of the user. It is another feature that such a subject can be more easily recognized by the user and/or that an awareness of the user of the subject can be increased. It is a further feature that a desire of the user to utilize the subject or a subject of a corresponding type and/or to engage in an activity involving the subject or a subject of a corresponding type is enhanced. It is a further feature that a subject potentially harmful for the user's health can be more easily ignored by the user and/or faded out from the user's consciousness and/or consciousness. It is another feature to achieve at least one of the above mentioned features by employing a sound emitted by the subject. It is another feature to address a specific aspect of the user's health, in particular at least one of the aspects addressed further below. It is a further feature to recognize a living situation of the user in which assisting the user in engaging in the healthier living style, in particular relative to the specific health aspect, is beneficial for the user and to assist the user in such a situation. It is another feature to propose a hearing device and/or a computer implemented medium having at least one of these advantages.

At least one of these features can be achieved by a method of providing an alert for a user comprising the features described herein and/or a hearing device comprising the features described herein and/or a computer-readable medium comprising the features described herein.

Accordingly, the present disclosure proposes a method of operating a hearing device configured to be worn at an ear of a user to assist the user in engaging in a healthy living style, the method comprising

In particular, by the modifying of the audio feature in the input audio signal, a perception of the audio feature in the output audio signal can be facilitated or impeded for the user relative to a perception of the sound emitted by the subject in the detected sound. In this way, the user can be assisted into engaging into the healthier living style by way of evoking his increased or decreased awareness relative to the subject suitable for impacting the user's health.

Independently, the present disclosure proposes a hearing device configured to be worn at an ear of a user, the hearing device comprising an input transducer configured to provide an input audio signal indicative of a sound detected in the environment of the user, a processor, and an output transducer configured to output an output audio signal, wherein the processor is configured to perform the method.

Independently, the present disclosure proposes a hearing system comprising a hearing device configured to be worn at an ear of a user and a communication device communicatively coupled to the hearing device, the hearing device comprising an input transducer configured to provide an input audio signal indicative of a sound detected in the environment of the user, and an output transducer configured to output an output audio signal, wherein the hearing device and/or the communication device comprises a processor configured to perform the method.

Independently, the present disclosure proposes a non-transitory computer-readable medium storing instructions that, when executed by a processor included in a hearing device, cause the hearing device to perform operations of the method

Subsequently, additional features of some implementations of the method of operating the hearing device and/or the hearing device and/or the computer-readable medium are described. Each of those features can be provided solely or in combination with at least another feature. The features can be correspondingly provided in some implementations of the method and/or the hearing device and/or the computer-readable medium.

In some implementations, during the determining whether the input audio signal comprises the audio feature, the input audio signal is input into a machine learning algorithm which separates the audio feature from the input audio signal and/or outputs a likelihood that the input audio signal comprises the audio feature, wherein the machine learning algorithm has been trained with previously recorded input audio signal characteristic of the sound emitted by the subject. In some instances, the sound samples may be recorded from the subject and/or one or more other subjects of a corresponding type suitable to impact the physical and/or mental health of the user.

In some implementations, after the determining whether the input audio signal comprises the audio feature, the machine learning algorithm is trained with the input audio signal and/or the audio feature matching the sound pattern when it is determined that the input audio signal comprises the audio feature matching the sound pattern.

In some implementations, the method further comprises receiving user control data indicative of whether the user desires said modifying of the audio feature, wherein the determining whether the input audio signal comprises the audio feature and/or the modifying of the audio feature in the input audio signal is performed depending on the user control data.

In some instances, the user control data may be received from a user interface. The user interface may be provided by the hearing device and/or a communication device communicatively coupled to the hearing device. For instance, the user interface may be provided as a push button and/or a touch screen allowing the user to enter a command whether he desires said modifying of the audio feature. The user interface may also be provided by speech recognition of the voice of the user, e.g., when detected by the input transducer, allowing the user to enter the command whether he desires said modifying of the audio feature. The command entered by the user via the user interface can thus be converted in the user control data.

In some implementations, the subject is of a type of subjects which are each suitable to impact an equal aspect of the physical and/or mental health of the user, wherein at least part of the subjects of the type emit a different sound and the sound pattern is characteristic of the different sound emitted by different subjects. In some instances, the sound pattern may be characteristic of the sound emitted by a plurality of the subjects of the type emitting a different sound. In some instances, the sound pattern may be characteristic of the sound emitted by each of the subjects of the type. In some instances, the processing unit is configured to determine that the input audio signal comprises the audio feature matching the sound pattern when the sound emitted from one or more of the subjects of the type is detected in the environment.

In some implementations, the aspect of the physical and/or mental health of the user includes

In some implementations, the method further comprises receiving, from a user interface, user control data indicative of one or more said aspects of the physical and/or mental health of the user which the user desires to engage in, wherein the determining whether the input audio signal comprises the audio feature and/or the modifying of the audio feature in the input audio signal is performed depending on the user control data. In some instances, the user control data may be received from a user interface. The user interface may be provided at the hearing device and/or a communication device communicatively coupled to the hearing device. For instance, the user interface may be provided as a push button and/or a touch screen allowing the user to select one or more aspects of the physical and/or mental health which he desires to engage in. The user interface may also be provided by speech recognition of the voice of the user, e.g., when detected by the input transducer, allowing the user to select by speech one or more aspects of the physical and/or mental health which he desires to engage in. The selection of one or more aspects of the physical and/or mental health entered by the user via the user interface can thus be converted in the user control data.

In some implementations, the method further comprises

In some implementations, the sensor data is indicative of the aspect of the physical and/or mental health of the user and the condition is determined to be fulfilled when said aspect of the physical and/or mental health of the user is determined to be below a threshold. For example, the threshold may be representative of a critical value of the drinking and/or eating behavior of the user and/or mood of the user and/or social activity of the user and/or physical activity of the user and/or nervousness and/or emotional stress of the user and/or sleeping behavior of the user and/or tooth brushing behavior of the user.

In some implementations, the physiological data is received from

In some implementations, the optical sensor configured to emit the light at a wavelength absorbable by an analyte contained in blood such that the physiological sensor data included in the optical sensor data comprises information about the blood flowing through tissue at the ear. In some implementations, the optical sensor is configured as a photoplethysmography (PPG) sensor such that the physiological sensor data included in optical sensor data comprises PPG data, e.g. a PPG waveform. In some implementations, the bioelectric sensor comprises a skin impedance sensor and/or an electrocardiogram (ECG) sensor and/or an electroencephalogram (EEG) sensor and/or an electrooculography (EOG) sensor.

In some implementations, the sound pattern is characteristic of

In some implementations, when the aspect of the physical and/or mental health of the user concerns the drinking behavior of the user, in particular a hydration level of the user and/or a state of dehydration, the sound pattern is characteristic of a sound of opening a receptacle and/or a sound of pouring and/or flowing and/or splashing water.

In some implementations, when the aspect of the physical and/or mental health of the user concerns the eating behavior of the user, in particular an unbalanced and/or inferior eating behavior of the user and/or a state of malnutrition, the sound pattern is characteristic of a sound related to food preparation and/or food consumption.

In some implementations, when the aspect of the physical and/or mental health of the user concerns the mood of the user, in particular a level of happiness or unhappiness and/or a state of depression, the sound pattern is characteristic of a sound of opening a receptacle and/or a sound of pouring and/or flowing and/or splashing water and/or a sound related to food preparation and/or food consumption and/or a motor sound and/or a sound typically occurring in a natural and/or rural environment and/or a sound of a wild animal and/or a sound of a domesticated animal and/or a sound of a baby and/or a sound of human laughter and/or giggling and/or a sound of crackling fire and/or a sound of blowing wind and/or a sound of a music instrument and/or a singing voice and/or a sound of a communication between humans and/or a voice of a significant other and/or a sound of footsteps and/or a voice of a significant other and/or a sound of footsteps and/or a sound of tooth brushing.

In some implementations, when the aspect of the physical and/or mental health of the user concerns the social activity of the user, in particular a level of social isolation and/or a state of loneliness, the sound pattern is characteristic of a sound of a communication between humans and/or a voice of a significant other and/or a sound of footsteps and/or a sound related to people performing a physical activity and/or a voice of a significant other.

In some implementations, when the aspect of the physical and/or mental health of the user concerns a nervousness and/or emotional stress of the user, the sound pattern is characteristic of a sound of opening a receptacle and/or a sound of pouring and/or flowing and/or splashing water and/or a sound related to food preparation and/or food consumption and/or a motor sound and/or a sound typically occurring in a natural and/or rural environment and/or a sound of a wild animal and/or a sound of a domesticated animal and/or a sound of a baby and/or a sound of human laughter and/or giggling and/or a sound of crackling fire and/or a sound of blowing wind and/or a sound of a music instrument and/or a singing voice and/or a sound of a communication between humans and/or a voice of a significant other and/or a sound of footsteps and/or a sound of tooth brushing.

In some implementations, when the aspect of the physical and/or mental health of the user concerns a physical activity of the user, in particular a fitness level and/or a lack of exercise, the sound pattern is characteristic of a sound of footsteps and/or a sound related to people performing a physical activity.

In some implementations, when the aspect of the physical and/or mental health of the user concerns a tooth brushing behavior of the user, in particular an insufficient level and/or a lack of dental hygiene, the sound pattern is characteristic of a sound of tooth brushing.

In some implementations, when the aspect of the physical and/or mental health of the user concerns a drinking and/or eating behavior of the user and/or a mood of the user and/or a social activity of the user and/or a nervousness and/or emotional stress of the user and/or a physical activity of the user and/or a body hygiene of the user, in particular a tooth brushing behavior of the user, the input audio signal is modified in the input audio signal such that an awareness of the user of a presence of the subject in the environment is increased. In particular, the input audio signal may be modified by augmenting the audio feature in the input audio signal such that, during the outputting of the audio signal, a perception of the augmented audio feature in the output audio signal is facilitated for the user relative to a perception of the sound emitted by the subject in the detected sound.

In some implementations, when the aspect of the physical and/or mental health of the user concerns a sleeping behavior of the user, in particular an insufficient amount of sleep and/or a sleep deprivation and/or a nervousness and/or emotional stress of the user, the sound pattern is characteristic of a sound of a household appliance and/or a sound of traffic noise and/or factory noise and/or a sound of a baby.

In some implementations, when the aspect of the physical and/or mental health of the user concerns a sleeping behavior of the user and/or a nervousness and/or emotional stress of the user, the input audio signal is modified in the input audio signal such that an awareness of the user of a presence of the subject in the environment is decreased. In particular, the input audio signal may be modified by fading and/or suppressing the audio feature in the input audio signal such that, during the outputting of the audio signal, a perception of the modified audio feature in the output audio signal is impeded for the user relative to a perception of the sound emitted by the subject in the detected sound.

In some implementations, the method further comprises

In some implementations, the condition is determined to be fulfilled when a current location of the user and/or current time, as indicated by the context data, matches a predetermined location and/or time. To illustrate, the engaging of the user in a healthy living style may thus be restricted to the predetermined location and/or time at which it can be more beneficial to the user.

In some implementations, said modifying the audio feature in the input audio signal comprises

In some implementations, the second audio feature superimposed on the first audio feature may be an audible audio feature, in particular such that the second audio feature is consciously perceivable by the user. In some implementations, the second audio feature superimposed on the first audio feature may be a subliminal audio feature, in particular such that the second audio feature is only subconsciously perceivable by the user.

In some implementations, said modifying the audio feature in the input audio signal comprises augmenting the audio feature in the input audio signal. In this way, an awareness of the user of a presence of the subject in the environment can be increased during the outputting of the output audio signal, in particular such that a perception of the audio feature in the output audio signal can be facilitated for the user relative to a perception of the sound emitted by the subject in the detected sound. The augmenting may comprise at least one of said directing a beamformer and/or increasing a sharpness and/or increasing a volume and/or altering a frequency of the audio feature and/or increasing a number of times in which the audio feature is included in the audio signal and/or extracting and altering a property of the audio feature and/or superimposing a second audio feature in the input audio signal.

In some implementations, said modifying the audio feature in the input audio signal comprises fading and/or suppressing the audio feature in the input audio signal. In this way, an awareness of the user of a presence of the subject in the environment can be decreased during the outputting of the output audio signal, in particular such that a perception of the audio feature in the output audio signal can be impeded for the user relative to a perception of the sound emitted by the subject in the detected sound. The fading and/or suppressing may comprise at least one of said decreasing a sharpness and/or decreasing a volume and/or altering a frequency of the audio feature and/or superimposing a second audio feature in the input audio signal. In particular, the second audio feature may be less perceivable than the first audio feature.

illustrates an exemplary hearing deviceconfigured to be worn at an ear of a user.

Hearing devicemay be implemented by any type of hearing device configured to enable or enhance hearing or a listening experience of a user wearing hearing device. For example, hearing devicemay be implemented by a hearing aid configured to provide an amplified version of audio content to a user, a sound processor included in a cochlear implant system configured to provide electrical stimulation representative of audio content to a user, a sound processor included in a bimodal hearing system configured to provide both amplification and electrical stimulation representative of audio content to a user, or any other suitable hearing prosthesis, or an earbud or an earphone or a hearable.

Different types of hearing devicecan also be distinguished by the position at which they are worn at the ear. Some hearing devices, such as behind-the-ear (BTE) hearing aids and receiver-in-the-canal (RIC) hearing aids, typically comprise an earpiece configured to be at least partially inserted into an ear canal of the ear, and an additional housing configured to be worn at a wearing position outside the ear canal, in particular behind the ear of the user. Some other hearing devices, as for instance earbuds, earphones, hearables, in-the-ear (ITE) hearing aids, invisible-in-the-canal (IIC) hearing aids, and completely-in-the-canal (CIC) hearing aids, commonly comprise such an earpiece to be worn at least partially inside the ear canal without an additional housing for wearing at the different ear position.

As shown, hearing deviceincludes a processorcommunicatively coupled to a memory, an input transducer, and an output transducer. Hearing devicemay include additional or alternative components as may serve a particular implementation. Input transducermay be implemented by any suitable device configured to detect sound in the environment of the user and to provide an input audio signal indicative of the detected sound, e.g., a microphone or a microphone array. Output transducermay be implemented by any suitable audio transducer configured to output an output audio signal to the user, for instance a receiver of a hearing aid, an output electrode of a cochlear implant system, or a loudspeaker of an earbud.

Memorymay be implemented by any suitable type of storage medium and is configured to maintain, e.g. store, data controlled by processor, in particular data generated, accessed, modified and/or otherwise used by processor. For example, memory may be configured to store instructions used by processorto process the input audio signal received from input transducer, e.g., one or more audio processing programs. As another example, memorymay be configured to store instructions to determine a property of the input audio signal, e.g., whether the audio signal comprises an audio feature matching a sound pattern. The instructions may include a database of a plurality of different sound patterns which may be accessed by processoras required. Memorymay comprise a non-volatile memory from which the maintained data may be retrieved even after having been power cycled, for instance a flash memory and/or a read only memory (ROM) chip such as an electrically erasable programmable ROM (EEPROM). A non-transitory computer-readable medium may thus be implemented by memory. Memorymay further comprise a volatile memory, for instance a static or dynamic random access memory (RAM).

Processoris configured to receive, from input transducer, an input audio signal indicative of a sound detected in the environment of the user; to determine whether the input audio signal comprises an audio feature matching a sound pattern characteristic of sound emitted by a subject suitable to impact a physical and/or mental health of the user; and, when the input audio signal comprises the audio feature, to modify the audio feature in a processing of the input audio signal so as to increase or decrease, during the outputting of the audio signal, an awareness of the user of a presence of the subject in the environment. Based on the processed input audio signal, an output audio signal can then be output by output transducer. These and other operations, which may be performed by processor, are described in more detail in the description that follows.

As illustrated, hearing devicemay further comprise a communication port. Communication portmay be implemented by any suitable data transmitter and/or data receiver and/or data transducer configured to exchange data with another device. For instance, the other device may be another hearing device configured to be worn at the other ear of the user than hearing deviceand/or a communication device such as a smartphone, smartwatch, tablet and/or the like.