Using a camera for hearing device algorithm training

Abstract

A system includes an image sensor, a hearing device, and a controller. The controller may include one or more processors and may be operatively coupled to the image sensor and the audio sensor. The controller may be configured to receive image data from the image sensor and sound data from the hearing device. The controller may further be configured to identify one or more optical components using the image data, each of the one or more optical components associated with an object or activity; determine one or more audio objects using at least the one or more optical components and the sound data, the one or more audio objects may each include an association between at least a portion of the sound data and the object or activity; and adjust an audio class using the one or more audio objects, the audio class associated with the object or activity.

What is claimed is: 
     
       1. A system, comprising:
 an image sensor configured to sense optical information of an environment and produce image data indicative of the sensed optical information; 
 a hearing device comprising:
 a housing wearable by a user; and 
 an audio sensor coupled to the housing and configured to sense sound of the environment and provide sound data using the sensed sound; and 
 
 a controller comprising one or more processors and operatively coupled to the image sensor and the audio sensor, the controller configured to receive the image data and sound data and to:
 identify one or more optical components using the image data, each of the one or more optical components associated with an object or activity; 
 determine one or more audio objects using at least the one or more optical components and the sound data, the one or more audio objects each comprising an association between at least a portion of the sound data and the object or activity; and 
 adjust an audio class using the one or more audio objects, the audio class associated with the object or activity. 
 
 
     
     
       2. The system according to  claim 1 , wherein
 each of the one or more optical components is associated with the activity; 
 the one or more audio objects each comprise an association between at least a portion of the sound data and the activity; and 
 the controller is configured to adjust the audio class using the one or more audio objects, the audio class associated with the activity. 
 
     
     
       3. The system according to  claim 1 , wherein the controller is configured to:
 determine a confidence value using the one or more audio objects, and 
 adjust the audio class in response to the determined confidence value exceeding a threshold confidence value. 
 
     
     
       4. The system according to  claim 1 , wherein the controller is configured to one of more of:
 adjust a range of an overall sound level of the audio class using the one or more audio objects; 
 adjust a range of one or more frequency-specific sound levels of the audio class using the one or more audio objects; and 
 adjust a range of one or more spectral or temporal sound characteristics of the audio class using the one or more audio objects. 
 
     
     
       5. The system according to  claim 1 , comprising one more motion sensors operatively coupled to the controller and configured to sense movement of the hearing device and provide movement data indicative of the sensed movement;
 wherein the controller is configured to identify the one or more optical components using the image data and the movement data. 
 
     
     
       6. The system according to  claim 1 , wherein the audio class is one or both of an environmental audio class and a personal audio class. 
     
     
       7. The system according to  claim 1 , wherein:
 the hearing device comprises one or more physiological sensors operably coupled to the controller and configured to sense one or more physiological characteristics of the user; and 
 the controller is configured to identify the one or more optical components further using the sensed physiological characteristics of the user. 
 
     
     
       8. The system according to  claim 1 , wherein the controller is configured to generate one or more hearing environment settings using the adjusted audio class. 
     
     
       9. The system according to  claim 1 , wherein the controller is configured to generate a new audio class in response to an absence of an existing audio class associated with the object or activity of the one or more audio objects or activities. 
     
     
       10. The system according to  claim 1 , wherein:
 the hearing device comprises one or more positional sensors operably coupled to the controller and configured to sense a location of the hearing device; and 
 the controller is configured to identify the one or more optical components further using the sensed location of the hearing device. 
 
     
     
       11. The system according to  claim 1 , wherein the controller is configured to adjust one or both of:
 one or more settings of the hearing device using the identified optical component; and 
 one or more settings of the hearing device using the determined one or more audio objects. 
 
     
     
       12. The system according to  claim 1 , wherein the controller is further configured to:
 identify an audio object of the determined one or more audio objects using the sound data in the absence of the image data; 
 determine at least one adjusted audio class using the audio object of the one or more audio objects; and 
 select one or more hearing environment settings using the at least one adjusted audio class. 
 
     
     
       13. The system according to  claim 1 , comprising a communication device operably coupled to the controller and configured to one or both of transmit and receive data, wherein the controller is configured to one or both of:
 transmit the one or more audio objects or the adjusted audio class to a database; and 
 receive object information from one or more objects in the environment via the communication device and identify the one or more optical components further using the received object information. 
 
     
     
       14. The system according to  claim 1 , wherein the controller is configured to:
 determine one or more assistive listening technologies using at least the one or more optical components; and 
 connect to the determined one or more assistive listening technologies. 
 
     
     
       15. A method, comprising:
 identifying one or more optical components using image data provided by an image sensor, each of the one or more optical components associated with an object or activity; 
 determining one or more audio objects using at least the one or more optical components and sound data provided by an audio sensor, the one or more audio objects each comprising an association between at least a portion of the sound data and the object or activity; and 
 adjusting an audio class using the one or more audio objects, the audio class associated with the object or activity. 
 
     
     
       16. The method according to  claim 15 , further comprising one or both of:
 adjusting one or more settings of the hearing device using the identified optical component; and 
 adjusting one or more settings of the hearing device using the determined one or more audio objects. 
 
     
     
       17. The method according to  claim 15 , further comprising one or more of:
 adjusting a range of an overall sound level of the audio class using the one or more audio objects; 
 adjusting a range of one or more frequency-specific sound levels of the audio class using the one or more audio objects; and 
 adjusting a range of one or more spectral or temporal sound characteristics of the audio class using the one or more audio objects. 
 
     
     
       18. The method according to  claim 15 , further comprising:
 sensing movement of the hearing device using one or more motion sensors; and 
 identifying the one or more optical components using the image data and the movement data. 
 
     
     
       19. The method according to  claim 15 , further comprising:
 sensing a physiological characteristic of a user using one or more physiological sensors; and 
 identifying the one or more optical components further using the sensed physiological characteristic of the user. 
 
     
     
       20. The method according to  claim 15 , further comprising:
 identifying an audio object of the determined one or more audio objects using the sound data in absence of the image data; 
 determining at least one adjusted audio class using the audio object of the one or more audio objects; and 
 selecting one or more hearing environment settings using the at least one adjusted audio class. 
 
     
     
       21. The method according to  claim 15 , wherein
 each of the one or more optical components is associated with the activity; 
 the one or more audio objects each comprise an association between at least a portion of the sound data and the activity; and 
 the method comprises adjusting the audio class using the one or more audio objects, the audio class associated with the activity. 
 
     
     
       22. The method according to  claim 15 , further comprising:
 determining a confidence value using the one or more audio objects, and 
 adjusting the audio class in response to the determined confidence value exceeding a threshold confidence value.