Audio biomarker for virtual lung function assessment and auscultation

Abstract

A mobile device application prompts and conducts audio and/or video tests using a microphone on a smartphone, tablet or laptop in order to record and analyze a patient's speech, cough, breathing and other sounds in order to diagnose the patient with Covid 19, another ailment, or as having normal ranges not indicative of disease. The mobile device's tests and protocols use program instructions, AI processing and other automated tools to facilitate the speed and reliability of the testing.

1 . A device having an application program for collecting user voice data and heartrate data interactively, comprising:
 a memory, including program instructions, for collecting user's voice data and heartrate interactively; and   a processor coupled to the memory for executing the program instructions to interact with the users to collect their voice and heartrate data and meta information based on pre-defined questions from their devices.   
     
     
         2 . The device according to  claim 1 , wherein the processor is configured to run on different operating systems including Windows, iOS, and Android. 
     
     
         3 . The device according to  claim 2 , wherein the processor is configured to further execute program instructions to perform tasks for which subjects take (a) lung reserve assessment with counting test, (b) speech assessment, (c) deep breathing assessment, (d) egophony screening test for pneumonia, and (e) cough pattern recognition test. 
     
     
         4 . The device according to  claim 3 , wherein the program instructions stored in the memory further include program instructions to provide user interface for labelling and tagging data with guideline from medical doctors. 
     
     
         5 . The device according to  claim 4 , wherein the program instructions stored in the memory further include program instructions for machine learning using multimodal voice and heartrate data collected from the current system and relevant data from elsewhere. 
     
     
         6 . The device according to  claim 5 , wherein the program instructions stored in the memory further include program instructions to perform ASR, VAD, spectral analyses, LPC analyses, voice pitch and voice formants and voice intensity analyses, heartrate estimation, and machine learning. 
     
     
         7 . The device according to  claim 6 , wherein the program instructions stored in the memory further include program instruction to perform the following analyses:
 (a) using ASR to recognize counting words, analyze the highest count a patient reached and a total duration of the counting;   (b) using ASR to analyze reading speech by the patient and analyze the subject's speaking rate, breaths per minute, PSD, pitch, formants, inter-word breath, and intra-word breath;   (c) using VAD to detect inhale, pause, and exhale and analyze the breath duration, breath PSD, and pause duration;   (d) using ASR to recognize the patient's /E/ and /A/ speech and analyze the closeness of the two sounds against the patient's baseline and a general normal baseline; and   (e) using VAD to detect a patient's cough and compare the patient's cough to the patient's own baseline cough, and classify the cough into specific categories (wet cough versus dry cough).   
     
     
         8 . The device according to  claim 7 , wherein the program instructions stored in the memory further include a program to run in different languages, including at least English and Spanish, to facilitate access. 
     
     
         9 . The device according to  claim 7 , wherein the program instructions stored in the memory further include program instructions to conduct analyses using multimodal voice and heartrate data and machine learning, to summarize the evaluation results, and to provide a risk score and report to review and provide to doctors. 
     
     
         10 . The device according to  claim 7 , wherein the program instructions stored in the memory further include program instructions to evaluate reliability of the voice-based and image-based heartrate data in VLFAA. 
     
     
         11 . The device according to  claim 7 , wherein the program instructions stored in the memory further include program instructions to further collect meta data through voice on the subject's age, gender, condition, medical history, and activities before testing. 
     
     
         12 . The device according to  claim 11 , wherein the program instructions stored in the memory further include program instructions to perform sentiment analysis to access the subject's anxiety level. 
     
     
         13 . A mobile device for diagnosing respiratory disease, comprising:
 a memory storing at least one application program capable of executing a plurality of tests each having a test protocol, the application program including program instructions;   a microphone and speaker;   a display;   a processor coupled to the memory, the display, the microphone and the speaker, the processor configured to execute the program instructions in order to: (i) instruct the user to perform an action including at least breathing, coughing, and reading text; (ii) activating the microphone to record the user following each instruction for each of the plurality of test protocols in respective recordings; (iii) analyzing and scoring the audio as according to criteria specific to each test protocol that are based on spectral data associated with the respective recordings; and (iv) determining a composite score for presence or absence of the disease based on the individual scores; (v) storing data, the test scores and the composite scores.   
     
     
         14 . The mobile device according to  claim 13 , wherein the processor is configured to instruct the user to perform each test according to its respective test protocol using the display of the mobile device. 
     
     
         15 . The mobile device according to  claim 13 , wherein the processor is configured to instruct the user to perform each test according to its respective test protocol using the speaker of the mobile device. 
     
     
         16 . The mobile device according to  claim 13 , wherein the mobile device further includes a network interface and wherein the processor is further configured exchange data with a remote database and server to store data generated during (i)-(v) on the server and in the database. 
     
     
         17 . The mobile device according to  claim 16 , wherein the server includes a processor and is configured to determine individual test scores and an aggregate score based on the data generated by the mobile device. 
     
     
         18 . The mobile device according to  claim 13 , wherein the processor determines each breath that the user takes based on a spectral analysis of the users' performance of each test. 
     
     
         19 . The mobile device according to  claim 18 , wherein the processor determines each breath that the user takes based on a spectral analysis of the users performance of each test as compared to a representative control. 
     
     
         20 . The mobile device according to  claim 18 , wherein the processor determines each breath that the user takes based on a spectral analysis of the users performance of each test as compared to a prior baseline of the user. 
     
     
         21 . The mobile device according to  claim 13 , wherein the processor determines PSD for the duration of a user's performance based on a spectral analysis of the users' performance of each test. 
     
     
         22 . The mobile device according to  claim 21 , wherein the processor determines a score for at least some of the tests based on the PSD of the users' performance of the at least some of the tests as compared to a representative control. 
     
     
         23 . The mobile device according to  claim 21 , wherein the processor determines a score for at least some of the tests based on the PSD of the users' performance of the at least some of the tests as compared to a prior baseline of the user. 
     
     
         24 . The mobile device according to  claim 13 , wherein the processor is configured to determine a presence of a disease based on the user's voice uttering /A/ or /E/ when instructed and the outcome of ASR and confidence scores using the utterances. 
     
     
         25 . The mobile device according to  claim 13 , wherein the scoring determines disease is more likely when the ASR fails to distinguish /A/ from /E/ or one or both of the confidence scores of the ASR determining /A/ or /E/ is low.