Semiautomated relay method and apparatus

Abstract

A captioning method for presenting captions to an assisted user (AU) during communication with a hearing user (HU) where the assisted user uses a captioned device and the hearing user uses a hearing user's device to facilitate the communication, the captioned device including a display screen and a speaker for presenting captions and broadcasting the hearing user's voice signals, respectively, the method comprising the steps of during an ongoing call between the AU and the HU, using an automated speech recognition (ASR) engine to generate initial ASR captions associated with the HU's voice signal, assessing at least one caption quality factor associated with prior initial ASR captions generated during the ongoing call, delaying broadcast of HU voice signal to the AU and based on the at least one caption quality factor, adjusting a duration of the HU voice signal broadcast delay.

What is claimed is: 
     
         1 . A method for captioning a hearing user's (HU's) voice during a call with an assisted user (AU), the method comprising the steps of:
 (a) storing a plurality of HU voice profiles and associated voice models for each of a plurality of HU device identifiers in a voice recognition database;   (b) subsequent to receiving an incoming call at an AU communication device;   (c) identifying an HU device identifier associated with the HU device used to initiate the incoming call;   (d) receiving HU voice signal during the call;   (e) comparing the HU voice signal to HU voice profiles associated with the HU device identifier to identify a current HU voice profile associated with the HU voice signal;   (f) selecting the voice model that is associated with the current HU voice profile as a current voice model;   (g) using the current voice model to transcribe the HU voice signal to text;   (h) presenting the text on a display screen of the AU communication device; and   (i) repeating steps (d) through (h) to continually identify a current HU voice model and use the current voice model to transcribe.   
     
     
         2 . The method of  claim 1  further including the steps of, determining that the HU voice signal does not match any of the stored HU voice profiles, using a default voice model to generate text and training the default voice model to generate a new voice model. 
     
     
         3 . The method of  claim 2  further including using the HU voice signal to generate a new voice profile and storing the new voice profile and the new voice model in the memory voice recognition database for subsequent use. 
     
     
         4 . The method of  claim 3  wherein the new voice model and new voice profile are stored along with the HU device identifier. 
     
     
         5 . The method of  claim 2  further including, upon determining that the HU voice signal does not match any of the stored HU voice profiles, having a call assistant (CA) transcribe the HU voice signal to text which is presented via the display screen while the new voice model is trained. 
     
     
         6 . The method of  claim 5  further including monitoring accuracy of the new voice model during training and, once accuracy exceeds a threshold level, switching from the CA generated text to use the new voice model to generate the text that is presented via the display screen. 
     
     
         7 . The method of  claim 1  wherein the AU communication device links to a remote relay for captioning services and wherein the HU voice profiles and voice models are stored at the relay. 
     
     
         8 . The method of  claim 1  wherein the HU voice profiles and voice models are stored in the AU communication device. 
     
     
         9 . The method of  claim 1  wherein each HU voice model is periodically modified as additional HU voice signal is processed to generate text. 
     
     
         10 . The method of  claim 9  wherein a call assistant CA corrects errors in the text and the system automatically modifies an HU voice model based on CA error corrections. 
     
     
         11 . The method of  claim 2  wherein the step of using a default voice model includes identifying HU voice signal characteristics and selecting one of a plurality of default voice models based on the identified HU voice signal characteristics. 
     
     
         12 . The method of  claim 1  wherein the HU communication device identifier is a phone number. 
     
     
         13 . The method of  claim 1  wherein the HU communication device identifier is a network address. 
     
     
         14 . A method for captioning a hearing user's (HU's) voice during a call with an assisted user (AU), the method comprising the steps of:
 during a voice call between an HU communication device and an AU communication device, receiving an HU voice signal;   using an automated speech recognition (ASR) engine to generate caption text for the HU voice signal;   storing the caption text in a memory device without initially presenting the text captions;   receiving a caption activation signal from the assisted user at a first time; and   presenting the caption text corresponding to a period prior to the first time to the AU via an AU communication device display screen.   
     
     
         15 . The method of  claim 14  wherein the step of  claim 14  wherein the AU communication device includes a user interface that includes a caption activation feature that the AU may use to generate the caption activation signal. 
     
     
         16 . The method of  claim 14  wherein the period prior to the first time includes a duration of 20 seconds or less. 
     
     
         17 . The method of  claim 14  further including broadcasting the HU voice signal in essentially real time to the AU via a speaker. 
     
     
         18 . The method of  claim 17  further including, upon receiving the caption activation signal, generating caption text for the HU voice signal as the HU voice signal is received and presenting the continuing caption text via the device display as that text is generated. 
     
     
         19 . A method for captioning a hearing user's (HU's) voice during a call with an assisted user (AU), the method comprising the steps of:
 during a voice call between an HU communication device and an AU communication device, receiving an HU voice signal;   broadcasting the HU voice signal via a speaker to the AU;   receiving a caption activation signal from the assisted user at a first time;   in response to receiving the caption activation signal;   using an automated speech recognition (ASR) engine to generate ASR text for the HU voice signal;   forming a link to a call assistant (CA) at a remote relay;   transmitting the HU voice signal to the CA for transcription to CA generated text;   receiving the CA generated text at the AU communication device;   prior to receiving the CA generated text at the AU communication device, presenting the ASR text via an AU communication device display screen; and   subsequent to receiving the CA generated text at the AU communication device, presenting the CA generated text via the AU communication device display screen.