Semiautomated Relay Method and Apparatus

Abstract

A system and method for presenting substantially simultaneous voice and text to an assisted user (AU) during a voice conversation between the AU and a hearing user (HU), the hearing user using an HU device to talk to the assisted user, the system comprising an AU captioned device including a device processor, a relay that includes a relay display, a relay speaker and a relay processor, wherein, at least one of the device processor and the relay processor is programmed to perform the steps of receiving an HU voice signal comprising a sequence of HU voice segments and assigning time stamps to each of the HU voice segments, wherein, the relay processor is programmed to perform the steps of generating text segments corresponding to each HU voice segment, storing each HU voice segment along with a corresponding text segment and a corresponding time stamp in a memory device, broadcasting the HU voice segments to a call assistant (CA) via the relay speaker and presenting each text segment via the relay display substantially contemporaneously with broadcast of the corresponding HU voice segment.

What is claimed is: 
     
         1 . A system for presenting substantially simultaneous voice and text to an assisted user (AU) during a voice conversation between the AU and a hearing user (HU), the hearing user using an HU device to talk to the assisted user, the system comprising:
 an AU captioned device including a device processor;   a relay that includes a relay display, a relay speaker and a relay processor;   wherein, at least one of the device processor and the relay processor is programmed to perform the steps of:   (i) receiving an HU voice signal comprising a sequence of HU voice segments; and   (ii) assigning time stamps to each of the HU voice segments;   wherein, the relay processor is programmed to perform the steps of:   (i) generating text segments corresponding to each HU voice segment;   (ii) storing each HU voice segment along with a corresponding text segment and a corresponding time stamp in a memory device;   (iii) broadcasting the HU voice segments to a call assistant (CA) via the relay speaker; and   (iv) presenting each text segment via the relay display substantially contemporaneously with broadcast of the corresponding HU voice segment.   
     
     
         2 . The system of  claim 1  wherein the relay processor assigns at least a subset of the time stamps to the HU voice segments. 
     
     
         3 . The system of  claim 1  wherein the AU device assigns at least a subset of the time stamps to the HU voice segments. 
     
     
         4 . The system of  claim 3  wherein the relay processor receives the HU voice signal from the AU device. 
     
     
         5 . The system of  claim 4  wherein the AU device transmits the HU voice segments and associated time stamps to the relay processor. 
     
     
         6 . The system of  claim 5  wherein the relay further transmits the text segments along with the time stamps to the AU device processor. 
     
     
         7 . The system of  claim 6  wherein the AU device further includes a device display and a device speaker, the device processor further programmed to perform the steps of, broadcasting the HU voice segments to AU via the device speaker and presenting each text segment via the device display substantially contemporaneously with broadcast of the corresponding HU voice segment via the device speaker. 
     
     
         8 . The system of  claim 1  wherein the step of generating text segments includes an automatic speech recognition (ASR) engine automatically generating the text segments from the HU voice segments. 
     
     
         9 . The system of  claim 8  wherein the relay links to an ASR engine provider server that generates the text segments and that assigns the time stamps to each text segment. 
     
     
         10 . The system of  claim 1  wherein the AU device processor assigns time stamps to each text segment and the relay assigns time stamps to each text segment. 
     
     
         11 . The system of  claim 1  wherein the relay further includes a user interface, the relay processor further programmed to monitor the interface for selection of at least a word within a text segment displayed on the relay display and, upon selection of at least a word within the displayed text, halting broadcast of the HU voice signal and rebroadcasting the at least a word via the relay speaker. 
     
     
         12 . The system of  claim 7  wherein the AU device further includes a user interface, the device processor further programmed to monitor the interface for selection of at least a word within a text segment displayed on the device display and, upon selection of at least a word within the displayed text, halting broadcast of the HU voice signal via the device speaker and rebroadcasting the at least a word via the device speaker. 
     
     
         13 . A system for presenting substantially simultaneous voice and text to an assisted user (AU) during a voice conversation between the AU and a hearing user (HU), the hearing user using an HU device to talk to the assisted user, the system comprising:
 an AU captioned device including a device processor, a device display and a device speaker;   a relay that includes a relay processor;   wherein, at least one of the device processor and the relay processor is programmed to perform the steps of:   (i) receiving an HU voice signal comprising a sequence of HU voice segments; and   (ii) assigning time stamps to each of the HU voice segments;   wherein, the relay processor is programmed to perform the steps of:   (i) generating text segments corresponding to each HU voice segment; and   (ii) transmitting the text segments and time stamps to the AU device;   wherein, the device processor is programmed to perform the steps of:   (i) receiving the text segments and time stamps;   (ii) storing each HU voice segment along with a corresponding text segment and a corresponding time stamp in a memory device;   (iii) broadcasting the HU voice segments to the AU via the device speaker; and   (iv) presenting each text segment via the device display substantially contemporaneously with broadcast of the corresponding HU voice segment.   
     
     
         14 . The system of  claim 13  wherein the AU device further includes a user interface, the device processor further programmed to monitor the interface for selection of at least a word within a text segment displayed on the device display and, upon selection of at least a word within the displayed text, halting broadcast of the HU voice signal via the device speaker and rebroadcasting the at least a word via the device speaker. 
     
     
         15 . The system of  claim 13  wherein the relay transmits the text segments immediately upon generation and wherein the device processor presents the text segments immediately upon reception, the step of presenting each text segment commensurately including visually distinguishing the text segment corresponding to the currently broadcast HU voice segment. 
     
     
         16 . The system of  claim 13  wherein the relay feeds the HU voice signal to an automatic speech recognition (ASR) engine which generates the test segments, the relay further including a call assistant (CA) station that includes a relay display, a relay speaker and a relay user interface, a CA using the station to view the text segments on the relay display while listening to the HU voice signal, the device processor further programmed to monitor an AU device interface for selection of a text segment on the device display that is subsequent to the text segment corresponding to a currently broadcast text segment and, upon receiving the text segment selection, halting broadcast of the HU voice signal, identifying the HU voice segment associated with the selected text segment and broadcasting the identified HU voice segment via the device speaker. 
     
     
         17 . The system of  claim 16  wherein, upon receiving the text segment selection, transmitting a text segment selection signal to the relay processor, upon receiving the text segment selection signal, the relay processor halting broadcast of the HU voice signal, identifying the HU voice segment associated with the selected text segment and broadcasting the identified HU voice segment via the relay speaker. 
     
     
         18 . A system for presenting substantially simultaneous voice and text to an assisted user (AU) during a voice conversation between the AU and a hearing user (HU), the hearing user using an HU device to talk to the assisted user, the system comprising:
 an AU captioned device including a device processor, a device display and a device speaker;   a relay that includes a relay processor, a relay display and a relay speaker;   wherein, each of the AU device and the relay performs the steps of:   (i) receiving an HU voice signal comprising a sequence of HU voice segments; and   (ii) storing time stamps with each of the HU voice segments;   (iii) presenting the text segments via the displays;   wherein the relay processor is further programmed to perform the steps of:   (i) broadcasting each voice segment via the relay speaker substantially commensurate with presenting the corresponding text segment via the relay display;   wherein the device processor is further programmed to perform the steps of:   (i) broadcasting each voice segment via the device speaker substantially commensurate with presenting the corresponding text segment via the device display;   (ii) monitoring for a signal from the AU to skip ahead in the voice segment broadcast; and   (iii) upon receiving the signal to skip ahead, transmitting a skip ahead signal to the relay thereby causing the relay to automatically skip ahead in HU voice segment broadcast.