Content output management based on speech quality

Abstract

Techniques for ensuring content output to a user conforms to a quality of the user's speech, even when a speechlet or skill ignores the speech's quality, are described. When a system receives speech, the system determines an indicator of the speech's quality (e.g., whispered, shouted, fast, slow, etc.) and persists the indicator in memory. When the system receives output content from a speechlet or skill, the system checks whether the output content is in conformity with the speech quality indicator. If the content conforms to the speech quality indicator, the system may cause the content to be output to the user without further manipulation. But, if the content does not conform to the speech quality indicator, the system may manipulate the content to render it in conformity with the speech quality indicator and output the manipulated content to the user.

1 .- 20 . (canceled) 
     
     
         21 . A computer-implemented method, comprising:
 receiving, from a first device, first audio data representing first speech;   determining a first indicator representing a first emotion corresponding to the first speech;   performing speech processing on the first audio data to determine first data representing first content responsive to the first speech;   determining, based at least in part on the first indicator, the first content is to be output according to the first emotion;   generating first metadata representing text-to-speech (TTS) processing is to be performed based at least in part on the first emotion;   performing, based at least in part on the first metadata, TTS processing on the first data to generate second audio data; and   causing output of the second audio data.   
     
     
         22 . The computer-implemented method of  claim 21 , further comprising:
 determining second data representing second content responsive to the first speech; and   determining, based at least in part on the first indicator, that the first content is to be output instead of the second content.   
     
     
         23 . The computer-implemented method of  claim 22 , further comprising:
 performing speech processing on the first audio data to determine natural language understanding (NLU) results data;   determining a first component associated with the NLU results data;   sending, to the first component, the NLU results data and the first indicator; and   receiving, from the first component, the first data and the second data.   
     
     
         24 . The computer-implemented method of  claim 22 , further comprising:
 performing speech processing on the first audio data to determine natural language understanding (NLU) results data;   determining a first component associated with the NLU results data;   sending, to the first component, the NLU results data and the first indicator;   determining a second component associated with the NLU results data;   sending, to the second component, the NLU results data and the first indicator;   receiving, from the first component, the first data; and   receiving, from the second component, the second data.   
     
     
         25 . The computer-implemented method of  claim 21 , further comprising:
 receiving third audio data representing second speech;   determining a second indicator representing a second emotion corresponding to the second speech;   performing speech processing on the third audio data to determine second data representing second content responsive to the second speech;   determining the second content corresponds to a second emotion;   determining, based at least in part on the second indicator, the second content is to be output according to a third emotion;   in response to the second content corresponding to the second emotion and the second content is to be output according to the third emotion, generating second metadata representing TTS processing is to be performed based at least in part on the third emotion;   performing, based at least in part on the second metadata, TTS processing on the second data to generate fourth audio data; and   causing output of the fourth audio data.   
     
     
         26 . The computer-implemented method of  claim 21 , wherein the second audio data matches the first emotion. 
     
     
         27 . The computer-implemented method of  claim 21 , further comprising:
 processing the first audio data to determine the first indicator.   
     
     
         28 . The computer-implemented method of  claim 21 , further comprising:
 processing image data corresponding to the first speech to determine the first indicator.   
     
     
         29 . The computer-implemented method of  claim 28 , wherein the image data represents a gesture of a non-facial body part of a user. 
     
     
         30 . The computer-implemented method of  claim 21 , wherein causing output of the second audio data comprises causing a second device, different from the first device, to output audio corresponding to the second audio data. 
     
     
         31 . A system comprising:
 at least one processor; and   at least one memory comprising instructions that, when executed by the at least one processor, cause the system to:
 receive, from a first device, first audio data representing first speech; 
 determine a first indicator representing a first emotion corresponding to the first speech; 
 perform speech processing on the first audio data to determine first data representing first content responsive to the first speech; 
 determine, based at least in part on the first indicator, the first content is to be output according to the first emotion; 
 generate first metadata representing text-to-speech (TTS) processing is to be performed based at least in part on the first emotion; 
 perform, based at least in part on the first metadata, TTS processing on the first data to generate second audio data; and 
 cause output of the second audio data. 
   
     
     
         32 . The system of  claim 31 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
 determine second data representing second content responsive to the first speech; and   determine, based at least in part on the first indicator, that the first content is to be output instead of the second content.   
     
     
         33 . The system of  claim 32 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
 perform speech processing on the first audio data to determine natural language understanding (NLU) results data;   determine a first component associated with the NLU results data;   send, to the first component, the NLU results data and the first indicator; and   receive, from the first component, the first data and the second data.   
     
     
         34 . The system of  claim 32 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
 determine speech processing on the first audio data to determine natural language understanding (NLU) results data;   determine a first component associated with the NLU results data;   send, to the first component, the NLU results data and the first indicator;   determine a second component associated with the NLU results data;   send, to the second component, the NLU results data and the first indicator;   receive, from the first component, the first data; and   receive, from the second component, the second data.   
     
     
         35 . The system of  claim 31 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
 receive third audio data representing second speech;   determine a second indicator representing a second emotion corresponding to the second speech;   perform speech processing on the third audio data to determine second data representing second content responsive to the second speech;   determine the second content corresponds to a second emotion;   determine, based at least in part on the second indicator, the second content is to be output according to a third emotion;   in response to the second content corresponding to the second emotion and the second content is to be output according to the third emotion, generate second metadata representing TTS processing is to be performed based at least in part on the third emotion;   perform, based at least in part on the second metadata, TTS processing on the second data to generate fourth audio data; and   cause output of the fourth audio data.   
     
     
         36 . The system of  claim 31 , wherein the second audio data matches the first emotion. 
     
     
         37 . The system of  claim 31 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
 process the first audio data to determine the first indicator.   
     
     
         38 . The system of  claim 31 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
 process image data corresponding to the first speech to determine the first indicator.   
     
     
         39 . The system of  claim 38 , wherein the image data represents a gesture of a non-facial body part of a user. 
     
     
         40 . The system of  claim 31 , wherein the instructions that cause the system to cause output of the second audio data comprise instructions that, when executed by the at least one processor, cause the system to cause a second device, different from the first device, to output audio corresponding to the second audio data.