Reusable multimodal application

Abstract

A method and system are disclosed herein for accepting multimodal inputs and deriving synchronized and processed information. A reusable multimodal application is provided on the mobile device. A user transmits a multimodal command to the multimodal platform via the mobile network. The one or more modes of communication that are inputted are transmitted to the multimodal platform(s) via the mobile network(s) and thereafter synchronized and processed at the multimodal platform. The synchronized and processed information is transmitted to the multimodal application. If required, the user verifies and appropriately modifies the synchronized and processed information. The verified and modified information are transferred from the multimodal application to the visual application. The final result(s) are derived by inputting the verified and modified results into the visual application.

1. A method of processing data by a mobile device, the method comprising:
 invoking a multimodal application; 
 receiving a selection of at least one of a sequential multimodality and a simultaneous multimodality, the sequential multimodality configured to sequentially switch between one or more modes of communication, and said simultaneous multimodality configured to substantially simultaneously communicate using the one or more modes of communication without switching between the one or more modes of communication; 
 receiving input information by the multimodal application in the one or more modes of communication using the selected multimodality; 
 transmitting data indicative of the input information received in the one or more modes of communication to a multimodal platform via a mobile communications network; 
 receiving, from the multimodal platform, synchronized and processed information comprising input information synchronized between the one or modes of communication, the multimodal application being operable to use the synchronized and processed information for continued execution using the one or more modes of communication; 
 presenting the synchronized and processed information for verification and modification in one or more input modes; and 
 rendering the synchronized and processed information. 
 
     
     
       2. The method of  claim 1 , wherein the one or more modes of communication comprise audio, visual and haptic modes. 
     
     
       3. The method of  claim 1 , further comprising:
 determining that the one or more modes of communication comprise audio and text; 
 determining that the input information comprises a search request; 
 transmitting the search request and associated search grammar derived by the multimodal application via the mobile communications network to the multimodal platform; 
 receiving search words for input to the multimodal application, the search words having been determined by synchronizing the input information to create synchronized information; 
 identifying predetermined grammar elements associated with the search request and the associated search grammar; and 
 processing the synchronized information and determining search words based on the identified grammar elements. 
 
     
     
       4. The method of  claim 1 , wherein the rendering is performed by a user interface application. 
     
     
       5. The method of  claim 1 , wherein the invoking the multimodal application is activated in response to a voice command. 
     
     
       6. The method of  claim 1 , wherein the multimodal application is operable to render a search interface that enables a user to navigate to a web page of interest through a voice input. 
     
     
       7. The method of  claim 1 , wherein the multimodal application is operable to render an input interface. 
     
     
       8. The method of  claim 1 , wherein the input information to the multimodal application is freeform text. 
     
     
       9. The method of  claim 1 , wherein the multimodal application is a web portal. 
     
     
       10. The method of  claim 1 , wherein the input information to the multimodal application is a uniform resource locator address. 
     
     
       11. The method of  claim 1 , wherein the sequential multimodality allows users to switch between visual and voice modes. 
     
     
       12. The method of  claim 1 , wherein the simultaneous multimodality allows users to simultaneously use voice and visual modes. 
     
     
       13. The method of  claim 1 , wherein the multimodal application further comprises global grammar elements that can be invoked by the user using predefined multimodal invocations. 
     
     
       14. A mobile device comprising a processor and memory, the device configured to:
 render, using a set of interface rules, a visual search interface; 
 execute a multimodal application operative to receive a selection of one of a sequential multimodality and a simultaneous multimodality, receive multimodal commands in the selected multimodality, and send information representing the multimodal commands to a multimodal platform communicatively coupled to a mobile communications network, the multimodal platform being configured to receive and synchronize the information and process the synchronized multimodal commands to create synchronized and processed information, the multimodal application being operable to use the synchronized and processed information for continued execution using the multimodal commands; 
 receive the synchronized and processed information; 
 present the synchronized and processed information for verification and modification in one or more input modes; and 
 display the synchronized and processed information. 
 
     
     
       15. A method for processing multimodal information from a plurality of mobile devices connected to a mobile network, the method comprising:
 receiving, via the mobile network, multimodal input information from a mobile device, the multimodal information having been processed by a multimodal application configured to execute on the mobile device and receive multimodal inputs in one or more modes of communication in a selected one of a sequential multimodality and a simultaneous multimodality, wherein the sequential multimodality is operative to enable the user to sequentially switch between the one or more modes of communication, and wherein the simultaneous multimodality is operative to enable the user to substantially simultaneously communicate in the one or more modes of communication; 
 synchronizing the multimodal input information to create synchronized information, the multimodal application being operable to use the synchronized information for continued execution using the one or more modes of communication; and 
 processing the synchronized information and transmitting the synchronized and processed information to the mobile device for rendering, verification, and modification of the synchronized and processed information in one or more input modes on the mobile device.