Multimodal text input by a keyboard/camera text input module replacing a conventional keyboard text input module on a mobile device

Abstract

Methods and modules for a multimodal text input in a mobile device are provided. Text may be input via keyboard or camera mode by holding the camera over written text. An image is taken of the written text, text is recognized, and output to an application by: activating a keyboard mode; providing an A-Z-keyboard in a first input field; activating the camera mode; capturing the text image and displaying the captured image in a second field of a device display; converting the captured image to character text by OCR and displaying the recognized character text on the display; outputting a selected character as the input text to the application upon a character selection, or outputting a selected part of the recognized character text as the input text to the application upon a selection of the part of the recognized character text via by a single keypress, control command, or gesture.

1 - 15 . (canceled) 
     
     
         16 . A method for multimodal text input in a mobile device via a keyboard or in a camera mode by holding the camera of the mobile device over a written text, such that an image is taken of the written text and the written text is recognized, wherein the input text is output to an application requesting the input text, the method comprising the following steps:
 a) activating a keyboard mode;   b) providing an A-Z-keyboard in a first field for text input;   c) activating the camera mode;   d) capturing the image of the written text and displaying the captured image with the written text in a second field of a display of the mobile device;   e) converting the captured image to character text by optical character recognition (OCR) and displaying the recognized character text on the display;   f) outputting a selected character as the input text to the application upon a selection of the character on the A-Z-keyboard, or   outputting a selected part of the recognized character text as the input text to the application upon a selection of the part of the recognized character text;   wherein the respective selection takes place by a single keypress or control command, or by a single gesture.   
     
     
         17 . The method according to  claim 16 , wherein the A-Z-keyboard is provided as a touch screen keyboard in a first field of the display being a touch screen display. 
     
     
         18 . The method according to one of the  claims 16 , wherein the recognized character text is displayed on the display either in a separate third field or as an overlay over the text within the captured image displayed in the second field. 
     
     
         19 . The method according to  claim 16 , wherein in step e) according to the recognized character text one or more suggestion candidates are determined by an algorithm in connection with a data base displayed in one or more third fields ( 6 ) or as another overlay within the second field, wherein one or more of the candidates are selectable by a keypress event. 
     
     
         20 . The method according to  claim 16 , wherein in step f) a keypress is for instance a mechanical keypress, a touch keypress or a swype gesture on a touch screen, on either a visible key, a hidden key, within one of the fields of the display, on the part of the recognized text or on another text on the display, for a certain selection. 
     
     
         21 . The method according to  claim 16 , wherein a respective size of the first field, the second field and the third field, if available, is adapted such that a surrounding frame always occupies a same frame or field size on the display, as an original standard keyboard module on the mobile device. 
     
     
         22 . The method according to  claim 16 , wherein in the activated camera mode the second and the one or more third fields, if available, are displayed adjacent to the keyboard. 
     
     
         23 . The method according to  claim 16 , wherein the steps d)-f) are executed repetitively, wherein the respective latest recognized text in the part of the respective latest captured image is analyzed for new text in regards to a previous recognized and output character text to the application, whereupon the control command is generated for the selection of the new text as the part of the recognized character text, as long as a certain keypress is detected for an ending of the repetitive selection and outputting of the respective new text to the application. 
     
     
         24 . The method according to  claim 16 , wherein in step f) the control command for the selection of the part of the recognized character text is generated automatically via a detection algorithm, wherein the detection algorithm recognizes whether the part of the recognized character text in a previous and in the current captured image is the same. 
     
     
         25 . The method according to  claim 16 , wherein:
 the keyboard mode is executed by a keyboard text input module; and   the camera mode is executed by a separate camera text input module;   wherein the execution of the keyboard text input module is independent from the execution of the camera text input module, but wherein the execution of the camera text input module is dependent on the execution of the keyboard text input module;   wherein the camera text input module is always active in the background to detect whether the keyboard text input module is activated; and   wherein if the keyboard text input module is detected to be activated or displayed, respectively, the camera mode is activated and at least the second field ( 5 ) and the recognized text are visible on the display ( 2 ) and selectable; and   else if the keyboard text input module is not detected anymore to be activated, the camera mode is also deactivated again.   
     
     
         26 . A method for multimodal text input in a mobile device via a keyboard or in a camera mode by holding the camera of the mobile device over a written text, such that an image is taken of the written text and the written text is recognized, wherein the input text is output to an application requesting the input text, the method comprising:
 activating a keyboard mode;   in the activated keyboard mode: providing an A-Z-keyboard in a first field for text input and a control key for a selection of the camera mode;   upon a selection of the camera mode: deactivating the keyboard mode and activating the camera mode and providing a control key for a selection of the keyboard mode;   in the camera mode: capturing the image of the written text and displaying the captured image with the written text in a second field of a display of the mobile device ;   in the camera mode: converting the captured image to character text by optical character recognition (OCR) and displaying the recognized character text on the display;   outputting a selected character as the input text to the application upon a selection of the character on the A-Z-keyboard, or outputting a selected part of the recognized character text as the input text to the application upon a selection of the part of the recognized character text;   upon the selection of the keyboard mode: deactivating the camera mode and activating the keyboard mode again;   wherein all of the respective selections take place by a single keypress or control command, or by a single gesture.   
     
     
         27 . A mobile device arranged to facilitate multimodal text input, the mobile device comprising:
 a display configured to display text and image content;   a keyboard configured to receive text;   a camera having an camera mode; and   a processor in communication with the display, the keyboard, and the camera, the processor being configured to receive text input via the keyboard or the camera mode in response to detecting the camera of the mobile device being held over a written text, such that an image is taken of the written text and the written text is recognized, where the input text is output to an executing application requesting the input text;   the processor configured to communicate with the camera to capture the image of the written text;   the display configured to display the captured image with the written text in a second field of the display;   the processor configured to:
 convert the captured image to character text by optical character recognition (OCR) and cause the recognized character text to be displayed on the display; and 
 output a selected character as the input text to the application upon a selection of the character on the keyboard, or output a selected part of the recognized character text as the input text to the application upon a selection of the part of the recognized character text, the respective selection takes place by a single keypress or control command, or by a single gesture. 
   
     
     
         28 . The mobile device as in  claim 27  wherein the processor is configured to generate the keyboard on the display using a touch screen interface, the keyboard being compatible with one or more standard applications, as for instance a phone application or an internet search application, running on the mobile device and requiring the text input and/or the first and the second means are adapted to display the first field and the second field and the third field, if available, occupying together always a same size on the display. 
     
     
         29 . A mobile device according to  claim 27 , wherein the keyboard and the camera mode are both active at the same time. 
     
     
         30 . A mobile device according to  claim 29 , wherein the processor is arranged to configure the keyboard as a A-Z touchscreen interface such that the A-Z touchscreen interface and the camera are active at the same time. 
     
     
         31 . A mobile device according to  claim 29 , wherein the processor is arranged to configured the keyboard as a touch screen interface;
 the processor being further configured to display the captured image in the second field and of the recognized text and enable by a single keypress or control command, or by a single gesture both:   the selection of the part of the character text; and   the selection of the respective character key; and   the immediate outputting of the input text to the application.   
     
     
         32 . A mobile device according to  claim 31 , wherein the single keypress or control command, or by the single gesture include:
 the selection of the part of the character text;   the selection of the respective character key; and   the immediate outputting of the input text to the processor.   
     
     
         33 . A mobile device according to  claim 27 , wherein the keyboard comprises two sub-modules,
 I) a first standard keyboard sub-module being invoked by the processor if in response to a request for the text input, the first standard keyboard sub-module being dependant on a second sub-module; and   II) the second sub-module being a separate camera sub-module, wherein the camera sub-module.   
     
     
         34 . A mobile device according to  claim 33 , wherein the second keyboard sub-module is adapted to be always active in the background detecting whether the second keyboard sub-module is activated or displayed, respectively. 
     
     
         35 . A mobile device according to  claim 33 , wherein the processor is configured to respond to the second keyboard sub-module being activated or displayed by activating the camera mode, such that the second field is displayed adjacent to the keyboard; or
 else to close or shut down, respectively, the camera mode in case the camera mode is still active, such that the camera mode is always and only as long activated and displayed as the second keyboard sub-module is activated by the application.