Method, apparatus, and system for providing interpretation result using visual information

Abstract

Provided is a method of providing an interpretation result using visual information, and the method includes: acquiring a spatial domain image including line-of-sight information of a user and gaze position information in the spatial domain image; segmenting the acquired spatial domain image into a plurality of images; detecting text areas including text for each of the segmented images; generating text blocks, each of which is a text recognition result for each of the detected text areas, and determining the text block corresponding to the gaze position information; converting a first language included in the determined text block into a second language that is a target language; and providing the user with a conversion result of the second language.

What is claimed is: 
     
       1. A method of providing an interpretation result using visual information, which is performed by an apparatus for providing an interpretation result using visual information, the method comprising:
 acquiring a spatial domain image including line-of-sight information of a user and gaze position information in the spatial domain image; 
 segmenting the acquired spatial domain image into a plurality of images; 
 detecting text areas including text for each of the segmented images; 
 generating text blocks, each of which is a text recognition result for each of the detected text areas, and determining the text block corresponding to the gaze position information; 
 converting a first language included in the determined text block into a second language that is a target language; and 
 providing the user with a conversion result of the second language, 
 wherein the generating of the text blocks, each of which is the text recognition result for each of the detected text areas, and the determining of the text block corresponding to the gaze position information includes:
 combining text blocks consecutively located within predetermined sections adjacent to each other into one text block; and 
 assigning a unique number to the text block or the combined text block, and 
 
 wherein the generating of the text blocks, each of which is the text recognition result for each of the detected text areas, and the determining of the text block corresponding to the gaze position information includes
 recognizing the text block corresponding to the unique number assigned in previous visual information as the same text block for a certain period of time. 
 
 
     
     
       2. The method of  claim 1 , further comprising arranging the spatial domain image and the gaze position information corresponding to the spatial domain area based on time. 
     
     
       3. The method of chair  1 , wherein the combining of the text blocks consecutively located within the predetermined sections adjacent to each other into one text block includes
 separating the text blocks having at least one of a different size and a different font and located in the same or adjacent text areas. 
 
     
     
       4. The method of  claim 1 , wherein the generating of the text blocks, each of which is the text recognition result for each of the detected text areas, and the determining of the text block corresponding to the gaze position information includes
 selecting the text block closest to the gaze position information among the generated text blocks and determining the text block. 
 
     
     
       5. The method of  claim 1 , wherein the generating of the text blocks, each of which is the text recognition result for each of the detected text areas, and the determining of the text block corresponding to the gaze position information includes,
 in response to point information of the user different from the gaze position information being present, selecting the text block closest to the point information of the user and determining the text block. 
 
     
     
       6. The method of  claim 5 , wherein the generating of the text blocks, each of which is the text recognition result for each of the detected text areas, and the determining of the text block corresponding to the gaze position information includes,
 in response to the point information of the user being detected in a plurality of pieces thereof, providing feedback information to use only one piece of point information among the plurality of pieces of point information. 
 
     
     
       7. The method of  claim 5 , wherein the generating of the text blocks, each of which is the text recognition result for each of the detected text areas, and the determining of the text block corresponding to the gaze position information includes,
 when the point information of the user continuously moves for a certain period of time within the same text block, selecting a text range pointed at by the point information as one independent user-specified text block and determining the text range. 
 
     
     
       8. The method of  claim 1 , wherein the generating of the text blocks, each of which is the text recognition result for each of the detected text areas, and the determining of the text block corresponding to the gaze position information includes,
 when the gaze position information sporadically shakes without being fixed for a certain period of time, providing a message guiding that the point information of the user is to be used. 
 
     
     
       9. The method of  claim 1 , wherein the segmenting of the acquired spatial domain image into the plurality of images includes:
 segmenting the acquired spatial domain image into the plurality of images on the basis of a pre-trained neural network algorithm; 
 determining, for each of the segmented images, an individual object corresponding to one of previously-specified classes; and 
 generating relative position information of the individual object based on the text block that is the text recognition result for the determined individual object. 
 
     
     
       10. The method of  claim 9 , wherein the converting of the first language included in the determined text block into the second language that is the target language includes converting the relative position information of the individual object corresponding to the determined text block into the second language, and
 wherein the providing of the converted second language to the user includes providing the user with the second language for the determined text block and the position information. 
 
     
     
       11. An apparatus for providing an interpretation result using visual information, the apparatus comprising:
 a visual information receiver configured to acquire a spatial domain image including line-of-sight information of a user and gaze position information in the spatial domain image; 
 a visual information converter configured to segment the acquired spatial domain image into a plurality of images, detect text areas including text for each of the segmented images, determine, among text blocks, each of which is a text recognition result for each of the detected text areas, a text block corresponding to the gaze position information, and convert a first language included in the determined text block into a second language that is a target language; and 
 an interpretation result provider configured to provide the user with a conversion result of the second language, 
 wherein the visual information converter is configured to combine text blocks consecutively located within predetermined sections adjacent to each other into one text block and assign a unique number to the text block or the combined text block, and 
 wherein the visual information converter is configured to recognize the text block corresponding to the unique number assigned in previous visual information as the same text block for a certain period of time. 
 
     
     
       12. The apparatus of  claim 11 , wherein the visual information converter is configured to select the text block closest to the gaze position information among the generated text blocks and determine the text block. 
     
     
       13. The apparatus of  claim 11 , wherein the visual information converter is configured to, in response to point information of the user different from the gaze position information being present, select the text block closest to the point information of the user and determining the text block. 
     
     
       14. The apparatus of  claim 11 , wherein the visual information converter is configured to segment the acquired spatial domain image into the plurality of images on the basis of a pre-trained neural network algorithm and determine, for each of the segmented images, an individual object corresponding to one of previously-specified classes, and
 wherein the interpretation result provider is configured to generate relative position information of the individual object based on the text block that is the text recognition result for the determined individual object. 
 
     
     
       15. The apparatus of  claim 14 , wherein the visual information converter is configured to convert the relative position information of the individual object corresponding to the determined text block into the second language, and
 wherein the interpretation result provider is configured to provide the user with the second language for the determined text block and the position information. 
 
     
     
       16. A system for providing an interpretation result using visual information, the system comprising:
 a communicator configured to acquire a spatial domain image including line-of-sight information of a user and gaze position information in the spatial domain image from an external device; 
 a memory in which a program for recognizing text on the basis of the spatial domain image and the gaze position information and provide an interpretation result for the recognized text is stored; 
 a processor configured to execute the program stored in the memory to: segment the acquired spatial domain image into a plurality of images; detect text areas including text for each of the segmented images; determine, among text blocks, each of which is a text recognition result for each of the detected text areas, a text block corresponding to the gaze position information; and convert a first language included in the determined text block into a second language that is a target language; and 
 a device configured to provide the user with a conversion result of the second language, 
 wherein the processor is configured to combine text blocks consecutively located within predetermined sections adjacent to each other into one text block and assign a unique number to the text block or the combined text block, and 
 wherein the processor is configured to recognize the text block corresponding to the unique number assigned in previous visual information as the same text block for a certain period of time.