System and method for linking streams of multimedia data to reference material for display

Abstract

A system for indexing displayed elements that is useful for accessing and understanding new or difficult materials, in which a user highlights unknown words or characters or other displayed elements encountered while viewing displayed materials. In a language learning application, the system displays the meaning of a word in context; and the user may include the word in a personal vocabulary to build a database of words and phrases. In a Japanese language application, one or more Japanese language books are read on an electronic display. Readings (‘yomi’) for all words are readily viewable for any selected word or phrase, as well as an English reference to the selected word or phrase. Extensive notes are provided for difficult phrases and words not normally found in a dictionary. A unique indexing scheme allows word-by-word access to any of several external multi-media references.

1. A system for linking source material to external reference materials for display, the system comprising:
 a source material image including a plurality of discrete pieces having links to external reference materials comprising any of textual, audio, video, and picture information, said source material image stored in an electronic database; 
 means for determining an  a beginning position address on said  of a source material stored in an electronic database for the beginning position of said source material image ;  
 means for cutting said  the source material image  into said  a plurality of discrete pieces;  
 means for determining a starting point address and an ending point address on said electronic database for a start point and an end point of said  of at least one of the plurality of discrete pieces of said image  based upon said  the beginning position of said source material image  address;  
 means for recording said  in a look-up table the starting and said  ending point addresses in a look-up table ;  
 means for linking at least one of the plurality of discrete pieces to at least one of a plurality of external reference materials by recording in the look-up table, along with the starting and ending point addresses of the at least one of the plurality of discrete pieces, a link to the at least one of the plurality of external reference materials, the plurality of external reference materials comprising any of textual, audio, video, and picture information;  
   means for displaying an image of the source material;    
 means for selecting a discrete portion of said  the displayed source material image;  
 means for determining the  a display address on said electronic database  of said  the selected discrete portion;  
 means for converting said  the display address of said  the selected discrete portion to an offset value from said  the beginning position address of said source material image ;  
 means for comparing said  the offset value with said recorded  the starting and ending point addresses of said discrete pieces  recorded in said  the look-up table to identify one of the plurality of discrete pieces;  
 means for selecting an external reference that corresponds to said look-up table start and end point address  one of the plurality of external reference materials corresponding to the identified one of the plurality of discrete pieces; and  
 means for retrieving the selected one of the plurality of external reference materials using a recorded link to the selected one of the plurality of external reference materials; and  
 means for reproducing said  displaying the retrieved external reference material.  
 
     
     
       2. The system of  claim 1 , further comprising:
 a linking engine for  wherein the means for linking said source material to said reference information  links at least one of the plurality of discrete pieces to at least one of a plurality of external reference materials on any of  a word-by-word and  or phrase-by-phrase basis.  
 
     
     
       3. The system of  claim 2 , said linking engine  further comprising:
 word cut means for dividing said source material into discrete pieces;  
 linking means for establishing at least one link between each of said discrete pieces and said reference information;  
 compiler  means for assembling an integrated  compiling the source material image from said  at least the plurality of discrete pieces; and  
 indexing  means for linking said assembledindexing at least one of the plurality of discrete pieces to saidand corresponding links to the plurality of external reference informationmaterials.  
 
     
     
       4. The system of  claim 3 , said linking engine  further comprising:
 means for building an index to link each of said source material pieces to said reference information  the look-up table from the indexed discrete pieces and the corresponding links to the plurality of external reference materials.  
 
     
     
       5. The system of  claim 4 , wherein said index  the look-up table links said source material pieces to said reference information  the identified one of the plurality of discrete pieces to at least a corresponding one of a plurality of external reference materials based upon the offset value of the offset of the starting and ending position addresses of said source material pieces from the beginning position address of said integrated source image . 
     
     
       6. The system of  claim 5 , wherein said offset locates said reference information to a corresponding source material piece  the identified one of the plurality of discrete pieces is identified based upon the offset occurrence  value being within a range defined by the value of the offsets of  the starting and ending point addresses of said source material pieces from said beginning position address of said integrated source image  the identified one of the plurality of discrete pieces. 
     
     
       7. The system of  claim 1 , further comprising:
 means for manipulating said stored  the source material image and the plurality of external reference information  materials with at least two user keys.  
 
     
     
       8. A method for linking source material to external reference materials for display, the method comprising the steps of:
 determining the  a beginning position address of a source material image  stored in an electronic database, said source material image including a plurality of discrete pieces having links to external reference materials comprising any of textual, audio, video, and picture information ;  
 cutting said  the source material image  into said  a plurality of discrete pieces;  
 determining a starting point address and an ending point address of said  at least one of the plurality of discrete pieces of said image  based upon said  the beginning position address of said source material image ;  
 recording said  in a look up table the starting and said  ending point addresses in a look up table ;  
 linking at least one of the plurality of discrete pieces to at least one of a plurality of external reference materials by recording in the look-up table, along with the starting and ending point addresses of the at least one of the plurality of discrete pieces, a link to the at least one of the plurality of external reference materials, the plurality of external reference materials comprising any of textual, audio, video, and picture information;  
   displaying an image of the source material;   
 selecting a discrete portion of said  the displayed source material image;  
 determining the  a display address of said  the selected discrete portion;  
 converting said  the display address of said  the selected discrete portion to an offset value from said  the beginning position address of said source material image ;  
 comparing said  the offset value with said recorded  the starting and ending point addresses of said discrete pieces  recorded in said  the look-up table to identify one of the plurality of discrete pieces;  
 selecting an external reference that corresponds to said look-up table start and end point addresses  one of the plurality of external reference materials corresponding to the identified one of the plurality of discrete pieces; and  
 retrieving the selected one of the plurality of external reference materials using a recorded link to the selected one of the plurality of external, reference materials; and  
 reproducing saiddisplaying the retrieved external reference material.  
 
     
     
       9. In a language learning method, a method for linking source material to external reference materials for display, the method comprising the steps of:
 reading a foreign language source material image including a plurality of discrete pieces having links to external reference materials comprising any of textual, audio, video, and picture information with an electronic viewer;  
 accessing reference materials on selected portions of said source material image;  
 determining the  a beginning position address of said  a foreign language source material image ;  
 cutting said  the source material image  into said  a plurality of discrete pieces;  
 determining a starting point address and an ending point address of said  at least one of the plurality of discrete pieces of said image  based upon said  the beginning position address of said source material image ;  
 recording said  in a look-up table the starting and said  ending point addresses in a look-up table ;  
 linking at least one of the plurality of discrete pieces to at least one of a plurality of external reference materials by recording in the look-up table, along with the starting and ending point addresses of the at least one of the plurality of discrete pieces, a link to the at least one of the plurality of external reference materials, the plurality of external reference materials comprising any of textual, audio, video, and picture information;  
   displaying an image of the source material;    
 selecting a discrete portion of said  the displayed source material image;  
 determining the  a display address of said  the selected discrete portion;  
 converting said  the display address of said  the selected discrete portion to an offset value from said  the beginning position address of said source material image ;  
 comparing said  the offset value with said recorded  the starting and ending point addresses of said discrete pieces  recorded in said  the look-up table to identify one of the plurality of discrete pieces;  
 selecting an external reference that corresponds to said look-up table start and end point addresses  one of the plurality of external reference materials corresponding to the identified one of the plurality of discrete pieces; and  
 retrieving the selected one of the plurality of external reference materials using a recorded link to the selected one of the plurality of external, reference materials; and  
 reproducing saiddisplaying the retrieved external reference material.  
 
     
     
       10. The method of  claim 9 , further comprising the step of  wherein the step of linking comprises:
 linking said source material to said reference information with a linking engine  at least one of the plurality of discrete pieces to at least a corresponding one of a plurality of external reference materials on any of  a word-by-word and  or phrase-by-phrase basis.  
 
     
     
       11. The method of  claim 10 , said linking step  further comprising the steps of:
 dividing said source material into discrete pieces;  
 establishing at least one link between each of said discrete pieces and said reference information;  
 assembling an integrated  compiling the source material image from said  at least the plurality of discrete pieces; and  
 linking said assembledindexing at least one of the plurality of discrete pieces to saidin the source material image and the corresponding links to the plurality of external reference informationmaterials.  
 
     
     
       12. The method of  claim 11 , said linking step  further comprising the step of:
 building an index to link each of said source material pieces to said reference information  the look-up table from the indexed discrete pieces of the source material image and the corresponding links to the plurality of external reference materials.  
 
     
     
       13. The method of  claim 12 , wherein said index  the look-up table links said source material pieces to said reference information  the identified one of the plurality of discrete pieces to at least a corresponding one of a plurality of external reference materials based upon the offset between the starting position address for said source material pieces and the beginning position address of said integrated source image  value. 
     
     
       14. The method of  claim 13 , wherein said offset locates said reference information to a corresponding source material piece  the identified one of the plurality of discrete pieces is identified based upon the offset occurrence  value being within a range defined by the value of the offsets of  the starting and ending position  point addresses of said source material pieces from said beginning position address of said integrated source image  the identified one of the plurality of discrete pieces. 
     
     
       15. In a language learning system, a system for linking source material to external reference materials for display, the system comprising:
 a text image including a plurality of discrete pieces having links to external reference materials comprising any of textual, audio, video, and picture information; 
 means for determining the  a beginning position address of said  text image ;  
 means for cutting said  the text image  into said  a plurality of discrete pieces;  
 means for determining a starting point address and an ending point address of said  at least one of the plurality of discrete pieces of said image  based upon said  the beginning position address of said source material image ;  
 means for recording said  in a look-up table the starting and said  ending point addresses in a look-up table ;  
 means for linking at least one of the plurality of discrete pieces to at least one of a plurality of external reference materials by recording in the look-up table, along with the starting and ending point addresses of the at least one of the plurality of discrete pieces, a link to the at least one of the plurality of external reference materials, the plurality of external reference materials comprising any of textual, audio, video, and picture information;  
   means for displaying an image of the text;   
 means for selecting a discrete portion of said  the displayed text image;  
 means for selecting a discrete portion of said  the displayed  
 means for determining the  a display address of said  the selected discrete portion;  
 means for converting said  the display address of said  the selected discrete portion to an offset value from said  the beginning position address of said source material image ;  
 means for comparing said  the offset value with said recorded  the starting and ending point addresses of said discrete pieces  recorded in said  the look-up table to identify one of the plurality of discrete pieces;  
 means for selecting an external reference that corresponds to said look-up table start and end point address  one of the plurality of external reference materials corresponding to the identified one of the plurality of discrete pieces; and  
 means for retrieving the selected one of the plurality of external reference materials using a recorded link to the selected one of the plurality of external reference materials; and  
 means for displaying said  the retrieved external reference material.  
 
     
     
       16. In a language learning method, a method for linking source material to external reference materials for display, the method comprising the steps of:
 determining the  a beginning position address of a  text image, said text image including a plurality of discrete pieces having links to external reference materials comprising any of textual, audio, video, and picture information ;  
 cutting said source material image  the text into said  a plurality of discrete pieces;  
 determining a starting point address and an ending point address of said  at least one of the plurality of discrete pieces of said image  based upon said  the beginning position address of said text image ;  
 recording said  in a look-up table the starting and said  ending point addresses in a look-up table ;  
 linking at least one of the plurality of discrete pieces to at least one of a plurality of external reference materials by recording in the look-up table, along with the starting and ending point addresses of the at least one of the plurality of discrete pieces, a link to the at least one of the plurality of external reference materials, the plurality of external reference materials comprising any of textual, audio, video, and picture information;  
   displaying an image of the cut text;   
 selecting a discrete portion of said  the displayed text image;  
 determining the  a display address of said  the selected discrete portion;  
 converting said  the display address of said  the selected discrete portion to an offset value from said  the beginning position of said text image  address;  
 comparing said  the offset value with said recorded  the starting and ending point addresses of said discrete pieces  recorded in said  the look-up table to identify one of the plurality of discrete pieces;  
 selecting an external reference that corresponds to said look-up table start and end point address  one of the plurality of external reference materials corresponding to the identified one of the plurality of discrete pieces; and  
 retrieving the selected one of the plurality of external reference materials using a recorded link to the selected one of the plurality of external reference materials; and 
 displaying said  the retrieved external reference material.  
 
     
     
       17. The method of  claim 8 , wherein cutting the source material into a plurality of discrete pieces is done manually. 
     
     
       18. The method of  claim 8 , wherein cutting the source material into a plurality of discrete pieces is done automatically. 
     
     
       19. The method of  claim 18 , wherein automatically cutting the source material into a plurality of discrete pieces is done using a grammar parser.  
     
     
       20. The method of  claim 18 , wherein automatically cutting the source material into a plurality of discrete pieces is done without using tags.  
     
     
       21. The method of  claim 18 , wherein automatically cutting the source material into a plurality of discrete pieces is done without reference to any tags which may be located in the source material.  
     
     
       22. The method of  claim 8 , wherein the link is a hyperlink.  
     
     
       23. The method of  claim 8 , wherein the link is an address of the selected one of the plurality of external reference materials.  
     
     
       24. The method of  claim 8 , wherein the link is reference information for retrieving the selected one of the plurality of external reference materials.  
     
     
       25. The method of  claim 8 , wherein determining a display address of the selected discrete portion is done without using tags. 
     
     
       26. The method of  claim 8 , wherein determining a display address of the selected discrete portion is done without reference to any tags which may be located in the source material.  
     
     
       27. The method of  claim 8 , wherein determining a display address of the selected discrete portion is done without reference to any hierarchical information which may be located in the source material.  
     
     
       28. The method of  claim 8 , wherein converting the display address of the selected discrete portion to an offset value from the beginning position address is done without using tags.  
     
     
       29. The method of  claim 8 , wherein converting the display address of the selected discrete portion to an offset value from the beginning position address is done without reference to any tags which may be located in the source material.  
     
     
       30. The method of  claim 8 , wherein converting the display address of the selected discrete portion to an offset value from the beginning position address is done without reference to any hierarchical information which may be located in the source material.  
     
     
       31. The method of  claim 8 , wherein comparing the offset value with the starting and ending point addresses recorded in the look-up table to identify one of the plurality of discrete pieces is done without using tags.  
     
     
       32. The method of  claim 8 , wherein comparing the offset value with the starting and ending point addresses recorded in the look-up table to identify one of the plurality of discrete pieces is done without reference to any tags which may be located in the source material.  
     
     
       33. The method of  claim 8 , wherein comparing the offset value with the starting and ending point addresses recorded in the look-up table to identify one of the plurality of discrete pieces is done without reference to any hierarchical information which may be located in the source material.  
     
     
       34. The method of  claim 8 , wherein retrieving the selected one of the plurality of external reference materials is done using a hyperlink.  
     
     
       35. The method of  claim 8 , wherein retrieving the selected one of the plurality of external reference materials is done using an address of the at least one external reference.  
     
     
       36. The method of  claim 8 , wherein the source material is text based source material.  
     
     
       37. The method of  claim 8 , wherein the source material is image based source material.  
     
     
       38. The method of  claim 8 , wherein the source material is graphic based source material.  
     
     
       39. The method of  claim 8 , wherein the source material is audio based source material.  
     
     
       40. The method of  claim 8 , wherein the source material is video based source material.  
     
     
       41. The method of  claim 8 , wherein the source material is a combination of two or more of text based source material, image based source material, graphic based source material, audio based source material, and video based source material.  
     
     
       42. The method of  claim 8 , wherein the plurality of external reference materials comprises a plurality of text based external reference materials.  
     
     
       43. The method of  claim 8 , wherein the plurality of external reference materials comprises a plurality of image based external reference materials.  
     
     
       44. The method of  claim 8 , wherein the plurality of external reference materials comprises a plurality of graphic based external reference materials.  
     
     
       45. The method of  claim 8 , wherein the plurality of external reference materials comprises a plurality of audio based external reference materials.  
     
     
       46. The method of  claim 8 , wherein the plurality of external reference materials comprises a plurality of video based external reference materials.  
     
     
       47. The method of  claim 8 , wherein at least one of the plurality of external reference materials is a combination of two or more of text based external reference material, image based external reference material, graphic based external reference material, audio based external reference material, and video based external reference material.  
     
     
       48. The method of  claim 8 , wherein linking at least one of the plurality of discrete pieces is done manually.  
     
     
       49. The method of  claim 8 , wherein linking at least one of the plurality of discrete pieces is done automatically.  
     
     
       50. The method of  claim 8 , wherein the electronic database is an electronic relational database.  
     
     
       51. The method of  claim 8 , wherein the electronic database is an electronic file.  
     
     
       52. The method of  claim 8 , wherein the electronic database is an electronic text.  
     
     
       53. The method of  claim 8 , wherein the beginning position address is a beginning location of the source material in the electronic database.  
     
     
       54. The method of  claim 53 , wherein each starting point address is a starting location of at lest one of the plurality of discrete pieces based upon the beginning location of the source material.  
     
     
       55. The method of  claim 53 , wherein the ending point address is an ending location of at least one of the plurality of discrete pieces based upon the beginning location of the source material.  
     
     
       56. The system of  claim 1 , wherein cutting the source material into a plurality of discrete pieces is done manually.  
     
     
       57. The system of  claim 1 , wherein cutting the source material into a plurality of discrete pieces is done automatically.  
     
     
       58. The system of  claim 57 , wherein automatically cutting the source material into a plurality of discrete pieces is done using a grammar parser.  
     
     
       59. The system of  claim 57 , wherein automatically cutting the source material into a plurality of discrete pieces is done without using tags.  
     
     
       60. The system of  claim 57 , wherein automatically cutting the source material into a plurality of discrete pieces is done without reference to any tags which may be located in the source material.  
     
     
       61. The system of  claim 1 , wherein the link is a hyperlink.  
     
     
       62. The system of  claim 1 , wherein the link is an address of the selected one of the plurality of external reference materials.  
     
     
       63. The system of  claim 1 , wherein the link is reference information for retrieving the selected one of the plurality of external reference materials.  
     
     
       64. The system of  claim 1 , wherein determining a display address of the selected discrete portion is done without using tags. 
     
     
       65. The method of  claim 1 , wherein determining a display address of the selected discrete portion is done without reference to any tags which may be located in the source material.  
     
     
       66. The system of  claim 1 , wherein determining a display address of the selected discrete portion is done without reference to any hierarchical information which may be located in the source material.  
     
     
       67. The system of  claim 1 , wherein converting the display address of the selected discrete portion to an offset value from the beginning position address is done without using tags.  
     
     
       68. The system of  claim 1 , wherein converting the display address of the selected discrete portion to an offset value from the beginning position address is done without reference to any tags which may be located in the source material.  
     
     
       69. The system of  claim 1 , wherein converting the display address of the selected discrete portion to an offset value from the beginning position address is done without reference to any hierarchical information which may be located in the source material.  
     
     
       70. The system of  claim 1 , wherein comparing the offset value with the starting and ending point addresses recorded in the look-up table to identify one of the plurality of discrete pieces is done without using tags.  
     
     
       71. The system of  claim 1 , wherein comparing the offset value with the starting and ending point addresses recorded in the look-up table to identify one of the plurality of discrete pieces is done without reference to any tags which may be located in the source material.  
     
     
       72. The system of  claim 1 , wherein comparing the offset value with the starting and ending point addresses recorded in the look-up table to identify one of the plurality of discrete pieces is done without reference to any hierarchical information which may be located in the source material.  
     
     
       73. The system of  claim 1 , wherein retrieving the selected one of the plurality of external reference materials is done using a hyperlink.  
     
     
       74. The system of  claim 1 , wherein retrieving the selected one of the plurality of external reference materials is done using an address of the at least one external reference.  
     
     
       75. The system of  claim 1 , wherein the source material is text based source material.  
     
     
       76. The system of  claim 1 , wherein the source material is image based source material.  
     
     
       77. The system of  claim 1 , wherein the source material is graphic based source material.  
     
     
       78. The system of  claim 1 , wherein the source material is audio based source material.  
     
     
       79. The system of  claim 1 , wherein the source material is video based source material.  
     
     
       80. The system of  claim 1 , wherein the source material is a combination of two or more of text based source material, image based source material, graphic based source material, audio based source material, and video based source material.  
     
     
       81. The system of  claim 1 , wherein the plurality of external reference materials comprises a plurality of text based external reference materials.  
     
     
       82. The system of  claim 1 , wherein the plurality of external reference materials comprises a plurality of image based external reference materials.  
     
     
       83. The system of  claim 1 , wherein the plurality of external reference materials comprises a plurality of graphic based external reference materials.  
     
     
       84. The system of  claim 1 , wherein the plurality of external reference materials comprises a plurality of audio based external reference materials.  
     
     
       85. The system of  claim 1 , wherein the plurality of external reference materials comprises a plurality of video based external reference materials.  
     
     
       86. The system of  claim 1 , wherein at least one of the plurality of external reference materials is a combination of two or more of text based external reference material, image based external reference material, graphic based external reference material, audio based external reference material, and video based external reference material.  
     
     
       87. The system of  claim 1 , wherein linking at least one of the plurality of discrete pieces is done manually.  
     
     
       88. The system of  claim 1 , wherein linking at least one of the plurality of discrete pieces is done automatically.  
     
     
       89. The system of  claim 1 , wherein the electronic database is an electronic relational database.  
     
     
       90. The system of  claim 1 , wherein the electronic database is an electronic file.  
     
     
       91. The system of  claim 1 , wherein the electronic database is electronic text.  
     
     
       92. The system of  claim 1 , wherein the beginning position address is a beginning location of the source material in the electronic database.  
     
     
       93. The system of  claim 92 , wherein each starting point address is a starting location of at least one of the plurality of discrete pieces based upon the beginning location of the source material.  
     
     
       94. The method of  claim 92 , wherein each ending point address is an ending location of at least one of the plurality of discrete pieces based upon the beginning location of the source material.  
     
     
       95. A system for linking textual source material to external reference material for display, the system comprising:
   means for determining a beginning position address of textual source material stored in an electronic database;        means for cutting the textual source material into a plurality of discrete pieces;        means for determining a starting point address and an ending point address of at least one of the plurality of discrete pieces based upon the beginning position address;        means for recording in a look-up table the starting and ending point addresses;        means for linking at least one of the plurality of discrete pieces to at least one of a plurality of external reference materials by recording in the look-up table, along with the starting and ending point addresses of the at least one of the plurality of discrete pieces, a link to the at least one of the plurality of external reference materials, the plurality of external reference materials comprising any of textual, audio, video, and picture information;        means for displaying an image of the textual source material;        means for selecting a discrete portion of the displayed textual source material image;        means for determining a display address of the selected discrete portion;        means for converting the display address of the selected discrete portion to an offset value from the beginning position address;        means for comparing the offset value with the starting and ending point addresses recorded in the look-up table to identify one of the plurality of discrete pieces;        means for selecting one of the plurality of external reference materials corresponding to the identified one of the plurality of discrete pieces;        means for retrieving the selected one of the plurality of external reference materials using a recorded link to the selected one of the plurality of external reference materials; and        means for displaying the retrieved external reference material.      
     
     
       96. A method for linking textual source material to external reference materials for display, the method comprising the steps of:
   determining a beginning position address of textual source material stored in an electronic database;        cutting the textual source material into a plurality of discrete pieces;        determining a starting point address and an ending point address of at least one of the plurality of discrete pieces based upon the beginning position address;        recording in a look up table the starting and ending point addresses;        linking at least one of the plurality of discrete pieces to at least one of a plurality of external reference materials by recording in the look-up table, along with the starting and ending point addresses of the at least one of the plurality of discrete pieces, a link to the at least one of the plurality of external reference materials, the plurality of external reference materials comprising any of textual, audio, video, and picture information;        displaying an image of the textual source material;        selecting a discrete portion of the displayed textual source material image;        determining a display address of the selected discrete portion;        converting the display address of the selected discrete portion to an offset value from the beginning position address;        comparing the offset value with the starting and ending point addresses recorded in the look-up table to identify one of the plurality of discrete pieces;        selecting one of the plurality of external reference materials corresponding to the identified one of the plurality of discrete pieces;        retrieving the selected one of the plurality of external reference materials using a recorded link to the selected one of the plurality of external reference materials; and        displaying the retrieved external reference material.