Methods and apparatus for the composition and communication of digital composition coded multisensory messages (DCC MSMs)

Abstract

Disclosed are methods and apparatus for composing and communicating Digital Composition Coded Multisensory Messages (DCC MSMs). The present invention also relates to the design, composition, creation, and communication of the multisensory messages. Multisensory messages, while rich in content and meaning, are to be composable from a great variety of platforms, from cell phones to mainframes.

The invention claimed is: 
     
       1. A method for communicating digital composition coded or structured multisensory messages in a communications system comprising a user input device, a rich message server, a receiving device and network connections, the method comprising:
 at a user input device:
 receiving a set of user input commands using coded instructions for composing a structured multisensory message from one or more multisensory objects, 
 wherein the multisensory objects include a combination of live sensor feeds or stored data residing at the user input device (user data) or at a remote server (server-side data), 
 wherein the user-input commands are created using an intuitive authoring tool and including one or more shorthand instruction codes drawn from a digital composition codebook, 
 wherein the digital composition codebook is a construction regimen of instruction codes which are system- or user-predefined, pre-recorded, or are formulated in real-time by the user of the user input device through a combination of buttons, actions, or gestures, and are learned by a combination of the user input device and the rich message server, utilizing signal processing and machine learning techniques, 
 wherein the one or more shorthand codes are composition instructions that define how the one or more multisensory objects are to be composed for presentation at the receiving device; and 
 encoding and communicating the combination of user input commands, user data and references to server-side data to the rich message server; 
 
 at the rich message server, composing, storing and transmitting a structured multisensory message as defined by the user, comprising:
 receiving and decoding the user input commands, the user data, and any references to server-side data from the remote user input device; 
 interpreting or learning a meaning of the user input commands having one or more shorthand instruction codes through context or training; 
 organizing the combination of multisensory objects in or referenced by the user data and server-side data; 
 compositing the one or more multisensory objects into a structured multisensory message in accordance with the user input commands for presentation on the receiving device and consistent with the capabilities of the receiving device and network connections; and 
 encoding and communicating the structured multisensory message to the receiving device; 
 
 at a receiving device, presenting a rich sensory experience as defined by the user, comprising:
 receiving and decoding a structured multisensory message from the rich message server; and 
 converting and rendering the structured multisensory message into a rich sensory experience as defined by the user, scaled to the capabilities of the receiving device. 
 
 
     
     
       2. The method of  claim 1 , wherein at least one of the one or more multisensory objects is stored remote from the user input device. 
     
     
       3. The method of  claim 1 , wherein the user-input command is user-specific and the interpretation of the user-input command is learned by a combination of user input device and the rich message server, utilizing signal processing and machine learning techniques. 
     
     
       4. The method of  claim 1 , wherein the rich message server encodes the composited multisensory message for presentation on the receiving device. 
     
     
       5. The method of  claim 1 , wherein the user-input commands are for the presentation of multisensory objects that represent more than two different senses. 
     
     
       6. A method of using a user device to send a digital composition coded or structured multisensory message, comprising:
 receiving at the user device as input from a user a set of user input commands using coded instructions for composing a structured multisensory message from one or more multisensory objects, 
 wherein the multisensory objects include a combination of live sensor feeds or stored data residing at the user device (user data) or at a remote server (server-side data), 
 wherein the user-input commands are created using an intuitive authoring tool and including one or more shorthand instruction codes drawn from a digital composition codebook, 
 wherein the digital composition codebook is a construction regimen of instruction codes which are system- or user-predefined, pre-recorded, or are formulated in real-time by the user of the user device through a combination of buttons, actions, or gestures, and are learned by either the user input device or a rich message server, 
 wherein the one or more shorthand codes are composition instructions that define how the one or more multisensory objects are to be composed for presentation in the structured multisensory message; and 
 encoding and transmitting the combination of user input commands, user data and server-side data to a rich message server for interpretation of the user-input commands, editing of the one or more multisensory objects in accordance with the user input commands, composition of the structured multi sensory message from the one or more multisensory objects, and encoding of the structured multisensory message prior to the presentation of the multisensory message to an end user. 
 
     
     
       7. The method of  claim 6 , wherein the user-input commands are received using tactile input by the user. 
     
     
       8. The method of  claim 6 , wherein the user-input commands are received using a camera, signal processing and machine learning techniques. 
     
     
       9. The method of  claim 6 , wherein the user-input commands are received using voice commands, signal processing and machine learning techniques. 
     
     
       10. The method of  claim 6 , wherein the user-input commands are received using eye movements, signal processing and machine learning techniques. 
     
     
       11. The method of  claim 6 , wherein the user-input commands are received using handwritings or drawings, signal processing and machine learning techniques. 
     
     
       12. The method of  claim 6 , wherein the user-input commands are received using finger movements, signal processing and machine learning techniques. 
     
     
       13. The method of  claim 6 , wherein the one or more shorthand codes are used to integrate multiple multisensory objects for presentation. 
     
     
       14. The method of  claim 6 , wherein the one or more shorthand codes are user-specific and the interpretation of the one or more shorthand codes is learned by a combination of the user input device and the rich message server. 
     
     
       15. The method of  claim 6 , wherein at least one of the one or more multisensory objects is stored remote from the user input device. 
     
     
       16. The method of  claim 6 , wherein the user-input commands are for the presentation of multisensory objects that represent more than two different senses. 
     
     
       17. A method of using a rich message server for serving digital composition coded or structured multisensory messages, comprising:
 receiving and decoding user-input commands for the composition of one or more multisensory objects into a structured multisensory message for presentation, 
 wherein the user-input commands are created by a user using an intuitive authoring tool and including one or more shorthand instruction codes drawn from a digital composition codebook, 
 wherein the digital composition codebook is a construction regimen of instruction codes which are system- or user-predefined, pre-recorded, or formulated in real-time by a user through a combination of buttons, actions, or gestures, and are learned by a combination of a user input device or the rich message server, utilizing signal processing and machine learning techniques, 
 wherein the one or more shorthand codes are composition instructions that define how the one or more multisensory objects are to be composed for presentation in the structured multisensory message; 
 interpreting or learning a meaning of the user-input commands by context or training; 
 compositing the one or more multisensory objects into a structured multisensory message in accordance with the user input commands consistent with the capabilities of a receiving device and network connections to be used; and 
 encoding and forwarding the composited multisensory message for presentation on an end-user's device. 
 
     
     
       18. The method of  claim 17 , wherein the one or more shorthand codes are used to integrate multiple multisensory objects for presentation. 
     
     
       19. The method of  claim 17 , wherein the rich message server learns the interpretation of the one or more shorthand codes. 
     
     
       20. The method of  claim 17 , wherein the rich message server encodes the composited multisensory message for presentation on the end-user's device. 
     
     
       21. The method of  claim 17 , wherein the user-input commands are for the presentation of multisensory objects that represent more than two different senses. 
     
     
       22. A method of using an end-user device to present a digital composition coded or structured multisensory message, comprising:
 receiving and decoding a composited multisensory message from a rich message server, the composited multisensory message being composited at the rich message server from one or more multisensory objects according to commands input by a user of a sending device, 
 wherein the commands are created using an intuitive authoring tool and including one or more shorthand instruction codes drawn from a digital composition codebook, 
 wherein the digital composition codebook is a construction regimen of instruction codes which are system- or user-predefined, pre-recorded or are formulated in real-time by the user of the sending device through a combination of buttons, actions, or gestures, and are learned by a combination of the sending device and a rich message server, utilizing signal processing and machine learning techniques, 
 wherein the one or more shorthand instruction codes are composition instructions that define how the one or more multi sensory objects are to be composed at the rich message server and composited for presentation at the end-user device; and 
 converting and rendering the structured multisensory message into a rich sensory experience as defined by the user of the sending device, scaled to the capabilities of the end-user device. 
 
     
     
       23. The method of  claim 22 , further comprising editing the composited multisensory message for presentation on the end-user device according to limitations and user settings of the end-user device. 
     
     
       24. The method of  claim 22 , wherein at least one of the one or more multisensory objects is available from the end-user device. 
     
     
       25. The method of  claim 24 , wherein at least one of the one or more multisensory objects is stored at the end-user device. 
     
     
       26. The method of  claim 24 , wherein at least one of the one or more multisensory objects is a live camera feed. 
     
     
       27. The method of  claim 22 , wherein the multisensory objects to be presented represent more than two different senses. 
     
     
       28. A system for communicating digital composition coded or structured multisensory messages, comprising:
 a user device for receiving a set of user input commands using coded instructions for composing a structured multisensory message from one or more multisensory objects, 
 wherein the multisensory objects include a combination of live sensor feeds or stored data residing at the user device (user data) or at a remote server (server-side data), 
 wherein the user-input commands are created using an intuitive authoring tool and including one or more shorthand instruction codes drawn from a digital composition codebook, 
 wherein the digital composition codebook is a construction regimen of instruction codes which are system- or user-predefined, pre-recorded, or are formulated in real-time by the user of the user device through a combination of buttons, actions, or gestures, and are learned by a combination of the user input device and the rich message server, utilizing signal processing and machine learning techniques, 
 wherein the one or more shorthand codes are composition instructions that define how the one or more multisensory objects are to be composed for presentation at a receiving device, and for encoding and communicating the combination of user input commands, user data and references to server-side data to a rich message server; 
 a rich message server for receiving and decoding the user input commands, the user data, and any references to server-side data from the remote user device, for interpreting or learning a meaning of the user input commands having one or more shorthand instruction codes through context or training, for organizing the combination of multisensory objects in or referenced by the user data and server-side data, for compositing the one or more multisensory objects into a structured multisensory message in accordance with the user input commands for presentation on the receiving device and consistent with the capabilities of the receiving device and network connections; and for encoding and communicating the structured multisensory message to the receiving device; and 
 a receiving device for receiving and decoding a structured multisensory message from the rich message server and for converting and rendering the structured multi sensory message into a rich sensory experience as defined by the user, scaled to the capabilities of the receiving device. 
 
     
     
       29. The system of  claim 28 , wherein the rich message server is configured to store at least one of the one or more multisensory objects. 
     
     
       30. The system of  claim 28 , wherein the user-input commands are user-specific and the interpretation of the user-input commands is learned by the rich message server. 
     
     
       31. The system of  claim 28 , wherein the rich message server is configured to edit the presentation of the composited multisensory message according to the capabilities of the receiving device. 
     
     
       32. The system of  claim 28 , wherein the multisensory objects to be presented represent more than two different senses. 
     
     
       33. A computer-implemented method for generating structured multisensory messages, the method comprising:
 receiving one or more of user input commands and one or more multisensory objects, a multisensory object comprising a digital representation of one or more sensory signals;   decoding the user input commands based on a plurality of instruction codes of a digital composition codebook; and   compositing a structured multisensory message from the one or more multisensory objects by organizing the one or more multisensory objects into the structured multisensory message, the structured multisensory message composed based on the user input commands and being consistent with at least one of capability of a receiving device and network conditions.   
     
     
       34. The method of claim 33, wherein a multisensory object includes a combination of one or more live sensor feeds and sensory data stored at a computer server. 
     
     
       35. The method of claim 33, wherein compositing the structured multisensory message comprising:
 multiplexing a first video clip with an audio clip into a composite video;   fading in the composite video for a predetermined period of time;   fading out the composite video and fading in a second video clip; and   generating the structured multisensory message, the structured multisensory message comprising instructions for combining the first video clip, the audio clip and the second video clip and presenting the structured multisensory message at the receiving device.   
     
     
       36. The method of claim 33, wherein compositing the structured multisensory message further comprising:
 compressing a live sensor feed of media represented by one or more sensory signals;   multiplexing the compressed live sensor feed of media with a stored media clip into a composite media;   generating the structured multisensory message, the structured multisensory message comprising instructions for combining the live sensor feed of media with the stored media clip and presenting the structured multisensory message at the receiving device.   
     
     
       37. The method of claim 33, wherein the digital composition book includes a plurality of instruction codes that define how to encode the one or more multisensory objects into the structured multisensory message for presentation at the receiving device. 
     
     
       38. A computer-implemented method of communicating user input for communicating structured multisensory messages, the method comprising:
 receiving one or more user input commands and one or more multisensory objects, a multisensory object comprising a digital representation of one or more sensory signals;   compositing and encoding the one or more multisensory objects into a structured multisensory message, the structured multisensory message being consistent with at least one of capability of a receiving device and network conditions; and   transmitting the structured multisensory message to the receiving device, wherein the receiving device is adapted to present the multisensory message in a scaled format defined by the capability of the receiving device.   
     
     
       39. The method of claim 38, wherein the user input commands are received using at least one of following:
 tactile input by the user;   live feed from a camera;   voice commands;   hand configurations of the user;   eye movements of the user;   handwritings of the user;   finger or hand movements of the user;   facial movements of the user; and   body movements of the user.   
     
     
       40. The method of claim 38, further comprising decoding the user input commands using a signal processing technique. 
     
     
       41. The method of claim 38, further comprising decoding the user input commands using a machine learning technique. 
     
     
       42. The method of claim 38, wherein transmitting the structured multisensory message to the receiving device comprises:
 responsive to the structured multisensory message comprising stored multisensory objects, transmitting instructions on decoding the structured multistory message on the receiving device.   
     
     
       43. The method of claim 42, wherein the instructions for decoding the structured multisensory message comprise instructions for combining and presenting the stored multisensory objects on the receiving device. 
     
     
       44. The method of claim 38, wherein transmitting the structured multisensory message to the receiving device comprises:
 responsive to the structured multisensory message comprising one or more stored multisensory objects and at least one live sensor feed of media, transmitting compressed live sensor feed of media and instructions on decoding the structured multisensory message on the receiving device.   
     
     
       45. The method of claim 44, wherein the instructions for decoding the structured multisensory message comprise instructions for:
 decompressing the compressed live sensor feed of media;   combining decompressed live sensor feed of media with the one or more stored multisensory objects into a composite media; and   transmitting the composite media to the receiving device.   
     
     
       46. A computer system for generating structured multisensory messages, the system comprising:
 a computer processor configured to perform steps comprising:
 receiving one or more user input commands and one or more multisensory objects, a multisensory object comprising a digital representation of one or more sensory signals; 
 decoding the user input commands based on a plurality of instruction codes of a digital composition codebook; and 
 compositing a structured multisensory message from the one or more multisensory objects by organizing the one or more multisensory objects into the structured multisensory message, the structured multisensory message composed based on the user input commands and being consistent with at least one of capability of a receiving device and network conditions. 
   
     
     
       47. The system of claim 46, wherein a multisensory object includes a combination of one or more live sensor feeds and sensory data stored at a computer server. 
     
     
       48. The system of claim 46, wherein compositing the structured multisensory message comprises:
 multiplexing a first video clip with an audio clip into a composite video;   fading in the composite video for a predetermined period of time;   fading out the composite video and fading in a second video clip; and   generating the structured multisensory message, the structured multisensory message comprising instructions for combining the first video clip, the audio clip and the second video clip and presenting the structured multisensory message at the receiving device.   
     
     
       49. The system of claim 46, wherein compositing the structured multisensory message comprises:
 compressing a live sensor feed of media represented by one or more sensory signals;   multiplexing the compressed live sensor feed of media with a stored media clip into a composite media;   generating the structured multisensory message, the structured multisensory message comprising instructions for combining the live sensor feed of media with the stored media clip and presenting the structured multisensory message at the receiving device.   
     
     
       50. The system of claim 46, wherein the digital composition book includes a plurality of instruction codes that define how to encode the one or more multisensory objects into the structured multisensory message for presentation at the receiving device. 
     
     
       51. A computer system for communicating user input for communicating structured multisensory messages, the system comprising:
 a computer processor configured to perform steps comprising:
 receiving one or more user input commands and one or more multisensory objects, a multisensory object comprising a digital representation of one or more sensory signals; 
 compositing and encoding the one or more multisensory objects into a structured multisensory message, the structured multisensory message consistent with at least one of capability of a receiving device and network conditions; and 
 transmitting the structured multisensory message to the receiving device, wherein the receiving device is adapted to present the multisensory message in a scaled format defined by the capability of the receiving device. 
   
     
     
       52. The system of claim 51, wherein the user input commands are received using at least one of following:
 tactile input by the user;   live feed from a camera;   voice commands;   hand configurations of the user;   eye movements of the user;   handwritings of the user;   finger or hand movements of the user;   facial movements of the user; and   body movements of the user.   
     
     
       53. The system of claim 51, further comprising decoding the user input commands using a signal processing technique. 
     
     
       54. The system of claim 51, further comprising decoding the user input commands using a machine learning technique. 
     
     
       55. The system of claim 51, wherein transmitting the structured multisensory message to the receiving device comprises:
 responsive to the structured multisensory message comprising stored multisensory objects, transmitting instructions on decoding the structured multistory message on the receiving device.   
     
     
       56. The system of claim 55, wherein the instructions for decoding the structured multisensory message comprise instructions for combining and presenting the stored multisensory objects on the receiving device. 
     
     
       57. The system of claim 51, wherein transmitting the structured multisensory message to the receiving device comprises:
 responsive to the structured multisensory message comprising one or more stored multisensory objects and at least one live sensor feed of media, transmitting compressed live sensor feed of media and instructions on decoding the structured multisensory message on the receiving device.   
     
     
       58. The system of claim 57, wherein the instructions for decoding the structured multisensory message comprise instructions for:
 decompressing the compressed live sensor feed of media;   combining decompressed live sensor feed of media with the one or more stored multisensory objects into a composite media; and   transmitting the composite media to the receiving device.   
     
     
       59. An apparatus for encoding user input commands for structured multisensory messages, the apparatus comprising:
 a computer processor for performing steps comprising:
 receiving one or more user input commands and one or more multisensory objects, a multisensory object comprising a digital representation of one or more sensory signals; 
 encoding the set of user input commands and references of the one or more multisensory objects into user input data; and 
   providing the user input data to a computer server to generate structured multisensory messages.   
     
     
       60. The user input device of claim 59, wherein the user input commands are received using at least one of following:
 tactile input by the user;   live feed from a camera;   voice commands;   hand configurations of the user;   eye movements of the user;   handwritings of the user;   finger or hand movements of the user;   facial movements of the user; and   body movements of the user.   
     
     
       61. The apparatus of claim 59, wherein the user input commands are created using an authoring tool having one or more shorthand instruction codes included in a digital composition codebook. 
     
     
       62. The apparatus of claim 59, wherein the computer processor is further for:
 receiving the structured multisensory messages from the computer server and presenting the structured multisensory messages to a user.   
     
     
       63. The apparatus of claim 62, wherein presenting a structured multisensory message comprises:
 rendering the structured multisensory message into a rich message based on the user commands;   scaling the rich message responsive to the capabilities of a receiving device; and   presenting the rich messages to a user of the receiving device.   
     
     
       64. The apparatus of claim 62, wherein presenting a structured multisensory message further comprises editing the structured multisensory message for presentation on the receiving device responsive to limitations and user settings of the receiving device. 
     
     
       65. The apparatus of claim 62, wherein presenting a structured multisensory message comprises:
 decompressing compressed live sensor feed of media;   combining the decompressed live sensor feed of media with one or more stored multisensory objects into a composite media; and   presenting the composite media to a user of the receiving device.   
     
     
       66. The user input device of claim 59, wherein the user input commands are received using at least two of following:
 tactile input by the user;   live feed from a camera;   voice commands;   hand configurations of the user;   eye movements of the user;   handwritings of the user;   finger or hand movements of the user;   facial movements of the user; and   body movements of the user.   
     
     
       67. The user input device of claim 59, wherein the user input commands are received using at least three of following:
 tactile input by the user;   live feed from a camera;   voice commands;   hand configurations of the user;   eye movements of the user;   handwritings of the user;   finger or hand movements of the user;   facial movements of the user; and   body movements of the user.   
     
     
       68. The user input device of claim 59, wherein the user input commands are received using at least four of following:
 tactile input by the user;   live feed from a camera;   voice commands;   hand configurations of the user;   eye movements of the user;   handwritings of the user;   finger or hand movements of the user;   facial movements of the user; and   body movements of the user.   
     
     
       69. The user input device of claim 59, wherein the user input commands are received using at least five of following:
 tactile input by the user;   live feed from a camera;   voice commands;   hand configurations of the user;   eye movements of the user;   handwritings of the user;   finger or hand movements of the user;   facial movements of the user; and   body movements of the user.