Ergonomic man-machine interface incorporating adaptive pattern recognition based control system

Abstract

An adaptive interface for a programmable system, for predicting a desired user function, based on user history, as well as machine internal status and context. The apparatus receives an input from the user and other data. A predicted input is presented for confirmation by the user, and the predictive mechanism is updated based on this feedback. Also provided is a pattern recognition system for a multimedia device, wherein a user input is matched to a video stream on a conceptual basis, allowing inexact programming of a multimedia device. The system analyzes a data stream for correspondence with a data pattern for processing and storage. The data stream is subjected to adaptive pattern recognition to extract features of interest to provide a highly compressed representation which may be efficiently processed to determine correspondence. Applications of the interface and system include a VCR, medical device, vehicle control system, audio device, environmental control system, securities trading terminal, and smart house. The system optionally includes an actuator for effecting the environment of operation, allowing closed-loop feedback operation and automated learning.

What is claimed is: 
     
       1. A programmable control responsive to an user input and a signal received from a signal source, comprising:
 a controller, operating according to a predetermined program, for receiving the user input and the signal and producing a control output, said controller producing a multivalued characterization of the signal with respect to at least one parameter;   a memory for storing data relating to an activity of the user;   a data processing system for predicting a most probable action of a user based on said stored data relating to said activity of the user and said characterized signal; and   a user feedback data presenting system comprising a display device for presentation of a sequence of programming options to the user, including said most probable action of the user, in a plurality of display images, each display image differing in available programming options.   
     
     
       2. The programmable control according to  claim 1  being for performing an action based on user input and an information content of a signal received from a signal source, further comprising:
 a user controlled direct manipulation-type input device, associated with said display device, having a device output, said device output being the user input; 
 a plant capable of performing a physical action, being responsive to an actuator signal; and 
 said controller, being for receiving data from said device output of said input device and a signal received from a signal source, and displaying user feedback data on said display device, 
 said user feedback data comprising a presentation of a sequence of programming options to the user on said display device, including said most probable action of the user, in a plurality of display images, each display image differing in available programming options, said sequence of programming options including at least one sequence of options sufficient to define an operable control program, and a presentation of additional programming options if said control program is not operable. 
 
     
     
       3. The system according to  claim 1 , being for processing a program comprising program material, in response to a viewer input, comprising:
 a user input processing system for determining a viewer preference;   said controller comprises a program material processing system for characterizing the program material, as said characterized signal, based on its content;   a correlator for correlating said characterized content of the program material with said determined viewer preference to produce a correlation index; and   a processor, selectively processing the program material based on said correlation index,   wherein said viewer preference is an input to said data processing system, and   said data processing system comprises said correlator.   
     
     
       4. The system according to  claim 3 , wherein said program material is encrypted, further comprising:
 a decryption system for decrypting the program material if it is selected to produce unencrypted program material and optionally an associated decryption event;   a memory for storing data relating to the occurrence of said decryption event; and   a central database for storing data relating to the occurrence of said decryption event in association with data relating to the viewer.   
     
     
       5. The system according to  claim 3 , wherein:
 said user input processing system monitors a pattern of user activity and predicts a viewer preference;   said program material processing system comprises:   a processor for preprocessing the program material to produce a reduced data flow information signal substantially retaining information relating to an abstract information content of the program material and selectively eliminating data not relating to said abstract information content of the program material and for characterizing said information signal based on said abstract information content; and   a comparing system for determining if said correlation index is indicative of a probable high correlation between said characterization of said information signal and said viewer preference and causing said stored program material to be processed by said processor based on said determination.   
     
     
       6. The system according to  claim 3 , wherein said processor comprises an image program material storage and retrieval system. 
     
     
       7. The system according to  claim 3 , further comprising a memory for storing a characterization of the program material; an input for receiving a feedback signal from the viewer indicating a degree of agreement with said correlation index determination, wherein said feedback signal and said stored characterization are used by said user input processing system to predict a new viewer preference. 
     
     
       8. The system according to  claim 3 , wherein said a program material processing system correlates and characterizes image information, further comprising:
 means for storing template data;   means for storing the image data;   means for generating a plurality of addressable domains from the stored image data, each of the domains representing a different portion of the image information;   means for creating, from the stored image data, a plurality of addressable mapped ranges corresponding to different subsets of the stored image data, the creating means including means for executing, for each of the mapped ranges, a procedure upon the one of the subsets of the stored image data which corresponds to the mapped range;   means for assigning identifiers to corresponding ones of the mapped ranges, each of the identifiers specifying for the corresponding mapped range an address of the corresponding subset of stored image data;   means for selecting, for each of the domains, the one of the mapped ranges which most closely corresponds according to predetermined criteria;   means for representing at least a portion of the image information as a set of the identifiers of the selected mapped ranges; and   means for selecting, from the stored templates, a template which most closely corresponds to the set of identifiers representing the image information.   
     
     
       9. The system according to  claim 8  wherein said correspondence of a template and the set of identifiers is determined by a processor executing a predetermined program for executing an algorithm selected from at least one of the group consisting of algorithms for selecting a minimum Hausdorff distance between the mapped range and the domain, for selecting the highest cross-correlation of the mapped range with the domain, and for selecting the lowest mean square error of the difference between the mapped range and the domain. 
     
     
       10. The system according to  claim 9  wherein said correspondence of a template trial and the set of identifiers is determined by selecting, for each domain, the mapped range with the minimum modified Hausdorff distance calculated as D[db,mrb]+D[1−db,1−mrb], where D is a distance calculated between a pair of sets of data each representative of an image, db is a domain, mrb is a mapped range, I−db is the inverse of a domain, and I-mrb is an inverse of a mapped range. 
     
     
       11. The system according to  claim 8 , wherein said means for representing further comprises means for:
 (a) determining a feature of interest of the image data,   (b) selecting a mapped range corresponding to the feature of interest,   (c) storing the identifiers of the selected mapped range,   (d) selecting a further mapped range corresponding to a portion of image data having a predetermined relationship to the feature of interest, and   (e) storing the identifiers of the further mapped range.   
     
     
       12. The system according to  claim 8 , wherein said image data comprises data having three associated dimensions obtained by an image processor executing an algorithm selected from the group consisting of an algorithm for synthesizing a three dimensional representation based on a machine based prediction derived from two dimensional image data, an algorithm for synthesizing a three dimensional representation derived from a time series of pixel images, and an algorithm for synthesizing a three dimensional representation based on a image data representing a plurality of parallax views having at least two dimensions, said set of identifiers representing data in each of the three associated dimensions. 
     
     
       13. A method of programming a device, comprising the steps of:
 providing an input for user commands and feedback;   storing information relating to the user commands and feedback in a memory;   providing a data input for external information;   predicting a subsequent user command based on the stored information relating to user commands and feedback, a status of the device, and external information from the data input;   presenting the predicted user command to the user;   accepting feedback from the user relating to the predicted subsequent user command;   modifying the predicted subsequent user command based on at least the feedback; and   executing the modified predicted user command.   
     
     
       14. The method according to  claim 13 , further comprising the step of identifying a user and storing an user identifier with said stored information relating to the user commands;
 said predicting step predicting a subsequent user command based on the stored information relating to user identifier, commands and feedback, a status of the device, and external information from the data input.   
     
     
       15. A programmable device comprising:
 a user input for receiving an input variable having a path between a first input state and a second input state, said user input comprising:
 path dependent user data having transitional path information between said first input state and said second input state; and 
 path independent user data comprising information about at least one of said first state and said second state; 
 a filter, separating sad path dependent user data as user characterization data and said path independent user data as instructions; 
 a memory for storing said user characterization data; 
 a processor for executing said instructions; and 
 a feedback device, presenting information relating to said instructions and said stored user characterization data. 
   
     
     
       16. The device according to  claim 15 , further comprising:
 a hierarchical command structure of said processor, said command structure having commands of different function; and   means for predicting a probability of execution of a plurality of commands based on said input,   said feedback device presenting commands based on at least said predicted probabilities.   
     
     
       17. The programmable control according to  claim 1 , wherein the signal comprises image information. 
     
     
       18. The method according to  claim 13 , wherein said stored information comprises a set of weights of a predictive algorithm. 
     
     
       19. The method according to  claim 13 , herein said external information comprises a perceptual data stream. 
     
     
       20. The programmable device according to  claim 15 , wherein a format of information presented from said feedback device is based on said path dependent user data. 
     
     
       21. A method for sending targeted content to a mobile device, the method comprising:
 identifying a user profile associated with a mobile device based on information received from the mobile device;   receiving a position of the mobile device, wherein the mobile device calculated the position using a positioning system;   selecting, based on the user profile and the position of the mobile device, customized information; and   sending, for display on the mobile device, the customized information to the mobile device.   
     
     
       22. The method of claim 21, wherein the information received from the mobile device includes preferences information, and wherein selecting further comprises:
 selecting, based on the preferences information, the customized information.   
     
     
       23. The method of claim 21, wherein the customized information comprises an advertisement. 
     
     
       24. The method of claim 23, wherein the advertisement concerns food. 
     
     
       25. The method of claim 23, wherein the advertisement concerns fuel. 
     
     
       26. The method of claim 23, wherein the advertisement concerns lodging. 
     
     
       27. The method of claim 21, wherein the positioning system is a geographic positioning system (GPS). 
     
     
       28. The method of claim 21, wherein the positioning system is a cellular base station. 
     
     
       29. The method of claim 21, wherein the information received from the mobile device includes user preferences information, and wherein sending further comprises:
 sending one or more customized billboards to the mobile device dependent on demographics of a user associated with the user profile, dependent on the position of the mobile device, and dependent on the user preferences information.   
     
     
       30. The method of claim 21, wherein the information received from the mobile device includes an itinerary, the method further comprising:
 generating the customized information dependent on the itinerary.   
     
     
       31. The method of claim 21, further comprising communicating with the mobile device based on the position of the mobile device and an itinerary for vehicle travel, wherein the itinerary relates to the customized information. 
     
     
       32. The method of claim 21, wherein the customized information is referred to as first customized information, and the mobile device is referred to as a first mobile device, the method further comprising:
 sending, for display on a second mobile device, second customized information.   
     
     
       33. The method of claim 32, the positioning system is referred to as a first positioning system, the method further comprising:
 receiving a position of the second mobile device, wherein the second mobile device calculated the position of the second mobile device using a second positioning system; and   selecting, based on the position of the second mobile device, the second customized information.   
     
     
       34. The method of claim 33, wherein the user profile is referred to as a first user profile, and wherein the second customized information includes advertising information, the method further comprising:
 identifying a second user profile associated with the second mobile device based on information received from the second mobile device;   selecting, based on the second user profile, the second customized information; and   sending the second customized information including the advertising information to the second mobile device.   
     
     
       35. The method of claim 34, wherein at least one of the first mobile device or the second mobile device is a phone. 
     
     
       36. The method of claim 21, further comprising controlling, based on the user profile, interaction with a user of the mobile device. 
     
     
       37. The method of claim 21, further comprising receiving feedback from a user associated with the mobile device via a touch screen of the mobile device, wherein the feedback is related to the customized information. 
     
     
       38. The method of claim 37, wherein the feedback relates to an advertisement contained in the customized information. 
     
     
       39. The method of claim 21, further comprising:
 receiving, in a user interaction, instructions from a user associated with the mobile device, wherein the instructions relate to the customized information;   providing, in the user interaction, feedback to the user; and   recording a result of the user interaction.   
     
     
       40. A system for sending targeted content to a mobile device, the system comprising:
 a receiver configured to receive a position of the mobile device, wherein the mobile device calculated the position using a positioning system;   a processor configured to:
 identify a user profile associated with the mobile device based on information received from the mobile device; and 
 select, based on the user profile and the position of the mobile device, customized information; and 
   a transmitter configured to send, for display on the mobile device, the customized information to the mobile device.   
     
     
       41. The system of claim 40, wherein:
 the receiver is configured to receive a request for the customized information; and   the transmitter is configured to send the customized information from a provider to the mobile device, wherein the customized information is dependent on the position of the mobile device, as determined by the positioning system during a time period associated with receiving the request, and dependent on at least one of the user profile or preferences information.   
     
     
       42. The system of claim 41, wherein the customized information is an advertisement. 
     
     
       43. The system of claim 42, wherein the advertisement concerns food. 
     
     
       44. The system of claim 42, wherein the advertisement concerns fuel. 
     
     
       45. The system of claim 42, wherein the advertisement concerns lodging. 
     
     
       46. The system of claim 41, wherein:
 the positioning system is a geographic positioning system (GPS); and   the GPS is configured to determine the position of the mobile device.   
     
     
       47. The system of claim 41, wherein:
 the positioning system is a cellular base station; and   the cellular base station is configured to determine the position of the mobile device.   
     
     
       48. The system of claim 41, wherein:
 the preferences information includes user preferences information; and   the transmitter is configured to send one or more customized billboards to the mobile device dependent on demographics of one or more users associated with the mobile device, dependent on the position of the mobile device, and dependent on the user preferences information.   
     
     
       49. The system of claim 41, wherein:
 the mobile device is a cellular phone; and   the cellular phone includes a display device.   
     
     
       50. The system of claim 49, wherein the display device includes a touch screen. 
     
     
       51. The system of claim 49, wherein the display device includes an interactive interface configured to receive instructions from one or more users and to display user feedback. 
     
     
       52. The system of claim 49, wherein the display device is a video display. 
     
     
       53. The system of claim 41, wherein the mobile device includes a gyroscope. 
     
     
       54. The system of claim 41, wherein the user profile includes a demographic profile. 
     
     
       55. The system of claim 41, wherein the mobile device is configured to identify one or more users based on a biometric. 
     
     
       56. The system of claim 55, wherein the biometric is at least one of a fingerprint, a voice sample, or a video pattern. 
     
     
       57. The system of claim 55, wherein the biometric comprises a fingerprint scan performed by the mobile device. 
     
     
       58. The system of claim 41, wherein the customized information includes an advertisement. 
     
     
       59. The system of claim 41, wherein the customized information includes a plurality of advertisements. 
     
     
       60. The system of claim 41, wherein the customized information is configured to be sent via a cellular network. 
     
     
       61. The system of claim 41, wherein a display device of the mobile device includes a touch screen display. 
     
     
       62. The system of claim 61, wherein the touch screen display is configured to display a graphical user interface. 
     
     
       63. The system of claim 41, wherein the user profile is based at least in part on prior actions of one or more users of the mobile device.