Ergonomic man-machine interface incorporating adaptive pattern recognition based control system

Abstract

An adaptive interface for a programmable system, for predicting a desired user function, based on user history, as well as machine internal status and context. The apparatus receives an input from the user and other data. A predicted input is presented for confirmation by the user, and the predictive mechanism is updated based on this feedback. Also provided is a pattern recognition system for a multimedia device, wherein a user input is matched to a video stream on a conceptual basis, allowing inexact programming of a multimedia device. The system analyzes a data stream for correspondence with a data pattern for processing and storage. The data stream is subjected to adaptive pattern recognition to extract features of interest to provide a highly compressed representation which may be efficiently processed to determine correspondence. Applications of the interface and system include a VCR, medical device, vehicle control system, audio device, environmental control system, securities trading terminal, and smart house. The system optionally includes an actuator for effecting the environment of operation, allowing closed-loop feedback operation and automated learning.

What is claimed is: 
     
       1. A programmable control responsive to an user input and a signal received from a signal source, comprising:
 a controller, operating according to a predetermined program, for receiving the user input and the signal and producing a control output, said controller producing a multivalued characterization of the signal with respect to at least one parameter;   a memory for storing data relating to an activity of the user;   a data processing system for predicting a most probable action of a user based on said stored data relating to said activity of the user and said characterized signal; and   a user feedback data presenting system comprising a display device for presentation of a sequence of programming options to the user, including said most probable action of the user, in a plurality of display images, each display image differing in available programming options.   
     
     
       2. The programmable control according to  claim 1  being for performing an action based on user input and an information content of a signal received from a signal source, further comprising:
 a user controlled direct manipulation-type input device, associated with said display device, having a device output, said device output being the user input; 
 a plant capable of performing a physical action, being responsive to an actuator signal; and 
 said controller, being for receiving data from said device output of said input device and a signal received from a signal source, and displaying user feedback data on said display device, 
 said user feedback data comprising a presentation of a sequence of programming options to the user on said display device, including said most probable action of the user, in a plurality of display images, each display image differing in available programming options, said sequence of programming options including at least one sequence of options sufficient to define an operable control program, and a presentation of additional programming options if said control program is not operable. 
 
     
     
       3. The system according to  claim 1 , being for processing a program comprising program material, in response to a viewer input, comprising:
 a user input processing system for determining a viewer preference;   said controller comprises a program material processing system for characterizing the program material, as said characterized signal, based on its content;   a correlator for correlating said characterized content of the program material with said determined viewer preference to produce a correlation index; and   a processor, selectively processing the program material based on said correlation index,   wherein said viewer preference is an input to said data processing system, and   said data processing system comprises said correlator.   
     
     
       4. The system according to  claim 3 , wherein said program material is encrypted, further comprising:
 a decryption system for decrypting the program material if it is selected to produce unencrypted program material and optionally an associated decryption event;   a memory for storing data relating to the occurrence of said decryrption event; and   a central database for storing data relating to the occurrence of said decryption event in association with data relating to the viewer.   
     
     
       5. The system according to  claim 3 , wherein:
 said user input processing system monitors a pattern of user activity and predicts a viewer preference;   said program material processing system comprises:   a processor for preprocessing the program material to produce a reduced data flow information signal substantially retaining information relating to an abstract information content of the program material and selectively eliminating data not relating to said abstract information content of the program material and for characterizing said information signal based on said abstract content; and   a comparing system for determining if said correlation index is indicative of a probable high correlation between said characterization of said information signal and said viewer preference and causing said stored program material to be processed by said processor based on said determination.   
     
     
       6. The system according to  claim 3 , wherein said processor comprises an image program material storage and retrieval system. 
     
     
       7. The system according to  claim 3 , further comprising a memory for storing a characterization of the program material; an input for receiving a feedback signal from the viewer indicating a degree of agreement with said correlation index determination, wherein said feedback signal and said stored characterization are used by said user input processing system to predict a new viewer preference. 
     
     
       8. The system according to  claim 3 , wherein said a program material processing system correlates and characterizes image information, further comprising:
 means for storing template data;   means for storing the image data;   means for generating a plurality of addressable domains from the stored image data, each of the domains representing a different portion of the image information;   means for creating, from the stored image data, a plurality of addressable mapped ranges corresponding to different subsets of the stored image data, the creating means including means for executing, for each of the mapped ranges, a procedure upon the one of the subsets of the stored image data which corresponds to the mapped range;   means for assigning identifiers to corresponding ones of the mapped ranges, each of the identifiers specifying for the corresponding mapped range an address of the corresponding subset of stored image data;   means for selecting, for each of the domains, the one of the mapped ranges which most closely corresponds according to predetermined criteria;   means for representing at least a portion of the image information as a set of the identifiers of the selected mapped ranges; and   means for selecting, from the stored templates, a template which most closely corresponds to the set of identifiers representing the image information.   
     
     
       9. The system according to  claim 8  wherein said correspondence of a template and the set of identifiers is determined by a processor executing a predetermined program for executing an algorithm selected from at least one of the group consisting of algorithms for selecting a minimum Hausdorff distance between the mapped range and the domain, for selecting the highest cross-correlation of the mapped range with the domain, and for selecting the lowest mean square error of the difference between the mapped range and the domain. 
     
     
       10. The system according to  claim 9  wherein said correspondence of a template and the set of identifiers is determined by selecting, for each domain, the mapped range with the minimum modified Hausdorff distance calculated as D[db,mrb]+D[1−db,1−mb], where D is a distance calculated between a pair of sets of data each representative of an image, db is a domain, mrb is a mapped range, I−db is the inverse of a domain, and I-mrb is an inverse of a mapped range. 
     
     
       11. The system according to  claim 8 , wherein said means for representing further comprises means for:
 (a) determining a feature of interest of the image data,   (b) selecting a mapped range corresponding to the feature of interest,   (c) storing the identifiers of the selected mapped range,   (d) selecting a further mapped range corresponding to a portion of image data having a predetermined relationship to the feature of interest, and   (e) storing the identifiers of the further mapped range.   
     
     
       12. The system according to  claim 8 , wherein said image data comprises data having three associated dimensions obtained by an image processor executing an algorithm selected from the group consisting of an algorithm for synthesizing a three dimensional representation based on a machine based prediction derived from two dimensional image data, an algorithm for synthesizing a three dimensional representation derived from a time series of pixel images, and an algorithm for synthesizing a three dimensional representation based on a image data representating a plurality of parallax views having at least two dimensions, said set of identifiers representing data in each of the three associated dimensions. 
     
     
       13. A method of programming a device, comprising the steps of:
 providing an input for user commands and feedback;   storing information relating to the user commands and feedback in a memory;   providing a data input for external information;   predicting a subsequent user command based on the stored information relating to user commands and feedback, a status of the device, and external information from the data input;   presenting the predicted user command to the user;   accepting feedback from the user relating to the predicted subsequent user command;   modifying the predicted subsequent user command based on at least the feedback; and   executing the modified predicted user command.   
     
     
       14. The method according to  claim 13 , further comprising the step of identifying a user and storing an user identifier with said stored information relating to the user commands;
 said predicting step predicting a subsequent user command based on the stored information relating to user identifier, commands and feedback, a status of the device, and external information from the data input.   
     
     
       15. A programmable device comprising:
 a user input for receiving an input variable having a path between a first input state and a second input state, said user input comprising:
 path dependent user data having transitional path information between said first input state and said second input state; and 
 path independent user data comprising information about at least one of said first state and said second state; 
 a filter, separating said path dependent user data as user characterization data and said path independent user data as instructions; 
 a memory for storing said user characterization data; 
 a processor for executing said instructions; and 
 a feedback device, presenting information relating to said instructions and said stored user characterization data. 
   
     
     
       16. The device according to clam  15 , further comprising:
 a hierarchical command structure of said processor, said command structure having commands of different function; and   means for predicting a probability of execution of a plurality of commands based on said input,   said feedback device presenting commands based on at least said predicted probabilities.   
     
     
       17. The programmable control according to  claim 1 , wherein the signal comprises image information. 
     
     
       18. The method according to  claim 13 , wherein said stored information comprises a set of weights of a predictive algorithm. 
     
     
       19. The method according to  claim 13 , wherein said external information comprises a preceptual data stream. 
     
     
       20. The programmable device according to  claim 15 , wherein a format of information presented from said feedback device is based on said path dependent user data. 
     
     
       21. A passenger automobile for use on public roads, the automobile being at least partially autonomous, the automobile comprising:
 a power plant configured to produce power to propel the automobile on the public roads;   a steering control system configured to control a direction of travel of the automobile on the public roads;   an acceleration control system configured to control an output of the power plant;   a braking control system configured to control deceleration of the automobile;   at least one of a velocity-measurement system configured to measure a velocity of the automobile on the public roads, an object-detection system configured to detect objects near the automobile, and a positioning system configured to determine a position of the automobile; and   a vehicle control system coupled to the power plant, the steering control system, the acceleration control system, the braking control system, and the at least one of the velocity-measurement system, the object-detection system, and the positioning system;   wherein:
 the vehicle control system is configured to control at least one of the steering control system, the acceleration control system, and the braking control system in response to signals from one or more of the velocity-measurement system, the object-detection system, and the positioning system, wherein said control is at least partially automated without input from a driver of the automobile; 
 the at least one of the velocity-measurement system, the object-detection system, and the positioning system includes the object-detection system; and 
 the vehicle control system is configured to present to the driver of the automobile predicted options to avoid a collision in response to the object-detection system.  
   
     
     
       22. The passenger automobile of claim 21, wherein the vehicle control system is configured to control, without input from a driver of the automobile, each of the steering control system, the acceleration control system, and the braking control system in response to signals from one or more of the velocity-measurement system, the object-detection system, and the positioning system.  
     
     
       23. The passenger automobile of claim 21, wherein the vehicle control system controls, without input from a driver of the automobile, at least one of the steering control system, the acceleration control system, and the braking control system by pre-activating the control system(s) so that activation of the control system(s) by the user is simplified.  
     
     
       24. The passenger automobile of claim 21, wherein the object-detection system is part of a collision-avoidance system including at least one of an imaging sensor, a radar system, and a LIDAR system.  
     
     
       25. The passenger automobile of claim 21, further comprising:
 a diagnostic system configured to generate diagnostic information about the automobile.    
     
     
       26. The passenger automobile of claim 25, further comprising:
 a wireless communication system;   wherein the vehicle control system is configured to transmit the diagnostic information from the diagnostic system through the wireless communication system.    
     
     
       27. The passenger automobile of claim 21, wherein:
 the vehicle control system is further configured to anticipate a travel condition and to prepare the automobile for the anticipated travel condition.    
     
     
       28. The passenger automobile of claim 27, wherein the vehicle control system is further configured to anticipate the travel condition based on at least one of voluntary inputs of the driver, implied inputs of the driver, current travel conditions, and past travel conditions.  
     
     
       29. The passenger automobile of claim 21, wherein the power plant is an engine.  
     
     
       30. The passenger automobile of claim 29, further comprising:
 a transmission control system;   wherein the vehicle control system is further configured to control the transmission control system in at least a partially automated manner without input from a driver of the automobile.    
     
     
       31. The passenger automobile of claim 21, further comprising:
 a traction control system;   wherein the vehicle control system is further configured to control the traction control system in at least a partially automated manner without input from a driver of the automobile.    
     
     
       32. The passenger automobile of claim 21, further comprising:
 a suspension control system;   wherein the vehicle control system is further configured to control the suspension control system in at least a partially automated manner without input from a driver of the automobile.    
     
     
       33. The passenger automobile of claim 21, further comprising:
 a climate control system;   wherein the vehicle control system is further configured to control the climate control system in at least a partially automated manner without input from a driver of the automobile.    
     
     
       34. The passenger automobile of claim 21, further comprising:
 an audio system;   wherein the vehicle control system is further configured to control the audio system in at least a partially automated manner without input from a driver of the automobile.    
     
     
       35. The passenger automobile of claim 21, further comprising:
 a wireless communication system;   wherein the vehicle control system is further configured to communicate with emergency services in the case of an accident or distress.    
     
     
       36. The passenger automobile of claim 21, wherein the object-detection system comprises a radar system.  
     
     
       37. The passenger automobile of claim 21, wherein the object-detection system comprises a LIDAR system.  
     
     
       38. The passenger automobile of claim 21, wherein the object-detection system comprises one or more imaging sensors.  
     
     
       39. The passenger automobile of claim 21, wherein the object-detection system comprises a neural network.  
     
     
       40. The passenger automobile of claim 21, wherein the vehicle control system comprises a neural network.  
     
     
       41. A method of operating a passenger automobile on a public road, the method comprising:
 operating a power plant of the automobile to propel the automobile on the public road in response to a vehicle control system of the automobile;   controlling a direction of the automobile in response to the vehicle control system;   controlling, in response to the vehicle control system, acceleration caused by the power plant;   controlling a braking of the automobile in response to the vehicle control system; and   wherein:
 at least one of the controlling of the direction of the automobile, the controlling of the acceleration, and the controlling of the braking of the automobile is performed based on results of one or more of the following in an at least partially automated manner without input from a driver of the automobile:
 measuring a velocity of the automobile; 
 detecting an object near the automobile, and 
 determining a position of the automobile; and 
 
   further comprising:
 predicting a collision; and 
 controlling at least one of operating of the power plant of the automobile controlling of the direction of the automobile in response to the predicted collision; and 
 presenting predicted options to avoid the collision to the driver of the automobile.  
   
     
     
       42. The method of claim 41, wherein the input from the driver of the automobile comprises at least one of a steering input, an acceleration input, a braking input, and a transmission input.  
     
     
       43. The method of claim 41, wherein each of the controlling of the direction of the automobile, the controlling of the acceleration, and the controlling of the braking of the automobile is performed based on results of one or more of the following in an at least partially automated manner without input from a driver of the automobile:
 measuring a velocity of the automobile;   detecting an object near the automobile, and   determining a position of the automobile.    
     
     
       44. The method of claim 41, wherein at least one of operating the power plant of the automobile, the controlling of the direction of the automobile, the controlling of the acceleration, and the controlling of the braking of the automobile is performed in response to at least one of the position of the automobile and the velocity of the automobile.  
     
     
       45. The method of claim 41, further comprising:
 diagnosing an operation of the automobile to generate diagnostic information.    
     
     
       46. The method of claim 45, further comprising:
 communicating, using a wireless communication system, the diagnostic information from the automobile.    
     
     
       47. The method of claim 41, further comprising:
 anticipating, using the vehicle control system, a travel condition of the automobile;   preparing, using the vehicle control system, the automobile for the anticipated travel condition.    
     
     
       48. The method of claim 47, wherein anticipating the travel condition comprises anticipating the travel condition based on at least one of voluntary inputs of the driver, implied inputs of the driver, current travel conditions, and past travel conditions.  
     
     
       49. The method of claim 41, further comprising:
 controlling a traction control system of the automobile in response to the vehicle control system.    
     
     
       50. The method of claim 41, further comprising:
 controlling a transmission control system of the automobile in response to the vehicle control system.    
     
     
       51. The method of claim 41, further comprising:
 controlling a suspension control system of the automobile in response to the vehicle control system.    
     
     
       52. The method of claim 41, further comprising:
 controlling a climate control system of the automobile in response to the vehicle control system.    
     
     
       53. The method of claim 41, further comprising:
 controlling an audio system of the automobile in response to the vehicle control system.    
     
     
       54. The method of claim 41, wherein detecting an object near the automobile comprises utilizing a radar system.  
     
     
       55. The method of claim 41, wherein detecting an object near the automobile comprises utilizing a LIDAR system.  
     
     
       56. The method of claim 41, wherein detecting an object near the automobile comprises utilizing one or more imaging sensors.  
     
     
       57. The method of claim 41, wherein detecting an object near the automobile comprises operating a neural network.  
     
     
       58. A non-transitory computer readable medium including instructions that, when executed by a computer in a passenger automobile traveling on a public road, implement an automobile control system to cause the computer to drive the automobile at least partially autonomously, the instructions comprising:
 instructions to implement a vehicle control system;   instructions to operate a power plant of the automobile to propel the automobile in response to the vehicle control system of the automobile; and   instructions to control a direction of the automobile in response to the vehicle control system;   instructions to control, in response to the vehicle control system, acceleration caused by the power plant;   instructions to control a braking of the automobile in response to the vehicle control system; and   instructions to accept a measurement of a velocity of the automobile;   wherein the vehicle control system controls at least one of the controlling of the direction of the automobile, the controlling of the acceleration, and the controlling of the braking of the automobile based on results of one or more of the following in an at least partially automated manner without input from a driver of the automobile:
 measuring a velocity of the automobile; 
 detecting an object near the automobile, and 
 determining a position of the automobile; and 
   further comprising:
 predicting a collision; and 
 controlling at least one of operating of the power plant of the automobile controlling of the direction of the automobile in response to the predicted collision; and 
 presenting predicted options to avoid the collision to the driver of the automobile.