Gesture stroke recognition in touch-based user interface input

Abstract

A method for recognizing gesture strokes in user input applied onto an electronic document, comprising: receiving data based on the user input, the data representing a stroke and comprising a plurality of ink points and a plurality of timestamps associated with the plurality of ink point; segmenting the plurality of ink points into a plurality of segments each corresponding to a respective sub-stroke and comprising a respective subset of the plurality of ink points; determining a scale of the electronic document; generating a plurality of feature vectors based on the plurality of segments; normalizing a subset of the feature vectors according to the scale; and applying the plurality of feature vectors to a trained stroke classifier to generate a vector of probabilities including a probability that the stroke is a non-gesture stroke and a probability that the stroke is a given gesture stroke of a set of gesture types.

1 . A method for recognizing gesture strokes in user input applied onto an electronic document via a touch-based user interface, comprising:
 receiving data generated based on the user input, the data representing a stroke and comprising a plurality of ink points in a rectangular coordinate space and a plurality of timestamps associated respectively with the plurality of ink points;   segmenting the plurality of ink points into a plurality of segments each corresponding to a respective sub-stroke of the stroke and comprising a respective subset of the plurality of ink points;   determining at least one scale of the electronic document;   generating a plurality of feature vectors based respectively on the plurality of segments, the feature vectors comprising features of the segments;   normalizing a subset of the features of the feature vectors according to the at least one scale; and   applying the plurality of feature vectors as an input sequence representing the stroke to a trained stroke classifier to generate a vector of probabilities including a probability that the stroke is a non-gesture stroke and a probability that the stroke is a given gesture stroke of a set of gesture types.   
     
     
         2 . The method of  claim 1 , wherein the at least one scale is predefined based on a document structure. 
     
     
         3 . The method of  claim 1 , wherein the at least one scale is calculated independently of a document structure. 
     
     
         4 . The method of  claim 3 , wherein the at least one scale is calculated, subsequently to receiving said data generated based on the user input, according to dimensions of the stroke. 
     
     
         5 . The method of  claim 1 , wherein generating the plurality of feature vectors based respectively on the plurality of segments comprises, for each segment of the plurality of segments corresponding to a respective sub-stroke:
 generating intrinsic geometric features that represent the shape of the respective sub-stroke, the intrinsic geometric features computed from the respective subset of the plurality of ink points associated with the sub-stroke; and   generating neighborhood features that represent spatial relationships between the sub-stroke and content that neighbors the sub-stroke, the neighborhood features computed from a relationship between the respective subset of the plurality of ink points associated with the sub-stroke and content that neighbors the sub-stroke,   wherein the content that neighbors the sub-stroke intersects a window centered with respect to the sub-stroke.   
     
     
         6 . The method of  claim 5 , wherein generating the intrinsic geometric features comprises generating statistical sub-stroke geometric features and/or global sub-stroke geometric features for the sub-stroke. 
     
     
         7 . The method of  claim 6 , wherein generating the statistical sub-stroke geometric features comprises, for each geometric feature of a set of intrinsic geometric features:
 determining respective values for the ink points of the segment corresponding to the respective sub-stroke; and   calculating one or more statistical measures based on the determined respective values.   
     
     
         8 . The method of  claim 6 , wherein generating the global sub-stroke geometric features for the sub-stroke comprises computing one or more of: a sub-stroke length, a count of singular ink points within the sub-stroke, and a ratio between the sub stroke length and a distance between a first and a last ink point of the sub-stroke. 
     
     
         9 . The method of  claim 5 , wherein generating the neighborhood features comprises generating one or more of:
 textual neighborhood features representing spatial relationships between the sub stroke and textual content that neighbors the sub-stroke;   mathematical neighborhood features representing spatial relationships between the sub-stroke and mathematical content that neighbors the sub-stroke; and   non-textual neighborhood features representing spatial relationships between the sub-stroke and non-textual content that neighbors the sub-stroke.   
     
     
         10 . The method of  claim 5 , wherein a first subset of the intrinsic geometric features and a first subset of the neighborhood features define a first group of features comprising features of the feature vectors which are constrained by the document structure. 
     
     
         11 . The method of  claim 10 , wherein the features of the first group of features are normalized according to a predefined scale. 
     
     
         12 . The method of  claim 10 , wherein a second subset of the intrinsic geometric features and a second subset of the neighborhood features define a second group of features comprising features of the feature vectors which are independent from the document structure. 
     
     
         13 . The method of  claim 12 , wherein the features of the second group of features are normalized according to a calculated scale independent of the document structure. 
     
     
         14 . A computing device, comprising:
 a processor; and   memory storing instructions that, when executed by the processor, configure the processor to perform a method according to  claim 1 .   
     
     
         15 . A computer program including instructions that when executed by a processor cause the processor to execute a method according to  claim 1 .