Method and apparatus of diagnostic test

Abstract

Method, apparatus and computer program for providing a personalized study plan to a learner through cognitive and behavioral diagnosis of the learner. A learner who uses a data input device such as a smart pen and a stylus pen by using data obtained from the data input device. The method, apparatus and computer program relate to technology for obtaining input data based on information inputted by a user for at least one question with the data input device, creating test behavior data on the user from the obtained input data, analyzing cognition and behavior of the user based on at least one of metadata on the at least one question and the created test behavior data, and providing a personalized study plan to the user through an algorithm using machine learning based on the cognition and behavior analysis.

1 . (canceled) 
     
     
         2 . A computer-implemented method comprising:
 identifying one or more strokes inputted by a user with a smart pen for at least one question, the one or more strokes characterized by coordinate values of points forming each of the one or more strokes, time information when the points are inputted, and pressure information of the points;   detecting delays associated with the user inputting the one or more strokes, wherein the delays are detected using changes in the coordinate values of the points forming the one or more strokes over time;   determining a delay time and a number of the detected delays;   determining a behavior pattern of the user based on a number of the one or more strokes, the determined delay time, and the number of the detected delays, wherein determining the behavior pattern comprises:
 comparing analysis information with a number of identified strokes, the delay time, and the number of the detected delays for the user, wherein the analysis information comprises, for each of multiple users associated with the at least one question, the number of the one or more strokes, the delay time, and the number of the detected delays; and 
 displaying a diagnosis result for the behavior pattern of the user based on the comparison; 
   determining at least one cognitive and behavioral diagnostic (CBD) factor from a plurality of CBD factors based on metadata for the at least one question and the analysis information, wherein the at least one CBD factor is selected from at least one of confidence, grit, reasoning, concept memory, deep understanding, calculation ability, ability to understand question, and test-taking strategy.   
     
     
         3 . The computer-implemented method of  claim 2 , further comprising:
 presenting, on a display, one or more questions to be solved by the user; and   in response to presenting the one or more questions to be solved by the user, identifying the one or more strokes inputted by the user with the smart pen.   
     
     
         4 . The computer-implemented method of  claim 2 , wherein detecting the delays associated with the user inputting the one or more strokes comprises:
 determining a time difference between a first time stamp and a second time stamp, the first time stamp associated with a first stroke of a character and a second stroke of a subsequent character; and   determining a pause during a specific time period when the user inputs the one or more strokes with the smart pen, wherein the pause is determined by counting a number of consecutive time intervals with the specific time period in which no stroke is inputted.   
     
     
         5 . The computer-implemented method of  claim 2 , wherein determining the delay time and the number of the detected delays comprises:
 determining the delay time between a first time stamp and a second time stamp, the first time stamp associated with a last stroke of a character and the second time stamp associated with a first stroke of a subsequent character; and   determining the number of the detected delays comprises determining the number of delay times across multiple time stamps associated with the one or more strokes inputted by the user.   
     
     
         6 . The computer-implemented method of  claim 2 , further comprising calculating a total time of use of the smart pen by the user inputting the one or more strokes. 
     
     
         7 . The computer-implemented method of  claim 6 , wherein calculating the total time of use of the smart pen by the user inputting the one or more strokes further comprises:
 calculating a first time of preparation by the user before inputting the one or more strokes;   calculating a second time when the user solves one or more questions with the smart pen by inputting the one or more strokes; and   calculating the total time of use of the smart pen based on the calculated first time of preparation by the user and the calculated second time when the user solves the one or more questions.   
     
     
         8 . The computer-implemented method of  claim 7 , wherein calculating the total time of use further comprises:
 determining a third time when the smart pen was lifted from a display following a first stroke of solving a first question of the one or more questions;   determining a fourth time when the smart pen was applied to the display for a second stroke for solving of a second question of the one or more questions, wherein the second question is solved after the first question; and   determining the first time of preparation by subtracting the third time from the fourth time.   
     
     
         9 . The computer-implemented method of  claim 2 , wherein displaying the diagnosis result for the behavior pattern of the user comprises:
 displaying data representative of a first subset of strokes of the one or more strokes where a first delay of the detected delays that exceeded a first threshold occurred, wherein the data representative of the first subset of strokes is displayed in a different color from a color used to input the one or more strokes; and   displaying data representative of a second subset of strokes of the one or more strokes where a second delay of the detected delays that exceeded a second threshold occurred, wherein the data representative of the second subset of strokes comprises one or more tags that illustrate delay times with the second subset of strokes.   
     
     
         10 . A computer-implemented method comprising:
 obtaining input data indicative of information inputted by a user for at least one question with a smart pen;   creating test behavior data on the user from the obtained input data, wherein the input data is characterized by coordinate values of points forming one or more strokes and time information when the points were inputted, and wherein the created test behavior data comprises a plurality of behavioral metrics;   determining at least one cognitive and behavioral diagnostic (CBD) factor from a plurality of CBD factors based on at metadata for the at least one question and the created test behavior data, wherein the determining comprises:
 determining a score of at least one behavioral metric for the user, the score determined using (i) a difference value between (i) the at least one behavioral metric for the user based on the test behavior data and (ii) a mean value (μ) of the at least one behavioral metric determined for a plurality of users, divided by (ii) a standard deviation value (σ) of the at least one behavioral metric determined for the plurality of users; 
 normalizing the score of the at least one behavioral metric; 
 determining a weighted average of the normalized score based on a predetermined weight for the at least one behavioral metric; 
 determining the at least one CBD factor of the user from the determined weighted average; 
 providing another question to the user based on the at least one determined CBD factor; and 
   providing a personalized study plan to the user through machine learning based on at the metadata or the created test behavior data.   
     
     
         11 . The computer-implemented method of  claim 10 , wherein providing the personalized study plan to the user comprises:
 determining a plurality of second CBD factors for the at least one question for the plurality of users including the user;   calculating, for each CBD factor of the plurality of second CBD factors, a similarity using at least two CBD factors associated with (i) at least two questions comprising the at least one question and (ii) two different users;   calculating, for each CBD factor of the plurality of second CBD factors, a cognitive gap metric using the calculated similarity; and   recommending the other question to the user based on the calculated cognitive gap metric.   
     
     
         12 . The computer-implemented method of  claim 11 , wherein recommending the other question to the user comprises:
 producing the calculated cognitive gap metric for each of combinations of the user and the at least one question;   identifying the other question having the cognitive gap metric that exceeds a threshold based on the calculated cognitive gap metric; and   recommending the identified other question to the user.   
     
     
         13 . The computer-implemented method of  claim 12 , wherein producing the calculated cognitive gap metric for each of combinations of the user and the at least one question comprises calculating the similarity using a cosine similarity approximation of the at least two CBD factors associated with (i) the at least two questions comprising the at least one question and (ii) the two different users. 
     
     
         14 . The computer-implemented method of  claim 10 ,
 wherein the at least one CBD factor is selected from at least one of confidence, grit, reasoning, concept memory, deep understanding, calculation ability, ability to understand question, and test-taking strategy; and   each of the CBD factors is expressed with a function based on at least one of at least one different behavioral metric and metadata on the question.   
     
     
         15 . The computer-implemented method of  claim 10 , wherein determining the score of the at least one behavioral metric for the user comprises determining a Z-score of at least one behavioral metric of the user. 
     
     
         16 . The computer-implemented method of  claim 15 , wherein normalizing the score of the at least one behavioral metric comprises normalizing the Z-score of the at least one behavioral metric of the user to a range of values using a minimum Z-score value a maximum Z-score value. 
     
     
         17 . A system comprising:
 one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
 obtaining input data from a smart pen based on information inputted by a user for at least one question with the smart pen; 
 creating test behavior data on the user from the obtained input data, wherein the input data comprises coordinate values of points forming one or more strokes and time information when the points were inputted, and wherein the created test behavior data comprises a plurality of behavioral metrics; 
 determining at least one cognitive and behavioral diagnostic (CBD) factor from a plurality of CBD factors based on at metadata for the at least one question and the created test behavior data, wherein the determining comprises:
 determining a score of at least one behavioral metric for the user, the score determined using (i) a difference value between (i) the at least one behavioral metric for the user based on the test behavior data and (ii) a mean value (μ) of the at least one behavioral metric determined for a plurality of users, divided by (ii) a standard deviation value (σ) of the at least one behavioral metric determined for the plurality of users; 
 normalizing the score of the at least one behavioral metric; 
 determining a weighted average of the normalized score based on a predetermined weight for the at least one behavioral metric; 
 determining the at least one CBD factor of the user from the determined weighted average; 
 providing another question to the user based on the at least one determined CBD factor; and 
 
 providing a personalized study plan to the user through machine learning based on at the metadata or the created test behavior data. 
   
     
     
         18 . The system of  claim 17 , wherein providing the personalized study plan to the user comprises:
 determining a plurality of second CBD factors for the at least one question for the plurality of users including the user;   calculating, for each CBD factor of the plurality of second CBD factors, a similarity using at least two CBD factors associated with (i) at least two questions comprising the at least one question and (ii) two different users;   calculating, for each CBD factor of the plurality of second CBD factors, a cognitive gap metric using the calculated similarity; and   
       recommending the other question to the user based on the calculated cognitive gap metric. 
     
     
         19 . The system of  claim 18 , wherein recommending the other question to the user comprises:
 producing the calculated cognitive gap metric for each of combinations of the user and the at least one question;   identifying the other question having the cognitive gap metric that exceeds a threshold based on the calculated cognitive gap metric; and   recommending the identified other question to the user.   
     
     
         20 . The system of  claim 19 , wherein producing the calculated cognitive gap metric for each of combinations of the user and the at least one question comprises calculating the similarity using a cosine similarity approximation of the at least two CBD factors associated with (i) the at least two questions comprising the at least one question and (ii) the two different users. 
     
     
         21 . The system of  claim 17 ,
 wherein the at least one CBD factor is selected from at least one of confidence, grit, reasoning, concept memory, deep understanding, calculation ability, ability to understand question, and test-taking strategy; and   each of the CBD factors is expressed with a function based on at least one of at least one different behavioral metric and metadata on the question.