Method, computing device, and computer-readable medium for processing plurality of tasks collected in crowdsourcing in subset unit

Abstract

The present invention relates to a method, a computing device, and a computer-readable medium for processing a plurality of tasks collected in crowdsourcing in a subset unit, and more particularly, to a method, a computing device, and a computer-readable medium for processing a plurality of tasks collected in crowdsourcing in a subset unit, so as to create a subset including task results performed by workers through the crowdsourcing, and provide the task results included in the created subset to reviewers, thereby deriving an inference result on the task result by using the subset as a unit when a review of each task result is completed.

What is claimed is: 
     
         1 . A method for, in a subset unit, processing a plurality of tasks collected through crowdsourcing performed on a computing device including at least one processor and at least one memory, the method comprising:
 a subset creation step of receiving a plurality of task results performed by a plurality of workers for a plurality of tasks including at least one unit task, and creating a subset including a predetermined number of task results based on a preset subset creation criterion;   a subset review step of providing task results less than the predetermined number included in the subset to a plurality of reviewers, and receiving a review result obtained by reviewing each of the task results from the reviewers;   a subset review completion step of determining whether each of the predetermined number of task results to which one or more review results are assigned through the subset review step satisfies a review completion criterion, and setting the subset to a review completion state when all of the predetermined number of task results included in the subset satisfy the review completion criterion; and   a subset inference step of deriving an inference result for each of the predetermined number of task results, based on reliability information of each of reviewers having performed a review on the predetermined number of task results included in the subset set to the review completion state, and based on one or more review results on each of the predetermined number of task results.   
     
     
         2 . The method of  claim 1 , wherein the subset creation criterion includes number information on the task results included in the subset,
 the review completion criterion includes maximum number information on reviews manageable by the reviewer for the predetermined number of task results included in the subset, and completion criterion information on task results, and   the completion criterion information on the task results includes at least one of minimum number information on review results with a specific value, maximum number information on the review results, and number information on a review result of reviewing a task result as an error.   
     
     
         3 . The method of  claim 1 , wherein the review completion criterion includes maximum number information on reviews manageable by the reviewer for the predetermined number of task results included in the subset, and
 the subset review step includes stopping providing one or more remaining task results, which are included in the subset and not reviewed by the specific reviewer, to the specific reviewer, when the specific reviewer reviews on one or more task results corresponding to the maximum number review information among the predetermined number of task results included in the subset.   
     
     
         4 . The method of  claim 1 , wherein the subset review completion step includes:
 determining whether the corresponding task result satisfies the review completion criterion, whenever a review result is received for each task result included in the subset through the subset review step; and   providing the task result to a plurality of reviewers through the subset review step when the task result does not satisfy the review completion criterion, and stopping providing the task result to the reviewers through the subset review step, when the task result satisfies the review completion criterion, to set the corresponding task result to a standby state.   
     
     
         5 . The method of  claim 1 , wherein the subset creation step includes:
 creating, when the subset creation criterion is changed while the subset is being created, the subset in the created state, based on the subset creation criterion before being changed; and   creating a subset created later based on the changed subset creation criterion.   
     
     
         6 . The method of  claim 1 , further comprising:
 a task result providing step of providing, based on the inference result derived through the subset inference step, information on whether the task result is passed to the worker having provided the task result included in the subset, and providing, based on the inference result, information on whether the review result by the reviewer having reviewed the task result is passed to the corresponding reviewer.   
     
     
         7 . A computing device including at least one processor and at least one memory to perform a method for processing a plurality of tasks collected through crowdsourcing in a subset unit, the computing device performing:
 a subset creation step of receiving a plurality of task results performed by a plurality of workers for a plurality of tasks including at least one unit task, and creating a subset including a predetermined number of task results based on the preset subset creation criterion;   a subset review step of providing task results less than the predetermined number included in the subset to a plurality of reviewers, and receiving a review result obtained by reviewing each of the task results from the reviewers;   a subset review completion step of determining whether each of the predetermined number of task results to which one or more review results are assigned through the subset review step satisfies a review completion criterion, and setting the subset to a review completion state when all of the predetermined number of task results included in the subset satisfy the review completion criterion; and   a subset inference step of deriving an inference result for each of the predetermined number of task results, based on reliability information of each of reviewers having performed a review on the predetermined number of task results included in the subset set to the review completion state, and based on one or more review results on each of the predetermined number of task results.   
     
     
         8 . A computer program stored on a computer-readable medium and including a plurality of instructions executed by at least one processor, the computer program comprising:
 a subset creation step of receiving a plurality of task results performed by a plurality of workers for a plurality of tasks including at least one unit task, and creating a subset including a predetermined number of task results based on a preset subset creation criterion;   a subset review step of providing task results less than the predetermined number included in the subset to a plurality of reviewers, and receiving a review result obtained by reviewing each of the task results from the reviewers;   a subset review completion step of determining whether each of the predetermined number of task results to which one or more review results are assigned through the subset review step satisfies a review completion criterion, and setting the subset to a review completion state when all of the predetermined number of task results included in the subset satisfy the review completion criterion; and   a subset inference step of deriving an inference result for each of the predetermined number of task results, based on reliability information of each of reviewers having performed a review on the predetermined number of task results included in the subset set to the review completion state, and based on one or more review results on each of the predetermined number of task results.