Predicting future performance of multiple workers on crowdsourcing tasks and selecting repeated crowdsourcing workers

Abstract

Systems and methods of a job distribution platform for aggregating performance data in a worker profile for workers in performing crowd sourced tasks are disclosed. In one aspect, embodiments of the present disclosure include a method, which may be implemented on a system, of generating and storing performance data when a worker performs one or more tasks distributed online to their respective computing devices by a job distribution platform which crowd sources tasks over a network to remote workers. The tasks can span current jobs and a history of previous jobs distributed to the worker and the job performance data for the worker is collected for current and previous jobs. New jobs can be assigned to a worker selected based on performance data of the worker. Other types of worker information may also be tracked and generated for multiple workers in various worker pools (channels), for example, psych questions or other more subjective data collected from a worker or contributor in addition to performance.

1 - 33 . (canceled) 
     
     
         34 . A method performed by a job distribution server to track performance data and select a worker to perform a new crowdsourcing task, the method comprising:
 generating a first performance data based on a first worker completing a first crowdsourcing task and a second performance data based on a second worker completing a second crowdsourcing task, wherein each of the first performance data and the second performance data includes a monitored speed rate of the first or second crowdsourcing task and an average speed rate among a plurality of workers to satisfactorily complete any crowdsourcing task;   aggregating all performance data associated with the first worker and all performance data associated with the second worker, wherein the aggregated performance data includes an average monitored speed rate differential indicating the difference between the monitored speed rate of each worker in completing the first and second crowdsourcing tasks and the average speed rate;   suspending the first worker from completing the new crowdsourcing task based on determining that the average monitored speed rate differential of the first worker falls below a threshold speed rate; and   re-routing the new crowdsourcing task from the first worker to the second worker.   
     
     
         35 . The method of  claim 34 , wherein said re-routing is based on determining that the average monitored speed rate differential of the second worker exceeds the threshold speed rate. 
     
     
         36 . The method of  claim 34 , wherein the first and second crowdsourcing tasks include a history of previously completed crowdsourcing tasks distributed to the first and second workers, where the first and second performance data for the first and second workers is collected for the first and second crowdsourcing tasks and the history of previously completed crowdsourcing tasks. 
     
     
         37 . The method of  claim 36 , further comprising:
 tracking the history of previously completed crowdsourcing tasks including the first and second crowdsourcing tasks.   
     
     
         38 . The method of  claim 34 , wherein the first and second performance data includes quantitative metrics indicating a rate or number of rejected results in performing the first and second crowdsourcing tasks. 
     
     
         39 . The method of  claim 34 , wherein the first and second performance data includes quantitative metrics indicating a rate or number of instances of bans from the first and second crowdsourcing tasks. 
     
     
         40 . The method of  claim 34 , wherein the first and second performance data includes quantitative metrics indicating a rate or number of instances that the first or second worker has been flagged for performance review. 
     
     
         41 . The method of  claim 34 , further comprising:
 tracking a rate in which the first or second worker completes the first or second crowdsourcing tasks.   
     
     
         42 . The method of  claim 41 , further comprising:
 tracking a number of instances that the first or second worker performs any of the first or second crowdsourcing tasks that exceed the average speed rate.   
     
     
         43 . The method of  claim 34 , wherein the first and second performance data includes quantitative metrics indicating a rate in which the first or second worker has completed one or more tasks exceeding the average speed rate. 
     
     
         44 . The method of  claim 34 , wherein the first and second performance data includes quantitative metrics indicating the monitored rate of speed in which the worker satisfactorily completes the first or second crowdsourcing task. 
     
     
         45 . The method of  claim 34 , further comprising:
 determining a value in completing any of the first or second crowdsourcing task by the first or second worker, wherein the first and second performance data includes quantitative metrics indicating the values of previous crowdsourcing tasks by the first or second worker.   
     
     
         46 . The method of  claim 34 , further comprising:
 determining an accuracy of work generated by the first and second workers using test jobs, wherein the first and second performance data includes quantitative metrics indicating the accuracy of the work performed by the first and second workers.   
     
     
         47 . The method of  claim 34 , further comprising:
 determining an experience level of the first and second workers based on time durations that the first and second workers have completed previous crowdsourcing tasks stored at the first and second performance data.   
     
     
         48 . The method of  claim 34 , wherein the first and second performance data further includes demographic data of the first and second workers. 
     
     
         49 . The method of  claim 34 , wherein first and second performance data further includes identification of devices used by the first and second workers to perform the first and second crowdsourcing tasks distributed by the job distribution server. 
     
     
         50 . The method of  claim 34 , wherein the first and second performance data further includes preferences of the first and second workers specified by the first and second workers, where the preferences of the first and second workers indicate a willingness to perform the first and second crowdsourcing tasks that involve objectionable content. 
     
     
         51 . The method of  claim 34 , wherein the first and second performance data further includes preferences of the first and second workers specified by the first and second workers, wherein the preferences of the first and second workers indicates a preferred time of day to perform the first and second crowdsourcing tasks. 
     
     
         52 . The method of  claim 34 , wherein the first and second performance data include compensation preferences which specify a preferred payment mechanism including-at least one of a direct deposit, indirect deposit, and mobile credit. 
     
     
         53 . The method of  claim 52 , wherein the compensation preferences further include at least one of a preferred wage, preferred currency, and preferred type of currency. 
     
     
         54 . The method of  claim 34  wherein the first and second crowdsourcing tasks are distributed to the first and second workers via multiple channels, wherein the first and second workers perform the first and second crowdsourcing tasks through multiple channels, and where each of the first and second crowdsourcing tasks are tracked by a unique identifier. 
     
     
         55 . A system to track performance data and select a worker to perform a new crowdsourcing task, the system comprising a job distribution server configured to:
 generate performance data upon completion of an assigned crowdsourcing task for each of a plurality of workers that have completed at least two past crowdsourcing tasks, wherein the performance data includes a monitored speed rate of the worker in completing the assigned crowdsourcing task and an average speed rate of the worker to complete any crowdsourcing task and the past crowdsourcing tasks;   aggregate all performance data associated with each worker of the plurality of workers, wherein the aggregated performance data includes an average monitored speed rate differential indicating a difference between the monitored speed rate of each worker and the average speed rate of the plurality of workers;   suspend a first worker from completing a new crowdsourcing task based on determining that an average monitored speed rate differential associated the first worker falls below a threshold; and   route the new crowdsourcing task from the first worker to a second worker that includes an average monitored speed rate differential that exceeds the threshold.   
     
     
         56 . The system of  claim 55 , further comprising:
 providing a test job to the first worker and second worker;   obtaining a set of results from the first worker and second worker based on the test job;   determining an accuracy of the results provided-by the first and second workers in performing the test job based on comparing the set of results provided-by the first and second workers; and   updating performance data associated with the first worker and second worker based on the determined accuracy of the results provided-by the first and second workers in performing the test job, wherein said routing the new crowdsourcing task from the first worker to the second worker is based on the updated performance data associated with the first worker and second worker.