Network-based content submission and contest management

Abstract

In one aspect, the present disclosure implements a method that identifies an annotator suited to judge one or more images submitted to a contest. In this regard, the method comprises enabling the annotator to generate scoring information in a plurality of prior contests and collecting the scoring information generated by the annotator. The method further includes calculating a metric that quantifies the ability the annotator to judge image quality relative to other annotators using the collected scoring information. If the metric is higher than a threshold amount, the method includes incentivizing the annotator to judge the images submitted to the contest.

1 . A method for identifying an annotator suited to judge one or more images submitted to a contest, the method comprising:
 enabling the annotator to generate scoring information in a plurality of prior contests;   collecting the scoring information generated by the annotator;   calculating a metric that quantifies the ability the annotator to judge image quality relative to other annotators using the collected scoring information; and   if the metric is higher than a threshold amount, incentivizing the annotator to judge the images submitted to the contest.   
     
     
         2 . The method as recited in claim  7 , wherein an ability to pick winners of the prior contests serves as the primary factor in the metric calculation that quantifies the ability the annotator to judge image quality. 
     
     
         3 . The method as recited in claim  7 , wherein calculating a metric includes using the scoring information generated in the plurality of prior contests includes quantifying the degree to which an annotator adheres to mainstream principles. 
     
     
         4 . The method as recited in claim  7 , wherein calculating a metric includes using the scoring information generated in the plurality of prior contests to quantify an annotator's ability to identify and rank images that have a desired criteria. 
     
     
         5 . The method as recited in claim  7 , wherein an annotator identified as being better in judging image quality as reflected in their metric score is compensated to participate in judging the contest. 
     
     
         6 . The method as recited in claim  7 , wherein incentivizing the annotator to judge images submitted to the contest includes requiring the user to perform at least one of the following types of work before a submission from a user is accepted: identifying a best image in a set, ranking images in a set, and selecting images that satisfy a geo-code constraint.