Using telemetry data in a distributed computing environment to address complex problems

Abstract

The disclosed technology concerns methods, apparatus, and systems for using telemetry data from a large number of remote computing devices to address complex problems otherwise prone to subjective inaccuracies. Particular embodiments disclosed herein involve classifying the difficulty of solving (or completing) an objective presented by a certain item of digital content. For instance, certain example embodiments involve collecting telemetry data from hundreds or thousands of users engaging with a respective content item, and, based at least in part on that telemetry data, assigning a difficulty classification to the content item.

What is claimed is: 
     
         1 . A method, comprising:
 by a central server in communication with a distributed plurality of remote computing devices via a network:
 transmitting a seed or initial-state data to the distributed plurality of the remote computing devices for use in a software application in which an initial state is determined by the seed or the initial-state data; 
 receiving telemetry data from the distributed plurality of remote computing devices, the telemetry data being indicative of how respective users at the computing devices interacted with the software application during execution of the application with the seed or initial-state data; and 
 assigning a difficulty categorization for the seed or the initial-state data based at least in part on the telemetry data, the assigned difficulty categorization being one from among a plurality of available difficulty categorizations. 
   
     
     
         2 . The method of  claim 1 , wherein the software application provides an interaction that begins from an initial state determined from the seed or initial-state data and thereafter proceeds until completion of an objective or failure to reach the objective without further modification from the central server. 
     
     
         3 . The method of  claim 1 , wherein the seed or the initial-state data is of an unknown difficulty. 
     
     
         4 . The method of  claim 1 , wherein the seed or the initial-state data is randomly selected from among a plurality of seeds or initial-state data. 
     
     
         5 . The method of  claim 1 , wherein the telemetry data received comprises an identification of the seed or the initial-state data and an indication of a result, the result being one of an indication that a game or scenario provided by the software application was completed or an indication that the game or scenario provided by the software application was not completed. 
     
     
         6 . The method of  claim 1 , wherein the telemetry data received comprises an identification of the seed or the initial-state data and an indication of a result, the result being one of an indication that a game or scenario provided by the software application was completed, that the game or scenario provided by the software application was abandoned, or that the game or scenario provided by the software application reached a point where no more progress was possible. 
     
     
         7 . The method of  claim 1 , wherein the telemetry data received comprises an indication of one or more of a score achieved or a number of moves made. 
     
     
         8 . The method of  claim 1 , wherein the telemetry data received comprises an indication of whether one or more milestones in progression toward achieving an objective were reached. 
     
     
         9 . The method of  claim 1 , wherein the assigning a difficulty categorization comprises:
 computing a difficulty-indicative data element from the telemetry data; and   applying the difficulty-indicative data element to a difficulty categorization table defining two or more difficulties to respective ranges of the difficulty-indicative element; and   assigning the difficulty categorization based on the application of the difficulty-indicative data element to the difficulty categorization table.   
     
     
         10 . The method of  claim 9 , wherein the assigning the difficulty categorization comprises:
 computing a difficulty-indicative data element for telemetry data for a plurality of seeds or initial-state data, the difficulty-indicative data being normalized to provide a common metric for the telemetry data that reduces effects from the seed or the initial-state data being used by different numbers of users at the plurality of the remote computing devices.   
     
     
         11 . The method of  claim 9 , wherein the difficulty-indicative data element is derived from a total number of users who successfully completed an objection and a total number of users who attempted to complete the objective. 
     
     
         12 . The method of  claim 1 , wherein the plurality of remote computing devices comprises 1000 or more remote computing devices. 
     
     
         13 . A system, comprising:
 a client computing device comprising a memory and one or more processors, the one or more processors being programmed to:
 receive, from a central server, seed data or initial-state data; 
 initiate a game or scenario that begins from an initial state determined from the seed data or the initial-state data and that proceeds without further modification from the central server; 
 terminate the game or scenario when the game or scenario is quit by the user or reaches a state in which no further progress is available to the user; and 
 transmit a data set to the central server indicative of user interactions with the game or scenario, the data set including an identification of the seed data or the initial-state data used and data indicating the user's progress toward reaching an objective for the game or scenario. 
   
     
     
         14 . The system of  claim 13 , wherein the seed data or the initial-state data is for a game or scenario of unknown difficulty. 
     
     
         15 . The system of  claim 13 , wherein the game or scenario is a first game or scenario, and wherein the one or more processors are further programmed to:
 allow the user to select a difficulty categorization for a subsequent game or scenario; and   receive, from the central server, seed data or initial-state data for the subsequent game or scenario, the seed data or initial-state data for the subsequent game or scenario having been assigned the selected difficulty categorization based at least in part on data sets received by the central server from multiple remote client computing devices indicating progress made by users of the multiple remote client computing devices toward reaching the objective for the game or scenario for the subsequent seed data or initial-state data.   
     
     
         16 . One or more computer-readable media storing computer-executable instructions, which when executed by a computer cause the computer to perform a method, the method comprising:
 receiving telemetry data from a plurality of client computing devices, the telemetry data including data that indicates progress toward an objective made by a user at a respective client computing device for an application that began from an initial state selected from a range of possible initial states, the selected initial state being determined from a seed value or an initial-state value;   computing one or more metrics from the telemetry data, the one or more metrics comprising metrics that aggregate the telemetry data into normalized values;   applying the one or more metrics to a difficulty classification table that correlates ranges of values of the one or more metrics to a respective difficulty classifications, the applying resulting in a selected difficulty classification being identified for the selected initial state; and   storing the seed or the initial-state data in a data set of available seeds or initial-state data having the selected difficulty classification.   
     
     
         17 . The one or more computer-readable media of  claim 16 , wherein the method further comprises:
 after the applying, receiving a request from a client computing device for a seed or initial-state data having the selected difficulty classification;   selecting a seed or initial-state data from the data set of available seeds or initial-state data having the selected difficulty classification; and   transmitting the selected seed or initial-state data having the selected difficulty classification to the client computing device.   
     
     
         18 . The one or more computer-readable media of  claim 16 , wherein the plurality of client computing device comprises 100 or more client computing devices. 
     
     
         19 . The one or more computer-readable media of  claim 16 , wherein the initial state corresponds to a unique shuffle of a standardized deck of cards. 
     
     
         20 . The one or more computer-readable media of  claim 16 , wherein the range of possible initial states is larger than 1×10 10 .