Machine-Learning Driven Recommendation Engine

Abstract

A data processing system implements a system for training and fine-tuning a plan recommendation model that recommends one or more insurance plans for a user based the cost of the insurance plans and the needs of the user. The plan recommendation model is trained and/or fine-tuned using training data that has been labeled by a human actuary to improve the accuracy of the model. The plan recommendation model is also fine tuned based on feedback received on recommendations made by the model to further improve the performance of the model.

What is claimed is: 
     
         1 . A data processing system comprising:
 a processor; and   a memory storing executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations of:
 accessing first insurance plan data from an insurance plan data store, the first insurance plan data identifying a plurality of first medical insurance plans provided by an employer to a plurality of first users of a benefits management platform; 
 accessing first user data from a user information datastore, the first user data comprising information associated with a plurality of first users indicative of medical plan coverage needs of the plurality of first users; 
 generating first candidate plan recommendations using a plan recommendation model, the plan recommendation model being configured to identify one or more candidate medical insurance plans selected from among the plurality of first medical insurance plans for a selected user selected from among the plurality of first users based on user data associated with the selected user, each candidate plan recommendation including an indication of the selected user and indication of the one or more candidate medical insurance plans selected from among the plurality of first medical insurance plans; 
 providing the first candidate plan recommendations to a first client device of an actuarial reviewer to present on an actuarial review user interface of the benefits management platform, 
 receiving actuarial recommendations from the first client device, the actuarial recommendations comprising a recommended plan selected from the first candidate plan recommendations suggested by the plan recommendation model and a confidence score associated with the recommended plan; 
 generating labeled training data for the plan recommendation model based on the actuarial recommendations; 
 fine-tuning the plan recommendation model using the labeled training data; 
 receiving a request for a plan recommendation from a second client device for a second user, the request for the plan recommendation requesting one or more medical insurance plans selected from among the plurality of first medical insurance plans based on second user data associated with the second user; 
 providing the second user data as an input to the plan recommendation model to obtain second candidate plan recommendations; and 
 sending the second candidate plan recommendations to the second client device to present on a plan recommendation user interface of the benefits management platform. 
   
     
     
         2 . The data processing system of  claim 1 , wherein the memory further stores executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations of:
 receiving the first user data from a plurality of first client devices of a plurality of first users via a user interface of an employee portal of the benefits management platform; and   storing the first user data in the user information datastore of the benefits management platform.   
     
     
         3 . The data processing system of  claim 1 , wherein the memory further stores executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations of:
 receiving the first insurance plan data from a first client device of the employer via a user interface of an employer portal of the benefits management platform; and   storing the first user data in the insurance plan data store of the benefits management platform.   
     
     
         4 . The data processing system of  claim 1 , wherein to generate the labeled training data for the plan recommendation model, the memory further stores executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations of:
 generating a plurality of labeled samples that include sample user data representing a sample user for which sample plan recommendations were generated, a recommended plan selected from the sample plan recommendations, and a confidence score associated with the recommended plan indicating a confidence in recommending the recommended plan from among the sample plan recommendations.   
     
     
         5 . The data processing system of  claim 4 , wherein the memory further stores executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations of:
 sanitizing the sample user data to remove personally identifiable information from the sample user data.   
     
     
         6 . The data processing system of  claim 4 , wherein the memory further stores executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations of:
 splitting the labeled training data into a first set of labeled training data and a second set of labeled training data;   fine-tuning the plan recommendation model using the first set of labeled training data;   providing the second set of labeled training data as an input to the plan recommendation model to obtain sample plan recommendations; and   comparing the sample plan recommendations to expected plan recommendations included in the second set of labeled training data to evaluate performance of the plan recommendation model.   
     
     
         7 . The data processing system of  claim 1 , wherein the memory further stores executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations of:
 presenting the second candidate plan recommendations on a subject matter expert review interface of an administrator portal of the benefits management platform;   receiving feedback on the second candidate plan recommendations from one or more subject matter experts; and   fine-tuning the plan recommendation model based on the feedback on the second candidate plan recommendations.   
     
     
         8 . The data processing system of  claim 1 , wherein the memory further stores executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations of:
 deploying the plan recommendation model to a production environment after fine-tuning the plan recommendation model using the labeled training data.   
     
     
         9 . The data processing system of  claim 1 , wherein the memory further stores executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations of:
 receiving second user data at the benefits management platform;   generating second candidate plan recommendations based on the second user data;   providing the second candidate plan recommendations to the first client device of the actuarial reviewer;   receiving second actuarial recommendations from the first client device, the second actuarial recommendations comprising a recommended plan selected from the first candidate plan recommendations suggested by the plan recommendation model and a confidence score associated with the recommended plan;   generating second labeled training data for the plan recommendation model based on the actuarial recommendations; and   fine-tuning the plan recommendation model using the second labeled training data.   
     
     
         10 . A method for operating a plan recommendation model, the method comprising:
 accessing first insurance plan data from an insurance plan data store, the first insurance plan data identifying a plurality of first medical insurance plans provided by an employer to a plurality of first users of a benefits management platform;   accessing first user data from a user information datastore, the first user data comprising information associated with a plurality of first users indicative of medical plan coverage needs of the plurality of first users;   generating first candidate plan recommendations using a plan recommendation model, the plan recommendation model being configured to identify one or more candidate medical insurance plans selected from among the plurality of first medical insurance plans for a selected user selected from among the plurality of first users based on user data associated with the selected user, each candidate plan recommendation including an indication of the selected user and indication of the one or more candidate medical insurance plans selected from among the plurality of first medical insurance plans;   providing the first candidate plan recommendations to a first client device of an actuarial reviewer to present on an actuarial review user interface of the benefits management platform;   receiving actuarial recommendations from the first client device, the actuarial recommendations comprising a recommended plan selected from the first candidate plan recommendations suggested by the plan recommendation model and a confidence score associated with the recommended plan;   generating labeled training data for the plan recommendation model based on the actuarial recommendations;   fine-tuning the plan recommendation model using the labeled training data;   receiving a request for a plan recommendation from a second client device for a second user, the request for the plan recommendation requesting one or more medical insurance plans selected from among the plurality of first medical insurance plans based on second user data associated with the second user;   providing the second user data as an input to the plan recommendation model to obtain second candidate plan recommendations; and   sending the second candidate plan recommendations to the second client device to present on a plan recommendation user interface of the benefits management platform.   
     
     
         11 . The method of  claim 10 , the method further comprising:
 receiving the first user data from a plurality of first client devices of a plurality of first users via a user interface of an employee portal of the benefits management platform; and   storing the first user data in the user information datastore of the benefits management platform.   
     
     
         12 . The method of  claim 10 , the method further comprising:
 receiving the first insurance plan data from a first client device of the employer via a user interface of an employer portal of the benefits management platform; and   storing the first user data in the insurance plan data store of the benefits management platform.   
     
     
         13 . The method of  claim 10 , wherein generating the labeled training data for the plan recommendation model further comprises:
 generating a plurality of labeled samples that include sample user data representing a sample user for which sample plan recommendations were generated, a recommended plan selected from the sample plan recommendations, and a confidence score associated with the recommended plan indicating a confidence in recommending the recommended plan from among the sample plan recommendations.   
     
     
         14 . The method of  claim 13 , the method further comprising:
 sanitizing the sample user data to remove personally identifiable information from the sample user data.   
     
     
         15 . The method of  claim 13 , the method further comprising:
 splitting the labeled training data into a first set of labeled training data and a second set of labeled training data;   fine-tuning the plan recommendation model using the first set of labeled training data;   providing the second set of labeled training data as an input to the plan recommendation model to obtain sample plan recommendations; and   comparing the sample plan recommendations to expected plan recommendations included in the second set of labeled training data to evaluate performance of the plan recommendation model.   
     
     
         16 . The method of  claim 10 , the method further comprising:
 presenting the second candidate plan recommendations on a subject matter expert review interface of an administrator portal of the benefits management platform;   receiving feedback on the second candidate plan recommendations from one or more subject matter experts; and   fine-tuning the plan recommendation model based on the feedback on the second candidate plan recommendations.   
     
     
         17 . The method of  claim 10 , the method further comprising:
 deploying the plan recommendation model to a production environment after fine-tuning the plan recommendation model using the labeled training data.   
     
     
         18 . The method of  claim 10 , the method further comprising:
 receiving second user data at the benefits management platform;   generating second candidate plan recommendations based on the second user data;   providing the second candidate plan recommendations to the first client device of the actuarial reviewer;   receiving second actuarial recommendations from the first client device, the second actuarial recommendations comprising a recommended plan selected from the first candidate plan recommendations suggested by the plan recommendation model and a confidence score associated with the recommended plan;   generating second labeled training data for the plan recommendation model based on the actuarial recommendations; and   fine-tuning the plan recommendation model using the second labeled training data.   
     
     
         19 . A machine-readable medium on which are stored instructions that, when executed, cause a processor of alone or in combination with other processors to perform operations of:
 accessing first insurance plan data from an insurance plan data store, the first insurance plan data identifying a plurality of first medical insurance plans provided by an employer to a plurality of first users of a benefits management platform;   accessing first user data from a user information datastore, the first user data comprising information associated with a plurality of first users indicative of medical plan coverage needs of the plurality of first users;   generating first candidate plan recommendations using a plan recommendation model, the plan recommendation model being configured to identify one or more candidate medical insurance plans selected from among the plurality of first medical insurance plans for a selected user selected from among the plurality of first users based on user data associated with the selected user, each candidate plan recommendation including an indication of the selected user and indication of the one or more candidate medical insurance plans selected from among the plurality of first medical insurance plans;   providing the first candidate plan recommendations to a first client device of an actuarial reviewer to present on an actuarial review user interface of the benefits management platform;   receiving actuarial recommendations from the first client device, the actuarial recommendations comprising a recommended plan selected from the first candidate plan recommendations suggested by the plan recommendation model and a confidence score associated with the recommended plan;   generating labeled training data for the plan recommendation model based on the actuarial recommendations;   fine-tuning the plan recommendation model using the labeled training data;   receiving a request for a plan recommendation from a second client device for a second user, the request for the plan recommendation requesting one or more medical insurance plans selected from among the plurality of first medical insurance plans based on second user data associated with the second user;   providing the second user data as an input to the plan recommendation model to obtain second candidate plan recommendations; and   sending the second candidate plan recommendations to the second client device to present on a plan recommendation user interface of the benefits management platform.   
     
     
         20 . The machine-readable medium of  claim 19 , wherein the machine-readable medium further stores executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations of:
 generating a plurality of labeled samples that include sample user data representing a sample user for which sample plan recommendations were generated, a recommended plan selected from the sample plan recommendations, and a confidence score associated with the recommended plan indicating a confidence in recommending the recommended plan from among the sample plan recommendations.