US2008015910A1PendingUtilityA1
Ranking-based method and system for evaluating customer predication models
Est. expiryJul 11, 2026(expired)· nominal 20-yr term from priority
G06Q 30/02G06Q 10/063G06Q 10/063112G06Q 10/0639G06Q 30/0202G06Q 30/0204
48
PatentIndex Score
0
Cited by
0
References
0
Claims
Abstract
A method and system perform ranking-based evaluations for regression models that are often appropriate for marketing tasks and are more robust to outliers than traditional residual-based performance measures. The output provided by the method and system provides visualization that can offer insights about local model performance and outliers. Several models can be compared to each other to identify the “best” model and, therefore, the “best” model data for the particular marketing task.
Claims
exact text as granted — not AI-modified1 . A computer-implemented ranking-based method for evaluating regression models by obtaining predictions y from one or more models to be evaluated on a test set of customers for which the true value y of the quantity of interest has been observed comprising the steps of:
storing the test set resides in an electronic database sorted in increasing order such that “large” corresponds to one of a plurality of customers and/or potential customers with largest perceived spending budget, and “small” corresponds to one of a plurality of customers and/or potential customers with smallest perceived spending budget; applying all models to customer data from the test set to obtain predictions for the spending for each model and customer; converting the predictions into ranks and storing for each model in one or more electronic databases as a model ranking table, a number of ranking order switches relative to ranking of the observed customer spending being calculated for each model and ranking order switches being defined as those changes in ranking position of the prediction relative to the order of the true observations y; calculating a measure of a magnitude of erroneous ranking from a weighted sum of ranking order switches; transforming the number of ranking switches and weighted sum of ranking order switches into a range of [−1, 1] wherein −1 corresponds to making all possible errors (inverse ranking) and 1 corresponds to a perfect model wherein said number of ranking switches has been transformed to represent a difference between a probability that the ranking of two customers and/or potential customers are in the same order versus the probability that two of the customers and/or potential customers are in different orders from the originally obtained rank; normalizing transformed measures of order switches into a range of [0,1] wherein 1 corresponds to perfect ranking and 0 corresponds inverse ranking; calculating a variance of measures of order switches and determining confidence intervals for each ranking measure; updating the model performance table with the ranking measures and their confidence intervals; and outputting the ranking measures and confidence levels as well as graphical representations thereof to a domain expert, who will choose based on this information the best model m* of the one or more models evaluated, and storing the predictions of ŷ=m*(x) in the optimal prediction table.
2 . The computer-implemented ranking-based method recited in claim 1 , wherein the step of calculating a measure of a magnitude of erroneous ranking from a weighted sum of ranking order switches comprises the steps of:
calculating using computing resources a number of ranking order switches wherein ranking order switches are those changes in ranking order position of said plurality of customers and/or potential customers assuming said ranking obtained in said obtaining step was erroneous, and calculating using computing resources a weighted sum of ranking order switches wherein said weighted sum of ranking order switches provides a measure of the magnitude of erroneous ranking.
3 . The computer-implemented ranking-based method of claim 2 , wherein said calculating using computing resources the number of ranking order switches step comprises a relationship:
T=Σ i<j 1{s i >s j }
wherein, a set of variables of said relationship includes:
T is a number of rank order switches, ad
s i is a rank order of one customer of said plurality of customers and/or potential customers,
i is the integer from 1 to (n−1), and
j is an integer from (i+1) to n, and
n is a size of a set of observed customers.
4 . The computer-implemented ranking based method recited in claim 2 , wherein said calculating using computing resources a weighted sum of order switches step comprising a relationship:
R=Σ i<j ( j−i )1 {s i >s j }
wherein, a set of variables of said relationship includes:
R is a weighted sum of order switches and
s i is a rank order of one customer of said plurality of customers and/or potential customers,
i is the integer from 1 to (n−1), and
j is an integer from (i+1) to n, and
n is a size of a set of observed customers.
5 . The computer-implemented ranking-based method recited in claim 1 , wherein the step of calculating a variance of measures of order switches and determining confidence intervals for each ranking measure comprises the steps of:
calculating using computing resources a variance that said measures of said ranking order switches; calculating using computing resources a confidence interval for each measure of said ranking order switches; and constructing using computing resources graphical representations of said ranking order switches.
6 . The computer-implemented ranking-based method recited in claim 5 , wherein said calculating using computing resources a variance implements a relationship:
Var
^
(
τ
^
)
=
(
2
n
(
-
1
)
)
2
·
2
·
(
2
∑
i
C
i
2
-
∑
i
C
i
-
(
2
n
-
3
)
n
(
n
-
1
)
·
(
∑
i
C
i
)
2
)
wherein, a set of variables of said set of relationships includes:
C
i
=
∑
j
<
i
1
{
y
i
>
y
j
}
+
∑
j
>
i
1
{
y
i
<
y
j
}
is a number of observations that are concordant with observation i,
i is the integer from 1 to (n−1),
j is an integer from (i+1) to n,
n is a size of a set of customers and/or potential customers, and
y is a predicted response of the model y=m(x).
7 . The computer-implemented ranking-based method recited in claim 1 , wherein the step of outputting the ranking measures and confidence levels comprises the steps of:
providing an output that presents an updated ranking of said plurality of customers and/or potential customers that corresponds to ranking from model obtained in said obtaining step with highest confidence interval; and/or providing an output of a model recommendation that corresponds to a best model of said one or more models evaluated.
8 . The computer-implemented ranking-based method recited in claim 1 , wherein the step of transforming said number of ranking switches and said weighting sum of order switches step implements a set of relationships:
τ
^
=
1
-
4
T
n
(
n
-
1
)
,
and
ρ
^
=
1
-
12
R
n
(
n
-
1
)
(
n
+
1
)
wherein, a set of variables of said set of relationships includes:
{circumflex over (τ)} is a probability that the ranking order has been changed,
n is a size of a set of customers and/or potential customers,
{circumflex over (p)} is a ranking correlation,
T is a number of rank order switches,
R is said weighted sum of order switches,
9 . The computer-implemented ranking-based method recited in claim 1 , wherein the step of outputting provides an updated ranking of a plurality of customers and/or potential customers that corresponds to ranking from a model obtained in said obtaining step with highest confidence interval.
10 . A computer system for implementing a ranking-based method for evaluating regression models by obtaining predictions ŷ from one or more models to be evaluated on a test set of customers for which the true value y of the quantity of interest has been observed comprising:
a database storing the test set resides in an electronic database sorted in increasing order such that “large” corresponds to one of a plurality of customers and/or potential customers with largest perceived spending budget, and “small” corresponds to one of a plurality of customers and/or potential customers with smallest perceived spending budget; means for applying all models to customer data from the test set to obtain predictions for the spending for each model and customer and onverting the predictions into ranks and storing for each model in one or more electronic databases as a model ranking table, a number of ranking order switches relative to ranking of the observed customer spending being calculated for each model and ranking order switches being defined as those changes in ranking position of the prediction relative to the order of the true observations y; calculating means for calculating a measure of a magnitude of erroneous ranking from a weighted sum of ranking order switches; means for transforming the number of ranking switches and weighted sum of ranking order switches into a range of [−1, 1] wherein −1 corresponds to making all possible errors (inverse ranking) and 1 corresponds to a perfect model wherein said number of ranking switches has been transformed to represent a difference between a probability that the ranking of two customers and/or potential customers are in the same order versus the probability that two of the customers and/or potential customers are in different orders from the originally obtained rank; said calculating means normalizing transformed measures of order switches into a range of [0,1] wherein 1 corresponds to perfect ranking and 0 corresponds inverse ranking and calculating a variance of measures of order switches and determining confidence intervals for each ranking measure; means for updating the model performance table with the ranking measures and their confidence intervals; and output means for outputting the ranking measures and confidence levels as well as graphical representations thereof to a domain expert, who will choose based on this information the best model m* of the one or more models evaluated, and storing the predictions of ŷ=m*(x) in the optimal prediction table.
11 . The computer system recited in claim 10 , wherein said calculating means calculates a number of ranking order switches wherein ranking order switches are those changes in ranking order position of said plurality of customers and/or potential customers assuming said ranking obtained in said obtaining step was erroneous, and a weighted sum of ranking order switches wherein said weighted sum of ranking order switches provides a measure of the magnitude of erroneous ranking.
12 . The computer system recited in claim 11 , wherein said calculating means implements a relationship:
T=Σ i<j 1{s i >s j }
wherein, a set of variables of said relationship includes:
T is a number of rank order switches, ad
s i is a rank order of one customer of said plurality of customers and/or potential customers,
i is the integer from 1 to (n−1), and
j is an integer from (i+1) to n, and
n is a size of a set of observed customers.
13 . The computer system recited in claim 11 , wherein said calculating means implements a relationship:
R=Σ i>j ( j−i )1 {s i >s j }
wherein, a set of variables of said relationship includes:
R is a weighted sum of order switches and
s i is a rank order of one customer of said plurality of customers and/or potential customers,
i is the integer from 1 to (n−1), and
j is an integer from (i+1) to n, and
n is a size of a set of observed customers.
14 . The computer system recited in claim 10 , wherein said calculating means calculates a variance that said measures of said ranking order switches, a confidence interval for each measure of said ranking order switches, and graphical representations of said ranking order switches.
15 . The computer system recited in claim 14 , wherein said calculating means implements a relationship:
Var
^
(
τ
^
)
=
(
2
n
(
-
1
)
)
2
·
2
·
(
2
∑
i
C
i
2
-
∑
i
C
i
-
(
2
n
-
3
)
n
(
n
-
1
)
·
(
∑
i
C
i
)
2
)
wherein, a set of variables of said set of relationships includes:
C
i
=
∑
j
<
i
1
{
y
i
>
y
j
}
+
∑
j
>
i
1
{
y
i
<
y
j
}
is a number of observations that are concordant with observation i,
i is the integer from 1 to (n−1),
j is an integer from (i+1) to n,
n is a size of a set of customers and/or potential customers, and
y is a predicted response of the model y=m(x).
16 . The computer system recited in claim 10 , wherein said output means provides an output that presents an updated ranking of said plurality of customers and/or potential customers that corresponds to ranking from model obtained in said obtaining step with highest confidence interval and/or provides an output of a model recommendation that corresponds to a best model of said one or more models evaluated.
17 . The computer system recited in claim 10 , wherein said transforming means implements a set of relationships:
τ
^
=
1
-
4
T
n
(
n
-
1
)
,
and
ρ
^
=
1
-
12
R
n
(
n
-
1
)
(
n
+
1
)
wherein, a set of variables of said set of relationships includes:
{circumflex over (τ)} is a probability that the ranking order has been changed,
n is a size of a set of customers and/or potential customers,
{circumflex over (p)} is a ranking correlation,
T is a number of rank order switches,
R is said weighted sum of order switches,
18 . A computer readable media containing code which implements a ranking-based method for evaluating regression models by obtaining predictions ŷ from one or more models to be evaluated on a test set of customers for which the true value y of the quantity of interest has been observed, the method comprising the steps of:
storing the test set resides in an electronic database sorted in increasing order such that “large” corresponds to one of a plurality of customers and/or potential customers with largest perceived spending budget, and “small” corresponds to one of a plurality of customers and/or potential customers with smallest perceived spending budget; applying all models to customer data from the test set to obtain predictions for the spending for each model and customer; converting the predictions into ranks and storing for each model in one or more electronic databases as a model ranking table, a number of ranking order switches relative to ranking of the observed customer spending being calculated for each model and ranking order switches being defined as those changes in ranking position of the prediction relative to the order of the true observations y; calculating a measure of a magnitude of erroneous ranking from a weighted sum of ranking order switches; transforming the number of ranking switches and weighted sum of ranking order switches into a range of [−1, 1] wherein −1 corresponds to making all possible errors (inverse ranking) and 1 corresponds to a perfect model wherein said number of ranking switches has been transformed to represent a difference between a probability that the ranking of two customers and/or potential customers are in the same order versus the probability that two of the customers and/or potential customers are in different orders from the originally obtained rank; normalizing transformed measures of order switches into a range of [0,1] wherein 1 corresponds to perfect ranking and 0 corresponds inverse ranking; calculating a variance of measures of order switches and determining confidence intervals for each ranking measure; updating the model performance table with the ranking measures and their confidence intervals; outputting the ranking measures and confidence levels as well as graphical representations thereof to a domain expert, who will choose based on this information the best model m* of the one or more models evaluated, and storing the predictions of ŷ=m*(x) in the optimal prediction table.Join the waitlist — get patent alerts
Track US2008015910A1 — get alerts on status changes and closely related new filings.
We store only your email — no account needed. See our privacy policy.