US2021019753A1PendingUtilityA1
System, Method, and Computer Program Product for Determining a Reason for a Deep Learning Model Output
Est. expiryJul 18, 2039(~13 yrs left)· nominal 20-yr term from priority
G06N 3/045G06N 3/08G06N 3/0499G06N 3/09G06Q 20/4016G06N 5/045G06F 17/16G06F 17/18G06N 3/0454
53
PatentIndex Score
0
Cited by
0
References
0
Claims
Abstract
A system, method, and product for determining a reason for a deep learning model output that obtain training data associated with training samples and first labels for the training samples; train a first model using the training samples and the first labels, training the first model generating predictions for the training samples; train a second model using the training samples and the predictions as second labels for the training samples; extract one or more weights of the trained second model; process, using the first model, input data including features associated with at least one sample, to generate output data, the output data including at least one prediction for the at least one sample; and apply the one or more extracted weights to the features to determine one or more contributions of one or more features of the features to the at least one prediction for the at least one sample.
Claims
exact text as granted — not AI-modifiedWhat is claimed is:
1 . A computer-implemented method comprising:
obtaining, with at least one processor, training data associated with a plurality of training samples and a plurality of first labels for the plurality of training samples; training, with at least one processor, a first model using the plurality of training samples and the plurality of first labels for the plurality of training samples, wherein training the first model generates a plurality of predictions for the plurality of training samples; training, with at least one processor, a second model using the plurality of training samples and the plurality of predictions as a plurality of second labels for the plurality of training samples; extracting, with at least one processor, one or more weights of the trained second model; processing, with at least one processor using the first model, input data including a plurality of features associated with at least one sample, to generate output data, wherein the output data includes at least one prediction for the at least one sample; and applying, with at least one processor, the one or more extracted weights to the plurality of features to determine one or more contributions of one or more features of the plurality of features to the at least one prediction for the at least one sample.
2 . The computer-implemented method of claim 1 , wherein the input data includes transaction data, wherein the at least one sample is associated with at least one transaction, and wherein the at least one prediction includes at least one probability of whether the at least one transaction is a fraudulent transaction.
3 . The computer-implemented method of claim 2 , wherein the one or more contributions are determined in real-time during processing of the at least one transaction in a transaction processing network.
4 . The computer-implemented method of claim 1 , wherein the first model includes a greater number of hidden layers than the second model.
5 . The computer-implemented method of claim 1 , further comprising:
ranking, with at least one processor, the plurality of features associated with the at least one sample according to the one or more contributions of the one or more features of the plurality of features to the at least one prediction for the at least one sample; and identifying, with at least one processor, at least one feature of the plurality of features with respect to at least one other feature of the plurality of features based on the ranked plurality of features as at least one reason code associated with the at least one prediction.
6 . The computer-implemented method of claim 1 , wherein the one or more extracted weights include a plurality of extracted weights, wherein the plurality of extracted weights is associated with a plurality of hidden units of at least one hidden layer of the trained second model, and wherein the one or more contributions of the one or more features of the plurality of features to the at least one prediction for the at least one sample is determined by:
calculating a score for each path of a plurality of paths of the one or more features through the second model based on the plurality of extracted weights and at least one activation function; and summing each score for each path of the plurality of paths to determine a contribution score for the one or more contributions of the one or more features.
7 . The computer-implemented method of claim 1 , further comprising:
determining, by at least one processor, a contribution score for each contribution of the one or more contributions of the one or more features of the plurality of features to the at least one prediction for the at least one sample according to the following Equations (1) to (4):
h
j
=
∑
i
=
1
K
x
i
*
u
ij
(
1
)
a
j
=
max
(
0
,
h
j
)
(
2
)
y
m
=
∑
j
=
1
H
a
j
*
v
jm
(
3
)
Contribution
(
x
i
->
y
m
)
=
∑
j
=
1
H
(
x
i
*
u
ij
)
*
(
a
j
*
v
jm
)
(
4
)
wherein Contribution (x i −>y m ) is the contribution score of a feature x i to a prediction y m , wherein K is a number of the plurality of features, wherein H is a number of hidden units in a hidden layer of the trained second model, wherein M is a number of output units of an output layer of the trained second model, wherein U and V are weight matrices, wherein a number of weight matrices U is equal to K×H, wherein a number of weight matrices V is equal to H×M, wherein u ij is a weight matrix applied to the feature on a path to a hidden unit h j , wherein a j is an activation unit of an activation layer between the hidden layer and the output layer of the trained second model, wherein v jm i is a weight matrix applied to an output of the activation unit a j on a path to the prediction y m .
8 . A computing system comprising:
one or more processors programmed and/or configured to:
obtain training data associated with a plurality of training samples and a plurality of first labels for the plurality of training samples;
train a first model using the plurality of training samples and the plurality of first labels for the plurality of training samples, wherein training the first model generates a plurality of predictions for the plurality of training samples;
train a second model using the plurality of training samples and the plurality of predictions as a plurality of second labels for the plurality of training samples;
extract one or more weights of the trained second model;
process, using the first model, input data including a plurality of features associated with at least one sample, to generate output data, wherein the output data includes at least one prediction for the at least one sample; and
apply the one or more extracted weights to the plurality of features to determine one or more contributions of one or more features of the plurality of features to the at least one prediction for the at least one sample.
9 . The computing system of claim 8 , wherein the input data includes transaction data, wherein the at least one sample is associated with at least one transaction, and wherein the at least one prediction includes at least one probability of whether the at least one transaction is a fraudulent transaction.
10 . The computing system of claim 9 , wherein the one or more contributions are determined in real-time during processing of the at least one transaction in a transaction processing network.
11 . The computing system of claim 8 , wherein the first model includes a greater number of hidden layers than the second model.
12 . The computing system of claim 8 , wherein the one or more processors are further programmed and/or configured to:
rank the plurality of features associated with the at least one sample according to the one or more contributions of the one or more features of the plurality of features to the at least one prediction for the at least one sample; and identify at least one feature of the plurality of features with respect to at least one other feature of the plurality of features based on the ranked plurality of features as at least one reason code associated with the at least one prediction.
13 . The computing system of claim 8 , wherein the one or more extracted weights include a plurality of extracted weights, wherein the plurality of extracted weights is associated with a plurality of hidden units of at least one hidden layer of the trained second model, and wherein the one or more contributions of the one or more features of the plurality of features to the at least one prediction for the at least one sample is determined by:
calculating a score for each path of a plurality of paths of the one or more features through the second model based on the plurality of extracted weights and at least one activation function; and summing each score for each path of the plurality of paths to determine a contribution score for the one or more contributions of the one or more features.
14 . The computing system of claim 8 , wherein the one or more processors are further programmed and/or configured to:
determine a contribution score for each contribution of the one or more contributions of the one or more features of the plurality of features to the at least one prediction for the at least one sample according to the following Equations (1) to (4):
h
j
=
∑
i
=
1
K
x
i
*
u
ij
(
1
)
a
j
=
max
(
0
,
h
j
)
(
2
)
y
m
=
∑
j
=
1
H
a
j
*
v
jm
(
3
)
Contribution
(
x
i
->
y
m
)
=
∑
j
=
1
H
(
x
i
*
u
ij
)
*
(
a
j
*
v
jm
)
(
4
)
wherein Contribution (x i −>y m ) is the contribution score of a feature to a prediction y m , wherein K is a number of the plurality of features, wherein H is a number of hidden units in a hidden layer of the trained second model, wherein M is a number of output units of an output layer of the trained second model, wherein U and V are weight matrices, wherein a number of weight matrices U is equal to K×H, wherein a number of weight matrices V is equal to H×M, wherein u ij is a weight matrix applied to the feature on a path to a hidden unit h j , wherein a j is an activation unit of an activation layer between the hidden layer and the output layer of the trained second model, wherein v i m is a weight matrix applied to an output of the activation unit a j on a path to the prediction y m .
15 . A computer program product comprising at least one non-transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to:
obtain training data associated with a plurality of training samples and a plurality of first labels for the plurality of training samples; train a first model using the plurality of training samples and the plurality of first labels for the plurality of training samples, wherein training the first model generates a plurality of predictions for the plurality of training samples; train a second model using the plurality of training samples and the plurality of predictions as a plurality of second labels for the plurality of training samples; extract one or more weights of the trained second model; process, using the first model, input data including a plurality of features associated with at least one sample, to generate output data, wherein the output data includes at least one prediction for the at least one sample; and apply the one or more extracted weights to the plurality of features to determine one or more contributions of one or more features of the plurality of features to the at least one prediction for the at least one sample.
16 . The computer program product of claim 15 , wherein the input data includes transaction data, wherein the at least one sample is associated with at least one transaction, wherein the at least one prediction includes at least one probability of whether the at least one transaction is a fraudulent transaction, and wherein the one or more contributions are determined in real-time during processing of the at least one transaction in a transaction processing network.
17 . The computer program product of claim 15 , wherein the first model includes a greater number of hidden layers than the second model.
18 . The computer program product of claim 15 , wherein the one or more processors are further programmed and/or configured to:
rank the plurality of features associated with the at least one sample according to the one or more contributions of the one or more features of the plurality of features to the at least one prediction for the at least one sample; and identify at least one feature of the plurality of features with respect to at least one other feature of the plurality of features based on the ranked plurality of features as at least one reason code associated with the at least one prediction.
19 . The computer program product of claim 15 , wherein the one or more extracted weights include a plurality of extracted weights, wherein the plurality of extracted weights is associated with a plurality of hidden units of at least one hidden layer of the trained second model, and wherein the one or more contributions of the one or more features of the plurality of features to the at least one prediction for the at least one sample is determined by:
calculating a score for each path of a plurality of paths of the one or more features through the second model based on the plurality of extracted weights and at least one activation function; and summing each score for each path of the plurality of paths to determine a contribution score for the one or more contributions of the one or more features.
20 . The computer program product of claim 15 , wherein the one or more processors are further programmed and/or configured to:
determine a contribution score for each contribution of the one or more contributions of the one or more features of the plurality of features to the at least one prediction for the at least one sample according to the following Equations (1) to (4):
h
j
=
∑
i
=
1
K
x
i
*
u
ij
(
1
)
a
j
=
max
(
0
,
h
j
)
(
2
)
y
m
=
∑
j
=
1
H
a
j
*
v
jm
(
3
)
Contribution
(
x
i
->
y
m
)
=
∑
j
=
1
H
(
x
i
*
u
ij
)
*
(
a
j
*
v
jm
)
(
4
)
wherein Contribution (x i −>y m ) is the contribution score of a feature to a prediction y m , wherein K is a number of the plurality of features, wherein H is a number of hidden units in a hidden layer of the trained second model, wherein M is a number of output units of an output layer of the trained second model, wherein U and V are weight matrices, wherein a number of weight matrices U is equal to K×H, wherein a number of weight matrices V is equal to H×M, wherein u ij is a weight matrix applied to the feature on a path to a hidden unit h j , wherein a j is an activation unit of an activation layer between the hidden layer and the output layer of the trained second model, wherein v jm is a weight matrix applied to an output of the activation unit a j on a path to the prediction y m .Join the waitlist — get patent alerts
Track US2021019753A1 — get alerts on status changes and closely related new filings.
We store only your email — no account needed. See our privacy policy.