US2025005370A1PendingUtilityA1
Multitask prompt tuning for parameter-efficient transfer learning
Est. expiryJun 29, 2043(~16.9 yrs left)· nominal 20-yr term from priority
G06N 3/08G06N 3/045G06N 3/096
57
PatentIndex Score
0
Cited by
0
References
0
Claims
Abstract
A source task prompt of each of a plurality of source tasks is decomposed as a multiplication of a shared prompt matrix shared across source tasks and a low-rank task-specific matrix. Prompt distillation is performed to transfer multitask knowledge to the shared prompt matrix by distilling knowledge from the source task prompts. Low-rank multiplicative updates are performed to the shared prompt matrix to transfer the multitask knowledge to one or more target tasks. The one or more target tasks (e.g., natural language processing) are carried out in accordance with the transferred knowledge
Claims
exact text as granted — not AI-modifiedWhat is claimed is:
1 . A method comprising:
decomposing, using a hardware processor, a source task prompt of each of a plurality of source tasks as a multiplication of a shared prompt matrix shared across the source tasks and a low-rank task-specific matrix; performing, using the hardware processor, prompt distillation to transfer multitask knowledge to the shared prompt matrix by distilling knowledge from the source task prompts; performing, using the hardware processor, low-rank multiplicative updates to the shared prompt matrix to transfer the multitask knowledge to one or more target tasks; and using the hardware processor, carrying out the one or more target tasks in accordance with the transferred knowledge.
2 . The method of claim 1 , wherein the decomposition is learned by performing the prompt distillation obtained from regular prompt tuning.
3 . The method of claim 1 , wherein each source task is represented as a prompt matrix.
4 . The method of claim 1 , wherein the decomposition enables efficient knowledge sharing across source tasks while still allowing each source task to maintain corresponding parameters for encoding task-specific knowledge.
5 . The method of claim 1 , wherein the decomposing of the source task prompt decomposes a k-th task into two decomposition parts, where P*∈ l×d denotes an initial shared task prompt across all source tasks and u k ∈ l , v k ∈ d are task-specific vectors for each task k, the task-specific vectors forming a rank-one matrix W k =u k ·v k T , which has a same dimension as the initial shared task prompt.
6 . The method of claim 5 , further comprising parametrizing a final shared task prompt {circumflex over (P)} for a k-th source task as:
P
^
k
=
P
*
∘
W
k
=
P
*
∘
(
u
k
·
v
k
T
)
where ∘ denotes a Hadamard product between two matrices, wherein a parameterization of prompt decomposition captures general information of a corresponding source task S by weights shared across the source tasks and weights W k encode task-specific knowledge in a low-rank subspace.
7 . The method of claim 1 , further comprising:
obtaining a teacher prompt P k t for a k-th source task using prompt tuning; and performing multitask training on the source tasks to jointly learn the single shared soft prompt via a knowledge distillation loss function.
8 . The method of claim 7 , further comprising:
randomly initializing a corresponding student prompt as =P*∘(u k ·v k T ), where all student prompts share P* and have corresponding task-specific vectors.
9 . The method of claim 7 , wherein a knowledge distillation loss is configured to transfer cross-task knowledge into the shared prompt matrix.
10 . The method of claim 9 , wherein a first distillation loss matches output probability distributions of student models and teacher models through minimizing a corresponding KL-Divergence,
ℒ
Logits
=
∑
k
∑
i
∈
S
k
KL
(
P
(
y
i
❘
"\[LeftBracketingBar]"
[
P
k
t
;
x
i
]
)
,
P
(
y
i
❘
"\[LeftBracketingBar]"
[
;
x
i
]
)
)
.
11 . The method of claim 10 , further comprising controlling, using a temperature T, a smoothness of an output distribution for both the teacher model and the student model as
p
j
=
1
z
exp
(
z
j
/
T
)
,
where z i is a logit score for class j and Z is a normalization factor and wherein an additional mean squared loss on hidden states of the teacher model is defined as:
ℒ
Hidden
=
∑
k
∑
i
∈
S
k
(
H
ki
s
-
H
ki
t
)
2
,
where H ki s , H ki t denotes hidden states of teacher and student networks, respectively, comprising a sequence of hidden vectors for an i-th input.
12 . The method of claim 9 , further comprising training a student source prompt for obtaining a single soft task prompt to be transferred to a target task using a total loss function defined by:
ℒ
Total
=
ℒ
PLM
+
λ
(
ℒ
Logits
+
ℒ
Hidden
)
,
wherein PLM =Σ k PLM k represents aggregated task losses for source tasks, and λ is a weight to balance an impact of distillation loss terms.
13 . The method of claim 1 , wherein the one or more target tasks comprise natural language processing.
14 . A computer program product, comprising:
one or more tangible computer-readable storage media and program instructions stored on at least one of the one or more tangible computer-readable storage media, the program instructions executable by a processor, the program instructions comprising:
decomposing a source task prompt of each of a plurality of source tasks as a multiplication of a shared prompt matrix shared across the source tasks and a low-rank task-specific matrix;
performing prompt distillation to transfer multitask knowledge to the shared prompt matrix by distilling knowledge from the source task prompts;
performing low-rank multiplicative updates to the shared prompt matrix to transfer the multitask knowledge to one or more target tasks; and
carrying out the one or more target tasks in accordance with the transferred knowledge.
15 . The computer program product of claim 14 , wherein the decomposition is learned by performing the prompt distillation obtained from regular prompt tuning.
16 . The computer program product of claim 14 , wherein each source task is represented as a prompt matrix.
17 . The computer program product of claim 15 , wherein the decomposing of the source task prompt decomposes a k-th task into two decomposition parts, where P*∈ l×d denotes an initial shared task prompt across all source tasks and u k ∈ l , v k ∈ d are task-specific vectors for each task k, the task-specific vectors forming a rank-one matrix W k =u k ·v k T , which has a same dimension as the initial shared task prompt.
18 . The computer program product of claim 14 , wherein the one or more target tasks comprise natural language processing.
19 . A system comprising:
a memory; and at least one processor, coupled to said memory, and operative to perform operations comprising:
decomposing a source task prompt of each of a plurality of source tasks as a multiplication of a shared prompt matrix shared across the source tasks and a low-rank task-specific matrix;
performing prompt distillation to transfer multitask knowledge to the shared prompt matrix by distilling knowledge from the source task prompts;
performing low-rank multiplicative updates to the shared prompt matrix to transfer the multitask knowledge to one or more target tasks; and
carrying out the one or more target tasks in accordance with the transferred knowledge.
20 . The system of claim 19 , wherein the decomposition is learned by performing the prompt distillation obtained from regular prompt tuning.
21 . The system of claim 19 , wherein the decomposition enables efficient knowledge sharing across source tasks while still allowing each source task to maintain corresponding parameters for encoding task-specific knowledge.
22 . The system of claim 19 , wherein the decomposing of the source task prompt decomposes a k-th task into two decomposition parts, where P*∈ l×d denotes an initial shared task prompt across all source tasks and u k ∈ l , v k ∈ d are task-specific vectors for each task k, the task-specific vectors forming a rank-one matrix W k =u k ·v k T , which has a same dimension as the initial shared task prompt.
23 . The system of claim 22 , further comprising parametrizing a final shared task prompt {circumflex over (P)} for a k-th source task as:
P
^
k
=
P
*
∘
W
k
=
P
*
∘
(
u
k
·
v
k
T
)
where ∘ denotes a Hadamard product between two matrices, wherein a parameterization of prompt decomposition captures general information of a corresponding source task S by weights shared across the source tasks and weights W k encode task-specific knowledge in a low-rank subspace.
24 . The system of claim 19 , further comprising:
obtaining a teacher prompt P k t for a k-th source task using prompt tuning; and performing multitask training on the source tasks to jointly learn the single shared soft prompt via a knowledge distillation loss function.
25 . The system of claim 19 , wherein the one or more target tasks comprise natural language processing.Join the waitlist — get patent alerts
Track US2025005370A1 — get alerts on status changes and closely related new filings.
We store only your email — no account needed. See our privacy policy.