US2026080696A1PendingUtilityA1
Method and system for analyzing pathological images based on magnification-aligned transformer (mat)
Assignee: GUANGDONG PROVINCIAL PEOPLES HOSPITALPriority: Sep 27, 2023Filed: Nov 25, 2025Published: Mar 19, 2026
Est. expirySep 27, 2043(~17.2 yrs left)· nominal 20-yr term from priority
G06T 2207/20081G06T 2207/20084G06T 7/0012G06V 2201/03G06V 10/267G06V 10/764G06V 10/454G06V 20/698G06T 2207/30024G06V 10/82G06N 3/088G06N 3/0464G06T 7/11
60
PatentIndex Score
0
Cited by
0
References
0
Claims
Abstract
A method for analyzing pathological images based on a magnification-aligned transformer (MAT) is provided, in which a pathological image dataset is identified and segmented to obtain pathological image patches; the pathological image patches is screened to obtain a patch set; an MAT classification network model including a self-supervised magnification alignment module and a global-local Transformer classification module is constructed; the MAT classification network model is trained for self-supervised magnification alignment using the patch set in the self-supervised magnification alignment module; the MAT classification network model is further trained using a convolutional neural network (CNN)-transformer; and a pathological image classification prediction result is obtained using the trained MAT classification network model. A system for implementing such method is also provided.
Claims
exact text as granted — not AI-modifiedWhat is claimed is:
1 . A method for analyzing pathological images based on a magnification-aligned Transformer (MAT), comprising:
acquiring a pathological image dataset composed of a plurality of whole-slide images (WSIs); identifying and segmenting a tissue region within each of the plurality of WSIs to obtain a mask corresponding to the tissue region; removing masks with a tissue area lower than a preset threshold; performing a patching operation on the tissue region based on the rest of the masks; constructing a MAT classification network model;
wherein the MAT classification network model comprises a self-supervised magnification alignment module and a global-local Transformer classification module;
the self-supervised magnification alignment module comprises two magnification-dependent feature extractors, wherein the two magnification-dependent feature extractors are structurally identical, and are respectively configured to extract features with fine-grained information and generate semantically aligned features with high similarity;
the global-local Transformer classification module comprises a global attention submodule and a local attention submodule; the global attention submodule is configured to explore global features among instances based on Transformer's capability to learn global information; and the local attention submodule is configured to explore local features among the instances based on convolutional neural network (CNN)'s capability to learn detail information;
training the MAT classification network model through steps of:
performing self-supervised magnification alignment training on the MAT classification network model to align low-magnification images to high-magnification images at a feature level to obtain a magnification-aligned feature representation;
exploring, by a transformer in the global attention submodule, global information of the magnification-aligned feature representation;
capturing, by a CNN in the local attention submodule, local information of the magnification-aligned feature representation;
aggregating pathological image features based on the global information and the local information to obtain aggregated features;
inferring, by the fully connected layer, the prediction result based on the aggregated features;
computing a prediction loss; and
obtaining a trained MAT classification network model through a backpropagation algorithm;
wherein a self-attention mechanism in the transformer of the global attention submodule covers all patches to obtain global attention; and the CNN in the local attention submodule, based on a fixed-size convolutional kernel and a sliding window mechanism, is more sensitive to adjacent patches; and
obtaining a pathological image classification prediction result using the trained MAT classification network model.
2 . The method of claim 1 , wherein the step of identifying and segmenting the tissue region within each of the plurality of WSIs to obtain the mask, and performing the patching operation on the tissue region based on the mask comprises:
performing binary classification on the pathological image dataset by using ImageNet to distinguish the tissue region from blank and contaminated regions, so as to obtain the mask corresponding to the tissue region; performing the patching operation on the tissue region based on the mask, wherein all WSIs are cropped into patches of a predetermined size; and inputting the patches of the predetermined size into a ResNet50 pre-trained on ImageNet or a pathology foundation model for feature extraction; wherein a size of the patches is scaled proportionally with a magnification of a corresponding WSI.
3 . The method of claim 1 , wherein the step of performing self-supervised magnification alignment training on the MAT classification network model comprises:
based on the two magnification-dependent feature extractors Φ High (·) and Φ Align (·), freezing parameters of the Φ High (·) to extract the high-magnification features; wherein the Φ High (·) is configured to receive a high-magnification image as input and output the high-magnification features; and inputting a low-resolution image having an identical field of view as the high-magnification image into the Φ Align (·) to generate the semantically-aligned features; and processing the high-magnification features and the semantically-aligned features by using an L1 loss function to reduce an absolute distance between features of different magnifications, so as to achieve semantic alignment, wherein the L1 loss function is expressed as:
L
MA
=
∑
i
=
1
n
❘
"\[LeftBracketingBar]"
X
i
-
x
i
❘
"\[RightBracketingBar]"
;
wherein X i is an output feature of an i-th patch from the Φ High (·), and x i is an output feature of an i-th patch from the Φ Align (·).
4 . The method of claim 1 , wherein the step of exploring, by the transformer in the global attention submodule, the global information of the magnification-aligned feature representation comprises:
inputting a feature F B×(N+1)×L of the patches into the global attention submodule; exploring the global information Out B×(N+1)×L of the magnification-aligned feature representation using the transformer in the global attention submodule through steps of:
generating a query (Q) vector, a key (K) vector and a value (V) vector using the fully connected layer;
generating a global attention matrix based on the self-attention mechanism using the Q vector and the K vector;
calculating a dot product of the global attention matrix and the V vector; and
concatenating the dot product with a randomly initialized class token to generate an output Out B×(N+1)×L as the global information, expressed as:
Q
=
MLP
(
F
B
×
N
×
L
)
;
K
=
MLP
(
F
B
×
N
×
L
)
;
V
=
MLP
(
F
B
×
N
×
L
)
;
Att
B
×
N
×
L
=
Softmax
(
Transpose
0
,
2
,
1
(
Q
×
K
)
)
;
Att
B
×
N
×
L
=
Concat
(
V
×
Transpose
0
,
2
,
1
(
Att
B
×
N
×
L
)
)
;
and
Out
B
×
(
N
+
1
)
×
L
=
Concatenate
(
Att
B
×
N
×
L
,
f
Class
1
×
L
)
;
wherein B represents a batch size; N represents the number of features; L represents a feature length; l represents a classification token feature; MLP represents a multi-layer perceptron; Q, K and V represent intermediate variables involved in conversion of the feature F B×N×L ; Att B×N×L represents an intermediate variable of the self-attention mechanism;
f
Class
1
×
L
is a class token, which is generated through a random initialization strategy and used to learn global instance information; and Transpose 0,2,1 (·) represents an operation that transposes dimensions of a tensor from (0, 1, 2) to (0, 2, 1).
5 . The method of claim 1 , wherein the step of capturing, by the CNN in the local attention submodule, the local information of the magnification-aligned feature representation comprises:
constructing an instance-level feature pyramid by using dilated convolutions in the CNN respectively with dilation rates of 1, 3 and 5 to capture instance information at three scales, respectively denoted as f 1 , f 2 and f 3 ; and fusing the instance information at the three scales by averaging to acquire the local information of the magnification-aligned feature representation, expressed as:
f
1
=
Conv
1
(
f
)
;
f
2
=
Conv
3
(
f
)
;
f
3
=
Conv
5
(
f
)
;
and
f
out
=
Mean
(
f
1
,
f
2
,
f
3
)
;
wherein f represents an input feature, Conv i (·) represents a dilated convolution with a dilation rate of i, Mean(·) represents an averaging pooling operation, and f out represents an output feature.
6 . The method of claim 1 , wherein the steps of inferring, by the fully connected layer, the prediction result based on the aggregated features, and computing the prediction loss comprise:
during a training process, computing a loss of the aggregated features through the fully connected layer using a cross-entropy loss function to generate augmented data, expressed as:
L
GLT
=
-
∑
i
=
i
n
y
i
log
;
wherein represents an output corresponding to an i-th WSI among the plurality of WSIs, and y i represents a label corresponding to the i-th WSI;
arranging the augmented data in the same order as that before aggregation of the pathological image features; and
inferring the prediction result using the fully connected layer of the MAT classification network model.
7 . The method of claim 1 , wherein the step of training the MAT classification network model further comprises:
before training, subjecting the magnification-aligned feature representation to data augmentation and class label embedding through steps of:
performing random sampling-based population on features of the magnification-aligned feature representation; and
concatenating populated features with a randomly initialized class token to serve as an input of a transformer layer.
8 . The method of claim 1 , wherein the pathological image features are aggregated based on the global information and the local information using an averaging operation.
9 . A system for implementing the method of claim 1 , comprising:
a feature engineering module; a model construction module; an alignment and attention training module; and an image testing module; wherein the feature engineering module is configured to perform:
acquiring the pathological image dataset composed of the plurality of WSIs;
identifying and segmenting the tissue region within each of the plurality of WSIs to obtain the mask corresponding to the tissue region;
removing masks with the tissue area lower than the preset threshold;
performing the patching operation on the tissue region based on the rest of the masks;
the model construction module is configured to construct the MAT classification network model; wherein the MAT classification network model comprises the self-supervised magnification alignment module and the global-local Transformer classification module; the self-supervised magnification alignment module comprises two magnification-dependent feature extractors, wherein the two magnification-dependent feature extractors are structurally identical, and are respectively configured to extract features with fine-grained information and generate semantically aligned features with high similarity; the global-local Transformer classification module comprises the global attention submodule and the local attention submodule; the global attention submodule is configured to explore global features among instances based on Transformer's capability to learn global information; and the local attention submodule is configured to explore local features among the instances based on CNN's capability to learn detail information; and the alignment and attention training module is configured to perform:
training the MAT classification network model through steps of:
performing self-supervised magnification alignment training on the MAT classification network model to align low-magnification images to high-magnification images at the feature level to obtain the magnification-aligned feature representation;
exploring, by the transformer in the global attention submodule, the global information of the magnification-aligned feature representation;
capturing, by the CNN in the local attention submodule, the local information of the magnification-aligned feature representation;
aggregating pathological image features based on the global information and the local information to obtain the aggregated features;
inferring, by the fully connected layer, the prediction based on the aggregated features;
computing the prediction loss; and
obtaining the trained MAT classification network model through the backpropagation algorithm;
wherein the self-attention mechanism in the transformer of the global attention submodule covers all patches in the magnification-aligned feature representation to obtain global attention; and the CNN in the local attention submodule, based on the fixed-size convolutional kernel and the sliding window mechanism, is more sensitive to adjacent patches in the magnification-aligned feature representation; and
the image testing module is configured to obtain the pathological image classification prediction result using the trained MAT classification network model.
10 . An electronic device, comprising:
at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory is configured for storing computer program instructions executable by the at least one processor; and the at least one processor is configured for executing the computer program instructions to implement the method of claim 1 .Join the waitlist — get patent alerts
Track US2026080696A1 — get alerts on status changes and closely related new filings.
We store only your email — no account needed. See our privacy policy.