Information aggregation in a multi-modal entity-feature graph for intervention prediction for a medical patient

Abstract

A computer-implemented method for providing event-specific intervention recommendations includes acquiring at least two data streams of a patient by using one or more sensors, wherein at least one the data streams includes images. At least one entity-feature-graph is generated based on the acquired at least two data streams of the patient. At least one intervention is selected based on the generated entity-feature-graph and a trained graph classification model. An information of the selected intervention is output to a user. The method has applications including, but not limited to, use cases in medical/healthcare for optimizing machine learning and supporting decision making.

1 . A computer-implemented method for providing event-specific intervention recommendations, the method comprising:
 acquiring at least two data streams of a patient by using one or more sensors, wherein at least one the data streams includes images;   generating at least one entity-feature-graph based on the acquired at least two data streams of the patient;   selecting at least one intervention based on the generated entity-feature-graph and a trained graph classification model; and   outputting an information of the selected intervention to a user.   
     
     
         2 . The method according to  claim 1 , wherein the trained graph classification model is trained by a learning process that considers confidence values assigned to each entity of a set of entities within the entity-feature-graph, the confidence values indicating confidence levels with which respective ones of the entities are extracted from the at least two data streams. 
     
     
         3 . The method according to  claim 2 , wherein the entity-feature-graph is transformed into an embedding space by a transformation process that makes use of distances between each pair of entities in the embedding space to encode corresponding probabilities that respective pairs of the entities are the same. 
     
     
         4 . The method according to  claim 3 , wherein the graph classification is performed based on using a graph neural network that is given as input the entity-feature-graph together with a probability information that indicates for each pair of entities of the set of entities a likelihood that both entities of a respective pair are the same. 
     
     
         5 . The method according to  claim 1 , further comprising:
 processing image data of the images contained in the at least one of the at least two data streams, which is the basis for decision making, and scanning the images for location information.   
     
     
         6 . The method according to  claim 5 , further comprising:
 determining based on the image data that the patient has suffered an accident; and   executing the selected intervention of establishing a phone connection of an emergency contact associated with the patient.   
     
     
         7 . The method according to  claim 5 , further comprising:
 determining based on the image data of the at least two data streams that the patient has undergone a surgery and is underactive; and   executing the selected intervention of adapting a therapy associated with the patient.   
     
     
         8 . The method according to  claim 7 , wherein it is determined based on the at least two data streams that the patient is watching television and the selected intervention includes increasing a difficulty of sports equipment of the patient. 
     
     
         9 . The method according to  claim 1 , wherein the graph classification model is learned based on training data extracted from historical data streams and previous intervention selection decisions. 
     
     
         10 . The method according to  claim 1 , wherein the at least two data streams include at least one data stream of text and/or at least one data stream of speech that is converted to text, and wherein the one or more sensors include at least one of a camera, sound recorder presence sensor, a temperature sensor, a sound-level sensor, and a door sensor. 
     
     
         11 . A computer system for providing event-specific intervention recommendations, the system comprising one or more processors configured to execute the following steps:
 acquiring at least two data streams of a patient by using one or more sensors, wherein at least one the data streams includes images;   generating at least one entity-feature-graph based on the acquired at least two data streams of the patient;   selecting at least one intervention based on the generated entity-feature-graph and a trained graph classification model; and   outputting an information of the selected intervention to a user.   
     
     
         12 . The system according to  claim 11 , wherein the trained graph classification model is trained by a learning process that considers confidence values assigned to each entity of a set of entities within the entity-feature-graph, the confidence values indicating confidence levels with which respective ones of the entities are extracted from the at least two data streams. 
     
     
         13 . The system according to  claim 12 , wherein the entity-feature-graph is transformed into an embedding space by a transformation process that makes use of distances between each pair of entities in the embedding space to encode corresponding probabilities that respective pairs of the entities are the same. 
     
     
         14 . The system according to  claim 13 , wherein the graph classification is performed based on using a graph neural network that is given as input the entity-feature-graph together with a probability information that indicates for each pair of entities of the set of entities a likelihood that both entities of a respective pair are the same. 
     
     
         15 . The system according to  claim 11 , wherein the one or more processors are further configured to:
 process image data of the images contained in the at least one of the at least two data streams, which is the basis for decision making, and scanning the images for location information.   
     
     
         16 . The system according to  claim 15 , wherein the one or more processors are further configured to:
 determine based on the image data that the patient has suffered an accident; and   execute the selected intervention of establishing a phone connection of an emergency contact associated with the patient.   
     
     
         17 . The system according to  claim 11 , wherein the graph classification model is learned based on training data extracted from historical data streams and previous intervention selection decisions. 
     
     
         18 . The system according to  claim 11 , wherein the at least two data streams include at least one data stream of text and/or at least one data stream of speech that is converted to text, and wherein the one or more sensors include at least one of a camera, sound recorder, presence sensor, a temperature sensor, a sound-level sensor, and a door sensor. 
     
     
         19 . A tangible, non-transitory computer-readable medium having instructions thereon which, upon being executed by one or more processors, alone or in combination, provide for execution of a method for providing event-specific intervention recommendations, the method comprising:
 acquiring at least two data streams of a patient by using one or more sensors, wherein at least one the data streams includes images;   generating at least one entity-feature-graph based on the acquired at least two data streams of the patient;   selecting at least one intervention based on the generated entity-feature-graph and a trained graph classification model; and   outputting an information of the selected intervention to a user.   
     
     
         20 . The tangible, non-transitory computer-readable medium according to  claim 19 , wherein the trained graph classification model is trained by a learning process that considers confidence values assigned to each entity of a set of entities within the entity-feature-graph, the confidence values indicating confidence levels with which respective ones of the entities are extracted from the at least two data streams.