Object recognition and behavioral analysis using semantic reasoning

Abstract

In one embodiment, a device detects an object in an area based on sensor data generated by a plurality of sensors deployed to that area. The device uses a semantic reasoning engine to determine an appearance type of the object, based on the sensor data. The device uses the semantic reasoning engine to determine a behavioral type of the object, based on the sensor data. The device makes, using the semantic reasoning engine, a determination that the behavioral type of the object does not match the appearance type of the object. The device provides an indication of the determination for display.

What is claimed is: 
     
         1 . A method comprising:
 detecting, by a device, an object in an area based on sensor data generated by a plurality of sensors deployed to that area;
 determining, by the device and by using a deep fusion reasoning engine that has a multi-level semantic architecture including sub-symbolic, conceptual, and symbolic reasoning layers, an appearance type of the object based on the sensor data and a knowledge base, wherein the knowledge base comprises a dynamic, graph-based structure storing information at different levels of abstraction, including sub-symbolic sensor features, conceptual-space embeddings, and symbolic relationships among object types in an ontology; 
 determining, by the device and by using the deep fusion reasoning engine, a behavioral type of the object based on the sensor data and the knowledge base; 
 making, by the device and by using the deep fusion reasoning engine, a determination that the behavioral type of the object does not match the appearance type of the object, wherein the determination is based on at least one of (i) a semantic similarity distance between the appearance type and the behavioral type that exceeds a threshold, and (ii) a determination that the behavioral type of the object changes over time; and 
 providing, by the device, an indication of the determination for display in a user interface. 
   
     
     
         2 . The method as in  claim 1 , wherein the sensor data comprises a video feed. 
     
     
         3 . The method as in  claim 1 , wherein the sensor data comprises one or more of: radar data, lidar data, or sonar data. 
     
     
         4 . The method as in  claim 1 , wherein the indication is an alert. 
     
     
         5 . The method as in  claim 1 , wherein the device provides an indication of the determination for display, based further on a location of the object in the area. 
     
     
         6 . The method as in  claim 1 , wherein the device provides an indication of the determination for display, based further on a trajectory of the object in the area. 
     
     
         7 . The method as in  claim 1 , wherein determining, by using the deep fusion reasoning engine, the behavioral type of the object comprises:
 determining that the behavioral type of the object is changing between different types over time; wherein the device makes the determination that the behavioral type of the object does not match the appearance type of the object based in part on the behavioral type of the object changing between different types over time.   
     
     
         8 . The method as in  claim 1 , wherein providing the indication of the determination for display comprises:
 providing the indication as an overlay for a map of the area.   
     
     
         9 . The method as in  claim 1 , wherein providing the indication of the determination for display comprises:
 providing the sensor data for display in conjunction with the indication.   
     
     
         10 . An apparatus, comprising:
 a network interface to communicate with a computer network;   a processor coupled to the network interface and configured to execute one or more processes; and   a memory configured to store a process that is executed by the processor, the process when executed configured to:
 detect an object in an area based on sensor data generated by a plurality of sensors deployed to that area; 
 determine, by using a deep fusion reasoning engine that has a multi-level semantic architecture including sub-symbolic, conceptual, and symbolic reasoning layers, an appearance type of the object based on the sensor data and a knowledge base, wherein the knowledge base comprises a dynamic, graph-based structure storing information at different levels of abstraction, including sub-symbolic sensor features, conceptual-space embeddings, and symbolic relationships among object types in an ontology; 
 determine, by using the deep fusion reasoning engine, a behavioral type of the object based on the sensor data and the knowledge base; 
 make, by using the deep fusion reasoning engine, a determination that the behavioral type of the object does not match the appearance type of the object, wherein the determination is based on at least one of (i) a semantic similarity distance between the appearance type and the behavioral type that exceeds a threshold, and (ii) a determination that the behavioral type of the object changes over time; and 
 provide an indication of the determination for display in a user interface. 
   
     
     
         11 . The apparatus as in  claim 10 , wherein the sensor data comprises a video feed. 
     
     
         12 . The apparatus as in  claim 10 , wherein the sensor data comprises one or more of: radar data, lidar data, or sonar data. 
     
     
         13 . The apparatus as in  claim 10 , wherein the indication is an alert. 
     
     
         14 . The apparatus as in  claim 10 , wherein the apparatus provides an indication of the determination for display, based further on a location of the object in the area. 
     
     
         15 . The apparatus as in  claim 10 , wherein the apparatus provides an indication of the determination for display, based further on a trajectory of the object in the area. 
     
     
         16 . The apparatus as in  claim 10 , wherein the apparatus determines, by using the deep fusion reasoning engine, the behavioral type of the object by:
 determining that the behavioral type of the object is changing between different types over time; wherein the apparatus makes the determination that the behavioral type of the object does not match the appearance type of the object based in part on the behavioral type of the object changing between different types over time.   
     
     
         17 . The apparatus as in  claim 10 , wherein the apparatus provides the indication of the determination for display by:
 providing the indication as an overlay for a map of the area.   
     
     
         18 . A tangible, non-transitory, computer-readable medium storing program instructions that cause a device to execute a process comprising:
 detecting, by a device, an object in an area based on sensor data generated by a plurality of sensors deployed to that area;   determining, by the device and by using a deep fusion reasoning engine that has a multi-level semantic architecture including sub-symbolic, conceptual, and symbolic reasoning layers, an appearance type of the object based on the sensor data and a knowledge base, wherein the knowledge base comprises a dynamic, graph-based structure storing information at different levels of abstraction, including the sub-symbolic sensor features, conceptual-space embeddings, and symbolic relationships among object types in an ontology;   determining, by the device and by using the deep fusion reasoning engine, a behavioral type of the object based no the sensor data and the knowledge base;   making, by the device and by using the deep fusion reasoning engine, a determination that the behavioral type of the object does not match the appearance type of the object, wherein the determination is based on at least one of (i) semantic similarity distance between the appearance type and the behavioral type that exceeds a threshold, and (ii) a determination that the behavioral type of the object changes over time; and   providing, by the device, an indication of the determination for display in a user interface.