Apparatus including NPU operated based on machine codes corresponding to ANN model

Abstract

An electronic device mounted on a fixed or a movable apparatus is provided. The electronic device may comprise an image signal processor (ISP) for at least one camera; a neural processing unit (NPU), including a plurality of processing elements (PEs), configured to: process an operation of an artificial neural network model trained to detect or track at least one object, based on an input feature map generated from at least one image, which is acquired via the ISP from the at least one camera, and output an inference result; and a signal generator generating a signal applicable to the at least one camera or the ISP.

What is claimed is: 
     
         1 . An apparatus comprising:
 at least one camera;   a memory configured to store a plurality of machine codes compiled for a plurality of artificial neural network (ANN) models;   a first semiconductor chip, provided for a neural processing unit (NPU), including:
 a first circuitry provided for an internal memory configured to store an input feature map and an output feature map, 
 a second circuitry provided for a direct memory access (DMA) configured to access and control the first circuitry, 
 a third circuitry provided for a plurality of processing elements (PEs) configured to process an operation of each ANN model, based on a machine code and the input feature map, and to output the output feature map, and 
 a fourth circuitry provided for a controller configured to control the first circuitry, the second circuitry and the third circuitry; 
   a second semiconductor chip, provided for a central processing unit (CPU), connected to the memory and the first semiconductor chip and configured to select the machine code in the memory; and   a signal generator configured to generate a signal applicable to control the at least one camera,   wherein when at least one object is detected or tracked with a confidence level lower than a corresponding threshold, the signal is generated to include a coordinate of the at least one object.   
     
     
         2 . The apparatus of  claim 1 , wherein the second semiconductor chip is further configured to deliver information on the selected machine code to the first semiconductor chip. 
     
     
         3 . The apparatus of  claim 1 , wherein the first semiconductor chip is configured to acquire the selected machine code from the memory. 
     
     
         4 . The apparatus of  claim 1 ,
 wherein the plurality of PEs are configured to receive the input feature map from the internal memory and output the output feature map to the internal memory, and   wherein the second semiconductor chip is further configured to receive an inference result from the first semiconductor chip.   
     
     
         5 . The apparatus of  claim 1 , further comprising:
 one or more sensors; and   an image signal processor (ISP).   
     
     
         6 . The apparatus of  claim 5 ,
 wherein the ISP generates the input feature map which has an adjustable dimension to be inputted to the first semiconductor chip.   
     
     
         7 . The apparatus of  claim 5 , wherein the one or more sensors are configured to sense a flight altitude or a flight speed. 
     
     
         8 . The apparatus of  claim 1 , wherein the plurality of ANN models include at least one ANN model trained to detect or track at least one object, based on an input feature map generated from at least one image, which is acquired from the at least one camera. 
     
     
         9 . The apparatus of  claim 1 ,
 wherein the coordinate is expressed by (a1, b1) and (a2, b2) with respect to two diagonal vertices of a rectangle which contains the at least one object.   
     
     
         10 . The apparatus of  claim 1 ,
 wherein a resolution, size or dimension of the input feature map is adjustable based on a determination data, and   wherein the dimension of the input feature map includes a horizontal size and vertical size of the input feature map and a number of channels.   
     
     
         11 . The apparatus of  claim 1 , further comprising:
 a signal generator configured to generate a signal for the input feature map,   wherein the generated signal includes at least one command for controlling generation of the input feature map or controlling at least one attribute of the input feature map.   
     
     
         12 . The apparatus of  claim 11 ,
 wherein the at least one attribute includes a size of the input feature map or a resolution of the input feature map.   
     
     
         13 . The apparatus of  claim 12 ,
 wherein the size or the resolution is adjusted based on a flight altitude or a flight speed.   
     
     
         14 . The apparatus of  claim 11 ,
 wherein the at least one command increases a number of the input feature map to be generated per second.   
     
     
         15 . The apparatus of  claim 11 ,
 wherein when the at least one attribute is a size of the input feature map, the size of the input feature map is reduced based on the generated signal, and a larger number of the input feature map is generated per second.   
     
     
         16 . The apparatus of  claim 11 ,
 wherein when a flight altitude rises or a flight speed increases, the generated signal is used to increase a resolution corresponding to the at least one attribute.   
     
     
         17 . The apparatus of  claim 1 ,
 wherein at least one image acquired from the at least one camera is partitioned into a plurality of image blocks based on a determination data, and   wherein the determination data is set to determine a number of the plurality of image blocks.   
     
     
         18 . The apparatus of  claim 17 ,
 wherein when a flight altitude increase, the number of the plurality of image blocks is increased.   
     
     
         19 . The apparatus of  claim 17 ,
 wherein when the number of the plurality of image blocks is greater than a first threshold, the plurality of image blocks are upscaled in size before being input to the NPU.   
     
     
         20 . The apparatus of  claim 1 , further comprising:
 an image preprocessor configured to divide at least one image acquired from the at least one camera into a plurality of image blocks,   wherein a number of the plurality of image blocks is determined based on a motion of the apparatus.