Pipelined Hardware Accelerator for Neural Network Post-Processing

This is a continuation of U.S. Patent Application No. 19/044,857, filed on February 4, 2025, which claims priority to Republic of Korea Patent Application No. 10-2024-0100508, filed on July 29, 2024, and Republic of Korea Patent Application No. 10-2024-0067309, filed on May 23, 2024, which are incorporated by reference in their entirety.

Humans have the intelligence to recognize, classify, infer, predict, control/decision making, and the like. Artificial intelligence (AI) is the artificial imitation of human intelligence.

The human brain is made up of tons of nerve cells called neurons. Each neuron is connected to hundreds to thousands of other neurons through connections called synapses. In order to mimic human intelligence, the operation of biological neurons and the connections between neurons are modeled in a neural network (NN) model. In other words, a neural network is a system of nodes connected in a layer structure that mimics neurons.

Embodiments relate to a post-processing circuit that is separate from a neural processing unit (NPU). The post-processing circuit includes an access circuit, a processing circuit and memory. The access circuit enables the post-processing circuit to communicate with the NPU via a bus. The processing circuit filters a plurality of bounding boxes received from the NPU and selectively outputs a subset of the bounding boxes that satisfy a predetermined condition as data output. The memory stores the data output from the processing unit.

In one or more embodiments, the data output includes for each of the bounding boxes in a region of an image and class scores indicative of probability that classes of objects being present in each bounding box.

In one or more embodiments, the post-processing circuit further includes a first computation circuit configured to select one or more classes for each bounding box as the subset of the data output by comparing class scores of classes for each bounding box.

In one or more embodiments, the post-processing circuit further includes a second computation circuit configured to extract one or more bounding boxes by comparing a class confidence score of each bounding box with a threshold confidence score. The class confidence score represents probability that an object of a class is present in each bounding box and derived from an object presence confidence score and the class scores. The object presence confidence score is included in the data output and is indicative of probability that an object is present in each bounding box.

In one or more embodiments, the second computation circuit is configured to determine the class confidence score as a product of the object presence confidence score and a class score with the subset of classes extracted by the first computation circuit.

In one or more embodiments, the memory further stores the subset of classes for each bounding box extracted by the first computation circuit.

In one or more embodiments, the memory includes a plurality of memory registers, and an address generation logic for accessing the plurality of memory registers.

In one or more embodiments, the processing circuit is configured to perform a non-maximum suppression (NMS) operation on the one or more bounding boxes extracted by the second computation circuit to remove redundant or overlapping bounding boxes of the plurality of bounding boxes.

Embodiments also relate to a system including a bus, a neural processing unit (NPU) and a post-processing circuit. The NPU is coupled to the bus and configured to perform at least multiply and accumulate operations on an input data to generate a plurality of bounding boxes. The post-processing circuit is coupled to the bus and includes an access circuit configured to communicate with the NPU via the bus to receive the plurality of bounding boxes, a processing circuit filters the plurality of bounding boxes and selectively outputs a subset of the bounding boxes that satisfy a predetermined condition as data output. The post-processing circuit includes memory that stores the data output from the processing unit.

Embodiments also relate to a method for performing operations associated with a neural network model. A neural processing unit (NPU) is coupled to the bus and performs at least multiply and accumulate operations on an input data to generate a plurality of bounding boxes. The plurality of bounding boxes are sent from the NPU to a post-processing circuit via a bus. The plurality of bounding boxes are filtered and a subset of the bounding boxes that satisfy a predetermined condition is selectively output as data output by the post-processing circuit. The data output is stored in an internal memory of the post-processing circuit.

Particular structural or step-by-step descriptions for examples according to the concept of the present disclosure disclosed in the present specification or application are merely exemplified for the purpose of explaining the examples according to the concept of the present disclosure.

Examples according to the concept of the present disclosure may be embodied in various forms. Examples according to the concept of the present should not be construed as being limited to the examples described in the present specification or application.

Examples according to the concept of the present disclosure may apply various changes. The present disclosure may take many forms. Accordingly, specific examples are illustrated in the drawings and described in detail in the present disclosure. However, this is not intended to limit the examples according to the concepts of the present disclosure to a specific disclosure form. Therefore, it should be understood that all changes, equivalents or substitutes included in the spirit and scope of the present disclosure are included in the present disclosure.

Terms such as first and/or second may be used to describe various components. However, the present disclosure should not be limited by the above terms. These terms are only used for the purpose of distinguishing one component from another. For example, without departing from the scope of rights according to the concept of the present disclosure, a first element may be termed a second element, and similarly, a second element may also be termed a first element.

When an element is referred to as being “connected to” or “in contact with” another element, it is understood that the other element may be directly connected to or in contact with the other element, but other elements may be disposed therebetween. On the other hand, when it is mentioned that a certain element is “directly connected” or “directly connected” to another element, it should be understood that no other element is present therebetween. Other expressions describing the relationship between elements, such as “between” and “immediately between” or “adjacent to” and “directly adjacent to”, etc., should be interpreted similarly.

3 In the present disclosure, expressions such as “A or B”, “at least one of A or/and B” or “one or more of A or/and B” may include all possible combinations thereof. For example, “A or B”, “at least one of A and B” or “at least one of A or B” may refer to both (1) including at least one A, (2) including at least one B, or () including both at least one A and at least one B.

As used herein, expressions such as “first”, “second”, “first or second” may modify various elements, regardless of order and/or importance. Said expressions are used only to distinguish one element from other elements, and does not limit the elements. For example, the first user apparatus And the second user device may represent different user device regardless of order or importance. For example, without departing from the scope of rights described in this disclosure, the first element may be named as the second element, and similarly, the second element may also be renamed as the first element.

Terms used in present disclosure are only used to describe specific examples, and may not be intended to limit the scope of other examples. The singular expression may include the plural expression unless the context clearly dictates otherwise. Terms used herein, including technical or scientific terms, may have the same meanings as commonly understood by one of ordinary skill in the art described in this document.

Among terms used in present disclosure, terms defined in a general dictionary may be interpreted as having the same or similar meaning as the meaning in the context of the related art. Unless explicitly defined in this document, it should not be construed in an ideal or overly formal sense. In some cases, even terms defined in the present disclosure cannot be construed to exclude examples of the present disclosure.

The terms used herein are used only to describe specific examples, and are not intended to limit the present disclosure. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as “comprise” or “having” are intended to indicate that the described feature, number, step, operation, component, part, or combination thereof is present. Accordingly, it should be understood that the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art. Unless explicitly defined in this disclosure, it is not to be construed in an ideal or overly formal sense.

Each feature of the various examples of the present disclosure may be partially or wholly combined or combined with each other. Various examples of the present disclosure are technically capable of various interlocking and driving as can be fully understood by those skilled in the art. Each of the examples of the present disclosure may be implemented independently of each other or may be implemented together in an association relationship.

In describing the examples, descriptions of technical contents that are well known in the technical field to which the present disclosure pertains and are not directly related to the present disclosure may be omitted. This is to more clearly convey the gist of the present disclosure without obscuring the gist of the present disclosure by omitting unnecessary description.

To facilitate understanding of the present disclosure, the following is a brief summary of terms used herein.

NPU: Abbreviation for neural processing unit, which may refer to a processor specialized for computing a neural network model independent of a CPU.

NN: Abbreviation for neural network, a network of nodes connected in a layer structure, mimicking the way neurons in the human brain are connected through synapses, to mimic human intelligence.

Information of a neural network: The information may include the structure of the network, information about the number of layers, information about the connection relationship of each layer, information about the parameters of each layer, information about the computational processing method, information about the activation function, the data type of the parameters of each layer (e.g., floating-point or integer), and the bitwidth of each parameter.

DNN: Abbreviation for deep neural network, which can refer to an increase in the number of hidden layers of a neural network to achieve higher artificial intelligence.

CNN: Abbreviation for convolutional neural network, a neural network that functions similarly to the visual cortex of the human brain in processing images. Convolutional neural networks are known to be well-suited for image processing and are known for their ability to extract features from input data and identify patterns in the features.

Transformer: The transformer neural network is a DNN based on attention techniques. It utilizes many matrix multiplication operations. A transformer can take an input value and parameters such as query (Q), key (K), and value (V) to obtain an output value, attentions (Q,K,V). Based on the output value (i.e., the attentions (Q,K,V)), the transformer can process various inference operations.

Kernel: Refers to the weights of the N x M matrix of convolutions. Each layer of the neural network model has a plurality of kernels, and the number of kernels may be referred to as the number of channels, the number of filters, and the like.

RTL code: The design of circuits for specific devices or systems is typically performed at the register transfer level (RTL). At this stage, the circuit is described in RTL code and subjected to functional verification through simulation. Once the RTL simulation is complete, the circuit design is converted into a gate-level representation through logic synthesis, followed by further verification at the gate level via simulation. A circuit for a neural processing unit, a circuit for a post-processing unit, or a system on chip (SoC) described below with reference to the drawings can be implemented using RTL code. RTL code represents the design of a digital circuit at a high level of abstraction, describing its operation and data flow based on clock cycles. RTL code is typically written in hardware description languages (HDLs), such as Verilog or VHDL. RTL code defines a digital circuit in terms of data transfer between registers and the logical operations performed on the data. It is widely used for simulation, synthesis, and verification during the design and development of integrated circuits (ICs). Synthesis refers to the process of converting RTL code into gate-level designs suitable for physical implementation on silicon. Key elements of RTL code include registers, combinational logic circuits, and state machines. Registers, defined by flip-flops or latches, serve as storage elements for holding data. Combinational logic circuits perform data operations such as addition, subtraction, and logical functions like AND/OR. State machines manage the sequence of operations and control the order of data transfers. Together, these elements define the structure and functionality of a circuit at the RTL stage, facilitating efficient simulation, verification, and synthesis for hardware implementation. In one example of the present disclosure, the post-processing unit may be provided as RTL code written in a hardware description language (HDL), such as Verilog or VHDL, for the purpose of intellectual property licensing.

0 1 Neural Network (NN) models are categorized according to the number of layers into 'single-layer neural networks' and 'multi-layer neural networks'. A typical multi-layer neural network consists of an input layer, a hidden layer, and an output layer. (1) The input layer is the layer that receives external data, and the number of neurons in the input layer is the same as the number of input variables. (2) The hidden layer is located between the input and output layers and receives signals from the input layer, extracts features, and passes them to the output layer. (3) The output layer receives signals from the hidden layer and outputs them to the outside. The input signals between neurons are multiplied by their respective weights, which have a value betweenand, and then summed up. If this sum is greater than the threshold of the neuron, the neuron is activated and implemented as an output value through the activation function.

On the other hand, increasing the number of hidden layers in a neural network to achieve higher artificial intelligence is called a deep neural network (DNN). There are many types of DNNs, but convolutional neural networks (CNNs) are known to extract features from input data and identify patterns in the features. A convolutional neural network (CNN) is a network structure in which the operations between neurons in each layer are implemented as the convolution of a matrix of input signals and a matrix of weight kernels.

3 FIG. Convolutional neural networks are neural networks that function similarly to the visual cortex of the human brain, which processes images. Convolutional neural networks are known to be suitable for image classification, object detection and the like. Convolutional neural networks are composed of convolutional operations, activation function operations, and pooling operations processed in a specific order (e.g.,). In a convolutional neural network, convolutional operations take up most of the computation time. Convolutional neural networks use a kernel in the form of a matrix to extract the features of each channel's image, and pooling to provide homeostasis against movement or distortion. In each channel, a feature map is obtained by the convolution of the input data and the kernel, and an activation function is applied to generate an activation map for that channel. Pooling can then be applied. The layer that finally classifies the pattern is located at the end of the convolutional neural network and can be exemplified by the fully connected layer. In the computational processing of convolutional neural networks, most of the operations are performed via convolutional or matrix multiplication.

However, for the efficiency and accuracy of neural network model operations related to image classification and object detection, post-processing operations such as additional filtering and deduplication of the output parameters (e.g., feature maps) may be performed.

Conventionally, in this case, the post-processing operations described above may be performed on a central processing unit (CPU) external to the neural processing unit and the data subsequently processed by the CPU may be stored in a memory external to the neural processing unit.

The memory external to the neural processing unit comprises a plurality of memory cells, each of which has a unique memory address. Whenever the neural processing unit recalls feature maps or weights stored in the main memory, or recalls other parameters, a latency of several clocks may be consumed to access the main memory cell corresponding to the address in the memory. These latencies may include column address strobe (CAS) latency and row address strobe (RAS) latency. Hence, the time and power consumed to read the necessary data and parameters (e.g., weights, feature maps, or kernels) from a memory external to the neural processing unit to the neural processing unit is significant.

The time and power for retrieving necessary data—such as parameters, including weights, feature maps, or kernels—from a memory external to the neural processing unit (NPU) to the NPU is considerable.

According to one example of the present disclosure, a post-processing unit may operate in conjunction with a NPU to efficiently perform operations, such as filtering and deduplication, on output parameters (e.g., feature maps), thereby enhance the efficiency and accuracy of neural network model computations. The post-processing unit may perform these operations without relying on a central processing unit (CPU) coupled to the NPU via a bus. Further, the post-processing unit may process computation data by, for example, filtering and deduplication results, without accessing main memory external to the NPU.

1 FIG.A 600 1000 600 1000 1000 600 2000 3000 4000 5000 600 1000 2000 3000 4000 5000 is an example diagram illustrating a system where a PPUA is separate from NPUA, in accordance with an example of the present disclosure. The system includes PPUA and NPUA, as well as multiple peripheral circuits. At least some of the circuit(s) of the system may be formed as circuit(s) in a system-on-chip (SoC). The NPUA may perform various neural network inference functions in communication with the PPUA, a CPU, a main memory, an image sensor, and a decoder. Each of the PPUA, NPUA, CPU, main memory, image sensor, or decodermay be formed as an independent circuit but is not limited thereto.

600 2000 600 2000 1000 600 1000 2000 600 1000 2000 3000 600 1000 2000 600 1000 2000 3000 3000 The PPUA may comprise circuitry formed on the same semiconductor die as the CPU. The PPUA and the CPUmay comprise circuitry integrated on the same semiconductor die. In one embodiment, the PPU 600A and the NPUA may be circuits integrated on the same semiconductor die. In another embodiment, the PPUA, the NPUA, and the CPUmay be circuits integrated on the same semiconductor die. In yet another embodiment, the PPUA, the NPUA, the CPU, and the main memorymay be circuits formed on the same semiconductor die. Additionally, the PPUA may comprise a semiconductor die interconnected with at least one of the NPUA or the CPU, each comprising a semiconductor die, utilizing chiplet technology. Additionally, the PPUA may comprise a semiconductor die interconnected with at least one of the NPUA, the CPU, and the main memory, each comprising a semiconductor die, utilizing chiplet technology. When chiplet technology is applied, it may further include an interposer. Alternatively, the main memorymay comprise a plurality of semiconductor dies utilizing chiplet technology.

1 FIG.A 1 FIG.B Each of the elements described above may be categorized by its operation function, and each of the elements may be implemented as a circuit board, a silicon substrate, a resistor element, a transistor, and the like. Thus, each element may be a semiconductor circuit with numerous transistors connected thereto, some of which may be difficult to identify and distinguish with the naked eye, and may be identified only by operation. Accordingly, each of the elements ofandmay be referred to as a circuit unit.

600 2000 3000 4000 5000 6000 1000 6000 1000 Each of the PPUA, CPU, main memory, image sensor, and decoderdescribed above may communicate via a busto send data to and receive data from the NPUA. According to one example of the present disclosure, the busmay be embodied as an advanced extensible interface (AXI) bus. However, without limitation, the NPUA may also be configured to be directly coupled to at least one of the elements described above.

1000 1000 600 1 FIG.A The NPUA may be defined as a processor specialized for the operation of a neural network model. In particular, the NPUA may be specialized for matrix operations or convolutional operations, which account for the majority of the computation in the neural network model. Although only a single NPU is illustrated in, two or more NPUs that operate in conjunction with PPUA may be provided in the system.

1000 100 200 300 400 500 600 1000 1000 The NPUA may include a controller, a direct memory access (DMA), a memory, a plurality of processing elements (PEs), a special function unit (SFU), and PPU. The elements of the NPUmay be distinguished by their operation functions, and each element may be formed utilizing circuit elements such as a resistor and a transistor. Thus, each element may be a semiconductor circuit with numerous connected transistors. Some of the aforementioned elements may be difficult to identify and distinguish with the naked eye and may be identified only by its operation. Accordingly, functional units of the NPUA may be referred to as circuits.

100 200 300 400 500 100 200 300 400 500 100 300 300 100 1000 320 1000 100 1000 The controllermay control operations associated with computing the neural network model performed by each of the DMA, the memory, the PEs, and a special function unit (SFU). The controllermay be directly coupled or indirectly coupled to each of the DMA, memory, a plurality of PEs, SFU, to enable communication between the components. For example, the controllermay allocate the capacity of each of the parameters in the memorybased on the capacity of the memory. The controllermay be configured to control the NPUA based on machine code (e.g., binary code) of a compiled neural network model. For example, a compilermay generate machine code that determines, based on hardware characteristics of the NPUA (e.g., number of PEs, amount of memory, functions provided by a special function unit, presence of a post-processing unit, and the like), a read/write sequence of data of the neural network model, a processing sequence of layers of the neural network, an operation sequence of convolutional multiplication, an operation sequence of matrix multiplication, and a read and write operation sequence of data of the DMA. Accordingly, the controllermay control the NPUA based on the machine code.

100 1000 3010 3000 2000 3010 1000 300 100 300 3010 1000 3010 The controllermay obtain schedule information that schedules an order of operations of the neural network model to be performed by the NPUbased on the directed acyclic graph (DAG) of the neural network model compiled by compilerstored in main memoryand executed by CPU. The compilermay determine an operation schedule that can accelerate the operation of the neural network model by determining the number of PEs of the NPU, the size of the memory, the size of the parameters of each layer of the neural network model, and the like. According to the computation schedule, the controllermay be configured to control the required number of PEs for each computation step and to control the read and write operations of the required parameters in the memoryfor each computation step. The compilermay efficiently schedule the operations according to the information of the hardware structure and performance of the NPUA. The compilermay determine data locality based on the order of the layers of the neural network, the order of the operations of the unit convolution and/or matrix multiplication and generate compiled machine code based on the order of the data required to compute the neural network model.

1000 320 320 3010 3010 320 1000 In some examples, the NPUA includes an embedded compiler. The embedded compilermay perform some operations in addition to or in place of external compiler. According to the configurations described above, compilerand/or compilerof the NPUA may generate machine code upon input of files in the form of various AI software frameworks. For example, AI software frameworks may include TensorFlow, PyTorch, Keras, XGBoost, mxnet, DARKNET, ONNX, and the like.

200 1000 3000 600 1000 3000 200 3000 600 1000 6000 The DMAmay allow the NPUA to directly access, read and/or write to the main memoryA and/or PPUA. The NPUmay read various data associated with the neural network model from the main memoryvia the DMA. The main memorymay be embedded in a system-on-chip (SoC) or may be configured as a separate memory device. Data for post-processing may also be sent to PPUA from NPUA via bus.

300 1000A and 300 300 3000 300 300 1000 3000 300 400 The memorymay be located in an on-chip region of the NPUmay perform caching or storing of data processed in the on-chip region. The memorymay also be referred to as cache memory. The memorymay read and store at least some of the data associated with performing operations associated with the neural network model from the main memory. The memorymay be configured to store all or part of the neural network model according to the memory capacity settings for each parameter and the data size for each layer of the neural network model. Representative parameters of data processed in the neural network model may include, among other data, attention parameters, KV cache, an activation map, an input feature map, an output feature map, and weights. Specifically, the memoryof NPUA may read and store parameters corresponding to input data from the main memory. Additionally, the memorymay read and store parameters corresponding to output data from the plurality of PEs.

300 300 300 The memorymay be embodied as one or more of read-only memory (ROM), static random-access memory (SRAM), dynamic random-access memory (DRAM), Resistive random-access memory (RAM), Magneto-resistive RAM, Phase-change RAM, Ferroelectric RAM, Flash Memory, high-bandwidth memory (HBM), and the like. According to one example of the present disclosure, the memorymay be embodied as SRAM, which is advantageous in terms of computational processing speed. Further, the memorymay be organized into at least one memory unit (e.g., a bank). The memory 300 may comprise homogeneous memory or heterogeneous memory.

300 300 300 300 300 The data stored in the memory unit of the memoryis not static and may change dynamically. By varying the memory allocation of the memoryto different types of parameters and data, the utilization rate of the memorymay be increased. In other words, the data stored in the memory banks of the memory(e.g., parameters of the neural network model) may be changed to any of the following as needed: attention parameters, KV cache parameters, activation map parameters, input feature map parameters, weights parameters, and output feature map parameters. Further, the size of data for each type of parameter stored by the memorymay vary for each computational step.

400 400 400 The PEsare hardware circuits that perform multiplication and accumulation (MAC) operations. The plurality of PEsmay be configured to receive, as input data, an input feature map and/or a kernel corresponding to a layer of the neural network, part of the layer of the neural network or multiple layers of the neural network. A processing element (PE) in the plurality of PEsmay be configured to perform functions such as addition, multiplication, accumulation, and the like to perform operations as defined by the neural network model. The PE may include, among other components, a multiply and accumulate (MAC) operator, and an arithmetic logic unit (ALU) operator.

400 In one example, a PE may take an input feature map or a part of the input feature map, perform convolution operations using the kernel and output an output feature map or a part of the output feature map. The plurality of PEsor the processing element PE may also be referred to as an artificial intelligence (AI) computing unit. In another example, the PE may perform a general matrix multiply (GEMM) operation or a matrix multiply operation on the input feature map using weights to output an output feature map or a part of the output feature map. More specifically, the PE may multiply the input feature map in the form of a matrix with a weight matrix, and then add a bias to the matrix to output an output feature map or a part of the output feature map in the form of a matrix. In the NPU, the matrix multiplication may be performed at a high speed by parallel processing, thereby enabling efficient processing of the matrix multiplication operation.

300 The PE may include circuitry designed to handle only integer type parameters as input. In such a case, the input parameters of the PE may be converted to integers of a specific bitwidth and be stored in the memory. Such PE may reduce the power consumption compared to PEs that support floating point and may be easier to implement as an on-device component.

500 500 1000 1000 The SFUmay process a number of activation functions for imparting nonlinearity to the output feature map. The activation functions processed by the SFUmay include, but are not limited to, SiLU functions, SoftMax functions, sigmoid functions, hyperbolic tangent (tanh) functions, ReLU functions, Leaky ReLU functions, Mahout functions, or ELU functions that result in nonlinear outputs with respect to inputs. It may be technically difficult to support all activation functions in the NPU. Therefore, the NPUmay also approximate various activation functions through a piecewise linear function approximation algorithm and piecewise linear function processing circuitry. These activation functions can be optionally applied after the MAC operation. The result of an operation applied with an activation function is referred to as an activation map.

500 500 500 500 500 500 500 In some embodiments, the SFUmay be configured to include a floating-point multiplier circuit for performing decimal point operations. In other embodiments, the SFUmay be configured to communicate with the PEs and may include circuitry designed to receive integer-type parameters from the PEs. In such a case, the SFUmay be further configured to include an inverse quantizer circuit configured to convert the integer-type parameter to a floating point-type parameter. The SFUmay be configured to process an activation function operation with the floating-point type parameters. Further, the SFUmay be configured to further include a quantization circuit configured to convert a parameter of a floating-point type at the end of the activation function operation to a parameter of an integer type. According to the above-described configuration, the SFUmay be configured to process the floating-point operation by de-quantizing the integer parameter when a floating-point operation is required, and to re-quantize the result. In other words, a NPU according to one example of the present disclosure may include a processing element circuit configured to process an integer type parameter and a special function circuit unit pipelined thereto, where the special function circuit unit includes a quantization circuit and a de-quantization circuit and may be configured to process an activation function operation with a floating-point type parameter. According to the above-described configuration, the SFUeffectively communicates with the PEs that support only integer parameters, and may directly convert and process the integer parameters without support from circuitry outside the NPU.

600 In some examples, the post-processing unit (PPU)may be configured to process a number of activation functions for imparting nonlinearity to the output feature map.

1 FIG.B 1 FIG.B 1 FIG.A 1 FIG.A 1 FIG.B 1 FIG.B 600 1000 600 1000 300 400 100 100 600 600 1000 600 1000 2000 1000 1000 1000 is an example diagram illustrating a system where PPUB is integrated within NPUA, in accordance with an example of the present disclosure. The system ofis different from the system ofin that PPUB is integrated into NPUB and communicates with the memoryand the PEsvia internal bus in NPUB. Further, controllerfurther controls various operations associated with PPUB since PPUB is located within NPUB whereas PPUA ofis located outside the NPUA and hence is controlled by other devices in the system (e.g., CPU). Otherwise, components illustrated inoperate in substantially in the same manner as those of, and their explanations are obviated herein for the sake of brevity. NPUA and NPUB are hereinafter collectively referred to as NPU.

2 FIG. 2 FIG. 2 FIG. 641 642 643 644 is a schematic diagram illustrating the PE according to one embodiment. Referring to, the PE may include, among other components, a multiplier, an adder, an accumulator, and a bit quantization unit. Various modifications may be made to PE ofto account for the computational characteristics of a target neural network model.

641 641 The multiplieris a circuit that multiplies the input (N)-bit data and the (M)-bit data. The result of the multiplieris output as (N+M)-bit data, where N and M are integers greater than zero. The first input that receives the (N)-bit data that changes dynamically whereas the second input that receives the (M)-bit data receive a parameter that remains relatively constant. For example, a set of weight parameters trained in a neural network model may be constant while the PE is processing a same layer of the neural network while input parameters such as activation parameters, feature map parameters, attention parameters, KV cache parameter computed with the set of weight parameters may vary frequently relative to the set of weight parameters.

A parameter that is variable means that the parameter is updated each time incoming input data to the neural network is updated. For example, the node data of each layer may be a MAC operation value reflecting the weight data of a neural network model where the node data of each layer in the neural network changes whenever the input video changes every frame. A parameter that is static means that the parameter is maintained regardless of the update of the incoming input data. For example, the weight data may remain constant if the neural network model is used to infer object detection of the video data.

The variable parameters fed to the first input may be node data of a layer of the neural network model. The node data of the neural network model may be one of input data of the input layer, accumulated values of the hidden layer, and accumulated values of the output layer. The constant parameters fed to the second input may be weight data of a connection network of the neural network model.

100 100 300 The controllermay improve memory reuse by taking into account the nature of the constant parameters. The variable parameters are computational values of each layer, and the controllermay identify reusable variable parameters based on the machine code of the compiled neural network model, and control the memoryto reuse the memory.

100 300 300 300 3000 3000 1000 3000 100 100 300 4 FIG. The constant parameters are the weight data of each connection network, and the controllermay recognize the constant parameters of the repeatedly used connection networks based on the structure data of the neural network model or the neural network data locality information and may control the memoryto reuse the parameters stored in the memory. Reusing the parameters means that the parameters stored in the memoryare not deleted, or copied or moved to the main memory, but are reused in the subsequent operation. According to the above-described configuration, it beneficially reduces power consumption according to the operation of the main memoryas shown in. Furthermore, it has the effect of eliminating the delay time that occurs when the NPUtransmits data to and from the main memory. The controllermay have information on reusable variable parameters and reusable constant parameters based on the machine code of the compiled neural network model. Accordingly, the controllermay be configured to control the memoryto reuse the parameters stored in the memory.

641 641 641 641 641 The processing element may constrain the operation of the multipliersuch that when a zero is input at an input of one of the first input and the second input of the multiplier, the multipliermay not perform an operation because the processing element knows that the result of the operation will be zero even if the operation is not performed. For example, when a zero is provided to an input of one of the first input and the second input of the multiplier, the multipliermay be configured to operate in a zero-skipping manner.

400 100 641 641 4 FIG. For zero-skipping, each of the PEs included in the plurality of PEsmay be enabled or disabled, respectively. The controllermay provide an enable or disable signal to each of the PEs on a clock-by-clock basis to each of the PEs. When the PE is disabled, the multipliermay be configured to deactivate according to a level of the first enable signal En1. Accordingly, the power consumed by the operation of the multipliermay be reduced. For example, information about the power consumption of the multiplier may be provided with reference to.

400 642 100 4 FIG. For zero skipping, each of the PEs included in the plurality of PEsmay be enabled or disabled, respectively. The controller 100 may provide an enable or disable signal to each of the PEs on a clock-by-clock basis to each of the PEs. When the PE is deactivated, the adder 642 may be configured to deactivate according to a level of the second enable signal En2. Accordingly, the power consumed by the operation of the addermay be reduced. For example, information on power consumption of the adder may be provided with reference to. In some examples, each PE may be designed to receive a respective control signal from the controllerfor controlling (i.e., enabling or disabling) the zero-skipping operation.

641 100 In some examples, each multiplierof each PE may receive a respective control signal from the controllerfor controlling zero-skipping operations. According to the configurations described above, the power consumption of the multipliers can be reduced by zero-skipping.

642 100 In some examples, each adderof each PE can be designed to receive a respective control signal from the controllerfor controlling the zero-skipping operation. According to the configurations described above, the power consumption of the adders can be reduced by zero-skipping.

641 642 100 In some examples, each of the multiplierand adderof each PE may be designed to simultaneously receive a respective control signal for controlling the zero-skipping operation from the controller. According to the configurations described above, the power consumption of the multipliers and adders may be reduced by zero-skipping.

In some examples, the weights are constant parameters generated by training, and the machine code from which the neural network model comprising the weights has been compiled may be programmed to input respective control signals for controlling the zero-skipping operation to each of the PEs at which values with zero weights are input.

The number of bits of data input to the first input and the second input may be determined according to the quantization of the node data and the weight data of the respective layers of the neural network model. For example, the node data of the first layer may be quantized to 5 bits and the weight data of the first layer may be quantized to 7 bits. In such a case, the first input may be configured to receive 5-bit data and the second input may be configured to receive 7-bit data, i.e., the number of bits of data input to each input may be different.

The PE may receive quantization information of the data input to each input. The neural network data locality information may include quantization information of the input data and output data of the PE.

1000 300 1000 The NPUmay control when the quantized data stored in the memoryis fed to the inputs of the PEs, the quantized bitwidth are converted in real time. That is, different layers may have different quantized bitwidth, and the PEs may be configured to generate input data by converting the bitwidth in real time by receiving bitwidth information from the NPUin real time as the bitwidth of the incoming data is converted.

643 642 641 643 643 643 643 643 643 The accumulatoruses the adderfor a number of (L) loops to accumulate the operation value of the multiplierand the operation value of the accumulator. Thus, the number of bits of data at the output and input of the accumulatormay be output as (N+M+log2(L))bits, where L is an integer greater than zero. When the accumulatorfinishes accumulating, the accumulatormay receive an initialization reset signal to initialize the data stored inside the accumulatorto zero. However, examples according to the present disclosure are not limited thereto. The accumulatorstores the accumulated value even when zero-skipping is enabled in the corresponding PE. Thus, subsequent values can be accumulated even when zero skipping is enabled.

644 643 644 100 1000 The bit quantization unitmay reduce the bitwidth in the data output from the accumulator. The bit quantization unitmay be controlled by the controller. The bitwidth of quantized data may be output as (X) bits, where X is an integer greater than zero. According to the above-described configuration, the plurality of PEs is configured to perform a MAC operation, and the Plurality of PEs may quantize and output the MAC operation result. Such quantization may further reduce power consumption as the number of (L)loops increases. Reducing power consumption may also reduce heat generation in edge devices. Reducing heat generation may beneficially reduce the possibility of malfunctioning operations caused by a high temperature of the NPU.

644 644 644 The output data X bits of the bit quantization unitcan be the node data of the subsequent layer or the input data of a convolutional product. If the neural network model is quantized, the bit quantization unitmay receive quantized information from the neural network model. However, the controller 100 may also analyze the neural network model to extract the quantized information. Thus, the output data X bits may be converted to a number of quantized bits to correspond to the quantized data size. The output data X bits of the bit quantization unitmay be stored in the memory 300 as a quantized bitwidth.

1000 641 642 643 644 644 643 100 644 The plurality of PEs of the NPUaccording to one example of the present disclosure includes a multiplier, an adder, an accumulator, and a bit quantization unit. The bit quantization unitmay reduce the number of bits of data in (N+M+log2(L))bits output from the accumulatorby the plurality of PEs to a number of bits in (X) bits. The controllermay control the bit quantization unitto reduce the number of bits in the output data by a predetermined number of bits from the least significant bit (LSB) to the most significant bit (MSB). Reducing the number of bits in the output data may beneficially reduce power consumption, computation, and memory usage. However, if the number of bits is reduced below a certain length, the inference accuracy of the neural network model may decrease rapidly. Therefore, the quantization level (i.e., the reduction of the number of bits in the output data) can be determined by comparing the degree of reduction in power consumption, computation, and memory usage with the degree of reduction in the inference accuracy of the neural network model. The quantization level can also be determined by setting a target inference accuracy for the neural network model and testing the accuracy of the inference with progressively lower bitwidth. The quantization level can be determined separately for each layer of the neural network model.

641 644 By adjusting the number of bits of (N) bit data and (M) bit data of the multiplier, and by reducing the number of bits of the operation value (X) bit by the bit quantization unit, the plurality of PEs may improve the MAC operation speed while reducing the power consumption, and also has the advantage of, among others, making the convolution operation of the neural network model more efficient.

3 FIG. is a schematic diagram illustrating a convolutional neural network relevant to the present disclosure. A convolutional neural network can be a combination of one or several convolutional layers, a pooling layer, and a fully connected layer. Convolutional neural networks have a structure suitable for learning and inference from two-dimensional data and can be trained using a backpropagation algorithm.

In examples of the present disclosure, the convolutional neural network has a kernel for each channel that extracts features of the input image for the channel. The kernel may be organized as a two-dimensional matrix and performs convolutional operations, as it traverses the input data. The size of the kernel can be arbitrary, and the stride at which the kernel traverses the input data can also be arbitrary. The result of the convolution over the entire input data per kernel may be referred to as a feature map or activation map.

In the following, a kernel may include a single set of weights or multiple sets of weights. The number of kernels for each layer may be referred to as the number of channels.

Since a convolutional operation is a combination of input data and kernels, an activation function may then be applied to add nonlinearity. When an activation function is applied to a feature map that is the result of a convolutional operation, it may be referred to as an activation map.

3 FIG. 64 128 256 Specifically, referring to, a convolutional neural network may include at least one convolutional layer, at least one pooling layer, and at least one fully connected layer. For example, convolution can be defined by two main parameters: the size of the input data (typically a 1×1, 3×3, or 5×5 matrix) and the depth of the output feature map (the number of kernels). These key parameters can be computed by convolution. These convolutions may start at depth 32, continue to depth, and end at depthorThe convolution operation may mean an operation of sliding a kernel of size 3×3 or 5×5 over an input image matrix that is input data, multiplying each weight of the kernel and each element of the input image matrix that overlaps, and then adding them all.

An activation function may be applied to the output feature map generated in this way to finally output an activation map. In addition, the weight used in the current layer may be transmitted to the subsequent layer through convolution. The pooling layer may perform a pooling operation to reduce the size of the feature map by down-sampling the output data (i.e., the activation map). For example, the pooling operation may include, but is not limited to, max pooling and/or average pooling.

The maximum pooling operation uses the kernel, and outputs the maximum value in the area of the feature map overlapping the kernel by sliding the feature map and the kernel. The average pooling operation outputs an average value within the area of the feature map overlapping the kernel by sliding the feature map and the kernel. As such, since the size of the feature map is reduced by the pooling operation, the number of weights of the feature map is also reduced.

The fully connected layer may classify data output through the pooling layer into a plurality of classes (i.e., inferenced values), and output the classified class and a score thereof. Data output through the pooling layer forms a three-dimensional feature map, and this three-dimensional feature map may be converted into a one-dimensional vector and input as a fully connected layer.

1 FIG.A 1 FIG.B 1000 400 1000 400 Referring toand, a neural network model processed by the NPUmay be related to image classification and object detection. The input data of the plurality of PEsof the NPUthat processes the neural network model described above may be image data, and the output data of the plurality of PEsmay be a plurality of bounding box data for the input image. Each of the plurality of bounding box data may include bounding box coordinate data and class data. The bounding box coordinate data may include height data, width data, x data, and y data.

Assuming that the shape of the bounding box is a rectangle, the bounding box coordinate data includes height data, width data, x data, and y data as described above. However, the shape of the bounding box is not limited to a square, but may be transformed into a pentagon or more polygon or a circle, and accordingly, the number and type of bounding box coordinate data may vary according to the shape of the bounding box. Further, the class data may include a plurality of classes categorized as existing inside the bounding box and a score thereof.

4 FIG. 4 FIG. 4 FIG. 300 1000 1000 is a schematic diagram describing energy consumption per unit operation of a NPU, according to one example of the present disclosure. Hereinafter,will be described with respect to the power consumption reduction technology of the memoryof the NPU. Referring to, this is a table schematically explaining energy consumed per unit operation of the NPU. Energy consumption can be divided into memory access, addition operation, and multiplication operation.

8 642 16 642 642 642 642 641 641 641 641 300 300 3000 3000 300 640 b “Add” refers to an 8-bit integer addition operation of the adder. An 8-bit integer addition operation may consume 0.03 pj of energy. “b Add” refers to the 16-bit integer addition operation of the adder. A 16-bit integer addition operation may consume 0.05 pj of energy. “32b Add” refers to a 32-bit integer addition operation of the adder. A 32-bit integer addition operation may consume 0.1 pj of energy. “16b FP Add” refers to a 16-bit floating-point addition operation of the adder. A 16-bit floating-point addition operation may consume 0.4 pj of energy. “32b FP Add” refers to a 32-bit floating-point addition operation of the adder. A 32-bit floating-point addition operation may consume 0.9 pj of energy. “8b Mult” refers to an 8-bit integer multiplication operation of the multiplier. An 8-bit integer multiplication operation may consume 0.2 pj of energy. “32b Mult” refers to a 32-bit integer multiplication operation of the multiplier. A 32-bit integer multiplication operation may consume 3.1 pj of energy. “16b FP Mult” refers to a 16-bit floating-point multiplication operation of the multiplier. A 16-bit floating-point multiplication operation may consume 1.1 pj of energy. “32b FP Mult” refers to a 32-bit floating-point multiplication operation of the multiplier. A 32-bit floating-point multiplication operation may consume 3.7 pj of energy. “32b SRAM Read" refers to a read access of 32 bits of data when the memoryis a static random-access memory (SRAM). Reading 32 bits of data from the memorymay consume 5 pj of energy."32b DRAM Read" refers to a read access of 32 bits of data when the main memoryis DRAM. Reading 32 bits of data from main memoryto memorycan consumepj of energy. The unit of energy is the pico-joule (pj).

1000 3000 300 128 When the NPUperforms 32-bit floating point multiplication versus 8-bit integer multiplication, the energy consumption per unit operation is approximately 18.5 times different. When reading 32-bit data from the main memoryconfigured as DRAM and reading 32-bit data from the memoryconfigured as SRAM, the energy consumption per unit operation is approximatelytimes different. In other words, from a power consumption perspective, power consumption increases as the number of bits in the data increases. Also, floating-point operations consume more power than integer operations. Also, reading data from DRAM increases power consumption dramatically.

300 1000 300 300 3000 300 1000 3000 Therefore, the memoryof the NPUmay be embodied as high-speed static memory such as an SRAM tube and not DRAM. However, the neural network processing unit according to examples of the present disclosure is not limited to SRAM. For example, the memorymay not include DRAM, and the memorymay be configured to include static memory configured to have relatively higher read and write speeds and consume relatively less power than the main memory. Accordingly, the memoryof the NPUaccording to one example of the present disclosure may be configured to have a relatively higher read and write speed and to consume relatively less power for the inference operations of the neural network model than the main memory.

3000 Static memories that can be driven at high speeds, such as SRAM, may include SRAM, magnetoresistive random access memory (MRAM), spin-transfer torque magnetic random-access memory (STT-MRAM), embedded magnetic random access memory (eMRAM), and orthogonal spin transfer magnetic random access memory (OST-MRAM). Furthermore, MRAM, STT-MRAM, eMRAM, and OST-MRAM are static memories and have non-volatile characteristics. Thus, a static memory capable of high-speed operation, such as SRAM, may beneficially obviate redundancy of providing additional memory to the main memoryfor rebooting after a power failure. However, examples according to the present disclosure are not limited thereto.

1000 300 300 According to the above-described configuration, the NPUreduces the power consumption by the DRAM during the inference operation of the neural network model. Furthermore, a memory cell of the SRAM of the memorymay comprise, for example, four to six transistors to store one bit of data. However, examples according to the present disclosure are not limited thereto. Further, a memory cell of the MRAM of the memorymay comprise, for example, one magnetic tunnel junction (MTJ) and one transistor to store one bit of data. However, examples according to the present disclosure are not limited thereto.

600 1000 600 600 610 620 640 630 650 5 FIG. 5 FIG. The following describes in detail the specific configuration and operation of PPUA that is separate from NPUA, according to one example of the present disclosure.is a schematic diagram illustrating PPUA according to one example of the present disclosure. Referring to, the PPUA according to one example of the present disclosure may include, among other components, a first computation unit, a second computation unit, an internal processing unit, an internal memoryand a DMA.

610 610 The first computation unitmay extract the highest scoring class of the plurality of classes associated with the one bounding box. The first computation unitmay perform a class-argmax operation to extract the index of the class having the highest score in the one bounding box and the class score of that class. The class score indicates, for each class corresponding to an object, the probability that the object is present in the bounding box.

620 620 610 The second computation unitmay selectively extract only those bounding boxes of the plurality of bounding boxes having class confidence scores that are above a threshold confidence score. The class confidence score represents probability or confidence that a specific object of a certain class is present in the bounding box. The class confidence score is determined as a product of an object presence confidence score and the class score. The object presence confidence score indicates the probability that an object is present in a bounding box but regardless of what of the classes the object belongs to. The second computation unitperforms a bounding box filtering operation to extract only those bounding boxes whose product of the object presence confidence score of the bounding box and the class score extracted from the first computation unitis above a certain threshold confidence score.

640 620 640 640 640 The internal processing unitmay post-process the data of the extracted bounding boxes in the second computation unit; that is, the internal processing unitmay decode the data of the extracted bounding boxes. Further, the internal processing unitmay perform a non-maximum suppression (NMS) operation on the data of the extracted bounding boxes. The internal processing unitmay be referred to as the central processing unit (CPU) of the PPU.

630 600 630 610 620 640 630 The internal memorymay store data required for computations performed by the PPU. That is, the internal memorymay store data that is input or output from the first computation unit, the second computation unit, and the internal processing unit. The internal memorymay be referred to as the memory of the PPU.

650 600 6000 600 1000 3000 600 1000 3000 650 2000 The DMAis a circuit that enables PPUA to receive processed data or send data for processing via bus. The data to PPUA may originate from NPUA or main memory. The data processed by PPUA may be sent to NPUA or memoryThe DMAsupports such data to be transferred efficiently with reduced intervention or no intervention from CPU.

5 FIG. 630 1 2 640 1 2 610 620 640 Referring to, the internal memorymay include a plurality of memory banks (e.g., DATA, OUTPUT, OUTPUT, and Code). A portion (e.g., DATA) of the plurality of memory banks may store a plurality of bounding box data output from the internal processing unit. Another portion (e.g., OUPUT, OUTPUT) of the plurality of memory banks may store data on a plurality of bounding boxes received from the first computation unitand the second computation unit. The other portion (e.g., Code) of the plurality of memory banks may store code data associated with post-processing data in the internal processing unit. However, the data stored in the plurality of memory banks is not limited to the above, and various types of data may be stored as needed.

640 Meanwhile, the inputs and outputs of the internal processing unitmay be transmitted through an advanced high-performance bus (AHB). The AHB refers to a high-performance bus protocol used primarily in system-on-chip (SoC) designs, which has the advantages of low power and scalability, which can improve the reliability and efficiency of the system.

6 FIG. 1 5 FIGS.A and 1000 600 is a schematic diagram illustrating a computation process of a NPU and a PPU according to one example of the present disclosure. For ease of description, reference will be made to the structure of the NPUA and the PPUA shown in.

100 110 120 130 140 150 The computation process Saccording to one example of the present disclosure may include an activation function operation step S, a class-argmax operation step S, a filtering operation step S, a decoding operation step S, and a non-maximum suppression (NMS) operation step S.

110 500 1000 500 In the activation function operation step S, the SFUof NPUA may process a number of activation functions for imparting nonlinearity to the output feature map. The activation functions processed by the SFUmay include, but are not limited to, a SiLU function, a Softmax function, a sigmoid function, a hyperbolic tangent (tanh) function, a ReLU function, a Leaky ReLU function, a Maxout function, or an ELU function that results in a nonlinear output value with respect to an input value.

1000 1000 On the other hand, not all activation functions may be supported in the NPU. Therefore, the NPUmay be programmed to approximate various activation functions via a piecewise linear function approximation algorithm and piecewise linear function processing circuitry. These activation functions can be optionally applied after the MAC operation. The operational value to which the activation function is applied may be referred to as the activation map.

500 500 500 500 500 500 For example, the SFUmay be configured to include circuitry that supports computing of the first order function. For example, the SFUmay be configured to include circuitry that supports computing the first order function and the second order function. For example, the SFUmay be configured to include circuitry to support first order function, second order function, and logarithmic function. For example, the SFUmay be configured to include circuitry to support first order function, second order function, logarithmic function, and exponential function. For example, the SFUmay be configured to include at least one of a first order function circuit, a second order function circuit, a third order function circuit, a logarithmic function circuit, an exponential function circuit, or similar function calculation circuits. The SFUmay program a particular activation function in different techniques.

500 500 500 Alternatively, the SFUmay program a particular activation function only as a first order function. For example, the SFUmay program a particular activation function only as a second order function. In other embodiments, the SFUmay program the particular activation function only as a third order function, a logarithmic function or an exponential function.

500 500 The SFUmay be configured to analyze the nonlinearity of the activation function to segment the plurality of segments into different widths. The SFUmay also analyze the nonlinearity of the activation function and segment each of the plurality of segments to an optimal width. However, the present disclosure is not limited thereto.

In the present disclosure, the activation function may be implemented in various forms that include characteristic segments. When segmenting an activation function into a plurality of segments, the number and width of the plurality of segments may be varied according to the different forms taken by the activation function.

For example, various activation functions, such as SiLU function, Softmax function, swish function, Mish function, sigmoid function, hyperbolic tangent function, hyperbolic tanh function, SELU function, Gaussian Error Linear Unit (GELU) function, SOFTPLUS function, ReLU function, Leaky ReLU function, Maxout function, ELU function, and the like, have various shapes that are divided into a plurality of characteristic intervals that include substantially linear intervals and/or nonlinear intervals. Accordingly, when approximating a nonlinear activation function in a hardware-processable manner, segmenting it by considering these characteristic intervals can result in a more efficient or close approximation of the activation function corresponding to the characteristics of each activation function. For example, the number and width of the segments may be determined by considering the substantially linear intervals, nonlinear intervals, and the like,

Accordingly, in a method for approximating an activation function according to the present disclosure, the concept of segment data is used for segmenting the activation function in consideration of these characteristic intervals of the activation function. The segment data includes discontinuity information of the activation function, derivative data, information of the hardware on which the activation function is processed, and the like, and also includes data processed therefrom.

7 FIG. 6 FIG. 120 610 120 610 is a diagram illustrating a class-argmax computation step performed by the PPU, according to one example of the present disclosure. In the class-argmax operation step Sof, the first computation unitextracts the class with the highest-class score from the plurality of classes included in a bounding box. That is, in the class-argmax operation step S, the first computation unitperforms a class-argmax operation to extract the index of the class with the highest class score in the bounding box and its class score.

630 1 1 1 1 2 7 FIG. 5 FIG. 14 FIG. Specifically, within one memory bank of the internal memory, for each of the plurality of bounding boxes, an object presence confidence score of the bounding box, coordinate of the bounding box, and indices of the multiple classes corresponding to the objects contained in the bounding box, and the score of each class may be stored. Referring to, the memory bank Bankmay include data for a plurality of bounding boxes. The memory bank Bankmay include a part of DATA memory bank, a part of OUTPUTmemory bank, and a part of OUTPU2 memory bank, described above with reference to. For example, the memory bank Bank1 may include data for a first bounding box BOXand data for a second bounding box BOX. Similarly, the memory bank Bank2, described below with reference to, may include another part of DATA memory bank, another part of OUTPUT1 memory bank and another part of OUTPUT2 memory bank.

7 FIG. 1 1 1, 1 2 2 1 It is assumed in the example ofthat the shape of the bounding boxes is a rectangle. The data for the first bounding box BOXmay include an object presence confidence score C predicting an object presence in the first bounding box BOX, and bounding box coordinate data for the first bounding box BOXsuch as height data H, width data W, x data X, and y data Y. x data X and y data Y indicate the x-coordinate and the y-coordinate of the first bounding box BOXin the image, respectively. Further, the data for the second bounding box BOXmay also include an object presence confidence score C predicting an object presence for the second bounding box BOX, and the second bounding box coordinate data: height data H, width data W, x data X, and y data Y. The memory bank Bankmay also include a plurality of dummy data to fill in empty or unused bits in the word width.

The shape of the bounding box is not limited to a rectangle, but may be transformed into a pentagon, a polygonal shape or a circle. The number and type of bounding box coordinate data may vary according to the shape of the bounding box.

1 1 1 1 The data for the first bounding box BOXmay include a plurality of class score data 0 to 33 for the objects included in the first bounding box BOX. For example, an object included in the first bounding box BOXmay be predicted to be one of several classes, and the data for first bounding box BOXmay include class score data 0 to 33 of the predicted classes. Further, the data for the second bounding box BOX2 may also include a plurality of class score data (e.g., 0 to 33) for the objects included in the second bounding box BOX2. For example, an object included in the second bounding box BOX2 may be predicted to be one of several classes, and the data for second bounding box BOX2 may include class score data (e.g., 0 to 33) of the predicted classes.

120 610 120 610 1 2 610 1 0 0 1 610 33 33 2 1 2 610 1 1 1 1 1 1 630 1 1, Then, in the class-argmax operation step S, the first computation unitextracts the highest scoring class among the plurality of classes included in each bounding box. That is, in the class-argmax operation step S, the first computation unitperforms a class-argmax operation to extract the index of the highest scoring class and its class score for the first bounding box BOXand those of the second bounding box BOX. For example, the first computation unitextracts, from the first bounding box BOX, the first-class index' and corresponding class score data, having the highest score among class score data 0 through 32 associated with the first bounding box BOX. The first computation unitalso extracts from the second bounding box BOX2, the last class index' and corresponding class score data, which is the highest-class score among the class score data 0 through 33 associated with the second bounding box BOX. The class index and class score data along with the bounding box coordinate data may be stored in memory bank Bank1. By extracting only, the index data and the corresponding score data of one class from each of the bounding boxes BOX, BOXand using or sending the extracted index data and its score data, the first computation unitcan reduce the size of data of each bounding box used or sent for subsequent processing. After storing the extracted data to memory bank Bank, the remaining data in memory bank Bankis deleted or overwritten by other data and the data in memory bank Bankis subject to subsequent processing. That is, after the extracted data is stored in the memory bank Bank, the remaining data in the memory bank Bankis not used. The extracted data in memory bank Bankbecomes the target of subsequent processing. In this way, the more efficient use of data space available in internal memoryis enabled. Alternatively, instead of moving the bounding box coordinate data, and the extracted class index and class score data to memory bank Bank, the memory locations of the bounding box coordinate data, and the extracted class index and class score data may be stored in memory bank Bankwhich may be later referenced by subsequent processes.

8 FIG. 8 FIG. 9 FIG. 7 FIG. 620 1 130 620 610 1 1 0 1 33 2 130 620 610 1 630 130 620 1 0 610 is a diagram illustrating a filtering computation step performed by the second computation unitof the PPU on the bounding box BOX, according to one example of the present disclosure. The process ofis repeated for other bounding boxes as well.is a diagram illustrating the result of a filtering operation performed by the PPU according to one example of the present disclosure. In the filtering operation step S, the second computation unitextracts only the bounding boxes having a class confidence score above a threshold confidence score from the plurality of bounding boxes. The class confidence score may correspond to a product of an object presence confidence score C of the bounding box and a class score data extracted from the first computation unit. In the example described above with reference to, the class confidence score for BOXwould be the product of object present score C of BOXand class score dataof BOXwhile the class confidence score for BOX2 would be the product of object present score C of BOX2 and class score dataof BOX. In the filtering operation step S, the second computation unitextracts only those bounding boxes whose product of object presence confidence scores C of the bounding boxes and the class score data extracted from the first computation unitare above a certain threshold confidence score thr. The information on the extracted or filtered bounding boxes is then stored in memory bank Bankof the internal memory. The information on the extracted or filtered bounding boxes may include the bounding box coordinate data, the class index and the class score. Alternatively, memory bank Bank 1 may store the memory locations of the bounding box coordinate data, and the extracted class index and class score data of the filtered for further processing. In the filtering operation step S, the second computation unitdoes not store in memory bank Bankthe data for the bounding boxes whose product of the object presence confidence score C of the bounding box and the class scoreextracted from the first computation unitis less than or equal to the certain threshold confidence score thr. Only the data of the filtered bounding boxes may be subject to further processing. In this way, the amount of processing for the subsequent processing may be reduced.

9 FIG. 130 620 610 640 640 640 In, it is assumed that there are N number of bounding boxes that have completed the class-argmax operation in the first computation unit 610. Accordingly, in the filtering operation step S, the second computation unitmay extract only two bounding boxes among the N bounding boxes whose product of the object presence confidence score C and the class score data extracted from the first computation unitis greater than a certain threshold confidence score thr. Only the data for the two filtered bounding boxes are subsequently processed by the internal processing unit. Accordingly, the size of data to be processed by the internal processing unitmay be reduced, allowing the internal processing unitto perform operations at a faster speed using smaller memory. Thus, the performance of the PPU may be improved because it depends on the computation speed of the internal processing unit.

10 FIG. 6 FIG. 10 FIG. 6 FIG. 140 640 150 is a diagram illustrating a decoding step performed by a PPU, according to one example of the present disclosure. In a subsequent decoding step Sof, the internal processing unitmay decode the data in the filtered bounding boxes. Specifically, referring to, the bounding box coordinate data is decoded so that it can be processed in the non-maximum suppression (NMS) operation step Softhat is subsequently performed by multiplication, addition, and subtraction operations of the height data H, width data W, x data X, and y data Y corresponding to the bounding box.

11 FIG. 150 620 is a diagram illustrating an NMS operation step performed by the PPU, according to one example of the present disclosure. Subsequently, in the NMS computation step S, the redundant or overlapping bounding boxes generated by the second computation unitmay be removed. The NMS refers to a post-processing step used in object detection tasks to remove redundant or overlapping bounding boxes generated by object detection algorithms, typically in neural network models such as You Only Look Once (YOLO) or Faster R-CNN. Through the NMS computation step, duplicative bounding boxes may be removed, and only non-duplicate bounding boxes can be retained for further processing.

The NMS computation step can be broken down to include a confidence score sorting step and a deduplication step. First, in the confidence score sorting step, the bounding box data is sorted based on the confidence score, which is the product of the object presence confidence score of the bounding box and the class score. In one embodiment, the data of the bounding boxes with the highest confidence score is sorted first, and the remaining bounding boxes are sorted in a descending order of the confidence score.

630 630 In the deduplication step, the bounding box with the highest confidence score is used as the reference, and the degree of overlap between the bounding box with the highest confidence score and other bounding boxes is determined. The degree of overlap between a bounding box (REF BOX) having the highest confidence score and another bounding box is typically measured using the Intersection over Union (IoU), which is the ratio of the overlap and union area between two bounding boxes. If the IoU between the bounding box (REF BOX) with the highest confidence score and the other bounding box exceeds a predefined threshold (for example, 0.5 or more), this indicates that there is significant overlap between the two boxes, and thus the other bounding box is removed. The removal of the other bounding box is performed by deleting the data associated with the other bounding box from internal memoryor making space of internal memoryoccupied by the data available for overwriting by other data. If the IoU between the bounding box (REF BOX) with the highest confidence score and the other bounding boxes is equal to or below a predefined threshold (for example, 0.5 or more), the other bounding box is retained. Through the non-maximum suppression (NMS) computation step, redundant bounding boxes can be removed from internal memory 630 and non-redundant bounding boxes can be retained, thereby improving the accuracy and reliability of the object detection system.

12 FIG. 8 FIG. is a diagram illustrating the amount of data reduced in the PPU, according to one example of the present disclosure. In, data fields and their sizes for a single bounding box in the neural network models of YOLO, face and pose, the overall size of the data for the neural network models, and the reduction of the size of the data after performing filtering operation or a class-argmax operation in combination with the filtering operation are illustrated.

100 Taking the top example of using the neural network model of YOLO, the PPU may be input with 50 KB of data for each of thebounding boxes, including an object presence confidence score and an index of a plurality of classes and a class score for the object contained inside the bounding box.

610 100 50 620 100 10 610 Then, in the class-argmax operation step, the first computation unitperforms a class-argmax operation to extract the index of the highest scoring class and the class score for each of thebounding boxes, thereby reducing the plurality of class data (class, key point) to two. As a result, the size of the data is reduced fromKB to 6.25 KB as an example. In the filtering operation step, the second computation unitmay filter the number of bounding boxes fromtoby removing the bounding boxes whose product of the object presence confidence score of the bounding box and the class score extracted from the first computation unitis below a certain threshold confidence score thr, thereby reducing the size of the data from 6.25 KB to to 0.625 KB.

620 100 10 In the next example where the neural network model is a face recognition model Face, in the filtering operation step, the second computation unitmay filter the number of bounding boxes fromtoby removing bounding boxes having a confidence score below a certain threshold confidence score thr, thereby reducing the size of the data from 6.25 KB to 0.625 KB. In this example, there is no separate reduction performed by a class-argmax operation.

620 100 10 In the bottom example, the neural network model is a pose detection model Pose. In the filtering operation step, the second computation unitmay filter the number of bounding boxes fromtoby removing the bounding boxes in which the product of the object presence confidence score and the class score is below a certain threshold confidence score (thr), thereby reducing the size of the data from 25 KB to 2.5 KB. In this example, there is also no separate reduction performed by a class-argmax operation.

13 FIG.A 13 FIG.B is a diagram of a directed acyclic graph (DAG) representation of an object detection neural network model input to a NPU including a PPU, according to one example of the present disclosure.is a diagram of a directed acyclic graph (DAG) representation of an object detection neural network model that is post-processed in a NPU and a PPU, according to one example of the present disclosure.

13 FIG.A 255 255 255 An object recognition neural network model represented by a DAG can be composed of multiple layers and nodes connected to the multiple layers. As illustrated in, the object detection neural network model may include a convolutional layer Conv, a multiplication layer Mul, and an addition layer Add operations. Specifically, the output of the convolutional layer Conv may bechannels of 80 x 80 bounding box data, the output of the multiplication layer Mul may bechannels of 80 x 80 bounding box data, and the output of the addition layer Add may bechannels of 80 x 80 bounding box data.

13 FIG.B 13 FIG.A 13 FIG.B 600 600 1000 600 500 1000 600 1000 As shown in, when the PPUis applied according to one example of the present disclosure, the object detection neural network model may include a convolutional layer Conv, a multiplication layer Mul, and an addition layer Add, and may further include a programmed activation function layer DX_PAF, a class argmax layer PP_Argmax, and a filter layer PP_Filter. That is, in the NPU and the PPUaccording to one example of the present disclosure, the compiler may modify the object detection neural network model shown into further include a programmed activation function layer (DX_PAF), a class argmax layer (PP_Argmax), and a filter layer (PP_Filter) shown in. The compiler may modify or improve the neural network model according to hardware information of the NPU(e.g., the presence of the PPUor the presence of the SFU, and the like) to use dedicated circuitry (e.g., the PPU or SFU) provided in the NPUand PPUfor accelerated computation on the NPU.

13 FIG.B In, for example, the number of types of anchor boxes of bounding boxes is three, so three convolution layers Conv, three multiplication layers Mul, three addition layers Add, three programmed activation function layers DX_PAF, three class argmax layers PP_Argmax, and three filter layers PP_Filter are shown. The An anchor box is a predefined bounding box used in object detection to generate candidate regions of various sizes and aspect ratios for identifying objects at specific locations. The number of each layer may vary according to the number of types of anchor boxes of bounding boxes.

13 FIG.B 128 128 128 80 630 630 In, the output of the convolutional layer Conv may bechannels of 80 x 80 bounding box data, the output of the multiplication layer Mul may bechannels of 80 x 80 bounding box data, and the output of the addition layer Add may bechannels of 80 x 80 bounding box data. However, as described above, the size of the bounding box data output from the class-argmax layer PP_Argmax is reduced so that seven channels ofx 80 bounding box data including the object presence confidence score, bounding box coordinate data, and class data that predicted the object presence is retained in internal memorywhile removing others are discarded from internal memory.

7 630 Then, by processing in the filter layer PP_Filter, only the bounding box data whose class confidence score is above the threshold confidence score among the 80x80 bounding box data ofchannels can be retained in internal memoryand the remaining data may be discarded from internal memory 630.

14 FIG. 14 FIG. 1 1 2 2 3 3 4 4 is a timing diagram illustrating a plurality of image data computation processes of a NPU including a PPU, according to one example of the present disclosure.illustrates a process of computing a plurality of image data in a NPU, divided into a first period Periodin which a first image data IMGis received by the NPU, a second period Periodin which a second image data IMGis received by the NPU, a third period Periodin which a third image data IMGis received by the NPU, and a fourth period Periodin which a fourth image data IMGis received by the NPU.

14 FIG. 14 FIG. 1 1 610 620 1 610 620 1 630 In Period1 of, the plurality of PEs performs convolution operations on the first image data IMGto output data for a plurality of bounding boxes for the first image data IMG. While the plurality of PEs performs the convolution operation, the first computation unitof the PPU performs a class-argmax operation on the outputted plurality of bounding box data, and the second computation unitof the PPU performs a filtering operation on the bounding box data. In, “computation” means the class-argmax operation and filtering operation. The data of the bounding boxes in the first image data IMG, which is the output of the first computation unitand the second computation unit, may be stored in the first memory bank Bankof the internal memory.

2 1 2 2 2 610 620 2 610 620 2 630 640 1 1 2 640 3 2 Periodbegins after the plurality of PEs finishes performing the convolution operations on the first image data IMG. In Period, the plurality of PEs performs convolution operations on the second image data IMGto output data on a plurality of bounding boxes in the second image data IMG. While the plurality of PEs performs the convolution operation, the first computation unitperforms a class-argmax operation on the outputted plurality of bounding box data, and the second computation unitperforms a filtering operation on the bounding box data. The data of the bounding boxes in the second image data IMG, which is the output of the first computation unitand the second computation unit, may be stored in the second memory bank Bankof the internal memory. Meanwhile, the internal processing unitperforms decoding and NMS operations Post on the data of the bounding boxes of the first image data IMG, which is the previous image data received from the first memory bank Bank. In Period, because the convolution operation time of the plurality of PEs takes longer than the decoding and NMS operation time of the internal processing unit, Periodbegins after the plurality of PEs finishes convolution operations on the second image data IMG.

3 3 3 610 620 14 FIG. In Periodof, the plurality of PEs performs convolution operations on the third image data IMGto output data on a plurality of bounding boxes in the third image data IMG. While the plurality of PEs performs the convolution operations, the first computation unitperforms a class-argmax operation on the outputted plurality of bounding box data, and the second computation unitperforms a filtering operation on the bounding box data.

3 610 620 630 Then, the data of the bounding boxes in the third image data IMG, which is the output of the first computation unitand the second computation unit, may be stored in the first memory bank Bank1 of the internal memory.

3 640 2 2 640 3 640 2 14 FIG. Meanwhile, in Periodof, while the plurality of PEs performs the convolution operations, the internal processing unitperforms decoding and NMS operations Post on the data of the bounding boxes of the second image data IMGinput from the second memory bank Bank. Because the decoding and NMS the operation time for the internal processing unittake longer than the convolution time of the plurality of PEs in Period, Period4 begins after the internal processing unitfinishes decoding and NMS operations on the data in the bounding boxes of the second image data IMG.

4 4 4 2 610 620 640 3 In Period, the plurality of PEs performs convolution operations on the fourth image data IMG, outputting data on multiple bounding boxes in the fourth image data IMG, identical to the sequence of operations described in Period. While the plurality of PEs performs the convolution operations, the first computation unitperforms a class-argmax operation on the outputted plurality of bounding box data, and the second computation unitperforms a filtering operation. Further, while the plurality of PEs performs the convolution operations, the internal processing unitperforms decoding and NMS operations Post on the data of the bounding boxes of the third image data IMG.

640 1 2 640 2 Again, at the later of the time when the internal processing unitfinishes performing the non-maximum suppression (NMS) operation on the first image data IMGand the time when the plurality of PEs finishes performing the operation on the second image data IMG, the plurality of PEs starts performing the operation on the third image data, or the internal processing unitperforms the decoding and NMS operation Post on the data in the bounding boxes of the second image data IMG.

610 620 As described above, the NPU including the PPU according to the present disclosure may perform a class-argmax operation by the first computation unitand a filtering operation by the second computation uniton the outputted plurality of bounding box data while the plurality of PEs performs a convolution operation. Accordingly, the NPU including the PPU according to the present disclosure may reduce the processing time of the post-processing operation because the time for the class-argmax operation and the filtering operation, which are part of the post-processing operation, is not required separately. In other words, a NPU comprising a PPU according to the present disclosure will only need time separately for the decoding and NMS operations, which are different parts of the post-processing operation, and will not need time separately for the class-argmax operation and the filtering operation, which are different parts of the post-processing operation.

128 Specifically, if the post-processing operation performed by the PPU, including a class-argmax operation, a filtering operation, a decoding operation, and an NMS operation are all performed, the size of the data that the post-processing operation processes may be 8.2 MB, and the data processing time may be 24 ms. On the other hand, if only decoding and NMS operations are performed during the post-processing operation, the size of the data that the post-processing operation processes may beKB, and the data processing time may be 1.29 ms.

610 620 The NPU and the PPU according to the present disclosure has the benefit of the first computation unitperforming a class-argmax operation on the outputted data of a plurality of bounding boxes and the second computation unitperforming a filtering operation while the plurality of PEs performs convolution operations, so that the additional time required for the post-processing operation and the amount of data to be processed may be reduced. Thus, it has the advantage of, among others, improving the computation speed of the NPU and the PPU according to the present disclosure.

640 Furthermore, while the plurality of PEs performs the convolution operation, the internal processing unitmay perform decoding and NMS operations on the data of the bounding boxes of the previous image data. As a result, the decoding and NMS operations during the post-processing operation of the previous image data may overlap with the convolution operation time of the subsequent image data, which has the advantage that the computation speed of the NPU may be further improved.

15 FIG. 210 220 230 240 is a flowchart of a method for performing operations associated with a neural network model. At least multiply and accumulate operations on an input data to generate a plurality of bounding boxes is performed Sby the NPU. The bounding boxes are sent Sfrom the NPU to a post-processing circuit via a bus. The bounding boxes are filtered Sand a subset of the bounding boxes that satisfy a predetermined condition as data output by the post-processing circuit are outputted. The data output is stored Sin a memory of the PPU.

According to an example of the present disclosure, a NPU may be provided.

The NPU may comprise a plurality of PEs configured to perform operations of a neural network model. The NPU may comprise a SFU configured to perform activation function operations on the data output from the plurality of PEs. A PPU separate from the NPU is provided to process data output from the NPU.

The neural network model may be an object detection model, the data output from the plurality of PEs may include data of a plurality of bounding boxes for image data, and each of the data of the plurality of bounding boxes may include an object presence confidence score, bounding box coordinate data, and class data.

The PPU may include a first computation unit configured to extract a highest-scoring class of a plurality of classes included in each of a plurality of bounding boxes, and a second computation unit configured to extract one or more bounding boxes from the plurality of bounding boxes that have a class confidence score equal to or greater than a threshold confidence score. The class confidence score may be a product of an object presence confidence score of the bounding box and a class score extracted by the first computation unit.

The PPU may include a first computation unit configured to perform a class-argmax operation that extracts an index of a class having a highest score and a class score thereof for a plurality of bounding boxes, and a second computation unit configured to perform a bounding box filtering operation configured to extract one or more bounding boxes having a product of an object presence confidence score of the bounding box and a class score extracted from the first computation unit, equal to or greater than a threshold confidence score.

The PPU may include an internal memory configured to store the data output from the first computation unit and the second computation unit.

The internal memory may include a plurality of memory banks, a portion of the plurality of memory banks may be configured to store output data of the first computation unit, and another portion of the plurality of memory banks may be configured to store output data of the second computation unit.

While the plurality of PEs is performing an operation, the first computation unit may perform the class-argmax operation, and the second computation unit may perform the bounding box filtering operation.

The PPU may include an internal processing unit configured to perform an NMS operation for a plurality of extracted bounding boxes, and through the NMS operation, redundant bounding boxes among the plurality of extracted bounding boxes may be removed.

While the plurality of PEs performs an operation on subsequent image data, the internal processing unit may perform the NMS operation on previous image data.

The internal processing unit may be configured to start the NMS operation for subsequent image data from the later of a completion time of the internal processing unit processing the NMS operation for previous image data and a completion time of the plurality of PEs processing the subsequent image data following the previous image data.

The plurality of PEs may be configured to start an operation for third image data from the later of a completion time of the internal processing unit processing the NMS operation for first image data and a completion time of the plurality of PEs processing second image data following the first image data.

The NPU may comprise a compiler configured to add a class-argmax layer and a filtering layer to an input neural network model.

Embodiments relate to an NPU including a plurality of PEs circuit, and a subsequent circuit. The plurality of PEs circuit generates data output by performing convolution operations of a neural network model. A PPU separate from the NPU is coupled to the NPU to receive the data output and extracts a subset of the data output.

In one or more embodiments, the data output includes, for each bounding box in a region of an image, class scores indicative of probability that classes of objects being presenting in each bounding box.

In one or more embodiments, the post-processing circuit includes a first computation circuit that selects one or more classes for each bounding box as the subset of the data output by comparing class scores of classes for each bounding box.

In one or more embodiments, the post-processing circuit further includes a second computation circuit that extracts one or more bounding boxes by comparing a class confidence score of each bounding box with a threshold confidence score. The class confidence score represents probability that an object of a class is present in each bounding box. The class confidence score is derived from the object presence confidence score and the class scores.

In one or more embodiments, the second computation circuit computes the class confidence score as a product of the object presence confidence score and a class score with the subset of classes extracted by the first computation circuit.

In one or more embodiments, the post-processing circuit further includes an internal memory coupled to the first computing circuit and the second computation circuit. The internal memory stores the subset of classes for each bounding box extracted by the first computation circuit, and stores data of the one or more bounding boxes extracted by the second computation circuit.

In one or more embodiments, the post-processing circuit further includes an internal processing circuit that performs a non-maximum suppression (NMS) operation on the one or more bounding boxes extracted by the second computation circuit.

In one or more embodiments, the internal processing circuit performs the NMS operation during a period in which the plurality of PEs performs the convolution operations.

In one or more embodiments, the internal processing circuit starts the NMS operation for a subsequent image subsequent to the image at a time that is later of (i) a completion time of the NMS operation for the image data and (ii) a completion time of the convolution operations on the image by the plurality of PEs circuit.

In one or more embodiments, the data output further includes coordinate data of each bounding box.

In one or more embodiments, the post-processing circuit further includes an internal memory that stores the subset of classes for each bounding box extracted by the first computation circuit.

In one or more embodiments, the first computation circuit performs the comparison of the class scores during a period in which the plurality of PEs circuit performs the convolution operations.

In one or more embodiments, the neural processing circuit further includes one or more processors, and memory. The memory stores instructions of a compiler. The instructions when executed by the one or more processors cause adding a class-argmax layer to generate the neural network model. The extracting of the subset of classes by the first computation circuit corresponds to operations of the class-argmax layer.

Embodiments relate to a PPU. The PPU may be implemented in register transfer level (RTL) code and designed to interface with a NPU (NPU) configured for object detection computations of a neural network model. The PPU may include a processing unit configured to filter a plurality of bounding boxes transmitted from the NPU and output only those that satisfy a particular condition and one or more input registers configured to store data output from the processing unit.

In one or more embodiments, the data may include one or more of an object presence confidence score indicating a probability that an object is present in each bounding box of the plurality of bounding boxes, a coordinate of each bounding box of the plurality of bounding boxes, and a class associated with each bounding box of the plurality of bounding boxes.

In one or more embodiments, the particular condition may comprise a class having a highest score among a plurality of classes and a class confidence score greater than or equal to a threshold confidence score.

In one or more embodiments, the class confidence score may be a product of an object presence confidence score and a score of the class.

In one or more embodiments, the PPU may include a memory unit configured to store an intermediate result and a final result of the processing unit.

In one or more embodiments, the memory unit may comprise a plurality of memory registers, and an address generation logic for accessing the plurality of memory registers.

In one or more embodiments, the PPU may be configured to perform a non-maximum suppression (NMS) operation on the plurality of bounding boxes.

In one or more embodiments, the NMS operation may be performed to remove redundant or overlapping bounding boxes of the plurality of bounding boxes.

In one or more embodiments, the NPU may comprise a plurality of PEs configured to perform operations of the neural network model, and a special function unit configured to receive output data of the plurality of PEs to perform activation function operations.

Embodiments relate to a circuit for a PPU. The circuit is configured to interface with a NPU (NPU) configured for object detection computations of a neural network model. The circuit may include a memory circuit configured to store data, of a plurality of bounding boxes, output from the NPU, and a processing circuit configured to filter the plurality of bounding boxes and output only those that satisfy a particular condition.

In one or more embodiments, the processing circuit may comprise a first computation circuit configured to output a highest scoring class of a plurality of classes included in each of the plurality of bounding boxes, and a second computation circuit configured to output one or more bounding boxes of the plurality of bounding boxes having a class confidence score above a threshold value.

In one or more embodiments, the processing circuit may comprise a first computing circuit configured to process a class-argmax operation to extract an index of a class having highest score and a score of the class from each of the plurality of bounding boxes.

In one or more embodiments, the processing circuit may comprise a second computing circuit configured to process a bounding box filtering operation to extract one or more bounding boxes from the plurality of bounding boxes for which a product of an object presence confidence score and the score of the class extracted by the first computation circuit is above a threshold value.

In one or more embodiments, the memory circuit may comprise a plurality of memory banks. A portion of the plurality of memory banks may be configured to store output data of the first computing circuit. Another portion of the plurality of memory banks may be configured to store output data of the second computing circuit.

In one or more embodiments, the processing circuit cooperating with the NPU may be configured to perform a non-maximum suppression (NMS) operation on the plurality of bounding boxes.

The examples of the present disclosure disclosed herein and in the drawings are provided solely to explain the technical content of the present disclosure and to facilitate understanding of the present disclosure, and are not intended to limit the scope of the present disclosure. It will be apparent to one of ordinary skill in the art to which the present disclosure belongs that other modifications based on the technical ideas of the invention may be practiced in addition to the examples shown herein.

[National R&D Project Supporting This Invention]

[Project Identification Number] Not assigned

[Task Number] ​ 00399936

[Name of Ministry] ​Ministry of Science and ICT

[Name of Task Management (Specialized) Institution] Institute of Information & Communications Technology Planning & Evaluation

[Research Project Title] Development of Unified Software Flatform of Semiconductor Technology Applicable for Artificial Intelligence

[Research Task Name] Development of Quality Performance Evaluation Test (BMT) Platform Technology for Edge AI Semiconductors

[Name of the organization performing the task] DeepX Co., Ltd.

[Research Period] 2024.04.01 ~ 2027.12.31