Method and apparatus with neural network based on accumulated update value

This application is a Continuation Application of U.S. patent application Ser. No. 16/106,703 filed on Aug. 21, 2018, which claims the benefit of Korean Patent Application No. 10-2017-0137374, filed on Oct. 23, 2017, in the Korean Intellectual Property Office, the disclosures of which are incorporated herein in their entirety by reference.

The present disclosure relates to methods and apparatuses with a neural network.

Technological automation of feature extraction, pattern recognition, and/or analyses, as only examples, has been implemented through processor implemented neural network models, as specialized computational architectures, that after substantial training may provide computationally intuitive feature extractions or recognitions, mappings between input patterns and output patterns, pattern recognitions of input patterns, or categorization in various forms. The trained capability of extracting such information or recognitions, generating such mappings, performing such pattern recognitions, or performing such categorizations may be referred to as a learning capability of the neural network. Such trained capabilities may also enable the specialized computational architecture to classify an input pattern or object(s), or portions of the input pattern or object(s), e.g., as a member that belongs to one or more predetermined groups. Further, because of the specialized training, such specially trained neural network may thereby have a generalization capability of generating a relatively accurate or reliable output with respect to an input pattern that the neural network may not have been trained for, for example. However, because such operations are performed through such specialized computation architectures, and in different automated manners than they would have been performed in non-computer implemented or non-automated approaches, they also invite problems or drawbacks that only occur because of the automated and specialized computational architecture manner in which they are implemented.

The neural network apparatus performs many operations on complex input data.

This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

In one general aspect, there is provided a processor-implemented neural network method, the method including: calculating individual update values for a weight assigned to a connection relationship between nodes included in a neural network; generating an accumulated update value by accumulating the individual update values in an accumulation buffer; and training the neural network by updating the weight using the accumulated update value in response to the accumulated update value being equal to or greater than a threshold value.

The threshold value may be a value corresponding to a least significant effective bit of the weight; and the updating may include: determining an effective update value based on the accumulated update value, adding the effective update value to the weight, and subtracting the effective update value from the accumulated update value of the accumulation buffer.

The effective update value may be a portion of the accumulated update value.

The effective update value may be a multiple of the least significant effective bit of the weight.

The method may further include determining whether the accumulated update value may be equal to or greater than the threshold value at a predetermined update period.

The method may further include accumulating the individual update values in the accumulation buffer until a next update period in response to the accumulated update value being smaller than the threshold value.

The weight may be a fixed point value including a first sign bit, a first integer part, and a first fractional part; the accumulated update value may be a fixed point value including a second sign bit and a second fractional part; and the updating may include adding, to the weight, a value of at least one bit that overlaps the first fractional part of the weight among bits representing the second fractional part of the accumulated update value.

The updating may include initializing the value of the at least one bit that overlaps the first fractional part to a same value as the second sign bit.

The method may further include adjusting a position of a decimal point of the accumulated update value.

The adjusting may include: moving the position of the decimal point of the accumulated update value in a lower bit direction of the accumulated update value in response to an average value of the individual update values being smaller than a second threshold value; and moving the position of the decimal point of the accumulated update value in an upper bit direction of the accumulated update value in response to the average value of the individual update values being greater than the second threshold value.

The second threshold value may be 1/a times the threshold value; and a may be a natural number.

The weight may be a dynamic fixed point value of which a bit number of the first fractional part may be adjusted.

The weight may be a floating point value including a first sign bit, a first exponent part, a first mantissa part, and a first bias; the accumulated update value may be a floating point value including a second sign bit, a second exponent part, a second mantissa part, and a second bias; and the updating may include adding an effective value of the accumulated update value included in an effective number range of the weight to the weight.

The method may further include adjusting the second bias of the accumulated update value.

The adjusting may include: increasing the second bias in response to the second exponent of the accumulated update value being greater than the threshold value; and decreasing the second bias in response to the accumulated update value being smaller than a second threshold value.

The second threshold value may be 1/b times the threshold value; and b may be a natural number.

A non-transitory computer-readable storage medium may store instructions that, when executed by a processor, cause the processor to perform the method.

In another general aspect, there is provided a neural network apparatus, the apparatus including: one or more processors configured to: calculate individual update values for a weight assigned to a connection relationship between nodes included in a neural network, generate an accumulated update value by accumulating the individual update values in an accumulation buffer, and train the neural network by updating the weight using the accumulated update value in response to the accumulated update value being equal to or greater than a threshold value.

The threshold value may be a value corresponding to a least significant effective bit of the weight; and the one or more processors are further configured to: determine an effective update value based on the accumulated update value, add the effective update value to the weight, and subtract the effective update value from the accumulated update value of the accumulation buffer.

The effective update value may be a portion of the accumulated update value.

The effective update value may be a multiple of the least significant effective bit of the weight.

The one or more processors are further configured to determine whether the accumulated update value may be equal to or greater than the threshold value at a predetermined update period.

The one or more processors are further configured to accumulate the individual update values in the accumulation buffer until a next update period in response to the accumulated update value being smaller than the threshold value.

The weight may be a fixed point value including a first sign bit, a first integer part, and a first fractional part; the accumulated update value may be a fixed point value including a second sign bit and a second fractional part; the one or more processors are further configured to add, to the weight, a value of at least one bit that overlaps the first fractional part of the weight among bits representing the second fractional part of the accumulated update value.

The one or more processors are further configured to initialize the value of the at least one bit that overlaps the first fractional part to a same value as the second sign bit.

The one or more processors are further configured to adjust a position of a decimal point of the accumulated update value.

The one or more processors are further configured to: move the position of the decimal point of the accumulated update value in a lower bit direction of the accumulated update value in response to an average value of the individual update values being smaller than a second threshold value; and move the position of the decimal point of the accumulated update value in an upper bit direction of the accumulated update value in response to the average value of the individual update values being greater than the second threshold value.

The second threshold value may be 1/a times the threshold value; and a may be a natural number.

The weight may be a floating point value including a first sign bit, a first exponent part, a first mantissa part, and a first bias; the accumulated update value may be a floating point value including a second sign bit, a second exponent part, a second mantissa part, and a second bias; and the one or more processors are further configured to add an effective value of the accumulated update value included in an effective number range of the weight to the weight.

The apparatus may further include a memory configured to store one or more programs, wherein the one or more processors are configured to calculate the individual update values, generate the accumulated update value, and train the neural network, in response to executing the one or more programs.

In another general aspect, there is provided a processor-implemented neural network method, the method including: calculating individual update values for a weight assigned to a connection relationship between nodes included in a neural network; generating an accumulated update value by adding the individual update values; and training the neural network by updating the weight using the accumulated update value in response to the accumulated update value being equal to or greater than a threshold value.

The updating may include: determining a portion of the accumulated update value to be an effective update value; and updating the weight by adding the effective update value to the weight.

The method may further include subtracting the effective update value from the accumulated update value; adding another individual update value to the accumulated update value; and re-updating the weight using the accumulated update value in response to the accumulated update value being equal to or greater than the threshold value.

The method may further include adjusting a size of the accumulated update value, by a factor, based on a comparison between a second threshold value and either one or both of an average value of the individual update values and the accumulated update value, wherein the updating may include updating the weight using the adjusted accumulated update value.

Additional aspects will be set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice of the presented embodiments.

The following detailed description is provided to assist the reader in gaining a comprehensive understanding of the methods, apparatuses, and/or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatuses, and/or systems described herein will be apparent after an understanding of the disclosure of this application. For example, the sequences of operations described herein are merely examples, and are not limited to those set forth herein, but may be changed as will be apparent after an understanding of the disclosure of this application, with the exception of operations necessarily occurring in a certain order. Also, descriptions of features that are known may be omitted for increased clarity and conciseness.

The features described herein may be embodied in different forms, and are not to be construed as being limited to the examples described herein. Rather, the examples described herein have been provided merely to illustrate some of the many possible ways of implementing the methods, apparatuses, and/or systems described herein that will be apparent after an understanding of the disclosure of this application.

Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to like elements throughout. In this regard, the present embodiments may have different forms and should not be construed as being limited to the descriptions set forth herein. Accordingly, the embodiments are merely described below, by referring to the figures, to explain aspects. As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed items. Expressions such as “at least one of,” when preceding a list of elements, modify the entire list of elements and do not modify the individual elements of the list.

As used herein, the terms ‘comprises’ or ‘comprising’ and the like should not be construed as necessarily including all of various elements or operations described in the specification and should be construed as excluding some of them, or may be interpreted as including additional components or operations.

Also, as used herein, the terms including ordinals such as “first” or “second” may be used to describe various elements, but the elements should not be limited by the terms. The terms are used only for the purpose of distinguishing one element from another.

The embodiments are directed to a method and an apparatus for processing a parameter in a neural network, and detailed descriptions thereof will be omitted with respect to matters widely known.

1 FIG. 10 is a diagram illustrating an example of a neural networkaccording to one or more embodiments.

1 FIG. 10 10 1 2 1 2 Thus, as illustrated in, the neural networkmay have a structure including an input layer, hidden layers, and an output layer. The neural networkmay perform an operation based on received input data (for example, Iand I) and generate output data (for example, Oand O) based on a result of the operation.

10 10 1 FIG. The neural networkmay be a deep neural network (DNN) or an n-layers neural network including two or more hidden layers. For example, as illustrated in, the neural networkmay be the DNN including an input layer (Layer 1), two hidden layers (Layer 2 and Layer 3), and an output layer (Layer 4). The DNN may be a fully connected neural network, convolutional neural network (CNN), a recurrent neural network (RNN), a deep belief network, a restricted Boltzman machine. The DNN may also include different or overlapping neural network portions respectively with such full, convolutional, recurrent connections. However, the DNN is not limited thereto.

10 The neural networkmay be configured to perform, as non-limiting examples, object classification, object recognition, voice recognition, and image recognition by mutually mapping input data and output data in a nonlinear relationship based on deep learning. Such deep learning is indicative of processor implemented machine learning schemes for solving issues, such as issues related to automated image or speech recognition from a big data set, as non-limiting examples. The deep learning may be implemented by mapping of input data and the output data through supervised or unsupervised learning or training, such that when trained the resultant machine learning model, engine, or example NN may intuitively map further input data to output data with a desired accuracy or reliability. Herein, it is noted that use of the term ‘may’ with respect to an example or embodiment, e.g., as to what an example or embodiment may include or implement, means that at least one example or embodiment exists where such a feature is included or implemented while all examples and embodiments are not limited thereto.

10 10 10 10 10 1 FIG. 1 FIG. When the neural networkincludes a DNN structure, the neural networkmay include more layers that may extract effective information, such as for pattern recognition or for classification purposes. In, the neural networkis illustrated as including four layers, but this is only an example, and the neural networkmay include fewer or more layers. For example, the neural network may include one or more input layers, three or more hidden layers, and one or more one output layers, for a total of five or more layers. In addition, the neural networkmay include layers of various structures different from those illustrated in.

10 10 1 FIG. Each of the layers included in the neural networkmay include a plurality of nodes referred to as ‘processing elements (PEs)’, ‘units’, or similar terms. The nodes may also be referred to as artificial neurons though such reference is not intended to impart any relatedness with respect to how the neural network architecture computationally maps or thereby intuitively recognizes information and how a human's neurons operate, i.e., the term artificial neuron is merely a term of art referring to the hardware implemented nodes of a neural network. For example, as illustrated in, the input layer (Layer 1) may include two nodes and the second layer (Layer 2) may include three nodes. However, this is only an example and each of the layers included in the neural networkmay include various numbers of nodes.

10 The nodes included in different layers included in the neural networkmay be connected or linked to one another to exchange data. For example, one node in a layer may receive data from other nodes in another layer and operate according to the received data, and output operation results to other nodes in yet another layer. Additionally, in a recurrent connection example, one node in a layer may receive data from itself, and/or from another node of the layer, from a previous time. The number of the connections may correspond to the number of the nodes included in the subsequent layer. For example, in adjacent fully connected layers, each node of a current layer may have a respective connection to each node of the subsequent layer, noting that in some examples such full connections may later be pruned or minimized during training or optimization.

An output value of each of the nodes may be referred to as activation, or a value which results from such a predetermined activation function of the corresponding node. The activation may thus be the output value of one node, and at the same time and due to corresponding connection(s) with a next layer, may be an input value for nodes included in the next layer. Each of the nodes may determine its activation based on resultant activations and weights received from the nodes included in a previous layer. A weight is a parameter used for calculating the activation at each node and may be a value assigned to a connection relationship between nodes. For example, an output from a previous layer's node may be provided to as an input to a node of a next or subsequent layer through a weighted connection between the previous layer's node and the node of the next layer, with the weight of the weighted connection being variously adjusted during the training of the neural network until the neural network is trained for a desired objective. There may be additional connections to the node of the next layer, such as for providing a bias connection value through a connection that may or may not be weighted and/or for providing the above example recurrent connection which may be weighted. During training and implementation such connections and connection weights may be selectively implemented, removed, and varied to generate or obtain a resultant neural network that is thereby trained and that may be correspondingly implemented for the trained objective, such as for any of the above example recognition objectives.

10 100 The architecture, selective connections between neighboring nodes, and corresponding connection weights may be varied during training of the neural network. The connection weights of the neural network may also be referred to as parameters of the neural network. For example, in a non-limiting supervised training example, the neural network may be trained based on labeled input image information or desired corresponding output images classifications and through backpropagation, as only an example. In the training, connection weightings between nodes of different hidden layers may be recursively adjusted until the corresponding neural network model is trained with a desired accuracy rate or below a maximum error rate, for example. The respectively trained neural network may be stored in a memory of the training or recognition apparatus. For example, the trained neural network may be stored in trained vectors, matrix or matrices, or other format, where elements of the matrix represent or suggest the corresponding trained weighted connections (parameters) of the corresponding neural network structure. The stored trained neural network may further include hyper-parameter information, which may define the specific structure or architecture of the corresponding neural network for which the example stored trained parameters correspond to. The hyper-parameters may define the number of hidden layers, the function of the layers, such as whether they are convolutional layers, how many and respective structures there are of such convolutional, de-convolutional, sampling, recurrent, and/or fully connected hidden layers of the neural network structure. The hyper-parameters may further include information of the configuration and values of any bias and/or contextual nodes in the neural network in differing examples, corresponding activation functions of the nodes, types of nodes, such as long short-term memory nodes in some examples, and define any further recurrent structures of the neural network, which may vary in differing embodiments. In the above example when there are multiple input layers, the hyper-parameters may for example also describe the architecture of the neural network for where within the neural network information acted on by a portion of the neural network subsequent to a first input layer is merged or considered with neural network information acted on by another portion of the neural network subsequent to a second input layer, or further where such portion and other portion of the neural network are divergent portions after a single input layer whose respective information is similarly subsequently merged. Similarly, in examples with separate output layers, the hyper-parameters may, for example, also describe the architecture of the neural network, including where within the neural network information acted on by an earlier portion of the neural network diverges or is separately considered within the neural network by separate, e.g., independent or dependent, portions of the neural network for ultimate respective consideration and provision to the separate output layers or separate pre-output layers that are then collectively acted on by a single output layer. Here, the example neural network information within the neural network may be information resident at, or represented by, respective layers of the neural network, and/or respective information resident at, or represented by, respective connections within the neural network. In some examples, such as where a hidden layer is a convolutional layer, the layer may output or generate one or more extracted feature information of information provided to the convolutional layer and dependent on the respectively applied kernels of the convolutional layer. In this example, the input of the convolutional layer may be two, three, or more dimensional information, such as a three dimensional volume, and the output of the convolutional layer may also be multi-dimensional information dependent on the set convolutional parameters, kernels, and architecture of the convolutional layer. The input to the neural network, as well as the example input data of the input layer, may similarly be singular or multi-dimensional information, such as in the form of pixels, time captured data, volume data, feature information, feature vectors, or feature maps, as non-limiting examples. Thus, the input data may be representative of image information, audio information, or other information. In an example where the neural networkis implemented with battery state monitoring for example, the input information may alternatively be various battery state information, e.g., voltage, current, temperature, etc., such as at different points in time or over periods of time.

1 FIG. Accordingly, returning to, each of the nodes may be a computational unit that receives one or more data values, e.g., through such weighted connections, and outputs the resulting activation and thereby may map inputs to outputs. The computational unit may correspond to the activation function for a node. As a non-limiting example, if σ is an activation function,

th th th th is a weight for the connection between the knode in the (i−1)layer and the jnode in the ilayer,

th is a bias connection value for the jlayer, and

th th is the resultant activation of the jnode in the ilayer, the activation

may be expressed by the below Equation 1.

1 FIG. As illustrated in, the activation of the first node in the second layer (Layer 2) may be expressed as

in addition

may have a value of

10 according to Equation 1. However, the above-described Equation 1 is only an example for describing the activation and the weight used for processing data in the neural network, but is not limited thereto. The activation may be a value obtained by processing through a rectified linear unit (ReLU) a value obtained by applying the activation function to a weighted sum of activations received from the previous layer.

10 10 The neural networkmay use a low-precision number system. The low-precision number system is a system that reduces a computational amount while reducing precision loss by using data having a relatively smaller number of bits for parameters and/or activation results than that of a typical high-precision neural network. For example, a typical high-precision neural network uses 32-bit or 64-bit floating point or fixed point data for parameters and activation results, while the neural networkmay use a low-precision number system that may be 8-bit or 16-bit floating point or fixed point data for the parameters and/or the activation results.

10 10 10 As noted, use of the neural networkusing 8-bit or 16-bit floating point or fixed point data means that the activation values, the weights, and the bias connection values of each of plural nodes, e.g., of one or more layers, of the neural networkhave an 8-bit or 16-bit floating point or fixed point data format. However, the neural networkthat uses the low-precise number system may use floating point or fixed point data with a smaller number of bits as parameters and is not limited to the above example.

10 10 10 10 10 10 10 2 12 FIGS.to In the neural network, a large number of data sets are exchanged between a plurality of mutually connected nodes and undergo many computation processes through layers. The neural networkthat uses the low-precise number system uses data having a smaller number of bits than that of the typical high-precision neural network during many computation processes, and thus a computational amount of the neural networkmay be reduced as compared with the typical high-precision neural network. Also, because the neural networkuses a smaller number of bits than that of the typical higher-precision neural network, the neural networkmay handle more complex data sets than the typical higher-precision neural network. with the same resources. However, even if the neural networkthat uses the low-precise number system reduces the computational amount, a technique for minimizing precision loss may be desirable. Methods of minimizing the precision loss in the neural networkthat use such a low-precise number system will now be described in greater detail with reference to.

2 FIG. is a flowchart illustrating an example of a method of processing parameters in a neural network according to some embodiments. The neural network may be a neural network that uses a low-precision number system. The parameters may be weights assigned to connection relationships between nodes included in the neural network. Hereinafter, the parameters to be processed in the neural network will be described as weights, but the parameters are not necessarily limited to the weights. The parameters may refer to any data processed in the neural network.

2 FIG. 3 3 FIGS.A toC 210 Referring to, in operation, the neural network apparatus may calculate individual update values for the weights assigned to the connection relationships between the nodes included in the neural network. A weight may be a floating point value or a fixed point value. The floating point value and the fixed point value will be described below with reference to.

3 FIG.A 3 FIG.B 3 FIG.C 30 35 is a diagram illustrating an example of a floating point valueaccording to some embodiments.is an example of a fixed point valueaccording to some embodiments.is a diagram illustrating another example of a fixed point value according to some embodiments.

3 FIG.A 30 310 320 330 340 Referring to, the floating point valuemay include a sign bit, an exponent part, a mantissa part, and a bias. A floating point means that a number is divided into a part indicating a decimal point part (that is, mantissa) and a part indicating a decimal point position (that is, exponent).

330 320 310 30 340 320 30 310 320 330 340 30 The mantissa partmay correspond to the part indicating the decimal point part. The exponent partmay correspond to the part indicating the decimal point position. The sign bitmay determine a sign of the floating point value. The biasmay be a value that is added to or subtracted from the exponent partand that is determined to represent a negative exponent. The floating point valuemay include the sign bit, bits corresponding to the exponent part, and bits corresponding to the mantissa part. The biasmay be previously determined with respect to the floating point valueand stored separately in either one or both of a memory and a local memory, such as an accumulation buffer.

310 320 330 340 30 Meanwhile, when the sign bitdenotes sign, the exponent partdenotes exponent, the mantissa partdenotes mantissa, and the biasdenotes bias, the floating point valuemay be a value following Equation 2.

3 FIG.B 35 315 325 335 345 Referring to, the fixed point valuemay include a sign bit, an integer part, a fractional part, and a decimal point. A fixed point means a notation representing decimal of a fixed digit number using a decimal point.

315 35 325 35 335 35 345 325 335 35 The sign bitmay determine a sign of the fixed point value. The integer partmay correspond to a part representing an integer of the fixed point value. The fractional partmay correspond to a part representing the decimal of the fixed point value. The decimal pointmay mean a point that is a reference for distinguishing the integer partand the fractional partof the fixed point value.

35 35 35 325 335 345 3 FIG.C 3 FIG.C Meanwhile, a value represented by the fixed point valuemay be described with reference to. Referring to, the fixed point valuemay be an 8-bit fixed point value. The fixed point valuemay also include the integer part, the fractional part, and the decimal point.

325 335 325 335 325 325 325 335 35 325 315 35 3 FIG.C Each of the bits representing the integer partand the fractional partmay have a value of 1 or 0. Also, as shown in, the bits representing the integer partand the fractional partmay sequentially have values of −8, +4, +2, +1, +0.5, +0.25, +0.125 and +0.0625. When a most significant bit of the integer partis 1, since a value represented by the most significant bit of the integer partis −8, even if other bits included in the integer partand the fractional parthave a certain value, the value represented by the fixed point valuemay be negative. The most significant bit of the integer partmay correspond to the sign bitthat determines the sign of the fixed point value.

3 FIG.C 35 35 However,is only an example, and the fixed point valuemay be a 16-bit fixed point value and may be a fixed point value having any suitable number of bits. Further, the fixed point valuemay represent a negative number by any one of a coding expression method, a complement method of 1, and a complement method of 2.

2 FIG. 1 FIG. 10 Returning back to, an individual update value may be a value that is calculated to update a weight, such as during the training of the neural networkof. A target to learn in the neural network is the weight. As noted above, the weight is updated repeatedly in a learning process in order for the neural network to gradually become trained to output accurate results that at least meet a learned accuracy threshold or do not exceed a maximum error threshold. As one of methods of updating the weight, for example, a gradient descent algorithm may be used. The gradient descent algorithm may mean an algorithm that minimizes an error by adjusting a parameter in a gradient direction of a loss function or an error function to find a lowest point of the loss function or the error function. For example, when the loss function denotes L, the parameter is 0, and a learning rate is n, the parameter may be adjusted according to Equation 3 below.

in the gradient direction of the loss function. For example, when the parameter is the weight,

may be the individual update value calculated to update the weight. Also, the learning rate y may be a varied determiner that controls how fast or aggressive such changes are made to the parameter is θ.

Generally, the individual update value

has a much smaller value than the weight. Thus, compared to the aforementioned typical high-precision neural network approaches, if a neural network uses a low-precision number system (e.g., 8 bits or 16 bits), the individual update value

may not be included in a range of values that the weight may represent. Thus, when the weight could be updated according to Equation 3, the individual update value

could be disregarded and the weight may not be updated. Although the weight needs to be updated repetitively, in order for the neural network to gradually output better results, the weight may not be updated due to a magnitude of the individual update value

and thus, it may be difficult or impractical to train such a low-precision neural network.

220 230 220 Rather, a neural network apparatus according to the present disclosure may perform operationsandsuch that the individual update values are not disregarded but reflected in a process of updating of the weight, so a low-precision neural network may be trained without such impediments and drawbacks. In operation, the neural network apparatus may accumulate the individual update values in an accumulation buffer. The accumulation buffer may be dedicated hardware for accumulating the individual update values and may be memory allocated to accumulate the individual update values. The individual update values may be accumulated in the accumulation buffer and become an accumulated update value. The accumulation buffer may store the accumulated update value.

10 1 FIG. In the neural network that uses the low-precision number system, such as the neural networkofas non-limiting examples, the neural network apparatus may accumulate the individual update values to a magnitude that may affect the weight such that the individual update values are not disregarded in the process of updating of the weight during training of the neural network.

230 In operation, the neural network apparatus may update the weight by using the accumulated update value if the accumulated update value of the accumulation buffer is equal to or greater than a threshold value. The neural network apparatus may determine whether the accumulated update value of the accumulation buffer is equal to or greater than the threshold value. The threshold value may be a value for determining whether the accumulated update value has been accumulated to the magnitude that may affect the weight. For example, the threshold value may be a value corresponding to a least significant effective digit of the weight.

−4 −4 −4 −4 −4 If the weight is a fixed point value, the least significant effective digit of the weight may be a digit corresponding to a least significant bit of the bits representing the weight. For example, if the least significant bit of the weight represents a 2digit, the least significant effective digit of the weight may be a 2digit, and the threshold value may be 2that corresponds to the least significant effective digit of 2. In an example, a 2digit may correspond to a smallest possible value greater than zero of an 8-bit fixed point data including a 4-bit integer part and a 4-bit fractional part.

0 2 0 1 1-8 −7 −7 −7 If the weight is a floating point value, the least significant effective digit of the weight may be a digit corresponding to the least significant bit of the bits representing an exponent part of the weight. However, if the weight is the floating point value, a bias may be considered to determine the least significant effective digit. For example, even if the least significant bit of the bits representing the exponent part of the weight represents a 2digit, if the bias is 8, the least significant effective digit of the weight is not a 2-2digit but a 2=2digit, and the threshold value may be 2that corresponds to the least significant effective digit of 2.

However, this is merely an example, and the threshold value may be any appropriate value between the value corresponding to the least significant effective digit of the weight and a value corresponding to a most significant effective digit of the accumulated update value.

If the accumulated update value is equal to or greater than the threshold value, since the accumulated update value has the magnitude that may affect the weight, the neural network apparatus may update the weight by using the accumulated update value.

Specifically, the neural network apparatus may determine a part of the accumulated update value that may be summed to an effective digit of the weight as an effective update value. The effective update value may mean a value actually used to update the weight in the accumulated update value. A range of values that weights may represent is limited by the number of bits allocated to the weights. Therefore, only values corresponding to the effective digit of the weight among values above the least significant effective digit of the weight may be added to the weight.

2 1 0 −1 −2 −3 −4 2 1 0 −1 −2 −3 −4 0 −2 −6 0 −2 −6 For example, when the weight is an 8-bit fixed point value including a 1-bit sign bit, a 3-bit integer part, and a 4-bit fractional part, the effective digit of the weight may be 2, 2, 2, 2, 2, 2, and 2digits. Due to the limit of the range of values that the weight may represent, in order to add a value to the weight, the weight needs to include at least one of values corresponding to any digit among 2, 2, 2, 2, 2, 2, and 2. For example, suppose that a value of 1.265625 (a decimal number) should be summed to the weight. Since the decimal number 1.265625 is equal to 1×2+1×2+1×2, 1×2and 1×2corresponding to the effective digit of the weight may be summed to the weight, whereas 1×2may not be summed to the weight due to the limit of the range of values that the weight may represent.

−4 2 1 0 −1 −2 −3 −4 2 1 0 −1 −2 −3 −4 2 1 0 −1 −2 −3 −4 −4 Meanwhile, a part that may be summed to the effective digit of the weight may include multiples of a value represented by the least significant digit of the weight. In the preceding example, the value corresponding to the least significant digit of the weight is 2, and the part of the accumulated update value that may be summed to the effective digit of the weight is 2, 2, 2, 2, 2, 2, and 2that are values corresponding to the 2, 2, 2, 2, 2, 2, and 2digits. All of 2, 2, 2, 2, 2, 2, and 2digits are the multiples of 2that is the value corresponding to the least significant digit.

Since the threshold value may be the value corresponding to the least significant digit of the weight, the effective update value may include multiples of the threshold value. For example, when the accumulated update value represents an accumulated update and the threshold value represents threshold, the effective update value may be expressed by Equation 4 below.

In Equation 4,

may be a quotient obtained by dividing Accumulated update by threshold. The effective update value may mean a value obtained by multiplying the quotient obtained by dividing the accumulated update value by the threshold value by the threshold value. Further, the effective update value may be the largest value among multiples of the threshold value that is smaller than the accumulated update value. For example, if Accumulated update is 2.2 (a decimal number) and threshold is 1 (a decimal number), the effective update value may be 2 that is the largest value among 1 and 2 that are multiples of the threshold value of 1 that is smaller than the accumulated update value of 2.2.

The neural network apparatus may add the effective update value to the weight. Since the effective update value is a value actually used to update the weight, the neural network apparatus may update the weight by adding the effective update value to the weight.

The neural network apparatus may also subtract the effective update value from the accumulated update value of the accumulation buffer. Since the effective update value of the accumulated update value of the accumulation buffer is reflected to the weight, the neural network apparatus may subtract the effective update value from the accumulated update value of the accumulation buffer. For example, since the weight is updated, the accumulated update value may be subtracted according to Equation 5 below.

In Equation 5, since

is the effective update value, the accumulated update value of the accumulation buffer may have a value obtained by subtracting the effective update value from a previous accumulated update value.

The neural network apparatus may continuously accumulate the individual update values in the accumulated update value of the accumulation buffer after updating the weight and may repeat the processes described above. The method of processing parameters in the neural network according to the present disclosure may accumulate individual update values much smaller than the value of the weight to a magnitude that may affect the weight and may generally apply the individual update values to the weight, instead of disregarding or discarding the individual update values, thereby minimizing a rounding error in the neural network that uses the low-precision number system. Also, the method of processing parameters in the neural network according to the present disclosure may minimize a precision loss by continuously maintaining a remaining part of the accumulated update value in the accumulation buffer after updating the accumulated update value to the weight.

Meanwhile, the neural network apparatus may determine whether the accumulated update value is equal to or greater than the threshold value at each predetermined update period. The update period may be previously determined by a user and may be determined by the neural network apparatus. Also, the update period may be set to any appropriate period to determine whether the accumulated update value is equal to or greater than the threshold value.

If the individual update values are relatively small, the update period may be determined to be relatively long. If the individual update values are relatively great, the update period may be determined to be relatively short. For example, the update period may be set to be repeated each time the individual update value is calculated, and may be set to be repeated each time the individual update values are calculated ten times but is not limited thereto. For example, the update period may be set to a period corresponding to n-times learning of the neural network.

The neural network apparatus may continue to accumulate the individual update values calculated to a next update period in the accumulation buffer if the accumulated update value is smaller than the threshold value. The neural network apparatus may again determine whether the accumulated update value is equal to or greater than the threshold value in the next update period and update the weight by using the accumulated update value if the accumulated update value is equal to or greater than the threshold value.

4 FIG. An overhead of the neural network may be reduced since the accumulated update value is determined to be equal to or greater than the threshold value in a predetermined period and the parameters used in the neural network apparatus are periodically updated. Referring to, an example of a method of periodically determining whether the accumulated update value is equal to or greater than the threshold value and updating the weight will be described below.

4 FIG. is a flowchart illustrating another example of a method of processing parameters in a neural network according to some embodiments.

4 FIG. 410 Referring to, in operation, time t may be initialized to 0, and an update period update_period and a threshold value threshold may be previously determined. The time t may have a time at which an individual update value is calculated as a basic unit. For example, t=4 may mean a time at which four individual update values are calculated but is not limited thereto. The update period may be 1, 4, or any suitable value equal to or greater than 1. The threshold value may be a value for determining whether the accumulated update value has been accumulated to a magnitude that may affect a weight.

420 In operation, a neural network apparatus may calculate individual update values for weights assigned to connection relationships between nodes included in the neural network. The individual update values may be gradient values for updating the weights by using a gradient descent algorithm.

430 440 420 In operation, the neural network apparatus may accumulate the individual update values in an accumulation buffer and add 1 at the time t. In operation, the neural network apparatus may determine whether the time t corresponds to the update period. If the time t does not correspond to the update period, the neural network apparatus may return to operationand calculate the individual update values again.

450 450 420 If the time t corresponds to the update period, the neural network apparatus may perform operation. In operation, the neural network apparatus may determine whether the accumulated update value of the accumulation buffer is equal to or greater than the threshold value. If the accumulated update value is smaller than the threshold value, the neural network apparatus may return to operationand calculate the individual update values again. At this time, the time t may be initialized to 0, and the above-described operations may be repeated for a next update period.

460 460 If the accumulated update value is equal to or greater than the threshold value, the neural network apparatus may perform operation. In operation, the neural network apparatus may update the weight by using the accumulated update value. For example, the neural network apparatus may update the weight by adding an effective update value according to Equation 4 above to the weight. Also, the neural network apparatus may subtract the effective update value from the accumulated update value of the accumulation buffer when the weight is updated.

5 FIG. is a diagram illustrating an example of a neural network that uses fixed point parameters according to some embodiments.

5 FIG. 50 510 520 530 540 55 515 535 50 540 540 530 Referring to, in the neural network that uses fixed point parameters, a weightmay be a fixed point value including a first sign bit, a first integer part, a first fractional part, and a first decimal point; and an accumulated update valuemay be a fixed point value including a second sign bit, a second fractional part, and a second decimal point. According to some embodiments, the weightmay be a dynamic fixed point value at which the first decimal pointis movable. Since the first decimal pointmoves, the number of bits (length) of the first fractional partmay be adjusted.

55 540 55 50 55 535 540 8 8 FIGS.A andB The accumulated update valuemay not include an integer part. For example, when the first decimal pointand the second decimal point are the same, since the accumulated update valuemay represent only values significantly smaller than the weight, the accumulated update valuemay not include the integer part but may include a fractional part (i.e. the second fractional part). Meanwhile, the second decimal point may be the same as the first decimal point, but may be different. The second decimal point may be adjusted depending on a situation. Hereinafter, a process of adjusting a position of the second decimal point will be described in detail with reference to.

50 55 The weightmay be a fixed point value of m bits (m is an arbitrary natural number). The accumulated update valuemay be a fixed point value of n bits (n is any natural number). m and n may be the same value or may be different values. In a neural network that uses a low-precision number system, each of m and n may have a value of 8 or 16, but is not limited thereto.

55 50 50 55 50 55 50 55 50 50 55 The accumulated update valuemay have a relatively small value compared to the weight. Thus, a value corresponding to each of the bits representing the weightmay be substantially greater than a value corresponding to each of the bits representing the accumulated update value. However, the value corresponding to any one of the bits representing the weightmay be equal to the value corresponding to any one of the bits representing the accumulated update value. To update the weight, since a part of the accumulated update valueneeds to be summed to the weight, some of lower bits of the weightmay overlap with some of upper bits of the accumulated update value.

550 50 55 When bitsoverlapping between the weightand the accumulated update valueare k bits (k is an arbitrary natural number), the neural network may obtain the substantially same effect as using fixed point data of (m+n−2−k) bits.

550 50 55 560 560 55 50 560 50 550 560 55 550 50 50 50 530 The bitsoverlapping between the weightand the accumulated update valuemay be determined with respect to a threshold value. The threshold valuemay be a value for determining whether the accumulated update valuehas been accumulated to a magnitude that may affect the weight. For example, the threshold valuemay be a value corresponding to a least significant effective digit of the weight. The overlapping bitsmay be bits representing a value equal to or greater than the threshold valueamong the bits representing the accumulated update value. The overlapping bitsmay overlap with some of the lower bits of the weightand may therefore be summed with the weight. Meanwhile, the lower bits of the weightmay represent the first fractional part.

50 55 55 50 550 530 50 535 55 50 550 The neural network apparatus may update the weightby using the accumulated update valueif the individual update values are accumulated and the accumulated update valueincreases to the magnitude that may affect the weight. For example, the neural network apparatus may add a value of at least one bitthat overlaps the first fractional partof the weightamong bits representing the second fractional partof the accumulated update valueto the weight. The value represented by the at least one bitthat overlaps may correspond to an effective update value.

550 530 50 535 55 50 6 FIG. Meanwhile, a padding method may be applied in a process of adding the value represented by the at least one bitthat overlaps the first fractional partof the weightamong the bits representing the second fractional partof the accumulated update valueto the weight. Hereinafter, with reference to, a process of adding an effective update value to a weight will be described in detail.

6 FIG. 614 630 is a diagram illustrating an example of a process of adding an effective update valueto a weightaccording to some embodiments.

6 FIG. 610 610 610 610 614 614 612 Referring to, an accumulated update valueis shown. The accumulated update valuemay be an accumulated value of individual update values. The accumulated update valuemay also be an n-bit fixed point value. Some of the bits representing the accumulated update valuemay be determined as the effective update value. The effective update valuemay correspond to bits excluding a sign bitamong bits located in an upper direction with respect to a threshold value.

614 630 620 612 616 614 616 612 620 630 620 630 620 630 612 6 FIG. Meanwhile, the effective update valuemay be padded so as to be added to the weightthat is an m-bit fixed point value. A padded effective update valuemay include the sign bit, a padding part, and the effective update value, as shown in. The padding partmay be filled with bits representing the same values as the sign bit. The padded effective update valuemay be summed with the weightsince the padded effective update valueis the m-bit fixed point value that is the same as the weight. For example, the padded effective update valuemay be added to or subtracted from the weightin accordance with the sign bit.

5 FIG. 550 50 550 515 550 50 55 55 550 550 515 Referring back to, the neural network apparatus may add the value represented by the at least one bitthat overlaps to the weightand then initialize the value of the at least one bitthat overlaps to the second sign bit. Since the value represented by the at least one bitthat overlaps is added to the weight, the accumulated update valuemust be subtracted by that value. The accumulated update valuemay be subtracted by the value represented by the at least one bitthat overlaps since the value of the at least one bitthat overlaps is initialized to the same value as the second sign bit.

55 550 530 50 55 55 550 530 50 50 55 55 55 515 55 55 550 55 550 −2 −2 −3 −3 −3 −3 For example, assume that the accumulated update valuewhich is a positive number is 0100011 (excluding sign bits) and the at least one bitthat overlaps the first fractional partof the weightis 01 that are 2 upper bits of the accumulated update value. Also, assume that a most significant bit of the accumulated update valuerepresents a 2digit. Since 0×2+1×2=2, which is the value of 01 represented by the at least one bitthat overlaps the first fractional partof the weight, is added to the weight, 2must be subtracted from the accumulated update value. If 2is subtracted from the accumulated update value, the accumulated update valuebecomes 0000011. Since the second sign bitof the accumulated update valueis 0 (i.e., since the accumulated update valueis the positive value), the at least one bitthat overlaps is initialized to 0. Unlike the preceding example, when the accumulated update valueis a negative value, the value of the at least one bitthat overlaps may be initialized to 1 which is the same value as the sign bit.

7 FIG. 5 FIG. is a flowchart illustrating an example of a method of processing parameters in a neural network that uses fixed point parameters according to some embodiments. The neural network may be a neural network that uses fixed point parameters as described in.

7 FIG. 710 720 Referring to, in operation, the time t may be initialized to 0, and a monitoring period, an update period, and a threshold value may be previously determined. In operation, a neural network apparatus may calculate individual update values for weights assigned to connection relationships between nodes included in the neural network.

730 740 745 In operation, the neural network apparatus may accumulate the individual update values in an accumulation buffer and add 1 at the time t. In operation, the neural network apparatus may determine whether the time t corresponds to the monitoring period. If the time t corresponds to the monitoring period, the neural network apparatus may perform operation.

745 In operation, the neural network apparatus may monitor an accumulated update value or the individual update values and adjust a position of a second decimal point of the accumulated update value based on a result of monitoring. Since the position of the second decimal point of the accumulated update value is adjusted, the number of bits overlapping between the weight and the accumulated update value may be adjusted and the bit number (length) of a fractional part of the accumulated update value may be adjusted.

The neural network apparatus may adjust the position of the second decimal point to minimize a precision loss or reduce an overhead based on a relative magnitude of the individual update values and the weights.

For example, the neural network apparatus may move the position of the second decimal point in a lower bit direction of the accumulated update value if an average value of individual update values is smaller than 1/a times the threshold value (a is an arbitrary natural number).

That the average value of the individual update values is smaller than 1/a times the threshold value means that the individual update values have significantly small values. Since the position of the second decimal point of the accumulated update value moves in the lower bit direction, the accumulated update value may represent a smaller value, and the individual update values may be more accurately reflected to the accumulated update value. The precision loss of the neural network may be minimized since the weight is updated based on the accumulated update value precisely reflecting the individual update values.

Also, when the average value of the individual update values is greater than 1/a times the threshold value, the neural network apparatus may move the position of the second decimal point in an upper bit direction of the accumulated update value.

8 8 FIGS.A andB That the average value of the individual update values is greater than 1/a times the threshold value means that the individual update values have relatively large values. The bits having a magnitude that may be disregarded among the bits representing the accumulated update value may be omitted since the position of the second decimal point of the accumulated update value moves in the upper bit direction. Thus, the overhead of the neural network may be reduced. However, the foregoing description is merely an example, and the neural network may adjust the position of the second decimal point or the bit number (length) of the second fractional part based on various criteria. Hereinafter, a process of adjusting a decimal point position of the accumulated update value will be described in detail with reference to.

8 FIG.A 8 FIG.B is a diagram illustrating an example of a process of adjusting a position of a decimal point of an accumulated update value according to some embodiments, andis a diagram illustrating another example of a process of adjusting a position of a decimal point of an accumulated update value according to some embodiments.

8 FIG.A 82 830 820 82 810 820 82 82 82 82 Referring to, a monitored accumulated update valueis shown where a currently accumulated valueand an individual update valueof the monitored accumulated update valuemay have considerably small magnitudes compared to a threshold value. The individual update valuemay not be accurately reflected to the accumulated update valuebecause there is a limit to a range of a value that the monitored accumulated update valuemay represent. Accordingly, a neural network may move the position of the decimal point of the accumulated update valuein a lower bit direction of the accumulated update value.

83 830 820 83 An accumulated update valuehaving the moved decimal point may more precisely reflect the currently accumulated valueand the individual update value. Meanwhile, since the decimal point moves, bits corresponding to a value that the accumulated update valuemay additionally represent may be filled with 0 or an arbitrary number.

8 FIG.B 84 840 84 840 810 850 810 84 84 84 84 Referring to, a monitored accumulated update valueis shown where an individual update valueof the monitored accumulated update valuehas a relatively large value even though the individual update valueis smaller than the threshold value, and a currently accumulated valuehas a value greater than the threshold value. A neural network apparatus may move the position of the decimal point of the accumulated update valuein an upper bit of the accumulated update valuein order to omit bits having a magnitude that may be disregarded among bits representing the accumulated update value, thereby preventing the accumulated update valuefrom exceeding a value that may be expressed by m bits during an update period.

7 FIG. 745 720 Returning back to, in operation, the neural network apparatus may adjust the position of the second decimal point of the accumulated update value, and then return to operationto calculate the individual update values again.

740 750 750 720 Meanwhile, if it is determined in operationthat the time t does not correspond to the monitoring period, the neural network apparatus may perform operation. In operation, the neural network apparatus may determine whether the time t corresponds to the update period. If the time t does not correspond to the update period, the neural network apparatus may return to operationand again calculate the individual update values. At this time, the time t may be initialized to 0, and the above-described operations may be repeated during a next update period.

760 760 720 If the time t corresponds to the update period, the neural network apparatus may perform operation. In operation, the neural network apparatus may determine whether the accumulated update value of the accumulation buffer is equal to or greater than the threshold value. If the accumulated update value is smaller than the threshold value, the neural network apparatus may return to operationto calculate the individual update values again.

770 770 If the accumulated update value is equal to or greater than the threshold value, then the neural network apparatus may perform operation. In operation, the neural network apparatus may update the weight by using the accumulated update value. For example, the neural network apparatus may update the weight by adding the effective update value according to Equation 4 above to the weight. Also, the neural network apparatus may subtract the effective update value from the accumulated update value of the accumulation buffer since the weight is updated. The neural network apparatus may adjust the position of the decimal point of the weight or the bit number (length) of the fractional part depending on a situation.

9 FIG. is a diagram illustrating an example of a neural network that uses floating point parameters according to some embodiments.

9 FIG. 90 910 920 930 940 95 915 925 935 945 Referring to, in the neural network that uses floating point parameters, a weightmay be a floating point value including a first sign bit, a first exponent part, a first mantissa part, and a first bias; and an accumulated update valuemay be a floating point value including a second sign bit, a second exponent part, a second mantissa part, and a second bias.

90 95 The weightmay be an m-bit (where m is a natural number) floating point value. The accumulated update valuemay be an n-bit (where n is any natural number) floating point value. m and n may be the same value or may be different values. In the neural network that uses a low-precision number system, each of m and n may have a value of 8 or 16, but is not limited thereto.

95 90 95 95 The accumulated update valuemay be an accumulated value of individual update values for the weight. However, due to a limit of the floating point value with a limited number of bits, the accumulated update valuemay be a value in which only an effective value of each of the individual update values included in an effective number range of the accumulated update valueis accumulated.

95 90 90 95 95 90 Also, due to the limit of the floating point value with the limited number of bits, only the effective value of the accumulated update valueincluded in the effective number range of the weightmay be added to the weight. The effective value of the accumulated update valuemay mean a part of the accumulated update valuethat may be added to an effective digit of the weight.

90 95 95 95 90 Since the weightis updated by the effective value of the accumulated update value, the accumulated update valuemay be subtracted by the updated part. Due to the limit of the floating point value with the limited number of bits, the accumulated update valuemay be different from a value obtained by subtracting the weightfrom the updated weight.

940 90 945 95 Meanwhile, the floating point value may be changed dynamically since a bias is changed. Therefore, even if an exponent part of the floating point value is the same, a relatively small value may be expressed according to the bias. A relative difference between the first biasof the weightand the second biasof the accumulated update valuemay be appropriately adjusted depending on a situation.

10 FIG. 9 FIG. is a flowchart illustrating an example of a method of processing parameters in a neural network that uses floating point parameters according to some embodiments. The neural network may be a neural network that uses floating point parameters as described in.

10 FIG. 1010 1020 Referring to, in operation, the time t is initialized to 0, and a monitoring period, an update period, and a threshold value may be previously determined. In operation, a neural network apparatus may calculate individual update values for weights assigned to connection relationships between nodes included in the neural network.

1030 1040 1045 In operation, the neural network apparatus may accumulate the individual update values in an accumulation buffer and add 1 to the time t. In operation, the neural network apparatus may determine whether the time t corresponds to the monitoring period. If the time t corresponds to the monitoring period, the neural network apparatus may perform operation.

1045 In operation, the neural network apparatus may monitor an accumulated update value or individual update values and adjust a second bias of the accumulated update value based on a result of monitoring. A range of a value that the accumulated update value may represent may be adjusted since the second bias of the accumulated update value is adjusted.

The neural network apparatus may adjust the second bias of the accumulated update value to minimize a precision loss or reduce an overhead based on a relative magnitude of the accumulated update value and the threshold value.

For example, the neural network apparatus may increase the second bias if a second exponent of the accumulated update value exceeds the threshold value. That the second exponent of the accumulated update value exceeds the threshold value may mean that the accumulated update value has a relatively large value. Since the second bias of the accumulated update value increases, bits having a magnitude that may be disregarded among bits representing the accumulated update value may be omitted. Thus, the overhead of the neural network may be reduced.

The neural network apparatus may also reduce the second bias if the accumulated update value is smaller than 1/b times the threshold value (b is an arbitrary natural number). That the accumulated update value is smaller than 1/b times the threshold value means that the individual update values have significantly small values. Since the second bias of the accumulated update value is reduced, the accumulated update value may express a smaller value, and the individual update values may be more precisely reflected to the accumulated update value. The precision loss of the neural network may be minimized since the weight is updated by the accumulated update value that precisely reflects the individual update values.

1045 1020 The neural network apparatus may adjust a position of the second bias of the accumulated update value in operationand then return to operationto calculate the individual update values again.

1040 1050 1050 1020 Meanwhile, if it is determined in operationthat the time t does not correspond to the monitoring period, the neural network apparatus may perform operation. In operation, the neural network apparatus may determine whether the time t corresponds to the update period. If the time t does not correspond to the update period, the neural network apparatus may return to operationand calculate the individual update values again. At this time, the time t may be initialized to 0, and the above-described operations may be repeated during a next update period.

1060 1060 1020 If the time t corresponds to the update period, the neural network apparatus may perform operation. In operation, the neural network apparatus may determine whether the accumulated update value of the accumulation buffer is equal to or greater than the threshold value. If the accumulated update value is smaller than the threshold value, the neural network apparatus may return to operationand calculate the individual update values again.

1070 1070 If the accumulated update value is equal to or greater than the threshold, the neural network apparatus may perform operation. In operation, the neural network apparatus may update the weight by using the accumulated update value. For example, the neural network apparatus may update the weight by adding an effective update value according to Equation 4 above to the weight. Also, the neural network apparatus may subtract the effective update value from the accumulated update value of the accumulation buffer since the weight is updated. The neural network apparatus may adjust a position of a decimal point of the weight or the bit number (length) of a fractional part depending on a situation.

2 4 7 FIGS.,, and Meanwhile, the method of processing parameters in the neural network shown inmay be recorded on a computer-readable recording medium having recorded thereon one or more programs including instructions for performing the method. Examples of the computer-readable recording medium may include magnetic media such as a hard disk, a floppy disk, and a magnetic tape, optical media such as CD-ROM and digital video disc (DVD), magneto-optical media such as a floptical disk, and a hardware apparatus, such as the ROM, the RAM, and the flash memory, specifically configured to store and execute program instructions. Examples of the program instructions may include not only machine language codes generated by a compiler but also high-level language codes executable by a computer by using an interpreter or the like.

11 FIG. 11 is a block diagram showing a configuration of a neural network apparatusaccording to some embodiments.

11 FIG. 11 FIG. 11 FIG. 12 FIG. 11 1110 1120 11 11 11 Referring to, the neural network apparatusmay include processorand memory. Only components related to the present embodiment are shown in the neural network apparatusshown in. Therefore, it is understood after an understanding of the disclosure of this application that the neural network apparatusmay further include components other than the components shown in. For example, the neural network apparatusmay further include a sensor module and a communication module such as demonstrated in.

11 11 11 2 10 FIGS.through 2 4 7 10 FIGS.,,and 2 4 7 10 FIGS.,,and 11 FIG. Meanwhile, the neural network apparatusmay correspond to the neural network apparatus of. The neural network apparatusmay also time-serially process the method shown in. Therefore, it will be understood that the method described with reference tomay be performed by the neural network apparatusofeven if omitted below.

11 11 The neural network apparatusmay be included in at least one of a home appliance, a mobile computing device, and a server, or may be connected to at least one of the home appliance, the mobile computing device, and the server by wired or wirelessly. The neural network apparatusmay also be implemented by a combination of a device and a server. A device that receives a user input and a server that trains a neural network may exist separately, or may be included in the same device, or in separate servers, and thus inference may be performed through communication between the device and the server when that server thereby implements the trained neural network. Inference may be performed in a single device, and is not limited to the above example.

1110 1110 1110 The processoris representative of one or more processors. For example, the processormay be implemented as an array of logic gates, and may be implemented as a combination of a general purpose microprocessor and a memory in which a program executable in the microprocessor is stored. For example, the processormay be a central processing unit (CPU), a graphics processing unit (GPU), a hardware accelerator, or the like.

1110 11 1110 11 1120 11 1110 11 1120 The processormay generally operate in order to control the neural network apparatus. For example, the processormay generally control the neural network apparatusby executing programs stored in the memoryincluded in the neural network apparatus. The processormay also perform functions of the neural network apparatusin parallel by executing the programs stored in the memory.

1110 1110 1110 11 2 10 FIGS.- The processormay perform a function of each of the nodes included in layers of the neural network. For example, the processormay perform a neural network operation or a matrix operation by using activation values and weights. The processormay also perform a function of the neural network apparatusdescribed in.

1110 Specifically, the processormay calculate individual update values for weights assigned to connection relationships between the nodes included in the neural network. The weight may be a floating point value or a fixed point value.

1110 1120 Also, the processormay accumulate the individual update values in an accumulation buffer. The accumulation buffer may be dedicated hardware for accumulating the individual update values and may be a part of the memoryallocated to accumulate the individual update values. The individual update values may be accumulated in the accumulation buffer to be an accumulated update value. The accumulation buffer may store the accumulated update value.

1110 The processormay determine whether the accumulated update value of the accumulation buffer is equal to or greater than a threshold value. The threshold value may be a value for determining whether the accumulated update value has been accumulated to a magnitude that may affect the weight. For example, the threshold value may be a value corresponding to a least significant effective digit of the weight.

1110 1110 The least one processormay update the weights by using the accumulated update value if the accumulated update value is equal to or greater than the threshold value. If the accumulated update value is equal to or greater than the threshold value, since the accumulated update value has the magnitude that may affect the weight, the processormay update the weight by using the accumulated update value.

1110 1110 The processormay determine a part of the accumulated update value that may be added to an effective digit of the weight as an effective update value. The processormay add the effective update value to the weight and subtract the effective update value from the accumulated update value of the accumulation buffer.

1110 The processormay accumulate individual update values much smaller than a value of the weight to a magnitude that may affect the weight and may generally apply the individual update values to the weight, instead of disregarding or discarding the individual update values, thereby minimizing a rounding error in the neural network that uses a low-precision number system.

1110 1110 1110 Meanwhile, the processormay determine whether the accumulated update value is equal to or greater than the threshold value at each predetermined update period. The update period may be previously determined by a user and may be determined by the processor. The processormay continue to accumulate the individual update values calculated to a next update period in the accumulation buffer if the accumulated update value is smaller than the threshold value.

1110 The processormay again determine whether the accumulated update value is equal to or greater than the threshold value in the next update period and update the weight by using the accumulated update value if the accumulated update value is equal to or greater than the threshold value.

11 It is determined whether the accumulated update value is equal to or greater than the threshold value according to a certain period and a parameter used in the neural network apparatusis periodically updated, and thus an overhead of the neural network may be reduced.

1110 When the weight is a fixed point value including a first sign bit, a first integer part, a first fractional part and a first decimal point, and the accumulated update value is a fixed point value including a second sign bit, a second fractional part and a second decimal point, the processormay add to the weight a value represented by at least one bit that overlaps the first fractional part of the weight among bits representing the second fractional part of the accumulated update value, and initialize the value represented by the at least one bit that overlaps to the same value as the second sign bit.

1110 1110 Also, the processormay monitor the accumulated update value and the individual update values and adjust a position of the second decimal point of the accumulated update value based on a result of monitoring. For example, the processormay move the position of the second decimal point in a lower bit direction of the accumulated update value if an average value of the individual update values is smaller than 1/a times the threshold value, and may move the position of the second decimal point in an upper bit of the accumulated update value if the average value is greater than 1/a times the threshold value.

1110 Meanwhile, if the weight is a floating point value including a first sign bit, a first exponent part, a first mantissa part, and a first bias, and the accumulated update value is a floating point value including a second sign bit, a second exponent part, and a second bias, the processormay add to the weight an effective value of the accumulated update value included in an effective number range of the weight.

1110 1110 Also, the processormay monitor the accumulated update value and adjust the second bias of the accumulated update value based on a result of monitoring. For example, the processormay increase the second bias if the second exponent of the accumulated update value exceeds the threshold value, and may reduce the second bias if the accumulated update value is smaller than 1/b times the threshold value.

1120 11 1120 11 1120 11 1120 The memoryis hardware storing various pieces of data processed in the neural network apparatus. For example, the memorymay store data processed and data to be processed by the neural network apparatus. The memorymay also store applications, drivers, etc., to be driven by the neural network apparatus. The memorymay be a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a dynamic random access memory (DRAM) ROM, Blu-Ray® or other optical disk storage, hard disk drive (HDD), solid state drive (SSD), or flash memory.

1120 1110 11 1110 The memorymay further include a built-in memory shared by the processorincluded in the neural network apparatusor a plurality of built-in memories supporting the processor.

12 FIG. is a diagram illustrating an example of an electronic system or device configured to implement a neural network inference operation.

12 FIG. 11 FIG. 11 FIG. 1200 1210 1220 1225 1230 1250 1260 1210 1220 1230 1250 1260 1240 1200 1220 1110 1230 1120 1225 1220 1220 1220 1225 1225 1230 1230 1200 1220 Referring to, an electronic system or deviceincludes a sensor, a processor, a local memory, a memory, a display, and a user interface (UI). The sensor, the processor,, the memory, the display, and the UIcommunicate with each other via a bus. The electronic system or devicemay correspond to any one or more or all of the above neural network (NN) processing apparatuses and implement any one or more or all of the above NN processing processes or methods. As a non-limiting example, the processormay correspond to the processorof, and/or the memorymay correspond to the memoryof. The local memorymay correspond to any of the above described temporary or local buffers/memories. In an example, the local buffers/memories may be memories of the processoror buffers/memories directly connected to the processor, e.g., configured for rapidly transferring data to/from the processorand the local memory, noting that alternatives are also available. The local memorymay further be allocated to temporarily store activations of a particular layer of the NN, or all layers of the NN, the ultimate output results of which may be stored in the memoryand/or respectively used for inputs to a next layer. In an example, except for purposes of an input to a next layer, the activations of each layer may be discarded and only final layer(s) output results of the NN stored to the memoryor used for another process, such as in cooperation with an unlocking and corresponding display operation of a mobile phone as the electronic system or device. As also noted, the processormay represent one or more processors that are configured as any or any combination of the above NN processing apparatuses, recognition apparatuses, rejection apparatuses, and/or verification apparatuses, as non-limiting examples.

1210 1210 1210 1220 1230 1210 The sensorincludes, for example, a microphone and/or an image sensor or camera to sense video data and audio data to recognize, reject, or verify an object, for example. The sensorsenses an image using a well-known scheme, for example, a scheme of converting an optical image to an electronic signal. An output of the sensoris transferred to the processoror the memory, and output of the sensormay also be transferred directly to, or operate as, an input layer of any of the NNs discussed herein.

1220 1220 1250 1200 1260 1220 1220 1220 1225 1220 1230 1 11 FIGS.through 1 11 FIGS.through The processormay be configured to perform one or more or all processes described with reference to. For example, to perform an object detection, recognition, rejection, and/or verification operations, the processormay detect, recognize, reject, and/or verify one or more objects in the input data using the neural network trained throughout the operations described above with respect to, for example. The result of any of the recognition, rejection, or verification operations may be output through the display. In addition, user adjustments or selective operations of the electronic devicemay be provided by UI, which may include a touch screen or other input device/system. In an example, the processormay be a graphics processor unit (GPU), a central processing unit (CPU), a reconfigurable processor, an application processor (AP), a neural processing unit (NPU), tensor processing unit (TPU), a HW accelerator, or otherwise an neuromorphic chip, or have any other type of multi- or single-processor configuration. In an example, the processoris further representative of multiple processors, e.g., with at least one processor being configured as such specialized neural network implementing processors and another processor configured to perform the other operations of the electronic device. Such specialized neural network implementing processors may also be configured in a secure area, with the corresponding processorand local memorybeing securely separated from other operations of other corresponding processorsand the memory, for example.

1 11 FIGS.through 1230 1220 1220 1200 1220 1200 1200 In addition to operations of one or more of the NN processing apparatuses and/or operations described in, as noted above, the memorymay further store instructions which, when executed by processor, cause the processorto perform additional operations, functions, and controls of the electronic system or device, such as a user interface of the electronic system based on inference operations performed by the processorusing the trained neural network. The electronic system or devicemay be connected to an external device, for example, a personal computer (PC) or a network, via an input/output device of the electronic system, to exchange data with the external device. The electronic system or devicemay be various electronic devices, as only non-limiting examples, a mobile device, for example, a mobile telephone, a smartphone, a personal digital assistant (PDA), a tablet computer or a laptop computer, a computing device, for example, a PC, a tablet computer or a netbook computer, an electronic product, for example, a television (TV), a smart TV, or a security device for gate control.

11 1110 1120 1200 1240 1220 1225 1210 1230 1250 1260 1 12 FIGS.through The neural network apparatus, the processor, the memory, electronic system or device, bus, processor, local memory, sensor, memory, display, and user interfaceinthat perform the operations described in this application are implemented by hardware components configured to perform the operations described in this application that are performed by the hardware components. Examples of hardware components that may be used to perform the operations described in this application where appropriate include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware, for example, by one or more processors or computers. A processor or computer may be implemented by one or more processing elements, such as an array of logic gates, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field-programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices that is configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes, or is connected to, one or more memories storing instructions or software that are executed by the processor or computer. Hardware components implemented by a processor or computer may execute instructions or software, such as an operating system (OS) and one or more software applications that run on the OS, to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to execution of the instructions or software. For simplicity, the singular term “processor” or “computer” may be used in the description of the examples described in this application, but in other examples multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. A hardware component may have any one or more of different processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SISD) multiprocessing, single-instruction multiple-data (SIMD) multiprocessing, multiple-instruction single-data (MISD) multiprocessing, and multiple-instruction multiple-data (MIMD) multiprocessing.

1 12 FIGS.through The methods illustrated inthat perform the operations described in this application are performed by computing hardware, for example, by one or more processors or computers, implemented as described above executing instructions or software to perform the operations described in this application that are performed by the methods. For example, a single operation or two or more operations may be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be performed by one or more processors, or a processor and a controller, and one or more other operations may be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may perform a single operation, or two or more operations.

Instructions or software to control computing hardware, for example, one or more processors or computers, to implement the hardware components and perform the methods as described above may be written as computer programs, code segments, instructions or any combination thereof, for individually or collectively instructing or configuring the one or more processors or computers to operate as a machine or special-purpose computer to perform the operations that are performed by the hardware components and the methods as described above. In one example, the instructions or software include machine code that is directly executed by the one or more processors or computers, such as machine code produced by a compiler. In another example, the instructions or software includes higher-level code that is executed by the one or more processors or computer using an interpreter. The instructions or software may be written using any programming language based on the block diagrams and the flow charts illustrated in the drawings and the corresponding descriptions in the specification, which disclose algorithms for performing the operations that are performed by the hardware components and the methods as described above.

The instructions or software to control computing hardware, for example, one or more processors or computers, to implement the hardware components and perform the methods as described above, and any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of a non-transitory computer-readable storage medium include read-only memory (ROM), random-access memory (RAM), flash memory, CD-ROMs, CD-Rs, CD+Rs, CD-RWs, CD+RWs, DVD-ROMs, DVD-Rs, DVD+Rs, DVD-RWs, DVD+RWs, DVD-RAMs, BD-ROMs, BD-Rs, BD-R LTHs, BD-REs, magnetic tapes, floppy disks, magneto-optical data storage devices, optical data storage devices, hard disks, solid-state disks, and any other device that is configured to store the instructions or software and any associated data, data files, and data structures in a non-transitory manner and provide the instructions or software and any associated data, data files, and data structures to one or more processors or computers so that the one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed over network-coupled computer systems so that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed fashion by the one or more processors or computers.

While this disclosure includes specific examples, it will be apparent after an understanding of the disclosure of this application that various changes in form and details may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered in a descriptive sense only, and not for purposes of limitation. Descriptions of features or aspects in each example are to be considered as being applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and/or if components in a described system, architecture, device, or circuit are combined in a different manner, and/or replaced or supplemented by other components or their equivalents. Therefore, the scope of the disclosure is defined not by the detailed description, but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents are to be construed as being included in the disclosure.

A neural network apparatus may perform many operations on complex input data. A technique of efficiently processing neural network operations is desired such that the neural network apparatus may analyze high-definition input data and extract information in real time. Thus, methods and apparatuses that reduce the computational amount for processing complex data input to the neural network while minimizing precision loss have been disclosed herein.

It should be understood that the embodiments described herein should be considered in a descriptive sense only and not for purposes of limitation. Descriptions of features or aspects within each embodiment should typically be considered as available for other similar features or aspects in other embodiments.

While one or more embodiments have been described with reference to the figures, it will be understood after an understanding of the disclosure of this application that various changes in form and details may be made therein without departing from the spirit and scope as defined by the following claims.