Apparatus and method for training low bit-precision deep neural network

This application claims the benefit of Korean Patent Application No. 10-2021-0002846, filed Jan. 8, 2021, which is hereby incorporated by reference in its entirety into this application.

The present invention relates generally to an apparatus and method for training a Deep Neural network (DNN) and, and more particularly to an apparatus and method for training a deep neural network, which use a new number system for training a deep neural network at low bit precision and an algorithm for automatically searching for a required precision of each layer.

A deep neural network (hereinafter referred to as a ‘DNN’), which is one type of machine-learning technology, denotes an artificial neural network including multiple hidden layers between an input layer and an output layer, and has been utilized in various fields such as object recognition, as well as speech recognition and translation.

In order to minimize the loss of accuracy, training of a deep neural network adopts a floating-point representation system having bit precision higher than that of 16 or more bits. Thus, the training of a deep neural network is dependent on a cloud server having high computing power.

In this case, since data of a user must be transmitted to the server, invasion of privacy and a long delay time may occur, and thus a problem may arise in that only limited functionality may be provided to the user.

In the past, in order to solve this problem, a tendency has arisen for the training of a neural network to be performed by the server and inference to be directly supported by a mobile device owned by each user. That is, low-bit quantization algorithms are applied to a neural-network inference process which has conventionally used an existing 32-bit floating-point representation or 64-bit floating-point representation. Accordingly, the development of low-power and high-speed mobile deep-neural-network inference processors has been promoted while bit precision for inference has decreased below 8 bits, and a deep-learning application, which has to date been dependent on a server, may be independently performed by a mobile device.

However, in this conventional method, among three stages of training (i.e., Feed-Forward: FF, error propagation: EP, and weight gradient update: WG), only the feed-forward stage can be accelerated by the mobile device, and the overall deep-neural-network training process still depends a lot on the server. The reason for this is the high-bit precision required by deep-neural-network training.

is a diagram for explaining the reason why typical deep-neural-network training requires bit precision higher than that of a feed-forward (FF) stage. Referring to, it can be seen that input activation (IA) values used in feed-forward have a distribution range as narrow as −10 Log(magnitude), whereas error values used in training have a distribution range of −20 Log(magnitude), evidently being wider than that of IA.

Due thereto, the inference (feed-forward) can sufficiently utilize low bit precision such as a 4-bit fixed-point representation, but training should use a floating-point system of 16 or more bits. In contrast, a floating-point system is characterized by having a number representation range wider than that of a fixed-point system because a number is represented to be divided into an exponent part and a fractional part. However, bit variability of a calculator using a floating point is deteriorated, thus making it difficult to minimize the amount of energy that is wasted depending on the optimal number of bits. Due thereto, there is a method for training a deep neural network using a fixed point, but high precision of 16 or more bits is still required for training even if a dynamic fixed point is utilized.

That is, existing deep-neural-network training requires high precision due to the wide distribution of errors appearing in the error propagation (EP) stage, and requires precision of at least 16 bits regardless of whether a floating-point system or a fixed-point system is adopted.

In this way, training of a deep neural network requiring precision of 16 bits or more not only requires a calculator that consumes large amounts of power, but also greatly increases memory access, and thus it is not suitable for use in mobile devices.

Also, the optimal precision required by a deep neural network varies greatly depending on the type of a data set, the type of a network, and the type and location of a layer. Accordingly, in order to find the optimal precision required by each deep neural network, the conventional technology mainly uses a method for retraining a deep neural network using low bit precision on the assumption that training of a deep neural network has been completed at high bit precision. When this method is used, the optimal number of bits may be determined only through repeated trial and error. Such a conventional method is not only disadvantageous in that high-bit training of a deep neural network is premised and retraining is required, but also problematic in that repeated trial and error must be undergone, thus requiring a long time to find the optimal number of bits.

Meanwhile, the importance of deep-neural-network training has gradually increased for distributed training, user optimization, adaptation to changing environments, etc., and thus there is presently required a method capable of supporting deep-neural-network training at low bit precision.

Accordingly, the present invention has been made keeping in mind the above problems occurring in the prior art, and an object of the present invention is to provide an apparatus and method for training a low-bit-precision deep neural network, which determine the number of integer bits of training data by adopting a dynamic fixed-point (DFXP) system and by introducing a Stochastic Dynamic FiXed-Point (SDFXP) system that supports both Stochastic Rounding (SR) and Stochastic Thresholding (ST), thus conveying an effect similar to the effect of training using a large number of bits even when using a small number of bits.

Another object of the present invention is to provide an apparatus and method for training a low-bit-precision deep neural network, which determine a similarity by comparing the result of a high-bit operation with the result of a low-bit operation during an operation in each of a feed-forward stage and an error propagation stage, and determine whether the precision of a corresponding layer is to be scaled based on the result of the determination, thus finding an optimal precision of each layer of the deep neural network.

A further object of the present invention is to provide an apparatus and method for training a low-bit-precision deep neural network, which can find the optimal precision of a corresponding layer during the progress of learning and perform a comparison between operations during only some initial iterations, among multiple iterations included in each epoch, thus minimizing the time required to scale the precision of each layer.

In accordance with an aspect of the present invention to accomplish the above objects, there is provided an apparatus for training a deep neural network including a hidden layer composed of n layers (where n is a natural number) between an input layer and an output layer, the apparatus including an input unit configured to receive training data required to train the deep neural network; and a training unit configured to train the deep neural network using the training data, wherein the training unit includes a training module configured to perform training using a preset first precision; a representation form determination module configured to determine a representation form for pieces of internal data generated during an operation procedure for the training and to determine a position of a decimal point of the pieces of internal data so that a permissible overflow bit in a dynamic fixed-point system varies randomly; and a layer-wise precision determination module configured to determine a precision of each layer during an operation in each of a feed-forward stage and an error propagation stage, which are performed for training, and to automatically change a precision of a corresponding layer based on a result of the determination.

In an embodiment, the representation form determination module may include a storage unit configured to store an overflow threshold, which is a maximum number of bits permitting the overflow; a random number generator configured to generate a random number between 0 and 1 in response to generation of the internal data; a multiplier configured to determine a permissible overflow range by multiplying the overflow threshold by the random number; and a decimal point position determination unit configured to determine the position of the decimal point of the pieces of internal data based on the permissible overflow range.

In an embodiment, the representation form determination module may further include a rounding unit configured to perform stochastic rounding on the internal data, and the decimal point position determination unit determines a position of a decimal point of internal data, to which the stochastic rounding is applied, based on the permissible overflow range.

In an embodiment, the layer-wise precision determination module may further perform a high-bit operation using a second precision higher than the first precision of each layer during the operation in each of the feed-forward stage and the error propagation stage, and thereafter determine the precision of the corresponding layer based on a similarity between a result of a low-bit operation using the first precision and a result of a high-bit operation using the second precision.

In an embodiment, the layer-wise precision determination module may store a first threshold value, which is preset as a reference for a similarity based on which whether the precision of each layer is to be changed is determined and a second threshold value, which is preset to a value greater than the first threshold value, and the layer-wise precision determination module may be configured to, when the similarity is less than or equal to the first threshold value, change the precision of the corresponding layer to a value higher than the first precision, when the similarity is equal to or greater than the second threshold value, change the precision of the corresponding layer to a value lower than the first precision, and when the similarity is between the first threshold value and the second threshold value, maintain the first precision.

In an embodiment, the layer-wise precision determination module may automatically change the precision of the corresponding layer, for all layers except the input layer and the output layer.

In an embodiment, the layer-wise precision determination module may automatically change the precision of each layer only during first m iterations, among multiple iterations included in each epoch, which is a training unit for each layer, and the training module may perform training during iterations ranging from an (m+1)-th iteration based on the precision of each layer automatically changed by the layer-wise precision determination module.

In an embodiment, the layer-wise precision determination module may configure a Finite-State Machine (FSM) based on results of the low-bit operation and the high-bit operation that are performed to automatically determine the precision of each layer during the first m iterations, and may change the precision of the corresponding layer based on the result of the configuration.

In an embodiment, the layer-wise precision determination module may be configured to, in the feed-forward stage, perform a high-bit operation by adding a bit corresponding to at least one of a feed-forward weight and an input activation value of the deep neural network, and in the error propagation stage, perform a high-bit operation by adding a bit corresponding to at least one of a backward weight and an error value of the deep neural network.

In accordance with another aspect of the present invention to accomplish the above objects, there is provided a method for training a deep neural network including a hidden layer composed of n layers (where n is a natural number) between an input layer and an output layer, the method including performing an input operation of receiving training data required to train the deep neural network; and performing a training operation of training the deep neural network using the training data so that the training is performed using first precision, which is preset for each layer, wherein performing the training operation may include performing a representation form determination operation of determining a representation form for pieces of internal data generated during an operation procedure for the training and determining a position of a decimal point of the pieces of internal data so that a permissible overflow bit in a dynamic fixed-point system varies randomly; and performing a layer-wise precision determination operation of determining a precision of each layer during an operation in each of a feed-forward stage and an error propagation stage, which are performed for training, and automatically changing a precision of a corresponding layer based on a result of the determination.

In an embodiment, performing the representation form determination operation may include performing an overflow threshold determination operation of determining an overflow threshold, which is a maximum number of bits permitting an overflow; performing a random number generation operation of generating a random number between 0 and 1 in response to generation of the internal data; performing a permissible overflow range determination operation of determining a permissible overflow range by multiplying the overflow threshold by the random number; and performing a decimal point position determination operation of determining the position of the decimal point of the pieces of internal data based on the permissible overflow range.

In an embodiment, performing the representation form determination operation may further include performing stochastic rounding on the internal data, and performing the decimal point position determination operation is configured to determine a position of a decimal point of internal data, to which the stochastic rounding is applied, based on the permissible overflow range.

In an embodiment, performing the layer-wise precision determination operation may include performing a high-bit operation using a second precision, which is a precision higher than the first precision, for each layer during the operation in each of the feed-forward stage and the error propagation stage; and determining a similarity between a result of the low-bit operation using the first precision and a result of the high-bit operation using the second precision, and after the precision of the corresponding layer is determined based on the determined similarity, the precision of the corresponding layer may be changed based on a result of the determination.

In an embodiment, performing the layer-wise precision determination operation may further include setting first and second threshold values, which are references for a similarity based on which whether the precision of each layer is to be changed is determined, the second threshold value being set to a value greater than the first threshold value, and performing the layer-wise precision determination operation is configured to, when the similarity is less than or equal to the first threshold value, change the precision of the corresponding layer to a value higher than the first precision, when the similarity is equal to or greater than the second threshold value, change the precision of the corresponding layer to a value lower than the first precision, and when the similarity is between the first threshold value and the second threshold value, maintain the first precision.

In an embodiment, performing the layer-wise precision determination operation may be configured to automatically determine the precision of the corresponding layer, for all layers except the input layer and the output layer.

In an embodiment, performing the layer-wise precision determination operation may be configured to automatically determine the precision of each layer only during first m iterations, among multiple iterations included in each epoch, which is a training unit for each layer, and performing the training operation may be configured to perform training during iterations ranging from an (m+1)-th iteration based on the precision of each layer automatically determined in the layer-wise precision determination operation.

In an embodiment, performing the layer-wise precision determination operation may further include a configuring a Finite-State Machine (FSM) based on results of the low-bit operation and the high-bit operation for automatically determining the precision of each layer during the first m iterations and the precision of the corresponding layer is changed based on a result of the configuration.

In an embodiment, performing the layer-wise precision determination operation may be configured to, in the feed-forward stage, perform a high-bit operation by adding a bit corresponding to at least one of a feed-forward weight and an input activation value of the deep neural network, and in the error propagation stage, perform a high-bit operation by adding a bit corresponding to at least one of a backward weight and an error value of the deep neural network.

Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. The present invention will be described in detail such that those skilled in the art to which the present invention pertains can easily practice the present invention. The present invention may be embodied in various different forms, and is not limited to the following embodiments. Meanwhile, in the drawings, parts irrelevant to the description of the invention will be omitted so as to clearly describe the present invention. It should be noted that the same or similar reference numerals are used to designate the same or similar components throughout the drawings. Descriptions of known configurations which allow those skilled in the art to easily understand the configurations will be omitted below.

In the specification and the accompanying claims, when a certain element is referred to as “comprising” or “including” a component, it does not preclude other components, but may further include other components unless the context clearly indicates otherwise.

is a schematic block diagram of an apparatus for training a deep neural network according to an embodiment of the present invention. Referring to, the apparatus for training a deep neural network including a hidden layer composed of n layers (where n is a natural number) between an input layer and an output layer according to the present invention includes an input meansand a training means.

The input meansmay be a means for receiving training data required to train the deep neural network, and may receive, for example, a data set (e.g., CIFAR-10 data set or the like) including images randomly selected for respective classes. This data set is called an epoch, and one epoch is composed of multiple iterations, wherein a weight update (weight gradient update) is performed once per iteration during training of the deep neural network.

The training meansis a means for training the deep neural network using the training data, received through the input means, and may include a training module, a representation form determination module, and a layer-wise precision determination module, as illustrated in.

The training moduleperforms training using the training data so that training is performed using a first precision preset for each layer. Typically, the first precision may be a high bit precision (e.g., 16-bit precision), but, in an embodiment of the present invention, the case where the first precision is set to a low bit precision (e.g., 8-bit precision) in order to apply the training apparatus to a mobile device will be described.

The representation form determination moduledetermines a representation form for representing pieces of internal data (e.g., pieces of data generated as the result of a convolution) that are generated during an operation (calculation) process for training. In particular, the representation form determination moduledetermines the position of the decimal point of the pieces of internal data so as to randomly vary a permissible overflow bit in a dynamic fixed-point system, thus enabling both a small number and a large number to be precisely represented.

That is, when an overflow occurs in a fixed-point system, a decimal point is generally moved so as to immediately increase the number of bits in an integer part. This case is characterized in that, in order to represent a large number, small numbers are inaccurately represented. Therefore, control may be performed such that a predetermined number of bits for which an overflow is permitted are set, and such that a decimal point is moved to increase the number of bits of the integer part only when an overflow exceeding the set number of bits occurs, thus enabling small numbers to be accurately represented. However, in this case, a problem may arise in that the number of bits assigned to the integer part becomes smaller, and thus accuracy may be lost as large numbers are inaccurately represented. These characteristics of the fixed-point system are not greatly problematic at a high bit precision in which the numbers of bits in an integer part and a fractional part can be secured as predetermined numbers of bits or more, but may be the cause of loss of accuracy at a low bit precision.

Therefore, the representation form determination modulemay be configured to include a storage unit for storing an overflow threshold, which is the maximum number of bits permitting an overflow; a random number generator for generating a random number between 0 and 1 in response to the generation of the internal data; a multiplier for determining a permissible overflow range by multiplying the overflow threshold by the random number; and a decimal point position determination unit for determining the position of the decimal point of the pieces of internal data based on the permissible overflow range. Further, the representation form determination modulemay randomly determine the permissible overflow bit within a range from 0 to the overflow threshold, thus solving the above problem. In this way, the present invention may be a scheme for leaving a margin in the number of overflow bits based on which it is determined whether to increase an integer part, may stochastically use the number of overflow bits, and may use the integer part, changed through the above procedure, during iterations ranging from a next iteration.

Also, the representation form determination modulemay further include a rounding unit for performing stochastic rounding on the internal data, and the decimal point position determination unit may determine the position of the decimal point based on the permissible overflow range for the internal data to which stochastic rounding is applied.

Here, stochastic rounding (SR) is intended to introduce the concept of probability into rounding, unlike typical rounding. In detail, unlike existing rounding, in which the value to be rounded is defined for a given value, the stochastic rounding (SR) is a scheme for taking into consideration the possibility of another type of rounding, and is characterized in that both rounded-up and rounded-down results may appear according to stochastic rounding and in that the probability that a value close to a rounded-up or rounded-down value will appear when rounding up or down is performed is higher than the probability in other cases. However, since the possibility that a value far away from the rounded-up or rounded-down value will appear is not unprecedented, a result obtained when rounding down is performed may also appear. Therefore, stochastic rounding may enable training of a low-bit deep neural network by reducing errors appearing due to rounding when the number of iterations is increased. For this stochastic rounding, reference is to be made to the prior art (e.g., a thesis entitled “DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients, Arxiv:1606.06160”).

The layer-wise precision determination moduledetermines the precision of each layer constituting the deep neural network and automatically changes the precision of the corresponding layer based on the result of the determination. In particular, unlike the conventional scheme in which a person is directly involved in determination or in which the results of inference of the deep neural network are checked, after which precision is predicted, the layer-wise precision determination modulemay be operated during training of the deep neural network, and may determine whether precision is to be scaled by comparatively checking a similarity between intermediate operation results of a high-bit operation and a low-bit operation, rather than final inference results. Therefore, the layer-wise precision determination moduleis characterized in that, as well as inference, the precision of errors and the precision of weights required in a training process may also be searched for.

For this operation, during an operation in each of an inference (feed-forward) stage and an error propagation stage, the layer-wise precision determination modulemay further perform, for each layer, a high-bit operation based on second precision, which is precision higher than the first precision, which is the initially set precision, and may determine the precision of the corresponding layer based on a similarity between the result of the low-bit operation using the first precision and the result of the high-bit operation using the second precision. That is, the layer-wise precision determination modulemay store a first threshold value, which is preset as the reference for a similarity based on which whether precision is to be changed for each layer is determined, and a second threshold value, which is preset to a value greater than the first threshold value. Further, the layer-wise precision determination modulemay be operated to change the precision of the corresponding layer to a value higher than the first precision when the similarity is less than or equal to the first threshold value, change the precision of the corresponding layer to a value lower than the first precision when the similarity is equal to or greater than the second threshold value, and maintain the first precision when the similarity is between the first threshold value and the second threshold value.

In other words, during the operation in each of the inference (feed-forward) stage and the error propagation stage, the layer-wise precision determination modulemay perform a procedure for performing a low-bit operation and a high-bit operation for each layer, deriving a similarity between respective result values of the low-bit operation and the high-bit operation, predicting that high precision is required and increasing the first precision when the similarity is low, and decreasing the first precision when the similarity is high, thus automatically finding the optimal precision of each layer and changing the current precision to the optimal precision.

Here, the layer-wise precision determination modulemay automatically change the precision of each layer, with respect to all layers constituting the deep neural network, except the input layer and the output layer.