Method and Apparatus with Neural Network Operation Using Sparsification

1. A processor-implemented neural network operation method, comprising: generating a first sparse data by sparsifying, during a first iteration, a first activation gradient based on a first threshold corresponding to a layer included in a neural network; determining a second activation gradient of a second iteration by performing a neural network operation based on the first sparse data and dense data; determining a second threshold for sparsifying the second activation gradient by updating the first threshold based on the second activation gradient; generating a second sparse data by sparsifying, during the second iteration, based on the second threshold, the second activation gradient; performing a neural network operation, using the second sparse data and the dense data, based on the second activation gradient and the second threshold, wherein the determining of the second threshold comprises determining the second threshold by updating the first threshold based on a target sparsity and a sparsity corresponding to a current iteration, the determining of the second activation gradient corresponds to a backpropagation operation, and the performing of the neural network operation based on the second activation gradient and the second threshold corresponds to an image inference operation of a terminal.

2. The method of claim 1, wherein the first activation gradient and the second activation gradient comprise any one of a gradient with respect to input activation, a gradient with respect to weight, and a gradient with respect to output activation.

3. The method of claim 1, wherein the determining of the second threshold comprises determining the second threshold by updating the first threshold over a predetermined number of iterations.

4. The method of claim 1 wherein the determining of the second threshold comprises determining the second threshold by multiplying the first threshold and a value of the target sparsity divided by the sparsity corresponding to the current iteration.

5. The method of claim 1, wherein the determining of the second threshold comprises: determining whether the second threshold exceeds a preset limited range; and correcting the second threshold to a value within the limited range in response to the second threshold exceeding the limited range.

6. The method of claim 1, wherein the determining of the second threshold comprises determining the second threshold by initializing the first threshold based on the second activation gradient.

7. The method of claim 1, wherein the dense data are stored in parallelized dense buffers.

8. The method of claim 1, wherein the performing of the neural network operation based on the second activation gradient and the second threshold comprises performing one or more multiply-accumulate (MAC) operations based on the second activation gradient.

9. The method of claim 1, further comprising generating a trained neural network by training the neural network using the second sparse data.

10. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the method of claim 1.

11. A neural network operation apparatus, comprising: a processor configured to: generate a first sparse data by sparsifying, during a first iteration, a first activation gradient based on a first threshold corresponding to a layer included in a neural network; determine a second activation gradient of a second iteration by performing a neural network operation based on the first sparse data and dense data; determine a second threshold for sparsifying the second activation gradient by updating the first threshold based on the second activation gradient; generate a second sparse data by sparsifying, during the second iteration, based on the second threshold, the second activation gradient; perform a neural network operation, using the second sparse data and the dense data, based on the second activation gradient and the second threshold, wherein the determining of the second threshold comprises updating the first threshold based on a target sparsity and a sparsity corresponding to a current iteration, the determining of the second activation gradient corresponds to a backpropagation operation, and the performing of the neural network operation based on the second activation gradient and the second threshold corresponds to an image inference operation of a terminal.

12. The apparatus of claim 11, wherein the first activation gradient and the second activation gradient comprise any one of a gradient with respect to input activation, a gradient with respect to weight, and a gradient with respect to output activation.

13. The apparatus of claim 11, wherein, for the determining of the second threshold, the processor is further configured to determine the second threshold by updating the first threshold over a predetermined number of iterations.

14. The apparatus of claim 11, wherein, for the determining of the second threshold, the processor is further configured to determine the second threshold by multiplying the first threshold and a value of the target sparsity divided by the sparsity corresponding to the current iteration.

15. The apparatus of claim 11, wherein, for the determining of the second threshold, the processor is further configured to: determine whether the second threshold exceeds a preset limited range; and correct the second threshold to a value within the limited range in response to the second threshold exceeding the limited range.

16. The apparatus of claim 11, wherein, for the determining of the second threshold, the processor is further configured to determine the second threshold by initializing the first threshold based on the second activation gradient.

17. The apparatus of claim 11, wherein the dense data are stored in parallelized dense buffers.

18. The apparatus of claim 11, wherein, for the performing of the neural network operation based on the second activation gradient and the second threshold, the processor is further configured to perform one or more multiply-accumulate (MAC) operations based on the second activation gradient.

19. The apparatus of claim 11, wherein the processor is configured to generate a trained neural network by training the neural network using the second sparse data.

20. A processor-implemented neural network operation method, comprising: generating a first sparse data by sparsifying, during a first iteration, based on a first threshold, a first activation gradient corresponding to a layer included in a neural network; determining a second activation gradient of a second iteration by performing a neural network operation based on the first sparse data and dense data; generating a second threshold by updating the first threshold based on a sparsity of the second activation gradient corresponding to a current iteration and a target sparsity; and performing a neural network operation based on the second activation gradient and the second threshold, wherein the determining of the second activation gradient corresponds to a backpropagation operation; and the performing of the neural network operation based on the second activation gradient and the second threshold corresponds to an image inference operation of a terminal.

21. The method of claim 20, wherein the performing of the neural network operation based on the sparsified first activation gradient comprises one or more multiply-accumulate (MAC) operators receiving the sparsified first activation gradient from one or more sparse buffers and performing the neural network operation.