Programming Method of an Activation Function and an Activation Function Programming Unit

1. A computing device, comprising: one or more processors; and memory storing instructions thereon, the instructions, when executed by the one or more processors, cause the one or more processors to: receive a target activation function of a first artificial neural network for implementing in a neural processor circuit, execute a second artificial neural network to determine an approximation activation function that approximates the target activation function and is programmable into the neural processor circuit to implement the first artificial neural network, wherein the second artificial neural network is trained to determine one or more parameters of the approximation activation function to reduce an error with respect to the target activation function, and cause the approximation activation function to be programmed into the neural processor circuit to implement the first artificial neural network.

2. The computing device of claim 1, wherein the second artificial neural network includes a first neural network section comprising at least one neuron and a second neural network section comprising at least one neuron, wherein a rectified linear unit (ReLU) function is applied to outputs of the at least one neuron of the first neural network section, and wherein a value to which the ReLU function is applied is an input to the at least one neuron of the second neural network section.

3. The computing device of claim 1, wherein the second artificial neural network comprises a first neural network section including at least one neuron and a second neural network section including at least one neuron, wherein each of the at least one neuron of the first neural network section includes a weight and a bias, and wherein the at least one neuron in the second neural network section include only weights.

4. The computing device of claim 1, wherein the approximation activation function comprises a plurality of segments including a segment of a first-order function.

5. The computing device of claim 1, wherein the approximation activation function comprises a plurality of segments, wherein a number of the plurality of the segments corresponds to hardware configuration of a subcircuit in the neural processor circuit programmable to store the approximation activation function, and wherein the hardware configuration is related to a number of comparators in the subcircuit.

6. The computing device of claim 1, wherein the second artificial neural network includes at least one neuron, and wherein the approximation activation function includes a plurality of segments, each of the segments separated by a breaking point of an output of the at least one neuron.

7. The computing device of claim 1, wherein the second artificial neural network includes a plurality of neurons, and wherein at least one of outputs of the plurality of neurons is pruned according to hardware information of a subcircuit in the neural processor circuit programmed to store the approximation activation function.

8. The computing device of claim 1, wherein the second artificial neural network includes a plurality of neurons, and wherein a number of the plurality of neurons is less than or equal to a number of comparators in a subcircuit in the neural processor circuit programmed to store the approximation activation function.

9. The computing device of claim 1, wherein the approximation activation function is defined by one or more slopes and one or more offsets.

10. The computing device of claim 9, wherein the instructions further cause the one or more processors to program the one or more slopes and the one or more offsets into a lookup table in the neural processor circuit.

11. An activation function programming method comprising: receiving a target activation function of a first artificial neural network for implementing in a neural processor circuit; executing a second artificial neural network to determine an approximation activation function that approximates the target activation function and is defined by one or more slopes and one or more offsets that are programmable in a lookup table of the neural processor circuit, wherein the second artificial neural network is trained to determine one or more parameters of the approximation activation function to reduce an error with respect to the target activation function; and causing the one or more slopes and the one or more offsets to be programmed into the lookup table of the neural processor circuit to implement the first artificial neural network.

12. The activation function programming method of claim 11, wherein the second artificial neural network includes a first neural network section comprising at least one neuron and a second neural network section comprising at least one neuron, wherein a rectified linear unit (ReLU) function is applied to outputs of the at least one neuron of the first neural network section, and wherein a value to which the ReLU function is applied is an input to the at least one neuron of the second neural network section.

13. The activation function programming method of claim 11, wherein the second artificial neural network comprises a first neural network section including at least one neuron and a second neural network section including at least one neuron, wherein each of the at least one neuron of the first neural network section includes a weight and a bias, and wherein the at least one neuron in the second neural network section include only weights.

14. The activation function programming method of claim 11, wherein the approximation activation function comprises a plurality of segments including a segment of a first-order function.

15. The activation function programming method of claim 11, wherein the approximation activation function comprises a plurality of segments, wherein a number of the plurality of the segments corresponds to hardware configuration of a subcircuit in the neural processor circuit programmable to store the approximation activation function, and wherein the hardware configuration is related to a number of comparators in the subcircuit.

16. The activation function programming method of claim 11, wherein the second artificial neural network includes at least one neuron, and wherein the approximation activation function includes a plurality of segments, each of the segments separated by a breaking point of an output of the at least one neuron.

17. The activation function programming method of claim 11, wherein the second artificial neural network includes a plurality of neurons, and wherein a number of the plurality of neurons is less than or equal to a number of comparators included in a subcircuit of the neural processor circuit.

18. The activation function programming method of claim 11, wherein the second artificial neural network includes a plurality of neurons, and wherein at least one of outputs of the plurality of neurons is pruned according to hardware information of a subcircuit in the neural processor circuit programmed to store the approximation activation function.