Method and Apparatus for Implementing a Deep Neural Network Performing an Activation Function

3. A method as claimed in claim 2, wherein the DNN uses the activation module to perform multiple activation functions in order to process the input data stream, and the method further comprises repeating the providing, monitoring, generating and loading steps in respect of each activation function in order to generate and load lookup data for the lookup table representing each of the multiple activation functions.

4. A method as claimed in claim 2, wherein the lookup table comprises, and is operable to switch between, two sets of lookup data and, on the activation module performing a series of activation functions, the loading of the generated lookup data of a next activation function in the series into the lookup table is performed concurrently with the performing of a first activation function in the series.

5. A method as claimed in claim 2, further comprising, on receiving the input data stream to the hardware, checking if an input value to the activation module lies outside the determined range of input values and, if the input value to the activation module lies outside the determined range of input values, using as an output value of the activation function the value of the activation function corresponding to the closest extreme of the determined range of input values.

6. A method as claimed in claim 2, wherein the monitoring the input to the activation module further comprises determining an offset that, when subtracted from each input value to the activation module, causes the range of input values to be centred about a predefined input value, the performing the activation function comprising subtracting the offset from each input value received at the activation module prior to looking up each input value in the lookup table.

7. A method as claimed in claim 2, wherein performing the activation function comprises, on receiving a first input value, looking up a pair of adjacent data points in the lookup table closest to the first input value and interpolating between a corresponding pair of values of the activation function so as to form an estimate of the value of the activation function corresponding to the first input value.

8. A method as claimed in claim 7, wherein a predefined number of most significant bits of the first input value are used as the lookup address into the lookup table and the remaining bits of the first input value are used in the interpolating between the corresponding pair of values of the activation function.

9. A method as claimed in claim 7, wherein the lookup table comprises first and second data stores, the first data store comprising a first set of data points and the second data store comprising a second set of data points such that for each adjacent pair of data points, one of the data points is in the first data store and the other data point is in the second data store, and the performing of the activation function for the first input value comprises simultaneously looking up each of the pair of adjacent points in their respective first or second data store.

10. A method as claimed in claim 1, wherein generating lookup data comprises calculating a set of curves approximating the activation function over the determined range of input values, each curve representing a portion of the activation function such that collectively the set of curves identify an output value for each input value within the determined range.

12. A method as claimed in claim 1, wherein generating lookup data comprises calculating a set of data points representing the activation function over the determined range of input values.

13. A method as claimed in claim 1, wherein monitoring the input to the activation module comprises identifying maximum and minimum input values to the activation module and using those maximum and minimum input values as the bounds of the determined range of input values.

14. A method as claimed in claim 1, wherein monitoring the input to the activation module comprises forming a histogram of input values representing the probability of occurrence of input values and using as the bounds of the determined range of input values a pair of input values between which a predefined or programmable proportion of the distribution of input values lies.

15. A method as claimed in claim 1, wherein the determined range of input values is less than the possible range of input values according to the bit length of the input values and the lookup data represents the activation function over less than that possible range of input values.

16. A method as claimed in any preceding claim, wherein the lookup data represents the activation function over a range equal to the determined range of input values.

17. A method as claimed in claim 1, wherein the number of entries in the lookup data representing the activation function over the determined range of input values is equal to the number of entries in the lookup table for the activation function.

18. A method as claimed in claim 1, wherein the method is performed subsequent to optimisation of the DNN.

20. Hardware for implementing a Deep Neural Network (DNN) comprising an activation module for performing an activation function, the activation module having a programmable lookup table for storing lookup data representing the activation function, and, in use, the activation module being configured to load into the lookup table first lookup data generated over a determined range of input values to the activation module for use in performing the activation function, wherein an expected range of input values to the activation module is determined by, at a representation of the hardware which is arranged to implement the DNN and to operate the DNN on calibration data, monitoring resultant input values to the activation module of the representation during operation of the DNN on the calibration data.