Method and System For Processing Big Data

1. A distributed acceleration system for distributed big data analytics processing comprising: a plurality of nodes each having a server processor, a server memory and a plurality of accelerators; the plurality of accelerators having a memory management unit, an accelerator memory, server interface and network interface to a plurality of accelerators located at other nodes; the server memory and the accelerator memory capable of storing and persisting input and output data set partitions across application execution stages; and the server interface is located between acceleration software located in the server memory which is capable of being executed by the server processor and wherein the server interface is capable of copying input and output streams of data pages between the plurality of accelerators and the server memory.

2. The system of claim 1 wherein the plurality of accelerators are capable of local data shuffle by sending and receiving data to each other over local integrated circuit wires.

3. The system of claim 2 wherein the data shuffle is performed for Apache Spark operations that require the local data shuffle including reduce operations.

4. The system of claim 1 wherein the accelerator is capable of cluster-wide data shuffle by sending and receiving data to or from other accelerators located at other nodes over the network interface.

5. The system of claim 4 wherein the data shuffling is performed for Apache Spark operations that require cluster-wide data shuffle including reduce operations.

6. The system of claim 1 , wherein the accelerators are selected from at least one of the group consisting of: a field programmable gate array (FPGA) chip and an application specific integrated circuit (ASIC).

7. The system of claim 1 , wherein the plurality of accelerators and the server processor are on the same integrated circuit.

8. The system of claim 1 , wherein the acceleration software comprises: a runtime scheduler that is capable of tracking explicit and implicit distributed data sets that reside in the accelerator memory.

9. The system of claim 1 , wherein the acceleration software is further capable of dispatching commands to allocate and deallocate distributed data sets in the accelerator memory.

10. The system of claim 1 , wherein the acceleration software is further capable of copying distributed data sets between the server memory and the accelerator memory.

11. The system of claim 1 , wherein the memory management unit comprises: a set of data page tables that includes one data page table for each data set entry in the data set table and wherein each date page table contains a plurality of data page table entries.

12. The system of claim 1 wherein the memory management unit comprises: an access register which monitors data pages which have not been accessed recently and provides this data page for allocation to a new data page; a dirty register which monitors data pages which have been written since allocation in the accelerator memory and upon which the memory management unit sends a message to the acceleration software requesting that the data page be saved to the server memory; and a free register which monitors data pages in the accelerator memory which are not currently allocated to any of the data sets.

13. A distributed acceleration method for distributed big data analytics processing comprising: storing and persisting input and output data set partitions across application execution stages at a plurality of nodes each having a server processor, a server memory and a plurality of accelerators, wherein the plurality of accelerators have a memory management unit, an accelerator memory, server interface and network interface to a plurality of accelerators located at other nodes; and executing acceleration software located in the server memory by the server processor; and copying input and output streams of data pages between the plurality of accelerators and the server memory by the server interface.

14. The method of claim 13 , further comprising: sending and receiving data between each of the plurality of accelerators through local data shuffle over local integrated circuit wires.

15. The method of claim 14 further comprising: performing the data shuffle for Apache Spark operations that require the local data shuffle including reduce operations.

16. The method of claim 13 further comprising: sending and receiving data to or from other accelerators located at other nodes over the network interface to accomplish cluster-wide data shuffle.

17. The method of claim 16 further comprising: performing the data shuffle for Apache Spark operations that require the local data shuffle including reduce operations.

18. The method of claim 13 , wherein the accelerators are selected from at least one of the group consisting of: a field programmable gate array (FPGA) chip and an application specific integrated circuit (ASIC).

19. The method of claim 13 , wherein the plurality of accelerators and the server processor are on the same integrated circuit.

20. The method of claim 13 , further comprising: tracking explicit and implicit distributed data sets that reside in the accelerator memory by a runtime scheduler of the acceleration software.

21. The method of claim 13 , further comprising: dispatching commands to allocate and deallocate distributed data sets in the accelerator memory by the acceleration software.

22. The method of claim 13 , further comprising: copying distributed data sets between the server memory and the accelerator memory by the acceleration software.

23. The method of claim 13 , wherein the memory management unit comprises: a set of data page tables that includes one data page table for each data set entry in the data set table and wherein each date page table contains a plurality of data page table entries.

24. The method of claim 13 , further steps performed by the memory management unit comprising: monitoring data pages which have not been accessed recently and providing this data page for allocation to a new data page by an access register; monitoring data pages by a dirty register which have been written since allocation in the accelerator memory; sending a message to the acceleration software requesting that the data page be saved to the server memory; and monitoring data pages by a free register in the accelerator memory which are not currently allocated to any of the data sets.