Centralized Parallel Burst Engine for High Performance Computing

1. An storage system for temporarily storing data from a high performance computing (HPC) system, comprising: a software client supported at a compute node of the HPC system, said software client configured to: capture file level data during a burst output phase of said compute node; and output said file level data and associated metadata via an output of the compute node; an intermediate storage system having at least a first burst engine including: an front-end interface for receiving said file level data and associated metadata from said compute node during said burst output phase, wherein said file level data and associated metadata is received at a first data rate; an internal memory for caching said file level data and said associated metadata; a burst engine controller configured to write said file level data and said associated metadata as block level data via a back-end interface; and at least a first set of dedicated non-volatile storage devices connected to said controller via said back-end interface that receive the block level data from the burst engine controller, wherein said first set of dedicated non-volatile storage devices have an aggregate data transfer rate that equals or exceed said first data rate; and a server including an internal memory and controller, wherein said server is configured to: receive a notification from said software client at the HPC node, said notification including logical block addresses of the block level data stored in the dedicated non-volatile storage devices of the intermediate storage system; retrieve block level data from said storage devices and interpret said metadata; and replay said block level data to a primary storage system in accordance with said metadata, wherein said block level data is replayed during a compute phase of said compute node of the HPC at a second data transfer rate that is less than said first data transfer rate.

2. The system of claim 1 , wherein a data input rate of said burst engine equals or exceed an aggregate data output rate of all connected compute nodes.

3. The system of claim 1 , further comprising: a plurality of burst engines, wherein each burst engine is directly connected to at least one compute node.

4. The system of claim 1 , wherein said set of dedicated non-volatile storage devices comprises; a set of solid state drives (SSDs).

5. The system of claim 1 , wherein said server is configured to notify said compute node that said storage devices are available upon completing replay of said block level data to said primary storage system.

6. The system of claim 1 , wherein said burst engine comprises: a PCIe controller card.

7. The system of claim 1 , wherein said set of dedicated non-volatile storage devices are connected to the back-end interface of said burst engine free of an intervening controller.

8. The system of claim 1 , wherein said burst engine controller is configured to write said block level data to said set of dedicated non-volatile storage devices with redundancy.

9. The system of claim 8 , wherein said burst engine controller is configured to write said block level data with RAID redundancy.