Method and Apparatus for Accelerated Data Translation Using Record Layout Detection

1. A method for low latency and high throughput data translation using record layout detection, the method comprising: a processor receiving a plurality of records, the records comprising record data content arranged in a first format, but exhibiting a plurality of different record layouts for the first format, wherein the processor comprises a processing pipeline that is controllable to translate records to a second format from any of the different record layouts for the first format, and wherein the processing pipeline is deployed on at least one of a reconfigurable logic device, a graphics processing unit (GPU), a multi-core processor, and a cell processor; the processor analyzing the record data content with respect to a plurality of conditions corresponding to the different record layouts to determine the record layouts for the records from among the different record layouts, wherein the processor includes a plurality of data analysis components arranged in parallel, and wherein the analyzing step, for each of a plurality of the records, comprises: the processor processing a record through the data analysis components in parallel, each of a plurality of the data analysis components (1) listening for data of interest in the record based on a byte offset from a start of record, (2) testing the record data content in the data of interest against a corresponding predicate condition to determine whether the corresponding predicate condition has been met, and (3) outputting data indicative of whether the tested record data content satisfies the corresponding predicate condition, wherein the corresponding predicate conditions for the data analysis components in the aggregate serve as criteria for determining whether the record exhibits any of the different record layouts; based on the output data from the data analysis components, the processor determining whether the record exhibits one of the different record layouts; and in response to a determination that the record exhibits one of the different record layouts, the processor generating data that associates the record with its determined record layout; defining the byte offsets for data of interest with respect to data analysis components based on data in a configuration table; the processor streaming the records and data indicative of their determined record layouts through the processing pipeline; controlling the processing pipeline based on the data indicative of the determined record layouts such that the controlled processing pipeline is configured to translate record data content arranged in the determined record layouts to the second format; and the controlled processing pipeline translating the records by simultaneously performing a plurality of translation tasks on different portions of the streaming record data content to arrange the record data content in the second format.

2. The method of claim 1 wherein the processing pipeline is deployed on a reconfigurable logic device.

3. The method of claim 2 wherein the reconfigurable logic device comprises a field programmable gate array (FPGA).

4. The method of claim 1 wherein the analyzing step is performed by a reconfigurable logic device.

5. The method of claim 4 wherein the reconfigurable logic device comprises the data analysis components arranged in parallel.

6. The method of claim 5 further comprising the reconfigurable logic device repeating the processing step, the determining step, and the generating step for a plurality of the streaming records in a pipelined fashion.

7. The method of claim 5 wherein the analyzing step further comprises: the processor determining a record length for the record; and the processor generating data that associates the determined record length with the record and its determined record layout.

8. The method of claim 5 further comprising: loading the predicate conditions into the data analysis components from the configuration table.

9. The method of claim 8 wherein the processing step further comprises each of a plurality of the data analysis components listening for data of interest in each record based on a field identifier in each record; and defining the field identifiers for data of interest with respect to data analysis components based on data in the configuration table.

10. The method of claim 1 wherein the analyzing step is performed by a field programmable gate array (FPGA).

11. The method of claim 1 wherein the processor comprises a first processor and a second processor, wherein the processing pipeline is deployed on the second processor, and wherein the analyzing step is performed by the first processor executing software.

12. The method of claim 11 further comprising a reconfigurable logic device tagging the analyzed records with data indicative of the records' determined record layouts, and wherein the streaming step comprises streaming the tagged records through the processing pipeline.

13. The method of claim 1 wherein the first format is a fixed field format.

14. The method of claim 13 wherein the second format is a mapped field format.

15. The method of claim 14 wherein the mapped field format is a mapped variable field format.

16. The method of claim 13 wherein the second format is a delimited data format.

17. The method of claim 1 wherein the first format is a mapped field format.

18. The method of claim 17 wherein the mapped field format is a mapped variable field format.

19. The method of claim 17 wherein the second format is a fixed field format.

20. The method of claim 17 wherein the second format is a delimited data format.

21. The method of claim 1 wherein the first format is a delimited data format.

22. The method of claim 21 wherein the second format is a mapped field format.

23. The method of claim 22 wherein the mapped field format is a mapped variable field format.

24. The method of claim 21 wherein the second format is a fixed field format.

25. The method of claim 1 further comprising: performing a data processing operation on the translated data in the second format.

26. The method of claim 25 wherein the translated data comprises a plurality of data characters in a plurality of fields, and wherein the data processing operation performing step comprises a processor selectively targeting a field of the translated data for the data processing operation without analyzing the data characters of the translated data.

27. The method of claim 26 wherein the translated data further comprises a plurality of record headers and a plurality of field headers, the record headers comprising data indicative of where boundaries exist between a plurality of records in the translated data, the field headers comprising data indicative of where boundaries exist between the fields in the records, and wherein the selectively targeting step comprises a processor selectively targeting a field for the data processing operation based on data in the field headers.

28. The method of claim 26 wherein the processor that performs the selectively targeting step and the data processing operation performing step comprises at least one of a reconfigurable logic device, a graphics processing unit (GPU), a multi-core processor, and a cell processor.

29. The method of claim 1 wherein the analyzing step is performed by at least one of a reconfigurable logic device, a graphics processing unit (GPU), a multi-core processor, and a cell processor.

30. An apparatus for low latency and high throughput data translation using record layout detection, the apparatus comprising: a processor that includes a processing pipeline, wherein the processing pipeline is deployed on at least one of a reconfigurable logic device, a graphics processing unit (GPU), a multi-core processor, and a cell processor; wherein the processor further comprises a plurality of data analysis components and a logic component; the processor configured to receive a plurality of records, the records comprising record data content arranged in the first format, but exhibiting a plurality of different record layouts for the first format; wherein the processor is further configured to perform record layout detection for each of a plurality of the records via a plurality of operations that (1) process a record through a plurality of the data analysis components in parallel, each of a plurality of the data analysis components configured to (i) listen for data of interest in the record based on a byte offset from a start of record, (ii) test the record data content in the data of interest against a corresponding predicate condition to determine whether the corresponding predicate condition has been met, and (iii) output data indicative of whether the tested record data content satisfies the corresponding predicate condition, wherein the corresponding predicate conditions for the data analysis components in the aggregate serve as criteria for determining whether the record exhibits any of the different record layouts, (2) based on the output data from the data analysis components, determine via the logic component whether the record exhibits one of the different record layouts, and (3) in response to a determination that the record exhibits on of the different record layouts, generate data that associates the record with its determined record layout; wherein the processor is further configured to define the byte offsets for data of interest with respect to data analysis components based on data in a configuration table; wherein the processor is further configured to stream the records and data indicative of their determined record layouts through the processing pipeline; and wherein the processing pipeline is controllable based on the data indicative of the determined record layouts such that the controlled processing pipeline is configured to translate the streaming records by simultaneously performing a plurality of translation tasks on different portions of the streaming record data content having the determined record layouts to arrange the record data content in the second format.

31. The apparatus of claim 30 wherein the processor comprises a reconfigurable logic device, the processing pipeline being deployed on the reconfigurable logic device.

32. The apparatus of claim 31 wherein the reconfigurable logic device comprises a field programmable gate array (FPGA).

33. The apparatus of claim 30 wherein the processor comprises a reconfigurable logic device, and wherein the reconfigurable logic device is configured to perform the analyze operation.

34. The apparatus of claim 33 wherein the first format is a fixed field format.

35. The apparatus of claim 34 wherein the second format is a mapped field format.

36. The apparatus of claim 35 wherein the mapped field format is a mapped variable field format.

37. The apparatus of claim 34 wherein the second format is a delimited data format.

38. The apparatus of claim 33 wherein the first format is a mapped field format.

39. The apparatus of claim 38 wherein the mapped field format is a mapped variable field format.

40. The apparatus of claim 38 wherein the second format is a fixed field format.

41. The apparatus of claim 38 wherein the second format is a delimited data format.

42. The apparatus of claim 33 wherein the first format is a delimited data format.

43. The apparatus of claim 42 wherein the second format is a mapped field format.

44. The apparatus of claim 43 wherein the mapped field format is a mapped variable field format.

45. The apparatus of claim 42 wherein the second format is a fixed field format.

46. The apparatus of claim 30 wherein the processor comprises a field programmable gate array (FPGA), and wherein the FPGA is configured to perform the analyze operation.

47. The apparatus of claim 30 wherein the processor comprises a first processor and a second processor, wherein the first processor comprises a software-based processor configured to execute software to perform the analyze operation, and wherein the processing pipeline is deployed on the second processor.

48. The apparatus of claim 47 wherein the processor further comprises a reconfigurable logic device, the reconfigurable logic device configured to tag the analyzed records with data indicative of the records' determined record layouts, and wherein the processor is further configured to stream the tagged records through the processing pipeline.

49. The apparatus of claim 30 wherein processor is further configured to perform the record layout detection operations for a plurality of the streaming records in a pipelined fashion.

50. The apparatus of claim 30 wherein the processor is further configured to, as part of the record layout detection operations: determine a record length for the record; and generate the data that associates the determined record length with the record and its determined record layout.

51. The apparatus of claim 30 wherein the processor is further configured to load the predicate conditions into the data analysis components from the configuration table.

52. The apparatus of claim 51 wherein each of a plurality of the data analysis components is configured to listen for data of interest in each record based on a field identifier in each record; and wherein the processor is further configured to define the field identifiers for data of interest with respect to data analysis components based on data in the configuration table.

53. The apparatus of claim 30 wherein the processor is further configured to perform a data processing operation on the translated data in the second format.

54. The apparatus of claim 53 wherein the translated data comprises a plurality of data characters in a plurality of fields, and wherein the processor is further configured to selectively target a field of the translated data for the data processing operation without analyzing the data characters of the translated data.

55. The apparatus of claim 54 wherein the translated data further comprises a plurality of record headers and a plurality of field headers, the record headers comprising data indicative of where boundaries exist between a plurality of records in the translated data, the field headers comprising data indicative of where boundaries exist between the fields in the records, and wherein the processor is further configured to selectively target a field for the data processing operation based on data in the field headers.

56. The apparatus of claim 54 wherein the processor comprises at least one of a reconfigurable logic device, a graphics processing unit (GPU), a multi-core processor, and a cell processor that performs the selective targeting operation and the data processing operation.

57. The apparatus of claim 30 wherein processor comprises at least one of a reconfigurable logic device, a graphics processing unit (GPU), a multi-core processor, and a cell processor that performs the analyze operation.

58. The apparatus of claim 30 wherein the processor comprises a hardware record layout detector, the hardware record layout detector comprising (1) the data analysis components arranged in a parallel orientation, and (2) the logic component.

59. The apparatus of claim 58 wherein the processor further comprises a reconfigurable logic device, wherein the hardware record layout detector is deployed on the reconfigurable logic device.

60. The apparatus of claim 30 wherein the processing pipeline is further configured to (1) identify field lengths for fields in the records based on the data indicative of the determined record layouts, (2) identify fields in the record data content based on the identified field lengths, and (3) arrange the identified fields of the record data content in the second format.

61. The apparatus of claim 60 wherein the processing pipeline is further configured to identify the field lengths based on a look up in a table that associates field lengths with data indicative of different record layouts.

62. The method of claim 1 wherein the controlling step comprises (1) identifying field lengths for fields in the records based on the data indicative of the determined record layouts, and (2) providing the identified field lengths to the processing pipeline.

63. The method of claim 62 wherein the translating step comprises the controlled processing pipeline (1) identifying fields in the record data content based on the provided field lengths, and (2) arranging the identified fields of the record data content in the second format.

64. The method of claim 62 wherein the identifying step comprises looking up the field lengths in a table that associates field lengths with data indicative of different record layouts.