Systems and techniques are provided for flexible data ingestion. Data including a file including a database table may be received at a computing device. The file may be in a non-standard binary format. The data including the file may be stored unaltered as a source data chunk. A processed data chunk may be generated from the source data chunk by converting the file to a standard binary format and storing the file in the processed data chunk without altering the source data chunk. A materialized data chunk may be generated from the processed data chunk by performing, with a database server engine of the computing device, a database operation on the database table of the file of the processed data chunk and storing the file in the materialized data chunk without altering the processed data chunk. The database table of the file of the materialized data chunk may be made available for querying by the database server engine.
Legal claims defining the scope of protection, as filed with the USPTO.
4. The method of claim 1, wherein generating, by an ingestion engine of the computing device, a processed data chunk from the source data chunk further comprises performing at least one of: determining a schema of the database table of the file of the source data chunk, validating data, removing garbage data, and casting of columns of the database table of the file of the source data chunk.
5. The method of claim 1, wherein the at least one database operation performed on the database table of the file of the processed data chunk comprises one or both of adding a column to the database table of the file of the processed data chunk and removing a column from the database table of the file of the processed data chunk.
6. The method of claim 1, wherein generating, by the ingestion engine of the computing device, a materialized data chunk from the processed data chunk further comprises sending, by the ingestion engine to the database server engine, a query that causes the database server engine to perform the at least one database operation.
7. The method of claim 1, wherein at least one operation is optionally performed when generating the processed data chunk, when generating the materialized data chunk, or when the database of the file of the materialized data chunk is made available for querying.
8. The method of claim 1, wherein the data comprising the file comprising the database table comprises a delta update or a snapshot update for a previously stored source data chunk.
9. The method of claim 1, wherein the non-standard binary format is comma separated values (csv) and wherein the standard binary format is one of parquet and XML.
13. The system of claim 10, wherein generating, by an ingestion engine of the computing device, a processed data chunk from the source data chunk further comprises performing at least one of: determining a schema of the database table of the file of the source data chunk, validating data, removing garbage data, and casting of columns of the database table of the file of the source data chunk.
14. The system of claim 10, wherein the at least one database operation performed on the database table of the file of the processed data chunk comprises one or both of adding a column to the database table of the file of the processed data chunk and removing a column from the database table of the file of the processed data chunk.
15. The system of claim 10, wherein the processor generates a materialized data chunk from the processed data chunk further by sending to the database server engine a query that causes the database server engine to perform the at least one database operation.
16. The system of claim 15, wherein at least one operation is optionally performed by the processor when the processor generates the processed data chunk, generates the materialized data chunk, or makes the database of the file of the materialized data chunk available for querying.
17. The system of claim 10, wherein the data comprising the file comprising the database table comprises a delta update or a snapshot update for a previously stored source data chunk.
18. The system of claim 10, wherein the non-standard binary format is comma separated values (csv) and wherein the standard binary format is one of parquet and XML.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
September 17, 2020
July 4, 2023
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.