Techniques for Configuring and Validating a Data Pipeline Deployment

1. A method, comprising: receiving a template that defines a plurality of job definitions; wherein each particular job definition of the plurality of job definitions corresponds to a particular data processing job, and wherein each particular job definition comprises: a code identifier that identifies code for processing the particular data processing job; a plurality of dataset dependency identifiers that identify a plurality of input datasets for the particular data processing job; a plurality of configuration parameters for processing the particular data processing job; for each particular job definition of the plurality of job definitions: based on the template, causing to be displayed a user interface for receiving a plurality of configuration parameter values for the plurality of configuration parameters for the particular job definition; receiving the plurality of configuration parameter values for the particular job definition; executing the corresponding particular data processing job for the particular job definition by executing the code for processing the particular data processing job, by using the input datasets for the particular data processing job and the plurality of configuration parameter values; in response to a command to perform a validation of a target data processing job that corresponds to a target job definition of the plurality of job definitions, executing the target data processing job for the target job definition by executing the code for processing the target data processing job, by using the input datasets for the target data processing job and the plurality of configuration parameter values for the target data processing job; applying one or more validation criteria to the target data processing job to generate a validation value that indicates a metric of accuracy of the plurality of configuration parameter values for the target data processing job; wherein the method is performed using one or more processors.

2. The method of claim 1 , wherein the method further comprises: receiving a list of a plurality of available templates; causing to be displayed in the user interface the plurality of available templates; receiving a user input for selection of the template from the plurality of available templates.

3. The method of claim 1 , wherein the method further comprises: receiving a global configuration parameter value that applies to each job definition of the plurality of job definitions; using the global configuration parameter value in the step of executing each of the corresponding particular data processing jobs.

4. The method of claim 1 , wherein the method further comprises: comparing the validation value to a pre-stored lower bound value and upper bound value; based on the comparison, displaying whether the validation value is within the lower bound value and upper bound value.

5. The method of claim 1 , wherein the plurality of job definitions define a data pipeline system for deduplication of data records.

6. The method of claim 1 , wherein the plurality of job definitions define a data pipeline system for featurizing, training, and applying a machine learning model.

7. The method of claim 1 , wherein the plurality of job definitions define a data pipeline system for data cleanup of data records.

8. The method of claim 1 , wherein the plurality of job definitions define a data pipeline system for joining of data sources.

9. The method of claim 1 , wherein the method further comprises: prior to executing the corresponding particular data processing job, storing the code for processing the particular data processing job, the input datasets for the particular data processing job, and the plurality of configuration parameter values in a code repository with version control.

10. One or more non-transitory computer-readable media storing instructions, wherein the instructions, when executed by one or more hardware processors, cause: receiving a template that defines a plurality of job definitions; wherein each particular job definition of the plurality of job definitions corresponds to a particular data processing job, and wherein each particular job definition comprises: a code identifier that identifies code for processing the particular data processing job; a plurality of dataset dependency identifiers that identify a plurality of input datasets for the particular data processing job; a plurality of configuration parameters for processing the particular data processing job; for each particular job definition of the plurality of job definitions: based on the template, causing to be displayed a user interface for receiving a plurality of configuration parameter values for the plurality of configuration parameters for the particular job definition; receiving the plurality of configuration parameter values for the particular job definition; executing the corresponding particular data processing job for the particular job definition by executing the code for processing the particular data processing job, by using the input datasets for the particular data processing job and the plurality of configuration parameter values; in response to a command to perform a validation of a target data processing job that corresponds to a target job definition of the plurality of job definitions, executing the target data processing job for the target job definition by executing the code for processing the target data processing job, by using the input datasets for the target data processing job and the plurality of configuration parameter values for the target data processing job; applying one or more validation criteria to the target data processing job to generate a validation value that indicates a metric of accuracy of the plurality of configuration parameter values for the target data processing job.

11. The one or more non-transitory computer-readable media of claim 10 , wherein the instructions further comprise instructions for: receiving a list of a plurality of available templates; causing to be displayed in the user interface the plurality of available templates; receiving a user input for selection of the template from the plurality of available templates.

12. The one or more non-transitory computer-readable media of claim 10 , wherein the instructions further comprise instructions for: receiving a global configuration parameter value that applies to each job definition of the plurality of job definitions; using the global configuration parameter value in the step of executing each of the corresponding particular data processing jobs.

13. The one or more non-transitory computer-readable media of claim 10 , wherein the instructions further comprise instructions for: comparing the validation value to a pre-stored lower bound value and upper bound value; based on the comparison, displaying whether the validation value is within the lower bound value and upper bound value.

14. The one or more non-transitory computer-readable media of claim 10 , wherein the plurality of job definitions define a data pipeline system for deduplication of data records.

15. The one or more non-transitory computer-readable media of claim 10 , wherein the plurality of job definitions define a data pipeline system for featurizing, training, and applying a machine learning model.

16. The one or more non-transitory computer-readable media of claim 10 , wherein the plurality of job definitions define a data pipeline system for data cleanup of data records.

17. The one or more non-transitory computer-readable media of claim 10 , wherein the plurality of job definitions define a data pipeline system for joining of data sources.

18. The one or more non-transitory computer-readable media of claim 10 , wherein the instructions further comprise instructions for: prior to executing the corresponding particular data processing job, storing the code for processing the particular data processing job, the input datasets for the particular data processing job, and the plurality of configuration parameter values in a code repository with version control.