Legal claims defining the scope of protection, as filed with the USPTO.
1. A computer-implemented method comprising: receiving, by a first trusted process a) executing in a first trusted processing environment and b) that manages data access for an untrusted processing environment, a data flow graph for an untrusted application, executing in the untrusted processing environment, that identifies one or more data objects and one or more untrusted functions; and managing, by a second trusted process i) executing in a second trusted processing environment and ii) that manages data access for an untrusted worker environment, execution of the untrusted functions to produce materialized data objects corresponding to the data objects by: determining, by the second trusted process, input data for one of the untrusted functions using the data objects; providing, by the second trusted process to an untrusted worker process in the untrusted worker environment, the input data to cause the untrusted worker process to execute the one of the untrusted functions using the input data; and receiving, by the second trusted process, output data generated by the untrusted worker process using the one of the untrusted functions and the input data.
2. The method of claim 1 comprising: transforming, by the first trusted process executing in the first trusted processing environment, the data flow graph into a revised data flow graph for the untrusted application, the revised data flow graph including one or more deferred data objects and one or more deferred operations each corresponding to one or more untrusted functions called by the untrusted application, wherein: managing, by the second trusted process i) executing the second trusted processing environment and ii) that manages data access for the untrusted worker environment, execution of the untrusted functions to produce the materialized data objects corresponding to the data objects comprises managing, by the second trusted process, execution of the untrusted functions corresponding to the deferred operations to produce the materialized data objects corresponding to the deferred data objects.
3. The method of claim 2 wherein: receiving, by the first trusted process a) executing in the first trusted processing environment and b) that manages data access for the untrusted processing environment, the data flow graph for the untrusted application, executing in the untrusted processing environment, that identifies the one or more data objects and the one or more untrusted functions comprises receiving the data flow graph that includes a data parallel pipeline used to produce the data flow graph; transforming, by the first trusted process executing the first trusted processing environment, the data flow graph into a revised data flow graph for the untrusted application comprises transforming the data flow graph into the revised data flow graph that includes one or more deferred data objects and one or more deferred parallel operations each corresponding to one or more untrusted functions called by the untrusted application; and managing, by the second trusted process, execution of the untrusted functions corresponding to the deferred operations to produce the materialized data objects corresponding to the deferred data objects comprises managing execution of the untrusted functions corresponding to the deferred parallel operations to produce materialized parallel data objects corresponding to the deferred data objects.
4. The method of claim 1 comprising: managing, by the first trusted process, data access for the untrusted application executing on a first virtual machine; and managing, by the second trusted process, data access for the untrusted worker process executing on a second virtual machine.
5. The method of claim 1 wherein: determining, by the second trusted process, the input data for the one of the untrusted functions using the data objects comprises determining an input batch of records that includes multiple, individual input records; providing, by the second trusted process to the untrusted worker process in the untrusted worker environment, the input data to cause the untrusted worker process to execute the one of the untrusted functions using the input data comprises providing the input batch of records to the untrusted worker process to cause the untrusted worker process to execute the one of the untrusted functions using the input batch of records; and receiving, by the second trusted process, the output data generated by the untrusted worker process using the one of the untrusted functions and the input data comprises collecting output records, received from the untrusted worker process, into an output batch.
6. The method of claim 1 comprising: receiving, from a client system, a request for execution of the untrusted application; and sending the output data to the client system.
7. The method of claim 1 wherein receiving, by the first trusted process a) executing in the first trusted processing environment and b) that manages data access for the untrusted processing environment, the data flow graph for the untrusted application comprises receiving, from the untrusted processing environment, a remote procedure call that identifies the data flow graph.
8. The method of claim 1 comprising determining, by the first trusted process executing in the first trusted processing environment, whether each of the one or more untrusted functions is valid, wherein providing, by the second trusted process to the untrusted worker process in the untrusted worker environment, the input data to cause the untrusted worker process to execute the one of the untrusted functions using the input data is responsive to determining that the one of the untrusted functions is valid.
9. A non-transitory computer-readable medium storing instructions operable when executed to cause at least one processor to perform operations comprising: receiving, by a first trusted process a) executing in a first trusted processing environment and b) that manages data access for an untrusted processing environment, a data flow graph for an untrusted application, executing in the untrusted processing environment, that identifies one or more data objects and one or more untrusted functions; and managing, by a second trusted process i) executing in a second trusted processing environment and ii) that manages data access for an untrusted worker environment, execution of the untrusted functions to produce materialized data objects corresponding to the data objects by: determining, by the second trusted process, input data for one of the untrusted functions using the data objects; providing, by the second trusted process to an untrusted worker process in the untrusted worker environment, the input data to cause the untrusted worker process to execute the one of the untrusted functions using the input data; and receiving, by the second trusted process, output data generated by the untrusted worker process using the one of the untrusted functions and the input data.
10. The computer-readable medium of claim 9 the operations comprising: transforming, by the first trusted process executing in the first trusted processing environment, the data flow graph into a revised data flow graph for the untrusted application, the revised data flow graph including one or more deferred data objects and one or more deferred operations each corresponding to one or more untrusted functions called by the untrusted application, wherein: managing, by the second trusted process i) executing the second trusted processing environment and ii) that manages data access for the untrusted worker environment, execution of the untrusted functions to produce the materialized data objects corresponding to the data objects comprises managing, by the second trusted process, execution of the untrusted functions corresponding to the deferred operations to produce the materialized data objects corresponding to the deferred data objects.
11. The computer-readable medium of claim 10 wherein: receiving, by the first trusted process a) executing in the first trusted processing environment and b) that manages data access for the untrusted processing environment, the data flow graph for the untrusted application, executing in the untrusted processing environment, that identifies the one or more data objects and the one or more untrusted functions comprises receiving the data flow graph that includes a data parallel pipeline used to produce the data flow graph; transforming, by the first trusted process executing the first trusted processing environment, the data flow graph into a revised data flow graph for the untrusted application comprises transforming the data flow graph into the revised data flow graph that includes one or more deferred data objects and one or more deferred parallel operations each corresponding to one or more untrusted functions called by the untrusted application; and managing, by the second trusted process, execution of the untrusted functions corresponding to the deferred operations to produce the materialized data objects corresponding to the deferred data objects comprises managing execution of the untrusted functions corresponding to the deferred parallel operations to produce materialized parallel data objects corresponding to the deferred data objects.
12. The computer-readable medium of claim 9 the operations comprising: managing, by the first trusted process, data access for the untrusted application executing on a first virtual machine; and managing, by the second trusted process, data access for the untrusted worker process executing on a second virtual machine.
13. The computer-readable medium of claim 9 wherein: determining, by the second trusted process, the input data for the one of the untrusted functions using the data objects comprises determining an input batch of records that includes multiple, individual input records; providing, by the second trusted process to the untrusted worker process in the untrusted worker environment, the input data to cause the untrusted worker process to execute the one of the untrusted functions using the input data comprises providing the input batch of records to the untrusted worker process to cause the untrusted worker process to execute the one of the untrusted functions using the input batch of records; and receiving, by the second trusted process, the output data generated by the untrusted worker process using the one of the untrusted functions and the input data comprises collecting output records, received from the untrusted worker process, into an output batch.
14. The computer-readable medium of claim 9 the operations comprising: receiving, from a client system, a request for execution of the untrusted application; and sending the output data to the client system.
15. The computer-readable medium of claim 9 wherein receiving, by the first trusted process a) executing in the first trusted processing environment and b) that manages data access for the untrusted processing environment, the data flow graph for the untrusted application comprises receiving, from the untrusted processing environment, a remote procedure call that identifies the data flow graph.
16. The computer-readable medium of claim 9 the operations comprising determining, by the first trusted process executing in the first trusted processing environment, whether each of the one or more untrusted functions is valid, wherein providing, by the second trusted process to the untrusted worker process in the untrusted worker environment, the input data to cause the untrusted worker process to execute the one of the untrusted functions using the input data is responsive to determining that the one of the untrusted functions is valid.
17. A system comprising one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising: receiving, by a first trusted process a) executing in a first trusted processing environment and b) that manages data access for an untrusted processing environment, a data flow graph for an untrusted application, executing in the untrusted processing environment, that identifies one or more data objects and one or more untrusted functions; and managing, by a second trusted process i) executing in a second trusted processing environment and ii) that manages data access for an untrusted worker environment, execution of the untrusted functions to produce materialized data objects corresponding to the data objects by: determining, by the second trusted process, input data for one of the untrusted functions using the data objects; providing, by the second trusted process to an untrusted worker process in the untrusted worker environment, the input data to cause the untrusted worker process to execute the one of the untrusted functions using the input data; and receiving, by the second trusted process, output data generated by the untrusted worker process using the one of the untrusted functions and the input data.
18. The system of claim 17 the operations comprising: transforming, by the first trusted process executing in the first trusted processing environment, the data flow graph into a revised data flow graph for the untrusted application, the revised data flow graph including one or more deferred data objects and one or more deferred operations each corresponding to one or more untrusted functions called by the untrusted application, wherein: managing, by the second trusted process i) executing the second trusted processing environment and ii) that manages data access for the untrusted worker environment, execution of the untrusted functions to produce the materialized data objects corresponding to the data objects comprises managing, by the second trusted process, execution of the untrusted functions corresponding to the deferred operations to produce the materialized data objects corresponding to the deferred data objects.
19. The system of claim 18 wherein: receiving, by the first trusted process a) executing in the first trusted processing environment and b) that manages data access for the untrusted processing environment, the data flow graph for the untrusted application, executing in the untrusted processing environment, that identifies the one or more data objects and the one or more untrusted functions comprises receiving the data flow graph that includes a data parallel pipeline used to produce the data flow graph; transforming, by the first trusted process executing the first trusted processing environment, the data flow graph into a revised data flow graph for the untrusted application comprises transforming the data flow graph into the revised data flow graph that includes one or more deferred data objects and one or more deferred parallel operations each corresponding to one or more untrusted functions called by the untrusted application; and managing, by the second trusted process, execution of the untrusted functions corresponding to the deferred operations to produce the materialized data objects corresponding to the deferred data objects comprises managing execution of the untrusted functions corresponding to the deferred parallel operations to produce materialized parallel data objects corresponding to the deferred data objects.
20. The system of claim 17 the operations comprising: managing, by the first trusted process, data access for the untrusted application executing on a first virtual machine; and managing, by the second trusted process, data access for the untrusted worker process executing on a second virtual machine.
Unknown
June 13, 2017
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.