Collision Evaluation For Log-Based Simulations

1. A method of testing software for operating a vehicle in an autonomous driving mode, the method comprising: accessing, by one or more processors from a storage system, log data collected by the vehicle; running, by the one or more processors, a first simulation using the software to control a first simulated vehicle; determining, by the one or more processors, whether a collision or a near collision associated with the first simulated vehicle and an agent vehicle identified by the log data has occurred; when the collision or the near collision is determined to have occurred, running, by the one or more processors, a second simulation using the log data to replace the agent vehicle with a model agent and using the software to control a second simulated vehicle; and determining, by the one or more processors, whether a collision or a near collision between the second simulated vehicle and the model agent has occurred.

2. The method of claim 1 , wherein the determining whether the collision or the near collision between the second simulated vehicle and the model agent has occurred includes monitoring, by the one or more processors, the second simulation to identify any collisions or near collisions between the second simulated vehicle and the model agent.

3. The method of claim 2 , wherein when no collisions or near collisions are identified, determining that the software has “passed” the second simulation.

4. The method of claim 2 , wherein when no collisions or near collisions are identified, determining that the software is validated for the second simulation.

5. The method of claim 1 , further comprising storing a plurality of model agents in the storage system; and selecting the model agent from the plurality of model agents stored in the storage system based on details of the model agent defined in the log data used to run the second simulation.

6. The method of claim 1 , further comprising: inserting characteristics of the agent vehicle in the log data into the model agent.

7. The method of claim 6 , wherein the characteristics include a location of the agent vehicle.

8. The method of claim 6 , wherein the characteristics include a heading of the agent vehicle.

9. The method of claim 6 , wherein the characteristics include an orientation of the agent vehicle.

10. The method of claim 6 , wherein the characteristics include a speed of the agent vehicle.

11. Apparatus for testing software for operating a vehicle in an autonomous driving mode, the apparatus comprising: a storage system; and one or more processors coupled with the storage system, wherein the one or more processors are configured to: access, from the storage system, log data collected by the vehicle; run a first simulation using the software to control a first simulated vehicle; determine whether a collision or a near collision associated with the first simulated vehicle and an agent vehicle identified by the log data has occurred; when the collision or the near collision is determined to have occurred, run a second simulation using the log data to replace the agent vehicle with a model agent and using the software to control a second simulated vehicle; and determine whether a collision or a near collision between the second simulated vehicle and the model agent has occurred.

12. The apparatus of claim 11 , wherein the one or more processors are configured to monitor the second simulation to identify any collisions or near collisions between the second simulated vehicle and the model agent.

13. The apparatus of claim 12 , wherein when no collisions or near collisions are identified, the one or more processors are configured to determine that the software has “passed” the second simulation.

14. The apparatus of claim 12 , wherein when no collisions or near collisions are identified, the one or more processors are configured to validate the software for the second simulation.

15. The apparatus of claim 11 , wherein the one or more processors are configured to store a plurality of model agents in the storage system, and select the model agent from the plurality of model agents stored in storage system based on details of the model agent defined in the log data used to run the second simulation.

16. The apparatus of claim 11 , wherein the one or more processors are configured to insert characteristics of the agent vehicle in the log data into the model agent.

17. The apparatus of claim 16 , wherein the characteristics include a location of the agent vehicle.

18. The apparatus of claim 16 , wherein the characteristics include a heading of the agent vehicle.

19. The apparatus of claim 16 , wherein the characteristics include an orientation of the agent vehicle.

20. The apparatus of claim 16 , wherein the characteristics include a speed of the agent vehicle.