Generalizable Medical Image Analysis Using Segmentation and Classification Neural Networks

1. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to implement: a first set of multiple segmentation neural networks, wherein each segmentation neural network in the first set has the same architecture but has been trained (i) on differently permuted training images, (ii) with differently initialized parameters, or (iii) both, from each other segmentation neural network in the first set, wherein each segmentation neural network in the first set is configured to: receive an input image of eye tissue captured using a first imaging modality; and process the input image to generate a segmentation map that segments the eye tissue in the input image into a plurality of tissue types; a set of multiple classification neural networks, wherein each classification neural network in the set has the same architecture but has been trained (i) on differently permuted training classification inputs, (ii) with differently initialized parameters, or (iii) both, from each other classification neural network in the set, wherein each classification neural network is configured to: receive a classification input derived from a segmentation map of eye tissue; and process the classification input to generate a classification output that characterizes the eye tissue; and a subsystem configured to: receive a first image of eye tissue captured using the first imaging modality; provide the first image as input to each of the segmentation neural networks in the first set to obtain one or more segmentation maps of the eye tissue in the first image; generate, from each of the segmentation maps, a respective classification input; and provide, for each of the segmentation maps, the classification input for the segmentation map as input to each of the classification neural networks to obtain, for each segmentation map, a respective classification output from each classification neural network; and generate, from the respective classification outputs for each of the segmentation maps, a final classification output for the first image.

2. The system of claim 1 , wherein the input image of eye tissue captured using the first imaging modality is a three-dimensional image comprising a plurality of voxels, and wherein the segmentation map assigns a respective tissue type from a predetermined set of tissue types to each of the voxels.

3. The system of claim 1 , wherein the first imaging modality is an Optical Coherence Tomography (OCT) scanner.

4. The system of claim 1 , wherein the subsystem is further configured to: provide a representation of at least one of the segmentation maps for presentation on a user device.

5. The system of claim 4 , wherein the representation of the segmentation map includes, for each of the plurality of tissue types, a two-dimensional thickness map overlaid on a projection of the first image.

6. The system of claim 4 , wherein the representation of the segmentation map includes a three-dimensional representation of the tissue that differentiates between tissue of different types as identified in the segmentation map.

7. The system of claim 1 , wherein the classification input for a given segmentation map is a down-sampled version of the given segmentation map, and wherein generating, from each of the segmentation maps, a respective classification input comprises down-sampling the segmentation map to generate the classification input.

8. The system of claim 1 , wherein the classification output that characterizes the eye tissue comprises a respective referral score for each of a plurality of referral decisions that represents a predicted likelihood that the referral decision is the most appropriate referral decision for a patient given a current state of the eye tissue.

9. The system of claim 8 , wherein generating the final classification output comprises combining the referral scores generated by the classification neural networks to generate a final referral score for each of the referral decisions that represents a final predicted likelihood that the referral decision is the most appropriate referral decision for the patient given the current state of the eye tissue.

10. The system of claim 1 , wherein the classification output that characterizes the eye tissue comprises a respective condition score for each of one or more eye-related conditions that represents a predicted likelihood that a patient has the condition given a current state of the eye tissue.

11. The system of claim 10 , wherein generating the final classification output comprises combining the condition scores generated by the classification neural networks to generate a final condition score for each of the conditions that represents a final predicted likelihood that the patient has the condition.

12. The system of claim 1 , wherein the classification output that characterizes the eye tissue comprises a respective progression score for each of one or more condition states that represents a predicted likelihood that a state of a corresponding eye-related condition will progress to the condition state at a particular future time given a current state of the eye tissue.

13. The system of claim 12 , wherein generating the final classification output comprises combining the progressions scores generated by the classification neural networks to generate a final progression score for each of the conditions that represents a final predicted likelihood that the state of a corresponding eye-related condition will progress to the condition state at the particular future time.

14. The system of claim 1 , wherein the classification output that characterizes the eye tissue comprises a respective treatment score for each of a plurality of treatments that represents a predicted likelihood that the treatment is the best treatment for a patient given a current state of the eye tissue.

15. The system of claim 14 , wherein generating the final classification output comprises combining the treatment scores generated by the classification neural networks to generate a final treatment score for each of the treatments that represents a final predicted likelihood that that the treatment is the best treatment for the patient.

16. The system of claim 1 , wherein the subsystem is further configured to: provide the final classification output for presentation on a user device.

17. The system of claim 1 , wherein each segmentation neural network in the first set is a convolutional neural network having a U-Net architecture.

18. The system of claim 1 , wherein each classification neural network comprises three-dimensional densely connected convolutional blocks.

19. The system of claim 1 , wherein the instructions further cause the one or more computers to implement: a second set of one or more segmentation neural networks, wherein each segmentation neural network in the second set is configured to: receive an input image of eye tissue captured using a second, different imaging modality; and process the input image to generate a segmentation map that segments the eye tissue in the input image into a plurality of tissue types; and wherein the subsystem is further configured to: receive a second image of eye tissue captured using the second imaging modality; provide the second image as input to each of the segmentation neural networks in the second set to obtain one or more segmentation maps of the eye tissue in the second image; generate, from each of the segmentation maps, a respective classification input; and provide, for each of the segmentation maps, the classification input as input to each of the classification neural networks to obtain, for each segmentation map, a respective classification output from each classification neural network; and generate, from the respective classification outputs for each of the segmentation maps, a final classification output for the second image.

20. One or more computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising: receiving a first image of eye tissue captured using a first imaging modality; providing the first image as input to each of multiple segmentation neural networks to obtain multiple segmentation maps of the eye tissue in the first image, wherein each segmentation neural network has the same architecture but has been trained (i) on differently permuted training images, (ii) with differently initialized parameters, or (iii) both, from each other segmentation neural network, wherein each segmentation neural network is configured to: receive an input image of eye tissue captured using the first imaging modality; and process the input image to generate a segmentation map that segments the eye tissue in the input image into a plurality of tissue types; generating, from each of the segmentation maps, a respective classification input; providing, for each of the segmentation maps, the classification input for the segmentation map as input to each of multiple classification neural networks to obtain, for each segmentation map, a respective classification output from each classification neural network, wherein each classification neural network has the same architecture but has been trained (i) on differently permuted training classification inputs, (ii) with differently initialized parameters, or (iii) both, from each other classification neural network, wherein each classification neural network is configured to: receive a classification input derived from a segmentation map of eye tissue; and process the classification input to generate a classification output that characterizes the eye tissue; and generating, from the respective classification outputs for each of the segmentation maps, a final classification output for the first image.

21. A method comprising: receiving a first image of eye tissue captured using a first imaging modality; providing the first image as input to each of multiple segmentation neural networks to obtain one or more segmentation maps of the eye tissue in the first image, wherein each segmentation neural network has the same architecture but has been trained (i) on differently permuted training images, (ii) with differently initialized parameters, or (iii) both, from each other segmentation neural network, wherein each segmentation neural network is configured to: receive an input image of eye tissue captured using the first imaging modality; and process the input image to generate a segmentation map that segments the eye tissue in the input image into a plurality of tissue types; generating, from each of the segmentation maps, a respective classification input; providing, for each of the segmentation maps, the classification input for the segmentation map as input to each of multiple classification neural networks to obtain, for each segmentation map, a respective classification output from each classification neural network, wherein each classification neural network has the same architecture but has been trained (i) on differently permuted training classification inputs, (ii) with differently initialized parameters, or (iii) both, from each other classification neural network, wherein each classification neural network is configured to: receive a classification input derived from a segmentation map of eye tissue; and process the classification input to generate a classification output that characterizes the eye tissue; and generating, from the respective classification outputs for each of the segmentation maps, a final classification output for the first image.