Implementations are disclosed for analyzing extracted individual plant components to make agronomic inferences about entire composite plant organs from which the individual plant components were harvested, and for using those agronomic inferences for various purposes. In various implementations, individual plant component(s) may be sampled from multiple plant components removed from composite plant organ(s) that previously included the plant components. Digital image(s) may be captured of the sampled individual plant component(s) and processed based on machine learning model(s) to generate agronomic inference(s) about the composite plant organ(s) that previously included the plurality of plant components. Based on the agronomic inference(s), computing device(s) may render output that includes a diagnosis or recommendation for the grower about the field or the crops, and/or agricultural equipment (e.g., robots) may be operated automatically.
Legal claims defining the scope of protection, as filed with the USPTO.
sampling a plurality of individual plant components from a plurality of plant components removed from one or more composite plant organs that previously included the plurality of plant components, wherein the one or more composite plant organs are part of crops in a field managed by a grower; capturing one or more digital images of the sampled one or more plurality of individual plant components; generating a plurality of individual plant component embeddings that represent the sampled plurality of individual plant components within the one or more digital images; determining similarity measures between the plurality of individual plant component embeddings and individual plant component reference embeddings generated previously based on the plurality of reference individual plant components; and based on the similarity measures and at least some of the ground truth agronomic observations about one or more of the reference composite plant organs that yielded the plurality of reference individual plant components, generating one or more agronomic inferences about the one or more composite plant organs that previously included the plurality of plant components; and processing the one or more digital images based on one or more machine learning models, wherein one or more of the machine learning models was trained previously based on training data associating a plurality of reference individual plant components with ground truth agronomic observations about one or more reference composite plant organs that yielded the plurality of reference individual plant components, wherein the processing includes: based on the one or more agronomic inferences, causing one or more computing devices to render output that includes a diagnosis or recommendation for the grower about the field or the crops. . A method implemented using one or more processors and comprising:
claim 1 . The method of, wherein the output comprises includes subfield recommendations for the field managed by the grower.
claim 1 . The method of, wherein the output includes one or more mid-crop-cycle agronomic recommendations.
claim 1 . The method of, wherein the output includes a local environmental zone map for the field.
claim 1 . The method of, wherein the plurality of individual plant components are corn kernels, and the one or more composite plant organs are ears of corn.
claim 1 . The method of, wherein the plurality of individual plant components are wheat seeds, and the one or more composite plant organs are heads of wheat.
claim 1 . The method of, wherein the plurality of individual plant components are sampled by a component onboard a combine harvester operating in the field.
claim 1 . The method of, wherein the processing includes segmenting the sampled plurality of individual plant components within the one or more digital images.
claim 1 determining second similarity measures between the plurality of individual plant component embeddings; and based on the second similarity measures, clustering the plurality of individual plant component embeddings into clusters, with each cluster representing an archetype composite plant organ. . The method of, wherein the similarity measures are first similarity measures, and further including:
sample a plurality of individual plant components from a plurality of plant components removed from one or more composite plant organs that previously included the plurality of plant components, wherein the one or more composite plant organs are part of crops in a field managed by a grower; capture one or more digital images of the sampled one or more plurality of individual plant components; generate a plurality of individual plant component embeddings that represent the sampled plurality of individual plant components within the one or more digital images; determine similarity measures between the plurality of individual plant component embeddings and individual plant component reference embeddings generated previously based on the plurality of reference individual plant components; and based on the similarity measures and at least some of the ground truth agronomic observations about one or more of the reference composite plant organs that yielded the plurality of reference individual plant components, generate one or more agronomic inferences about the one or more composite plant organs that previously included the plurality of plant components; and process the one or more digital images based on machine learning models, wherein one or more of the machine learning models was trained previously based on training data associating a plurality of reference individual plant components with ground truth agronomic observations about one or more reference composite plant organs that yielded the plurality of reference individual plant components, wherein to process includes to: based on the one or more agronomic inferences, cause one or more computing devices to render output that includes a diagnosis or recommendation for the grower about the field or the crops. . A system comprising one or more processors and memory storing instructions that, in response to execution by the one or more processors, cause the one or more processors to:
claim 10 . The system of, wherein the output includes subfield recommendations for the field managed by the grower, one or more mid-crop-cycle agronomic recommendations, or a local environmental zone map for the field.
claim 10 . The system of, wherein the plurality of individual plant components are corn kernels, and the one or more composite plant organs are ears of corn.
claim 10 . The system of, wherein the plurality of individual plant components are wheat seeds, and the one or more composite plant organs are heads of wheat.
claim 10 . The system of, wherein the plurality of individual plant components are sampled by a component onboard a combine harvester operating in the field.
claim 10 . The system of, wherein the processing includes segmenting the sampled plurality of individual plant components within the one or more digital images.
obtain one or more digital images of a plurality of individual plant components that were sampled from a plurality of plant components removed from one or more composite plant organs that previously included the plurality of plant components, wherein the one or more composite plant organs are part of crops in a field managed by a grower; generate a plurality of individual plant component embeddings that represent the sampled plurality of individual plant components within the one or more digital images: determine similarity measures between the plurality of individual plant component embeddings and individual plant component reference embeddings generated previously based on the plurality of reference individual plant components; and based on the similarity measures and at least some of the ground truth agronomic observations about one or more of the reference composite plant organs that yielded the plurality of reference individual plant components, generate one or more agronomic inferences about the one or more composite plant organs that previously included the plurality of plant components; and process the one or more digital images based on one or more machine learning models, wherein one or more of the machine learning models was trained previously based on training data associating a plurality of reference individual plant components with ground truth agronomic observations about one or more reference composite plant organs that yielded the plurality of reference individual plant components, wherein to process includes to: based on the one or more agronomic inferences, cause one or more computing devices to render output that includes a diagnosis or recommendation for the grower about the field or the crops. . At least one non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to:
claim 16 . The non-transitory computer-readable medium of, wherein the output includes subfield recommendations for the field managed by the grower, one or more mid-crop-cycle agronomic recommendations, or a local environmental zone map for the field.
claim 16 . The non-transitory computer-readable medium of, wherein the plurality of individual plant components are corn kernels, and the one or more composite plant organs are ears of corn.
claim 16 . The non-transitory computer-readable medium of, wherein the plurality of individual plant components are wheat seeds, and the one or more composite plant organs are heads of wheat.
claim 16 . The non-transitory computer-readable medium of, wherein the plurality of individual plant components are sampled by a component onboard a combine harvester operating in the field.
Complete technical specification and implementation details from the patent document.
In many large-scale agricultural fields, crop yield may be one of the few data points that can be collected reliably and/or economically. For example, yield monitors may be operated onboard combine harvesters to measure the rate at which clean grain enters a grain tank. This measurement conveys how much of the plant component of interest—corn kernels, wheat seeds, etc.—was obtained, but does not provide any explanation as to why that amount was obtained.
In many cases it is possible to determine causes and/or contributing factors of crop yield by examining physical characteristics of larger portions of crops, such as entire plants or composite plant organs. For example, the visual appearance of an ear of corn can be observed to identify conditions such as phosphate shortages, insufficient population, nitrogen deficiency/surplus, low fertility, potash shortage, or over/under hydration, to name a few. Before such observation can be performed, however, individual plant components such as corn kernels or wheat seeds often are removed from composite plant organs and intermixed. Consequently, it may no longer be possible to use conventional techniques to perform analytics on composite plant organs or based on specific locations, e.g., because any geospatial references of individual plant components are lost.
Implementations are described herein for analyzing extracted individual plant components to make agronomic inferences (e.g., predictions) about entire composite plant organs such as ears of corn or heads of wheat from which the individual plant components were harvested. Additionally, techniques are described herein for using these agronomic inferences to generate subfield diagnoses, recommendations, and/or to control automated agricultural equipment.
In various implementations, a method may be implemented using one or more processors and may include: sampling one or more individual plant components from a plurality of plant components removed from one or more composite plant organs that previously included the plurality of plant components, wherein the one or more composite plant organs are part of crops in a field managed by a grower; capturing one or more digital images of the sampled one or more individual plant components; processing the one or more digital images based on one or more machine learning models to generate one or more agronomic inferences about the one or more composite plant organs that previously included the plurality of plant components, wherein one or more of the machine learning models was trained previously based on training data associating reference individual plant components with ground truth agronomic observations about one or more reference composite plant organs that yielded the reference individual plant components; and based on the one or more agronomic inferences, causing one or more computing devices to render output that includes a diagnosis or recommendation for the grower about the field or the crops.
In various implementations, the output may include subfield recommendations for the field managed by the grower, one or more mid-crop-cycle agronomic recommendations, and/or a local environmental zone map for the field, to name a few examples. In various implementations, the individual plant components may be corn kernels, and the composite plant organs may be ears of corn. In various implementations, the individual plant components may be wheat seeds, and the composite plant organs may be heads of wheat.
In various implementations, the one or more individual plant components may be sampled by a component onboard a combine harvester operating in the field. In various implementations, the processing may include segmenting the sampled one or more individual plant components within the one or more digital images.
In various implementations, the sampled one or more individual plant components may include a sampled plurality of individual plant components, and the processing may include generating a plurality of individual plant component embeddings that represent the sampled plurality of individual plant components within the one or more digital images.
In various implementations, the method may include: determining similarity measures between the plurality of individual plant component embeddings and individual plant component reference embeddings generated previously based on the plurality of reference individual plant components; and based on the similarity measures and at least some of the ground truth agronomic observations about one or more of the reference composite plant organs that yielded the reference individual plant components, generating one or more of the agronomic inferences about the one or more composite plant organs that previously included the plurality of plant components.
In various implementations, the method may include determining similarity measures between the plurality of individual plant component embeddings; and based on the similarity measures, clustering the plurality of individual plant component embeddings into clusters, with each cluster representing an archetype composite plant organ.
In addition, some implementations include one or more processors (e.g., central processing unit(s) (CPU(s)), graphics processing unit(s) (GPU(s), and/or tensor processing unit(s) (TPU(s)) of one or more computing devices, where the one or more processors are operable to execute instructions stored in associated memory, and where the instructions are configured to cause performance of any of the aforementioned methods. Some implementations also include one or more non-transitory computer readable storage media storing computer instructions executable by one or more processors to perform any of the aforementioned methods. Yet other implementations include agricultural vehicles, such as robots, that are equipped with edge processor(s) configured to carry out selected aspects of the present disclosure.
It should be appreciated that all combinations of the foregoing concepts and additional concepts described in greater detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein.
Implementations are described herein for analyzing extracted individual plant components to make agronomic inferences (e.g., predictions) about entire composite plant organs such as ears of corn or heads of wheat from which the individual plant components were harvested. Additionally, techniques are described herein for using these agronomic inferences to generate subfield diagnoses, recommendations, and/or to control automated agricultural equipment.
In various implementations, individual plant components that have been separated from composite plant organs—corn kernels from ears of corn or wheat seeds from heads of wheat, for instance—may be sampled, e.g., using a grain isolation mechanism, such as a seed meter, which is deployed as part of a combine harvester. Digital images capturing these sampled individual plant components may then be processed using one or more vision techniques (e.g., rules-based vision techniques such as adaptive thresholding, contour detection, or the watershed algorithm, or techniques based on the application of machine learning model(s)) to generate agronomic inference(s). These agronomic inference(s) may include, for instance, inferences or predictions about an entire composite plant organ from which the individual plant components were harvested. With ears of corn, the agronomic inference(s) may include inferences that are normally made based on visual evaluation of entire ears of corn, such as predictions regarding abiotic and/or biotic factors, phosphate shortages, insufficient population, nitrogen deficiency/surplus, low fertility, potash shortage, over/under hydration, and/or plant-to-plant competition, to name a few.
In some implementations, agronomic inference(s) generated using these various vision techniques may be used to provide subfield recommendations and/or guidance to growers about specific portions of fields. As a combine harvester is traversed through a field, its position may be tracked, e.g., using technologies such as the Global Positioning System (GPS) to obtain position coordinates periodically/continuously. The harvester's tracked position coordinates may be at least loosely associated with contemporaneously harvested individual plant components. The agronomic inferences generated from digital images depicting these sampled individual plant components may then also be associated with the position coordinates, and hence, specific portions of a field. Thus, for instance, a grower may receive subfield recommendations for the field managed by the grower, e.g., as a heat map or a map segmented into portions for which different recommendations and/or diagnoses are made.
As noted previously, rules-based vision techniques (e.g., adaptive thresholding, contour detection, watershed) and/or machine-learning vision techniques may be employed to generate agronomic inferences. In the latter case, various types of machine learning models may be trained to facilitate generation of various types of agronomic inferences and/or predictions. In some implementations, one or more convolutional neural networks (CNNs) of various configurations may be trained to segment the images into different semantic portions, including portions representing individual plant components. Additionally or alternatively, the CNN(s) may be trained to extract various features of those segmented individual plant components, e.g., in the form of discrete and/or continuous embeddings or vectors.
These extracted features (e.g., embeddings) may be mapped, e.g., by what will be referred to herein as “mapping” machine learning model(s) (or layers of larger machine learning models), to ground truth agronomic observations about reference individual plant components. The reference individual plant components may not have been harvested from composite plant organs until after ground truth agronomic observations about the composite plant organs (e.g., detected physical traits, diagnoses, measurements, etc.) were obtained by humans, machines, robots, etc. Consequently, the mapping machine learning model(s) can be trained by applying data indicative of the reference individual plant components as inputs into the mapping machine learning model(s) to generate predicted agronomic inference(s), and training the mapping machine learning model(s) based on comparing the predicted agronomic inference(s) to the ground truth agronomic observations, e.g., using techniques such as gradient descent, back propagation, cross entropy, etc. Once trained, the mapping machine learning models can be applied to data indicative of sampled individual plant components (e.g., for which little or no ground truth data is available about composite plant organs) to generate agronomic inferences(s) about the composite plant organs from which those individual plant components were harvested. Various types of machine learning models may be trained as the mapping machine learning model(s), such as a multilayer perceptron (e.g., a neural network), transformer (of a type sometimes used for large language models), hidden Markov model(s), CNNs, support vector machines, etc.
In some implementations, individual plant components may be grouped into clusters based on their similarities to each other. Each cluster may represent an inferred composite plant organ that is an “archetype” that models “real” composite plant organ(s) from which individual plant components of the cluster likely would have been extracted. For example, a plurality of individual corn kernel embeddings may be generated to represent a sampled plurality of individual corn kernels that are depicted in digital image(s) captured in a combine harvester. Similarity measures may be determined between the plurality of individual corn kernel embeddings, e.g., using techniques such as Euclidean distance, dot product, cosine similarity, etc. The plurality of individual corn kernel embeddings may then be grouped into clusters of similar corn kernel embeddings. Each cluster may represent an archetype ear of corn that would likely have yielded corn kernels having those qualities found in the cluster.
In some implementations, individual plant component embeddings may be used to determine agronomic inference(s) about composite plant parts. For example, similarity measures may be determined between the plurality of individual corn kernel embeddings and reference individual corn kernel embeddings generated previously based on a plurality of reference corn kernels for which ground truth agronomic observations are available. Based on these similarity measures, suitable ground truth agronomic observations may be attributed to the individual corn kernel embeddings generated based on the sample corn kernels.
1 FIG. 1 FIG. schematically illustrates one example environment in which one or more selected aspects of the present disclosure may be implemented, in accordance with various implementations. The example environment depicted inrelates to the agriculture domain, but this is not meant to be limiting. Techniques described here may be useful in any domain in which individual components are extracted from larger composite components.
1 FIG. 1 FIG. 102 104 110 102 106 112 108 112 102 The environment ofincludes one or more farmsand an agronomic inference systemconnected by one or more computer networks. Farmalso includes one or more client devices, one or more fieldsthat are used to grow one or more crops, and agricultural equipment such as a combine harvesterthat is configured, among other things, to capture images depicting individual plant components as described herein. Field(s)may be used to grow various types of crops that may produce plant parts of economic and/or nutritional interest. These crops may include but are not limited to strawberries, tomato plants, soybeans, corn, lettuce, spinach, beans, cherries, nuts, cereal grains (e.g., wheat), berries, rice, flax, grapes, and so forth. One farmis depicted in detail infor illustrative purposes. However, there may be any number of farms for which agronomic inferences generated as described herein may be useful.
106 106 1 FIG. An individual (which in the current context may also be referred to as a “user”) may operate a client deviceto interact with other components depicted in. Each client devicemay be, for example, a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device of a vehicle of the participant (e.g., an in-vehicle communications system, an in-vehicle entertainment system, an in-vehicle navigation system), a standalone interactive speaker (with or without a display), or a wearable apparatus that includes a computing device, such as a head-mounted display (“HMD”) that provides an AR or VR immersive computing experience, a “smart” watch, and so forth. Additional and/or alternative client devices may be provided.
104 106 104 106 104 106 106 107 Agronomic inference systemcomprises a non-limiting example of a computing system on which techniques described herein may be implemented. Each of client devicesand agronomic inference systemmay include one or more memories for storage of data and software applications, one or more processors for accessing data and executing applications, and other components that facilitate communication over a network. The computational operations performed by client deviceand/or agronomic inference systemmay be distributed across multiple computer systems. Client devicemay operate a variety of different applications that may be used, for instance, to analyze various agricultural inferences. For example, client deviceoperates an application(e.g., which may be standalone or part of another application, such as part of a web browser), which a user can use to trigger generation of and/or view agronomic inferences generated as described herein.
104 116 118 122 104 114 120 116 118 122 114 108 120 118 119 122 116 118 122 104 106 108 104 In various implementations, agronomic inference systemmay include a sampling module, an inference module, and a training module. Agronomic inference systemmay also include one or more databases,for storing various data used by and/or generated by modules,, and/or. For example, databasemay store data such as images, e.g., captured in a combine harvester, that depict individual plant components (e.g., kernels of corn, wheat seeds) stripped from larger composite plant organs (e.g., ears of corn, heads of wheat), other sensor data gathered by farm equipment and/or personnel on the ground, user-input data, weather data, and so forth. Databasemay store machine learning model(s) that are applied by inference moduleto generate agronomic inferencesand/or are trained by training module. In some implementations one or more of modules,, and/ormay be omitted, combined, and/or implemented in a component that is separate from agronomic inference system, such as on client deviceand/or in combine harvester. In some implementations, agronomic inference systemmay be considered cloud-based computing resources as it may be implemented across one or more computing systems that may be referred to as the “cloud.”
116 108 108 114 116 114 116 108 4 FIG. In some implementations, sampling modulemay be configured to sample images that depict individual plant components such as kernels of corn or seeds of wheat that have been extracted (e.g., stripped), e.g., by agricultural equipment such as combine harvester, from larger composite plant organs such as ears of corn or heads of wheat. In some implementations, images captured by combine harvesteror other similar agricultural equipment may be stored in database, and sampling modulemay sample these images from databasefor further processing. In other implementations, sampling modulemay be integral with combine harvesteror other similar agricultural equipment and may be configured to control not only capturing images of individual plant components, but also to control one or mechanical apparatus (one example is depicted in) that are designed to sample individual plant components from a larger stream of plant components being harvested.
118 116 120 119 118 119 Inference modulemay be configured to process images of individual plant components provided by sampling moduleusing one or more machine learning models stored in databaseto generate agronomic inferencesas described herein. Various types of machine learning models may be applied by inference moduleto generate these agronomic inferences, which may include predictions and/or classifications. Additionally, various types of machine learning models may be used to generate semantically rich embeddings that are applied as inputs and/or intermediate representations across the various machine learning models. These various machine learning models may include, but are not limited to, recurrent neural networks (RNNs), long short-term memory (LSTM) networks (including bidirectional), gated recurrent unit (GRU) networks, graph neural networks (GNNs), transformer networks (e.g., the same as or similar to those often used as large language models), feed-forward neural networks, convolutional neural networks (CNNs), support vector machines, random forests, decision trees, etc. As used herein, a “transformer” may include, but is not necessarily limited to, a machine learning model that incorporates a “self-attention” mechanism, and that is usable to process an entire sequence of inputs at once, as opposed to iteratively. One non-limiting example of a transformer is a transformer trained based on the Bidirectional Encoder Representations from Transformers (BERT) concept.
122 118 120 122 Training modulemay be configured to train the various machine learning models used by inference moduleto generate agronomic inferences as described herein. These machine learning models may include those stored in database, as well as other machine learning models that are employed to encode various modalities of input data into embeddings. In various implementations, training modulemay be configured to train transformers and other types of models to generate agronomic inferences based on images of extracted individual plant components such as corn kernels or wheat seeds.
2 FIG.A 2 FIG.A demonstrates the disjointed nature of multi-phase testing, development, and deployment of a varietal of crop commercially. Multiple phases of a typical agricultural operation are depicted in. Genotype, environment, and management (G×E×M) data is obtained from multiple phases of the agricultural operation. For instance, “PHASE 1” shown at left may correspond to early pipeline trials and/or research and development of the agricultural operation in which particular varietals of a crop such as corn or wheat are initially tested, e.g., in a small test bed. The next phase to the right, “PHASE 2,” may correspond to pre-commercial trials of the agricultural operation in which the crop varietals are tested on a somewhat larger scale. The next phase to the right, “PHASE 3,” may correspond to more advanced agronomic development trials in which different management techniques are tested a wider scale to finalize agronomic techniques that will be used commercially. This may continue until “PHASE N,” which may correspond to the crop being grown commercially on a wide scale.
During each phase, a crop yield may be generated. However, as shown by the broken chains, the G×E×M is fragmented between phases, and consequently, the crop yield data is also fragmented. Put another way, there is insufficient data to connect knowledge across the various domains associated with the multiple phases, including the environments and agronomic systems deployed in each. Crop yield alone does not provide sufficient information to diagnose why a specific commercial field or portion thereof (e.g., an acre) is underperforming, overperforming, etc.
2 FIG.B 2 FIG.A 1 FIG. 104 104 119 230 230 230 230 As shown in, agronomic inference systemmay facilitate the linking of G×E×M and yield data across the various phases/domains depicted inusing kernel-level or individual grain/seed-level imagery. This may allow accurate agronomic diagnoses and recommendations to be generated during later phases. In particular, agronomic inference systemis able to capture the G×E×M data at each phase and link it together to generate agronomic inferences (in) such as a mapshowing portions of an agricultural field that exhibit different inferred agronomic traits and/or recommendations. Based on such a map, a grower is able to apply fine-tuned, granular agronomy to diagnose and/or properly manage crops growing in different portions of the field. For example, different colored portions of mapmay correspond to different agronomic recommendations, such as apply more/less fertilizer (e.g., mapmay be a nitrogen prescription map), apply more/less irrigation, treat with particular herbicide or pesticide, etc.
3 FIG. 2 FIG.A 3 FIG. 104 118 332 334 118 332 108 334 334 schematically depicts an example of how to link the previously disjoint data depicted in. Starting at left, agronomic inference system, e.g., by way of inference module(not depicted in), may process digital image(s)depicting individual plant components to generate plant component embedding(s). For example, inference modulemay process digital image(s)captured inside combine harvesterusing a CNN and/or transformer to generate plant component embeddings. Embeddingsmay be, for instance, continuous vector embeddings.
104 334 104 118 122 338 336 336 340 336 340 122 120 340 336 2 FIGS.A-B Before, during, or after agronomic inference systemgenerates plant component embeddings, agronomic inference system, e.g., by way of inference moduleor training module, may process reference digital imagesdepicting reference individual plant components that were extracted during earlier phases of, such as during R&D, when high-quality reference G×E×M datawas available. Reference G×E×M datamay be ground truth observational data that corresponds to (e.g., be used as training labels for learning) the types of agronomic inferences that are generated using techniques described herein. Based on this processing, individual plant component reference embedding(s)may be generated. Reference G×E×M datamay be linked to individual plant component reference embedding(s), e.g., by training moduletraining one or more machine learning models stored in databaseto learn mapping(s) between individual plant component reference embedding(s)and reference G×E×M data.
340 336 118 332 332 118 334 334 118 332 332 104 330 The machine learning model(s) trained to learn mapping(s) between individual plant component reference embedding(s)and reference G×E×M data(referred to herein as “mapping machine learning models”) can then be used, e.g., by inference module, to generate agronomic inference(s) about composite plant organs (e.g., ears of corn, heads of wheat) from which the individual plant components depicted in digital imageswere extracted. For example, visual features of digital imagescan be encoded, e.g., by inference module, into embedding(s). Then, embeddingsmay be processed, e.g., by inference moduleusing mapping machine learning model(s), to generate agronomic inferences about archetype composite plant organs that would likely have yielded the individual plant components depicted in digital images. Based on these agronomic inferences, as well as based on other data such as position coordinates associated with imagesdepicting individual plant components, it is possible for agronomic inference systemto generate diagnoses (e.g., plant stress, nitrogen deficiency, dehydration, etc.) and/or recommendations, such as a prescription mapfor managing a field.
4 4 FIGS.A andB 4 FIG.A 4 FIG.B 456 108 452 456 schematically depict a non-limiting example of how images of individual corn kernelsmay be captured, e.g., inside of a combine harvester, in accordance with various implementations.is a side view andis a front view. A grain isolation/singulation mechanism(e.g., a seed meter or other similar mechanical apparatus) is provided that is configured to rotate and collect corn kernels(or other types of seeds or grains, depending on the application) on an individual and/or group basis for purposes such as controlled deposit, counting, etc.
450 452 452 108 452 454 452 450 450 456 454 118 120 456 4 FIG.A In various implementations, a light sourcesuch as an LED light may be placed proximate grain isolation/singulation mechanismand may be configured to emit various colors of electromagnetic radiation (e.g., light) depending on factors such as the type of individual plant component being photographed, physical characteristics of grain isolation/singulation mechanismand/or the agricultural equipment (e.g., combine harvester) in which grain isolation/singulation mechanismis deployed, and so forth. A vision sensorsuch as various types of 2D and/or 3D digital cameras may also be placed proximate grain isolation/singulation mechanism, e.g., on the opposite side from light sourceas depicted in. In the depicted configuration, light sourcemay provide back lighting that passes through corn kernelsand is captured by vision sensor. This may provide rich visual data that can then be processed, e.g., by inference moduleusing one or more machine learning models stored in database, to generate agronomic inferences about whole ears of corn or even entire corn plants from which corn kernelswere harvested.
5 FIG. 5 FIG. 5 FIG. 5 FIG. 500 500 104 illustrates a flowchart of an example methodfor practicing selected aspects of the present disclosure. For convenience, operations of methodwill be described as being performed by a system configured with selected aspects of the present disclosure, such as agronomic inference system. Other implementations may include additional operations than those illustrated in, may perform operation(s) ofin a different order and/or in parallel, and/or may omit one or more of the operations of.
502 116 108 108 502 452 450 454 504 116 118 454 452 4 FIG. At block, the system, e.g., by way of sampling moduleoperating within combine harvester, may sample one or more individual plant components from a plurality of plant components removed from one or more composite plant organs that previously included the plurality of plant components. In various implementations, the composite plant organ(s) may be part of crops in a field managed by a grower. The types of crops that can be harvested by combine harvester, and from which individual plant components can be sampled in block, may include, but are not limited to, corn (maize), flax (linseed), wheat, rice, oats, rye, barley, sorghum, soybeans, rapeseed, sunflowers, and so forth. For example, and as depicted in, grain isolation/singulation mechanismmay be operated to isolate (at least temporarily) individual plant components such as corn kernels so that those components may be irradiated by light sourceand captured in digital image(s) by vision sensor. To this end, at block, the system, e.g., by way of sampling moduleand/or inference moduleusing vision sensor, may capture one or more digital images of the sampled (e.g., temporarily isolated from other components using grain isolation/singulation mechanism) one or more individual plant components.
506 118 At block, the system, e.g., by way of inference module, may process the one or more digital images based on one or more machine learning models, such as the mapping models described herein, to generate one or more agronomic inferences about the one or more composite plant organs that previously included the plurality of plant components. In the context of corn, for instance, the agronomic inferences may include, but are not limited to, phosphate shortages, too much or too little nitrogen, low fertility (e.g., caused by insufficient fertilizer), other abiotic and/or biotic factors, plant-to-plant competition, quantification of the impact of beneficial microbes on crops, and/or too much or too little irrigation/hydration, to name a few.
122 336 122 118 338 340 122 336 122 2 2 FIGS.A andB 3 FIG. 3 FIG. In various implementations, one or more of the machine learning models may have been trained previously, e.g., by training module, based on training data associating reference individual plant components with ground truth agronomic observations about one or more reference composite plant organs that yielded the reference individual plant components. For example, ground truth agronomic observations may include the G×E×M data depicted inin association with the earlier phases (e.g., PHASE 1, PHASE 2, PHASE 3) and/or the reference G×E×M datadepicted in. In various implementations, during training, training module(or inference module) may process reference digital imagesand/or embeddingsgenerated therefrom using a mapping machine learning model to generate agronomic inferences, such as inferred seed productivity and/or density, inferred applied nitrogen rate (“N rate” in) over time and/or per product, inferred fungicide applied, and so forth. These agronomic inferences may be compared, e.g., by training module, to ground truth reference G×E×M datato determine error(s). Based on these error(s), training modulemay train the mapping machine learning model, e.g., using techniques such as gradient descent, back propagation, cross entropy, and so forth.
5 FIG. 3 FIG. 506 506 118 334 502 118 504 Referring back to, the processing of blockmay include a variety of different sub operations. For instance, at blockA, the system, e.g., by way of inference module, may generate individual plant component embeddings (e.g.,in) that represent the individual plant components sampled at block. For example, inference modulemay process the digital image(s) captured at blockusing a CNN or transformer to generate semantically rich individual plant component embeddings.
506 118 334 340 3 FIG. At blockB, the system, e.g., by way of inference module, may determine similarity measures between individual plant component embeddings (e.g.,) and individual plant component reference embeddings (e.g.,in) generated previously. In various implementations, these similarity measures may be determined using techniques such as Euclidean distance, dot product, cosine similarity, etc.
506 336 506 118 119 3 FIG. 1 FIG. Based on the similarity measures determined at blockB, as well as on ground truth agronomic observations (e.g.,in) about reference plant organs that yielded the reference individual plant components at blockC, the system, e.g., by way of inference module, may generate agronomic inferences (e.g.,in). For instance, based on the similarity measures, the individual plant component embeddings (e.g., corn kernels) may be most similar to (e.g., clustered with) individual plant component reference embeddings generated from composite plant organs (e.g., ears of corn) that had exhibited visual characteristics indicative of nitrogen deprivation. Thus, nitrogen deprivation may be inferred for composite plant organs (e.g., whole ears of corn that were discarded before there was an opportunity to observe them directly and draw agronomic conclusions from those direct observations) from which the individual plant components were harvested.
508 106 510 Based on the agronomic inferences, at block, the system may cause one or more computing devices, such as client device, to render output that includes a diagnosis or recommendation for a grower about a field, portion of a field, and/or crops growing in the field or portion thereof. In various implementations, the output may include, for instance, subfield recommendations for the field managed by the grower, one or more mid-crop-cycle agronomic recommendations, a local environmental zone map (e.g., rendered as a heat map) for the field, and so forth. In other implementations, at block, the agronomic inferences and/or diagnoses may be used to control agricultural equipment. For example, an autonomous agricultural vehicle or robot (e.g., a rover, drone) may be operated (e.g., autonomously, semi-autonomously) based on agronomic inferences generated using techniques described herein to perform various remedial actions. As one example, a rover may traverse through a field and apply nitrogen to those portions of the field that were inferred using techniques described herein to be nitrogen deficient.
6 FIG. 610 610 614 612 624 625 626 620 622 616 610 616 is a block diagram of an example computing devicethat may optionally be utilized to perform one or more aspects of techniques described herein. Computing devicetypically includes at least one processorwhich communicates with a number of peripheral devices via bus subsystem. These peripheral devices may include a storage subsystem, including, for example, a memory subsystemand a file storage subsystem, user interface output devices, user interface input devices, and a network interface subsystem. The input and output devices allow user interaction with computing device. Network interface subsystemprovides an interface to outside networks and is coupled to corresponding interface devices in other computing devices.
622 610 610 User interface input devicesmay include a keyboard, pointing devices such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touch screen incorporated into the display, audio input devices such as voice recognition systems, microphones, and/or other types of input devices. In some implementations in which computing devicetakes the form of a HMD or smart glasses, a pose of a user's eyes may be tracked for use, e.g., alone or in combination with other stimuli (e.g., blinking, pressing a button, etc.), as user input. In general, use of the term “input device” is intended to include all possible types of devices and ways to input information into computing deviceor onto a communication network.
620 610 User interface output devicesmay include a display subsystem, a printer, a fax machine, or non-visual displays such as audio output devices. The display subsystem may include a cathode ray tube (CRT), a flat-panel device such as a liquid crystal display (LCD), a projection device, one or more displays forming part of a HMD, or some other mechanism for creating a visible image. The display subsystem may also provide non-visual display such as via audio output devices. In general, use of the term “output device” is intended to include all possible types of devices and ways to output information from computing deviceto the user or to another machine or computing device.
624 624 500 1 4 FIGS.- Storage subsystemstores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystemmay include the logic to perform selected aspects of the methoddescribed herein, as well as to implement various components depicted in.
614 625 624 630 632 626 626 624 614 These software modules are generally executed by processoralone or in combination with other processors. Memoryused in the storage subsystemcan include a number of memories including a main random-access memory (RAM)for storage of instructions and data during program execution and a read only memory (ROM)in which fixed instructions are stored. A file storage subsystemcan provide persistent storage for program and data files, and may include a hard disk drive, a floppy disk drive along with associated removable media, a CD-ROM drive, an optical drive, or removable media cartridges. The modules implementing the functionality of certain implementations may be stored by file storage subsystemin the storage subsystem, or in other machines accessible by the processor(s).
612 610 612 Bus subsystemprovides a mechanism for letting the various components and subsystems of computing devicecommunicate with each other as intended. Although bus subsystemis shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.
610 610 610 6 FIG. 6 FIG. Computing devicecan be of varying types including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computing devicedepicted inis intended only as a specific example for purposes of illustrating some implementations. Many other configurations of computing deviceare possible having more or fewer components than the computing device depicted in.
While several implementations have been described and illustrated herein, a variety of other means and/or structures for performing the function and/or obtaining the results and/or one or more of the advantages described herein may be utilized, and each of such variations and/or modifications is deemed to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are meant to be exemplary and that the actual parameters, dimensions, materials, and/or configurations will depend upon the specific application or applications for which the teachings is/are used. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific implementations described herein. It is, therefore, to be understood that the foregoing implementations are presented by way of example only and that, within the scope of the appended claims and equivalents thereto, implementations may be practiced otherwise than as specifically described and claimed. Implementations of the present disclosure are directed to each individual feature, system, article, material, kit, and/or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and/or methods, if such features, systems, articles, materials, kits, and/or methods are not mutually inconsistent, is included within the scope of the present disclosure.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
May 22, 2023
June 16, 2026
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.