Machine-Learning-Enabled Tool Changer for Mushroom Crop Management System

This is a continuation patent application which claims priority under 35 U.S.C. § 120 to U.S. Ser. No. 18/904,925, filed Oct. 2, 2024, which is hereby incorporated by reference in its entirety, including without limitation, the specification, claims, and abstract, as well as any figures, tables, appendices, or drawings thereof. This application also claims priority under 35 U.S.C. § 119(e) to provisional patent application U.S. Ser. No. 63/551,215, filed Feb. 8, 2024. The provisional patent application is hereby incorporated by reference in its entirety herein, including without limitation: the specification, claims, and abstract, as well as any figures, tables, appendices, or drawings thereof.

The present disclosure relates generally to techniques for the cultivation and harvest of agricultural crops, and in particular for the automated cultivation and harvest of mushrooms.

In typical commercial mushroom growing operations, mushrooms are grown in growing beds on the surface of casing soil over substrate in a series of weekly intervals called flushes. Each flush is picked several times per day over a five-day period, and typically two to three flushes are harvested. The size at which the mushrooms are picked depends on market requirements.

European and North American commercial production of button mushrooms typically occurs on “Dutch Style” substrate filled shelves, using a two or three flush cropping cycle. The substrate is typically a composted mixture of wheat straw, animal manure, and gypsum. The substrate is pasteurized, inoculated, and colonized with spawn of a selected mushroom strain. The substrate is covered with a casing soil of peat and lime mixture in a layer approximately 45 to 50 mm deep, which is then ruffled with compost added to the casing to mix mushroom mycelium into the casing.

Traditionally, commercial mushroom farm operations rely on manual labour to harvest the mushrooms. Manual labour is costly, however, and difficult to optimize. Mushrooms typically grow at such a rate that the mushrooms approximately double in size every 24 hours. Using manual labour, each flush is picked only two or three times per day for the duration of the flush, meaning that a mushroom bed may become overgrown between pickings due to the growth rate of mushrooms, or that the growth of some mushrooms may be negatively impacted due to the build-up of CObeneath the surface. In order to prevent overgrowth of a mushroom bed, a flush can be picked more frequently, but picking at a higher frequency is difficult and costly to accomplish with manual labour. When a bed becomes overgrown, the mushrooms may run out of room and grow into each other, thereby reducing yield, increasing stem growth, and/or causing deformation of each individual mushroom thereby adversely affecting the quality and value of the harvested mushrooms.

The automated mushroom harvesting apparatus and system by Bourdeau et al. disclosed in WIPO International Publication Number WO 2023/010198 A1 solves many of the challenges associated with the automated picking of cultivated mushrooms. The system includes a tool change and cleaning station comprising a tool rack having a plurality of slots for supporting a plurality of end effectors. The plurality of end effectors includes end effectors of different sizes for harvesting different sizes of mushrooms.

There remains, however, a need for improved techniques to optimize the total yield and overall effectiveness of automated mushroom cultivation and harvest systems which addresses at least some of the shortcomings of previous solutions and provides yet further advantages, thereby providing a material value over prior techniques.

It is to be understood that the accompanying drawings are used for illustrating the principles of the embodiments and exemplifications of the subject-matter discussed herein. Hence the drawings are illustrated for simplicity and clarity, and not necessarily drawn to scale and are not intended to be limiting in scope. Reference characters/numbers are used to depict the elements of the subject-matter discussed that are also shown in the drawings. The same corresponding reference characters/numbers are given to a corresponding component or components of the same or similar nature, which may be depicted in multiple drawings for clarity. In particular, specific embodiments or categories of embodiments of an element designated by a particular reference character may be distinguished by means of a suffix, wherein the specific embodiment designated by a reference character having a suffix is a species of the more general element having the same reference character lacking the suffix. For example, an element shown in the drawings and designated by the reference character ###n is a species of the more general element designated by reference character ###, and thus possesses all of the features of the more general element. Text may also be included in the drawings to further clarify certain principles or elements of the invention. It should be noted that features depicted by one drawing may be used in conjunction with or within other drawings or substitute features of other drawings. It should further be noted that common and well-understood elements for creating a commercially viable version of the embodiments discussed herein are often not depicted to facilitate a better view of the principles and elements of the subject-matter discussed herein. Throughout the drawings, sometimes only one or fewer than all of the instances of an element visible in the view are designated by a lead line and reference character, for the sake only of simplicity and to avoid clutter. It will be understood, however, that in such cases, in accordance with the corresponding description, that all other instances are likewise designated and encompassed by the corresponding description.

Improved techniques for automated cultivation and harvest of mushrooms are disclosed herein. More particularly, improved techniques for automated selection of crop management tools by robotic mushroom crop managers are disclosed herein. The techniques include a robotic mushroom crop manager operative to generate a machine-learning-based model based on previous actions performed on a mushroom bed using tools selected from a plurality of tools, and to continuously or periodically train the model based on mushroom bed data collected while the robotic mushroom crop manager performs a crop management program. The techniques also include an automated crop management tool selection system including a plurality of like robotic mushroom crop managers communicatively coupled with a mushroom crop management tool coordinator operative to generate a machine-learning-based model based on previous actions performed on a plurality of mushroom beds by the plurality of like robotic mushroom crop managers using tools selected from a like plurality of tools, and to continuously or periodically train the model based on mushroom bed data collected while the robotic mushroom crop managers perform respective crop management programs.

With reference to, an automated mushroom crop management systemmay have a plurality of robotic mushroom crop managersA,B . . .N communicatively coupled with a mushroom crop management tool coordinatorvia a network.

Each robotic mushroom crop managerA,B . . .N may be an individual instance of the robotic mushroom crop managershown in. The robotic mushroom crop managermay have a robotic mushroom crop manager controller, and crop management equipment. The robotic mushroom crop managermay have or be operative to cooperate with a tool exchange plateincluding a plurality of crop management toolsA,B . . .N. The robotic mushroom crop managerand the tool exchange platemay be, include, or be a similar to or a modification of, embodiments of a mushroom harvesting robot and tool change station, respectively, as described and shown in WIPO International Publication Number WO 2023/010198 A1, the entirety of which is incorporated herein by reference.

The robotic mushroom crop manager controllermay be operative to control the crop management equipmentto perform a crop management program. The robotic mushroom crop manager controllermay have a processor, a memory, a storage, a communications interface, and input/output equipment. The memorymay store instructions operable by the processorusing the storage, the communications interface, and the input/output equipmentto performs the functions described herein. In particular, the memorymay store a crop management program engineand a mushroom bed data collector. The memorymay also store a mushroom bed data processor. The robotic mushroom crop manager controllermay interface with the crop management equipmentin order to communicate with and to control the crop management equipmentas described herein.

The crop management equipmentmay have at least one crop management deviceand sensors. The sensorsmay include optical imagers(intended to encompass either a single optimal imager or a plurality of optical imagers as required by context), and may also include other sensors(likewise). The robotic mushroom crop manager controlleris operable to use the crop management program engineto control the crop management equipmentto use the crop management deviceto perform a crop management program, which may be stored in the storage. At the same time, the robotic mushroom crop manager controlleris operable to use the mushroom bed data collectorto use the sensorsto collect mushroom bed data, as described herein.

With reference to, the processoris operative to use the crop management program engineto control the crop management equipmentto perform the crop management programrelative to a mushroom bedcontaining a number of mushrooms(only one of which is identified by a reference character and lead line to avoid clutter) in a growing medium. The growing mediummay be of any suitable form or composition. For example, the growing mediummay include a casing soil, which may include a peat and lime mixture, layered atop a substrate, which may include a composted mixture of wheat straw, animal manure, and gypsum, which is pasteurized, inoculated, and colonized with spawn of a selected mushroom strain. The crop management programmay include a sequence of actions to be performed by the crop management equipment, including actions to be performed by the crop management device. Without limitation, such actions may include: moving the crop management equipmentto or above any location on the mushroom bed; using the crop management deviceto harvest a specific mushroomat a particular location in the mushroom bed; and using the crop management deviceto move or otherwise disturb the growing mediumat a particular location in the mushroom bed.

In particular, the processormay generate and operate a mushroom bed modelof the mushroom bed, and to perform the crop management programbased at least partly on the mushroom bed model. The crop management program enginemay further be operable, for a given one of the sequence of actions in the crop management program, and for a given state of the mushroom bedaccording to the mushroom bed model, to select one of the crop management toolsA,B . . .N for performance of that action by the crop management device.

One non-limiting embodiment of crop management equipmentis shown in. The crop management devicecomprises a robotic armoperatively mounted to a carriage assembly. The robotic armcomprises a shoulder, an upper armpivotally mounted to the shoulder, an elbow, and a forearmpivotally mounted to the upper armat the elbow. An end effectoris releasably mounted to the free endof forearm.

The sensorsmay be mounted to or proximal the crop management deviceof the crop management equipmentso as to be operable to sense the mushroom bedat or about the current position of the crop management device. In particular, at least some of the optical imagers, which may be digital cameras, may be coupled to the crop management equipmentadjacent or proximal the crop management devicein such a way as to provide a field of view containing the crop management deviceand an area of the mushroom bedin which the crop management deviceis operable to harvest mushroomsin the field of view. For example, one of the optimal imagersmay be a harvesting imager, covered by a transparent harvesting imager shield, mounted at an undersideof the forearm, which may be proximal the end effectorrelative to the elbow. In this way, a field of view of the harvesting imager, which may be a plan or overhead view, may contain a portion of the mushroom bedbeneath the end effectorin use, and may also contain at least a portion of the end effectoritself. Another one of the optimal imagersmay be a trimming imager, covered by a transparent trimming imager shield, mounted to the robotic armor the carriage assemblyand positioned, oriented, and configured in such a way as to provide an elevation, side, or perspective view of the portion of the end effectorand portion of the mushroom bedbeneath the end effectorwhen in use, which field of view may correspond to the field of view of the harvesting imager, in that they are directed to a common object viewed from different angles. For example, the trimming imager may be mounted at an underside of theof the forearm, but now proximal the elbowrelative to the end effector, and aimed toward the end effector. Alternatively, the trimming imager may be mounted at an underside of the upper arm, the shoulder, or to the carriage assembly, and positioned, oriented, and configured to provide the field of view described herein.

As noted above, the sensorsmay include other sensors, which may or may not be limited by field of view in this way. The other sensorsmay include one or more of an air temperature sensor, an air humidity sensor, a motion sensor, an orientation sensor, a light sensor, a soil pH sensor, a soil moisture sensor, a soil temperature sensor, a soil nutrient sensor, a soil pest/insect sensor, and a soil pollution sensor. Any one or more of the other sensorsmay be positioned or mounted at the robotic arm, which may also be at an underside of the robotic arm

The mushroom bed data collectormay be operable by the processorto collect using the sensorsa stream of data about the state and conditions of the mushroom bedincluding the growing mushroomsand optionally also the growing medium(collectively, “mushroom bed data”). In particular, the mushroom bed data collectormay use the optical imagersto collect a continuous stream of images of the mushroom bedin the field of view of the optimal imagers. The optical imagersmay be operated to continuously collect images as the crop management deviceis continuously moved from position to position above the mushroom bedwhile performing the crop management program. The mushroom bed data collectormay be further operable to collect using the other sensors, when provided, a stream of data about the state and conditions of the mushroom bedcorresponding to the nature of such other sensors.

In particular, the mushroom bed data may include, quantify, or enable determination of one or more properties or characteristics of the mushroom bed, the growing mushrooms, and optionally the growing medium. A non-limiting list of such properties or characteristics of the mushroomsmay include: size; shape; density; defects; marks; quality grade; anomalies; surface texture; underside spacing between cap and growing medium; stem length; and stem orientation. When included, a non-limiting list of the properties or characteristics of the growing mediummay include: pH, moisture, temperature, nutrient quantities, pest/insect quantities, and pollution quantities. The mushroom bed data may be position-aware, in that it is associated with (which may be in the form of metadata) a collection location on the mushroom bedwhere the mushroom bed data was collected by the sensors. For example, the mushroom bed data may be indexed according to a virtual partitioning of the mushroom bed. For example, as shown in, the mushroom bedmay be divided by a set of gridlinesinto a set of mushroom bed cells, sometimes called ‘sliding windows’, and the mushroom bed data may be collected and indexed in accordance with the mushroom bed cells. Such mushroom bed cellsmay have any appropriate size, shape, or dimensions. In some embodiments, the mushroom bed cellsare squares having a side dimension of from about 0.5″ to about 5″, or at least about 1″, or about 1″, although other dimensions are possible and contemplated. Any suitable alternative arrangement may be used, including for example, a hexagonal tiling arrangement. The mushroom bed data may also be time-aware, in that it is associated with (which may be in the form of metadata) a collection time at which the mushroom bed data was collected by the sensors. For example, when the mushroom bed data is or includes images of the mushroom bed, the images may be indexed, labelled, or otherwise associated with a location on the mushroom bedwhere the image was collected by the optimal imagers, and may also be indexed, labelled, or otherwise associated with a time at which the image was collected by the optimal imagers. The position and/or the time may be generated by the crop management equipmentitself and received by the robotic mushroom crop manager controller, or it may be generated by the robotic mushroom crop manager controller.

The sensorsmay be operable to collect mushroom bed data in any desired time interval. For example, the sensorsmay be operated to collect mushroom bed data every 1-1000 ms, although other time intervals are contemplated. The mushroom bed data collectormay receive raw mushroom bed data from sensorsusing any communicative connection between the robotic mushroom crop manager controllerand the crop management equipment. The connection may be a wired connection, a wireless connection, and may use the communications interfaceto receive the raw mushroom bed data. The raw mushroom bed data may then be stored in the storage. The processormay operate the mushroom bed data processoras part of a computer vision system to process the raw mushroom bed data into pre-processed mushroom bed data, which may also be stored in the storage. For example, the mushroom bed data processormay be operable to process images collected by the optical imagersto augment, enhance, colour-correct, convert, or compress such images, or to identify, parameterize, or otherwise any of the properties and characteristics described above.

The processormay also be operable to train and operate a mushroom bed modelof the mushroom bedbased on the mushroom bed data. In particular, the transformed mushroom bed data generated by the mushroom bed data processormay include mushroom bed data vectors configured for ingestion by the mushroom bed model. The mushroom bed data vectors may include or enable determination of any quantifiable properties or characteristics of the mushroom bed, mushrooms, and optionally the growing medium, as described herein. When the mushroom bed data is or includes a stream of images, including position-indexed and time-indexed images, as described herein, the mushroom bed data vectors may be or include the images in any suitable encoding, which may include or be labelled by, which may be by metadata, corresponding locations and times. Any suitable object-detection techniques or metrics may be used, which may include intersection-over-union similarity measures. For example, one non-limiting mushroom bed data vector includes any combination of at least some, or all, of the following attributes: time stamp; mushroom bed cell index (e.g. row and column indices); window size; light configuration (e.g. RGB values); distance/depth (which may be in combination with light configuration as RGB-D values); motion quantities (which may be speed, velocity, inertia values of camera, sensors, or related structure); environmental data (such as moisture, temperature, surface substrate quality; substrate grade); and metrics of mushrooms in window (such as size, density, number of high density clumps, proportion of high quality mushrooms). Other quantities and measures are possible and contemplated.

As discussed above, the mushroom bed data may enable a determination of properties or characteristics of the mushroomsincluding without limitation: size; shape; density; defects; marks; quality grade; anomalies; surface texture underside spacing between cap and growing medium; stem length; and stem orientation. When the collected mushroom bed data is or includes a stream of images, as described herein, the mushroom bed modelmay be trained to generate and predict such properties or characteristics, as described further below.

Specifically, the processormay be operable to train the mushroom bed modelby using a comparer. In an initial training stage, the mushroom bed data processormay be used to generate mushroom bed data vectors based on mushroom bed data received as described herein, where the corresponding known mushroom bed state vectors encoding the properties and characteristics of the mushroom bed, including the mushrooms, and optionally the growth mediumare determined by an additional procedure.

For example, with reference to, the mushroom bedmay be an actual mushroom bedwith live, growing mushroomsin a real growth medium, and the known mushroom bed state vectors encoding the properties and characteristics of the mushroom bedmay be determined manually, which may be by manual inspection. Alternatively, and with reference to, the mushroom bedmay a synthetic mushroom bed″, with synthetic mushrooms″, and optionally synthetic growth medium″, fabricated purposefully to possess preconfigured mushroom bed state vectors encoding a predetermined variety of the mushroom bed (including mushroom) properties and characteristics. In any case, the mushroom bed data vectors so received and generated may include or be labelled with the known mushroom bed state vectors. The mushroom bed modelmay then process such labelled mushroom bed data vectors using the comparerto determine differences between the known mushroom bed state vectors and the predicted mushroom bed state vectors generated by the mushroom bed modelbased on the received mushroom bed data vectors, to learn to predict the corresponding known mushroom bed state vectors, and thus the known mushroom bed state.

The processormay be operable to train the mushroom bed modelusing any suitable techniques known in the art. A network architecture or topology may be established, and layers may be added which are associated with respective optimization functions, activation functions, and/or loss functions. One or more artificial neural networks may be used, and each may of any suitable type, including without limitation convolutional neural networks, recurrent neural networks, and deep learning neural networks. The mushroom bed modelmay involve one artificial neural networks, or may involve multiple different artificial neural networks. The mushroom bed modelmay include instructions using supervised or unsupervised machine learning, involving identifying and recognizing patterns in the mushroom bed data (in the form of the mushroom bed data vectors) to enable recognition of mushroom bed states of the mushroom bed. The mushroom bed data and the differences between the actual and predicted mushroom bed state described herein which may be used to train the artificial neural network may be encoded in any suitable manner, such as, without limitation, an N-dimensional tensor, a matrix, or an array. Training of the artificial neural network may involve parameters initialized to random values, which are changed with each iteration, using any appropriate algorithm to converge to predetermined values. Such algorithms may include, for example, gradient descent, stochastic gradient descent (SGD), minibatch SGD, nonlinear conjugate gradient, limited-memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS), or Levenberg-Marquardt (LMA) algorithms. Training of the artificial neural network may employ any appropriate statistical model, which may be a multinomial logistic regression model, a random forest model, a decision tree, a logistic regression model, or a gradient boosting model.

As indicated above, the robotic mushroom crop managermay also have or be operative to cooperate with a tool exchange plate. In the embodiment shown in, the tool exchange plateforms a part of the crop management equipment, and is mounted to the carriage assembly. The tool exchange platesupports a plurality of crop management tools(only one of which is identified with a reference character and lead line in order to avoid clutter). The crop management toolsare sized, shaped, and configured for coupling with and use by the robotic armend effectorto perform corresponding actions described herein. For example, different crop management toolsmay have different sizes or configurations suitable for the harvesting of different sizes and/or types of mushrooms. Different crop management toolsmay include, without limitation: suction cups with any suitable diameters, depths, dimensions, or configurations to accommodate any corresponding size, shape, or type of mushroom; grabbers with any number of fingers, such as 3, 4, 5, or 6, which may have knuckles or elbows; pipe-like tools with sharp, razor-like end to encircle a mushroom, which may be a mushroom pin, pass beyond it, and twist or otherwise disturb the substrate around it, in a circular manner; pipe-like tools with a moving section to close under a mushroom, which may be a mushroom pin, to lift it for transplantation, which may be in a circular manner; pipe-like tools with an end flattened to press on a mushroom, which may be a mushroom pin, to be destroyed but left in place to rot; irrigation tools; fertilization tools; spoon-like tools for substrate movement, manipulation, or transplantation; and tools with one or more environmental sensors embedded for real-time probing. Other types and arrangements of tools are possible and contemplated.

In particular, for each of the actions performable by the crop management devicerelative to the mushroom bedin accordance with the crop management program, a crop management devicemay employ a crop management toolselected from a plurality of crop management toolsA,B . . .N. In particular, for a given set of conditions of the mushroom bed, a particular crop management toolmay be optimally functional for the performance of the action required by the crop management program. For example, different crop management toolsmay have different sizes, shapes, or configurations suitable for the harvesting of different sizes and/or types of mushrooms. Different crop management toolsmay have different sizes or configurations suitable for the movement or disturbance of the growing mediumat a particular location in the mushroom bed. The foregoing are non-limiting examples. As such, optimal performance of the crop management programmay involve for at least some of the actions performed in accordance with the crop management programa selection of a particular crop management toolfrom the plurality of crop management toolsA,B . . .N which is optimal for the performance of that action given the corresponding conditions of the mushroom bed.

For this purpose, the crop management program enginemay be operable by the processorfor selecting a crop management toolfrom the plurality of crop management toolsA,B . . .N based on mushroom bed data collected from the mushroom bed. In particular, the processormay be operable to continuously or periodically train a tool selection modelbased on the mushroom bed data of the mushroom bedincluding at least the mushrooms, and optionally the growing medium, which may be based on mushroom bed state vectors generated by the mushroom bed model. The processormay be operable to train the tool selection modelto select a specified crop management toolselected from the plurality of crop management toolsA,B . . .N to perform a specified action on the mushroom bedin order to produce a corresponding change in state of the mushroom bed. The change in state of the mushroom bedmay include a change in state of mushroomsand optionally growing mediumat a specified future time. Any appropriate future time may be specified, and may be from 1 second to 1 hour (3600 seconds). More distant future times are also contemplated. With respect to each such selection and corresponding action, the processormay be operable to train the tool selection modelbased on feedback regarding a desired change in state of the mushroom bed, as further described below.

In particular, the processormay be operable to train the tool selection modelby using an outcome optimizer. As discussed above, the crop management program enginemay be operable to control the crop management equipmentaccording to a crop management program, and the crop management programmay include a sequence of actions to be performed by the crop management device. Without limitation, such actions may include: harvesting a specific mushroomat a particular location in the mushroom bed; and moving or otherwise disturbing the growing mediumat a particular location in the mushroom bed. Each such action may be characterized by parameters defining a desired or optimal outcome. For example, where the action is harvesting a mushroom, the desired or optimal outcome may include one or more of: minimizing damage to the mushroom; minimizing time required to harvest the mushroom; minimizing damage to adjacent or surrounding mushrooms; or minimizing disturbance of adjacent or surrounding growing medium. With respect to such actions, each one of the plurality of crop management toolsA,B . . .N may produce a different actual outcome based on different properties of the particular crop management tool, and different properties of the mushroom bed, such as different sizes and/or types of mushrooms, or different kinds or conditions of growing medium. Given such different properties of the mushroom bed, one of the plurality of crop management toolsA,B . . .N when used to perform the required action may produce an actual outcome which is closer to a desired or optimal outcome than the other ones of the crop management toolsA,B . . .N. Therefore, for any given current state of the mushroom bed(such as, a given current state of a particular mushroom) and a given action to be performed on the mushroom bedin accordance with the crop management program (such as, to harvest the mushroom), the processormay be operable to train the tool selection modelto predict which one of the plurality of crop management toolsA,B . . .N is likely to produce an actual outcome which is closer to the desired or optimal outcome of the action as compared to the other ones of the crop management toolsA,B . . .N.

In particular, and as discussed above, during performance of the crop management program, the mushroom bed data collectormay be operable to use the sensorsof the crop management equipmentto periodically or continuously collect mushroom bed data from the mushroom bed. In particular, the optical imagersmay be operable to collect a stream of images of the mushroom bedin the respective fields of view of the optimal imagersas the crop management deviceis operated to perform actions on the mushroom bed. More particularly, the mushroom bed data collectormay be operable to use the sensorsof the crop management equipmentto collect mushroom bed data from the mushroom bedat a time before the performance of a given action on the mushroom bed, and then also at a post-action time after the performance of the action on the mushroom bed. In this way, when the crop management program enginecontrols the crop management equipment, and in particular the crop management deviceto select a crop management toolfrom a plurality of crop management toolsA,B . . .N, and to use the selected crop management toolto perform a corresponding action on the mushroom bed, the mushroom bed data collectormay be operable to use the sensorsof the crop management equipmentto collect mushroom bed data from the mushroom bedat the pre-action time before the performance of the action and then also at the post-action time after the performance of the action on the mushroom bed. The processor, using the mushroom bed data collector, may then be operable to determine an actual outcome of the performance of the action on the mushroom bedusing the selected crop management toolbased on differences between the pre-action mushroom bed state, which may in the form of mushroom bed state vectors generated by the mushroom bed model, at the pre-action time and the post-action mushroom bed state, which again may in the form of mushroom bed state vectors generated by the mushroom bed model, at the post-action time, and to determine an outcome difference, which may be by using the comparer, between the actual outcome and a desired outcome in accordance with the crop management program.

In particular, the processormay use the outcome optimizerto compare the actual outcome and the desired outcome of the action performed on the mushroom bedusing the crop management toolselected from the plurality of crop management toolsA,B . . .N. The comparermay receive from the mushroom bed modela post-action mushroom bed state vector of the mushroom bed based on mushroom bed data collected at the post-action time after performance of the action by the selected crop management tool, which encodes characteristics and properties of the mushroom bedat the post-action time, and further receive from the mushroom bed modela pre-action mushroom bed state vector of the mushroom bed based on mushroom bed data collected at the pre-action time before performance of the action by the selected crop management tool, which encodes characteristics and properties of the mushroom bedat the pre-action time, and the pre-action mushroom bed state vector and the post-action state vector to determine the actual outcome of the action. The processormay then use the outcome optimizerto compare the actual outcome of the action so generated with a desired outcome based on desired properties and characteristics of the mushroom bedin accordance with the crop management program. As discussed above, a non-limiting list of properties or characteristics of mushroomsincludes: size; shape; density; defects; marks; quality grade; anomalies; surface texture underside spacing between cap and growing medium; stem length; and stem orientation. And, when included, a non-limiting list of the properties or characteristics of the growing mediumincludes: soil pH, soil moisture, presence and/or concentration of gases (such as CO), soil temperature, soil nutrient, soil pest/insect, and soil pollution. Thus, the crop management program may include a desired outcome relative to any one or more of such properties and characteristics of the mushroom bedto be caused by the performance of the action by the crop management device, and the processormay be operable to use the outcome optimizerto compare and determine differences between the actual outcome with respect to such one or more properties and characteristics with the desired outcome of the action.

For example, where the action is harvesting a mushroom, the desired or optimal outcome may include one or more of: minimizing damage to the mushroom; minimizing time required to harvest the mushroom; minimizing damage to adjacent or surrounding mushrooms; or minimizing disturbance of adjacent or surrounding growing medium. The processor, using the comparerand the outcome optimizer, may then use these differences to train the tool selection modelto learn to predict the actual outcomes of the performance of corresponding actions on the mushroom bedusing each of the plurality of crop management toolsA,B . . .N, and in this way to predict for a given action in the crop management program which one of the plurality of crop management toolsA,B . . .N is likely to produce an actual outcome which is closest to the desired outcome-that is, which is an optimal outcome.

For any given action performed using a selected crop management tool, a time interval between the performance of the action and collection of mushroom bed data to determine an actual mushroom bed state as described may be any desired time interval. For example, the time interval may be about 0.1 to about 600 seconds, or about 1 to about 60 seconds, or about 5 to about 10 seconds, although other time intervals are contemplated. In order to use the comparerand outcome optimizerto compare and determine differences between the actual outcome with respect to one or more properties and characteristics of the mushroom bedwith the desired outcome of the action, as described above, the processormay, for a given action, be operable to store tool selection data in a pending data structure, which may be stored in the storage, which associates the mushroom bed state at the time of the action, the time of the action, and the selected crop management toolused to perform the action. Then, at or about an end of the time interval, the processormay be operable to fetch the tool selection data from the pending data structurefor use by the comparerand the outcome optimizerto compare and determine differences between the actual outcome with respect to one or more properties and characteristics of the mushroom bedwith the desired outcome of the action, as described.

For example, where the action performed by one of the plurality of crop management toolsA,B . . .N is to harvest a specific mushroom, the desired or optimal outcome may include minimizing damage to the mushroom. The crop management program enginemay be operable to select one of the plurality of crop management toolsA,B . . .N to perform the harvesting action. The mushroom bed data collectormay be operable to control the crop management equipmentsensors, and particularly the optical imagers—including particularly the g imager and the trimming imager described above—to periodically or continuously collect images of the mushroombefore, during, and after harvesting of the mushroomby the selected crop management tool. For example, the mushroom bed data collectormay collect images of the mushroomas it is lifted by the selected crop management tool, as the mushroomstem is trimmed, as the trimmed mushroomis carried to a collection box, as the mushroomis dropped in the collection box, and when the mushroomis resting in the collection box. In particular, the harvesting imager may provide a plan or overhead view of the mushroomimmediately before it is grasped and lifted by the selected crop management tooland after it is dropped in the collection box, while the trimming imager, which provides an elevation, side, or perspective view of the portion of the end effectoras described above, may provide a view of the mushroomonce it has been grasped and lifted by the selected crop management tooland while it is being moved to the collection box. In this way, the collection of images with respect to the interaction between the selected crop management tooland the mushroommay be comprehensive.

Thus, the processorof the robotic mushroom crop manager controllermay be operable, based on the mushroom bed data so collected, to use the mushroom bed modelto model and characterize the mushroomat selected points, periodically, or throughout the harvesting action as described, and then to use the comparerand outcome optimizerto compare and determine differences between the actual outcome of the mushroomwith respect to one or more properties and characteristics with the desired outcome of the harvesting action, as described. This may include, for example, an amount or degree of damage caused to the mushroom, which may be either to its cap or stem. The determined differences between the actual outcome with the desired outcome may then be used to further train the tool section model.

The processormay be operable to train the tool selection modelusing any suitable techniques known in the art. A network architecture or topology may be established, and layers may be added which are associated with respective optimization functions, activation functions, and/or loss functions. One or more artificial neural networks may be used, and each may of any suitable type, including without limitation convolutional neural networks, recurrent neural networks, and deep learning neural networks. The tool selection modelmay involve one or more of the same artificial neural networks, or may involve different artificial neural networks. The tool selection modelmay include instructions using supervised or unsupervised machine learning, involving identifying and recognizing patterns in the mushroom bed data (in the form of the mushroom bed data vectors) to enable recognition of mushroom bed states of the mushroom bed, and to enable prediction of a selected crop management tool to produce an optimal actual outcome. The mushroom bed data, used to generate mushroom bed state vectors as described, and the differences between the actual and desired outcomes described herein which may be used to train the artificial neural network may be encoded in any suitable manner, such as, without limitation, an N-dimensional tensor, a matrix, or an array. Training may be performed in any suitable manner, and may include iterative training using labeled training data as described herein. Training of the artificial neural network may involve parameters initialized to random values, which are changed with each iteration, using any appropriate algorithm to converge to predetermined values. Such algorithms may include, for example, gradient descent, stochastic gradient descent (SGD), minibatch SGD, nonlinear conjugate gradient, limited-memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS), or Levenberg-Marquardt (LMA) algorithms. Training of the artificial neural network may employ any appropriate statistical model, which may be a multinomial logistic regression model, a random forest model, a decision tree, a logistic regression model, or a gradient boosting model.

In order to optimize the speed and efficiency of selection of an optimal crop management toolfrom the plurality of crop management toolsA,B . . .N given a required action of the crop management program, the processormay be further operable to use a selection library manager. As described above, the processormay be operable to use the comparerand outcome optimizerto compare and determine differences between the actual outcome of a given action performed using a selected crop management toolwith respect to one or more properties and characteristics with the desired outcome of the action. At a certain point, the likelihood that a given crop management toolmay produce an optimal outcome as compared to the other ones of the plurality of crop management toolsA,B . . .N is sufficiently great that, thereafter, for a given action, when the relevant properties and characteristics of the mushroom bedfall within respective preconfigured ranges the selection of that crop management toolmay also be preconfigured, such that use of the tool selection modelto determine a selected crop management toolmay be avoided.

To this end, the processormay operate the selection library managerto store in a selection library datastore, stored in the storage, predetermined tool selection data, which may be a record in a look-up table, associating a predetermined optimal one of the plurality of crop management toolsA,B . . .N with a given action included in the crop management program and a preconfigured combination of preconfigured ranges of the properties and characteristics of the mushroom bed. The selection library managermay be configured to do so when a likelihood that a given crop management toolmay produce an optimal outcome as compared to the other ones of the plurality of crop management toolsA,B . . .N exceeds a predefined threshold. In some embodiments, the selection library manageris configured to do so on any given successful action, that is when the determined differences between the actual outcome of a given action performed using a selected crop management toolwith respect to one or more properties and characteristics with the desired outcome of the action, as described above, matches within a predefined threshold. In other words, the selection library managermay be configured to store in the selection library datastorepredetermined tool selection data on the very first instance of a successful action according to the noted predefined threshold.

Then, in general, as the processoruses the crop management program engineto perform the crop management program, for each action performable in accordance therewith, the processormay use the selection library managerto determine when an optimal tool selection threshold is met for a given one of the plurality of crop management toolsA,B . . .N in association with a given action in the crop management program and given preconfigured ranges of the properties and characteristics of the mushroom bed, which may be preconfigured ranges of dimensions of a current mushroom bed state vector. For example, the given action may be harvesting a selected mushroom, and the characteristics of the mushroom bedmay include one or more of: a size, shape, and orientation of the mushroom, including an orientation of the stem and spacing between an underside of the cap and top surface of the growing medium; and number, sizes, spacing, and orientations of surrounding mushrooms. The processor, before using the tool selection modelas described above to select one of the plurality of crop management toolsA,B . . .N to perform the given action based on a current mushroom bed state, may first instead use the selection library managerto access the selection library datastoreto determine whether the selection library datastorecontains predetermined tool selection data, which may be a record, encoding the above-described association-that is, for the given action, defines preconfigured ranges of the properties and characteristics of the mushroom bedinto which the current mushroom bed state falls. In this way, the processormay avoid using the tool selection modelto select a crop management tool, and instead select the crop management toolidentified in the predetermined tool selection data. The selection library managermay operate in this way using less time and fewer computational resources than the tool selection model, and in this way the overall performance of the crop management program by the crop management program enginemay likewise use less time and fewer computational resources.

The processormay further operate the selection library managerto update the selection library datastore, stored in the storage, as the crop management programis performed. In particular, if and when a given action in the crop management program and given ranges of the properties and characteristics of the mushroom bedis performed using one of the plurality of crop management toolsA,B . . .N based the selection library datastorecontaining a record with matching predetermined tool selection data, as described above, but the action does not in fact produce an actual outcome within the preconfigured similarity with the desired outcome, as described above, then the processormay be operable to use the selection library managerto access the selection library datastoreto remove that record.

Thus, a robotic mushroom crop manageras shown and described above is thus operable to optimize tool selection for use by harvesting equipment to perform actions of a crop management program on a mushroom bed. For example, the robotic mushroom crop manageris operable to optimize selection of a crop management tool from a plurality of crop management tools available on a tool exchange plate for harvesting mushrooms from the mushroom bed, so as to minimize damage caused to the mushrooms and also to optimize time spent harvesting the mushroom. The robotic mushroom crop manageris operable to train a tool selection modelbased on collected mushroom bed data. A selection library manageris operable, for a given crop management program action and mushroom bed state, to determine an optimal crop management tool by lookup in a selection library datastore. In this way, the robotic mushroom crop managermay be progressively configured for optimal crop management tool selection over time.

In order to accelerate and optimize the process of crop management tool selection by a particular robotic mushroom crop manager, however, a plurality of like or similar robotic mushroom crop managermay cooperate in such a way as to share, in effect, the lessons learned with respect to crop management tool selections made by all of them, so that any one of the robotic mushroom crop managermay, in a given situation, benefit from the experiences of all of the others.

For this purpose, and as shown inand as described above, the automated mushroom crop management systemmay have a plurality of robotic mushroom crop managersA,B . . .N communicatively coupled via a network. Each one of the plurality of robotic mushroom crop managersA,B . . .N may be configured as described above, and in particular may be configured to develop its respective selection librarybased on successful actions performed by its crop management equipmentusing a selected one of the plurality of crop management toolsA,B . . .N as described above. The automated mushroom crop management systemmay be configured such that one or more, which may be each, of the plurality of robotic mushroom crop managersA,B . . .N is operable to benefit from the experience gained from all of them with respect to the selection of an optimal crop management toolfor the performance of a given action.

Specifically, one or more, which may be each, or which may be very one, of the robotic mushroom crop managersA,B . . .N may store in its respective storagea robotic mushroom crop manager (RMCM) list datastorewhich stores records corresponding respectively to one or more of the other robotic mushroom crop managersA,B . . .N. Each such record may store, among other things, an identifier and a communication address, which may be a network address, of the corresponding robotic mushroom crop manager. Records corresponding to all or a subset of the other robotic mushroom crop managersA,B . . .N may be stored in this way. In particular, for a given one of the robotic mushroom crop managersA,B . . .N, records corresponding to only a subset of the other robotic mushroom crop managersA,B . . .N may be stored, based on a similarity of respective parameters of the subset of other robotic mushroom crop managersA,B . . .N and the given one of the robotic mushroom crop managersA,B . . .N. Such parameters may include, without limitation: location of the other robotic mushroom crop manager; and type or types of mushroomsof the mushroom bedassociated with that other robotic mushroom crop manager.

Thus, and as described above, when the processoruses the crop management program engineto perform the crop management program, and uses the selection library managerto access the selection library datastoreto determine whether the selection library datastorecontains predetermined tool selection data for the given action, and when it determines that the selection library datastoredoes not contain predetermined tool selection data for the given action, the processormay then be operable, which may be by use of the selection library manager, the access the RMCM listto determine whether it contains a record of other robotic mushroom crop managersA,B . . .N having respective parameters similar within a predefined threshold to parameters of the robotic mushroom crop manager. For example, the predefined threshold may be that a respective location of the other robotic mushroom crop manageris within a predefined distance of a location of the robotic mushroom crop manager, within a predefined distance of a location of certain crop management equipment, and/or a type of the mushroomsof the mushroom bedassociated with that other robotic mushroom crop manageris the same or similar within a predefined threshold to those of the robotic mushroom crop manager. In such case, the processormay be operable, which may be using the selection library manager, to generate a tool selection request message, which may encode in association the given action in the crop management programand a current mushroom bed state vector, and to use the communications interfaceto communicate, which may be via the network, to those other robotic mushroom crop managerswhose parameters are the same or similar within the predefined threshold. The processormay be configured either to wait up to a preconfigured waiting time for a reply before continuing to perform the crop management program, or it may be configured to skip the action involved, and may further be configured to reschedule the action to a later point in the crop management program, which may be done either immediately or after the preconfigured waiting time. For each of the other robotic mushroom crop managersto which the tool selection request message was sent, and respectively throughout, the processorthereof may be operable to use the communication interfaceto receive the tool selection request message, and may further be operable, which may be using the selection library manager, to access the selection library datastoreto determine whether it stores predetermined tool selection data which matches the given action and the mushroom bed state vector encoded in the tool selection request message, which may be in the same manner as is done by the sending robotic mushroom crop managerwith respect to its own selection library datastoreas described above. In the case that the selection library datastoredoes indeed store such predetermined tool selection data, then the processorof the corresponding other robotic mushroom crop managermay be operable, which may be using the corresponding selection library manager, to generate a tool selection reply message encoding the optimal tool selection encoded in that predetermined tool selection data, and may further be operable to use the corresponding communications interfaceto transmit the tool selection reply message to the original robotic mushroom crop managerwhich had sent the tool selection request message. The processorof that original robotic mushroom crop managermay then be operable using the corresponding communications interfaceto receive the tool selection reply message. Then, the processormay be operable using the selection library managerto store in the selection library datastorepredetermined tool selection data associating that optimal tool selection with the mushroom bed state vector, in the same manner as described above generally. As such, when in performance of the crop management programthe processoragain encounters that given action, it is operable using the selection library managerto access the selection library datastore, which now stores the predetermined tool selection data associating that optimal tool selection with the mushroom bed state vector, and thus is operable to use the crop management equipmentto perform the given action using that optimal crop management toolas described.

Additionally, or alternatively, the automated mushroom crop management systemmay have a plurality of robotic mushroom crop managersA,B . . .N communicatively coupled with a mushroom crop management tool coordinatorvia a network. An embodiment of the mushroom crop management tool coordinatoris shown in. The mushroom crop management tool coordinatormay have a processor, a memory, a storage, a communications interface, and input/output equipment. The memorymay store instructions operable by the processorusing the storage, the communications interface, and the input/output equipmentto perform the functions described herein. In particular, the memorymay store a selection request processor, and a library submission processor.

As described above, each robotic mushroom crop manager controllermay be operable to use its selection library managerto determine when an optimal tool selection threshold is met for a given one of the plurality of crop management toolsA,B . . .N in association with a given action in the crop management program and a given preconfigured ranges of the properties and characteristics of the mushroom bed. As described above, the selection library managermay store in a selection library datastorepredetermined tool selection data associating the predetermined optimal one of the plurality of crop management toolsA,B . . .N with the given action included in the crop management program and the preconfigured combination of preconfigured ranges of the properties and characteristics of the mushroom bed. In such case, processormay further be operable to use the communication interfaceto transmit to the mushroom crop management tool coordinatora library submission request encoding the predetermined tool selection data. The processorof the mushroom crop management tool coordinatormay be operable to use the communications interfaceto receive the library submission request, and to use the library submission processorto process the library submission request. In particular, the library submission processormay be operable to determine, as described below, whether to add the predetermined tool selection data encoded in the library submission request to the selection libraryof the mushroom crop management tool coordinator, for use as further described below.

Specifically, the mushroom crop management tool coordinatormay be operable to receive library submission requests from a plurality of robotic mushroom crop manager controllers, as described. Until a threshold number of library submission requests is met, the processormay be operable to store the received library submission requests in a pending datastorein the storage. The memorymay further store a selection library manager, and once the threshold number of library submission requests is met, the processormay be operable to use selection library managerto process the stored library submission requests.

In particular, each stored library submission request may encode corresponding predetermined tool selection data as described above, associating a predetermined optimal one of the plurality of crop management toolsA,B . . .N with a given action and a preconfigured combination of preconfigured ranges of the properties and characteristics of the mushroom bed. The corresponding predetermined tool selection data of the stored library submission requests may agree or disagree in certain respects. For example, the predetermined tool selection data of a first one of the library submission requests, for a given action and a first preconfigured combination of preconfigured ranges of the mushroom bed, may indicate selection of a particular one of the plurality of crop management toolsA,B . . .N, while the predetermined tool selection data of a second one of the library submission requests, for the same action and a second preconfigured combination of preconfigured ranges of the mushroom bed which overlaps the first preconfigured combination, may indicate selection of a different one of the plurality of crop management toolsA,B . . .N. In other words, the two library submission requests may disagree as to the optimal tool selection where the mushroom bed conditions are otherwise similar within a predefined threshold. Conversely, the predetermined tool selection data of a first one of the library submission requests, for a given action and a first preconfigured combination of preconfigured ranges of the mushroom bed, may indicate selection of a particular one of the plurality of crop management toolsA,B . . .N, while the predetermined tool selection data of a second one of the library submission requests, for the same action and a second preconfigured combination of preconfigured ranges of the mushroom bed which overlaps the first preconfigured combination, may indicate selection of the same one of the plurality of crop management toolsA,B . . .N. In other words, the two library submission requests may agree as to the optimal tool selection where the mushroom bed conditions are otherwise similar. In general, the stored library submission requests may agree or disagree in this way, within a range of degrees.