Control System, Control Method and Machine, Storage Medium and Program Product

The present application is based on and claims priority to CN Patent Application No. 202410415004.X filed on Apr. 7, 2024, the disclosure of which is incorporated by reference herein in its entirety.

The present disclosure relates to the technical field of smart construction machine, in particular to a control system, a control method and machine, a storage medium and a program product.

At present, smart construction machine has received more and more attention and has made some progress. Current smart construction machine substantially uses a sequential architecture of perception-planning-control.

According to an aspect of the present disclosure, a control system for a construction machine is provided. The control system comprises: an environment perception module configured to receive environment information and/or control result information, output perception result information based on the environment information and/or the control result information, receive a system adjustment instruction, and adjust a state of the environment perception module based on the system adjustment instruction; a decision-making planning module configured to receive an operation task and the perception result information, generate a motion trajectory based on the operation task and the perception result information, output the motion trajectory, predict the motion trajectory to obtain prediction result information, receive feedback result information corresponding to the motion trajectory, and output the system adjustment instruction to the environment perception module based on the prediction result information and the feedback result information; and a control performing module configured to receive the motion trajectory, perform the operation task based on the motion trajectory, obtain the feedback result information and the control result information during the performing of the operation task, output the feedback result information to the decision-making planning module, and output the control result information to the environment perception module.

In some embodiments, the environment perception module is further configured to output the perception result information directly to the control performing module; and the control performing module is further configured to receive the perception result information and perform an operation corresponding to the perception result information based on the perception result information.

In some embodiments, the decision-making planning module is configured to calculate and obtain an adjustment parameter based on the prediction result information and the feedback result information, and output the system adjustment instruction based on the adjustment parameter, wherein the system adjustment instruction comprises the adjustment parameter, and the adjustment parameter is configured to adjust the state of the environment perception module; and the environment perception module is configured to extract the adjustment parameter from the system adjustment instruction and adjust the state of the environment perception module based on the adjustment parameter.

In some embodiments, the feedback result information comprises a reaction result of an operation object of the construction machine.

In some embodiments, the perception result information comprises the environment information; or the perception result information comprises a logical judgment result generated based on the environment information and/or the control result information. In some embodiments, the environment perception module comprises an environment perception element and a multi-degree-of-freedom platform, wherein the environment perception element is installed on the multi-degree-of-freedom platform; and the adjustment parameter comprises an internal parameter value of the environment perception element and/or a parameter of the multi-degree-of-freedom platform.

In some embodiments, the environment perception module is further configured to transmit the environment information to the decision-making planning module, wherein the environment information is first data information with an element coordinate system of the environment perception module as a coordinate system; and the decision-making planning module is further configured to convert the first data information into second data information with a reference coordinate system of the construction machine as a coordinate system based on a transformation matrix between the element coordinate system and the reference coordinate system, and generate the motion trajectory based on the operation task and the second data information.

In some embodiments, the decision-making planning module is further configured to adjust the motion trajectory based on the perception result information of the environment perception module, and output the motion trajectory adjusted to the control performing module; and the control performing module is further configured to control motion of the construction machine based on the motion trajectory adjusted.

In some embodiments, the decision-making planning module is further configured to adjust the motion trajectory based on the feedback result information of the control performing module, and output the motion trajectory adjusted to the control performing module; and the control performing module is further configured to control motion of the construction machine based on the motion trajectory adjusted.

In some embodiments, the decision-making planning module is further configured to send a control instruction to the control performing module to control an operational state or operational speed of the control performing module to make the operational state or operational speed of the control performing module match an adjusted state of the environment perception module.

According to another aspect of the present disclosure, a control method for a construction machinery is provided. The control method comprises: receiving, by an environment perception module, environment information and/or control result information, and outputting perception result information based on the environment information and/or the control result information; receiving, by a decision-making planning module, an operation task and the perception result information, generating a motion trajectory based on the operation task and the perception result information, outputting the motion trajectory, and predicting the motion trajectory to obtain predicting result information; receiving, by a control performing module, the motion trajectory, performing the operation task based on the motion trajectory, obtaining feedback result information and the control result information during the performing of the operation task, outputting the feedback result information to the decision-making planning module, and outputting the control result information to the environment perception module; receiving, by the decision-making planning module, the feedback result information corresponding to the motion trajectory, and outputting a system adjustment instruction to the environment perception module based on the prediction result information and the feedback result information; and receiving, by the environment perception module, the system adjustment instruction, and adjusting a state of the environment perception module based on the system adjustment instruction.

In some embodiments, the control method further comprises: outputting the perception result information directly to the control performing module by the environment perception module; and receiving, by the control performing module, the perception result information, and performing an operation corresponding to the perception result information based on the perception result information.

In some embodiments, the outputting, by the decision-making planning module, the system adjustment instruction to the environment perception module based on the prediction result information and the feedback result information comprises: calculating and obtaining, by the decision-making planning module, an adjustment parameter based on the prediction result information and the feedback result information, wherein the adjustment parameter is configured to adjust the state of the environment perception module; and outputting, by the decision-making planning module, the system adjustment instruction based on the adjustment parameter, wherein the system adjustment instruction comprises the adjustment parameter; and the adjusting, by the environment perception module, the state of the environment perception module based on the system adjustment instruction comprises: extracting, by the environment perception module, the adjustment parameter from the system adjustment instruction; and adjusting, by the environment perception module, the state of the environment perception module based on the adjustment parameter.

In some embodiments, the feedback result information comprises a reaction result of an operation object of the construction machine.

In some embodiments, the perception result information comprises the environment information; or the perception result information comprises a logical judgment result generated based on the environment information and/or the control result information.

In some embodiments, the control method further comprises: transmitting the environment information to the decision-making planning module by the environment perception module, wherein the environment information is first data information with an element coordinate system of the environment perception module as a coordinate system; and converting, by the decision-making planning module, the first data information into second data information with a reference coordinate system of the construction machine as a coordinate system based on a transformation matrix between the element coordinate system and the reference coordinate system, and generating the motion trajectory based on the operation task and the second data information.

In some embodiments, the control method further comprises: adjusting, by the decision-making planning module, the motion trajectory based on the perception result information of the environment perception module, and outputting the motion trajectory adjusted to the control performing module; and controlling, by the control performing module, motion of the construction machine based on the motion trajectory adjusted.

In some embodiments, the control method further comprises: adjusting, by the decision-making planning module, the motion trajectory based on the feedback result information of the control performing module, and outputting the motion trajectory adjusted to the control performing module; and controlling, by the control performing module, motion of the construction machine based on the motion trajectory adjusted.

In some embodiments, the control method further comprises: sending a control instruction to the control performing module by the decision-making planning module to control an operational state or operational speed of the control performing module to make the operational state or operational speed of the control performing module match an adjusted state of the environment perception module.

According to another aspect of the present disclosure, a control system for a construction machinery is provided. The control system comprises: a memory; and a processor coupled to the memory, wherein the processor is configured to, based on instructions stored in the memory, perform the control method as described previously.

According to another aspect of the present disclosure, a construction machine is provided. The construction machine comprises the control system as described previously.

According to another aspect of the present disclosure, a computer-readable storage medium is provided. The computer-readable storage medium has computer program instructions stored thereon that, when executed by a processor, implement the control method as described previously.

According to another aspect of the present disclosure, a computer program product is provided. The computer program product comprises instructions that, when executed by a processor, cause the processor to perform the control method as described previously.

According to another aspect of the present disclosure, a computer program is provided. The computer program comprises instructions that, when executed by a processor, cause the processor to perform the control method as described previously.

Other features and advantages of the present disclosure will become clear from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings.

Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that: unless additionally specified, the relative arrangements, numerical expressions and numerical values of the components and steps expounded in these examples do not limit the scope of the present disclosure.

At the same time, it should be understood that, for ease of description, the dimensions of various parts shown in the accompanying drawings are not drawn according to actual proportional relations.

The following descriptions of at least one exemplary embodiment which are in fact merely illustrative, shall by no means serve as any delimitation on the present disclosure as well as its application or use.

The techniques, methods, and devices known to a common technical person in the relevant art may not be discussed in detail, but where appropriate, the techniques, methods, and devices should be considered as part of the description.

Among all the examples shown and discussed here, any specific value shall be construed as being merely exemplary, rather than as being restrictive. Thus, other examples in the exemplary embodiments may have different values.

It is to be noted that: similar reference numerals and letters present similar items in the following accompanying drawings, and therefore, once an item is defined in one accompanying drawing, it is not necessary to make further discussion on the same in the subsequent accompanying drawings.

In the related art, smart construction machine focuses on smart operation, and since the tasks, objects and environments of a construction machine operation are more intricate and volatile than the driving task and road environment of self-driving automobile, the sequential architecture of perception-planning-control might no longer meet the operation requirements in some circumstances. For example, when a loader works, a walking mechanism and a bucket move at the same time to complete a combined action. If decision-making is not well-planned, the bucket will block the visual field of the sensor pointing to the traveling direction. During operation of the excavator, the slewing mechanism, the jib, the arm and the bucket move at the same time to complete a complex operation action. No matter where the vehicle-mounted sensor is installed, it will move along with the machine. In some circumstances, the sensor no longer points to the traveling direction or the operation direction, and so forth. Therefore, the sequential architecture of perception-planning-control of the construction machine in the related art might lead to relatively poor coordination of the construction machine during the process of performing operations.

In view of this, the embodiment of the present disclosure provides a control system or control method for a construction machine to improve the coordination of the construction machine during the process of performing operations.

is a structural block view schematically showing a control system for a construction machine according to some embodiments of the present disclosure. As shown in, the control system comprises: an environment perception module, a decision-making planning moduleand a control performing module. For example, the control system may be applied in smart construction machine such as an excavator and a loader and unmanned smart systems.

The environment perception moduleis configured to receive environment information and/or control result information, output perception result information based on the environment information and/or the control result information, receive a system adjustment instruction, and adjust a state of the environment perception module based on the system adjustment instruction.

That is, the environment perception modulemay receive the environment information and output the perception result information based on the environment information; or the environment perception modulemay receive the control result information and output the perception result information based on the control result information; or the environment perception modulemay receive the environment information and the control result information, and output the perception result information based on the environment information and the control result information.

For example, when the construction machine just starts working, the environment perception module may receive the environment information and output the perception result information based on the environment information; the decision-making planning module generates and outputs a motion trajectory based on an operation task and the perception result information; the control performing module performs the operation task based on the motion trajectory, obtains control result information and outputs the control result information to the environment perception module, so that the environment perception module may receive the control result information. At this time, since the environment perception module has been receiving the environment information, the environment perception module receives the environment information and/or the control result information.

In some embodiments, the environment perception module comprises an environment perception element and a multi-degree-of-freedom platform. The environment perception element is installed on the multi-degree-of-freedom platform. For example, the environment perception element comprises, but is not limited to, a perception element (also referred to as a sensor) such as a camera, a laser radar or a millimeter-wave radar. The multi-degree-of-freedom platform may be a controllable platform or a multi-degree-of-freedom structure. For example, the multi-degree-of-freedom platform comprises, but is not limited to, some structures on the construction machine or an individually provided device.

For example, an installation location and an installation method of the environment perception element are determined according to the operation task.

If an internal parameter of the environment perception element may be changed (for example, a focal length or a frequency may be changed), a program interface may be provided to the decision-making planning module to control the change of its internal parameter.

For example, the environment perception element is installed on the multi-degree-of-freedom platform, so that the decision-making planning module can command an actuator to change a base point position and a perspective of the environment perception element within a certain range according to the operation task.

In some embodiments, the environment information comprises: environment image information shot by a camera and/or three-dimensional information (for example, the three-dimensional information comprises distance information, direction information or the like) detected by a radar. The control result information which is the change of own attitude and state of the construction machine in operation, and output by the control performing module, comprises, for example, attitude information (for example, a boom angle) of a boom or the like.

In the embodiment of the present disclosure, the environment perception moduleis configured to receive at least one of the environment information or the control result information, and output the perception result information to the decision-making planning modulebased on the at least one of the environment information or the control result information.

For example, the perception result information comprises the environment information. That is, the environment perception module may output the environment information as the perception result information directly to the decision-making planning module.

For another example, the perception result information comprises: a logical judgment result generated based on the environment information and/or the control result information. That is, the environment perception module may generate the logical judgment result based on the environment information and/or the control result information. For example, if the environment perception module determines that there is an obstacle ahead of the construction machine (for example, an excavator) according to the environment information and/or the control result information, the environment perception module generates a logical judgment result that there is an obstacle ahead. Then, the environment perception module outputs the logical judgment result as the perception result information to the decision-making planning module.

For another example, the environment perception module may also output the environment information and the above-described logical judgment result together as the perception result information to the decision-making planning module.

In the embodiment of the present disclosure, the environment perception moduleis further configured to receive a system adjustment instruction from the decision-making planning moduleand a state of the environment perception module based on the system adjustment instruction. This facilitates the environment perception module to achieve a relatively favorable perception function. For example, the state of the environment perception module comprises: an internal parameter value of the environment perception element and/or a parameter of the multi-degree-of-freedom platform. That is, the environment perception modulemay adjust the internal parameter value (for example, a focal length or the like) of the environment perception element and/or the parameter of the multi-degree-of-freedom platform (for example, an azimuth angle of the multi-degree-of-freedom platform, comprising a horizontal angle and/or a pitch angle, and the like) based on the system adjustment instruction. For another example, the state of the environment perception module further comprises a position or attitude of the environment perception element, or the like.

In some embodiments, the environment perception module is an environment perception circuit. That is, the environment perception module may be implemented by a circuit. In other embodiments, the environment perception module comprises a software part, or is implemented by combining software and hardware.