Work Support System for Excavator

The present application is a continuation application of International Application No. PCT/JP2023/045244 filed on Dec. 18, 2023, which is based on and claims priority to Japanese Patent Application No. 2022-203642 filed on Dec. 20, 2022. The contents of these applications are incorporated herein by reference.

The present disclosure relates to a work support system for an excavator.

In the related art, a work support system for supporting excavation motions and the like of an excavator in order to increase the work efficiency of the excavator, is known. For example, there is a work support system of the related art for evaluating the work quality of a plurality of operators, learning the parameters of the excavator operation model based on the operation data of the best operator, and supporting the operator based on the learned operation model.

In this kind of excavator work support system, by investigating and recognizing in advance earth and sand information such as soil quality of the location to be excavated (excavation object) by the excavator, the motion content of the excavator corresponding to the earth and sand information can be set in the excavation motion.

According to an embodiment of the present invention, there is provided a work support system for an excavator is provided, the work support system including the excavator; an environment detection device configured to detect environment information of a work site of the excavator; and a simulation device configured to acquire the environment information detected by the environment detection device during a motion of the excavator, and to generate a three-dimensional virtual space model of the work site.

In the actual work site, even if the earth and sand information is held beforehand, the earth and sand information changes easily depending on the excavation site, excavation depth, etc. Therefore, even if the work support system sets the motion content of the excavator based on the earth and sand information, the actual excavator may not be able to perform excavation as scheduled.

The present disclosure provides a work support system for an excavator capable of further improving the work efficiency of an excavator, by updating environment information in association with the motion of the excavator and setting motion contents based on the environment information.

Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. In each of the drawings, the same components are denoted by the same reference numerals and duplicate descriptions may be omitted.

First, the work support system SYSaccording to the embodiment of the present invention will be described with reference to.is a diagram illustrating a work support system SYSaccording to the embodiment.

An excavatorapplied to the work support system SYSincludes a lower traveling body, an upper turning bodymounted on the lower traveling bodyso as to be able to turn through a turning mechanism, an excavation attachment AT, and a cabin.

The lower traveling bodyof the excavatorof the present embodiment has a pair of right and left crawlersC. The crawlersC are driven by a traveling hydraulic motorM which is a traveling actuator mounted on the lower traveling body.

The upper turning bodyis mounted on the lower traveling bodyin a turnable manner via a turning mechanism. The turning mechanismis driven by a turning hydraulic motorA which is a turning actuator mounted on the upper turning body. The turning actuator may be an electric actuator (a turning electric generator).

A boomis mounted on the upper turning body. An armis mounted on the tip of the boom, and a bucketserving as an end attachment is mounted on the tip of the arm. The boom, the arm, and the bucketform an excavation attachment AT serving as an example of an attachment. The boomis driven by a boom cylinder, the armis driven by an arm cylinder, and the bucketis driven by a bucket cylinder. The boom cylinder, the arm cylinder, and the bucket cylinderform an attachment actuator. The end attachment may be a slope bucket.

The boomis supported to be vertically turnable with respect to the upper turning body. A boom angle sensor Sis attached to the boom. The boom angle sensor Sdetects a boom angle α which is a turning angle of the boom. The boom angle α is, for example, a rising angle from a state in which the boomis fully lowered. Therefore, the boom angle α becomes maximum when the boomis fully raised.

The armis rotatably supported with respect to the boom. An arm angle sensor Sis attached to the arm. The arm angle sensor Sdetects an arm angle β which is a rotation angle of the arm. The arm angle β is, for example, an opening angle from the fully closed state of the arm. Therefore, the arm angle β becomes maximum when the armis fully opened.

The bucketis rotatably supported with respect to the arm. A bucket angle sensor Sis attached to the bucket. The bucket angle sensor Sdetects the bucket angle γ which is the rotation angle of the bucket. The bucket angle γ is the opening angle from the state in which the bucketis closed most. Therefore, the bucket angle γ is maximum when the bucketis fully opened.

Each of the boom angle sensor S, the arm angle sensor Sand the bucket angle sensor Smay use only an acceleration sensor or may be a combination of an acceleration sensor and a gyro sensor. Alternatively, the boom angle sensor Smay be a stroke sensor attached to the boom cylinder, a rotary encoder, a potentiometer, or an inertial measurement device. The same applies to the arm angle sensor Sand the bucket angle sensor S.

The upper turning bodyis provided with the cabinserving as an operator cab, and a power source such as an engineis mounted. The upper turning bodyis provided with a space recognition device, an orientation detection device, and a positioning device, and is also provided with various sensors of the excavator, such as a machine body inclination sensor Sand a turning angular velocity sensor S. Further, an operation device, an output device, a controllerand the like are provided inside the cabin. In the present specification, for convenience of explanation, the side of the upper turning bodywhere the excavation attachment AT is attached is referred to as the front side, and the side where the counterweight is attached is referred to as the rear side.

The space recognition deviceis a device for recognizing a three-dimensional real space (environment information) around the excavator. The space recognition deviceis configured to measure the direction and distance to a recognized object from the space recognition deviceor the excavator. The space recognition deviceincludes, for example, an ultrasonic sensor, a millimeter wave radar, a monocular camera, a stereo camera, a LiDAR, a distance image sensor, an infrared sensor, or any combination thereof. In the present embodiment, the space recognition deviceincludes a front sensorF attached to the front end of the upper surface of the cabin, a rear sensorB attached to the rear end of the upper surface of the upper turning body, a left sensorL attached to the left end of the upper surface of the upper turning body, and a right sensorR attached to the right end of the upper surface of the upper turning body. The space recognition devicemay have an upper sensor (not illustrated) attached to the excavatorthat recognizes an object present in the space above the upper turning body.

The orientation detection devicedetects information related to the relative relationship between the orientation of the upper turning bodyand the orientation of the lower traveling body. The orientation detection devicemay be formed of, for example, a combination of a geomagnetic sensor attached to the lower traveling bodyand a geomagnetic sensor attached to the upper turning body. Alternatively, the orientation detection devicemay be formed of a combination of a GNSS receiver attached to the lower traveling bodyand a GNSS receiver attached to the upper turning body. The orientation detection devicemay be a rotary encoder, a rotary position sensor, or any combination thereof. In a configuration in which the upper turning bodyis driven to turn by a turning electric generator, the orientation detection devicemay be formed of a resolver. The orientation detection devicemay be attached to, for example, a center joint provided in relation to the turning mechanismfor implementing relative rotation between the lower traveling bodyand the upper turning body.

The orientation detection devicemay be formed of a camera attached to the upper turning body. In this case, the orientation detection deviceapplies known image processing to the imaging information captured by the camera to extract an image of the lower traveling bodyincluded in the imaging information. Then, the orientation detection deviceidentifies the longitudinal direction of the lower traveling bodyfrom the image of the lower traveling body, and derives an angle formed between the longitudinal direction of the upper turning bodyand the longitudinal direction of the lower traveling body. The longitudinal direction of the upper turning bodyis derived from the mounting position of the camera. In particular, because the crawlerC protrudes from the upper turning body, the orientation detection devicecan identify the longitudinal direction of the lower traveling bodyby detecting the image of the crawlerC. The orientation detection devicemay be integrated into the controller. The camera may also use a space recognition device.

The positioning deviceis configured to measure the position of the upper turning body. In the present embodiment, the positioning deviceis a GNSS receiver, which detects the position of the upper turning bodyand outputs the detected value to the controller. The positioning devicemay be a GNSS compass. In this case, because the positioning devicecan detect the position and the orientation of the upper turning body, the positioning devicealso functions as an orientation detection device.

The machine body inclination sensor Sdetects the inclination of the upper turning bodywith respect to a predetermined plane. In the present embodiment, the machine body inclination sensor Sis an acceleration sensor that detects the tilt angle of the upper turning bodyaround the longitudinal axis and around the lateral axis, with respect to the horizontal plane. The longitudinal axis and the lateral axis of the upper turning body, for example, are orthogonal to each other and pass through an excavator center point which is a point on the turning axis of the excavator.

The turning angular velocity sensor Sdetects the turning angular velocity of the upper turning body. In the present embodiment, the turning angular velocity sensor Sis a gyro sensor, but it may be a resolver, a rotary encoder, or any combination thereof. The turning angular velocity sensor Smay detect the turning velocity. The turning velocity may be calculated from the turning angular velocity.

Hereinafter, at least one of the boom angle sensor S, the arm angle sensor S, the bucket angle sensor S, the machine body inclination sensor S, or the turning angular velocity sensor Sis also referred to as a posture detection device. The posture of the excavation attachment AT is detected, for example, based on the respective outputs of the boom angle sensor S, the arm angle sensor S, and the bucket angle sensor S.

The operation deviceis a device provided in the cabinfor an operator to operate the excavator. For example, the operation devicehas an operating lever and an operating pedal for controlling the drive of the actuator of the excavator. The actuator includes at least one of a hydraulic actuator or an electric actuator.

Further, the operation devicehas an information input device (for example, a right console box, a left console box) for an operator of the excavatorto input information to the controller. The information input device may be, for example, a switch panel installed close to the display device of the output device. Alternatively, the information input device may be a touch panel applied as a display device or a voice input device such as a microphone placed in the cabin. The information input device may also be a communication device for obtaining information from the outside.

The output deviceincludes at least one of a display device and a sound output device. The display device is a liquid crystal display installed in the cabin. The display device may be a display of a portable terminal such as a smartphone. The sound output device includes at least one of a device for outputting sound to an operator in the cabinor a device for outputting sound to an operator outside the cabin. The sound output device may be a speaker of a portable terminal.

The controlleris a control device for controlling the excavator. In the present embodiment, the controlleris formed of a computer including one or more processors, a memory (volatile memory, non-volatile memory), and the like. The one or more processors read and execute a program corresponding to each function from the memory. For example, each function includes a machine guidance function for guiding a manual operation of the excavatorby an operator, or a machine control function for automatically (or autonomously) operating the excavator. The controllermay include a contact avoidance function for automatically operating or braking the excavatorto avoid contact between the excavatorand an object existing around the excavator.

That is, the work support is an expression including automatically operating the excavatoron behalf of the operator, assisting the operation of the excavatorby the operator, and providing operation information to the operator of the excavator. For the work support of the excavator, the work support system SYSapplies the controllermounted on the excavatorand performing each function, and the environment detection device(see) providing information to the controller. However, in addition to the work support system SYSbeing formed of the excavatoralone, a management apparatuscapable of communicating with the excavatormay be applied outside the excavator. By applying the management apparatus, the work support system SYScan support the motions of a plurality of excavators(or other work machines) so as to be interlocked. Further, the work support system SYSmay include an external space recognition devicehaving the same function as the space recognition device, outside the excavator.

Next, an example of the work support system SYSaccording to the embodiment will be described with reference to.is a functional block diagram illustrating a configuration example of the work support system SYS.

The excavatorincludes a controller, a space recognition device, an orientation detection device, a positioning device, various solenoid valves, various actuators, and a communication device Tas a configuration of the work support system SYS. On the other hand, the management apparatusincludes a computer bodyfor performing various processes of the work support system SYSand a communication device T.

The external space recognition devicedetects the state of the work site where the excavatoris located. The detection of the state of the work site includes, for example, measurement of distance, shape, and direction in addition to imaging the work site. For example, the external space recognition deviceincludes an ultrasonic sensor, a millimeter wave radar, a monocular camera, a stereo camera, a LiDAR, a distance image sensor, an infrared sensor, or any combination thereof installed in the work site. The external space recognition devicecommunicates, either wirelessly or via a wired connection, with at least one of the communication device Tof the excavatoror the communication device Tof the management apparatus, and successively transmits detection information of the detected state of the work site.

The controllerof the excavatorincludes a virtual space generation part, an excavator state identification part, an excavation state estimation part, an actuator driving part, a determination part, an operation prediction part, an operation intervention part, and a motion simulatoras functional blocks. Although the virtual space generation part, the excavator state identification part, the excavation state estimation part, the actuator driving part, the determination part, the operation prediction part, the operation intervention part, and the motion simulatorare distinguished for convenience of explanation, they do not need to be physically distinguished, and they may be composed entirely or partially of common software components or hardware components.

The computer bodyof the management apparatusincludes, as functional blocks, a determination part, an operation prediction part, an operation intervention part, and a motion simulator. Although the determination part, the operation prediction part, the operation intervention part, and the motion simulatorare distinguished for convenience of explanation, they do not need to be physically distinguished, and may be composed entirely or partially of common software components or hardware components. The determination part, the operation prediction part, the operation intervention part, and the motion simulatorhave the same functions as the determination part, the operation prediction part, the operation intervention part, and the motion simulatorof the controller. The management apparatusmay include a virtual space generation part(see the dotted line in) having the same function as the virtual space generation partof the controller. Hereinafter, each functional block of the controlleris described as a representative example, and descriptions of each functional block of the computer bodyare omitted. The work support system SYSwill suffice to have the functions of virtual space generation parts,, the determination parts,, the operation prediction parts,, the operation intervention parts,, and the motion simulators,in at least one of the excavatoror the management apparatus.

The virtual space generation partgenerates a three-dimensional virtual space model on the virtual three-dimensional coordinates in the virtual space generation partbased on the detection information of the space recognition device, the orientation detection device, and the positioning device, and the information of the operation device. The three-dimensional virtual space model is formed into a virtual rectangular parallelepiped, cube, sphere, or hemisphere according to the imaging range of the space recognition device. The three-dimensional virtual space model may be image information displayed on the display device of the output device. In this case, the three-dimensional virtual space model is a three-dimensional topographic image and is formed by computer graphics.

Typically, the three-dimensional virtual space model is information having a plurality of layers in which object information is superimposed on topographic information indicating the topography around the excavatorvisible to the operator seated in the cabin. Hereinafter, changeable information (parameters) applied to the three-dimensional virtual space model is referred to as environment information. The environment information includes topographic information and object information. The virtual space generation partperforms known image processing on the detection information captured by at least one of the front sensorF, the rear sensorB, the left sensorL or the right sensorR of the space recognition device, and extracts topographic information and object information included in the detection information. At this time, for example, the object included in the current imaging information may be extracted by comparing a plurality of pieces of past imaging information with the current imaging information. Then, the virtual space generation partarranges the extracted topographic information and object information in a three-dimensional virtual space model, and reproduces virtual environment information surrounding the cabin(operator) in the three-dimensional virtual space model.

The object information extracted for reproducing the three-dimensional virtual space model includes, for example, soil to be excavated (including mounds, holes, walls, ditches, etc.), static objects other than excavated objects, the excavatoritself, work machines such as another excavator, vehicles, animals including people, plants, and the like. In the arrangement of the object information with respect to the three-dimensional virtual space model, the virtual space generation partcan utilize information such as distance and direction between the excavatorand the object information measured by the ultrasonic sensor, millimeter wave radar, LiDAR, and the like. Thus, various objects in the work site where the excavatorexists and coordinates of the objects are appropriately reproduced in the three-dimensional virtual space model.

Further, the virtual space generation partmay receive the detection information of the external space recognition deviceand generate a three-dimensional virtual space model imaged by the external space recognition device. Alternatively, the virtual space generation partmay be configured to generate one three-dimensional virtual space model by integrating the detection information of the space recognition deviceof the excavatorand the detection information of the external space recognition device.

is a diagram illustrating an example of a three-dimensional virtual space modelgenerated by the virtual space generation part. For example, a virtual excavatorobtained by reproducing the excavatorin a real space, is arranged in the three-dimensional virtual space modelof the virtual space generation part. The virtual excavatorcorresponds to the shape, position, posture, etc., of the excavatorat an actual work site. The position, posture, and the like of the virtual excavatorin the three-dimensional virtual space modelare identified by an excavator state identification partdescribed below, and are determined, for example, based on the detection information of at least one of the space recognition deviceor the orientation detection device. The position, posture, and the like of the virtual excavatormay be determined or adjusted by using the detection information of a posture detection device, the positioning device, and the like.

In the three-dimensional virtual space model, the excavation object(mound in FIG.), which is the topographic information existing around the virtual excavator, and a virtual dump truck(discharge object), which is the object information, are arranged, including the shape itself. Although the illustration of other environment information is omitted in, the object information extracted from the detection information is appropriately arranged in the three-dimensional virtual space model. Further, the virtual space generation partmay generate the three-dimensional virtual space modelby leaving an image that is difficult to extract in the detection information as the background of the three-dimensional virtual space model.

Further, the virtual space generation partaccording to the present embodiment associates (adds) various kinds of additional information with the topographic information and object information reproduced in the three-dimensional virtual space model.

For example, in addition to the shape, position, or posture, additional information of the excavator such as an identification number, a type, a working time, a type and a posture of the excavation attachment AT, and earth and sand information (additional information) such as a weight, a volume, and a density of the earth and sand put in the bucket, are added to the virtual excavator. The additional information of the excavator may be estimated based on the detection information of the environment detection device, or information previously stored in the controllermay be used. Further, the earth and sand information stored in the excavatorcan be estimated based on information obtained by detecting the load applied when the excavatoris performing excavation, detected by a pressure sensor, a load sensor, or the like, which is a part of the environment detection device.

That is, the environment detection deviceis a device that detects information that affects the environment information constituting the three-dimensional virtual space modelinside or outside the excavator. The environment detection devicemay include various sensors such as a space recognition device, an orientation detection device, a positioning device, a posture detection device (the boom angle sensor S, the arm angle sensor S, the bucket angle sensor S, the machine body inclination sensor S, and the turning angular velocity sensor S), a pressure sensor not illustrated, a load sensor, and an operation sensor of the operation device. The environment detection devicemay also include detection information of a device (the external space recognition device, other work machines and vehicles (for example, a dump truck)) installed outside the excavator.

Further, for example, additional information of an excavation object such as the sediment accumulation amount, weight, density, hardness, and soil quality of the earth and sand is added to the excavation objectof the three-dimensional virtual space model. The additional information of an excavation object is estimated by the excavation state estimation partdescribed below based on the detection information of the space recognition device. Alternatively, the additional information of an excavation object may be estimated by the pressure and load applied to the bucketof the excavatorwhen the excavatoroperates, the weight of earth and sand recognized by the dump truck when the earth and sand is discharged to the dump truck, an image, or the like. Alternatively, a part of the additional information of the excavation object may be extracted from the design data of the work site stored in advance in the controller.

On the other hand, in addition to the shape, position, and posture, additional information of the discharge object such as identification number, type, size of the loading platform, state of the loading platform, and working time are added to the virtual dump truckof the three-dimensional virtual space model. Further, earth and sand information (additional information of the discharge object) such as weight, volume, and density of earth and sand loaded on the loading platform of the virtual dump truck(the discharge object) may be added to the loading platform of the virtual dump truck. The additional information of the discharge object may be estimated based on object information extracted from the detection information of the space recognition device, or information previously stored in the controllermay be used. The earth and sand information is added to the virtual dump truckof the three-dimensional virtual space modelby detecting, at the actual dump truck, the weight, image, and the like of the earth and sand that changes when the earth and sand is discharged from the excavator, and receiving the information.

The additional information to be added to the environment information may include various kinds of information other than the above. For example, when a person or another work machine exists around the excavator, the object information thereof is arranged in the three-dimensional virtual space model. Additional information of the relative distance to the excavatorcan be added to the object information. Further, additional information such as identification number, type, and working time can be added to the information of other work machines.

The virtual space generation partmay integrate the design data of the work site stored in the controllerinto the three-dimensional virtual space model. The design data has the completed shape of the excavation of the work site, and the virtual space generation partsuperimposes and displays, on the three-dimensional virtual space model, graphics such as computer graphics indicating the position of the completed shape. Further, the design data may include the state of earth and sand (position, shape, soil quality, hardness) of the excavation site investigated in advance, and this information may be added to the topographic information of the three-dimensional virtual space model.

Then, the virtual space generation partsequentially updates the environment information of the three-dimensional virtual space modelbased on the detection information of the space recognition device, the orientation detection device, and the positioning device, information of the operation device, and communication information from the external space recognition deviceand the dump truck. For example, when the excavatorexcavates the excavation object of the work site, the environment information of the three-dimensional virtual space modelmay be changed based on the detection information at that time. Even if the earth and sand information (additional information) of the excavation objectis added in advance, the earth and sand information may be different when the excavatorhas actually performed the excavation. In such a case, the virtual space generation partchanges or corrects the earth and sand information based on the detection information detected when the excavatoractually has actually performed the excavation. Thus, the virtual space generation partcan generate the three-dimensional virtual space modelthat is even closer to the work site environment.

Referring back to, the excavator state identification partis configured to identify the state of the excavator(posture of the excavation attachment AT, etc.) including the position and orientation of the excavator. The position of the excavatoris, for example, the latitude, longitude, and altitude of the reference point of the excavator. The excavator state identification partidentifies the position of the excavatorbased on the output of the positioning device, and identifies the orientation of the excavatorbased on the output of the orientation detection device. The position and orientation of the excavatoridentified by the excavator state identification partare reflected on the virtual excavatorof the three-dimensional virtual space model.