Method for Controlling the Positioning of an Autonomous Forklift Relative to an Annular Load to Be Lifted

The present invention relates to the field of autonomous vehicles for the automated transportation of loads, such as autonomous forklifts.

Autonomous vehicles for transporting loads are increasingly being used to increase productivity and improve logistics management in factories or in warehouses.

Automated forklifts are one example of such vehicles and make it possible for example for a load to be loaded, transported and positioned at height without human intervention.

However, in environments such as factories or warehouses, human intervention is still required in addition to the automated operations, for example to control the satisfactory progress of these operations or to perform tasks that cannot be carried out by machines alone. These environments are therefore shared between humans and autonomous machines.

Personal safety is of fundamental importance in such working environments and accordingly requires that specific procedures be put in place.

For example, during operations to pick a load from storage shelving, the position of the forklift relative to the load must in particular be controlled.

If the positioning of the forklift relative to the load is incorrect, there is a risk that the load will tip when it is picked, and therefore the risk of an accident.

In light of the above, the aim of the invention is to increase the safety of load picking operations by a forklift.

The invention relates to a method for controlling the positioning of an autonomous forklift relative to an annular load to be lifted and transported, said forklift comprising a vertically movable fork provided with at least one arm.

The method comprises a step of acquiring at least one image of the annular load by means of at least one 3D camera rendering a point cloud of coordinates measured in a coordinate system associated with said 3D camera, said 3D camera being fastened to the autonomous forklift centred relative to the fork and having a line of sight oriented forwards along a longitudinal axis of said forklift.

The method further comprises a step of digitally processing said image obtained in the acquisition step, comprising a sub-step of detecting at least one circle located on a front face of the annular load and corresponding to the inner diameter or the outer diameter of the annular load, and a sub-step of calculating an offset, in a plane perpendicular to the line of sight, between the centre of said detected circle and the centre of said 3D camera, said offset representing a centring error of the forklift relative to the annular load.

The method also comprises a sub-step of determining a plane corresponding to the front face of the annular load, and a sub-step of calculating the angle between the normal to said determined plane and the line of sight of said 3D camera, said angle representing an angle error of the forklift relative to the annular load.

The method further comprises a step of stopping the forklift if at least one of the calculated angle error or centring error values is greater, as an absolute value, than a predefined limit value.

“Front face” of the annular load is given to mean the frontal face of the load that is oriented on the same side as the camera.

Such a method makes it possible to determine the relative position between the load and the forklift and to increase the safety of operations by stopping the forklift if at least one centring error or angle error value exceeds a predefined limit value.

Advantageously, the digital processing step can further comprise sub-steps of calculating the depth and the outer diameter of the annular load, said step of stopping the forklift being carried out if the calculated values of the depth and the outer diameter of the annular load do not correspond to predefined values.

The load is picked by the forklift only if the depth and outer diameter measurements correspond to predefined values, thus making it possible to ensure that the load in front of the forklift matches the load to be picked, and therefore further increase the safety of operations.

In one embodiment, the method further comprises a prior step of the forklift receiving an assigned mission comprising the predefined values of the depth and the outer diameter of the annular load to be transported.

16 17 According to one feature, the depth of the annular load is calculated on the basis of the inner circle corresponding to the inner diameter of said annular load and having a diameter D, an inner circle located on a rear face of the annular load and corresponding to an inner surface of said annular load and having a diameter D, and the distance d between said 3D camera and the front face of the load, using the following equation:

“Rear face” of the annular load is given to mean the frontal face of the load that is oriented on the opposite side from the camera. The front and rear faces define the thickness of the load.

16 17 According to another feature, the depth of the annular load is equal to the maximum difference between the coordinates of the points of an inner surface of the annular load between the inner circle corresponding to the inner diameter of said annular load and having a diameter D, and an inner circle located on a rear face of the annular load and corresponding to an inner surface of said annular load and having a diameter D, said difference being measured along the axis of the coordinate system oriented along the line of sight of said 3D camera.

Advantageously, the acquisition step can comprise the following successive sub-steps: a first sub-step of acquiring at least one image, a sub-step of advancing the forklift towards the annular load by a predefined distance, and a second sub-step of acquiring at least one image carried out when the forklift has covered said predefined distance.

For example, the sub-steps of calculating the centring error and the angle error of the forklift relative to the annular load are carried out after the first acquisition sub-step, and the sub-steps of calculating the depth and outer diameter of the annular load are carried out after the second acquisition sub-step.

For example, the detection sub-step comprises detecting the inner circle corresponding to the inner diameter of said load, and detecting the outer circle corresponding to the outer diameter of the annular load.

Advantageously, the digital processing step comprises a sub-step of filtering the images obtained in the acquisition step. This filtering sub-step is carried out before the other sub-steps of the digital processing step and makes it possible to reduce the amount of data to be processed by filtering the points of the images obtained in the acquisition step in order to keep only the points that form part of a region of particular interest. By limiting the calculations to the reduced areas of the region of particular interest, the method obtains results more quickly without adversely affecting the quality thereof. By limiting the data to the regions of interest, the size of the memory required for the command and digital image processing module associated with the 3D cameras is also reduced.

According to one advantageous feature, the image acquisition step comprises a prior sub-step of calculating an optimum gain of said 3D camera used for image acquisition.

In one particular embodiment, the step of acquiring at least one image of the load is carried out by a single 3D camera. As a variant, the acquisition step can be carried out by a plurality of 3D cameras.

According to another aspect, the invention relates to an autonomous forklift comprising a vertically movable fork provided with at least one arm, at least one 3D camera for acquiring at least one point cloud data image, said 3D camera being fastened to the autonomous forklift centred relative to the fork and having a line of sight oriented forwards along a longitudinal axis of said forklift, a command and digital image processing module associated with the 3D camera, a wireless telecommunication device, and a unit for controlling said forklift configured to implement a method as described above.

Preferably, the line of sight of said 3D camera is horizontal and parallel to the longitudinal axis of said arm of the fork.

Preferably, the line of sight of said 3D camera is situated at a distance of less than 100 mm from a horizontal plane containing said arm of the fork.

1 FIG. 1 depicts the main elements of an autonomous forkliftaccording to one embodiment of the invention.

1 The architecture of the forkliftis given by way of example and does not limit the invention to the architectural configuration depicted alone. It will be understood that the invention also relates to forklifts designed to operate in manual mode and which have been adapted to enable a second autonomous operating mode.

1 2 3 4 4 4 4 1 FIG. a b The autonomous forkliftillustrated incomprises a lifting membercomprising a carriagethat bears a forkcomprising two arms,that are spaced apart laterally and extend towards the front of the forklift. As a variant, the forkcould comprise just one arm.

3 3 3 3 3 3 a b c a The carriageforms a frame. The carriageis provided with a horizontal upper crossmember, and two arms,extending the upper crossmembervertically downwards.

4 4 4 3 4 4 4 4 3 3 a b a b a b b c The forkalso comprises two uprights′,′, which are fastened to the carriageand each bear one of the arms,. Each of uprights′,′is fastened to one of the arms,of the carriage.

4 4 4 4 4 4 a b a b a b The arms,of the fork are generally used for insertion into entry openings provided in the transport pallets bearing the loads to be lifted. The uprights′,′allow the arms,to be raised so that a pallet to be transported or another type of load can be lifted and so that a pallet or another type of load can be positioned or collected at height.

4 3 5 4 4 5 4 4 4 4 a b a b a b 1 FIG. The forkcan move in translation in a vertical plane V defined by the carriage, along a vertical mastof the forklift. The uprights′,′can slide along the mast. The arms,of the fork can move between an uppermost position and a lowermost position that is illustrated inand that corresponds to a running position. In the lowermost position, the arms,are situated at a distance from the ground.

4 4 4 4 4 4 4 4 a b a b a b The longitudinal axes of the arms,are parallel. These longitudinal axes are oriented parallel to a horizontal axis X and define a horizontal plane H referred to as the lifting plane. The arms,of the forkare perpendicular to the vertical plane V. The arms,of the forkcan preferably also move laterally relative to each other.

4 4 a b As a variant, the arms,could also be telescopic or retractable and/or able to be oriented angularly about their longitudinal axis.

1 6 1 1 As is known per se, the forkliftis provided with a drive systemenabling the forkliftto move. The drive system comprises at least one electric motor or combustion engine (not shown) providing drive to the wheels of the forklift.

1 7 8 9 7 8 3 FIG. The forkliftis also provided with an on-board locator device, an on-board wireless telecommunication deviceand an on-board control unit() receiving information from the locator deviceand from the wireless telecommunication devicein order to autonomously command the movement of the forklift.

1 10 11 11 3 FIG. The forkliftis also provided with at least one 3D camerafor measuring time of flight (TOF), and an associated command and digital image processing module(), referred to as the vision module, which commands the capturing of images by the 3D camera and receives and interprets the images captured by the 3D camera. In the remainder of the description, the modulewill be referred to as the vision module.

10 10 10 10 10 2 FIG. As is known per se, the 3D camerais able to capture an image of an object and render a point cloud of relative coordinates measured with respect to a frame of reference associated with the 3D camera. The frame of reference associated with the 3D cameracomprises a coordinate system R made up of three orthogonal axes X1, Y1, Z1, as illustrated in. Here, the axis Z1 of the coordinate system is oriented along the line of sight of the 3D camera, and the axis Y1 of the coordinate system is oriented along a horizontal axis Y perpendicular to the line of sight of the 3D camera. A 3D camera is able to measure distances along all three axes of the coordinate system associated therewith. For example, as illustrated, the horizontal, vertical or depth offset relative to the 3D camerais measured along the axes Y1, X1 or Z1 respectively.

10 7 10 11 10 11 The 3D camerais separate from the locator device. Here, the 3D camerais separate from the vision module. Alternatively, the 3D cameraand the vision modulecould form a single assembly.

9 6 7 8 The control unitcomprises the hardware and software for commanding the operation of the drive systemon the basis of the information received from the locator deviceand the telecommunication device.

9 6 11 11 9 9 2 The control unitalso commands the operation of the drive systemon the basis of the data from the vision moduleand is configured to communicate therewith. Alternatively, the vision modulecould be integrated into the control unit. The control unitalso makes it possible to command the autonomous movement of the lifting member.

1 2 FIGS.and 10 4 1 10 As illustrated in, the 3D camerais centred relative to the forkand its line of sight is oriented forwards along a longitudinal axis of the forklift. Preferably, the line of sight of the 3D camerais horizontal and parallel to the lifting plane H.

10 3 10 4 10 3 10 3 10 The 3D camerais fastened to the carriageat a constant height relative to the lifting plane H and without any possibility of moving relative to said carriage. The 3D camerais able to move conjointly with the fork. Here, the 3D camerais fastened to the lower part of the carriageabove the lifting plane H. In some variants (not shown), the 3D cameracan be fastened to the carriagelevel with the lifting plane H or below said plane. Preferably, the line of sight of the 3D camerais situated at a relative distance of less than 100 mm from the plane formed by the arms of the fork, that is the lifting plane H.

10 1 As described in greater detail below, the 3D camerais able to capture images of the environment in front of the forklift, in order to determine the actual position of the loads to be lifted and transported by the forklift.

8 9 1 1 The wireless telecommunication deviceis configured to communicate with the control unitand with a computerized warehouse management system (WMS) (not shown) that is remote from the forkliftand intended to manage the operations of a storage warehouse and command a fleet of forklifts.

1 1 1 Each forkliftreceives information in the form of periodic electronic messages sent by the WMS regarding missions that are assigned to it relating to journeys to be made and/or loads to be transported. Each forkliftis capable of sending the WMS electronic messages representing the status of the missions assigned to it. A forkliftcan for example report an error encountered during the performance of a mission by sending an error code.

1 12 13 20 12 12 12 5 FIG. 3 5 FIGS.to The operating principle of the autonomous forkliftfor the operation of picking an annular loadsituated on a shelving structureand the method′ for controlling the relative positioning between the forklift and the loadillustrated inwill now be described with reference to. An annular loadgenerally comprises a support consisting of a hollow cylindrical core, generally provided with solid or spoked circular flanges, intended to receive a payload that is wound around the core. The payload can for example comprise a wire, a cable, a fabric or a rubber strip, etc. Alternatively, the annular loadcan be a tyre.

20 20 8 The method′ starts with a prior receiving stepin which the wireless telecommunication deviceof the forklift receives an instruction relating to an annular load to be transported, constituting a mission assigned to the forklift by the WMS. The instruction comprises information relating to the type of load to be transported, the dimensions of the load, the current position of the load and the destination thereof. Particularly with regard to the dimensions of the load, the instruction comprises the values of the depth of the inner surface and the outer diameter of the load.

1 1 In one variant embodiment, the instruction relating to an annular load to be transported can be predefined and contained in a memory of the forklift. The values of the depth of the inner surface and outer diameter of the load can particularly be predefined and contained in a memory of the forklift.

20 21 10 11 12 9 1 11 10 10 12 1 11 10 10 3 FIG. 4 FIG. The method′ continues with a stepof acquiring at least one image of the annular load by means of the 3D camera, which is commanded by the vision module. The image is acquired at a predefined distance D1 () from the load. When the forklift is situated at this predefined distance D1, the control unitstops the forkliftand sends an image acquisition request to the vision module, which commands the capturing of images by the associated 3D camera. The 3D camerathus captures at least one image of the loadsituated in front of the forkliftand sends it to the vision module. A simplified example of a captured image is illustrated in. The images captured by the 3D camerarender a point cloud of coordinates X1, Y1, Z1 in the coordinate system R associated with the 3D camera.

21 21 a Preferably, the acquisition stepcomprises a prior sub-stepof calculating an optimum gain of the 3D camera used for image acquisition. The gain is considered to be optimum when the brightness values of an image are evenly distributed over the entire dynamic range of the image sensor. The dynamic range of a sensor is equal to the difference between the highest light levels and the lowest light levels that can be detected by the sensor.

In order to spread the brightness values over the entire dynamic range and to avoid outliers, the optimum gain is obtained on the basis of the maximum value of the dynamic range of the image sensor of the 3D camera and the brightness values of an image previously acquired in the given working conditions and with a unity gain, and on the basis of the ratio between the maximum value of the dynamic range and the brightness value that corresponds to the ninth decile of said image.

21 23 11 10 11 After the acquisition step, the method continues with a stepof digitally processing the images obtained, carried out by the vision module. It should be noted that the digital processing of the images can start as soon as the image captured by the 3D camerais made available to the vision module.

23 14 16 12 17 12 a 4 FIG. The digital processing step comprises a sub-stepof detecting circles,located on a front face of the load, illustrated in, and a circlelocated on a rear face of the load.

14 12 16 17 12 14 16 17 The circles are detected such that the circleis the circle that is observed on the front face of the load and corresponds to the outer surface of the load, and the circles,are respectively the circles that are observed on the front face and the rear face thereof and correspond to the inner surface of the load. The circles,andare detected using known circle detection methods, for example the Hough transform method.

14 16 17 23 14 16 10 1 12 b 2 FIG. The sub-step of detecting inner circles,and outer circleis followed by a sub-stepof calculating an offset measured in a plane perpendicular to the line of sight (i.e. along the axes X1 and Y1 of the coordinate system R in), between the centre of one of the inner circles,detected and the centre of the 3D camerarepresented by the origin of the coordinate system R. This offset represents the centring error of the carriagerelative to the load.

12 23 14 16 23 17 e e 14 16 17 17 16 4 FIG. The outer and inner diameters of the loadare determined in a sub-step. In this step, these outer and inner diameters are considered to be equal to the diameters D, D() of the outer circleand inner circlethat are observed on the front face of the load. During this sub-step, the diameter Dof the inner circleis also determined. The value of the diameter Dis less than the value of the diameter Ddue to the vanishing angle of the image.

23 23 12 c The digital processing stepalso comprises a sub-stepof determining a plane that corresponds to the front face of the load. This process can for example be carried out using an iterative method based on a RANSAC (random sample consensus) algorithm that eliminates outliers.

23 23 10 c d This sub-stepis followed by a sub-stepof calculating the angle between the normal to the plane determined and the line of sight of the 3D camera. This angle represents the angle error of the carriage relative to the load.

23 23 12 f The digital processing stepalso comprises a sub-stepof calculating the depth P of the annular load.

16 17 16 17 23 12 10 e In one embodiment, the depth P is calculated on the basis of the diameter values Dand Dof the circles,determined in sub-step, and the value of the distance d between the front face of the loadand the 3D camerathat is determined thereby, by applying the following equation:

12 15 12 10 Alternatively, in another embodiment, for example when the contrast between the front and rear faces of the loadis insufficient, the depth P of the load is calculated as being equal to the maximum difference between the coordinates of the points of the inner surfaceof the annular load, measured along the axis of the coordinate system R oriented along the line of sight of the 3D camera.

23 24 23 23 20 b d After the digital processing step, the method continues with a stepof comparing the values calculated during sub-steps,to predefined values sent by the WMS in stepor stored in the memory of the forklift.

24 9 1 25 During the comparison step, if the calculated value of the centring error of the forklift relative to the load is greater, as an absolute value, than a predefined associated limit value, the control unitcommands the forkliftto stop for correction by an operator (step).

9 1 25 Similarly, if the calculated value of the angle error of the forklift relative to the load is greater, as an absolute value, than a predefined associated limit value, the control unitcommands the forkliftto stop for correction by the operator (step).

These two limit values can be determined according to the permissible tolerances of the centring errors and angle errors relative to the safety requirements for lifting and transport operations.

24 23 23 20 9 1 25 e f 14 During step, the values calculated during sub-steps,are also compared to the dimensions of the load received in stepor stored in the memory of the forklift. If the calculated value of the outer diameter Dof the load is greater than the predefined outer diameter value received, the control unitcommands the forkliftto stop for correction by the operator (step).

20 9 1 25 Similarly, if the calculated value of the depth of the load is greater than the predefined depth value received in stepor stored in the memory of the forklift, the control unitcommands the forkliftto stop for correction by the operator (step).

25 9 During stepof stopping the forklift, the control unitcan send the WMS an error message representing the error encountered.

26 Conversely, if the calculated values correspond to the associated predefined values, the control unit of the forklift controls the lifting and transport operation (step).

21 12 21 In the exemplary embodiment described above, the stepof acquiring one or more images is carried out at the distance D1 from the loadonly. Alternatively, the acquisition stepcan comprise a first sub-step of acquisition at the distance D1, followed by a second sub-step of acquisition at a shorter distance D2 from the load, which therefore occurs after the forklift has advanced.

10 12 This two-stage acquisition can be beneficial for loads with large dimensions, for which the viewing angle of the 3D cameramight be too small to capture both the inner surface and the outer surface of the load. In this case, the first acquisition sub-step is carried out so that the outer diameter of the load can be calculated, and the second acquisition sub-step is carried out so that the inner diameter and the depth of the inner surface of the loadcan be calculated.

23 23 23 23 23 23 b d f e b f In this case, the sub-steps,of calculating the centring error and the angle error of the forklift relative to the annular load can be carried out after the first acquisition sub-step, and the sub-steps,of calculating the depth and outer diameter of the annular load can be carried out after the second acquisition sub-step. Alternatively, it could be possible to carry out sub-stepstoafter the second acquisition sub-step.

23 23 23 14 16 f e a In the exemplary embodiments described, the sub-steps,of calculating the depth and outer diameter of the annular load are provided in order to increase the safety of operations. As a variant, these steps could be omitted. In this case, in the detection sub-step, only one of the circlesorcan be detected.

23 23 23 23 15 12 11 23 g g g In the exemplary embodiment illustrated, the digital processing stepcomprises a prior additional sub-stepof filtering points. This sub-stepis carried out before the other sub-steps of the digital processing stepand makes it possible to reduce the amount of data to be processed by filtering the points of the images obtained in the acquisition step in order to keep only the points that form part of a region of particular interest. For example, the inner surfaceof the loadcan constitute a region of particular interest. By limiting the calculations to the reduced areas of the region of particular interest, the method obtains results more quickly without adversely affecting the quality thereof. By limiting the data to the regions of interest, the size of the memory required for the vision moduleis also reduced. Alternatively, this filtering sub-stepcould be omitted.