Workplace monitoring and forecasting of operator’s behavior in HRC applications

Operator’s safety in collaborative robotic systems should take precedence to compose an operator-friendly, efficient, and flexible HRC solution. This requirement is satisfied by the application of safety sensor & control network that monitors the collaborative workspace and tracks the operator redundantly with high accuracy.

SHERLOCK workspace monitoring solution

SHERLOCK workspace monitoring solution

In SHERLOCK, one step further, the monitoring solution consists of an additional, soft safety feature that evaluates the work cell’s past and present state and proactively avoids potential HR contact, by performing a forecast of the future trajectory of the operator using state of the art AI techniques, under the name of FTF. The trajectory planner of the robot generates paths that, if possible, avoid overlapping with the operator’s future ones.

Prior to the integration of the physical robotic cell, a custom motion planner and controller was developed, to simulate human motion within the implemented virtual representation of the cell. After executing a process simulation, the virtual robot and virtual human data were acquired, processed and used to format a synthetic dataset.

Digital representation of the solar panel’s assembly cell

Digital representation of the solar panel’s assembly cell

For the generation of the real-life dataset, the solution used data that are captured from the implemented sensor network during the process to perform the prediction. More specifically, the operator’s position tracking is performed by combining an IR and an RGBD sensor, that capture the (x, y) coordinates of the human. The position of the robot is acquired from the developed COMAU – ROS drivers, returning thus the joint rotations within the ROS environment. Several process executions were performed in the physical cell, in order to collect and process the human-robot data. The datasets then, were used to train a multilayer, LSTM based network that forecasts human motion during the process execution.

FTF enabled HRC

FTF enabled HRC

The Shopfloor Digital Representation (SDR) module, that is responsible for the trajectory planning of the robot, receives the FTF result, and perceives it as an obstacle (ROS primitive object), thus generating a robot path that avoids it. If the goal position of the robot overlaps with the generated obstacles, the robot readjusts its speed (reducing its linear value), as this would minimize potential impact.