Figure 1: Right sequence: object being picked up as seen by the vision camera. Left Sequence: General view of the object being picked up

The tests were successful, as 90% of the objects used for the test were correctly identified, located, grasped, and moved to their target location. These results give a high confidence that the prototype will reach the desired performance when deployed at the ECORESET pilot site.

2.    ECORESET pilots

In December 2021, CERTH, IBEC, ROBOTNIK and SADAKO travelled to the ECORESET premises in Athens to perform a first phase of the pilot integration. During this visit, the camera framework designed by Sadako for the Disassembly workbench was installed alongside the workbench designed and built by ECORESET. This camera framework is designed to accommodate the cameras for operator gesture detection as well as the ones for object detection for cobot disassembly tasks, as shown in the following images:

Figure 2: Left image: constructed camera framework over the disassembly workbench. Right image: CAD view of the designed camera framework

This first integration step was an opportunity of successfully testing the operator location detection and the gesture detection software in their final environment, as well as for other the partners present during this visit to test their latest software and hardware developments:

Figure 3: HR Recycler partners working during the Ecoreset pilots

In particular, an integration test involving IBEC, ROBOTNIK and SADAKO was designed to test IBEC’s central task planner:

Figure 4: IBEC-ROBOTNIK-SADAKO integration test. Sadako’s vision software output is visible on the monitor on the top left of the image

The test was designed so that the pallet truck had to perform a sequence given by the task planner which involved raising and lowering the pallet in a given sequence. The task planner and interaction manager were configured so that a specific gesture made by the operator and detected by the gesture recognition software would interrupt the pallet truck sequence, and another one would resume it. This test was successful and showed the correct flow of information between the different sensors, software, and robotic actuators.

The latest progress in the HR-Recycler project involves a great deal of effort towards the realization of integration tests at the pilot sites. The results gathered from these preliminary tests are promising and hint at a successful deployment of the complete HR-Recycler pipeline during the following year.

PROMOTING RELIABLE HUMAN-ROBOT INTERACTIONS VIA AN INTERACTION MANAGER APPLICATION.

Human-Robot Interactions (HRI) have a leading role in hybrid human-robot collaborative environments (i.e recycling plants for Waste Electrical and Electronic Equipment, WEEE. See https://www.hr-recycler.eu/), where qualified human workers need to interact with their robotic co-workers reliably and effectively. However, in this type of hybrid environment, HRI becomes challenging given the complexity of the agents involved, the work to be done, and the environment where the interactions occur.

To assure effective and safe interactions in the context of HR-Recycler, at IBEC (https://specs-lab.com/) we are developing a control module able to orchestrate bidirectional interactions between human workers and their robotic counterparts. For this purpose, an Interaction Manager has been designed to inform the human worker about the state of its workplace.

The Interaction Manager is integrated within a ROS infrastructure from which has access to other cognitive modules’ information, such as the Worker Model, the Moral Engine or the Task Planner. Additionally, the Interaction Manager provides a visual interface that allows the humans in the working environment (the recycling plant) to interact with the non-human agents via a tablet device (Figure 1).

Figure 1. Tablet with the HR-Recycler Interaction Manager App displaying information regarding worker’s demographics, moral preferences and status of the Task Planner.

This application visualizes the worker’s information when detected by SADAKO’s computer-based vision modules. Specifically, the worker’s ID, role, language, and trust level is provided by the Worker Model. Moreover, once the worker is identified and the Moral Engine adjusts the robot’s actuation speed and the safety distance, these personalized parameters are shown in the tablet, allowing the worker to better understand the robot’s performance.

The workflow status of the a-cell has been divided into two disassembling parts allowing the agents involved to work in parallel. In the “Disassembling Process A”, the robot’s workflow state is displayed by querying the Task Planner current status. Moreover, a progress bar reports how far is the robot to complete its disassembling process. The “Disassembling Process B” panel will enable the worker to notify when the device has been fully disassembled.

Importantly, the HR-Recycler’s Interaction Manager anticipates that human-robot interactions will not always occur in trouble-free situations. Given the complexity of disassembling tasks that robots must perform, some additional difficulties may arise, which will be notified by an alert panel reporting the error code and its source (Figure 2). Then it is time for the human worker to decide how the problem will be solved. To that end, relevant options are displayed and the selected solution will be sent back to the Task Planner.

Figure 2. Task Planner alert message providing information of the problem source and offering options to proceed.

Another source of exceptions in the normal workflow is the violation of personalized safety measures. In the case that the human worker surpasses the Human-Robot safety distance, another alert panel is dedicated to reporting such an event (Figure 3). In this case, the robot normal functioning will not be resumed until the current distance to the robot exceeds the safety threshold. Both distances are displayed in real-time.

Figure 3. Moral Engine test at Sadako’s WEEE reproduced recycling plant. The moral alert message reports that the safety threshold has been crossed.

Future developments on the Interaction Manager visual interface will include the automatic personalization of language settings and feedback provision from other Interaction Manager functionalities, such as the recognition of gesture-based HRI. Altogether, this Interaction Manager application and its tablet interface will serve as an additional communication channel ensuring reliable and effective Human-Robot Interaction, considering real-time environmental demands and being adaptive to each worker’s preferences.

New tools for human robot collaboration

In the context of HR-recycler project the collaboration between human and robot is a crucial aspect allowing a safe and dynamic sharing of the operational area. Due to peculiar environment conditions of the recycling factory that include presence of dust, dirty surfaces, and a not completely structured setup, the use of a robust and flexible device for the safety detection and monitoring in the classification and disassembling areas is required.

After an extended investigation of the several options present on the market, a specific solution based on radar technology by the company Inxpect has been selected. This product presents interesting features that fit quite well with the requirements of recycling factory: resistance to environmental disturbances, high sensitivity, 3D safety monitoring, several configurable fields of detection.

The radar (Radio Detecting and Ranging) is a well-known detection system that uses radio waves for detection and localization. It can measure the distance, angle, or velocity of objects and it consists of a transmitter, a receiver and a processor to carry out the low-level computations. Radio waves (pulsed or continuous) emitted from the transmitter reflect off the object and return to the receiver, giving information about the object location and speed. Electromagnetic waves are extremely resistant to disturbances like dust, debris, smoke or darkness, and they travel close to the speed of light (300,000 km/s).

Figure 1: safety radar sensor and its own controller

For the HR-Recycler projects after some preliminary feasibility studies, COMAU together with the sensors’ supplier and the involved partners opted for a solution that includes six different radar sensors placed in specific spots of the cell. These are arranged to cover all the possible interference area between human and machine, in particular monitoring the dangerous area where an industrial and not-collaborative robot (COMAU NJ 220) is used in the classification process for the pick and place of WEEE components.

The safety signals from the sensor are collected in a specific controller that send the output to a safe PLC. The robot controller is in turn connected to the safe PLC trough safe signals allowing an overall safe management of the cell. When the operator accesses a safeguarded area, the sensor triggers the movement and the robot is stopped immediately; moreover, until the presence is detected inside the specific area (even micro movements as breathing can be detected), the system is prevented from restarting, avoiding in this way any dangerous condition for the worker.

The two different monitoring states of the sensors, namely access and presence of the worker to/in the dangerous area, are showed in the following pictures:

Figure 2: access monitoring of HR-recycler robot station

If the worker enters inside the working area the safety radar sensors detect its movement (figure 2) and consequently stop the robot; after the access, the sensors change their configuration (figure 3) and prevent the restart of the system until the worker is inside the area.

Figure 3: presence monitoring of HR-recycler robot station

The complete cell will be setup and tested in the following months in COMAU premises, where a replica of the real recycling cell has been arranged for testing purposes. After the assessment of the overall solution the entire cell will be moved and deployed in the end user facilities for the final evaluation of the industrial process.

Preliminary pilot setup

During the period 14-22/12/2021, the preliminary pilot setup in the frame of the HR Recycler Project took place in the WEEE recycling plant of Ecoreset, with the participation of CERTH, SADAKO, ROBOTNIC, GAIKER and ECORESET.

Remarkable progress has been achieved regarding among others:

• The completion of the construction of the working benches
• The camera set up and recording
• The pallet truck initial set up and localization
• Installation of the interaction manager with real-time gesture detection

The completion of the set-up will continue early next year in the plant of Ecoreset. The first pilot run is expected within Q1 2022.