September 2020

Examples of the workshop’s presentations. The full program of the workshop can be found HERE.

 ORGANIZERS

– Apostolos Axenopoulos: Centre for Research and Technology Hellas – Information Technologies Institute
– Dimitrios Giakoumis: Centre for Research and Technology Hellas – Information Technologies Institute
– Eva Salgado:  Etxebarria, Tecnalia
– Vicky Vouloutsi, Institute for Bioengineering of Catalonia (IBEC), SPECS -Lab,
– Anna Mura: Institute for Bioengineering of Catalonia (IBEC), SPECS -Lab,

Autonomous pallet transportation in factory floor environments

Multi-pallet detection in factory floor environments

Within the HR-Recycler a novel pallet-truck AGV for the autonomous transportation of the pallets between classification and the workstations will be developed to enable automation in intra-factory logistics transportations of WEEE devices within a collaborative factory floor environment. The pallet-truck to be developed should be endowed with multi-pallet detection and pallet pick up navigation and control capabilities.

Technical Challenges

Pallet-truck AGVs’ operation in human populated factory environments is a challenging task, especially when there is a demand to operate without following fixed paths defined by guide wires, magnetic tape, magnets, or transponders embedded in the floor. There are several methods that tackle the task of autonomous pallet transportation and they are usually relied on computer vision approaches varying for indoor/outdoor environments. Yet, most of them are devoted to operate in predetermined paths and their global navigation is controlled by a central system typically linked to the Enterprise Resources Planning (ERP) of the factory.

Robotic Solution: Multi-pallet detection and docking strategy

A dedicated method has been developed by CERTH-ITI for the pallet-truck AGV developed by Robotnik partner within the HR-Recycler project. The solution comprises a vision-based method that enables safe and autonomous operation of pallet moving vehicles that accommodate pallet detection, pose estimation, docking control and pallet pick up in such industrial environments. A dedicated perception topology has been applied relying on stereo vision and laser-based measurements installed on-board a nominal pallet truck model. Pallet detection and pose estimation is performed with model-based geometrical pattern matching on point cloud data, exploiting CAD models of universal types of pallets, while robot alignment to candidate pallet is performed with a dedicated visual servoing controller. To allow safe and unconstrained operation, a human-aware navigation method has been developed to cope with human presence both during global path planing and during pallet-docking navigation phase. The developed method has been assessed in Gazebo simulation environment with a pallet truck model and proved to have real-time performance achieving increased accuracy in navigation, pallet detection and pick-up (see Figure1)

The Visual Analytics Lab (VARLab) of CERTH-ITI

Figure 1 In first figure, the robot approaches the pallet form-up area in Gazebo simulation environment. In the second figure the multi-pallet detection algorithm is applied along with the global planner algorithm that navigates the robot towards the alignment state. A point-to-point visual servoing controller drives the pallet-truck towards the selected pallet, as illustrated in third figure. The last figure outlines the docking and final pick-up of the pallet with a joint state controller.

Lifelong mapping for dynamic factory floor environments

Mapping the factory floor environments

The HR-Recycler project aims to automate the transportation of WEEE devices and disassembled components within a collaborative factory floor environment. To achieve this, different type of AGVs dedicated to serve specific roles during the disassembly process will be developed. To enable AGVs navigation in such environments, simultaneously localization and mapping (SLAM) methods will be utilized. However, when it comes to highly dynamic environments such as factory floors with moving objects the built maps with classic SLAM are soon get obsolete and the robot navigation performance degrades.

Technical Challenges

Despite the leaps of progress that have been made in the field of mobile robotics in recent years, one major challenge that AGVs still face is that of long-term autonomous operation in dynamic environments. In the HR-Recycler scenario where the AGV operates in a factory floor (see Figure 1), it has to deal with changing conditions where other robots, workers, moving objects such as pallets and even commodities move around the factory environment. In this scenario with the typical SLAM mapping the static objects (such as walls) will constitute only a fraction of the existing information in the map during robot navigation. If we consider this map as the ground truth, and use it disregarding ongoing changes, it is very challenging to maintain stable robot localization even if a robust localization filter will be employed.

Figure 1 Gazebo simulation of the BIANATT S.A. (End-User) factory floor environment. Note that the majority   of the existing information in the illustrated infrastructure corresponds to dynamic objects.

Robotic Solution: Life-long mapping

CERTH-ITI developed an essential solution on solving this issue by employing the life-long mapping approach which will has the ability to distinguish static and dynamic areas and handle this information accordingly during robot navigation. The ability to identify areas that exhibit high or low dynamics can improve the navigation of mobile robots as well as improve the process of long-term mapping of the environment. We utilized temporal persistence modeling in order to predict the state of cells in the life-long map by gathering rare observations from the on-board the robot sensors. This allows the modeling of the objects persistence in the map and provides to the system with prior knowledge regarding the occupancy of the area where robot operates. The method allows robot navigation by avoiding congested areas deteriorating the re-plans leading thus the robot to its target location without unnecessary maneuvering. Simulation results on life-long mapping with temporal persistence modelling are outlined in Figure 2, which illustrates the static metric map (left) and the probability of the dynamic areas through temporal persistence modeling (right).

The Visual Analytics Lab (VARLab) of CERTH-ITI

Figure 2  The outcome of life-long mapping . The left image illustrates the metric map produced by the classical 2D SLAM, while the  right image corresponds to the dynamic obstacles persistence probability calculated from the temporal persistence modeling.

HR-Recycler aims at developing a hybrid human-robot collaborative environment for the disassembly of waste of electrical and electronic equipment. Within this environment, humans and robots will be working collaboratively by sharing different processing and manipulation tasks.

The HR-Recycler project will implement learning paradigms in virtual reality (VR) settings, for which a recycling plant will be modeled in 3D to be displayed in a VR environment. One objective is to capture accurately the interaction of the workers with the environment and especially the objects present in the factory. Another objective is to allow workers to practice disassembly procedures and to properly interact with the environment. It is necessary to deliver a VR environment as realistic as possible and to model the procedures so that they can lead to more efficient training of the workers.

Within the project DIGINEXT is developing an effective virtual training system, targeting the WEEE recycling industry. The authoring tool (Procedure Editor) aims at easing the creation of WEEE disassembly virtual training experiences by minimizing production efforts.

Unlike a generic 3D tool, DIGINEXT’s solution does not require any specific programming or 3D modeling skills and allows us to adopt a smoother and faster workflow as compared to classical solutions based on a 3D engine. This means that a field expert can create procedures for virtual training even with very limited programming skills.

The tool is used by following these simple steps:

The first step consists of 3D modeling and animation: setting the scene (building, furniture) and selecting the WEEE that will be recycled.

The 3D model is next split and each part that requires an operation (e.g. screws) has an action attributed (e.g. unscrew, open, etc.). The figure below shows the list of parts for a microwave oven, and for a specific screw it can be seen that it is a “screwable” part that has 2 possible actions “screw” and “unscrew” (circled in red).

The procedure is then constructed very simply by assembling boxes within the graphical environment, by attributing these actions in a linear way that follows the processing as done in the actual factories.

Once the procedure is finished it can be either played using the mouse (see picture below) or exported following the S1000D norm and be used e.g. for I.A. training by other members of the project’s consortium.

Michael Darques, DXT

With the advancement of technology, the adopted processes need to accompany this advance. The main objective of the changes seeks to meet continuous improvement, in order to improve working conditions and the satisfaction of employees who work at the factory, reducing the risks to human health and safety. The processes of recycling waste electrical and electronic equipment are composed of numerous dangerous, monotone, and time-consuming tasks that must be replaced. It is here that automatic processes based on robotics appear as a solution. Through the robotics solutions proposed in the HR-Recycler project, workers and robots will collaborate in a synchronized manner. With this, the risk to the health and safety of workers due to the handling of potentially hazardous waste will be reduced and workers will be able to focus on tasks raising the level of quality, value, and qualification. And as a result, the companies will be able to increase their recycling rates.

It was expected to have collected images between partners Sadako and Interecycling. But unfortunately, with some delays and the difficult situation of COVID-19 that is affecting the whole world, this stage is still to be accomplished. We are hopeful that this whole scenario will soon improve rapidly in order to complete this stage.

Regarding the recordings, we will collect data for the Classification step in the same way as we have done in the previous recording campaigns, in the other end-users partners, Indumetal and Bianatt. For the Disassembly step, we will record hand-held camera data, later used for the co-bot operation, as well as whole-body images used for action recognition software.

The recordings will not be fundamentally different from those made previously, in the end-users mentioned above. In order to improve and to make better the results already obtained, we can try some calibration of the camera for the disassembly step and use the images later at Sadako to assess operational performance rather than only to build neural networks. This camera calibration would allow us to better locate the operator in space and use all the developed software functionalities.

So with this stage and recordings, we hope to gather data to improve our existing neural networks, both for the Classification and Disassembly steps, and to use some of this data to measure our operational performance. To get to know a little better the work developed in the area of ​​image collection you can consult the blog post of SDK from last 26/06/2020 (https://www.hr-recycler.eu/blog/).

Ana Catarina, INT
Albert Tissot, SDK

Monitoring and reviewing the impact assessment [in HR Recycler]

 In HR Recycler the consortium aims to develop a collaborative human-robot  to aid workers in the performance of their heavy tasks in recycling plants of waste electrical and electronic (WEEE) materials. The objective is that workers and robots “collaborate” in a joint and synchronised manner. What are the impacts of such a development? How it might affect workers rights and society as a whole? One main tool to ensure legal and ethical compliance is the performance of an impact assessment.

An impact assessment evaluates the origin, nature and severity of impacts that the (processing) operations, real or hypothetical, of a specific project entail. The ‘architecture’ for impact assessment typically consists of three main elements, the ‘framework’, the ‘method’ and the ‘template’. A framework constitutes an ‘essential supporting structure’ or organizational arrangement for something, which, in this context, concerns the policy for impact assessment, and defines and describes the conditions and principles thereof. In turn, a method, which is a ‘particular procedure for accomplishing or approaching something’, concerns the practice of impact assessment and defines the consecutive and/or iterative steps to be undertaken to perform such a process. A method corresponds to a framework and can be seen as a practical reflection thereof. These two elements have been identified in D2.2, being public. Lastly, a template for the assessment process can be seen as a practical implementation of a method (i.e. a procedure therefor, comprising consecutive and/or iterative steps) for impact assessment, itself reflecting a framework therefor (i.e. conditions and principles).

Building on the TARES impact assessment (Truthfulness, Authenticity, Respect, Equity and Social Responsibility), elaborated previously in another blogpost, VUB continues to identify legal, ethical and societal impacts that the project technology might entail, and to provide recommendations. This exercise necessitates broadening the picture of the project, so as to include relevant societal concerns, instead of solely focusing on legal matters (e.g. data protection).

To report the first version of the TARES impact assessment, an initial template had been sent out, where all partners had to answer specific questions from a technical and end-user point of view. The process of impact assessment shall be receptive and collaborative, i.e. technology developers work alongside the assessor’s team. During the first version of the TARES impact assessment, several issues were identified, such as that workers are considered to be ‘vulnerable subjects’, and consent to participate in the research project is not deemed to be freely given by data protection authorities. For instance, as a recommendation, an independent observer role to collect consent forms is warranted. The results of the report are confidential, for the time being, due to legitimate secrecy, and are illustrated in D2.3. However, a summary of the process may be published in the future.

The next tasks comprise mostly monitoring, review and update – to be achieved periodically with three deliverables – reports. The continuity of the TARES impact assessment aims at anticipating the risks and at adopting a mitigation strategy with recommendations for the further development of the technology. As the system is integrated, tested and evaluated, VUB will repeat this impact assessment, in three phases every 6 months, reporting after each phase of the project pilots. This means that a similar questionnaire, duly adapted, will be shared with all partners, so as to monitor closely the progress of each phase and the compliance to the mitigation strategy and recommendations; the latter will be adjusted to take into account the responses received. Assessors are currently revising their IAs, because things do change and there is a need to keep up in order to appropriately address upcoming issues. By doing this, the impact assessment aspires to be a ‘living instrument’, that evolves with the project and is able to assess ongoing changes and impacts.

To that end, VUB and specifically d.pia.lab recently announced for public consultation the third policy brief, which proposes a template for a report from the process of data protection impact assessment (DPIA); this reflects the best practice for impact assessment and, at the same time, conforms to the requirements of the General Data Protection Regulation GDPR. By utilizing the template as well as its subsequent modifications after the public consultation, VUB aims to better structure and perform the monitoring and review phase (including updates) of the assessment process in HR-Recycler.

If you need wish to receive further information, do not hesitate to subscribe to the HR-Recycler newsletter here.

Nikos Ioannidis & Sergi Vazquez Maymir

August 2020