Blog

The 3D model is next split and each part that requires an operation (e.g. screws) has an action attributed (e.g. unscrew, open, etc.). The figure below shows the list of parts for a microwave oven, and for a specific screw it can be seen that it is a “screwable” part that has 2 possible actions “screw” and “unscrew” (circled in red).

The procedure is then constructed very simply by assembling boxes within the graphical environment, by attributing these actions in a linear way that follows the processing as done in the actual factories.

Once the procedure is finished it can be either played using the mouse (see picture below) or exported following the S1000D norm and be used e.g. for I.A. training by other members of the project’s consortium.

Michael Darques, DXT

With the advancement of technology, the adopted processes need to accompany this advance. The main objective of the changes seeks to meet continuous improvement, in order to improve working conditions and the satisfaction of employees who work at the factory, reducing the risks to human health and safety. The processes of recycling waste electrical and electronic equipment are composed of numerous dangerous, monotone, and time-consuming tasks that must be replaced. It is here that automatic processes based on robotics appear as a solution. Through the robotics solutions proposed in the HR-Recycler project, workers and robots will collaborate in a synchronized manner. With this, the risk to the health and safety of workers due to the handling of potentially hazardous waste will be reduced and workers will be able to focus on tasks raising the level of quality, value, and qualification. And as a result, the companies will be able to increase their recycling rates.

It was expected to have collected images between partners Sadako and Interecycling. But unfortunately, with some delays and the difficult situation of COVID-19 that is affecting the whole world, this stage is still to be accomplished. We are hopeful that this whole scenario will soon improve rapidly in order to complete this stage.

Regarding the recordings, we will collect data for the Classification step in the same way as we have done in the previous recording campaigns, in the other end-users partners, Indumetal and Bianatt. For the Disassembly step, we will record hand-held camera data, later used for the co-bot operation, as well as whole-body images used for action recognition software.

The recordings will not be fundamentally different from those made previously, in the end-users mentioned above. In order to improve and to make better the results already obtained, we can try some calibration of the camera for the disassembly step and use the images later at Sadako to assess operational performance rather than only to build neural networks. This camera calibration would allow us to better locate the operator in space and use all the developed software functionalities.

So with this stage and recordings, we hope to gather data to improve our existing neural networks, both for the Classification and Disassembly steps, and to use some of this data to measure our operational performance. To get to know a little better the work developed in the area of ​​image collection you can consult the blog post of SDK from last 26/06/2020 (https://www.hr-recycler.eu/blog/).

Ana Catarina, INT
Albert Tissot, SDK

Monitoring and reviewing the impact assessment [in HR Recycler]

 In HR Recycler the consortium aims to develop a collaborative human-robot  to aid workers in the performance of their heavy tasks in recycling plants of waste electrical and electronic (WEEE) materials. The objective is that workers and robots “collaborate” in a joint and synchronised manner. What are the impacts of such a development? How it might affect workers rights and society as a whole? One main tool to ensure legal and ethical compliance is the performance of an impact assessment.

An impact assessment evaluates the origin, nature and severity of impacts that the (processing) operations, real or hypothetical, of a specific project entail. The ‘architecture’ for impact assessment typically consists of three main elements, the ‘framework’, the ‘method’ and the ‘template’. A framework constitutes an ‘essential supporting structure’ or organizational arrangement for something, which, in this context, concerns the policy for impact assessment, and defines and describes the conditions and principles thereof. In turn, a method, which is a ‘particular procedure for accomplishing or approaching something’, concerns the practice of impact assessment and defines the consecutive and/or iterative steps to be undertaken to perform such a process. A method corresponds to a framework and can be seen as a practical reflection thereof. These two elements have been identified in D2.2, being public. Lastly, a template for the assessment process can be seen as a practical implementation of a method (i.e. a procedure therefor, comprising consecutive and/or iterative steps) for impact assessment, itself reflecting a framework therefor (i.e. conditions and principles).

Building on the TARES impact assessment (Truthfulness, Authenticity, Respect, Equity and Social Responsibility), elaborated previously in another blogpost, VUB continues to identify legal, ethical and societal impacts that the project technology might entail, and to provide recommendations. This exercise necessitates broadening the picture of the project, so as to include relevant societal concerns, instead of solely focusing on legal matters (e.g. data protection).

To report the first version of the TARES impact assessment, an initial template had been sent out, where all partners had to answer specific questions from a technical and end-user point of view. The process of impact assessment shall be receptive and collaborative, i.e. technology developers work alongside the assessor’s team. During the first version of the TARES impact assessment, several issues were identified, such as that workers are considered to be ‘vulnerable subjects’, and consent to participate in the research project is not deemed to be freely given by data protection authorities. For instance, as a recommendation, an independent observer role to collect consent forms is warranted. The results of the report are confidential, for the time being, due to legitimate secrecy, and are illustrated in D2.3. However, a summary of the process may be published in the future.

The next tasks comprise mostly monitoring, review and update – to be achieved periodically with three deliverables – reports. The continuity of the TARES impact assessment aims at anticipating the risks and at adopting a mitigation strategy with recommendations for the further development of the technology. As the system is integrated, tested and evaluated, VUB will repeat this impact assessment, in three phases every 6 months, reporting after each phase of the project pilots. This means that a similar questionnaire, duly adapted, will be shared with all partners, so as to monitor closely the progress of each phase and the compliance to the mitigation strategy and recommendations; the latter will be adjusted to take into account the responses received. Assessors are currently revising their IAs, because things do change and there is a need to keep up in order to appropriately address upcoming issues. By doing this, the impact assessment aspires to be a ‘living instrument’, that evolves with the project and is able to assess ongoing changes and impacts.

To that end, VUB and specifically d.pia.lab recently announced for public consultation the third policy brief, which proposes a template for a report from the process of data protection impact assessment (DPIA); this reflects the best practice for impact assessment and, at the same time, conforms to the requirements of the General Data Protection Regulation GDPR. By utilizing the template as well as its subsequent modifications after the public consultation, VUB aims to better structure and perform the monitoring and review phase (including updates) of the assessment process in HR-Recycler.

If you need wish to receive further information, do not hesitate to subscribe to the HR-Recycler newsletter here.

Nikos Ioannidis & Sergi Vazquez Maymir

August 2020

Exploring the Environment: Haptic regression

The HR-Recycler project aims at creating a hybrid collaboration environment for disassembly of Waste of Electrical and Electronic Equipment (WEEE) materials, where humans and robots collaborate with the goal of disassembling WEEE products. Even for a single device type (e.g., microwave), recycling plants are interested in many variations that come in different shapes, weights, and disassembly steps. The complexity is further exacerbated when the mechanical structures of these devices are damaged which is often the case for disposed WEEE materials. When disassembling devices without knowing their construction plans, fixtures are easily not detected by visual sensors – either because they are not visible, such as snap-in fixtures, or because they are not properly recognized, such as recessed screws. In these situations, a solution is required for understanding the object and identifying its components.  Hence, a major challenge in disassembling an unknown object is to overcome these various sources of uncertainty. The shape of an object can be approximated well from vision and depth sensors. However, the information may be incomplete as visually occluded structures and the material composition that affects mechanical characteristics remains out of reach for such sensors. Thus, we are working at providing a robot with the ability to exploit its force sensing capability to refine the knowledge about these objects by interacting with the object.

Technical Challenges

By observing how robot force applied to the object affects interaction dynamics between them, we can improve the knowledge about the object properties, particularly material properties and estimation of structural features. This approach known as tactile and haptic exploration can significantly improve the results of visual object identification methods for complex objects with hidden structures.

Robotic Solution: Haptic SLAM

Haptic SLAM is dedicated to modeling the environment using haptic sensory data without visual feedback, for which haptic regression is an important technology. The entire space of the environment is divided into grids on which the probabilistic environmental model is constructed. Specifically, the probabilistic density functions of the objects in the environment are updated based on collected haptic data using the Bayesian inference and haptic regression. To improve the efficiency of large state space, occupancy and inference grids are applied and adopted with the discretized representation of the workspace. The represented model is built from iteratively collected sensor data given an initial belief over the geometry of the object.  An imminent challenge lies in the refinement of the estimated geometric shape of the underlying object. The critical point is to infer the shape candidates represented via analytical parameterization of different shapes from the initial sensor data and add this prior knowledge to further measurements. As a result, the overall dataset allows not only to update the geometric decomposition but also the material decomposition of the object. An example of the modeling process of an object can be referred in Figure 1, and the modeling results of the object are illustrated in Figure 2.

The Industry 4.0 concept refers to the so-called fourth industrial revolution, which involves digital transformation of the industry with the integration and digitization of all the industrial processes that make up the value chain. The proper integration of the technologies that industry 4.0 offers, enable significant benefits such as greater productivity and better resource management, more efficient decision making based on real information, optimal and integrated production processes, increased flexibility to achieve massive and personalized production in real time, reduction of operational time and reduction of defects percentage in factories. Therefore, Industry 4.0 is characterized by adaptability, flexibility and efficiency.

 The current ‘linear economy’ where everyday products are made, used and disposed should no longer work for businesses, society or environment, as earth’s raw materials are not limitless. Nowadays, there is a need for finding a new path of economic development that includes dealing with waste generation. Hence, there is a need for a transition into a new circular business model that recycles and reuses such waste with the objective of transforming it into higher value-added products to meet the current demands of society.

The integration of I4.0 technologies with circular economy practices have the capacity to significantly reduce  natural resources consumption. New and innovative solutions could make possible to extract even more raw materials, especially critical raw materials, from waste components in order to return them to the cycle. Thus, becoming circular in industrial settings not only involves saving a considerable amount of time, raw materials and consequently, money. It also opens all kinds of opportunities to innovation. To progress in this path, the recycling plant of the future should integrate all kind of I4.0 solutions, having as a result a smart plant, capable of easily adapting to changes in the composition and quantity of the materials to be treated and ensuring at the same time workers safety, avoiding human contact with waste as much as possible. HR-Recycler will help Indumetal in this approach, since it will create a hybrid collaboration environment, where humans and robots will harmoniously share and undertake at the same time different processing and manipulation tasks.

This is not only an improvement for the company’s competitiveness but it is also a step forward towards improving the WEEE collection rate in the European Union. For instance, in 2017, the collection rate of WEEE was 46 % in the EU and its recycling rate was 38,8 %. These rates are no longer acceptable for transitioning EU industry into a circular economy model.

In Indumetal we believe that circular economy strategies are the key for successfully implementing Industry 4.0, at the same time that Industry 4.0 provides technology to activates those strategies.

SADAKO explains their role in HR-Recycler and illustrates how their developed technologies and models are applied to various stages of the project.

As an AI and computer vision company, the role of Sadako in the HR-Recycler project is to provide the crucial vision algorithms to the robotic systems, allowing the perception of the environment in which the machines evolve.

Because of the wide variety of vision-dependent systems being employed in this project (different types of robot arms and Automatically Guided Vehicles, central factory planner), many different challenges are encountered which need each their own specific solution. All of the designed solutions rely on the use of RGB-D cameras and computers capable of Neural Network inference, and will be a good illustration of the many capabilities and applications of computer vision.

Until now, Sadako has developed solutions for the Classification and Disassembly stages of the recycling process, as well as for the AGV navigation.

Classification: 3D perception and object recognition for bin picking
Disassembly: Video-inference for gesture recognition, needed for Human-Robot collaboration and task status management
AGV: Human location perception, needed for optimal AGV path planning and human safety

• Classification:

This stage is the first of the recycling process, where WEEE arrives mixed in baskets and has to be sorted by type of object (Microwave, Flat Panel Display, PC Tower, Emergency Lamp)
The goal in this stage of the process is to identify the type and 3D orientation of each object lying in the basket, and provide the robotic arm with a picking decision. This is achieved through the use of Deep Learning vision algorithms and point cloud processing

• Disassembly:

The following process step, Disassembly, is where the previously sorted objects are dismantled and their parts are sorted by material to be sent to their final recycling location. The dismantling is made by an operator, with the support of a collaborative robotic arm. In order to achieve collaboration with the robot, Sadako has developed software based on Neural Networks capable of detecting actions in a video, in contrast with standard computer vision that detects objects in an image. This software is able to interpret the gestures being made by the operator, allowing him or her for example to stop or start the robot at any time. The software also allows the monitoring of the dismantling task’s status.

• AGV:

AGVs are used to transport WEEE objects between each stage of the dismantling process, reducing high-strain and repetitive physical work for the operators. To ensure that the AGV trajectory does not interfere with the movement of humans around the factory, the central system has to be aware of their position. To achieve this, Sadako has developed a solution based on Openpose, a software that extracts the position of the joints of humans on an image, crossed with depth information provided by the camera, thus allowing to perceive the position and orientation of operators in three dimensions. This software can be deployed at the Disassembly workstation as well as onboard the AGVs.

In its compromise of developing technology to improve the world and reduce health risks endured by humans in the industry, Sadako is staying strongly active and working in the progress of the HR-Recycler project, despite the hard times imposed by the pandemic situation

In two sessions held at the BIANATT  Aspropyrgos installations on July 8-9 and September 23-25, the procedures of dismantling FPD screens, desktop towers, emergency lighting, and microwave ovens, as well as the procedure of classifying WEEE appliances, were recorded in a special high power desktop computer using high-performance cameras.  Both sessions were coordinated and recorded by CERTH researchers.

BIANATT provided all the equipment and installed a customized construction to allow image recording from many different angles. Below you may see photos of the installation that was used for this purpose.

The recordings will be utilized by CERTH to produce Deep Learning models capable of detecting the aforementioned appliances as well as their internal parts during the disassembly process.

Comau explains the approach used on tools selection for WEEE scenarios.

Comau is working on different aspect of the WEEE management related to two main scenarios of the recycling process: the first related to classification of the part to be processed and the second to the disassembling phase in which the part are dismantled and the different materials and components are collected and selected for the final sorting phase.

From the robotics point view this two scenarios present some similarity about the main challenges to be solved even if some different approach and peculiar solutions can be adopted. In general the wide range of parts that can be managed is be very different in terms of:
– weight (from less than 1 kg up to 20 kg) ,
– shape (sharp or smooth edges)
– dimensions (from few cm up to 1 meter o more)
– materials (plastics, metals, glasses, etc)
So one the main requirement is the versatility of the robotics system about the tools to be adopted.

For the classification case an appropriate sensor base platform will be able to detect the parts in the bin but anyway a flexible gripper for the end effector able to handle this wide variety of objects is needed. After a benchmarking analysis on the possible solutions that meet these requirements available in the market, the first tests carried out have been executed with a vacuum grippers, called Kenos and manufactured by Piab (https://www.piab.com/en-us/products/kenos-vacuum-gripping-systems/). It has been designed and optimized to be used in various application and represents a flexible tool for the handling manipulation of several products with different shapes, dimensions and compactness.

The first results were very positive: the system has been able to handle all the parts considered as test case in the project (LCD, PC Tower, Emergency lamps, Microwaves oven) with an easy orientation of the gripper and a smart adaptation to the different parts in random surface positions.

In disassembling phase similar problem needs to be faced but in this case the use of a collaborative platform and a small payload robot introduce some issues to take into account as the safe management of interaction with operators as well as the high variety of tool to be used for the dismounting of the parts: also in this case in preliminary test has been adopted the novel KENOS vacuum gripper by PIAB, certified as collaborative and the PLUTO automatic screwdriver by KOLVER (https://kolver.it/prodotti-elenco/4-Serie-PLUTO-MITO-NATO) integrated in a customized solution mounted directly on the robot flange.

The preliminary test conducted on the emergency lamp case showed very good result in terms of reliability and reconfigurability of the system related to the different application that can be executed. Next steps to be performed can take in consideration the use of an appropriate tool changer useful in order to enhance the potentiality of the system and cover also possible task as wire cutting or handling of small objects inside the parts.
The optimization of the current solutions, the adoption of other tools and related tool changer will be face during the rest of the project.

CERTH explains their work in AI-enabled Cell-level perception methods.

The objective of this task is to provide the envisaged robot with the ability to perceive the industrial environment so as to effectively collaborate with the humans and assist them with the Waste Electrical and Electronic (WEEE) device recycling process. Towards this direction, CERTH recorded multiple recycling procedures of four WEEE devices, namely PC Towers, Microwave Ovens, Flat Panel Displays and Emergency Lamps, focusing on the human-robot collaboration aspect during the WEEE device disassembly, thus creating a WEEE device component detection dataset by manually annotating a number of the recorded frames.

Given this dataset, CERTH deployed state of the art Computer Vision methods utilizing Deep Learning techniques in order to train a Convolutional Neural Network (CNN) capable of detecting the four WEEE devices as well as their respective components during the various stages of the disassembly process. Additionally, CERTH enhanced these methods by proposing an anchoring mechanism targeting specifically the detection of small objects.

GAIKER goes on with the definition of the real-world scenarios in which the novel WEEE recycling approach based on human-robot collaborations will be demonstrated and validated

GAIKER continues the progress in the task of outlining the scenarios of the pilot demonstrations, whose ultimate aims are to test and evaluate, in real operating environments, the performance of the solutions developed within the HR-Recycler Project to manage WEEE (Waste Electric and Electronic Equipment). This advance is supported by the regular interaction with the partners of the Consortium that, as evidenced after the 1st Project Review Meeting held in Brussels (Belgium) at the end of January 2020, are doing significant and consistent progresses in the development of hardware and software solutions.

In the definition of scenarios, GAIKER is considering how to integrate all the relevant technological advances generated during the implementation of the scheduled tasks, involving user-cases definition, factory-level modelling, cell-level perception, collaborative schemes definition and robotic equipment design and programming. GAIKER is also striving to keep in close contact with the other project partners to understand, update and gather any relevant information that can help in the definition of more precise, realistic and functional pilot scenarios.

Until now, the greatest contributions have been:
– The definition of the use-cases and current recycling scenarios by the industrial WEEE recyclers involved in the HR-Recycler.
– The initial modelling of the factory floor and human workers, including work cell conceptual configuration, virtual factory 3D simulations and process orchestration.
– The demonstrated capabilities of advanced artificial intelligence-based perception in terms of automatic object recognition and manipulation, and human motion prediction.
– The preliminary definition of the robotic equipment, as articulated arms and automated vehicles, and the software for controlling robotic motion and interactions.
– The establishment of foundations for human-robot safe collaboration and adaptative interaction.

GAIKER has been thoughtfully registering, analyzing and integrating all these advances, particularly focusing on those aspects most closely linked to the key performance indicators defined to evaluate the HR-Recycler system. Thus, GAIKER has identified the main elements of the scenarios, including work cells and plant floor layouts, robot and worker movements (sifting and manipulation), detailed procedure requirements, material flows, human-robot interactions, safety issues and process orchestration. Based on these data GAIKER has begun to build the conceptual model for the pilot scenarios. This is, naturally, a flexible and additive model and GAIKER will be aware of any further novelty and progress in the projected tasks.