October 2021

The ambitious H2020 HR-Recycler project answers the need to create a new human-robot collaborative environment, where WEEE can be disassembled in a safe manner. To achieve an efficient collaboration in this field 12 European partners have joint forces and are working on various elements required create the hybrid human-robot recycling plant for electrical and electronic equipment of the future.

Within HR-Recycler CS GROUP – France (formerly DIGINEXT) focuses on the development of an effective virtual training system for the WEEE recycling industry. The dedicated tool (Procedure Editor) enables the creation of WEEE disassembly training experiences, based on predefined procedures (see image below), as described in a previous blog post.

Once the procedures created, they can either be implemented on the relevant object or exported in a format useable for VR training of robots and human workers. Taking this into consideration, CS GROUP has therefore implemented an S1000D export, described in more details below.

This type of export can be performed by anyone and results, within HR-Recycler, in two different files, as illustrated in the figure below. The content of each file is as follows:

• File 1: S1000D xml – containing the required persons, required parts and the procedure.
• File 2: 3D data xml – containing 3D file URLs, submodel URLs for each part and animation data

Below are included a few specific examples on the level of information provided for each element.

The first image shows the list of people, required to perform a certain procedure along with their person category code. Consequently, the extracted file provides clear information on the type of personnel and competences required to perform a certain task.

Furthermore, in the S1000D Export file it is possible to see the sequence of steps to be performed in order to complete a specific procedure. For example, it shows the:

• type of personnel/operator required to perform a procedure (highlighted in blue),
• the action to be performed, e.g. “unscrew” (highlighted in red)
• the reference of the tool and the associated video sequence (highlighted in green)

For training purposes, it is useful to be able to visualise the different steps of the procedures, that might be complex. The figure below shows the list of animations to be played for each step of the procedure.

This approach has made it possible to easily share information on procedures with relevant partners and enable the VR training of robots and human workers.

AR-Enabled Safety and Human-Robot Collaboration in HR-Recycler

CERTH has developed an AR-enabled system to monitor and communicate with robots in the WEEE recycling environment, through the use of optical and projective AR techniques. Such a system is essential in the harsh and noisy industrial WEEE recycling environment, where noise makes vocal communication difficult, and the scale and clutter of the environment pose challenges to achieving overview of the overall process for workers and supervising personnel.

The AR system introduced helps overcome the above challenges and introduce additional safety aspects to the overall process. At the most fundamental level, projective AR is used to indicate to workers the workspaces and intentions of the robots. For instance, the Autonomous Ground Vehicles (AGV) in HR-Recycler include projectors that highlight on the ground the immediate trajectory of the robot, and luminous strips that indicate the intention of the robot to move towards the left or right direction. Similarly, the workstations are equipped with projectors that indicate the workspace of the collaborative robots, as well as current targets for the robots to drop off components extracted from the devices. Projective AR is a ubiquitous technique that does not require the user to wear any equipment and provides an additional degree of safety in the shop floor.

In addition to projective AR, HR-Recycler has developed an optical AR system, where advanced robot supervision and communication is made possible through the use of an AR Head-Mounted Display (HMD). Through the HMD, the user is able to observe the current state and the plan of the robot or send goals for pick-and-place. The navigation plan and planned arm trajectory are visualized in an intuitive manner, overlaid on top of the actual robot, allowing the user to adjust their actions in accordance to the current robot plan. In addition, through registration of the HMD with the robot reference frame, it is possible to detect whether the user has entered the workspace of the robot, and in this case the robot is paused and a warning is presented to the user.

Overall the developed AR system increases safety within the shop floor and improves the overview of the workers and supervisors, thereby improving overall process efficiency.

The challenge of measuring emotions

Action-reaction, but how to measure it if it is an emotion?

The central nervous system plays a fundamental role in detecting and understanding emotions, cognitive processes and a series of other psychosocial constructs such as trust. The responses of the nervous system are relatively specific and show different patterns of activation depending on the situations and the emotional state. Any psychological process entails an emotional experience of greater or lesser intensity and of different quality, which is why the emotional reaction is something omnipresent in every psychological process. Likewise, all emotion involves a multidimensional experience with at least three response systems: cognitive (subjective), behavioural (expressive) and physiological (adaptive). Analysing, for example, the different dimensions of anger, it is possible to distinguish between the cognitive process (feeling angry), the behavioural (frowning) and the physiological (heart rate variation, among others).

Based on this premise, there are a whole series of psychophysiological signals that have been used to identify the various emotional processes:

• Electrical activity of the brain: By means of an electroencephalogram (EEG) it is possible to capture the cortical activity of the brain using electrodes in contact with the scalp. These types of signals offer a lot of information, although their analysis can be very complex since each component (electrode) contains information of a temporal nature (its amplitude), modal (frequency of the wave) and topographic (location in the brain). Although traditionally the acquisition of this signal required precise and specialized instruments, today the necessary equipment is much more accessible.
• Cerebral blood flow: Functional magnetic resonance imaging (fMRI) is a non-invasive technique that allows the measurement of brain activity. This technique is based on detecting areas of the brain with a higher concentration of blood flow, under the premise that these specific areas have greater activity compared to others with a lower flow. Although it offers great advantages at the research level, it requires bulky equipment and very high cost.
• Variation in heart rate: It is one of the main physiological signals linked to the feeling of security or danger. At the level of analysis, the period between beats is usually considered, distinguishing between low frequency pulsations if this period is below a threshold (classically 0.15 Hz) or high frequency if said threshold is exceeded. The main techniques to obtain this metric are the electrocardiogram (ECG), which records the activity of the heart, and photoplethysmography (PPG), which uses a controlled light beam to calculate blood flow based on the amount of reflected light.
• Skin galvanic response (GSR): Also called electrodermal activity, electrodermal activity or EDA. It reflects the electrical conductance of the skin by measuring the potential difference generated between two electrodes, generally located on the phalanges of two continuous fingers of the non-dominant hand. The excitation of the skin, normally produced in stressful situations, causes a dilation of the pores which, in turn, causes a decrease in the electrical resistance of the skin. It is necessary to distinguish its tonic or basal component and its phasic component. The former undergoes slow variations and often reflects unwanted experimental changes (for example, changes in the temperature conditions of the experiment), while the latter reflects rapid changes that are linked to the study stimulus.
• Muscle activity: Muscle activity can be measured using a pair of electrodes aligned with their kinematic axis (direction in which they expand and contract). The application can be used in a variety of situations. For example, electrodes can be positioned on both sides of the eyes (either vertically or horizontally) to detect eye movements. This specific test, called electrooculography (EOG) offers very useful information when cleaning the encephalographic signal of possible unwanted artifacts. Other generic applications include the detection of involuntary movements using electromyograms (EMG) in response to certain stimuli.
• Ocular behaviour: Ocular behaviour offers very valuable information on certain aspects, both physiological (for example, dilation and contraction of the pupil, blinking frequency, etc.) and behavioural (for example, drift and gaze fixation). Depending on whether the experimental process is carried out on a fixed platform (such as a computer) or freely available (participant in movement), this information can be extracted using fixed eye tracking instruments (attached to the visualization system) or mobiles (wearable systems in the form of glasses).

In short, analysing and measuring our nervous system to study or even understand our reactions, our emotions, to certain stimuli or situations is much more than a technological challenge. Designing and correctly using elements or devices that analyse our body and its signals is a challenge for science, for technology and for its possible multisectoral applications where the basis is to know the individual and their possible behaviours.