Home

D-3-2-CORBYS-Physical-Physiological-sensing-devices

1. BLE IMU sensor unit mai Back BlueGiga if a h ttyacm BLE protocol i Charging USB USB USB ae est uni Charger 3 uae HUB m i Charging GPN switch unit M Chest unit y iE A Battery powered BT protocol peenu HSS SENSORS Other HW SW Figure 6 Chest unit in Human Sensory system The Chest Unit shown in green in Figure 6 is used for measuring ECG heart rate temperature at the chest of the patient In addition it can measure acceleration and rotation of the patient The Chest Unit and the belt are connected by a mechanical and electrical connector device The majority of electronics as well as radio and battery are to be hosted in the Chest Unit The belt contains contact electrodes for heart rate measurements and ECG The Chest Unit is manufactured in a rugged plastic material and is designed without any sharp edges It is protected against splash water and entry of foreign objects but it is not designed to be immersed in water see Figure 7 The Chest unit integrates both activity and physiological sensors in the same device A 3 axis accelerometer is used to detect activity and a 3 axis gyroscope used for measuring rotation Both the accelerometer and gyroscope raw data are provided as part of the HSS dataset to the cognitive modules Skin temperature is measured by an infrared IR sensor Heart rate measurement is based on detection of electrical signals
2. v P E 1z z1 cr 5 lal 0 4 s cm 5 0 Sem X oving trial 70 311 mE R poner y X Y Z 5 55 70 frm 1 23642 ery 4H2 RA si v v a P E X Y Z 55 70 0 imm 7 40 2 z s XYZ 5 40 70 mm 9 3341 v w IY 45 0 5 10cm A P Figure 48 Source localisation results for raw EEG left and EEG data filtered right recorded during target 25 a target 25 b target 45 c and target 45 d for subject 3 EEG filtered shows activation in Brodmann area 4 in the upper alpha mu band 10 5 12 Hz 60 D3 2 Physical Physiological sensing devices Rev 1 0 Brodmann Area BA Best Match Raw EEG EEG filtered Target 25 a BA 37 19 39 21 BA 7 5 4 3 Target 25 b BA 37 19 39 21 BA 7 5 4 3 Target 45 c BA 37 19 39 21 BA 7 5 4 3 Target 45 d BA 37 19 39 21 BA 4 3 5 6 Table 5 The 4 Brodmann area localized trough sLORETA that show prominent activity in the raw EEG and in the EEG filtered Experiment II Fig 49 displays the EEG signal recorded during relaxation and walking for subject 4 Notice that the locomotion creates
3. User Interface User interface designed to meet the needs of the various users in exchanging information between the robot and human user Brain Computer Interface BCI The sensor system measuring the brain waves using EEG and detecting patterns identifying movement actions Human HSS The sensors measuring aspects of the human physiology and movement Sensory patterns System HSS Chest Unit CU Sensor unit located at chest of patient IMU Sensor Unit Sensor Unit located at back of patient Il D3 2 Physical Physiological sensing devices Rev 1 0 Chest Belt Belt around patient chest with Chest Unit at front and IMU Sensor Unit at back HSS Controller Computer and software system that receives data from HSS sensors and forward these to the General Purpose Network Low Level Control Localised control of actuators usually torque current or position control Sensory data is passed to the real time control actuation commands are calculated and sent to the actuators Smart Actuators Highly integrated mechatronic units incorporating a motor and the complete motion control electronics in one single unit Generic CORBYS Robot Control Components Cognitive System Incorporates situation awareness and intention detection to enable optimal man machine interaction towards achievement of set goals in the specific usage context Executive Layer The executive layer is responsibl
4. CORBYS ROS FRAMEWORK BLE BlueGiga if ethO _ ttyftdi ttyacm Eth USB USB_ USB USB BLE protocol Charger USB HUB GPN switch unit Bluetooth COM port if BT protocol Legend iss Hw_ HSS SW HSS SENSORS Other HW SW Figure 2 Human Sensory System overview 7 USB powered IMU sensor unit Back Battery powered IMU sensor unit i _ Back EG Charging Chest unit m Charging Chest unit _ Battery powered Figure 2 gives an overview of all HSS modules that will be presented in this chapter 2 1 HSS architecture The Human Sensory System HSS consists of a set of HSS Sensor Modules that interface with the HSS Controller which in turn interfaces with the CORBYS network as illustrated in Figure 3 The HSS Sensor Modules are one or more sensors connected to a sensor controller handling communication The HSS Controller handles all communication with the CORBYS Demonstrators via the CORBYS General Purpose Network The module based architecture facilitates the use of HSS modules in multiple Cognitive Robotics applications D3 2 Physical Physiological sensing devices Rev 1 0 Human Sensory System HSS HSS Sensor module 1 HSS Sensor module 2 HSS Sensor module n Figure 3 Human Sensory system
5. Significant ERD ERS was computed by a bootstrap algorithm with a significance level of 0 05 Graimann et Pfurtscheller 2006 Figure 55 1 0 5 0 0 5 1 1 50 37 5 Frequency Hz 12 6 Frequency Hz w N P nn n N in Time s Figure 55 Example of bootstrap algorithm in a ERD ERS time frequency map D Feature extraction and classification A classifier was built to detect each condition i e if the subject is paying attention or not to the leg movement task The features extracted were the spectral power of channels with significant desynchronization located in the motor cortex area in alpha and beta bands individually selected by visual inspection of the ERD ERS from the most discriminant electrode Welch method was used to compute the spectral power with a sliding window of 128 samples and a 120 samples overlapping Features were concatenated z score normalized and used to train a linear discriminant analysis LDA classifier A 10 fold cross validation procedure was performed to assess the generalization properties of the classifier 3 3 3 Results Figure 56 displays the ERD ERS maps for the 5 subjects that participated in the experiment Subjects 3 4 and 5 showed a higher desynchronization during attention to the motion than in the other condition This 68 D3 2 Physical Physiological sensing devices Rev 1 0 desynchronization was more prominent in both bands alpha and beta for subjects
6. 3 and 5 and in the beta band for subject 4 For Subject 2 the results are the contrary i e higher activity in the distraction task which could be due to the way that this subject performed the mental task For subject 1 is difficult to visually identify the differences as it seems that the desynchronization in alpha is lower but the beta is higher in the attention condition than in the other so more elaborated tools to quantify this desynchronization might need to be applied Gomez et al 2012 In summary in all the cases individually is possible to visually detect differences between the two mental states Frequency Hz Frequency Hz Frequency Hz Frequency Hz Frequency Hz Figure 56 ERD ERS maps for the five subjects from top row Subject 1 to bottom row Subject 5 for both conditions First Passive Motion Attention 50 ines Channel C2 w en Passive Motion Distraction Channel C2 Time s column Passive Motion Attention Second Column Passive Motion Distraction Black dashed line individualized alpha band Red dashed line individualized beta band Time s A Linear Discriminant Analysis LDA classifier was used to differentiate between both conditions Three 69 D3 2 Physical Physiological sensing devices Rev 1 0 different sets of features were used to train our classifier In the first one only power in alpha band was used in the second one only be
7. 7 13 Hz and 15 30 Hz respectively Rehabilitation programs for patients with these injuries are usually based on the execution of repetitive movements to regain muscle control or to delay the loss of mobility due to the disease One limitation in these rehabilitation therapies is that excessive repetitive movements could lead to a lack of patient engagement compromising the adherence to the therapy In this direction it is well known that cognitive processes such as attention mediate in the rehabilitation and play an important role in the success of the therapy Tee et al 2008 The possibility to monitor the patient attention could be a key issue in rehabilitation because it directly measures the cognitive process and indirectly the adherence to the therapy To date very few works have addressed the characterization of attention during the execution of motor tasks An fMRI study has shown that the degree of attention to the motor task modulates brain activity in sensory motor areas in such a way that focused attention produces higher activations of the motor brain rhythms Johansen Berg et Matthews 2002 This result has been confirmed in the EEG domain for passive upper limb mobilization Antelis et al 2012 In addition the last paper shows that it is possible to build an offline classifier to distinguish between two conditions passive movement with without attention to the motor task The present work builds in this direction by extending the pre
8. Feedback stimulus that informs the subject about the correctness of their response to a specific task The feedback stimulus presented after the accomplishment of a task informs the subject about the correctness of his response and therefore provide the critical information that would enable the error detection Feedback stimulus provided by other subsystems can be auditory visual somatosensory etc The error marker input is a time marker that informs the BCI subsystem when the feedback is presented to the subject Signal processing Decoding Refer to BCI8 BCI8 Feedback error related Information about the presence of the potential decoding flag feedback error related potential in the brain signal The feedback error related potential decoding flag output can take the following values present and absent depending on the presence of the feedback error related potential in the brain signal Signal processing Decoding part of the requirement will be addressed in Task 3 5 Deliverable D3 3 Integration in the BCI architecture will be addressed in Task 3 4 Deliverable D3 4 BCI9 Decoding accuracy It provides a numerical value e g a percentage related to the ability of the BCI subsystem in detecting the feedback error related potential Refer to BCI8 BCI10 Attention states decoding Information about the subject s attention level flag during a specific task The attention states decoding flag output prov
9. HSS Controller Engineer GUI s The CORBYS GUI developed as part of the CORBYS ROS framework provides a highly customizable interface which covers the visualization of all the sensor data produced by the HSS Controller No additional GUI is expected to be developed to display the HSS Controller sensor data However a specific GUI module will be developed in order to display the set of available sensor units and initialize training sessions by choosing the sensors to be used by the patient This HSS Controller specific GUI will also display information about the state and status of the HSS Controller based on the heart beat messages sent by the HSS Controller According to the CORBYS ROS framework the interactions between the GUI and the HSS Controller will be implemented via the central ROS parameter server The exact parameters to be used will be defined and documented in D3 4 2 5 10 HSS Controller Initialization The HSS Controller has two main modes A standby mode in which sensors are discovered and the state of their battery is monitored In this mode no data is produced on the CORBYS ROS topic the list of available sensors their serial number and battery status are just populated in the ROS parameter server The second mode is a training session mode in which the HSS Controller is connected to the sensors and continuously transfers data on the CORBY ROS topics On startup the HSS Controller automatically goes to standby mode The following s
10. M 2012 Wearable Wireless Multi parameter Sensor Module for Physiological Monitoring pHealth 2012 Proceedings of the 9th International Conference on Wearable Micro and Nano technologies for Personalized Health pages 210 215 2012 94 D3 2 Physical Physiological sensing devices Rev 1 0 Matteucci M Carabalona R Casella M Di Fabrizio E Gramatica F Di Rienzo M Snidero E Gavioli L and Sancrotti M 2007 Micropatterned dry electrodes for braincomputer interface Journal Microelectronic Engineering 84 2 Miiller Putz G R Scherer R Brunner C Leeb R and Pfurtscheller G Better than random A closer look on BCI results International Journal of Bioelectromagnetism vol 10 pp 52 55 2008 Pascual Marqui R Standardized low resolution brain electromagnetic tomography SLORETA technical details Methods Find Exp Clin Pharmacol 2002 24D 5 12 Pfurtscheller G Lopes da Silva F H Event related EEG MEG synchronization and desynchronization basic principles Clinical neurophysiology official journal of the International Federation of Clinical Neurophysiology 1 November 1999 volume 110 issue 11 Pages 1842 1857 Pfurtscheller G Muller Putz G R Pfurtscheller J and Rupp R EEG Based Asynchronous BCI Controls Functional Electrical Stimulation in a Tetraplegic Patient EURASIP J Adv Sig Proc 2005 19 3152 3155 2005 Pons J L Wearable Robots Biomechatronic Exoskeletons Wile
11. One Sensor ID per individual sensor value This option leads to a large number of sensors about 24 without the EEG sensors Since many sensors are sampled at the same rate this requires to duplicate many timestamps in the output messages e One Sensor ID per physical sensor unit This option allows reducing the number of sensors to the minimum but is impractical because the sensors of a single unit have different sampling rates which means that not all data can be populated for each update e One Sensor ID per HSS internal topic This is the solution which was chosen because it minimizes the required bandwidth on the CORBYS network no duplication of timestamps and no missing data Message SensorID HSS Sensor Data Reading Format float32 Size Rate ms HSS_Chest_Phi OxA1 161 heart_rate skin_temp activity posture battery 5 3000 HSS_Chest_IMU 0xA2 162 accel_x accel_y accel_z gyro_x gyro_y gyro_z 6 52 HSS_Chest_ECG 0xA3 163 ecg1 ecg2 ecg3 ecg8 8 32 HSS_Back_TH OxA4 164 skin_temp temp1 humidity1 temp2 humidity2 battery 6 3000 HSS_Back_IMU OxA5 165 ax ay az gx gy gz mx my mz qw qX qy qz 13 100 HSS_Env_TH OxA6 166 skin_temp temp1 humidity1 temp2 humidity2 battery 6 3000 HSS_Env_IMU OxA7 167 ax ay az gx gy gZ mx my mz qw qx qy qz 13 100 HSS_EMG OxAx P Figure 24 Data format on the CORBYS Sensor _ Data _HSS topic 33 D3 2 Physical Physiological sensing
12. Topic Sensors Data EEG The ROS Message sent will include EEG data and BCI decoding outputs from those decoders enables in the current demonstrator state Further information will be reported in D3 4 42 D3 2 Physical Physiological sensing devices Rev 1 0 3 1 1 Evaluation of EEG acquisition systems that could reduce the noise level 3 1 2 Introduction EEG systems take an important place in BCI applications as they provide an excellent way to describe brain signals with good time resolution and lower cost than other brain activity acquisition systems A key parameter of studies of EEG systems is the signal to noise ratio SNR as it provides a measurement of the signal quality In the EEG context this parameter is affected by a significant number of internal and external sources which influence the recorded signal independent of the system used for recording Sensor scalp contact i e to guarantee low contact impedance between the electrode and scalp and the capability to automatically remove or reduce noise sources are also important aspect of the acquisition system that determines the quality of the recording There are several studies in literature about EEG acquisition technologies such as those with focus on classical gel based electrodes and novel water based electrodes applied in specific brain computer interface Searle and Kirkup 2010 Matteucci et al 2007 Research teams have also developed working prototypes of dry EEG sensors
13. a movement artifact that is more visible at low frequencies Relax T ahuan fm pale mn fOr A A arrn ANNANN 9 02 D DE 08 1 Seconds Seconds Figure 49 2 seconds portion of EEG signal recorded during relaxation left and walking right for subject 4 Walking T Figure 50 presents the power spectral densities for subject 4 of the EEG signal recorded during walking and relaxation The walking condition has a greater PSD at low frequency range i e up to 4 Hz reflecting the effect of the low frequency movement as it is observed in Figure 49 This artifact is also present in high frequency 30 40 Hz and more prominent over central and parietal areas Notice also that the walking creates a desynchronization in the alpha mu rhythms in the central areas and synchronization around beta bands 61 D3 2 Physical Physiological sensing devices Rev 1 0 _ Walking Relax NG FCI N FCz sf AN i i FC2 a i CPz IPIE CP4 A ee E Ta Figure 50 Logarithmic power spectral density of the EEG signal recorded during relaxation black and walking blue for subject 4 Walking Relax i FCI Ht SQ ss ine cl i s i S4 SS lt iN Gey CP3 ir ir p ee a E t SSS 5 Hz Figure 51 Average logarithmic power spectral density of the EEG sign
14. accel_z HSS_Chest_IMU Int16 m s T gyro_x HSS_Chest_IMU Int16 deg s gyro_y HSS_Chest_IMU Int16 2 deg s 2 gyro_z HSS_Chest_IMU Int16 deg s ie ecg 1 8 HSS_Chest_ECG Int16 mV 2 2 skin_temp HSS_Back_TH Int16 0 01 C 20 40 2 0 1 temp1 HSS_Back_TH Int16 0 01 C 10 50 0 1 humidity1 HSS_Back_TH Int16 0 01 0 100 0 1 5 temp2 HSS_Back_TH Int16 T No Sensor humidity2 HSS_Back_TH Int16 2 No Sensor battery HSS_Back_TH Ulnt8 1 0 100 1 1 skin_temp HSS_Env_TH Int16 0 01 C 20 40 2 0 1 No Meaning temp1 HSS_Env_TH Int16 0 01 C 10 50 2 0 1 humidity1 HSS_Env_TH Int16 0 01 0 100 0 1 5 temp2 HSS_Env_TH Int16 7 si No Sensor humidity2 HSS_Env_TH Int16 z 2 No Sensor battery HSS_Env_TH Ulnt8 1 0 100 1 1 ax HSS_ Back Env _IMU Int16 m s 2 ay HSS_ Back Env IMU Int16 m s az HSS_ Back Env IMU Int16 m s gx HSS_ Back Env IMU Int16 deg s fe 2 gy HSS_ Back Env IMU Int16 deg s tg 2 2 gz HSS_ Back Env IMU Int16 deg s 2 mx HSS_ Back Env IMU Int16 nT m Kd my HSS_ Back Env IMU Int16 a nT m T 2 mz HSS_ Back Env _IMU Int16 nT m T qw HSS_ Back Env IMU Int16 2 2 2 qx HSS_ Back Env _IMU Int16 2 qy HSS_ Back Env IMU Int16 F 2 qz HSS_ Back Env IMU Int16 2 ti si F Figure 25 HSS Controller sensor data specification Figure 25 details the scaling applied to the raw data t
15. an API to configure the Chest Unit and receive the data from its different sensors e The ROS Publisher uses the Chest Unit driver to subscribe to the sensor data and forward the data on to the HSS Controller internal ROS topics e The UDP GUI Server is implemented for debugging purposes It forwards the data exchanged on the serial port to a UDP socket A test application can be connected to display the sensor data All these components are modelled using the ThingML languages which allow automatically generating C C implementations for Linux and ROS packages 7 http www thingml org 28 D3 2 Physical Physiological sensing devices Rev 1 0 amiy THES Front end GUI Dataog Test Engineer D amp HSS_Chest_Phi Interface HSS_Chest_IMU Cc HSS_Chest_ECG DE BE cD OL pa C Oo lt Serial Port 115200 bps 10 ftdi rules Roving Networks AT commands Transparent serial over Bluetooth Get a fixed device for the Roving Networks Bluetooth Adapter BUS usb SYSFS idProduct 6001 SYSFS idVendor 0403 SYSFS serial AE01AAEF SYMLINK bluesmirf Roving Networks Bluetooth Module Linux Ubuntu 10 04 Figure 20 Architecture of the Chest Unit Driver The test and engineer interface on the top right corner of Figure 20 is a Java application which implements its own driver and GUI for the Chest Unit sensors It can be connected locally or remotely to the HSS
16. an amplifier can decrease the dependency of the electrode contact impedance on the contrary low input impedance causes load of bio signal source and it results in damaging of the signal The input impedance in the Porti 16 is of two orders of magnitude higher than in the g USBamp The need of low contact impedance can be mitigated partly by the high input impedance of the amplifier and the shielding of the electrode wires or connectors as with the TMSi equipment as well as by using local impedance adapters as the active electrodes of g Tec electrodes CMRR is a measure of the ability of a test instrument to reject interference that is common to both of its measurement inputs which is equal in both amplifiers Mains interference i e 50 60 Hz can be prevented with active guarded shielded leads and electrodes also it is possible to reject this interference using a notch filter of main frequency i e 50 Hz by hardware as g USBamp or by software as well From a theoretical point of view the high input impedance of the amplifiers could be a technical parameter that makes the TMSi amplifier more appropriate for CORBYS as both amplifiers have very similar characteristics However this decision needs to be confirmed by the experimental evaluation as there are many other aspects that mediate in the EEG recordings 3 1 4 Evaluation scenarios This section addresses the influence of electrical motor noise in the EEG recordings of both EEG acquisition s
17. and demonstrated that the signal obtained can be largely comparable to wet electrodes Popescu 2007 Among those existing sources of noise that can affect the EEG signals the electromagnetic noise caused by external devices such as motors and the power supply are those that could have a significant impact on the use of BCI in CORBYS scenario This deliverable focuses on a technical evaluation of the quality of EEG recordings of two commercial systems to find adequate equipment for the CORBYS scenario This is because in CORBYS the EEG system will be forced to work close to a robot in movement which is a source of electromagnetic activity and noise and might affect the EEG readings This document reports the evaluation of two commercial EEG systems in scenarios similar to the CORBYS 1st demonstrator The first one is the Porti amplifier a multichannel device of TMSi Twente Medical System International BV Netherlands with passive electrodes of water solution The second one is g USBAmp a biosignal amplifier produced by g Tec Graz Austria with active electrodes of gel solution The first section describes the technical specifications of these two commercial EEG systems The second section presents an analysis of both EEG systems in two different experiments The objective of the experiments was to study the possible effect of electromagnetic noise in the EEG signals caused by electric motors in two different scenarios the first experiment was
18. and the Self Organizing Information Anticipatory Architecture SOIAA endows robotic systems with cognitive capabilities and are responsible for identifying the current state of the system and maintaining a cognitive image of the environment including representation of the humans interacting with the system and generating high level commands The physiological data will be used as input via SAWBB to the reasoning sub systems within the Situation Assessment architecture for observation and learning of patterns identification of deviation from established reference points and rectifications suggested to the human in the loop therapist in case of Demonstrator I EEG data is largely utilized by SOIAA for detection of intention and attention of motion Details of cognitive modules will be described in documents deliverables under WP4 and WPS This deliverable D3 2 gives detailed information about development of Human Sensory System sensors and controller software The main document contains an overall description of the system however some implementation details for the sensor systems and test specifications are given in the project internal appendices To meet challenges in the integration of BCI related algorithms for detection of cognitive processes in real time robotic applications i e CORBYS first demonstrator studies on the impact of different artifacts have been performed and documented in the current deliverable Ocular mechanical and
19. been increased from seconds up to 4 milliseconds ESUMS http www sintefannualreport com 201 1 en with your heart in his hands 12 D3 2 Physical Physiological sensing devices Rev 1 0 S ChestBelt Test Application e boll x SensoriD 25944 FW revision 3 0 3 Overrun o 0 BW 3380 tha Mode RawGyroMode Live Data Conn Restored Bluetooth update int Posture Activity IMU Timestamp 554 Timestamp lesa Timestamp id Temperature Heart Rate 114 2 Ba ChestBelt Heart Rate and ECG Graphs Figure 8 A screen dump of the Java application receiving sensor data from Chest unit 2 2 1 Chest Unit Test The Chest Unit has been thoroughly tested in lab by SINTEF and externally towards reference systems before use in the CORBYS project Hence only the modified configuration is tested in the CORBYS project This covers Gyroscope Raw accelerometer data Time stamping Higher Bluetooth data rates 2 2 1 1 Gyroscope The gyroscope has been tested with respect to static rotation using a turntable This has been used to verify scaling of the outputs and rotation axis The combined message with gyroscope and accelerometer data introduced in CORBYS has been verified towards the Java application at the PC Visual inspection of the graphs has been done as a check of data usefulness w
20. brain injuries such as brain stroke or spinal cord injury can cause problems in patient s movement as the Central Nervous System CNS or the efferent channels from the CNS are compromised Brain Computer Interfaces BCI is a technology that addresses this problem by creating a new channel to bypass the injury and recover the communication between the non affected CNS and the limb muscles The principle is to measure the brain activity to extract some meaningful information to move the limbs by for instance functional electrical stimulation FES Pfurtscheller et al 2005 or by robotic exoskeletons Pons 2008 The electrophysiology of the motor skills is usually characterized by the event related synchronization desynchronization ERS ERD which refers to the increase or decrease in synchrony of the neural population of a determined area of the brain cortex Pfurtscheller et Lopes da Silva 1999 This technique has been used to quantify changes in EEG signal by calculating the increase or decrease of spectral power during a mental process compared to the brain activity during a reference time or baseline There are associated techniques to visualize Graimann et al 2002 and compute Gomez et al 2012 neural changes in motor areas of the brain For instance the movement of a limb has been characterized by this method and thus it is known that neural population of the central brain cortex area motor cortex desynchronize in alpha and beta bands
21. devices Rev 1 0 Figure 24 presents how the data from the internal sensor topics is provided in CORBYS_Sensor_ Samples as well as the sampling rate of each sensor Variables marked with a are currently not populated or populated with a non specified value and hence should not be used at this point The amount of data generated by the HSS controller sensors will use some bandwidth on the CORBYS GPN As an early estimation of the required bandwidth given the size if the messages and their sampling rate the required bandwidths are 2 53kByes s to transmit the sensor data and 3 32kBytes s when taking headers and timestamps into account The overhead of the ROS framework might require some additional bandwidth however these numbers seem reasonable and should not pose any problem to the CORBYS GPN Subscribers to the Sensor_Data_HSS topic should use the IDs provided in Figure 24 and the corresponding macros in the CORBYS_Common source folder to decode the data according to the specified sensor reading format Name Internal Topic Internal Type Scaling Unit Mininum Maximum Resolution Accuracy Notes heart_rate HSS_Chest_Phi Int16 0 1 BPM 25 230 0 1 1 skin_temp HSS_Chest_Phi Int16 0 01 C 20 40 2 0 1 activity HSS_Chest_Phi Ulnt8 2 tg 2 Enumerated values posture HSS_Chest_Phi Ulnt8 ve 2 Enumerated values battery HSS_Chest_Phi Ulnt8 1 0 100 1 1 accel_x HSS_Chest_IMU Int16 Hd m s 2 accel_y HSS_Chest_IMU Int16 m s T
22. electromagnetical contaminations on the EEG signal have been addressed through the execution of several experiments Due to the result obtained the TMSi system has been chosen as EEG system for the CORBYS demonstrator I In addition the Independent Component Analysis ICA has been showed to be a feasible technique to use in artifact removal within a simulation of the CORBYS rehabilitation scenario An initial analysis of passive lower limb movements with attention or non attention to the motor task have been performed in order to improve rehabilitation programs where the movement repetitions could lead to a 2 D3 2 Physical Physiological sensing devices Rev 1 0 lack of patient engagement compromising the adherence to the therapy Results reported showed the feasibility to distinguish between the two conditions Dataset from human sensors including BCI data have been recorded analysed and provided to the cognitive partners as an early integration activity This have served the basis for which all data is handled establishing protocols for sampling handling data with different time scales and interfaces for testing various information theoretic tools The safety of the Human Sensory System and of the Brain Computer Interface is discussed and FMEA risk analysis is provided There are no safety concerns for the Human Sensory System for the clinical testing in WP9 The relevant requirements from D2 1 are analysed and compared to the curre
23. from the heart measured on the skin Extraction of heart rate from the complex analogue signals is performed in the digital domain by the chest unit processor system Power is provided by a rechargeable battery and 11 D3 2 Physical Physiological sensing devices Rev 1 0 components are selected for minimum power consumption Wireless communication is using the Bluetooth SPP protocol stack Figure 7 Belt with Chest unit sensor Microcontroller software is based on the real time kernel wC OS II and optimized for low power consumption by putting microcontroller and radio into sleep mode when idle Sensors are read at regular intervals and data is transmitted using Bluetooth avoiding requests from the client application To ease sensor data analysis a Java based monitoring application for recording and visualization of sensor data is developed and shown in Figure 8 The main development of the Chest Belt and Chest Unit has been done by SINTEF in a project called ESUMS However the embedded microcode has been enhanced to fit the usage in the CORBYS project providing more sensor data at a better timing accuracy to the cognitive system than required for use in the ESUMS project e Gyroscope functionality has been added e The sample rate has been increased from 300ms to 52ms better follow the movement of the patient e The wireless data rate has been doubled compared to previous versions e The resolution of the sample timestamps has
24. high level cognitive control modules 2 a semantically driven self awareness module and 3 a cognitive framework for anticipation of and synergy with human behaviour based on biologically inspired information theoretic principles The CORBYS control architecture will be validated within the two CORBYS demonstrators The first demonstrator is the novel mobile robot assisted gait rehabilitation system CORBYS Further information about the design challenges of CORBYS can be found on the CORBYS web page www corbys eu Additionally general information about the field of Cognitive Robotics can be found on the EU Framework Program 7 web pages on Cognitive Systems and Robotics http cordis europa eu fp7 ict cognition One of the main CORBYS objectives is development of advanced sensing module for assessing the physical and psychological state of human in robots environment The physiological sensing devices in the CORBYS project consist of the Brain Computer Interface BCI and the Human Sensory System HSS Non invasive Brain Computer Interface using EEG performs online detection of human cognitive information such as intention of leg motion feedback error related potential and attention states The Human Sensory System consists of a set of physiological sensors to measure patient effort and movement The sensors are grouped into four categories i e physiological sensors movement sensors environmental sensors and mechanical sensing tec
25. in D3 4 in month 26 22 D3 2 Physical Physiological sensing devices Rev 1 0 2 5 Human Sensory System controller EMG sensor unit EMG interface Cabinet IMU sensor unit Environment H troller P SS controller PC H USB powered LINUX ROS IMU sensor unit Back CORBYS Battery powered ROS FRAMEWORK IMU sensor unit BLE Back BlueGiga if Ai aie a ttyacm BLE protocol ging USB USB USB ERTER est uni Charger USB HUB Charging GPN switch unit Chest unit Bluetooth Battery powered COM port if ROY BT protocol Legend iiss Hw Ass sw Other HW SW Figure 16 HSS controller in Human Sensory System 2 5 1 HSS Controller Hardware The HSS controller software will be running on an industrial ruggedized Intel Atom based computer mounted in the Carrier Frame of Demonstrator I At this point the HSS Controller Software is developed and tested in a VirtualBox virtual machine with a hard drive of 16GB and 2GB of allocated RAM 23 D3 2 Physical Physiological sensing devices Rev 1 0 2 5 2 CORBYS GPN and Robot Operating System ROS The HSS Controller is one module of the overall CORBYS System as described in deliverable D2 2 The integration between the different CORBYS
26. incorrect data due to incorrect sensor location there will be a therapist procedure to validate sensor data before each training session 73 D3 2 Physical Physiological sensing devices Rev 1 0 POTENTIAL POTENTIAL POTENTIAL DETECTION Pos FUNCTION failure MODE CAUSES EFFECTS METHOD SEV OCC Recommended Action s SEV OCC DET RPN 1 Human connection loss to Network error no sensor data ROS heartbeat 7 5 1 FS must check HSS heart 0 Sensory GPN to cognitive beat and handle any System modules detected errors according to Controller FS specification failure time lag runtime error for delay on sensor Timestamped 7 5 3 Cognitive modules must 7 5 2 HSS controller data to cognitive sensor data check timestamps hardware modules may Timestamps are lead to incorrect synchronized by cognitive ROS framework recognition failure time lag Configuration missing or delay Timestamp on 7 5 3 Cognitive modules must 7 5 2 error on sensor data sensor data check timestamps Cognitive to cognitive cognitive modules should handle modules may modules detects incomplete datasets Data lead to incorrect missing data from HSS to be verified in cognitive Timestamps are pre session procedures recognition synchronized by ROS framework 2 Human measurement Sensors pick up incorrect sensor HSS controller 7 4 6 0 Sensory failure external noise data to cognitive detects out of System modules range sensor W
27. of the main contamination introduced in the EEG by locomotion and head movements These artifacts are likely to be present in the CORBYS 1 demonstrator The results indicate that artifactual components can conceal or mimic EEG alpha and beta rhythms over the entire scalp as previously described for EMG contamination Goncharova et al 2003 and are also relevant at low frequencies Independent Component Analysis technique was applied to remove the artifact contamination generated as it appears to be an effective method for removing artifacts from EEG data Jung et al 1998 The results suggest the feasibility of the use of this technique for artifact removal in CORBYS The key issue for a BCI system is to what extent the artifacts components interferes with its goal Notice that decoder specific analysis is extremely important in order to study the feasibility of removing gait related artifacts allowing correct interpretation of cognitive related process In addition to this work in the section 3 3 Detection of attention during assisted passive leg motion of this deliverable the ICA method was used to filter eye blinking and muscular artifacts Results obtained indicate that ICA does not interfere with the attention decoding process on the contrary it improves its performances 64 D3 2 Physical Physiological sensing devices Rev 1 0 3 3 Detection of attention during assisted passive leg motion 3 3 1 Introduction Neurological disorders or
28. on the patient Mounting of individual sensor components directly on the user s skin Based on user safety concerns the physiological measurements system might not be used on patient groups such as Patients with electronic implants e Patients with certain dermatologic conditions Patients with limitations in cognitive capabilities Others to be decided The physiological sensors will be mounted and removed by trained clinical rehabilitation professionals Certain physiological sensors for example electrode based can be placed at optimum measurement locations directly on the skin of the patient HSS should not be used by patient with electronic implants Usage scenario and sensor mounting will be described as part D7 4 See HSS29 HSS31 Mounting of individual sensor components directly on the user s skin Less optimal but more user friendly locations can be used See HSS29 HSS32 Placement of sensor components on the patient All sensor components should be clearly marked in order to reduce the risk of placing sensors in incorrect measurement positions e g 83 See HSS29 D3 2 Physical Physiological sensing devices Rev 1 0 mix left and right HSS33 Placement of sensor Preferably automated Therapist GUI shall be used for components on the patient detection mechanisms to sanity check of sensor data at avoid the risk of incor
29. patient chest Humidity sweat Used in the assessment of physical effort The humidity is measured on the back of the patient Movement sensors Inertial Measurement Units IMU The IMU consists of a 3 axis accelerometer a 3 axis gyroscope and optionally a magnetometer The IMU is measuring velocity orientation and gravitational forces In CORBYS it is used for measuring patient balance and movement One IMU is located at the chest and one at the back of the patient The IMU at the back also includes a magnetometer Environmental sensors Environment temperature Exterior temperature used in humidity sweat calculations Located externally on the HSS Controller Table 1 CORBYS HSS Sensors These sensors are grouped into the following sensor modules e Chest unit a unit connected to a belt around the patient chest measuring heart rate ECG skin temperature and IMU data See Figure 5 e IMU sensor unit connected to the same belt as chest unit but at the back of the patient The IMU sensor unit measures skin temperature humidity and activity via the IMU This IMU also includes a magnetometer See Figure 5 e Environment temperature an IMU sensor unit is attached to the mobile platform and is used for environment humidity measurements e EMG sensors 8 sensors all together Two sensors are connected to each thigh and similarly two sensors to each calf for each leg of
30. seconds Scale 1000uV 1 Scale 100uV N W a N a gt Figure 39 shows power spectrum averaged across channels of EEG recorded for both systems for Subject 2 A peak at 20 Hz and at its fundamental harmonic frequencies i e 40 Hz 60 Hz etc reflects the electric noise contamination 50 D3 2 Physical Physiological sensing devices Rev 1 0 gt a Motor ON Motor OFF gt o T wo a T W EEG system 1 Power spectrum uV7 Hz 20 30 Frequency Hz Figure 39 EEG spectrum recording during motor ON blue and motor OFF red conditios averaged across 10 channels registered with TMSi left and g Tec right system for subject 2 Subject 1 Subject 2 Subject 3 r e n cy 4 log t0 Noisy spectrum Free noisy spectrum Jog 10 Noisy spectrum Free noisy spectrum log10 Noisy spectrum Free noisy spectrum Figure 40 Ratio between power spectrum of noisy and free noise one for Porti blue and g USBamp red sytem of each subject Since both amplifiers have different gain across frequency it is not possible to compare their EEG or corresponding PSD To avoid this phenomenon the ratio between the power spectrums of both conditions is used this is presented in Figure 40 for all subjects Notice that the level in Noise condition in g USBamp is more th
31. the CORBYS project 71 D3 2 Physical Physiological sensing devices Rev 1 0 4 Safety Work with safety aspects of the CORBYS demonstrator I and of all its components is started and will be documented in documents that will accompany the deliverable on the final demonstrator D7 4 M36 covering safety standards and procedures In the following safety aspects of HSS and BCI module are given together with FMEA risk analysis 4 1 Chest unit The chest unit is a battery powered device using wireless communication This ensures patient safety with respect to electrical shock The CORBYS Chest Unit uses conductive rubber electrodes for ECG measurement These gives signal quality comparable to the medical electrodes also when the patient is very active A disadvantage is possible skin irritation when sweat collects under the rubber and this may cause some skin irritation during prolonged use However CORBYS training sessions will be of limited duration The sensors are integrated in the chest unit and automatically placed correctly when the patient puts on the device The sensor belt will attract sweat and dirt after use and should be rinsed in lukewarm water no soap once a week and then left to air dry However one must remember to detach both the chest unit and the IMU unit from the belt before washing as the units are not water resistant The units can be wiped with a soft damp cloth and towel dried 4 2 IMU unit The IMU unit is a batte
32. the muscles of the face neck and on the scalp and are caused by movement chewing swallowing muscle twitches anxiety tremor or general muscle tension Van de Velde et al 1998 The EMG activity also spans a broad frequency range distribution and even weak muscle contractions produce an EMG activity that can mimic or obscure the EEG activity over the entire scalp Goncharova et al 2003 In the CORBYS gait rehabilitation system scenario the subject will walk with assistance of a robotic device In this context the EEG data will be affected by typical artifacts such as EOG and EMG but also electromagnetic artifacts due to the mobile platform and mechanical artifacts associated with head movements and locomotion The study of the impact of the electromagnetic noise i e noise due to the DC motors is addressed in the section 3 1 Evaluation of EEG acquisition system that could reduce the noise of this deliverable Several research groups have studied the EEG artifacts during human locomotion For instance Gwin et al 2010 studied the brain activity during walking and running on a treadmill in a controlled scenario The results showed that during the walking condition the artifacts slightly contaminate the EEG signals in an event related analysis while during the running conditions the EEG signals are strongly affected by movement artifacts This deliverable addresses the mechanical artifacts and their removal process based on Independent C
33. 1 0 applications The skin temperature for the environment unit is not meaningful in the context of the demonstrator since the sensor is mounted on the chassis of the demonstrator e 100 ms for the IMU data This value might be adjusted in order to cope with the Bluetooth Smart bandwidth limitations The current implementation provides all IMU data raw accelerometer gyros and magnetometer data as well and fused attitude data from the accelerometers and gyros as a quaternion To HSS Front end GUI Datalog sata ESSA Fest Engineer HSS_Back_IMU aS En Th B terface HSS_Env_IMU A kd 2 oa IMU Sensor Generated from ThingML Bluegiga proprietary binary protocol for Bluetooth 4 0 Serial Port 115200 bps 46 bluegiga rules Get a fixed device for the bluegiga BLE dongle at dev bled112 ATTRS idVendor 2458 ATTRS idProduct 0001 MODE 0660 GROUP dialout SYMLINK bled112 Bluegiga BLED112 Figure 22 Architecture of the IMU Sensors Driver Linux Ubuntu 10 04 Figure 22 presents the structure of the IMU sensor driver The bottom part of the figure represents the operating system drivers which link to the BLED112 dongle The BLED112 dongle implements an USB CDC ACM driver which is recognized by the Linux kernel as a virtual serial port and automatically mounted as a dev ttyACMX device X varies depending on the number of devices plugged to the computer In order to get a fixed d
34. 2 Patient inspection should be inflamatic uncomfortable or response and added to post training responses or pain unable to wear inspection session procedure sensor Table 7 Safety analysis for HSS 75 D3 2 Physical Physiological sensing devices Rev 1 0 The safety design of the Brain Computer Interface BCI submodule has two different levels of analysis hardware and software 4 7 EEG unit The amplifier Porti System TMSi of the BCI complies with the following safety requirements e CE0044 meets all the requirements of the MDD 93 42 EEC MDD Classification IIa rule 10 e Applied standards IEC 60601 1 1988 A1 1991 A2 1995 Medical electrical equipment Part 1 General requirements for safety IEC 60601 1 2 2001 Medical electrical equipment Part 1 2 Electromagnetic compatibility e Safety class IEC 60601 1 Internally or externally powered type CF 4 8 BCI Software The Software of the BCI submodule provides the following output regarding the state of the BCI sensors 1 EEG sensor failure information about the actual state of the EEG sensor indicating if it is working properly or not e g it can happen that an electrode gets disconnected or broken during the rehabilitation session causing wrong recordings The EEG sensor failure will allow then to have a complete overview on the BCI submodule status 4 9 BCI Safety analysis The table below presents the risk analysis for t
35. 2 Physical Physiological sensing devices Rev 1 0 Implementation Documentation Development of the CORBYS related decoding algorithms Attention states Feedback error related potential Integration of the decoding algorithms in the BCI software architecture BCI module testing Bluetooth acquisition Decoders activation deactivation Measurement failure Training procedures Decoding Procedure ROS network integration Generation and integration of simulated EEG data node version for simulation Design of BCI output messages including EEG data and decoding outputs Design of BCI configuration parameters from Parameter Server Integration of BCI architecture in ROS node template Development of GUI ROS node Synchronization Evaluation of BCI processing delays Evaluation of submodule sampling frequency and cycle time Evaluation of synchronization between HSS and BCI data Integration and Testing The Brain Computer Interface submodule will be integrated and tested within the CORBYS mobile platform in WP7 and WP8 Testing of functional and other requirements will be analysed in WP8 7 References Anderer P Roberts S Schlogl A Gruber G Klosch G Herrmann W Rappelsberger P Filz O Barbanoj MJ Dorffner G Saletu B Artifact processing in computerized analysis of sleep EEG a review Neuropsychobiology 1999 40 150 157 Antelis J M Montesano L Giralt X Casals A and Minguez J Detect
36. 6 59 D3 2 Physical Physiological sensing devices Rev 1 0 The source localization of each trial was computed by sLORETA EEG data were converted to cross spectrum using default frequencies delta 1 5 6 Hz theta 6 5 8 Hz lower alpha 8 5 10 Hz upper alpha 10 5 12 Hz lower beta 12 5 18 Hz beta 18 5 21 Hz upper beta 21 5 30 Hz and all bands 1 5 30 Hz Figure 48 a d presents localization results for the raw EEG left and EEG filtered right recorded during target 25 a target 25 b target 45 c and target 45 d for subject 3 The source localization from the raw EEG filtered in the upper alpha is in BA 37 19 39 21 for all conditions respectively see Table 5 where this neural generators are not apparently related to motor behaviour Notice that in this case the artifacts act as noise for the sLORETA which is not able to locate activity on the motor cortex The source localization from the EEG free of artifacts and filtered in the upper alpha mu shows a prominent activity in the Brodmann area BA4 primary motor cortex on the precentral gyrus These results are consistent with previous description of motor cortex activation associated with motor execution and imagery Dyson et al 2010 Hanakawa et al 2003 Pfurtscheller and Lopes da Silva 1999 This result shows that after the filtering the primary motor cortex is one of the most relevant sources of the EEG Table 5
37. Controller Chest Unit module in order to visualize the low level communications with the sensors Since the debug application implements its own driver for the Chest Unit this allows comparison of the data provided by the debug application and the data forwarded on the ROS topic which in turn enables validation of the implementation of the HSS Controller driver 2 5 6 IMU Sensors Driver The structure and design of the IMU sensor driver is similar to the Chest Unit driver Figure 21 presents an overview its architecture 29 D3 2 Physical Physiological sensing devices Rev 1 0 3s IMU Sensors Engineer HSS_Back_TH Test Interface HSS_Env TH timestamp Long sequence Int16 skin_temp Int16 temp1 Int16 humidity1 Int16 temp2 Int16 humidity2 Int16 battery Int8 HSS_Back_TH IMU Back Units HSS_Env_TH BGAPI 6 wg a Bluegiga protocol over UDP ua BLED112 100ms HSS_Back_IMU HSS_Env_IMU timestamp Long sequence Int16 magneto Int16 3 accel Int16 3 HSS_Back IMU 9970 Int16 3 HSS Env IMU quaternion Int16 4 ws USB Serial lt n Bluetooth lt Smart 4 0 t IMU Sensor l Driver z Discover units and connect to 2 units back and environment IMU Environment Unit mounted on the demonstrator Publish data on 4 ROS topics 2 from the back unit and 2 from the environmental unit Figure 21 Interface of the IMU Sensor Driver The IM
38. MU Environment humidity and temperature sensor unit A special version of the IMU is used for measuring environment temperature and humidity These measurements are needed as a reference value for calculating the humidity at the patient The unit is mounted on the mobile unit and connected to the HSS controller using a USB cable The external sensor voltage regulator and the IMU unit is embedded into a small box e The external sensor is the same type that is used for IMU located on the patient s back Voltage regulator is used for providing power from the USB plug to the external sensor and the IMU environment unit The IMU unit will not have any battery to avoid procedures for charging the unit The unit will be operating as long as the HSS controller computer is running e All parts are integrated into a single box to avoid cabling between multiple boxes Cabinet USB Reg IMU sensor unit HSS controller PC Environment Sensor H Eth USB USB USB_ USB BLE BlueGiga if BLE protocol Legend HSS SENSORS Other HW SW Figure 14 IMU environment sensor unit internals The wireless interface is the same as for IMU back sensor unit The IMU unit may be enabled providing movement information for the mobile platform 2 3 3 IMU Sensor Unit Test The IMU sensor is currently in an internal design veri
39. ON OF ATTENTION DURING ASSISTED PASSIVE LEG MOTION cccccccccccscccsescceecececccecececeeeceseseceseseneeenens 65 4 SAFETY ccoiseiiscaciccccelessisevescscsesdocvecensscvesssdesdacecucdscdsccscbesasseduossscscvadesessecssoee cecesustsosenescebosseossosescocbosesescecsessossees 72 4 1 CHEST UNIT matela a N tenttsacnctansudeehenceabercnnendentsscccetenamtechinesanetenstodeatasaccesanontecheassinescarsodsttanes 72 4 2 DEA DIN i AER ESN AE ESSE AN ES E A E R ESN E E R RA 72 4 3 EMG UNITS maane e cues e a e a e a aoa e a a a aa a a aaa a Ss 72 4 4 HSS CONTROLLER COMPUTER iireriritiaaa ai a a e s aaa iea aar i aie RESE 72 4 5 OFFLINE CHARGER e ara aE E E E eO AEE O EEEE EEA EE E OEE 72 4 6 SN D AD EN AA RAET IANA AAAA E EI TAEAE AA IA IAIA R EAA 73 4 7 JD AEA DI i AAE AA E EE EEE A A A A AA 76 4 8 BCOISOFTWARE tesassceticstve ere E a E EA A E a rE ES 76 4 9 BEESAFETY ANALYSIS a a a a a i a aa a i a a a a a a aias 76 5 REQUIREMENTS FROM D2 1 cccccccsccccesesccesssesccecssenscecssesevecstescsedavesctecssenseedsvencsecssecssesaveccvecsdecssedsvecdseseues 79 6 CONCLUSIONS AND FUTURE WORK ssessesessessesessessosesscsesscssescosesscsesscsessessesecsessesesseseeseseessoseseesesseses 91 7 REBERE NCES E EE ETE T T A T A E 93 D3 2 Physical Physiological sensing devices Rev 1 0 Executive Summary This deliverable document D3 2 reports on work related to the development of Human Sensory System HSS and Brain Computer Interface BCI as performed in Work Pack
40. PO CORBYS Cognitive Control Framework for Robotic Systems BOTS Tee TS PROGRAMME CORBYS Cognitive Control Framework for Robotic Systems FP7 270219 Deliverable D3 2 Physical Physiological sensing devices Contractual delivery date Month 20 Actual submission date 30st September 2012 Start date of project 01 02 2011 Duration 48 months Lead beneficiary BBT Responsible person Marco Creatura Revision 1 0 Project co funded by the European Commission within the seventh Framework Program Dissemination Level Public PP Restricted to other program participants including the Commission Services Restricted to a group specified by the consortium including the Commission Services Confidential only for members of the consortium including the Commission Services D3 2 Physical Physiological sensing devices Document History Rev 1 0 Author s Revision Date Contributions Anders Liverud 0 1 02 07 2012 Draft document structure Marco Creatura 0 2 23 07 2012 Revision of the BBT contribution Anders Liverud 0 3 10 09 2012 Added IMU Sensor module documentation Anders Liverud 0 4 14 09 2012 Reset of track changes after much editing Steffen Dalgard Anders Liverud 0 6 21 09 2012 Draft document Marco Creatura 0 8 24 09 2012 BCI section Anders Liverud 0 9 26 09 2012 Input from review on HSS Marco Creatura 0 10 26 09 2012 BCI section internal revie
41. Physiological sensing devices Rev 1 0 No 20 271 7500 30 Targus 5 070 18388 3750 15 Roving RTS Much other activity on computer 68529 Roving RTS Much other activity on computer Table 2 Bluetooth bandwidth measurement results 16 D3 2 Physical Physiological sensing devices Rev 1 0 2 3 IMU Sensor Unit The IMU sensor is used in two settings in the CORBYS Demonstrator I system connected to the Chest Belt on the back of the patient and on the mobile platform for measuring environment humidity and temperature 2 3 1 IMU Sensor Back unit EMG sensor unit EMG interface Cabinet IMU sensor unit Environment HSS controller PC 7 USB powered LINUX ROS IMU sensor unit HSS controller SW Back Battery powered IW IMU sensor unit CORBYS ROS FRAMEWORK BLE Back BlueGiga if Ai A r ttyacm BLE protocol ging USB USB USB ee Charger eS uni uae HUB m0 Charging GPN switch unit Chest unit E P ERTE Battery powered BT protocol kogong HSS SENSORS Other Hw sw Figure 11 IMU sensor unit in Human Sensory system The IMU sensor is used for humidity and IMU measurements on the patient back and for environment temperature measurements The development for the IMU sensor was star
42. S format which is used by all CORBYS modules define CORBYS_NODES_HSS 42 CORBYS_Heart_Beat flags uint32 HSS_Chest_Phi HSS_Back_TH N N heades Maan pea timestamp Long HSS_Env_TH sequence Int16 timestamp Long APRE r E HSS Chest Phi heart_rate Int16 sequence Int16 i CORBYS_Message_Header i CORBYS HSS_Chest_IMU skin_temp Int16 skin_temp Int16 U re one G PN HSS Chest ECG activity Int8 temp1 Int16 eee T E eee ener cua tene T posture Int8 humidity1 Int16 h battery Int8 temp2 Int16 eader humidity2 Int16 HSS_Back_TH HSS_Chest_IMU battery Int8 BD srny oO HSS_Back_IMU timestamp Long Sensor Data HSS sequence Int16 HSS_Back_IMU sensorDatal gt ee accel Int16 3 HSS _EnvIMU P o We i PSS Enan gyra Anne timestamp Long CORBYS_Sensor Samples HSS_Env_IMU sequence Int16 PA ince HSS_Chest_ECG accel Int16 3 SRO E ie teen timestamp Long gyro Int16 3 i HSS_EMG_Data sequence Int16 magneto Int16 3 samples ai raw_ecg int16le quatemion ines Gone Sensor Reading e HSS internal topics and messages a timestamp time reading float32 CORBYS topics and messages Figure 23 From the Internal topics to the CORBYS topics 32 D3 2 Physical Physiological sensing devices Rev 1 0 Figure 23 presents the ROS messages used by the HSS Controller front end The left hand side of the
43. U Unit sensors use the Bluetooth Smart 4 0 protocol for which no support is available in Linux To communicate with the sensors the HSS Controller uses an external Bluetooth Smart 4 0 module which implements the protocol stack and provides and API over a USB virtual serial port The chosen module is a Bluegiga BLED112 USB dongle The IMU Sensor driver connects to two different IMU sensors One is mounted on the back of the chest belt of the patient and the other is mounted on the CORBYS demonstrator mobile platform and used as an environmental sensor The environment device is a single unit which has an address that can be hard coded in the HSS Controller The back unit is one of several units available for training sessions and needs to be discovered and configured in the initialization phase of each training session At least two units need to be available in order to be able charge one while the other is being used on a patient The IMU Sensor driver outputs data on 4 different topics 2 topics for the environment unit and 2 for the back unit The topic messages have been designed to group together data which have common update rates e 3 seconds for the temperature and humidity measurements and the battery status The temp2 and humidity2 value are currently not populated in the CORBYS demonstrator but provided for future 8 http www bluegiga com BLED112 Bluetooth low energy dongle 30 D3 2 Physical Physiological sensing devices Rev
44. age 3 Sensing systems for assessing dynamic system environments including humans of the CORBYS project Based on deliverable documents D2 1 Requirements and Specification and D2 2 Detailed Specifications of the System this document describes the realisation of the Human Sensory System and the analysis of Brain Computer Interface processes related to development and integration of the BCI module in the CORBYS 1 demonstrator To meet challenges in physiological monitoring in robotics systems a human sensory system has been realized Compared to available commercial sensor systems the CORBYS human sensory system has features that make it suitable for robotics environments It is compact easy to attach and remove and does not disturb or cause any discomfort for the user The sensors provide relevant physiological measures and can coexist with robotic systems like gait rehabilitation robots Data is provided on a standardized robotics software interface The sensors have been realized through development of a chest belt with sensors situated on the chest and back of the patient Physiological parameters like heart rate with ECG humidity skin temperature are measured together with velocity and orientation These parameters will be used by the cognitive modules of the CORBYS Demonstrator I robot assisted gait rehabilitation system for assessing physical effort identifying psyco physiological states and for identifying intentions of the patient Simi
45. al recorded during relaxation black and walking blue Figure 51 presents the average i e all 4 subjects of the experiment logarithmic power spectral density for EEG signal recorded during walking and relaxation The spectral landscape is similar for all the channel locations peaking at low frequency and decreasing smoothly as the frequency is higher The spectra show a 62 D3 2 Physical Physiological sensing devices Rev 1 0 peak in the alpha band 8 13 Hz that is largest in the centro parietal locations i e CP3 CPz and CP4 Individual PSD displays sharper spectral peaks than the average one As in the previous experiment ICA decomposition was computed on the walking dataset separately for each subject The ICA components were selected by visual inspection and depending on the subject the components chosen varied between 7 and 8 Figure 52 presents some EEG trials without artifacts reconstructed through reprojection Subject Walking Subject Walking Filtered ee er ee ee S s sei SSS oe ee DAI repr Me Me pr fot 0 2sec Figure 52 2 seconds portion of raw EEG left and EEG filtered right recorded during walking for all subjects The filtered signal is obtained by removing the selected ICA components identified as artifacts 63 D3 2 Physical Physiological sensing devices Rev 1 0 3 2 4 Discussion This deliverable reports on the analysis which describe temporal and spectrual characteristics
46. ame data provided as listed Same as HSS6 gait parameter assessments in HSS6 HSS8 Sensor output related to Same data provided as listed Same as HSS6 identification of psyco in HSS6 physiological states HSS9 Sensor output related to Same data provided as listed This will be handled in D5 3 for identification of intentional states in HSS6 80 adaptive walking mode This is where SOIAA provides gait trajectories from BCI intention data HSS sensor data which facilitate the patient and orthosis being able to walk freely in a structured environment D3 2 Physical Physiological sensing devices Rev 1 0 sensor probes on the patients No limitations are stated at this stage HSS10 Data processing and signal The project needs to sum up HSS controller will be running on analysis requirements all the data processing separate computer with sufficient requirements both capacity processing power for HSS tasks and platform wise HSS11 Integrating physiological Discussions between BBT and All sensor measurements are sensor measurements SINTEF regarding finding a presented to cognitive modules as SINTEF with BCI BBT shared platform for integrating individual ROS topics BCI and sensor signals HSS sensor data will be provided for cognitive modules with consistent timings allowing for various latencies to provide a full picture of the patient over time HSS12 Interfacing
47. an two orders of magnitude greater than in the TMSi system for frequencies between 0 5 60Hz and for all the subjects For both systems the ratio frequency spectra show a peak at 20 Hz and at its fundamental harmonic frequencies i e 40 Hz 60 Hz etc although in case of g Tec noise data is spread over all frequencies while it is not the case for the TMSi system 3 1 5 Discussion Two commercial EEG systems are presented in this document Results show that TMSi equipment was less affected by electrical noise than g Tec system There could be two reasons result in this effect differences in technical characteristics in the electrodes or the differences in the technical features of the amplifiers While the electrodes could affect the data quality due to the impedance between sensor and scalp the amplifier could alleviate deficiency regards signal quality In summary the TMSi system has a better performance in the presence of motor activity and unknown sources of electrical noise which is the best replication that has been devised to emulate the working conditions of the CORBYS experimentation 51 D3 2 Physical Physiological sensing devices Rev 1 0 3 2 Analysis and removal of movements artifacts during locomotion 3 2 1 Introduction Wide range of artifacts can occur in EEG recordings One possible categorization of artifacts is based on their origin technical originated from outside the human body such as the 50 60 Hz power line noise ch
48. and magnetometer at patient chest and back To be implemented in D3 4 EMG muscular activity Mechanical sensing force torque angular joint movements force pressure distribution will be implemented by sensors in Demonstrator I in WP7 HSS2 Sensor locations The actual sensors selected will be defined in the detailed specification Implemented sensor and functionality Homing positions of inertial sensors is handled at session start up by therapist A sensor for measuring environment temperature and humidity is developed and will be attached on mobile platform Requirements that will be handled by mechanical sensors in Demonstrator I in WP7 Positions of robot mechanical support to the patient such as limb fixation and patient movement actuators HSS3 Patient user size Adult users Height weight circumferences will be discussed with clinical partners HSS will fit all patients relevant for CORBYS usage HSS4 Sensor output of primary parameter values Definition on how human sensor parameters are shared with the rest of the CORBYS system as well as in export to Implemented Sharing is done through ROS topics details are described in HSS Controller section 79 D3 2 Physical Physiological sensing devices Rev 1 0 log files Details are to be defined Logging to file is used in early integration phases will also be available as ROS bags See HSS
49. anges in electrode impedances etc and physiological originated from a variety of bodily activities such as potentials introduced by eye or body movements muscular activity cardiac activity etc While some of these artifacts are easily identified others may have similar characteristics to the neural activity and therefore difficult to recognize and to eliminate The presence of artifacts in EEG signal has an impact on the analysis and could lead to unreliable results if they interfere with the neural process under study This is the reason why one important part of biomedical signal processing is to understand the noise and the artifacts in order to minimize their impact either in the analysis or in the development of the technology The two physiological artifacts being mostly studied are the eye movements EOG and in the body movement EMG EOG artifacts are due to electrical eye activity propagated throughout the body and recorded at the scalp surface Schl6 gl et al 2007 They are generally high amplitude patterns in the brain signal caused by the blinking of the eyes or low amplitude patterns caused by movements such as rolling the eyes Anderer and Roberts 1999 The EOG amplitude is attenuated approximately with the square of the distance Croft and Barry 2000 and thus contaminates mostly the frontal EEG channels The EOG activity spans a wide frequency range being maximal at frequencies below 4Hz The EMG artefacts are generated in
50. architecture 2 1 1 Sensor Module architecture Sensor modules differ in type and functionality The simplest module can consist of a single sensor and a communication module transmitting raw data to the HSS controller HSS Sensor Modules contain multiple sensors and a sensor controller handling multi sensor interpretation as well as transmitting data The sensors are grouped into 4 main groups physiological sensors movement sensors environmental sensors and mechanical sensing technologies Mechanical sensors are not in the scope of HSS Figure 4 shows the gait rehabilitation HSS Sensor Module measurement positions along with their use in the gait rehabilitation application Figure 4 Measurement positioning of the gait rehabilitation HSS Sensor Module Table 1 lists all the sensors in CORBYS Human Sensory System D3 2 Physical Physiological sensing devices Rev 1 0 Sensor Purpose Gait rehabilitation measurement position Physiological sensors Electromyography EMG Measuring the electrical activity produced by the skeletal muscles Measured on thigh and calf both front and rear muscle on both legs A total of 8 sensors Heart rate and ECG signal Used in the assessment of physical effort The heart rate is derived from the ECG signal measured on the patient chest The full ECG signal is also provided Skin temperature Monitor skin temperature Skin temperature is measured on the
51. bundle together the data with common update rates e 3 seconds for skin temperature heart rate posture and activity data The battery status has been added to this message even if its actual update rate is around 30 seconds e 52ms for the IMU data accelerometers and gyroscopes 26 D3 2 Physical Physiological sensing devices Rev 1 0 e 32ms for 8 samples chunks of ECG data which has an update rate of 4ms At this stage chunk of 8 samples seems to be a good trade off between bandwidth and latency It might be adjusted if required by the nodes using the data On the sensor side the chest units are connected to the HSS controller via Bluetooth using the Serial Port Profile SPP The SPP is supported by recent Linux kernel modules but turned out to be unstable in Ubuntu 10 04LTS To avoid any instability the HSS controller is setup to use an external Bluetooth adapter which implements its own Bluetooth stack as well as the SPP The chosen adapter is a BlueSMiRF Bluetooth modem which is based on a Roving Networks Bluetooth module The HSS controller communicates with the Bluetooth using a standard serial interface ChestBelt Engineer Test Interface Publish data on 3 ROS topics 3s HSS_Chest_Phi timestamp Long ChestBelt protocol over UDP Chest Unit USB Serial Sensors gt Driver Roving Networks Bluetooth Module Discover several sensors but connect to one belt at the time sequence Int16
52. cally mounted and provided with a device handle named dev ttyUSBX where X varies In order for the Chest Unit driver to always be able to connect to the right device we have used the Linux UDEV service in order to automatically create a symbolic link with a fixed name whenever the specific FTDI chip is connected Figure 20 presents the UDEV rule added to create a link from dev bluesmirf to the appropriate device This rule includes the serial number of the FTDI chip used with the Bluetooth module This serial number should be changed to match the actual chip being used on the HSS Controller The dev bluesmirf can be used as a standard serial port with an 115200 baud rate 8 data bits 1 stop bit and no parity 8N1 Hardware flow control is supported all the way from the computer to the FTDI chip Bluetooth module and Bluetooth SPP The Chest Unit driver itself consists of four main components blue in Figure 20 e The Roving Network AT Serial driver This component manages all the communications with the serial device It uses AT commands to interact with the Bluetooth stack of the module when no sensor is connected and it decodes serial packets coming from the sensor once a connection has been established It provides a simple API for discovering and connecting to Bluetooth devices and allows sending and receiving packets to the Chest Unit e The Chest Unit Driver encodes and decodes serial packets according the Chest Unit protocol It provides
53. ch as attention affect rehabilitation exercises and they play a key role in the result of the therapy Presented work shows that the attention modulates brain activity during passive leg movements as observed in the EEG signal An off line classifier was developed to differentiate between two states attention or distraction to the motor task The results show the feasibility to distinguish between both conditions above the chance level for all the subjects This is the first step in the development of the attention decoding unit of CORBYS This research could be improved in several directions For instance source localization techniques will be needed to confirm that the EEG changes observed between conditions have the neural origin in the motor cortex as demonstrated by fMRI studies Johansen Berg et P Matthews 2002 In addition to this the subjects had to focus in the screen but their legs were not concealed from his or her gaze which could have an influence on the motor activity Although this aspect influences both conditions further investigation is required Future work will consider the movement artifacts analysed in the section 3 2 Analysis and removal of movements artifacts during locomotion of this deliverable and the automation of feature selection process Eventually notice that the final next step is to build an online classifier to assess the feasibility of decoding attention during motion in real time which is the user scenario in
54. controller sections HSSS5 Safety related sensor output Status information or flags Project has not identified need for information derived from should be raised if sensor sensor fusion from HSS for Safety sensor fusion of multiple readings indicate a potentially Handling of potentially hazardous sensors hazardous situation To be situations will be discussed in discussed with partners with D7 4 stakes in the design of CORBYS control system HSS6 Sensor output related to HSS will provide human Data will be processed by physical effort assessments sensor data related to physical SAWBB to be described in D4 3 effort which is heart rate Initial discussions have taken ECG humidity temperature place between SINTEF and UR on EMG and movement data input requirements for the Situation Assessment architecture with respect to physiological sensor output Based on these requirements sampling rate timestamp resolution etc have been revised for heart rate ECG The revised ESUMS HSS dataset from SINTEF was received by UR and is currently being utilised as part of effort undertaken in WP4 Similarly activity sensing output IMU data provided as input to the Situation Assessment architecture UR is making use of revised HSS ESUMS dataset in which gyroscope was included Also SOIAA will make use of this data in the adaptive walking mode that will be presented in D5 2 and DS5 3 HSS7 Sensor output related to S
55. d in the ROS parameter server using the HSS Controller GUI Once the selection of the sensors has been made the drivers attempt to connect to these sensors e Log all versions of software hardware and firmware The first step after connecting to the sensor is to query the sensors for their versions and status in order to populate the parameter server with complete information about the sensors being used and confirm that the connection is properly established e Synchronization of the clocks of the sensors with the HSS Controller In order to get accurate time information the wireless sensors need to be synchronized with the HSS Controller clock which is itself synchronized with other CORBYS nodes Experiments will be made in order to evaluate if a time synchronization over the air allows for a sufficient accuracy In case wireless transmission delays are too unpredictable the clock synchronization could be done using serial communication with the sensor on the charging station In that case the synchronization would have to be made before the sensors are installed on the patient e Configuration of the sensors and subscription to the data At this stage the sensor drivers will configure the sensors according the required CORBYS update rates and start collecting data from the sensors e Start forwarding data on the CORBYS GPN After all sensors have been configured and the driver has stated forwarding data on the HSS internal topics the front
56. developed under a controlled situation where a DC brush motor was used as proposed by related partner developing mobile platform i e there will be the motors used for the powered orthosis and for the wings in the mobile platform the second experiment involved a less controlled scenario where the source of electromagnetic noise was caused by a generic AC motor 43 D3 2 Physical Physiological sensing devices Rev 1 0 Figure 29 Commercial EEG systems On the left passive electrode water solution and Porti 16 of TMSi On the right active electrode gel solution and g USBamp of g Tec 3 1 3 EEG Systems Evaluation of technical specifications The electroencephalogram EEG is an acquisition system that measures electrical brain activity through electrodes placed on the surface of scalp These sensors measure neural activity by electrical potentials voltage over time between a signal electrode and a reference one Every EEG system includes three acquisition blocks the electrodes the wires interface sensor amplifier and the amplifier filtering amplifying and signal conversion Electrodes The brain activity is acquired by electrodes which build an interface between the scalp skin and the metal of the wire The electrode is a metallic sensor that converts the local differences of the concentration of charged ions into an electrical potential signal However the measurement of the electrical activity of target physiological pr
57. e 49 D3 2 Physical Physiological sensing devices Rev 1 0 The experimental design is shown in Figure 37 with two conditions motor ON and motor OFF A trial was composed of a rest time of 3 seconds and baseline task of 5 seconds The subject was asked to avoid blinking during baseline period A total of 60 trials were recorded 30 trial of each condition i START Rest Baseline END TRIAL 1 TRIAL 2 ee TRIAL n 1 TRIAL n Electrical mini bike ON Electrical mini bike OFF Figure 37 Time diagram of protocol in the experimental procedure Data processing and results Artifact filtering was performed by visual inspection of the baseline recordings in both conditions Epochs of one second were discarded if a physiological artifact was identified EEG power spectrum was calculated by a sliding window periodogram of one second with 30 ms of overlapping and then averaging To compute the periodogram a hamming window was used with a resolution of 0 25 Hz 1024 points using zero padding and power line notch filtered at 50 Hz and bandpass filtered between 0 5 and 60 Hz Figure 38 shows EEG signal affected by noise i e 20 Hz sinusoidal contamination l i i Wi o in Hh ih ht ee Figure 38 Example of four EEG channels contaminated by electric mini bike recorded using TMSi left with scale of 100 uV and g Tec right with scale of 1000 uV system over a time window of 4
58. e is is T BE ac E a on Pr m E ho mrana A e o E EMG L L AL t 1 4 4 i 4 Toren Witre Saa 10 ls te te ar aral P 5 r Ri Ci Fer 8 Fed F grel P iG a ig a a oral Pr P doe a ein hale ah oa NT aa AT MA TT WPAN A A A Sa Oe ee E iaa a MENTA GIIA G AT ETM D No Gel a u Tre 1a wnase Sure 19 ra s a n fe Fe fe nn rem Nee mee Nee ml ee Ne a Sea al ay r ps a tS rs Ta EOG lateral movement s gr 1 z geez2223239 EOG blinking ge922733999220229 Tere 0 winana Bam 10 Figure 46 This figure shows for subject 3 scalp maps right and the component reprojected back to the sensor space left of the four ICs that contain artifacts muscular activity electrode without gel eye blinking and eye lateral movement 58 D3 2 Physical Physiological sensing devices Rev 1 0 For one subject the Figure 46 shows several components eliminated and some EEG trials without artifacts reconstructed through reprojection Figure 47 Notice how the presence of the main artifacts due to the head movement is mitigated although in the last two conditions some EMG is still present Relaxation Relaxation Filtered a Figure 47 This figure shows for subject 3 two seconds portion of EEG signal left recorded during relaxation target 25 target 25 target 45 and target 45 The same EEG free of artifacts right The components eliminated are displayed in Figure 4
59. e behaviour is solved in the spinal cord 55 D3 2 Physical Physiological sensing devices Rev 1 0 3 2 3 Results The channels located in central areas were selected for further analysis as they are the location where the legs movements are better observed Pfurtscheller and Lopes da Silva 1999 The following nine channels were chosen FC1 FCz FC4 C1 Cz C2 CP3 CPz and CP4 Experiment I Relaxation S Sn J reap 4 50uV ii 0 2sec rapa A A ir arrra A ana aa o E Da os on g E 1 1 is 2 Target 25 Target 25 oy Figure 43 This figure displays for subject 3 two epochs of EEG signal recorded during relaxation top target 25 middle left target 25 middle right target 45 bottom left and target 45 bottom right Figure 43 presents the EEG signal recorded during relaxation and target 25 and 45 where the EEG is contaminated by the EMG artifact generated from the head movement The influence of the artifacts in the frequency domain is that the power is higher when the angular movement is higher and this effect is more prominent at high frequencies 10 40 Hz Figure 44 displays the result for one subject and Figure 45 the 56 D3 2 Physical Physiological sensing devices Rev 1 0 results of the mean of all the subjects Notice that the spectra shows a peak in the alpha band 8 13 Hz that is largest in the centro parietal location i e CP3 CPz and CP4 in the not motion state and this act
60. e for translating the high level plans cognitive inputs into low level actions invoking actions at the appropriate times monitoring the action execution and handling exceptions The executive layer can also allocate and monitor resource usage Communication Server Task manager Manage subscriptions of sensor data between different control modules The sensor data to the cognitive modules are not flowing through the Communication Server but are forwarded directly The task manager manages operation modes to be executed by the system Performs specific tasks when the operation mode is changed FPGA Reflexive Module Field Programmable Gate Array FPGA based hardware subsystem of Situation Awareness architecture SAWBB for acceleration of robot reflexive behaviour Safety Module Verification that actuator output is in line with the commanded output and that it satisfies safety related position velocity current and or torque constraints Real Time Data Server Real time data server is a software module responsible for communicating sensor data from real time RT bus to other software modules This excludes communication of RT modules with sensors and actuators which communicate with sensors and actuators directly in order to preserve RT control behaviour Real Time Network RTN General Purpose Network GPN Sensor network for real time safety critical data transmission Network for robot control and int
61. e s sensor measurement sensor module out of no sensor data BCI software 8 3 1 failure power detects when the sensor module is out of power 77 D3 2 Physical Physiological sensing devices Rev 1 0 measurement instrumental environmental incorrect Signal 3 failure noise sensor data to processing other module s algorithm of the BCI software mitigate the potential effects measurement patient movement noise incorrect Signal 8 failure sensor data to processing other module s algorithm of the BCI software mitigate the potential effects measurement physiological noise incorrect Signal 9 failure sensor data to processing other module s algorithm of the BCI software mitigate the potential effects Table 8 Safety analysis for BCI 78 D3 2 Physical Physiological sensing devices 5 Requirements from D2 1 Rev 1 0 This section lists the relevant requirements for Human Sensory System and Brain Computer Interface from D2 1 and how these have been fulfilled or implemented Req HSS1 Name Sensors implemented in the CORBYS system Description The actual sensors selected will be defined in the detailed specification Realization Implemented Heart rate and ECG Skin temperature at chest and back Humidity sensors at patient back Inertial measurement units 3 axis accelerometer gyroscope
62. ed on Hyv rinen et Oja 2000 to eliminate the components from blinking EMG artifacts and electrical noise from the mini bike Each component was reprojected back to the sensor space and time and time frequency representations of these reprojections with the associated spatial filter were visually inspected to assess whether the component was artifacted The components free from artifacts were reprojected back to the sensor space to obtain an artifact free EEG This EEG was filtered and then for each trial the 3s 3s interval with respect to the mini bike onset was extracted see Figure 54 These new trials were again visually 67 D3 2 Physical Physiological sensing devices Rev 1 0 inspected to assure they are artifact free Then for each trial the across trials average was subtracted for each condition in order to remove the evoked potential caused by either the beep sound from the mini bike device when it was activated and or the sudden start of the motion The ERD ERS during the motion process was obtained by Pj Ppas Pbas 100 where Pj is the power of each time frequency bin of the j trial from the onset of the mini bike Os to 3s and Pbas means power of the baseline 3s to Os Power spectra were computed by the Welch method with a sliding window of 128 samples with 120 samples of overlapping between consecutive windows ERD ERS is represented as a time frequency plot where time varies from 3s to 3s and frequency from 0 Hz to 50 Hz
63. emperature at low medium and high H1 Humidity sensor humidity and compare results with Sensirion development kit results 1 second update rate Compare IR temperature readout at 1 seconds readout interval with values from ESUMS IR1 Skin temperature Chest Belt high low and medium temperature M1 Magnetometer Compare readout with other device reference device to be specified Al Accelerometer Test against g in x y z directions G1 Gyroscope Mount unit at stepper motor and compare readout with stepper motor speed at various speeds Cl Communication Test max distance before communication lost C2 Communication Communication performance tested and presented at pHealth 2012 Liverud et all 2012 Table 3 IMU sensor Unit design verification test Tests have so far uncovered that battery charging and discharging does not work fully as intended and needs to be corrected This will probably be corrected by an embedded firmware update for the IMU Sensor Unit Full tests results will be provided in D3 4 4 http www sensirion com en products humidity temperature humidity sensor sht2 1 21 D3 2 Physical Physiological sensing devices Rev 1 0 2 4 EMG sensors an Hss sensors ter HW SW Figure 15 EMG sensors in Human Sensory System The purpose of the EMG sensors is to measure the muscle activity of the patient EMG will be measured on the thigh and calf of each leg The CORBYS EMG system will be presented
64. encephalographic signal acquired by Integration in the BCI the BCI hardware architecture will be addressed in Task 3 4 Deliverable D3 4 Integration in the CORBYS system will be addressed in WP6 BCI4 Filtered EEG Electroencephalographic signal filtered from Signal processing Decoding occurring artefacts part of the requirement has been partially addressed in the current deliverable i e section 3 1 and 3 2 where an analysis of EEG hardware and EEG artefacts has been performed Further analysis will be addressed in Task 3 3 and 3 5 Deliverable D3 3 Integration in the BCI architecture will be addressed in Task 3 4 Deliverable D3 4 Integration in the CORBYS system will be addressed in WP6 BCI5 Intention of legs motion Information about which leg the subject is Signal processing Decoding decoding flag going to move It also provides a no part of the requirement will 88 D3 2 Physical Physiological sensing devices Rev 1 0 movement output value The intention of legs motion decoding flag output can take the following values right leg left leg and no be addressed in Task 3 3 Deliverable D3 3 Integration in the BCI percentage related to the ability of the BCI subsystem in detecting the intention of legs motion movement architecture will be addressed in Task 3 4 Deliverable D3 4 BCI6 Decoding accuracy It provides a numerical value e g a Refer to BCI5 BCI7 Error marker
65. end will start forwarding the data on the CORBYS topic At this stage the HSS Controller is still in initialization mode and the subscriber to the sensor data should use the incoming data only for initialization checks e Basic checks of the selection and positioning of the sensors The CORBYS GUI should be used to visualize the data coming from the HSS Controller and confirm that all sensors are providing sensible data and that all sensors are properly mounted In particular o ECG Signal should be visualized to confirm that the electrodes are properly placed A drop of water can be used to moisten the electrodes and the belt size should be adjusted for a good contact between the electrodes and the skin o Chest IMU and Back IMU data should be checked to confirm the correct orientation of the Sensors o Skin temperature should be checked for both the Chest and Back sensors to make sure that no obstacles block the IR sensors o Specific checks will be required for the EMG sensors e Switch to training mode After the sensor mounting and connection has been confirmed by the user the HSS controller switches to the training session mode in which the data can be exploited by subscribers 36 D3 2 Physical Physiological sensing devices Rev 1 0 At the end of a training session the HSS controller stops forwarding sensor data disconnects from the sensor and goes back the standby mode 2 5 11 HSS Controller Test The HSS controller is desig
66. er 91 D3 2 Physical Physiological sensing devices Rev 1 0 Evaluate effect of noise for human sensory system Electromagnetic noise from actuators motors radios etc at the mobile platform and in the environment may interfere with senor measurements The possible effect of this will be investigated and reported in D3 4 Physical integration The Human Sensory System will be integrated with Demonstrator I from WP7 as a part of WP8 System Integration When Demonstrator I is ready for integration this work will start by integrating the HSS computer in the carrier frame together with external antennas and cabling to the general purpose network HSS will then be tested in Demonstrator I environment and results will be compared to results from similar tests prior to integration This activity is dependent on WP7 and WP8 work and may not be completed for D3 4 In this present deliverable three topics related to the design of a BCI submodule within the CORBYS gait rehabilitation system have been addressed In section 3 1 Evaluation of EEG acquisition systems that could reduce the noise level two EEG systems i e TMSi and g Tec have been analysed in order to choose the most appropriate for the CORYS gait rehabilitation system Due to the results obtained where the g Tec system showed to be more affected by electrical noise contamination TMSi has be chosen as EEG system for the CORBYS 1st demonstrator In section 3 2 Analysis and removal of
67. erface to the cognitive modules Demonstrator Specific Technology Comp onents Mobile Robotic Gait Pelvis Link Rehabilitation Mechanical interface between the mobile platform and the powered orthosis equipped with an appropriate actuation and sensing system System Powered Orthosis Exoskeleton system to help the patient in moving his her legs and receiving an appropriate rehabilitation therapy Mobile Platform Reconnaissance robot Vision System for Investigation of The platform for the entire system including Pelvis Link Powered Orthosis necessary computational storage and power supply modules as well as motored wheels for movement Cameras of the 2 demonstrator used for environment perception including human tracking 7DOF lightweight robot arm mounted on the Mobile Platform used for the object manipulation for contaminated area sample drawing Hazardous Robot Arm Environments Mobile Platform Mobile platform of the 2 demonstrator which consists of a variable drive system that is equipped with chains It is used for mounting of the robot arm and sensors for environment perception as well as sensors for platform navigation and robot arm control Containers for samples are also placed on mobile platform IV D3 2 Physical Physiological sensing devices Rev 1 0 Table of Contents CORBYS DEFINITION OF TERMS fSiscccsscdeteastesesseccucesesccucssscucecevesdexseeseccsvesacacvsuesa
68. evice inode the HSS controller uses a custom UDEV rule which creates a dev bled112 symbolic link for the BLED112 dongle The UDEV rule is generic for all BLED112 dongles and assumes that only one is plugged to the HSS Controller In case several dongles need to be used specific rules which include the serial numbers of the dongles should be created The dev bled112 devices is a standard virtual serial port with a 115200 baud rate 8 data bits 1 stop bit and no parity 8N1 The BLED112 dongle uses a proprietary packet based protocol over this serial link The implementation of the IMU Sensor driver is composed of four layers and a UDP GUI server which is used for diagnosis and debugging purposes These five components are modelled using the ThingML languages and the C C and ROS implementation code is automatically generated e The Bluegiga transport protocol manages the connection with the dev bled112 device and sends and receives packets conforming to the Bluegiga proprietary protocol 31 D3 2 Physical Physiological sensing devices Rev 1 0 e The BGLIB layer provides an API to interact with the Bluetooth Smart stack of the BLED112 dongle API commands are transformed to packets to be transmitted to the dongle and incoming packets coming from the dongle are decoded and forwarded as messages or callbacks the layer above e The IMU Sensor driver uses the Bluetooth Smart API provided by the BGLIB layer and implements the specific services pr
69. evices Rev 1 0 HSS Controller Engineer GUI Chest Unit Sensors HSS_Chest_Phi HSS_Chest_IMU HSS_Chest_ECG Bluetooth HSS_Back_TH Config_Data HSS_Back_IMU IMU CORBYS Sensors G P N BLE HSS_Env_TH HSS_Env_IMU Sensor_Data_HSS rear Heart_Beat HSS_EMG_Data EMG A Sensors Not implemented in D3 2 Prototype Figure 18 Architecture of the HSS Controller Software Each component described in Figure 18 runs in its own process ROS node in the HSS controller in order to avoid having the failure of one component bring down the whole HSS controller The communication between the drivers and the front end component uses a set of ROS topics which publish the data received by the drivers These ROS topics are internal to the HSS controller and should not be used by other CORBYS nodes The HSS front end receives the data from all drivers performs some sanity checking on the data and forwards sensor data to the CORBYS Sensor_Data_ HSS topic The following sub sections describe the components of the HSS controller software 2 5 5 Chest Unit Driver Figure 19 presents the interfaces of the Chest Unit driver In the initialization phase the driver needs to discover the chest units within range and read their battery status Once a session is initialized the driver is connected to only one chest unit collects the sensor data and publishes it on three separate ROS topics These three topics were designed to
70. fication test where all features are tested against specifications The planned tests are listed in Table 3 ID Module Test description From completely empty battery charge battery Measure charging time Record charging B1 Battery curve from console printout From fully charged battery measure usage time for measuring IR temperature and dual humidity sensor before battery empty B2 Battery Record data using IMU Sensor Unit Java app and console From fully charged battery measure usage time for measuring all sensors before battery B3 Battery empty Record data using IMU Sensor Unit Java app and console 20 D3 2 Physical Physiological sensing devices Rev 1 0 Ll LED Test led at power on off charging low battery may change BU1 Button Push button for 3 seconds to switch on similarly 3 seconds to switch off BU2 Button Short push to make unit start advertising Measure standby current before and after running unit with all sensors multimeter average Pl Power usage of 200 samples P2 Power usage Measure current usage not transmitting any data P3 Power usage Measure current usage transmitting IR temperature and dual humidity sensors P4 Power usage Measure current transmitting data from all sensors Test that the unit is automatically switched off after 5 minutes when not connected on P5 Power usage Bluetooth Compare humidity sensor readouts for humidity and t
71. figure presents the ROS messages provided by the sensor drivers These messages are specific to each individual sensor and use the data types provided by the sensor no scaling or any other form of processing is done in the sensor driver nodes The right hand side of the figure presents the structure of the ROS messages exchanged on the CORBYS GPN The CORBYS Sensor Data message is a generic structure which is designed to represent data coming from all the different sensors Each message is composed of a header which provides the origin of the data and a set of sensor samples Each sensor sample corresponds to a specific sensor ID and can contain a set of sensor readings Each reading has a timestamp and a set of sensor data represented as 32bit floating point numbers This structure allows representing in a single message data coming from different sensors as well as several sets of readings for a single sensor 2 5 8 1 HSS Controller data interface The HSS Controller main functionality is to collect the data from the different sensors and publish it on the Sensor_Data_HSS topic The front end receives the raw data from the sensors the data needs to be processed scaled and converted to a set of 32bit floats which can be provided to the CORBYS topic Other CORBYS nodes should not subscribe to the HSS internal topics but only to the HSS_Sensor_Data topic Several alternatives can be used to define the sensor IDs on the Sensor Data HSS topic e
72. gorithm does not compensate for the delay in the messages The result is an uncertainty in the absolute accuracy in the range 0 250ms There are currently no specific requirements for the absolute accuracy the minimum required accuracy will be investigated and added to D3 4 These requirements will be implemented in the final version of the HSS Controller http www bluetooth org Technical Specifications adopted htm D3 2 Physical Physiological sensing devices Rev 1 0 The delay variation for the Bluetooth Smart communication between IMU sensor unit and a computer has not been yet tested however it is expected to be better than the delay variation measured for the Chest unit The current implementation adds timestamp when sensor data arrives at the computer The relative delay between samples will be low however the absolute accuracy will not be better than the delay characteristics of the Bluetooth Smart link This will be tested during setup of the system and test results will be presented in D3 4 10 D3 2 Physical Physiological sensing devices Rev 1 0 2 2 Chest Unit EMG sensor unit EMG interface Cabinet IMU sensor unit Environment H troller P SS controller PC A a LINUX ROS IMU sensor unit HSS controller SW Back Battery powered CORBYS ROS FRAMEWORK x
73. he BCI submodule Medium and high risks are related to EEG signal contamination i e patient movement noise and physiological noise section 3 1 Analysis evaluation and removal of movement artifacts from EEG measurement during locomotion of this deliverable addresses the mechanical artefacts and their removal process Removal algorithms mitigate the effect of the artifacts on the EEG signal where the detection value in the table indicates how much the filter can reduce this contamination This value will change since it is strictly dependent on the decoding algorithm implementation actually under study 76 D3 2 Physical Physiological sensing devices Rev 1 0 POTENTIAL POTENTIAL DETECTION Recommended Pos FUNCTION failure MODE POTENTIAL CAUSES EFFECTS METHOD SEV OCC DET RPN Action s SEV OCC DET RPN 1 BCI software GPN connection network error no sensor data ROS heartbeat 8 4 1 loss to other module s data package GPN network error delay on Data timestamp 7 4 1 Check Timestamp 7 4 1 loss sensor data 2 BCl sensors measurement incorrect sensor setup no sensor data BCI software 7 5 2 failure to other detects module s incorrect sensor setup measurement incorrect sensor no sensor data BCI software 7 2 1 failure connection to other detects module s incorrect sensor connection measurement faulty sensor no sensor data BCI software 7 3 3 failure to other detects faulty modul
74. he HSS controller front end ROS module integration with the CORBYS demonstrator a set of datasets will be provided as ROS bags which can be played to simulate the presence of the HSS sensor modules for the HSS controller front end This enables usage of the HSS controller front end without having the HSS sensor modules present An aggregated dataset containing data from HSS BCI and hardware sensors should be made This will give synchronized data from all modules for a training session 38 D3 2 Physical Physiological sensing devices Rev 1 0 2 6 Offline charging station for wireless sensors EMG sensor unit EMG interface Cabinet IMU sensor unit Environment H troller P SS controller PC T USB powered siaal LINUX ROS CORBYS ROS FRAMEWORK IMU sensor unit Back Battery powered IW IMU sensor unit HSS controller SW BLE a BlueGiga if Sa ae F ttyacm BLE protocol ging USB USB USB est uni Charger i USB HUB m i Charging GPN switch unit Ghastunit Bluetooth COM portit Vz Battery powered BT protocol Legend iiss HW Other HW SW Figure 26 HSS controller in Human Sensory system The chest unit and the IMU sensor need to be charged regularly There will be two sets of sensors availab
75. he extended 10 5 international system For both EEG systems data were digitized at a sampling rate of 256 Hz band pass filtered at 0 5 60 Hz and recorded simultaneously with the same computer using BBT proprietary software Experiment 1 Noise influence of a DC brush motor In this section both EEG systems are exposed to electromagnetic capacitive or inductive noise from a DC brush motor as these types of motors were as specified in D2 2 Experimental protocol Experiments were carried out in a real scenario environment with ambient light and without any particular restriction on the background noise and luminance Subjects were seated in a comfortable chair approximately 100 cm from a LCD monitor which displayed the tasks instructions A DC brush motor i e PITTMAN 46 D3 2 Physical Physiological sensing devices Rev 1 0 GM9236E 349 R1 12 VCD 500 CPR was used to analyze its influence in EEG recording Motor was working with constant velocity at 17 motor revolutions per second RPS and it was located at two different positions from the subject s head respectively distance _1 0 5 m and distance 2 1 m DC BRUSH MOTOR Figure 31 Subject during experimental procedure EEG was recorded during relaxing task with noise influence produced of DC brush motor The experimental design is shown in Figure 32 with three conditions distance_1 distance_2 and motor off Each condition was recorded for 60 trials where after each co
76. he minimum and maximum value and the unit for all the individual sensor data published by the HSS Controller As an indication the table also provides the resolution and accuracy of each sensor The missing information in this table as well as the specification of the EEG data will be added in deliverable 3 4 In the event of missing sensor data or invalid sensor data the HSS front end will not output the corresponding data on the Sensor_Data_HSS topic A common strategy for handling these cases should be defined for all CORBYS Sensor data topics Solutions such as outputting zero data or NaN Not a Number values in 34 D3 2 Physical Physiological sensing devices Rev 1 0 combination with some flags could be used to keep populating the topics at the expected rate while clearly marking the data as abnormal 2 5 8 2 HSS Controller status and flags In addition to the Sensor _Data_HSS topic the HSS Controller has to publish heartbeat messages on the general CORBYS Heart_Beat topic These heartbeats are processed by the CORBYS functionality supervisor node in order to check the health of all CORBYS nodes Heartbeat messages include a 32bit flag which can be use to indicate the status of the node At this point no common format for these flags has been defined yet but the HSS Controller will use the flag value to indicate the status of each individual driver node and the presence of valid data at the expected rate for the different sensors 2 5 9
77. heart_rate Int16 skin_temp Int16 activity Int8 posture Int8 battery Int8 See iad 32ms HSS_Chest_ECG timestamp Long sequence Int16 raw_ecg Int16 8 52ms HSS_Chest_IMU timestamp Long sequence Int16 accel Int16 3 gyro Int16 3 Figure 19 Interface of the Chest Unit Driver For testing and debugging purposes a graphical user interface can be connected to the Chest Unit driver This graphical interface communicates with the driver over UDP which allows executing it either locally on the HSS Controller or remotely on a separate computer This interface mainly displays and logs the data coming from the sensor It might also be used to send commands to the sensor for debugging purposes This functionality will be disabled in the deployed HSS Controller in order to avoid conflicting commands reaching the sensors 6 https www sparkfun com products 158 27 D3 2 Physical Physiological sensing devices Rev 1 0 Figure 20 presents the architecture of the Chest Unit driver and shows the three main part of the driver the operating system part the Chest Unit driver and the testing GUI respectively On the operating system side bottom of Figure 20 the Bluetooth module is connected to the HSS Controller via a FTDI USB to TTL serial chip The Linux kernel includes a driver for the FTDI chip and recognizes it is a standard serial port By default the device is automati
78. his experiment lasted approximately 12 minutes Experiment II The subjects were in a standing position with a table on wheels emulating the walking device with the instrumentation located above Figure 42b The reaching point was set to 5 meters away from the start position The experiment consisted in trials of one condition where the subjects walked from the starting point to the reference point Each trial started with a 3 seconds interval of time where the subjects relaxed followed by a variable duration between 10 15 seconds depending on the velocity of the subject in reaching the reference point execution interval where the subjects were walking until the reaching point and then it finished with a 5 seconds rest period The experiment comprised 22 trials and lasted approximately 9 minutes 54 D3 2 Physical Physiological sensing devices Rev 1 0 45 25 0 25 a b Figure 42 a Experimental setup and protocol of Experiment I to look at the targets subjects need to turn the head 25 and 45 b Experimental setup and protocol of Experiment II subjects were asked to walk pushing ahead a table on wheels emulating a walking device D Data Processing Power spectral density PSD was obtained using Welch s method For each trial the PSD was estimated on every Is in the band 1 40 Hz with 0 25Hz resolution and then averaged across trials For each subject the EEG data was filtered by Independent Component A
79. hnologies BCI and HSS are connected to controllers that convert and transmit data to the cognitive framework in the CORBYS system 1 1 Document Scope The present document corresponds to Deliverable 3 2 in the CORBYS project and is the outcome of work in CORBYS Task 3 1 on Sensoring data acquisition fusion and interpretation as well as Task 3 5 Human robot sharing of cognitive information 1 2 Document Structure This document is structured as follows After a brief introduction the Human Sensory System and BCI implementation are described The next chapter focuses on safety followed by a chapter in which relevant requirements as identified in WP2 are listed and discussed The last section concludes the document and also presents a list of future work D3 2 Physical Physiological sensing devices Rev 1 0 1 3 Associated Documents The following documents give additional perspectives for the present work D2 1 Requirements and Specification State of the Art Prioritised End User Requirements Ethical Aspects D2 2 Detailed Specification of the System System Architecture Specification with control and data flow module interdependencies user scenarios etc D3 1 Sensor Network D3 2 Physical Physiological sensing devices 2 Human Sensory System realization EMG interface Cabinet HSS controller PC Rev 1 0 EMG sensor unit IMU sensor unit Environment LINUX ROS
80. ides a numerical value indicating the user s level of attention Progress report on the Signal processing Decoding related part of the requirement has been addressed in the current deliverable Final analysis will be addressed in D3 3 89 D3 2 Physical Physiological sensing devices Rev 1 0 Integration in the BCI architecture will be addressed in Task 3 4 Deliverable D3 4 BCI11 Decoding accuracy It provides a numerical value e g a percentage related to the ability of the BCI subsystem in detecting the attention states Refer to BCI10 90 D3 2 Physical Physiological sensing devices Rev 1 0 6 Conclusions and future work This report describes the work related to development of Human Sensory System HSS and Brain Computer Interface BCI performed in Work Package 3 Sensing systems for assessing dynamic system environments including humans The human sensory system has been realized through development of a lightweight chest belt with sensors at chest and back of patient measuring physiological parameters during gait rehabilitation training sessions Infrastructure for transmitting sensor data wirelessly with a predictable low latency to a computer synchronizing and time stamping data and interfacing the user interface and cognitive modules has been provided The sensor modules are based on state of the art low power components that are highly integrated and optimized f
81. ill be given and shut down times targeted in D3 4 for the entire CORBYS system Time allocated for the physiological sensor system alone is TBD CCM10 Connection to sensing Sensing network sub system ROS is used for interfacing HSS network sub system should provide pre processed and cognitive modules ROS sensor data for cognitive topics are described in the HSS modules in appropriate time Controller sections rate SIREF2 Sub systems to be The documentation required This document together with D3 4 integrated must be accompanied by sufficient in order to integrate sub system components into a CORBYS Users manual and source code will be sufficient 84 D3 2 Physical Physiological sensing devices Rev 1 0 documentation complete system such as documentation for sub system Functional specification integration Mechanical design drawings User manual Source code Interface definitions Installation guidelines CTREF Conformance testing test Test plans shall have unique HW replaceable components will 7 protocol design definition of test objects be marked with serial number physical components shall be SW replaceable components will uniquely marked and software be marked with ID and revision shall have correct version numbering It shall further contain information about test site test date and test personnel The test protocol when feasible will be designed with the following informa
82. ion of movements with attention or distraction to the motor task during robot assisted passive movements of the upper limb International Conference of the IEEE Engineering in Medicine and Biology Society San Diego USA 2012 Costa D Vitti M De Oliveira Tosello D Electromyographic study of the sternocleidomastoid muscle in head movements Electromyogr Clin Neurophysiol 1990 93 D3 2 Physical Physiological sensing devices Rev 1 0 Croft RJ Barry RJ Removal of ocular artifact from the EEG a review Neurophysiol Clin 2000 30 1 5 19 Dyson M F Sepulveda J Q Gan Localisation of cognitive tasks used in EEG based BCIs Clinical Neurophysiology Volume 121 Issue 9 September 2010 Gargiulo G Calvo RA Bifulco P Cesarelli M Jin C Mohamed A van Schaik A 2010 A new EEG recording system for passive dry electrodes Clin Neurophysiol 121 5 686 93 Gomez J Aguilar M Horna E and Minguez J Quantification of Event Related Desynchronization Synchronization at Low Frequencies in a Semantic Memory Task International Conference of the IEEE Engineering in Medicine and Biology Society San Diego USA 2012 Goncharova II McFarland DJ Vaughan TM Wolpaw JR EMG contamination of EEG spectral and topographical characteristics Clin Neurophysiol 2003 Graimann B Huggins J E Levine S P and Pfurtscheller G Visualization of significant ERD ERS patterns in multichannel EEG and ECoG data Clinical neurophysiolog
83. ireless data Cognitive modules should Body validate sensor data to be Sensors discussed with cognitive partners measurement Defect sensor no sensor data HSS controller 7 3 HSS controller reports errors failure incorrect sensor to cognitive detects to FS which handle detected connection modules inoperable errors according to FS missing sensor specification Cognitive data modules should handle incomplete datasets measurement Defect sensor or incorrect sensor Therapist detect 7 3 2 0 failure incorrect sensor data to cognitive in pre training location during modules session setup procedure Therapist procedure to be developed also included in safety document 74 D3 2 Physical Physiological sensing devices Rev 1 0 measurement Defect sensor or incorrect sensor HSS controller 3 6 failure incorrect sensor data to cognitive detects out of location training modules range sensor session data Cognitive modules should validate sensor data to be discussed with cognitive partners measurement Sensor module no sensor data HSS controller 6 1 Status on sensor modules is failure run out of power to cognitive monitor sent via ROS heartbeat to modules remaining FS FS must handle any battery capacity detected errors according to FS specification Cognitive modules should handle incomplete datasets biocompatibility irritations patient Patient 5
84. ith respect to body movements 2 2 1 2 Raw accelerometer data 13 D3 2 Physical Physiological sensing devices Rev 1 0 Raw accelerometer data has been provided and tested before the use in CORBYS but then at a lower rate of 300 milliseconds The rate is increased to 52 milliseconds for CORBYS as requested by the cognitive partners to be able to detect gait movements A regression test of the accelerometer functionality with activity and posture algorithms has been performed The new combined message with gyroscope and accelerometer data has been tested towards the Java application at the PC Visual inspection of the graphs has been done as a check of data usefulness with respect to body movements 2 2 1 3 Timestamping The enhanced timestamp resolution has been tested by analysing log files created at the computer side The information has been compared to the known sample rate for the different sensors No deviations have been found 2 2 1 4 Bluetooth delay variation Test made using Bluetooth communication between Chest unit and a computer showed that the link introduced a delay variation in the range 0 250msec To compensate for this a timestamp is attached to each measurement The chest unit has a local timer used to add timestamp before sending to the computer This solution gives an accuracy of relative timing between samples In order to have an absolute accuracy the local timer needs to be synchronized with the computer clock The cur
85. ivity desynchronizes in the motion condition as it is the typical response in a motor behaviour Pfurtscheller and Lopes da Silva 1999 Target 25 Target 25 Target 45 Target 45 Relaxation 5 Hz Figure 44 This figure displays for subject 3 the logarithmic power spectral density of the EEG signal recorded during relaxation black target 25 green target 25 red target 45 blue and target 45 cyan Taraet 25 _ Target 25 4 5 Hz Figure 45 Average logarithmic power spectral density of the EEG signal recorded during relaxation black target 25 green target 25 red target 45 blue and target 45 cyan 57 D3 2 Physical Physiological sensing devices Rev 1 0 For each subject the ICA decomposition was computed on the EEG concatenated of the five conditions i e 0 targets 25 and 45 where the size of the EEG data satisfies the condition of the minimum amount of data needed to obtain ICA good performance Groppe et al 2009 The ICA algorithm decomposed the EEG into spatially fixed and temporally independent components ICs The components that contained artifacts were eliminated and then the remaining components were reprojected back to the sensor space Jung et al 1998 ol im a 7 n fn fe fe f
86. larly the Human Sensory System may be used for providing physiological data for CORBYS Demonstrator II the Reconnaissance Robot The infrastructure for transmitting sensor data to the cognitive framework has been developed Sensors are transmitting sensor data wirelessly with a predictable low latency to a computer The computer is running controller software that is synchronizing and time stamping sensor data before data is provided to the cognitive modules therapist and engineering user interfaces through the general purpose network The sensor modules are based on state of the art components that are highly integrated and optimized for long term wireless physiological monitoring Effort has been put into compact integrated design and low power consumption enabling long term physiological monitoring D3 2 Physical Physiological sensing devices Rev 1 0 HSS BCI EEG sensors Intention of motion Attention to motion Detection of errors Executive and Cognitive Control Cognitive Framework Physiological sensors SOIAA SAWBB Heart rate EMG Movement Humidity Skin temp Figure 1 Human sensors interaction with other modules in CORBYS Demonstrator I Figure 1 illustrates how the Human Sensory System HSS and Brain Computer Interface BCI interact with the cognitive framework and executive and cognitive control part of CORBYS Demonstrator I The cognitive framework composed of the Situation Awareness Blackboard SAWBB
87. ld to the pedals by an elastic strip Subjects were instructed to completely relax the limbs and to avoid voluntary leg movements during the experiment The mini bike and the experimental setup are displayed in Figure 53 The experiment consisted of trials of two conditions with attention or distraction to the motor task In the first condition subjects focused their attention to the legs during the execution time while in the second one the subjects were instructed to perform mental algebraic computations as a distractor to the passive leg movement Johansen Berg et Matthews 2002 Antelis et al 2012 The algebraic computations were subtractions starting from a 3 digits voluntarily selected number and subtracting a one digit figure voluntarily selected Each trial started with a 5 seconds interval of time where the subjects relaxed minimizing movements and blinking while they were informed of the condition attention or distraction they had to 66 D3 2 Physical Physiological sensing devices Rev 1 0 perform the bike is stopped followed by a 5 seconds execution time where the subject was mentally performing the task bike is on and it finished with a 5 seconds rest time where subjects were allowed to perform minimal movements and blinking bike is stopped Figure 54 shows the structure of one trial The experiment consisted in 3 series of 30 trials with 15 trials of each condition presented in random order There was a rest period of 1 min
88. le one set in use while the other set is charging Charging is done in the offline charging station The charging time is less than 2 hours and the following operational time is more than 2 hours exact numbers will be provided in D3 4 The charge level can be monitored over Bluetooth and shown at the therapist GUI During the training session only the charge level for the sensors in use are shown Between training sessions the charge level for all sensors can be shown Location of the charger will be decided when the EMG system is specified in D3 4 the goal is to have a common location for charging of all sensors There are three alternatives for powering the charger e The HUB is powered from the HSS controller computer USB interface Location has to be on the mobile platform e The HUB is powered from an external 24VDC power supply Location has to be on the mobile platform 39 D3 2 Physical Physiological sensing devices Rev 1 0 The HUB is powered from an external 220VAC power supply Location has to be on a table somewhere near CORBYS Demonstrator I U paa OA M Chest unit Chest unit Bl u eto oth BI u eto oth aiueroot aiueroor Figure 27 HSS offline charging station 40 D3 2 Physical Physiological sensing devices Rev 1 0 3 Brain Computer Interface BCI CORBYS project focuses on a robotic system that have a symbiotic relationship with humans One of Its objective concerns
89. ling Thus the inner 44 D3 2 Physical Physiological sensing devices Rev 1 0 wire of the input cable is not affected by any capacitive coupling which means that there will be no artifact when the cables are moving On the other hand g Tec provides no information about the specific cable shielding used for their systems In CORBYS it is expected to find artifacts from wire movements due to the user locomotion and thus the TMSi wires seem to be more appropriate as the movement of the cables will be difficult to avoid notice that this statement is mainly due to the lack of information about the g Tec wires Amplifier The amplifier includes the following functions filtering amplifying and signal conversion The next table resumes the technical specifications of the commercial amplifiers related to the signal quality such as input impedance the common mode rejection ratio CMRR Table 4 Technical specifications of the Porti 16 and g USBamp amplifiers Porti 16 g USBamp Input referred noise lt 2 uVpp Fs 128 Hz lt 0 3 uV RMS 0 1 10Hz Input impedance gt 102 Q gt 10 Q CMRR gt 100 dB gt 100 dB Connector Micro coax active shielding 1 mm 2 pin touch proof Special features Active guarded shielded leads 50 60Hz selectable hardware notch and electrodes filter One of the most relevant amplifier parameters is the input impedance as it alleviates the need of a low impedance contact to the skin High input impedance in
90. lues of the execution mode input are Training motion Training feedback Training attention Training artefacts Decoding motion Decoding feedback Decoding attention Decoding motion amp feedback Decoding motion amp attention Decoding feedback amp attention Decoding motion amp feedback amp attention Stop The independence of the cognitive related tasks allows their simultaneous execution in the decoding process i e Decoding motion amp feedback Decoding attention amp feedback etc A stop input value has been also added to Signal processing Decoding part of the requirement will be addressed in Task 3 3 and 3 5 Deliverable D3 3 Integration in the BCI architecture will be addressed in Task 3 4 Deliverable D3 4 87 D3 2 Physical Physiological sensing devices Rev 1 0 allow the interruption of the running processes Motion feedback and attention are the abbreviations for intention of legs motion feedback error related potential and attention states respectively BCI2 Configuration File Values of the optional parameters a default To be addressed in Task setting is provided 3 4 Deliverable D3 4 A list of possible optional parameters is available below Number of electrodes to be used Sampling rate of the EEG signal Decoders parameters A complete list will be provided depending on the results of the ongoing CORBYS research BCI3 Raw EEG Electro
91. modules is done through the CORBYS General Purpose Network GPN The CORBYS GPN is a standard TCP IP network over Ethernet To ease the integration of the different module the CORBYS system uses the Robot Operating System ROS framework The ROS framework runs on top of the operating system and TCP IP network and provides standard ways to describe and publish both services and data in a distributed network ROS Robot Operating System provides libraries and tools to help software developers create robot applications It provides hardware abstraction device drivers libraries visualizers message passing package management and more ROS is licensed under an open source BSD license ROS facilitates communication between processes running on the same or on different machines The communication between processes can take place either by a blackboard style paradigm with publishers and subscribers called Topics or in a server client mode called Services A ROS node is a process that performs computation and runs in a separate process using a socket IP address and port Nodes are combined together into a graph and communicate with one another using streaming topics remote procedure call RPC services and the Parameter Server These nodes are meant to operate at a fine grained scale a robot control system will usually comprise many nodes For example one node controls a laser range finder one node controls the robot s wheel motors one node perfo
92. movement artifacts from during locomotion the description of temporal and spectral characteristics of the main contamination introduced in the EEG by locomotion and head movements have been described ICA technique was applied to remove the artifacts contamination generated showing its feasibility within a simulation of the CORBYS rehabilitation user scenario Due to the importance of the decoder specific analysis in the removing process of gait related artifacts further evaluations are needed once the CORBYS decoders have been implemented Section 3 3 Detection of attention during assisted passive leg motion shows how the attention modulates brain activity during passive leg movements An off line classifier has been developed to differentiate between attention and non attention to the motor task Results showed the feasibility to distinguish between both conditions above the chance level for all the subjects This is the first step in the design a BCI system to decode passive lower limb movements with attention and non attention to the motor task Future work will consider the automation of the features selection process and build an online classifier to assess the feasibility of decoding attention during real time motion The remaining work for the Brain Computer Interface will be presented mainly in D3 3 month 38 and D3 4 month 26 This will include Design of a Brain Computer Interface software architecture Analysis Design 92 D3
93. nalysis ICA algorithm FastICA based on Hyv rinen et Oja 2000 to eliminate the components from blinking and EMG artifacts FastICA algorithm is based on a fixed point iterative method that maximizes the non Gaussianity as a measure of statistical independence Hyvarinen et Oja 2000 The assumption is that the number of sources is the same as the number of electrodes i e 32 Each component was reprojected back to the sensor space time frequency representations of these reprojections with the associated spatial maps were visually inspected to assess whether the component was artifacted The components free from artifacts were reprojected back to the sensor space to obtain an artifact free EEG For source localization Standardised Low Resolution Brain Electromagnetic Tomography sLORETA Pascual Marqui 2002 was employed sLORETA is a linear method to compute from EEG data a statistical map that gives the location of neural generators within the brain The source localization was used to assess whether the filtered EEG is free of artifacts as follows the source activity of the pre filtered and post filtered EEG was computed and checked whatever it corresponds to the motor cortex Notice that if the artifacts do not contaminate the EEG then the motor cortex will be one of the neural generators This strategy was used only in the first experiment as the walking behaviour does not involve the motor cortex in a large extent since the majority of th
94. ndition the subject has 2 minutes of rest Each trial was composed of a 3 seconds resting time and 5 seconds baseline executing task The subject was asked to avoid blinking during the baseline condition Motor on distance_1 REST Motor on distance_2 REST Motor OFF Figure 32 Time diagram of protocol in the experimental procedure Data processing and results For each trial the EEG data from the baseline were visually inspected and one second epochs were discarded if a physiological artifact was identified EEG power spectrum was calculated by a sliding window periodogram of one second with 30 ms of overlapping and then averaged To compute the periodogram a 1s hamming window was used with a resolution of 0 25 Hz 1024 points using zero padding and power line is notch filtered at 50 Hz For each condition the power spectrum density PSD was averaged across channels and then averaged across subjects Figure 33 shows the PSDs for each condition and amplifier The only visible difference is around 56 Hz where it seems to be an artifact as up to our knowledge this abnormal activity cannot be attributed to any known neural activity This artifact appears in all three conditions of the g Tec system even when the motor is OFF and it appears only in one condition of the TMSi system i e motor ON This suggests that the nature 47 D3 2 Physical Physiological sensing devices Rev 1 0 of the artifact is not due to the DC motor The rest of
95. ned to allow testing its components in isolations and to ease the testing of its integration in the CORBYS demonstrator This section provides a short description of the different testing phases for the HSS Controller The specification of the tests and results will be provided in D3 4 2 5 11 1Unit tests of the driver modules All the HSS Controller sensor drivers are independent processes connected to the sensors on one side and outputting sensor data on the HSS Controller internal ROS topics For each sensor a testing application can be connected at the lowest level of the driver The debugging GUI implements it own version of the sensor driver Inconsistencies and faults can be detected by logging the sensor data produced by the HSS Controller driver and comparing it to the data produced by the debugging GUIs 2 5 11 2Unit tests of HSS Front End The front end of the HSS controller front end can be tested in isolation of the sensor drivers by using ROS bags in order to capture data sequences on the internal HSS ROS topic and playing these data to the HSS Controller front end The output data of the front end can be compared to the data played on the internal topics in order to detect potential malfunctions 2 5 11 3 Robustness tests An important aspect of the HSS controller is its robustness with respect to potential failures Special attention will be made at testing the ability of the HSS Controller to detect failures and recover from them Te
96. nt implementation Most requirements of the Human Sensory System are fulfilled in this deliverable the remaining requirements will be fulfilled in D3 4 Regarding BCI most requirements will be fulfilled in D3 3 and D3 4 The Human Sensory System will be extended in Deliverable D3 4 in month 26 The main addition will be EMG sensors to measure muscle activity but improvements in the sensors and the controller will also be added The Demonstrator I development will at that time be ready for early integration and hence the Human Sensory System will be enhanced to fit with the cognitive modules and the engineering and therapist user interfaces Options to simulate the Human Sensory System from recorded data will be added so to ease system integration The analysis Brain Computer Interface submodule will be advanced in D3 3 month 38 and D3 4 month 26 The main addition will include the design of BCI software architecture addressing also network integration and synchronization issues and the implementation of the decoding CORBYS related decoding algorithms D3 2 Physical Physiological sensing devices Rev 1 0 1 Introduction The focus of CORBYS is on robotic systems that have a symbiotic relationship with humans Such robotic systems have to cope with highly dynamic environments as humans are demanding curious and often act unpredictably CORBYS will design and implement a cognitive robot control architecture that allows the integration of 1
97. ntaining 2200 message packets send over a period of two minutes o Delay variation v Figure 10 Test 4 Person walking The graph shows the distribution of delay for 2200 packets X axis Delay in milliseconds Y axis Number of packets 2 2 1 5 Bluetooth transfer rate When enabling raw data at higher data rates there were a concern whether the data were above the practical Bluetooth transfer rate From previous projects SINTEF had experienced large variations related to CPU load on the computer Different brands of Bluetooth dongles had also shown different performance We chose to use a dongle from TARGUS and a dongle from Roving The TARGUS dongle has the Bluetooth protocol stack running on the computer while Roving has an external protocol stack running on a processor inside the dongle Running an external protocol stack offloads the computer and makes the communication more robust with respect to CPU load on the computer A special test program was made generating a test sequence that could detect lost and duplicated data The tests showed that it was possible to run at data rates of 7500 bytes sec but data loss was frequent It also showed that the Roving dongle handled much higher data rates than the TARGUS dongle The goal to transfer 2000 bytes sec was achieved by both dongles Test duration Rate bytes sec Sequence Device Loss sequence sec Handshake Comment sec length bytes 15 15 D3 2 Physical
98. ocesses can contain a number of undesired components for example 50 Hz mains interference electrode offset potential drift in offset potential or fluctuations caused by mechanical influences such as movements An EEG sensor is tested concerning its technical specifications such as active passive level of noise DC behaviour and variability frequency response impedance and its stability weight wearability and sensor material However the main challenge to EEG electrodes is to get a good low impedance contact to the skin for comparison about different electrode technologies refer to CORBYS Deliverable D2 1 Section 15 State of the Art in Non Invasive Brain Computer Interface The other point need to be concerned is whether the contact of the electrode is affect by the locomotion of the human From the technical specifications there is not a clear reason to state which of these electrodes is better suited for this purpose Wires Both devices have shielded wires which send the EEG signals that are measured at the electrode into the amplifier The main advantage of the shielded cable is that movements of the cable and environmental noise do not influence the signals in the cable Thus this source of noise does not affect the recorded EEG signals in the CORBYS scenario In TMSi equipment the cables are covered with active shielding so that the signal is immune to the cable movement artifacts and to mains interference i e 50 or 60 Hz coup
99. omponent Analysis ICA technique In the present study two experiments were conducted the first one addresses lateral head movement and the second one studied the walking movement with a walker device The relevance for the CORBYS project is due to the presence of linear and angular head movement during locomotion Hirasaki et al 1999 the second one emulates the walking condition of the 1 CORBYS demonstrator Although in the clinical EEG the artifacts are addressed with generalist filters in the brain 52 D3 2 Physical Physiological sensing devices Rev 1 0 computer interface technology community they are addressed with filter design for each user application This is the reason why this study analyzes both average and individual artifacts 3 2 2 Methods A Subjects Data were collected from 4 healthy subjects 3 women and one man 23 25 3 59 years None of them had a history of a neurological psychiatric disorder or was under chronic medication The group included 3 rights and one left hand dominant people The participants were duly informed about the experiment before they signed the consent form B Data Collection EEG was recorded using two g Tec amplifiers with 32 active electrodes The electrodes were placed at FP1 FP2 AF3 AF4 F7 F3 Fz F4 F8 FC3 FC1 FCz FC2 FC4 T7 C3 C1 Cz C2 C4 T8 CP3 CPz CP4 P7 P3 Pz P4 P8 O1 Oz and O2 according to the international 10 10 system Figure 41a The gro
100. or long term wireless physiological monitoring The Chest Belt can easily be fit to and removed from the patient at start and end of a training session and will not cause any discomfort but still provide relevant physiological sensor data to the cognitive modules of the CORBYS Demonstrator I Patient safety is handled by using a wireless battery powered sensor modules The remaining work for the Human Sensory System will be presented in D3 4 in month 26 This will include EMG system Description of the sensor modules Time to mount sensors on patient Driver component for the EMG sensors Complete testing description Chest unit IMU unit HSS controller SW Complete specification of charging of wireless devices Operational and charging time for all units Localization of charging station ROS network integration Complete set of test data exploitable for simulating the execution of the HSS Controller without actual sensors Front end implementation fully compliant with the final CORBY ROS Guideline Implementation of the training initialization process and supporting GUI Integration with Task Manager and Executive Supervisor Synchronization issues Implementation of an appropriate mechanism for time synchronization with the wireless sensor either over the air or through serial communication on the charging station Evaluation of latency and time stamping offsets at the different layers of the HSS Controll
101. ovided that the IMU Sensor Unit It provides a specific API to the layer above which allow connecting to IMU Sensor units subscribing to the different sensors it contains and collecting sensor data e The ROS publisher is the top layer of the stack It subscribes to data from the two IMU Units to use for a training session and forwards the sensor data on the HSS Controller as ROS topics The test and engineer interface on the top right corner of Figure 22 is a Java application which implements its own driver and GUI for the IMU Unit sensors It can be connected locally or remotely to the HSS Controller IMU Sensor driver in order to visualize the low level communications with the BLED112 dongle Since the debug application implements its own driver for the Bluegiga protocol and IMU Sensor comparing the data provided by the debug application and the data forwarded on the ROS topic allow validating the implementation of the HSS Controller driver 2 5 7 EMG Sensors Driver Not implemented as part of D3 2 Once the EMG sensor will be selected a specific driver will be implemented to provide the sensor data internal HSS ROS topics in the same way as other sensors 2 5 8 HSS Controller Front End and ROS Interface The role of the HSS front end is to collect all the data coming from the sensors and provide this data on the CORBYS general purpose network using the SensorData HSS ROS topic The sensor data on this topic is represented using a common CORBY
102. physiological Wireless sensors are connected to sensor measurement system General Purpose Network GPN with the main CORBYS for interfacing cognitive modules cognitive robot control and control system HSS13 Mains requirements It must be anticipated that HSS Controller will be running at some of the measurement 24VDC equipment will require 220V 50Hz HSS14_ Sensor network architecture The project needs to compile a Worst case values will be requirements summary of all sensors and specified as part of the GPN actuators with detailed specifications in D3 4 operation characteristics and worst case values in order to specify the total sensor network architecture HSS15 Online access to past Online access to past and Human sensory system values can rehabilitation sessions possibly ongoing therapy be recorded at the GPN and will sessions implies a software be described in D3 4 architecture solution as well as probably a WiFi node on the CORBYS system HSS16 CORBYS system A shared understanding of All sensor data will be time intermittence delay and signal propagation will have stamped D3 4 will discuss synchronisation to be reached between the synchronization of computers on requirements for sensors partners GPN and actuators HSS sensor data delays are discussed in section 2 1 2 HSS17 Number of physiological The detailed number is TBD Implemented There will be one Chest unit at front and one IMU sensor unit at the back of
103. rect session startup and during training placement sessions This will be described as part of D7 4 HSS34 Physiological sensor Sensors should not cause The Chest belt has been tested biocompatibility issues irritations inflamatic with patients and no responses or pain during the biocompatibility issues has been designated duration of observed CORBYS rehabilitation EMG sensors will be covered in sessions D3 4 Sensors can be temporarily attached to the patient using e g medical grade adhesive tape HSS35 Physiological sensor EC Medical device standard This is an optional requirement biocompatibility issues biocompatibility testing of all and no testing towards EC materials interfacing the Medical device standard patient biocompatibility has been done However informal testing has been dine see HSS34 HSS36 Physiological sensor Physiological sensor Cleaning and replacement of Chest hygienic issues interfacing the patient s skin Belt will be described in D7 4 directly should be possible to clean or replace from patient to patient Single use probes Multiple use probes that have smooth surfaces and that can be cleaned in appropriate detergents HSS37 Time required to mount or For a trained user it should be Chest unit and IMU sensor unit dismount all physiological possible to mount all sensors should be possible to mount within sensors within the maximum time one minute required for the entire start up EMG mounting time w
104. rent synchronization algorithm does a single handshake during connect This algorithm does not compensate for the delay in the messages used The result is an uncertainty in the absolute accuracy in the range 0 250ms Test datasets with sensor data from the Chest Unit have been recorded using the Java monitoring application Based on these datasets the delay variation has been analysed The figure below shows delay variation for a person standing This gives a static distance for the communication path This is also expected to be the case for the demonstrator I where the patient is attached to the powered orthosis and the Bluetooth interface to the HSS controller is placed on the mobile platform cabinet The delay variation is given in milliseconds one line per 10 milliseconds The graph is based on a dataset containing 1500 message packets send over a period of one minute 70 60 50 MANN O 20 40 60 80 100 120 140 160 180 200 220 240 260 280 Delay variation v Figure 9 Test 1 Person standing The graph shows the distribution of delay for 1500 packets X axis Delay in milliseconds Y axis Number of packets 14 D3 2 Physical Physiological sensing devices Rev 1 0 The next figure shows a person walking in a corridor The person is starting and stopping The computer is not moved with the patient Here some longer delays show up This is probably due to some retransmissions The graph is based on a dataset co
105. rms localization one node performs path planning one node provide a graphical view of the system and so on The use of nodes in ROS provides several benefits to the overall system There is additional fault tolerance as crashes are isolated to individual nodes Code complexity is reduced in comparison to monolithic systems Implementation details are also well hidden as the nodes expose a minimal API to the rest of the graph and alternate implementations even in other programming languages can easily be substituted The ROS nodes of the CORBYS system are all running Ubuntu Linux version 10 04LTS 32bits and the Electric version of ROS These choices are expected to provide a stable platform for the development and use of the CORBYS system A set of common CORBYS ROS messages and topics have been defined and allow for the integration between the HSS Controller and the cognitive modules 2 5 3 HSS Controller setup and operating system Figure 17 details the installation steps for setting up the HSS controller The first step is a standard installation of Ubuntu Linux A single user account is setup on the HSS Controller Login hsscontroller Password corbys The next installation step installs some required packages and security updates from Ubuntu repositories http www ros org wiki 24 D3 2 Physical Physiological sensing devices Rev 1 0 et AEAEE This script describes the steps to setup the HSS Controlle HIEP yr ever erer er
106. ry powered device using wireless communication This ensures patient safety with respect to electrical shock The unit has no electrical contact with the patient 4 3 EMG units The EMG sensors will be covered in D3 4 4 4 HSS controller computer The HSS Controller will be located on the mobile platform for CORBYS demonstrator I The controller is powered from the mobile platform batteries 24VDC It will not be physically connected to the patient Electrical safety insulation towards the grid power supply 220VAC is handled by the Central Power System on the mobile platform 4 5 Offline charger The sensor unit shall not be charged while attached to the patient This means that the offline charger will not be physically connected to the patient The charger will be powered from the HSS controller or by use of a separate power supply 72 D3 2 Physical Physiological sensing devices Rev 1 0 4 6 Safety analysis As part of the safety work for demonstrator I FMEA safety analysis has been performed for all components of the system as reported in Deliverable 7 1 The analysis for the HSS is shown in the figure below It shows that there are no high risks connected to the HSS The medium risks will be addressed at the system level risk analysis and handled by other modules Incomplete datasets due to failing or missing sensors must be handled by receivers of data from HSS all sensor data shall be defined with a valid range of values To avoid
107. scenssesseecessbscadesssececsesaecesuseccsesee Hl EXECUTIVE SUMMARY veetee sk eseese toeristi riea oeae tist eisoes eseeto tiaa oaase otiosi oeseri sebis deoti ioes 1 1 INTRODUC TION a a O eases a Sa a aa Too araa e a oora aaea tesir as sse 4 1 1 DOCUMENT SCOPE anas aia a a E Ea E sa e a in svecdveuteck as i E a Ee ai asa ai 4 1 2 DOCUMENT STRUCTURE venaya a a a A A a O EENET 4 1 3 ASSOCIATED BOT CIN IANH EA S A O OE A 5 2 HUMAN SENSORY SYSTEM REALIZATION ccccccssssccccsssccccsssecccsssccccecsecccesscccceseseccesesecceseseceeseseceees 6 2 1 TAS SSARGHITEGTURE EOR EEEE AEE E E tanta EE EO E SEE O TEN EEA TOES 6 X2 CHEST UNI nre a a a a e a r a n a ade a a a a r a i 11 2 3 IMU SENSOR UNIT Tione duck ieri ieda r e a aae aa aea edie ne Sae iai s eda e ana Esne iaoi cies tee 17 2 4 EMG SENSOR Sreo agaa aaa ag aa a aa g a aa gE ie 22 2 5 HUMAN SENSORY SYSTEM CONTROLLER orestes eriei eiiie dieu everseevdavagseveacasvecdacueneiedvadsevsadaeueds 23 2 6 OFFLINE CHARGING STATION FOR WIRELESS SENSORS ccccccccsccscscceeceeseeceesceceeeeeeeeeeeeeeeeeeeeeeeeeeeeeseeeseseseseseneeens 39 3 BRAIN COMPUTER INTERFACE BC ccccccssssssssssssssssssssssssssssssesssssssssssssssssssssssssssssssssssssssseseees 41 3 1 1 EVALUATION OF EEG ACQUISITION SYSTEMS THAT COULD REDUCE THE NOISE LEVEL esssssscceeessssteceeeeecenees 43 3 2 ANALYSIS AND REMOVAL OF MOVEMENTS ARTIFACTS DURING LOCOMOTION cscseseseeeeeeerereeerereeeeereeeeees 52 3 3 DETECTI
108. sing the proprietary Bit amp Brain Technologies BBT software 53 D3 2 Physical Physiological sensing devices Rev 1 0 C Experimental Procedure The experimental procedure comprised a baseline recording and two experiments It lasted about 30 minutes Baseline Subjects were seated in a comfortable position with eyes open and were asked to try to minimize movements and blinking for an interval of 5 minutes Experiment I The subjects were in a standing position in front of a fixation cross placed in the center of a wall 2 meters away from them 0 initial position straight ahead and four targets located at an angle of 25 and 45 with the 0 Figure 42a The experiment consisted of successive trials of four different conditions that correspond to movements to align the head with the targets i e target 25 target 25 target 45 and target 45 and return to the initial position Each trial started with a 3 seconds interval of time where the subjects relaxed in the initial position and minimized movements and blinking followed by a variable depending on the velocity of the subject head movement execution interval where the subjects moved the head from the initial position to the target specified and move it back and then finished with a 2 seconds rest time where subjects were allowed to minimally move and blink The experiment had 10 trials for each condition with a rest period of 30 seconds between conditions T
109. sting will include scenarios with a high occurrence probability such as e Loss of connection with a sensor e Crash of a driver and or of the radio adapter Poor wireless connection leading to packet loss and or high latency e Failure of a sensor and or abnormal sensor values 2 5 11 4Q0S and extra functional properties tests Specific tests will be performed to evaluate typical data latency and time stamping offset in order to ensure that the CORBYS requirements are met The time stamping offset mostly depends on the time synchronization between the HSS Controller and the sensor devices Latency can be introduces at every stage of the sensor data processing and distribution It is expected that the main source of latency is the wireless communications between the sensor units and the HSS controller but latency introduced by the HSS internal communication though ROS topics the processing time of the drivers and front end as well as distribution of the data on the CORBYS ROS topic will be evaluated 2 5 11 5Integration tests 37 D3 2 Physical Physiological sensing devices Rev 1 0 To test the integration of data from the HSS controller with the reset of the CORBYS demonstrator a set of front end datasets will be provided as ROS bags which can be replayed to simulate the presence of the HSS Controller on the ROS network Multiple data sets will be provided for the different phases of the execution of the HSS controller Also to test t
110. sults of the ongoing 3 4 Deliverable D3 4 CORBYS research e g laptop netbook personal digital assistant etc BCISW10_ EEG system montage A fast and easy EEG system montage cap and The choice of the CORBYS electrodes placement is required Associated EEG hardware has been with these requirements the most appropriate addressed in the current system will be used deliverable section 3 1 The EEG system chose is water based log time and uncomfortable preparation issues have been solved BCISW11 EEG system portability A reduced size and weight EEG system is The choice of the CORBYS required Associated with these requirements EEG hardware has been addressed in the current deliverable section 3 1 Even if the priority in the EEG hardware selection has been given to the performance of the EEG 86 D3 2 Physical Physiological sensing devices Rev 1 0 system in a noisy scenario this requirement has been accomplished BCI1 Execution Mode Indicates which operation between training and decoding is going to be used BCI requires a machine free training stage before users can work the technology The training process modifies some internal parameters that successively the decoding process uses This procedure must be observed for each cognitive related task is planned to be used A training phase is also needed for the artefacts removal processing Training artefacts The possible va
111. ta band power and in the third one both previous features were used together Bands and channels were visually and manually selected for each subject to calculate the features The final results are shown in Table 6 Subject Accuracy alpha Accuracy beta Accuracy alphat beta 1 75 00 63 33 81 67 2 62 86 82 86 80 00 3 57 14 70 00 64 29 4 61 67 70 00 66 67 5 76 67 75 00 81 67 Table 6 Results for each band of each subject 100 T T T T T T Hi alpha 90 Theta 7 Wi alpha beta 80 70 4 si l oenen 50 4 Accuracy 40 4 30 4 10 4 Subject 1 Subject 2 Subject3 Subject 4 Subject 5 Mean Figure 57 Results for each band of each subject dotted line chance level according to Miiller Putz et al 2008 p lt 0 05 As shown in Table 6 the classifier performance for all subjects is over the chance level which is situated in 62 5 according to Miiller Putz et al 2008 Beta band features achieved higher accuracy for subjects 2 3 and 4 while for subjects 1 and 5 the combination of both bands features was the better election The mean accuracy is 77 24 selecting the best result of each subject showing that it is possible to build a classifier over the chance level and thus showing the feasibility of building an off line system to differentiate the two conditions 3 3 4 Discussion 70 D3 2 Physical Physiological sensing devices Rev 1 0 Cognitive processes su
112. ted in an internal SINTEF project called TRALE Liverud et all 2012 however most of the firmware development and device manufacturing were done in the CORBYS project The following sections give a technical description of the implementation of the sensor unit including basic application and user information The IMU sensor unit shown in Figure 12 integrates both activity and physiological sensors in the same device A combined 3 axis accelerometer and gyroscope in addition to a magnetometer form an inertial measurement unit IMU Skin temperature is measured by an infrared IR sensor Additionally two IC ports are available for external sensors In CORBYS usage a combined humidity temperature sensor is fitted measuring the humidity at the patient back Power is provided by a rechargeable battery and components are selected for minimum power consumption Wireless communication is by the new Bluetooth Smart 17 D3 2 Physical Physiological sensing devices Rev 1 0 technology previously called Bluetooth Low Energy This is a feature of the Bluetooth 4 0 standard providing low latency low power short range communication Profiles are defined for sensor values like temperature and heart rate allowing Smartphones or other devices to receive data without proprietary drivers The Bluetooth Smart technology is included in the Continua Health Alliance design guidelines Figure 12 The IMU sensor unit without and with enclosure measuring 54
113. tem configurations Reconnaissance Operator The person steering the robot by remote control robot for f Hazardous Area The person that robot follows in a team work on investigation of Investigation of Examination Officer hazardous areas RA Engineer A professional dealing with the CORBYS system based on a need to do os technical maintenance repairs or system configurations CORBYS Domain Knowledge Sensor Fusion Method used to combine multiple independent sensors to extract and refine information not available through single sensors alone Situation Assessment Estimation and prediction of relation among objects in the context of their environment Cognitive Control Capability to process variety of stimuli in parallel to filter those that are the most important for a given task to be executed to create an adequate response in time and to learn new motor actions with minimum assistance Kawamura et al 2008 Human Robot Interaction Neural Plasticity Ability of a robotic system to mutually communicate with humans Ability of neural circuits both in the brain and the spinal cord to reorganise or change function Cognitive Processes Processes responsible for knowledge and awareness they include the processing of experience perception and memory CORBYS Technology Components SAWBB Situation Awareness Blackboard SOIAA Self Organising Informational Anticipatory Architecture
114. tended duration of Anticipation 8 hour sessions See HSS21 continuous usage of physiological sensors HSS23 Intended duration of If CORBYS becomes a See HSS21 continuous usage of community walker gait physiological sensors assistance system usage sessions could last from morning to evening HSS24 Electrical measurement Within Consortium electronic This requirement will be handled system safety systems are acceptable as long in the safety discussion in D7 4 as they are tested and deemed safe for the CORBYS users e g complete user shielding from 220V 50Hz HSS25 Electrical measurement CE Medical device standard See HSS24 system safety electrical safety 82 D3 2 Physical Physiological sensing devices Rev 1 0 HSS26 Electrical measurement system safety Medical device CE approvals on all sensor components For a commercial product after CORBYS See HSS24 HSS27 Sensor systems should not be invasive or excessively obtrusive In vivo implanted sensor systems are not a part of the CORBYS physiological measurement system Sensor concepts probing human fluidic samples blood urine saliva etc are excluded Sensor concepts probing human body openings such as rectal core temperature measurements and breath air gas analysis are excluded Fulfilled in HSS designed HSS28 HSS29 HSS30 Limitations in the range of acceptable users Mounting and removal sensors
115. the CORBYS demonstrator The EEG system that showed to be less affected by electrical noise contamination will be chosen for the CORBYS gait rehabilitation system In robotic related rehabilitation programs it has been suggested that human cognitive processes such as motor intention attention and higher level motivational states play an important role in the success of the therapy Tee et al 2008 In this context the BCI module will online detect cognitive processes of interest for the CORBYS gait rehabilitation system such as intention of leg motion feedback error related potential and attention states Section 3 3 Detection of attention during assisted passive leg motion reports the progress in designing a BCI system to decode passive lower limb movements with attention and non attention to the motor task 41 D3 2 Physical Physiological sensing devices Rev 1 0 BCI Ros Node Inputs Heartbeat Config_Data Outputs EEG dafa EEG_decodings ad hoc link EEG sensor failure Figure 28 Brain Computer Interface submodule overview Figure 28 presents an overview of the BCI submodule It will be integrated in the architecture of the 1 CORBYS demonstrator as a ROS Node task 6 1 Architecture decomposition and definition The inputs required from the BCI i e configuration parameters and system state D2 2 will be read from the ROS Parameter Server its outputs will be sent from the BCI ROS Node using a ROS
116. the PSD is very similar in three conditions and thus there is no other apparent effect of artifacts on the EEG This result is confirmed by the Figure 34 which shows PSDs resulted from both EEG systems in the 3 conditions recorded Figure 35 shows the ratio between the PSDs of motor ON and motor OFF conditions for each subject this ratio eliminates the effect of the different gains of the amplifiers and allows the comparison The results show that there is not apparent difference below 20 Hz that is the range where the difference should be due to the RPS of the motor about 17 Hz Greater difference is present in the range 20 50 Hz it can t be attributed to the DC motors since in the analysis performed in Figure 33 and Figure 346 this is not visible Artifacts above 50 Hz are not due to the motor activity TMS g TEC 25 25 Motor OFF Motor OFF Large distance Large distance Short distance Short distance 20 7 20 7 N N zaspi 315 E E B B B 8 a a a NOD ges T 4 10 a oa 5 5 0 h 0 0 20 40 60 0 20 40 60 Frequency Hz Frequency Hz Figure 33 Average power spectrum density for every conditions motor OFF large distance short distance using TMSi system left and g Tec one right Short distance Large distance Motor OFF Power spectru m uV Hz Power spectru m uV Hz Power spectru m uV Hz o 20 ao 60 o 20 ao 60 o 20 ao 60 Frequency Hz Frequency Hz Frequency Hz Figure 34 Average power spectrum densi
117. the development of a perception system for assessing the physical and mental state of the environment including humans The perception system includes multimodal physiological sensing devices such as the Brain Computer Interface BCI and the Human Sensory System HSS that are important for human motor control and learning The Non invasive Brain Computer Interface using EEG will detect human cognitive information in real time used by the robot cognitive control architecture for perception of the human mental state In particular BCI will decode cognitive processes such as intention of leg motion feedback error related potential and attention states In the CORBYS gait rehabilitation system scenario the subject will be walking assisted by a robotic device In this context the EEG data will be affected by typical artifacts such as EOG and EMG but also electromagnetic artefacts due to the mobile platform i e motors used for the powered orthosis and for the mobile platform s wings and mechanical artefacts associated with head movements and locomotion In section 3 2 Analysis and removal of movements artifacts during locomotion ocular and mechanical artefacts and their removal process based on Independent Component Analysis ICA technique are addressed Section 3 1 Evaluation of EEG acquisition systems that could reduce the noise level focuses on the study of the electromagnetic noise i e noise due to the DC motors and their impact in
118. the patient D3 2 Physical Physiological sensing devices Rev 1 0 Figure 5 Belt with Chest unit sensors and IMU sensors at back 2 1 2 Communication links and timing issues The Human Sensory System is linked to Demonstrator I via the General Purpose Network a TCP IP network This network is defined in WP 3 The various HSS sensors are connected to the HSS controller via Bluetooth wireless technology The Chest unit uses standard Bluetooth communication via SPP protocol Serial Port Profile The IMU sensor unit uses Bluetooth Smart a part of the Bluetooth 4 0 specification formerly named Bluetooth Low Energy for communication this is an emerging low power low latency communication protocol providing defined profiles for various sensor devices All computers on the General Purpose Network GPN will have synchronized clocks the accuracy is expected to be in the range 1 5msec Testing made using Bluetooth communication between the Chest unit and a computer showed that the link introduced a delay variation in the range 0 250msec To compensate for this a timestamp is added to each measurement The chest unit has a local timer and a timestamp is added before sending to the computer This solution has good accuracy for inter sample timing In order to obtain an absolute accuracy the local timer needs to be synchronized with the computer clock The current synchronization algorithm does a single handshake during connection this al
119. the patient One IMU sensor unit will be located at the mobile platform to measure environment humidity 81 D3 2 Physical Physiological sensing devices Rev 1 0 and temperature Postponed Four EMG sensors located at each patient leg HSS18 Number of physiological For ease of use purposes it See HSS17 sensor probes on the will be desirable to combine patients several sensors into single devices thereby reducing the experienced system complexity HSS19 Signal connection of Sensor data signals will be Implemented physiological sensors to the sent through electrical Physiological sensors are CORBYS system wires cables transmitted using Bluetooth to HSS controller HSS controller use wired Ethernet for communication over GPN HSS20 Signal connection of For ease of use purposes See HSS19 physiological sensors to the possibilities to make some CORBYS system sensor units transmit data using wireless communication protocols will be considered HSS21 Intended duration of Anticipation A therapy The chest unit and IMU sensor continuous usage of session will last up to 2 hours unit will have battery capacity for physiological sensors more than 2 hours Final battery capacity measurements will be provided in D3 4 Two sets of units will follow the demonstrator One set for use and another charging Charging time is less than 2 hours There will be one charging station following the demonstrator HSS22 In
120. ties for three conditions recording motor on located 0 5 m far subject s head motor on located 1 m far subject s head and motor 48 D3 2 Physical Physiological sensing devices Rev 1 0 Ratio between long distance and motor off Ratio between short distance and motor off 2 log10 Nolsy spectrum Free noisy spectrum log10 Noisy spectrum Free noisy spectrum 0 10 20 30 40 50 70 10 20 30 40 50 Frequency Hz Frequency Hz Figure 35 Ratio between power spectrum of noise conditions and motor OFF for TMSi blue and g USBamp red systems of each subject Noise condition is considered motor located 1 m left and 0 5 m right from subject s head Experiment 2 Noise influence of an AC electrical engine Experiments were carried in a real scenario environment with ambient light without any particular restriction on the background noise and luminance Subjects were seated in a comfortable chair approximately 60 cm from a LCD monitor which displayed the tasks instructions Close to the subject there is an electric mini bike activated by remote control by the subject From an electrical point of view this device behaves as a solenoid which works in different frequencies depending on the speed bike and for the experimentation the slowest speed motor was used Remote Electric control mini bike Figure 36 Subject during experimental procedure EEG was recorded during relaxing task with noise influence produced by an electric mini bik
121. tion for each test item Unique test item number Description of test activity Description of expected test result which should be in accordance with target specifications Check box field for entering test result with the following alternatives Passed Failed Field for entering test observation in particular observations when the Failed box was checked Req Name Description Realization BCISWI1 BCI communication External interface that communicates with To be addressed in WP6 i e interface other subsystems using a TCP IP messages ROS protocol BCISW2 Therapist GUI The graphical user interface GUI allows the To be addressed in WP6 therapist to interact with the BCI software and WP7 i e User interface design and implementation Demonstrator development BCISW3 Subject GUI User GUI In the training and decoding process subjects It has been already are asked to perform some tasks addressed in D2 2 Further analysis will be 85 D3 2 Physical Physiological sensing devices Rev 1 0 addressed in WP6 and WP7 i e User interface design and implementation Demonstrator development the most appropriate system will be used BCISW4 BCI software portability The BCI software can run over different To be addressed in Task operating systems Windows Linux etc 3 4 Deliverable D3 4 BCISW5 EEG sensor cap size The cap is available in 3 si
122. ub section briefly describe the initialization steps of the HSS Controller at startup and when transitioning to the training session mode 2 5 10 LHSS Controller startup On startup the HSS Controller automatically starts the sensor drivers and the HSS front end as system daemons The drivers start in standby mode and scan for available sensors to collect their serial numbers status and battery levels The front end also starts in standby mode and monitors the executions status of the sensor drivers All nodes wait for the CORBYS ROS core and parameter server to be available Once the parameter server is available the HSS front end populates it with the list of available sensors and their characteristics The HSS Controller stays in this mode until a session is initialized 2 5 10 2Training session initialization In order to start a training session and publish sensor data on the CORBYS ROS topic the HSS controller needs to go through a number of steps e Discovery of the sensor within range and battery status This is continuously done in the standby mode and the set of available sensor is kept updated in the ROS parameter server 35 D3 2 Physical Physiological sensing devices Rev 1 0 e Selection of the desired sensors and connection The user of the demonstrator has to choose among the available sensors the ones to use for the training session At this point the sensor should be set up on the patient and the selection should be populate
123. und electrode was placed on FPz and the reference on the left earlobe EEG signals were sampled at 256 Hz bandpass filtered 0 5 60 Hz with a Butterworth filter of order 4 and power notch filtered at 50 Hz Vertical and horizontal EOG were acquired with the ground electrode on the right mastoid and the reference electrode on the left mastoid Fig 1b EOG signals were recorded with the gUSBamp amplifier from g Tec at a sampling frequency of 256 Hz bandpass filtered 0 5 60 Hz with a Butterworth filter of order 4 and power notch filtered at 50 Hz Two unipolar EMG electrodes were placed on both left and right sternocleidomastoid muscle as they are activated with the head rotation movements Costa et all 1990 see Figure 41c The ground electrode was placed on the left forearm and the reference electrode was placed on the right forearm EMG signals were registered with the gUSBamp amplifier from g Tec at a sampling frequency of 256 Hz bandpass filtered 0 5 100 Hz with a Butterworth filter of order 4 and power notch filtered at 50 Hz a b c Figure 41 a Scalp electrode position according to the American EEG Society 1994 b subject wearing an EEG cap with 32 active electrodes c superficial muscular electrodes EMG on the right sternocleidomastoid muscle The data recording was carried out with three g Tec amplifiers two of them for the EEG and the other one for the EMG and EOG synchronized The experiment was executed u
124. ute between series The experiment lasted 25 minutes Trial i er Trial 5s 5s 5s Trial i 1 a Ot LCi User relax avoiding User mental task User rest minimal movement and blinking condition attention or movement and blinking distraction Bike stopped Bike stopped Bike on Bike onset Figure 54 Example of a trial darker grey data processed B Data recording and mini bike 1 EEG system EEG data was recorded by a TMSi amplifier with 16 electrodes according to the 10 10 system FC3 FC1 FCz FC2 FC4 C3 C1 Cz C2 C4 CP3 CP1 CPz CP2 CP4 Pz Ground was located in FPz and the reference on the right earlobe Notice that the large majority of electrodes were situated close to Cz as it is usually the sensor where the leg motor cortical activation is observed Graimann et Pfurtscheller 2006 The EEG signal was acquired with a sampling rate of 256 Hz power line notch filtered 50 or 60 Hz and bandpass filtered from 0 5 Hz to 60 Hz The acquisition and experimental software was property of Bit amp Brain Technologies 2 Mini bike A mini stationary bike YF612 Tecnovita by BH was used to move the user s legs The angular velocity of the pedals was 2m rad s The activation of the mini bike was manually controlled by the supervisor of the experiment C Data processing For each subject the EEG data was filtered by an Independent Component Analysis ICA algorithm FastICA bas
125. vious research to the lower limbs The CORBYS project proposes a robot assisted gait rehabilitation system The objective of this deliverable is to report the progress developed by Bit amp Brain Technologies BBT to design a BCI system to decode passive lower limb movements with attention or non attention to the motor task To study the viability of this decoder an experimental setup was built where passive leg movements were performed by a mini stationary bike device while the subject was paying attention to the movement or executing a distractive task from the leg motion The study spanned the electrophysiology and the development of an off line classifier to differentiate between both conditions 65 D3 2 Physical Physiological sensing devices Rev 1 0 3 3 2 Methods A Experiment 4 right footed and one left footed healthy subjects participated in the experiment aged 20 to 28 years mean 24 4 and standard deviation 3 28 They all signed an informed consent Volunteers were seated in a comfortable chair in front of a computer screen Their feet were held to the pedals of a mini stationary bike that is motor assisted to actively move the user s legs a b Figure 53 a Mini bike and b Experimental setup The subjects were seated in a comfortable position far enough from the mini stationary bike to allow a smooth passive movement of the legs and to avoid possible contact between the feet and the floor or furniture The feet where he
126. w changes Anders Liverud 0 11 27 09 2012 HSS sections updated Marco Creatura 0 12 28 09 2012 Executive Summary Introduction and BCI section including Safety Requirements and Conclusion updated Marco Creatura 1 0 30 09 2012 Release version II D3 2 Physical Physiological sensing devices Rev 1 0 CORBYS Definition of Terms Term CORBYS Demonstrators The 1 CORBYS Demonstrator Demonstrator I Definition Mobile Robot assisted Gait Rehabilitation System The 2 CORBYS Demonstrator Demonstrator II Reconnaissance Robot for Investigation of Hazardous Environments RecoRob CORBYS Roles User CORBYS End user Any user interacting with the CORBYS systems for example in case of gait rehabilitation system users with the following roles a patient therapist or an engineer In the case of reconnaissance robot users with the following role tele operator or a hazardous area examination officer The companies entities that use exploit aspects of CORBYS technology in their commercial products or services Mobile Robotic Patient The person receiving gait rehabilitation therapy aided by the CORBYS Gait system Rehabilitation Therapist The medical professional configuring and assessing rehabilitation System Roles therapy aided by the CORBYS system Engineer A professional dealing with the CORBYS system based on a need to do technical maintenance repairs or sys
127. x34x15 mm Microcontroller software is based on the real time kernel wC OS II and optimized for low power consumption by putting microcontroller and radio into sleep mode when idle Sensors are read at regular intervals and data transmitted using the Bluetooth Smart attribute indicate operation avoiding requests from the client application To ease sensor data analysis a Java based monitoring application for recording and visualization of sensor data is developed and shown in Figure 13 hnttp www continuaalliance org products design guidelines html 18 D3 2 Physical Physiological sensing devices Rev 1 0 Le Status Connected Manufacturer SINTEF Hardware revision A Serial Number 36176 Model TRALE Firmware revision 1 0 0 Skin Temperature 25 53 C Subscribe Interval ms 2000 Read write g time 13 44 53 059 thet 12C Humidity Sensors RH Sensor1 NiA C NIA Interval ms 3000 Magnetometer A Subscribe Interval ms 200 Time 13 44 53 209 Ae Combined IMU Data Quaternion Angles deg x 14 roll 179 80 y 48 pitch 179 82 o wwz yw 41 33 w 5789 Lea v Subscribe Time hssss3350 E Battery Figure 13 A screen dump of the Java application receiving sensor data from IMU sensor unit 19 D3 2 Physical Physiological sensing devices Rev 1 0 2 3 2 I
128. y official journal of the International Federation of Clinical Neurophysiology 1 January 2002 volume 113 issue 1 Pages 43 47 Graimann B and Pfurtscheller G Quantification and visualization of event related changes in oscillatory brain activity in the time frequency domain Progress in Brain Research no 159 pp 79 97 2006 Groppe DM Makeig S Kutas M Identifying reliable independent components via split half comparison Neuroimage 2000 g Tec Technologies n d g Tec EEG system online Available at lt http www gtec at gt Accessed 20 September 2012 Gwin J T Gramann K Makeig S and Fer D P 2010 Removal of movement artifact from highdensity eeg recorded during walking and running J Neurophysiol 103 3526 3534 Hanakawa T Immisch I Toma K Dimyan MA Van Gelderen P Hallett M Functional properties of brain areas associated with motor execution and imagery J Neurophysiol 2003 Feb 89 2 989 1002 Hirasaki E Moore S Raphan T Cohen B Theodore Effects of walking velocity on vertical head and body movements during locomotion Exp Brain Res 1999 Hyv rinen A Oja E Independent Component Analysis Algorithms and Applications Neural Networks 2000 Johansen Berg H and Matthews P Attention to movement modulates activity in sensori motor areas including primary motor cortex Experimental Brain Research vol 142 pp 13 24 2002 Liverud A E Vedum J Fleurey F and Seeberg T
129. y Books 2008 Searle A and Kirkup L 2010 A direct comparison of wet dry and insulating bioelectric recording electrodes Physiological Measurement 21 2 271 283 Popescu F Fazli S Badower Y Blankertz and Mller KR 2007 Single trial classification of motor imagination using 6 dry eeg electrodes PLoS ONE 2 2 Schl gl A Keinrath C Zimmermann D Scherer R Leeb R Pfurtscheller G A fully automated correction method of EOG artifacts in EEG recordings Clin Neurophysiol 2007 Tee K P Guan C Ang K K Phua K S Wang C and Zhang H Augmenting Cognitive Processes in Robot Assisted Motor Rehabilitation Proceedings of the 2nd Biennial IEEE RAS EMBS International Conference on Biomedical Robotics and Biomechatronics pp 698 703 2008 Teplan M Fundamentals of EEG measurement Measurement Science Review 2002 TMSi Twente Medical Systems International n d EEG TMSi system online Available at lt http http www tmsi com gt Accessed 20 September 2012 Tzyy Ping Jung Colin Humphries Te Won Lee Scott Makeig Martin J Mckeown Vicente Iragui and Terrence J Sejnowski Extended ICA removes artifacts from electroencephalographic recordings In NIPS 97 1998 MIT Press Van de Velde M van Erp G Cluitmans PJ Detection of muscle artefact in the normal human awake EEG Electroencephalogr Clin Neurophysiol 1998 Aug 107 2 149 58 95
130. y Teer ere ty vere rr yr eter erers Tr vr TTT eT ere TT TTT er Install Ubuntu 10 04 LTS from the iso image Install all updates Install packets for git svn ssh Java and Maven2 for hss controller JAVA debug UI sudo apt get install subversion default jdk eclipse openssh server maven2 git core EN AAAA AAA PAA AAEE AAAA ENAA AA E AIEA lala aa ade Insta ROS FEE E T TTT A AAAA A A A A A A A A A AA A AAAA EEEE EEIEIEE TEETE IEIET HEHEHE sudo sh c echo deb http packages ros org ros ubuntu lucid main gt etc apt sources list d ros latest list wget http packages ros org ros key O sudo apt key add sudo apt get update sudo apt get install ros electric desktop ful1 echo source opt ros electric setup bash gt gt bashrc bashre mkdir ros _workspace echo export ROS_PACKAGE_PATH ros_workspace ROS_PACKAGE_PATH gt gt bashrc echo export ROS_WORKSPACE ros_workspace gt gt bashrc bashre echo ROS_PACKAGE_PATH Se eee a er ee eat one Insta QT 4 8 1 HARA HE HE a a a a a a a a Download QtSdk online 1inux x86 v1 2 1 run from QT web page my a a en ey Meee nee geet opt cd opt chmod u x QtSdk online 1inux x86 v1 2 1 run QtSdk online 1inux x86 v1 2 1 run pT GN AATA AA AEEA A ATAA NAA i EA E a leh ial Lida Insta QWT HARA A aE AE HE EE AEE HE EEE aE a Ea a a a a a a a a a Download qwt 6 0 1 zip from qwt web page cd opt cp home hsscontro ler Down loads qwt 6 0 1
131. ystems in two different experiments The first experiment was developed under a controlled situation where a DC brush motor was used as proposed by related partner developing mobile platform the second experiment involved a less controlled scenario where the source of electromagnetic noise was caused by a 45 D3 2 Physical Physiological sensing devices Rev 1 0 generic AC motor Data recording Both experiments used the same data recording setup Three subjects participated in both experiments EEG signals were recorded simultaneously from g Tec and TMSi systems Data acquired from g Tec system was recorded from 10 active electrodes placed F3 Fz F4 C3 Cz C4 P3 Pz P4 and Oz according to the 10 10 system Ground and reference electrodes were placed on FPz and on the right earlobe respectively Data acquired from TMSi equipment was recorded from 10 water electrodes placed next to g Tec electrodes at the following locations AFF3 AFFz AFF4 FOC3 FOCz FOC4 OCP3 OCPz OCP4 and POOz according to the 10 5 system see Figure 30 Tmsi gTec x AFF3 AFFz AFFs BR kz fe e FOC3 FOCz FOC4 G oS d oe al OcP3 OCPz OCPs o Pa Pa he POOz e Oz Figure 30 On the Left EEG montage of gel electrodes in blue g Tec and water electrodes in green TMSi On the right picture of real montage gel electrodes red and water electrodes white Electrodes were placed over a g Tec cap according to t
132. zes small medium Requirement that needs and large the most appropriate one needs to always to be accomplished be chosen depending on the subject head since the quality of the circumference Anyway the medium sized cap Signal depend on it is suitable for over 95 of all adult subjects In the present deliverable it was accomplished BCISW6 EEG Electrodes location The EEG electrodes are inserted via small In this deliverable different and number holes in the cap Their position on the scalp EEG setup has been used indicated on the cap according to the extended depending on the task to international 10 20 system and number focus on This ISSUE will be depends on what brain areas are activated gadr arsed in detai during i T the development of Task 3 3 during a specific cognitive task Ongoing and 3 5 Deliverable D3 4 CORBYS research will identify those Tasks 3 3 and 3 5 BCISW7 Subject screen User The graphical user interface GUI displays To be addressed in WP7 screen commands to the subject e g a visual cue i e Demonstrator indicating that the subject has to start development walking BCISW8 Therapist screen The graphical user interface GUI allows the To be addressed in WP6 therapist to interact with the BCI software and WP7 i e User interface design and implementation Demonstrator development BCISW9 BCI processing unit The minimum computing power needed To be addressed in Task depend on the re
133. zip unzip qwt 6 0 1 zip cd qwt 6 0 1 sudo make install sudo su The following is required for the CORBYS_GUI to run echo usr local qwt 6 0 1 lib gt etc ld so conf d qwt conf ldconfig HEE EEE AE EAE EAE EEE AEE EEA A AAA A AAA A AE A A A A Checkout CORBYS SVN CODE HEE EE EE EAE EAE AEE AEE EE AEA AEA AE AAA A A AE svn checkout https corbys eu svn CORBYS Figure 17 HSS Controller Installation Script 2 5 4 HSS Controller Software This section describes the drivers and software parts of the HSS controller which makes the link between the sensor units and the CORBYS general purpose network Figure 18 presents a high level view of the architecture of the HSS Controller software The HSS software is composed of a set of driver components on the left hand side of Figure 18 which manage the connection with individual sensors and of a front end component which provides the interface of the HSS controller on the CORBYS GPN The interface of the HSS controller is made of 3 ROS topics which have been defined in collaboration with other CORBYS modules The Config Data topic is a common topic for all modules and allows distributing the configuration of the CORBYS system to all its modules The Heart Beat topic is also used by all modules to publish status information at regular intervals Finally the Sensor_Data_HSS topic is specific to the HSS Controller and used to publish the sensor data 25 D3 2 Physical Physiological sensing d

D-3-2-CORBYS-Physical-Physiological-sensing-devices

Contents

Download Pdf Manuals

Related Search

Related Contents